Lab Overview Presentation

To get a quick overview of the work we have done in this lab within the past five years, take a look at our lab presentation. We update it continuously!

Recent & Upcoming Activities

• April 2020: At ECIR 2020 Lambda-Lab will be present with two full papers (Curriculum Learning Strategies for IR: An Empirical Study on Conversation Response Ranking led by Gustavo Penha and Diagnosing BERT with Retrieval Heuristics led by Arthur Câmara) and a presentation of a 2019 accepted Information Retrieval Journal article.
  
  
• March 2020: Nirmal Roy will present his short paper Exploring Users’ Learning Gains within Search Sessions at CHIIR 2020.
  
  
• December 2019: we will present our edX Log File Analysis Tool (ELAT) at the LAK 2020 conference! It requires no server, no programming knowledge and has no setup costs. Fully browser-based! Take a look at ELAT. The paper preprint is available here.
  
  
• November 2019: Sara Salimzadeh and Peide Zhu join Lambda-Lab working on search for complex domains; Sara is also a member of the AI for Fintech Lab (an ICAI lab hosted at TU Delft).

• September 2019: PhD student Nirmal Roy joins Lambda-Lab, working in the area of search as learning.
  
  
• August 2019: A couple of Lambda-Lab shirts made their way to our members!
  
  
• July 2019: Three members of Lambda-Lab (Arthur, Gustavo, Claudia) are at ACM SIGIR 2019 in Paris!
  
  
• May 2019: Guanliang Chen (MOOC Analytics: Learner Modeling and Content Generation) and Dan Davis (Large-Scale Learning Analytics: Modeling Learner Behavior & Improving Learning Outcomes in Massive Open Online Courses) defend their PhD theses.

• April 2019: Three members of Lambda-Lab (Arthur, Gustavo, Claudia) are at the 41st European Conference on Information Retrieval, presenting a demo on node-indri and a long paper on axiomatic IR for neural models.
  
  
• April 2019: Yue Zhao defends his PhD thesis on Learning Analytics Technology to Understand Learner Behavioral Engagement in MOOCs.

• February 2019: Dimitrios Bountouridis is at FAT* 2019, presenting our work on SIREN, a prototype to understand the effects of recommender systems in online news environments (check the blog post on the paper).

Why the Name?

It is tradition in our research group to form teams around Greek letters, usually a letter that is associated with a major topic of the team. And thus we picked Lambda to highlight Learning. Sigma (which would have been another obvious choice due to Search) was already taken.

Research Focus

Lambda-Lab brings together research expertise in data science and information retrieval (IR). Core IR topics (such as collaborative search, conversational search and data-hungry ranking models) are investigated as well as applications of data science and IR to application domains, most prominently the domain of Massive Open Online Learning. TU Delft is one of the largest MOOC providers in Europe and as such offers a unique research environment to investigate and explore human learning in a computational and large-scale manner. With more than 80 TU Delft MOOCs running on the edX platform and more than 2 million registered learners (end of 2018), we have a rare opportunity to empirically explore and evaluate learning interventions (which are often designed around learners' searching for information).

Researchers Involved

Several researchers of the WIS group contribute to this research line:

• Claudia Hauff is an Associate Professor at WIS and the lab leader of Lambda-Lab. She is the daily supervisor of all MSc and PhD projects within this research line. Contact her via c.hauff@tudelft.nl with any questions about Lambda-Lab.
• Felipe Moraes' PhD is financed by NWO (TOP2 grant) and works on collaborative search questions in the setting of MOOCs.
• Arthur Câmara's PhD is financed by NWO (VIDI grant) and focuses on search for learning, with a special focus on neural IR models.
• Gustavo Penha's PhD is financed by NWO (VIDI grant) and zooms in on the problem of conversational search in the context of MOOCs.
• Nirmal Roy's PhD revolves around search as learning.
• Sara Salimzadeh's PhD revolves around search in complex domains and is financed by the ICAI Fintech lab.
• Peide Zhu's PhD revolves around conversational search / search as learning.

Past members include:

• Dan Davis (PhD student, 2015-2019, supported by CEL) researched the impact of the online learning environment on learners and learning. His PhD thesis is available here.
• Guanliang Chen (PhD student, 2015-2019, supported by TU Delft's Extension School) explored data sources beyond the MOOC platforms themselves to explore learner modeling and content generation. His PhD thesis can be found here.
• Yue Zhao (PhD student, 2014-2019, supported by a CSC grant) employed data analytics to make sense of MOOC learners' behaviour. His PhD thesis can be found here.
• Mónica Marrero (research engineer, 2017-2018).
• Dimitrios Bountouridis (postdoc, 2018-19) worked on fairness and transparency in the context of news recommenders.

Lambda-Lab is also the host of a number of Master students:

• Alex Balan investigated conversational search problems (thesis defended in December 2019).
• Francisco Morales (thesis defended in February 2020) and Wanning Yang (thesis defended in December 2019) investigated the use of IR technologies for NLP problems.
• Kun Jiang's thesis (defended in August 2019) was on question answering and how to improve the first step of the QA pipeline (i.e. the retrieval system).
• Nirmal Roy explored word embeddings and how to use them to improve ad-hoc retrieval (thesis defended in August 2019).
• Daan Rennings innvestigated the axiomatic approach to information retrieval, specifically for neural IR models. His thesis led to an ECIR paper and was defended in April 2019.
• Kilian Grashoff explored the impact of group size on collaborative search. He defended his thesis in January 2019, and contributed to a DESIRES 2018 paper and an Information Retrieval Journal article (2019).
• Sindu Sindunuraga Rikarno Putra investigated to what extent search is a viable alternative to learning from MOOC videos. He defended his thesis in August 2018. Sindu's thesis work contributed to a SIGIR 2018 demo, a DESIRES 2018paper and a full paper at CIKM 2018.
  
• Ioana Jivet explored how to increase self-regulation in MOOC learners through the use of an interactive learning tracker. She defended her thesis in September 2016.  Currently, Ioana is working as a PhD student at the Welten-instituut. Her thesis work contributed to a full paper at LAK 2017.
• Yingying Bao investigated the prevalence of cheating in MOOCs. She defended her thesis in February 2017. Her thesis work contributed to a short paper at EDM 2017.
• Jochem de Goede designed a workbench to increase the reproducibility of MOOC experiments.
• Sambit Praharaj collaborated with researchers from Lugano to bring data analytics to the higher-education classroom. His thesis work contributed to a full paper at UMAP 2017.

We also collaborate on projects/papers with external researchers:

• Vassileios Triglianos and Cesare Pautasso, University of Lugano.
• Tarmo Robal, Tallinn University of Technology.
• René Kizilcec, Stanford University.
• Markus Krause, Telefonica.

Software / datasets

We are regularly producing software artifacts and datasets in our lab. Here we list the more mature contributions we have made:

• MANtIS is a multi-domain information seeking dialogues dataset derived from Stack Exchange.
• ELAT is a edX Log File Analysis Tool that runs completely in your browser. No server, no setup costs, no programming knowledge required.
• SearchX is a scalable collaborative search system, developed to research large-scale search and sensemaking experiments.
• APONE is an academic environment for online experiments. We have used in our research and teaching.
• SIREN is a simulation framework for understanding the effects of recommender systems in online news environment.

Publications

+++ 2019 +++

Ryan Clancy, Nicola Ferro, Claudia Hauff, Jimmy Lin, Tetsuya Sakai, Ze Zhong Wu:
The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019). SIGIR 2019: 1432-1434

Felipe Moraes, Kilia Grashoff and Claudia Hauff:
On the Impact of Group Size on Collaborative Search Effectiveness.
Accepted at the Information Retrieval Journal.

Felipe Moraes, Claudia Hauff:
node-indri: Moving the Indri Toolkit to the Modern Web Stack. ECIR (2) 2019: 241-245
Best Demo Award.

Daniël Rennings, Felipe Moraes, Claudia Hauff:
An Axiomatic Approach to Diagnosing Neural IR Models. ECIR (1) 2019: 489-503

Dimitrios Bountouridis, Jaron Harambam, Mykola Makhortykh, Mónica Marrero, Nava Tintarev, Claudia Hauff:
SIREN: A Simulation Framework for Understanding the Effects of Recommender Systems in Online News Environments. FAT 2019: 150-159

+++ 2018 +++

Dan Davis, Claudia Hauff, Geert-Jan Houben:
Evaluating Crowdworkers as a Proxy for Online Learners in Video-Based Learning Contexts. PACMHCI 2(CSCW): 42:1-42:16 (2018)

Felipe Moraes, Sindunuraga Rikarno Putra, Claudia Hauff:
Contrasting Search as a Learning Activity with Instructor-designed Learning. CIKM 2018: 167-176

Sindunuraga Rikarno Putra, Kilian Grashoff, Felipe Moraes, Claudia Hauff:
On the Development of a Collaborative Search System. DESIRES 2018: 76-82

Yue Zhao, Tarmo Robal, Christoph Lofi, Claudia Hauff:
Can I Have a Mooc2Go, Please? On the Viability of Mobile vs. Stationary Learning. EC-TEL 2018: 101-115

Dan Davis, Vasileios Triglianos, Claudia Hauff, Geert-Jan Houben:
SRLx: A Personalized Learner Interface for MOOCs. EC-TEL 2018: 122-135

Dan Davis, Guanliang Chen, Claudia Hauff, Geert-Jan Houben:
Activating learning at scale: A review of innovations in online learning strategies. Computers & Education 125: 327-344 (2018)

Guanliang Chen, Dan Davis, Markus Krause, Efthimia Aivaloglou, Claudia Hauff, Geert-Jan Houben:
From Learners to Earners: Enabling MOOC Learners to Apply Their Skills and Earn Money in an Online Market Place. TLT 11(2): 264-274 (2018)

Guanliang Chen, Claudia Hauff, Geert-Jan Houben:
Feature Engineering for Second Language Acquisition Modeling. BEA@NAACL-HLT 2018: 356-364

Tarmo Robal, Yue Zhao, Christoph Lofi, Claudia Hauff:
IntelliEye: Enhancing MOOC Learners' Video Watching Experience through Real-Time Attention Tracking. HT 2018: 106-114

Guanliang Chen, Jie Yang, Claudia Hauff, Geert-Jan Houben:
LearningQ: A Large-Scale Dataset for Educational Question Generation. ICWSM 2018: 481-490

Tarmo Robal, Yue Zhao, Christoph Lofi, Claudia Hauff:
Webcam-based Attention Tracking in Online Learning: A Feasibility Study. IUI 2018: 189-197

Dan Davis, René F. Kizilcec, Claudia Hauff, Geert-Jan Houben:
The half-life of MOOC knowledge: a randomized trial evaluating knowledge retention and retrieval practice in MOOCs. LAK 2018: 1-10

Dan Davis, Daniel Seaton, Claudia Hauff, Geert-Jan Houben:
Toward large-scale learning design: categorizing course designs in service of supporting learning outcomes. L@S 2018: 4:1-4:10

Sindunuraga Rikarno Putra, Felipe Moraes, Claudia Hauff:
SearchX: Empowering Collaborative Search Research. SIGIR 2018: 1265-1268

Mónica Marrero, Claudia Hauff:
A/B Testing with APONE. SIGIR 2018: 1269-1272

Yue Zhao, Tarmo Robal, Christoph Lofi, Claudia Hauff:
Stationary vs. Non-stationary Mobile Learning in MOOCs. UMAP (Adjunct Publication) 2018: 299-303

+++ 2017 +++

Yue Zhao, Christoph Lofi, Claudia Hauff:
Scalable Mind-Wandering Detection for MOOCs: A Webcam-Based Approach. EC-TEL 2017: 330-344

Yingying Bao, Guanliang Chen, Claudia Hauff:
On the Prevalence of Multiple-Account Cheating in Massive Open Online Learning. EDM 2017

Dan Davis, Ioana Jivet, René F. Kizilcec, Guanliang Chen, Claudia Hauff, Geert-Jan Houben:
Follow the successful crowd: raising MOOC completion rates through social comparison at scale. LAK 2017: 454-463

Guanliang Chen, Dan Davis, Markus Krause, Claudia Hauff, Geert-Jan Houben:
Buying time: enabling learners to become earners with a real-world paid task recommender system. LAK 2017: 578-579

Yue Zhao, Dan Davis, Guanliang Chen, Christoph Lofi, Claudia Hauff, Geert-Jan Houben:
Certificate Achievement Unlocked: How Does MOOC Learners' Behaviour Change? UMAP (Adjunct Publication) 2017: 83-88

Vasileios Triglianos, Sambit Praharaj, Cesare Pautasso, Alessandro Bozzon, Claudia Hauff:
Measuring Student Behaviour Dynamics in a Large Interactive Classroom Setting. UMAP 2017: 212-220

+++ 2016 +++

Dan Davis, Guanliang Chen, Tim Van der Zee, Claudia Hauff, Geert-Jan Houben:
Retrieval Practice and Study Planning in MOOCs: Exploring Classroom-Based Self-regulated Learning Strategies at Scale. EC-TEL 2016: 57-71
Best Student Paper Award.

Vasileios Triglianos, Cesare Pautasso, Alessandro Bozzon, Claudia Hauff:
Inferring Student Attention with ASQ. EC-TEL 2016: 306-320

Dan Davis, Guanliang Chen, Claudia Hauff, Geert-Jan Houben:
Gauging MOOC Learners' Adherence to the Designed Learning Path. EDM 2016: 54-61

Guanliang Chen, Dan Davis, Claudia Hauff, Geert-Jan Houben:
Learning Transfer: Does It Take Place in MOOCs? An Investigation into the Uptake of Functional Programming in Practice. L@S 2016: 409-418
Best Paper Nominee.

Guanliang Chen, Dan Davis, Jun Lin, Claudia Hauff, Geert-Jan Houben:
Beyond the MOOC platform: gaining insights about learners from the social web. WebSci 2016: 15-24