Adaptive Learning Systems in Programming Education: A Prototype for Enhanced Formative Feedback - Lohr, Dominic; Berges, Marc-Pascal; Chugh, Abhishek; Striewe, Michael (Proceedings of DELFI 2024, 2024)
Formative feedback is crucial in programming education, yet many learning systems fall short, concentrating mostly on pinpointing errors rather than guiding learners on how to resolve them. This is particularly unhelpful for novices who often lack advanced skills like debugging. Feedback is considered more valuable when it addresses error causes rather than just symptoms. However, this is challenging using only conventional methods like unit testing. Identifying error causes requires detailed information about both the error and the learner. Our proposed prototype introduces a new approach to integrating programming exercises into adaptive learning systems. It directly categorizes student code into so-called answer classes using a combination of static and dynamic code analysis. When integrated with data derived from a learner model, this approach enables tailored feedback that lowers the barrier to learning programming while keeping motivation hig
Keywords: programming education; adaptive feedback; CS1

ALeA : Advancing Personalized Learning with Adaptive Assistance and Semantic Annotation - Grelka, Felix; Lohr, Dominic; Berges, Marc-Pascal (Workshop KI und Bildung 2024, 2025)

Efficient Exam Correction at Scale: Streamlining Paper-Based Assessments with the VoLl-KOrN System - Betzendahl, Jonas; Lohr, Dominic; Berges, Marc-Pascal; Kohlhase, Michael (Digitales Lehren und Lernen an der Hochschule Strategien - Bedingungen - Umsetzung, 2025)
This paper introduces VoLl-KOrN – a tool designed for digitizing and optimizing paper-based examination processes at universities. By utilizing QR codes on exam papers, VoLl-KOrN allows for submissions to be scanned and automatically assigned to students. The web interface allows educators to view, organize, and assess scanned exams. Its innovation lies in its use of answer classes that cluster similar responses, allowing automatic or manual scoring with a focus on objectivity and consistency. Additionally, the system enables a systematic analysis of exam data to improve teaching strategies and identify learning gaps. This enables further opportunities like interconnecting with learning platforms to, e.g., update a learner model based on exam performanc
Keywords: Assessment; Answer classes; learning analytics; educational technology; digital teaching; learner model

Exploring the Potential of Large Language Models to Generate Formative Programming Feedback - Kiesler, Natalie; Lohr, Dominic; Keuning, Hieke (Proceedings of the 2023 IEEE ASEE Frontiers in Education Conference, 2023)
Ever since the emergence of large language models (LLMs) and related applications, such as ChatGPT, its performance and error analysis for programming tasks have been subject to research. In this work-in-progress paper, we explore the potential of such LLMs for computing educators and learners, as we analyze the feedback it generates to a given input containing program code. In particular, we aim at (1) exploring how an LLM like ChatGPT responds to students seeking help with their introductory programming tasks, and (2) identifying feedback types in its responses. To achieve these goals, we used students' programming sequences from a dataset gathered within a CS1 course as input for ChatGPT along with questions required to elicit feedback and correct solutions. The results show that ChatGPT performs reasonably well for some of the introductory programming tasks and student errors, which means that students can potentially benefit. However, educators should provide guidance on how to use the provided feedback, as it can contain misleading information for novices.<br
Keywords: ChatGPT; large language models; feedback; feedback types; introductory programming

Introducing a self-study course for learning textual programming at highschool with APFEL and ALeA - Lohr, Dominic; Wechsler, Lars; Berges, Marc-Pascal (Workshop KI und Bildung 2024, 2025)

Learning Support Systems based on Mathematical Knowledge Management - Berges, Marc-Pascal; Betzendahl, Jonas; Chugh, Abhishek; Kohlhase, Michael; Lohr, Dominic; Müller, Dennis (Lecture Notes in Computer Science, 2023)
To cater to the increasingly diverse student bodies, higher education has to personalize education. In times of stagnant educational budgets and staffing problems, this can only be achieved via adaptive, interactive learning support services. In this paper we show how these can be generated by modeling the domain, the learner competencies, and the rhetoric and didactic relations among learning objects, re-using existi

Learning with ALeA: Tailored experiences through annotated course material - Kruse, Theresa; Berges, Marc-Pascal; Betzendahl, Jonas; Kohlhase, Michael; Lohr, Dominic; Müller, Dennis (INFORMATIK 2023. Designing Futures: Zukünfte gestalten, 2023)
We present ALeA, an adaptive learning assistant for university courses. The intelligent tutoring system (ITS) provides services for both, learners and instructors, based on semantically annotated course material. The system can generate learning tools like flashcards, guided tours, and quizzes, which can be tailored to individual learning progress. Currently, the system is only available for courses from the field of computer science, like artificial intelligence. We plan to expand it for further courses from different field
Keywords: learning assistant; self-learning; learning analytics; adaptive learning

„Let Them Try to Figure It Out First“ – Reasons Why Experts (Do Not) Provide Feedback to Novice Programmers - Lohr, Dominic; Kiesler, Natalie; Keuning, Hieke; Jeuring, Johan (Proceedings of the 29th ACM Conference on on Innovation and Technology in Computer Science Education, 2024)
A recent ITiCSE working group investigated when and how experts give feedback and hints at steps novice programmers take when solving programming problems. Based on the feedback literature and an analysis of expert feedback on steps, the working group designed guidelines for when and how to give feedback. The feedback provided by educators using these guidelines on a number of sequences of student steps varied a lot. In this paper, we try to answer the question of why educators give feedback at particular steps to novice learners of programming. We prepared six authentic sequences of student steps when solving an introductory programming task. The preprocessed sequences were used in a survey to gather information about when and why an expert would give feedback. Respondents annotated each step from one sequence with if and why they would give feedback at that step. Our survey received 47 responses. We qualitatively analyzed the responses, resulting in a coding scheme consisting of 19 different reasons for why experts intervene (or not) when novice learners work on introductory programming tasks. We found a considerable variety of reasons experts give for when and how to help students with feedback and hints. Also, sometimes one expert uses a reason at a step to explain why they do intervene, and another expert uses the same reason at the step to not intervene. The categories of experts' feedback indicators will pave the way for several future studies and applications, including learning systems trying to resemble expert feedback strategies.<br
Keywords: Learning programming; expert feedback; feedback rationale; feedback guidelines; novice programmers

Leveraging Large Language Models to Generate Course-specific Semantically Annotated Learning Objects - Lohr, Dominic; Berges, Marc-Pascal; Chugh, Abhishek; Kohlhase, Michael; Müller, Dennis (Journal of Computer Assisted Learning, 2025)
Background: Over the past few decades, the process and methodology of automated question generation (AQG) have undergone significant transformations. Recent progress in generative natural language models has opened up new potential in the generation of educational content. Objectives: This paper explores the potential of large language models (LLMs) for generating computer science questions that are sufficiently annotated for automatic learner model updates, are fully situated in the context of a particular course, and address the cognitive dimension understand. Methods: Unlike previous attempts that might use basic methods like ChatGPT, our approach involves more targeted strategies such as retrieval-augmented generation (RAG) to produce contextually relevant and pedagogically meaningful learning objects. Results and Conclusions: Our results show that generating structural, semantic annotations works well. However, this success was not reflected in the case of relational annotations. The quality of the generated questions often did not meet educational standards, highlighting that although LLMs can contribute to the pool of learning materials, their current level of performance requires significant human intervention to refine and validate the generated conten
Keywords: automated question generation; computer science; generative AI; GPT-4; large language models; retrieval-augmented generation

Steps Learners Take when Solving Programming Tasks, and How Learning Environments (Should) Respond to Them - Jeuring, Johan; Keuning, Hieke; Marwan, Samiha; Bouvier, Dennis; Izu, Cruz; Kiesler, Natalie; Lehtinen, Teemu; Lohr, Dominic; Petersen, Andrew; Sarsa, Sami (Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 2, 2022)
Every year, millions of students learn how to write programs. Learning activities for beginners almost always include programming tasks that require a student to write a program to solve a particular problem. When learning how to solve such a task, many students need feedback on their previous actions, and hints on how to proceed. In the case of programming, the feedback should take the steps a student has taken towards implementing a solution into account, and the hints should help a student to complete or improve a possibly partial solution. Only a limited number of learning environments for programming give feedback and hints on intermediate steps students take towards a solution, and little is known about the quality of the feedback provided. To determine the quality of feedback of such tools and to help further developing them, we create and curate data sets that show what kinds of steps students take when solving programming exercises for beginners, and what kind of feedback and hints should be provided. This working group aims to 1) select or create several data sets with steps students take to solve programming tasks, 2) introduce a method to annotate students' steps in these data sets, 3) attach feedback and hints to these steps, 4) set up a method to utilize these data sets in various learning environments for programming, and 5) analyse the quality of hints and feedback in these learning environments.<br
Keywords: automated; learning programming; tutoring systems

Term Extraction for Domain Modeling - Kruse-Kurbach, Theresa; Lohr, Dominic; Berges, Marc-Pascal; Kohlhase, Michael; Moghbeli Damaneh, Halimeh; Schütz, Marcel (Proceedings of the 22. Fachtagung Bildungstechnologien (DELFI), 2024)
Addressing learners' individual needs in large lectures is challenging, especially if we want to tailor teaching materials, learning interventions, and feedback to individual students or sub-cohorts with special needs and educational biographies. AI-based adaptive learning systems (ALS) can help. Given a fine-grained domain model, we work on a system that can maintain a learner model and use it to generate learner-adaptive course materials: targeted explanations, flashcards, or quizzes from collections of learning objects that reference the domain model. In this system, the domain model consists of a knowledge graph of theories, each introducing a set of concepts and their definitions, properties, and relations. The conceptual and semantic relations – i.e., the terminological dependency relation between concepts – in the knowledge graph, together with a model of cognitive processes, shape the learner model. In our experience, given sufficient student activity in the system, the quality and coverage of the domain model are key determining factors for the quality of the learner-adaptivity of the system. The domain model is essential to the system and requires a large investment. Concept definitions and statements of properties and relations can be taken from course material or textbooks. In other words, the development of domain models should be computer-assisted, not only for quality assurance but also for efficiency reasons. In this paper, we present and evaluate experiments on using natural language processing (NLP) techniques, in particular term extraction, to generate term lists for domain models. We focus on the following research question: To what extent can automatic keyword extraction support domain model creation? To answer this question, we extract terms from an introductory computer science course to compare the results of different automatic term extraction tools with those of manual term extraction.<br
Keywords: domain modeling; term extraction; adaptive learning system; programming

The Potential of Answer Classes in Large-scale Written Computer-Science Exams - Lohr, Dominic; Berges, Marc-Pascal; Kohlhase, Michael; Rabe, Florian (Hochschuldidaktik Informatik (HDI) 2021, 2023)
Students’ answers to tasks provide a valuable source of information in teaching as they result from applying cognitive processes to a learning content addressed in the task. Due to steadily increasing course sizes, analyzing student answers is frequently the only means of obtaining evidence about student performance. However, in many cases, resources are limited, and when evaluating exams, the focus is solely on identifying correct or incorrect answers. This overlooks the value of analyzing incorrect answers, which can help improve teaching strategies or identify misconceptions to be addressed in the next cohort. In teacher training for secondary education, assessment guidelines are mandatory for every exam, including anticipated errors and misconceptions. We applied this concept to a university exam with 462 students and 41 tasks. For each task, the instructors developed answer classes – classes of expected responses, to which student answers were mapped during the exam correction process. The experiment resulted in a shift in mindset among the tutors and instructors responsible for the course: after initially having great reservations about whether the significant additional effort would yield an appropriate benefit, the procedure was subsequently found to be extremely valuable.
Keywords: computer science education; task analysis; answer classes

The Y-Model - Formalization of Computer-Science Tasks in the Context of Adaptive Learning Systems - Lohr, Dominic; Berges, Marc-Pascal; Kohlhase, Michael; Müller, Dennis; Rapp, Max (IEEE German Education Conference (GeCon), 2023)
Students' answers to tasks provide a valuable source of information in teaching as they result from applying cognitive processes to a learning content addressed in the task. Due to steadily increasing course sizes, analyzing student answers is frequently the only means of obtaining evidence about student performance. However, in many cases, resources are limited, and when evaluating exams, the focus is solely on identifying correct or incorrect answers. This overlooks the value of analyzing incorrect answers, which can help improve teaching strategies or identify misconceptions to be addressed in the next cohort. In teacher training for secondary education, assessment guidelines are mandatory for every exam, including anticipated errors and misconceptions. We applied this concept to a university exam with 462 students and 41 tasks. For each task, the instructors developed answer classes – classes of expected responses, to which student answers were mapped during the exam correction process. The experiment resulted in a shift in mindset among the tutors and instructors responsible for the course: after initially having great reservations about whether the significant additional effort would yield an appropriate benefit, the procedure was subsequently found to be extremely valuable
Keywords: computer science education; task analysis; answer classes

Towards Criteria for Valuable Automatic Feedback in Large Programming Classes - Lohr, Dominic; Berges, Marc-Pascal (Hochschuldidaktik Informatik (HDI) 2021, 2021)
Automatically generated feedback systems and assessments are popular tools to deal with the rapidly increasing number of students, especially in STEM-related subjects such as Computer Science.However, in the development of such systems, the focus is often on compensating resource issues only, and the quality of the systems’ feedback is not considered. To determine what kind of feedback participants of large introductory programming courses (CS1) use and what expectations they have regarding effective feedback on programming difficulties, a survey was conducted in a large introductory programming class (300 students). The results show that during online semester, students made less use of possibility to ask tutors questions in lab sessions. This is in line with statements of experienced tutors who were previously interviewed. Moreover, the results show that it is not sufficient to merely point to errors in program code but that students need more detailed information concerning the underlying cause of the error, especially in automated system
Keywords: Automatic feedback; CS1; programming education

Towards Giving Timely Formative Feedback and Hints to Novice Programmers - Jeuring, Johan; Keuning, Hieke; Marwan, Samiha; Bouvier, Dennis; Izu, Cruz; Kiesler, Natalie; Lehtinen, Teemu; Lohr, Dominic; Peterson, Andrew; Sarsa, Sami (Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE, 2022)
Every year, millions of students learn how to write programs. Learning activities for beginners almost always include programming tasks that require a student to write a program to solve a particular problem. When learning how to solve such a task, many students need feedback on their previous actions, and hints on how to proceed. For tasks such as programming, which are most often solved stepwise, the feedback should take the steps a student has taken towards implementing a solution into account, and the hints should help a student to complete or improve a possibly partial solution. This paper investigates how previous research on feedback is translated to when and how to give feedback and hints on steps a student takes when solving a programming task. We have selected datasets consisting of sequences of steps students take when working on a programming problem, and annotated these datasets at those places at which experts would intervene, and how they would intervene. We have used these datasets to compare expert feedback and hints to feedback and hints given by learning environments for programming. Although we have constructed extensive guidelines on when and how to give feedback, we observed plenty of disagreement between experts. We also found several differences between feedback given by experts and learning environments. Experts intervene at specific moments, while in learning environments students have to ask for feedback themselves. The contents of feedback is also different; experts often give (positive) feedback on subgoals, which is not supported by most environments.
Created from Fastlane, Scopus look-up
Keywords: automated feedback; feedback and hints; learning environments; learning programming; sequences of programming steps

Von Autonomem Fahren bis Zahnarzt - Vorstellungen von Schüler:innen zu Künstlicher Intelligenz und ihre Integration in den Informatikunterricht - Lindner, Annabel; Müller-Unterweger, Michaela; Löffler, Patrick; Lohr, Dominic; Berges, Marc-Pascal (Informatikunterricht zwischen Aktualität und Zeitlosigkeit, 2023)
Der vorliegende Beitrag beschreibt die Erhebung von Schüler:innenvorstellungen zum Thema Künstliche Intelligenz (KI) im Rahmen der didaktischen Rekonstruktion des Themenfeldes für den schulischen Informatikunterricht. Zudem werden Ansätze zur Integration der ermittelten Konzepte und Fehlvorstellungen im Unterricht durch angemessene Beispielkontexte und Aktivitäten illustriert. Anhand einer offenen Fragestellung wurden Ideen und Vorstellungen von Schüler:innen der 9. und 10. Jahrgangsstufe des Gymnasiums zu KI erhoben und qualitativ analysiert, um zentrale Anknüpfungspunkte für die curriculare Behandlung des Themenfeldes zu identifizieren. Die Konzepte wurden anschließend mit den bereits in der Literatur identifizierten Aspekten in Beziehung gesetzt. Dabei zeigten sich starke Überschneidungen mit den bereits vorliegenden Forschungsergebnissen, was zur Konsolidierung und Generalisierung dieser Ergebnisse beiträgt. Unter anderem nehmen die Schüler:innen selbständiges Lernen und Handeln als zentrale Charakteristika von KI-Systemen wahr und beschreiben komplexe Informatiksysteme, die ihnen überwiegend aus dem Alltag bekannt sind. Basierend auf den Untersuchungsergebnissen werden mögliche thematische Kontexte, Aktivitäten und didaktische Implikationen für das Unterrichten des Themenfeldes diskutiert.<br
Keywords: Schüler:innenvorstellungen; Künstliche Intelligenz; DidaktischeRekonstruktion; Lehrplan

You’re (Not) My Type - Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks? - Lohr, Dominic; Keuning, Hieke; Kiesler, Natalie (Journal of Computer Assisted Learning, 2025)
Background: Feedback as one of the most influential factors for learning has been subject to a great body of research. It plays a key role in the development of educational technology systems and is traditionally rooted in deterministic feedback defined by experts and their experience. However, with the rise of generative AI and especially Large Language Models (LLMs), we expect feedback as part of learning systems to transform, especially for the context of programming. In the past, it was challenging to automate feedback for learners of programming. LLMs may create new possibilities to provide richer, and more individual feedback than ever before. Objectives: This paper aims to generate specific types of feedback for introductory programming tasks using LLMs. We revisit existing feedback taxonomies to capture the specifics of the generated feedback, such as randomness, uncertainty, and degrees of variation. Methods: We iteratively designed prompts for the generation of specific feedback types (as part of existing feedback taxonomies) in response to authentic student programs. We then evaluated the generated output and determined to what extent it reflected certain feedback types. Results and Conclusion: The present work provides a better understanding of different feedback dimensions and characteristics. The results have implications for future feedback research with regard to, for example, feedback effects and learners' informational needs. It further provides a basis for the development of new tools and learning systems for novice programmers including feedback generated by A
Keywords: feedback; introductory programming; generative AI; GenAI; ChatGPT; GPT-4; large language models; feedback classification; feedback types