A Solution for Automated Grading of QDA Homework - Kaufmann, Andreas; Riehle, Dirk; Krause, Julia; Harutyunyan, Nikolay (HICSS Proceedings, 2023)
Teaching research methods is important in any curriculum that prepares students for an academic career. While theoretical frameworks for qualitative theory building can be adequately conveyed through lecturing, the practices of qualitative data analysis (QDA) cannot. However, using experiential learning techniques for teaching QDA methods to large numbers of students presents a challenge to the instructor due to the effort required for the grading of homework. Any homework involving the coding of qualitative data will result in a myriad of different interpretations of the same data with varying quality. Grading such assignments requires significant effort. We approached this problem by using methods of inter-rater agreement and a model solution as a proxy for the quality of the submission. The automated agreement data serves as the foundation for a semi-automated grading process. Within this paper, we demonstrate that this proxy has a high correlation with the manual grading of submission
Keywords: Advances in Teaching and Learning Technologies; homework evaluation; inter-rater agreement; qualitative data analysis

A systematic literature review of pre-requirements specification traceability - Mucha, Julia; Kaufmann, Andreas; Riehle, Dirk (Requirements Engineering, 2024)
Requirements traceability (RT) is the ability to link requirements to other software development artifacts. In pre-requirements (pre-RS) traceability, requirements are linked to their origin, such as interviews with stakeholders, meeting protocols, or legacy systems. Compared with post-RS traceability, which links requirements to source code and other later artifacts, pre-RS traceability has seen much less research. This article presents a systematic literature review of pre-RS traceability based on 77 articles published between 1992 and 2022, aiming to provide a comprehensive overview of its use cases, benefits, problems, and solutions. Through the analysis of existing literature, this review identifies gaps for future research and establishes a foundation for future investigations in the field of pre-RS traceability.<br
Keywords: Pre-requirements specification traceability; Requirements traceability; Requirements engineering; Systematic literature review

A validation of QDAcity-RE for domain modeling using qualitative data analysis - Kaufmann, Andreas; Krause, Julia; Harutyunyan, Nikolay; Barcomb, Ann; Riehle, Dirk (Requirements Engineering, 2022)
Using qualitative data analysis (QDA) to perform domain analysis and modeling has shown great promise. Yet, the evaluation of such approaches has been limited to single-case case studies. While these exploratory cases are valuable for an initial assessment, the evaluation of the efficacy of QDA to solve the suggested problems is restricted by the common single-case case study research design. Using our own method, called QDAcity-RE, as the example, we present an in-depth empirical evaluation of employing qualitative data analysis for domain modeling using a controlled experiment design. Our controlled experiment shows that the QDA-based method leads to a deeper and richer set of domain concepts discovered from the data, while also being more time efficient than the control group using a comparable non-QDA-based method with the same level of traceability.
Keywords: Domain model; Domain modeling; Qualitative data analysis; Requirements engineering; Controlled experiment

Domain Modeling Using Qualitative Data Analysis - Kaufmann, Andreas (Friedrich-Alexander-Universität Erlangen-Nürnberg, 2021)
The creation of domain models from qualitative input relies heavily on experience. An uncodified ad-hoc modeling process is still common and leads to poor documentation of the requirements analysis. In this thesis, we present a novel method for domain analysis based on qualitative data analysis (QDA). The method helps identifying inconsistencies, ensures a high degree of completeness, and inherently provides traceability from analysis results back to stakeholder input. In our approach, the QDAcity-RE method, the research process of theory building facilitates domain analysis within the requirements elicitation phase of a software development project. We show how an iterative process of concurrent data collection and analysis can be applied to requirements engineering (RE), including open, axial, and selective coding of qualitative data. The traceability of domain model elements back to original statements by stakeholders, generated by our method, does not have to be created and maintained separately after the fact. The traces are documented in an analysis artifact called the code system, which evolves iteratively with the analysis process. The code system can act as a universal model from which a variety of artifacts can be derived, describing both behavioral and structural aspects. This thesis focuses on the creation of conceptual domain models using QDAcity-RE, but peripherally also demonstrates this capability through the generation of behavioral models and a software requirements specification. We applied and evaluated our method for domain modeling in four exploratory projects in the domains of medical imaging diagnostics, railway systems, HR development, and qualitative research. We show that by applying QDA to domain analysis, structural elements and relationships needed to derive a UML class diagram can be extracted from a code system based on interviews with domain experts. Constant comparison and theoretical sampling assist in integrating differing domain descriptions into an abstract model. While the analysis process still requires interpretations and modeling decisions, our method provides more guidance than existing domain analysis approaches and a thorough documentation of these decisions. In addition, codes and memos ensure traceability between the original data and the derived model and assist in connecting several RE artifacts, ensuring a high degree of inter-model consistency. We validated our claim that QDAcity-RE helps an analyst gain a deeper understanding of a problem domain through a controlled experiment.<br
Keywords: Domain Modeling; Qualitative Data Analysis; Requirements Engineering

Improving Traceability of Requirements Through Qualitative Data Analysis - Kaufmann, Andreas; Riehle, Dirk (Software Engineering, 2015)

Supporting Interview Analysis with Autocoding - Kaufmann, Andreas; Barcomb, Ann; Riehle, Dirk (53rd Hawaii International Conference on System Sciences (HICSS 2020), 2020)
Interview analysis is a technique employed in qualitative research. Researchers annotate (code) interview transcriptions, often with the help of Computer-Assisted Qualitative Data Analysis Software (CAQDAS). The tools available today largely replicate the manual process of annotation. In this article, we demonstrate how to use natural language processing (NLP) to increase the reproducibility and traceability of the process of applying codes to text data. We integrated an existing commercial machine–learning (ML) based concept extraction service into an NLP pipeline independent of domain specific rules. We applied our prototype in three qualitative studies to evaluate its capabilities of supporting researchers by providing recommendations consistent with their initial work. Unlike rule based approaches, our process can be applied to interviews from any domain, without additional burden to the researcher for creating a new ruleset. Our work using three example data sets shows that this approach shows promise for a real–life application, but further research is neede
Keywords: QDA; NLP; interview analysis; CAQDAS; machine learning; autocoding

The Benefits of Pre-Requirements Specification Traceability - Krause, Julia; Kaufmann, Andreas; Riehle, Dirk; Jung, Martin (Proceedings of the 30th International IEEE Requirements Engineering Conference (RE 2022), 2022)
Requirements traceability is the ability to trace requirements to other software engineering artifacts. Traceability can be classified as either pre- or post-requirements specifications (RS) traceability. Pre-RS traceability is the ability to trace between requirements and their origin. However, the benefits of pre-RS traceability are often not clear. In this article, we systematically lay out the benefits of pre-RS traceability. We present results from both a literature review and a qualitative survey of practitioners involved with documenting and utilizing such trace links. We find that the benefits strongly depend on the practitioners, their tasks, and the project environment. Awareness of these relationships supports a clearer understanding of the benefits of pre-RS traceability and thus motivates successful implementation of the required practices. The results of our research motivates the adoption of pre-RS traceability and present problem areas for future research.<br
Keywords: Requirements Engineering; Requirements Traceability; Pre-Requirements specification Traceability; Literature review; Qualitative Survey

The Code System of a Systematic Literature Review on Pre-Requirements Specification Traceability - Krause, Julia; Kaufmann, Andreas; Riehle, Dirk (2020)
Tracing requirements back to their origin is important to understand the context and to identify all potential concerns or conflicts in the genesis of the requirements. The so-called pre-requirements specification (pre-RS) traceability is part of requirements traceability (RT) and is recognized as an important factor for long-term project success. Different approaches have been developed to connect requirements and their origin, but still, there is no general established solution. Mostly it is seen as an effort without significant benefits leading to resistance towards its implementation. To guide future research, a comprehensive understanding of the use cases, benefits, challenges, and existing approaches encountered is vital. However, the landscape of the scientific literature on this field remains somewhat fragmented, with pre-RS traceability often only considered in the periphery. We, therefore, conducted a systematic literature review, which identified 67 relevant papers. The literature was coded using qualitative data analysis (QDA) methods. In this technical report, we present the whole resulting code system of the QDA and describe the relationships between the identified themes, focusing on the relevance, problems, solutions, and techniques of pre-RS traceability.<br
Keywords: Pre-Requirements Specification Traceability; Requirements Traceability; Requirements Engineering; Qualitative Data Analysis

The QDAcity-RE method for structural domain modeling using qualitative data analysis - Kaufmann, Andreas; Riehle, Dirk (Requirements Engineering, 2017)
The creation of domain models from qualitative input relies heavily on experience. An uncodified ad-hoc modeling process is still common and leads to poor documentation of the analysis. In this article we present a new method for domain analysis based on qualitative data analysis. The method helps identify inconsistencies, ensures a high degree of completeness, and inherently provides traceability from analysis results back to stakeholder input. These traces do not have to be documented after the fact. We evaluate our approach using four exploratory studie
Keywords: Domain modeling; Domain model; Requirements engineering; Requirements elicitation; Qualitative data analysis

Uncovering the Periphery: A Qualitative Survey of Episodic Volunteering in Free/Libre and Open Source Software Communities - Barcomb, Ann; Kaufmann, Andreas; Riehle, Dirk; Stol, Klaas Jan; Fitzgerald, Brian (IEEE Transactions on Software Engineering, 2020)
Free/Libre and Open Source Software (FLOSS) communities are composed, in part, of volunteers, many of whom contribute infrequently. However, these infrequent volunteers contribute to the sustainability of FLOSS projects, and should ideally be encouraged to continue participating, even if they cannot be persuaded to contribute regularly. Infrequent contributions are part of a trend which has been widely observed in other sectors of volunteering, where it has been termed “episodic volunteering” (EV). Previous FLOSS research has focused on the Onion model, differentiating core and peripheral developers, with the latter considered as a homogeneous group. We argue this is too simplistic, given the size of the periphery group and the myriad of valuable activities they perform beyond coding. Our exploratory qualitative survey of 13 FLOSS communities investigated what episodic volunteering looks like in a FLOSS context. EV is widespread in FLOSS communities, although not specifically managed. We suggest several recommendations for managing EV based on a framework drawn from the volunteering literature. Also, episodic volunteers make a wide range of value-added contributions other than code, and they should neither be expected nor coerced into becoming habitual volunteers.
Keywords: Community management; episodic volunteering; free software; open source software; peripheral developer

Using Natural Language Processing to Support Interview Analysis - Kaufmann, Andreas (Friedrich-Alexander-Universität Erlangen-Nürnberg, 2014)

Using Students as a Distributed Coding Team for Validation through Intercoder Agreement - Kaufmann, Andreas; Barcomb, Ann; Riehle, Dirk (2016)