Continuous, Real-Time Emotion Annotation: A Novel Joystick-Based Analysis Framework

Sharma K, Stulp F, Van Den Broek EL, Castellini C (2020)

Publication Type: Journal article

Publication year: 2020

Journal

IEEE Transactions on Affective Computing Institute of Electrical and Electronics Engineers

Book Volume: 11

Pages Range: 78-84

Article Number: 8105870

Journal Issue: 1

DOI: 10.1109/TAFFC.2017.2772882

Abstract

Emotion labels are usually obtained via either manual annotation, which is tedious and time-consuming, or questionnaires, which neglect the time-varying nature of emotions and depend on human's unreliable introspection. To overcome these limitations, we developed a continuous, real-time, joystick-based emotion annotation framework. To assess the same, 30 subjects each watched 8 emotion-inducing videos. They were asked to indicate their instantaneous emotional state in a valence-arousal (V-A) space, using a joystick. Subsequently, five analyses were undertaken: (i) a System Usability Scale (SUS) questionnaire unveiled the framework's excellent usability; (ii) MANOVA analysis of the mean V-A ratings and (iii) trajectory similarity analyses of the annotations confirmed the successful elicitation of emotions; (iv) Change point analysis of the annotations, revealed a direct mapping between emotional events and annotations, thereby enabling automatic detection of emotionally salient points in the videos; and (v) Support Vector Machines (SVM) were trained on classification of 5 second chunks of annotations as well as their change-points. The classification results confirmed that ratings patterns were cohesive across the participants. These analyses confirm the value, validity, and usability of our annotation framework. They also showcase novel tools for gaining greater insights into the emotional experience of the participants.

Authors with CRIS profile

Claudio Castellini

Involved external institutions

Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR) / German Aerospace Center

Germany (DE) Universiteit Utrecht (UU) / Utrecht University

Netherlands (NL)

How to cite

APA:

Sharma, K., Stulp, F., Van Den Broek, E.L., & Castellini, C. (2020). Continuous, Real-Time Emotion Annotation: A Novel Joystick-Based Analysis Framework. IEEE Transactions on Affective Computing, 11(1), 78-84. https://dx.doi.org/10.1109/TAFFC.2017.2772882

MLA:

Sharma, Karan, et al. "Continuous, Real-Time Emotion Annotation: A Novel Joystick-Based Analysis Framework." IEEE Transactions on Affective Computing 11.1 (2020): 78-84.

BibTeX: Download