A Context-aware Conversational Interface for Controlling Internet-of-Things Devices - Gui, Zhou; Freund, Michael; Harth, Andreas (IoT '24: Proceedings of the 13th International Conference on the Internet of Things, 2024)
This paper presents a demonstration of a context-aware conversational interface for interacting with multiple IoT devices using multi-turn natural language commands. Our system facilitates device discovery and identification through a Knowledge Graph (KG) and addresses the interoperability issue by using the Web of Things (WoT) specifications as an intermediate abstraction layer. The proposed system comprises four major components: a context-aware multi-turn dialogue interface, a device discovery and identification module using a domain-specific KG, a text-to-code module to parse natural language commands into an executable code format, and a customized code execution engine. We demonstrate our system using two Philips Hue smart lamps and one Elgato Stream Deck controller. The Philips Hue smart lamp device can be controlled standalone via natural language, allowing users to adjust the power state, the color, and the brightness. Also, the lamp can be integrated with the Elgato Stream Deck Controller to enable user-defined automation rules following the Trigger Action programming (TAP) paradigm.</smal
Keywords: Conversational Interface; Internet of Things; Trigger Action Programming; Natural Language Understanding; Knowledge Graph

A Natural Language Interface for IoT Systems Using the Web of Things Abstraction - Gui, Zhou; Freund, Michael; Harth, Andreas (IoT '23: Proceedings of the 13th International Conference on the Internet of Things, 2024)
We present a demo of a Natural Language Interface (NLI) for controlling Internet of Things (IoT) devices using the Web of Things (WoT) specification as an intermediate abstraction layer. All interaction information of a device is stored in a Knowledge Graph using the thing description ontology. The central component of the NLI is a sequence-to-sequence neural network model for text to code translation. We build a data corpus based on the functionalities of a Philips Hue smart lamp and use the corpus to train the text to code model. Our demonstration illustrates how to control the power state, the light colour, and the brightness of a Philips Hue smart lamp using natural language commands. The implementation of an NLI system based on the WoT specification represents an approach towards the development of easy-to-use and interoperable IoT system
Keywords: Human-centered computing; Human computer interaction (HCI); Natural language interfaces; Information systems; Resource Description Framework (RDF)

Design and Implementation of a Natural Language Interface for Controlling the Web of Things Devices - Gui, Zhou; Kuty, Layla; Harth, Andreas (Natural Language Processing and Information Systems, 2024)
We present a systematic workflow for developing a Natural Language Interface (NLI) using Knowledge Graphs (KGs) and the Web of Things (WoT) specification as an abstraction layer. We first synthesize a data corpus from a given device interface description using manually created templates and parameter sheets. Then we paraphrase the synthesized data corpus with pre-trained Large Language Models (LLMs). The resulting corpus serves as training data for a text-to-code encoder- decoder neural network model, enabling the mapping of diverse natural language commands into an executable code format. To the best of our knowledge, no existing data corpus or NLI system has been tailored for WoT device interactions. Our work provides a baseline and can be ex- tended to a broader range of real world WoT use cases.
Keywords: Natural Language Interface; Knowledge Graphs; Large Language Models; Web of Things

Open Information Extraction on German Wikipedia Texts - Klose, Christian; Gui, Zhou; Harth, Andreas (CEUR Workshop Proceedings (CEUR-WS), 2022)
Knowledge Graphs are becoming a fundamental building block for semantic search and voice assistants. This paper deals with the automated Knowledge Graph Construction from unstructured data. Predominantly, the focus is on Open Information Extraction (Open IE), an unsupervised learning approach that attempts to extract triples from plain text independent of their domain. Hence, it is the first step towards automated Knowledge Graph Construction. Previous work mainly applied Open IE to English texts. In this paper, the focus is on German texts. Due to the lack of German Open Information Extraction datasets, a dataset on the basis of Wikipedia is created. Two Open Information Extraction Systems for German are introduced. Finally, the performance of the systems are evaluate
Keywords: Open Information Extraction; Natural Language Processing; Knowledge Graph Construction

Towards a Data Driven Natural Language Interface for Industrial IoT Use Cases - Gui, Zhou; Harth, Andreas (Proceedings of the 2021 IEEE International Conference on Human-Machine Systems, ICHMS 2021, 2021)
The ubiquitous availability of sensors and smart devices makes IoT networks more and more complex to manage and control. A natural language interface (NLI) would allow users to interact with the devices via human language by translating the user command into a machine-interpretable meaning representation, often called logical forms.Despite the rapid development of conversational interfaces in smart home and personal intelligent assistant use cases, there are limited research and applications in industrial sensor and actuator networks, usually referred to as Industrial Internet of Things (IIoT). In this paper, we show an early phase design principle of a semantic representation to express IIoT device interactions and propose a data-focused workflow of IIoT automation system architecture.
CRIS-Team Scopus Importer:2021-11-19
Keywords: Industrial Internet of Things; Knowledge Graph; Natural Language Interface; Semantic Representation