JUIC-IoT: Just-In-Time User Interfaces for Interacting with IoT Devices in Mixed Reality

The number of deployed Internet of Things (IoT) devices is continuously increasing. While Mixed Reality (MR) allows hands-free interaction, creating MR User Interfaces (UI) for each IoT device is challenging, as often a separate interface has to be designed for each individual device. Additionally, approaches for automatic MR UI generation often still require manual developer intervention. To address these issues, we propose the JUIC-IoT system, which automatically assembles Just-in-Time MR UIs for IoT devices based on the machine-understandable format W3C Web of Things Thing Description (TD). JUIC-IoT detects an IoT device with object recognition, uses its TD to prompt an LLM for automatically selecting appropriate UI components, and then assembles a UI for interacting with the device. Our evaluation of JUIC-IoT shows us that the choice of LLM and the TD of a device are more crucial than the formulation of the input prompts for obtaining a usable UI. JUIC-IoT represents a step towards dynamic UI generation, thereby enabling intuitive interactions with IoT devices.

Lucien Ledermann, Jannis Strecker-Bischoff, Kimberly Garcia, and Simon Mayer. 2025. JUIC-IoT: Just-In-Time User Interfaces for Interacting with IoT Devices in Mixed Reality. In Companion of the the 2025 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp Companion ’25), October 12–16, 2025, Espoo, Finland. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3714394.3754371