Magic Gaze: Enabling Seamless Control of IoT Devices Through Eye Tracking

Hands-free control offers natural and intuitive interaction with devices, particularly in scenarios where traditional input methods are impractical. We introduce an extensible framework that integrates eye tracking, object detection, and gesture recognition to study intended and unintended interactions with Internet of Things (IoT) devices. To develop our framework, we conducted a structured experiment with 9 participants, focusing on identifying natural and intuitive interaction behaviors in different situations. The results showed that users intuitively combined gaze- and head-based gestures, showing the potential of head/gaze combinations as input mechanisms, specifically for directional movements. On this basis, we propose a system for hands-free interaction and control of IoT devices with intuitive gaze- and head-based gestures. We report on our promising findings as well as on limitations with respect to accurately distinguishing intention in real-world conditions. All our code is publicly available, ensuring the reproducibility and extension of our findings.

Kenan Bektaş, Tobias Ettling, Simon Mayer, and Jannis Strecker-Bischoff. 2026. Magic Gaze: Enabling Seamless Control of IoT Devices Through Eye Tracking. In 2026 Symposium on Eye Tracking Research and Applications (ETRA ’26), June 01–04, 2026, Marrakesh, Morocco. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3797246.3804834

@inproceedings{10.1145/3797246.3804834,
author = {Bekta\c{s}, Kenan and Ettling, Tobias and Mayer, Simon and Strecker-Bischoff, Jannis},
title = {Magic Gaze: Enabling Seamless Control of IoT Devices Through Eye Tracking},
year = {2026},
isbn = {9798400725197},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3797246.3804834},
doi = {10.1145/3797246.3804834},
abstract = {Hands-free control offers natural and intuitive interaction with devices, particularly in scenarios where traditional input methods are impractical. We introduce an extensible framework that integrates eye tracking, object detection, and gesture recognition to study intended and unintended interactions with Internet of Things (IoT) devices. To develop our framework, we conducted a structured experiment with 9 participants, focusing on identifying natural and intuitive interaction behaviors in different situations. The results showed that users intuitively combined gaze- and head-based gestures, showing the potential of head/gaze combinations as input mechanisms, specifically for directional movements. On this basis, we propose a system for hands-free interaction and control of IoT devices with intuitive gaze- and head-based gestures. We report on our promising findings as well as on limitations with respect to accurately distinguishing intention in real-world conditions. All our code1 is publicly available, ensuring the reproducibility and extension of our findings.},
booktitle = {Proceedings of the 2026 Symposium on Eye Tracking Research and Applications},
articleno = {91},
numpages = {8},
location = {
},
series = {ETRA '26}
}