Hi there!

Hands-free control offers natural and intuitive interaction with devices, particularly in scenarios where traditional input methods are impractical. We introduce an extensible framework that integrates eye tracking, object detection, and gesture recognition to study intended and unintended interactions with Internet of Things (IoT) devices. To develop our framework, we conducted a structured experiment with 9 participants, focusing on identifying natural and intuitive interaction behaviors in different situations. The results showed that users intuitively combined gaze- and head-based gestures, showing the potential of head/gaze combinations as input mechanisms, specifically for directional movements. On this basis, we propose a system for hands-free interaction and control of IoT devices with intuitive gaze- and head-based gestures. We report on our promising findings as well as on limitations with respect to accurately distinguishing intention in real-world conditions. All our code is publicly available, ensuring the reproducibility and extension of our findings.

Kenan Bektaş, Tobias Ettling, Simon Mayer, and Jannis Strecker-Bischoff. 2026. Magic Gaze: Enabling Seamless Control of IoT Devices Through Eye Tracking. In 2026 Symposium on Eye Tracking Research and Applications (ETRA ’26), June 01–04, 2026, Marrakesh, Morocco. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3797246.3804834

@inproceedings{10.1145/3797246.3804834,
author = {Bekta\c{s}, Kenan and Ettling, Tobias and Mayer, Simon and Strecker-Bischoff, Jannis},
title = {Magic Gaze: Enabling Seamless Control of IoT Devices Through Eye Tracking},
year = {2026},
isbn = {9798400725197},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3797246.3804834},
doi = {10.1145/3797246.3804834},
abstract = {Hands-free control offers natural and intuitive interaction with devices, particularly in scenarios where traditional input methods are impractical. We introduce an extensible framework that integrates eye tracking, object detection, and gesture recognition to study intended and unintended interactions with Internet of Things (IoT) devices. To develop our framework, we conducted a structured experiment with 9 participants, focusing on identifying natural and intuitive interaction behaviors in different situations. The results showed that users intuitively combined gaze- and head-based gestures, showing the potential of head/gaze combinations as input mechanisms, specifically for directional movements. On this basis, we propose a system for hands-free interaction and control of IoT devices with intuitive gaze- and head-based gestures. We report on our promising findings as well as on limitations with respect to accurately distinguishing intention in real-world conditions. All our code1 is publicly available, ensuring the reproducibility and extension of our findings.},
booktitle = {Proceedings of the 2026 Symposium on Eye Tracking Research and Applications},
articleno = {91},
numpages = {8},
location = {
},
series = {ETRA '26}
}

In pilot training, deviation from standard procedures is a significant concern. To provide student pilots with objective feedback in post-flight debriefing, we captured pilots' view and gaze with the Pupil Core eye-tracker. Then we conducted a preliminary evaluation to test the feasibility of existing scene segmentation models for gaze-mapping. We used an OpenCV baseline model for coarse inside vs. outside-analysis, a fine-tuned Detectron2 model for specific instrument segmentation, and Segment Anything Models (SAM 2 and SAM 3) for human-in-the-loop analysis. The baseline was fast but fragile, failing in common flight scenarios; the Detectron2 model was powerful but inflexible and unsuitable for general use; and SAM 3 was promising, offering generalizability for post-flight analysis despite noisy digital displays. A qualitative preliminary evaluation of SAM with Visual Flight Rules shows that it can be beneficial in eye movement analysis. We identified poor data quality in bright cockpit environments and ergonomics as main limitations.

Sebastian Oes, Kenan Bektaş, Jannis Strecker-Bischoff, and Simon Mayer. 2026. ClearSkies: A Preliminary Study of Gaze-Mapped Scene Segmentation in Training Aircraft Cockpits. In 2026 Symposium on Eye Tracking Research and Applications (ETRA ’26), June 01–04, 2026, Marrakesh, Morocco. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3797246.3805852

@inproceedings{10.1145/3797246.3805852,
author = {Oes, Sebastian and Bektas, Kenan and Strecker-Bischoff, Jannis and Mayer, Simon},
title = {ClearSkies: A Preliminary Study of Gaze-Mapped Scene Segmentation in Training Aircraft Cockpits},
year = {2026},
isbn = {9798400725197},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3797246.3805852},
doi = {10.1145/3797246.3805852},
abstract = {In pilot training, deviation from standard procedures is a significant concern. To provide student pilots with objective feedback in post-flight debriefing, we captured pilots’ view and gaze with the Pupil Core eye-tracker. Then we conducted a preliminary evaluation to test the feasibility of existing scene segmentation models for gaze-mapping1. We used an OpenCV baseline model for coarse inside vs. outside-analysis, a fine-tuned Detectron2 model for specific instrument segmentation, and Segment Anything Models (SAM 2 and SAM 3) for human-in-the-loop analysis. The baseline was fast but fragile, failing in common flight scenarios; the Detectron2 model was powerful but inflexible and unsuitable for general use; and SAM 3 was promising, offering generalizability for post-flight analysis despite noisy digital displays. A qualitative preliminary evaluation of SAM with Visual Flight Rules shows that it can be beneficial in eye movement analysis. We identified poor data quality in bright cockpit environments and ergonomics as main limitations.},
booktitle = {Proceedings of the 2026 Symposium on Eye Tracking Research and Applications},
articleno = {33},
numpages = {3},
location = {
},
series = {ETRA '26}
}

ABIS is an international workshop organized by the SIG on Adaptivity and User Modeling in Interactive Software Systems of the German Gesellschaft für Informatik. For more than 30 years, the ABIS workshop has served as a highly interactive forum for discussing the state of the art in personalization, user modeling, and related areas. The 2026 edition will focus on personalization and recommendations, with particular attention to adaptation on the Web and its societal effects. To explore and discuss the effects of the adaptive Web on society, the workshop aims to bring together researchers and practitioners to share insights and discuss emerging findings and future developments. Our goal is to identify current trends, newly observed effects, and future research directions, with the overarching aim of fostering the development of this discipline and encouraging new collaborations.

Laura Stojko, Eelco Herder, Jannis Strecker-Bischoff, Julia Seitz Sänger, Thomas Neumayr, Enes Yigitbas, and Mirjam Augstein. 2026. ABIS 2026: The Effects of the Adaptive Web on Society. In 18th ACM Web Science Conference (WebSci Companion ’26), May 26–29, 2026, Braunschweig, Germany. ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/3795513.3810983

@inproceedings{10.1145/3795513.3810983,
author = {Stojko, Laura and Herder, Eelco and Strecker-Bischoff, Jannis and Seitz S\"{a}nger, Julia and Neumayr, Thomas and Yigitbas, Enes and Augstein, Mirjam},
title = {ABIS 2026: The Effects of the Adaptive Web on Society},
year = {2026},
isbn = {9798400724923},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3795513.3810983},
doi = {10.1145/3795513.3810983},
abstract = {ABIS is an international workshop organized by the SIG on Adaptivity and User Modeling in Interactive Software Systems of the German Gesellschaft f\"{u}r Informatik. For more than 30 years, the ABIS workshop has served as a highly interactive forum for discussing the state of the art in personalization, user modeling, and related areas. The 2026 edition will focus on personalization and recommendations, with particular attention to adaptation on the Web and its societal effects. To explore and discuss the effects of the adaptive Web on society, the workshop aims to bring together researchers and practitioners to share insights and discuss emerging findings and future developments. Our goal is to identify current trends, newly observed effects, and future research directions, with the overarching aim of fostering the development of this discipline and encouraging new collaborations.},
booktitle = {Companion Publication of the 2026 18th ACM Web Science Conference},
pages = {108–109},
numpages = {2},
keywords = {Adaptive Web, Personalization, Recommendations, Societal Effects},
location = {
},
series = {WebSci Companion '26}
}