Publications

Personalization of online realities is today ubiquitous to support decision making or reduce information overload. Recently, through the expanding capabilities and pervasiveness of Mixed Reality and Ubiquitous Computing technologies, we observe increasing personalization also of physical reality. This might yield more convenient, efficient and inclusive everyday interactions. However, it may readily lead to serious societal consequences such as the loss of shared worlds and the emergence of perceptual filter bubbles. To mitigate such harms while retaining the benefits of personalization, it is important to understand how ubiquitous personalization systems may operate responsibly. Responding to this need, we propose a conceptual model that overcomes the limitations of established personalization models and expands their applicable scope to physical, virtual, and hybrid environments. We validated our model in relation to existing literature and show how it provides a conceptual foundation for the analysis and study of responsible personalization systems that create individually and societally beneficial Personalized Realities.

Jannis Strecker, Simon Mayer, and Kenan Bektaş. 2025. Towards Societally Beneficial Personalized Realities: A Conceptual Foundation for Responsible Ubiquitous Personalization Systems. In Designing Interactive Systems Conference (DIS ’25), July 5–9, 2025, Funchal, Portugal. ACM, New York, NY, USA, 23 pages. https://doi.org/10.1145/3715336.3735709

Extended Reality (XR) technologies enable the personalized mediation of an individual's perceivable reality across modalities, thereby creating a Personalized Reality (PR). While this may lead to individually beneficial effects in the form of more efficient, more fun, and safer experiences, it may also lead to perceptual filter bubbles since individuals are exposed predominantly or exclusively to content that is congruent with their existing beliefs and opinions. This undermining of a shared basis for interaction and discussion through constrained perceptual worldviews may impact society through increased polarization and other well-documented negative effects of filter bubbles. In this paper, we argue that this issue can be mitigated by increasing individuals' awareness of their current perspective and providing avenues for development, including through support for engineered serendipity and fostering of self-actualization that already show promise for traditional recommender systems. We discuss how these methods may be transferred to XR to yield valuable tools to give people transparency and agency over their perceptual worldviews in a responsible manner.

Jannis Strecker, Luka Bekavac, Kenan Bektaş, and Simon Mayer. 2025. Change Your Perspective, Widen Your Worldview! Societally Beneficial Perceptual Filter Bubbles in Personalized Reality. In Proceedings of Purposeful XR: Affordances, Challenges, and Speculations for an Ethical Future at CHI 2025 (Purposeful XR ’25). 6 pages. https://www.alexandria.unisg.ch/handle/20.500.14171/122376

Inspired by the concepts of diminishing reality and ad-blocking in browsers, this study investigates the perceived benefits and concerns of blocking physical, real-world content, particularly ads, through Extended Reality (XR). To understand how users perceive this concept, we first conducted a user study (N=18) with an ad-blocking prototype to gather initial insights. The results revealed a mixed willingness to adopt XR blockers, with participants appreciating aspects such as customizability, convenience, and privacy. Expected benefits included enhanced focus and reduced stress, while concerns centered on missing important information and increased feelings of isolation. Hence, we investigated the user acceptance of different ad-blocking visualizations through a follow-up online survey (N=120), comparing six concepts based on related work. The results indicated that the XR ad-blocker visualizations play a significant role in how and for what kinds of advertisements such a concept might be used, paving the path for future feedback-driven prototyping.

Christopher Katins, Jannis Strecker, Jan Hinrichs, Pascal Knierim, Bastian Pfleging, and Thomas Kosch. 2025. Ad-Blocked Reality: Evaluating User Perceptions of Content Blocking Concepts Using Extended Reality. In CHI Conference on Human Factors in Computing Systems (CHI '25), April 26–May 01, 2025, Yokohama, Japan. ACM, New York, NY, USA, 18 pages. https://doi.org/10.1145/3706598.3713230

@inproceedings{10.1145/3706598.3713230,
author = {Katins, Christopher and Strecker, Jannis and Hinrichs, Jan and Knierim, Pascal and Pfleging, Bastian and Kosch, Thomas},
title = {Ad-Blocked Reality: Evaluating User Perceptions of Content Blocking Concepts Using Extended Reality},
year = {2025},
isbn = {9798400713941},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3706598.3713230},
doi = {10.1145/3706598.3713230},
abstract = {Inspired by the concepts of diminishing reality and ad-blocking in browsers, this study investigates the perceived benefits and concerns of blocking physical, real-world content, particularly ads, through Extended Reality (XR). To understand how users perceive this concept, we first conducted a user study (N = 18) with an ad-blocking prototype to gather initial insights. The results revealed a mixed willingness to adopt XR blockers, with participants appreciating aspects such as customizability, convenience, and privacy. Expected benefits included enhanced focus and reduced stress, while concerns centered on missing important information and increased feelings of isolation. Hence, we investigated the user acceptance of different ad-blocking visualizations through a follow-up online survey (N = 120), comparing six concepts based on related work. The results indicated that the XR ad-blocker visualizations play a significant role in how and for what kinds of advertisements such a concept might be used, paving the path for future feedback-driven prototyping.},
booktitle = {Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems},
articleno = {626},
numpages = {18},
keywords = {Extended Reality, Content Curation, Visualization, Physical Ad Blocker},
location = {Yokhama, Japan},
series = {CHI '25}
}

Technological advancements, particularly in artificial intelligence, significantly transform our society and work practices. Computational thinking (CT) has emerged as a crucial 21-century skill, enabling individuals to solve problems more effectively through an automation-oriented perspective and fundamental concepts of computer science. To ensure the effective integration of CT into educational curricula, it is crucial to develop efficient assessment frameworks that allow teachers to measure and promote student CT proficiency. Therefore, our aim is to develop a short test to measure CT among undergraduate students. To this end, we consider two performance tests: the Computational Thinking test (CTt) and the Algorithmic Thinking Test for Adults (ATTA). We use items from both instruments to compile a short test. Based on a sample of 290 second-year non-computer science undergraduate students, we provide evidence on the quality of our test. Besides classical test theory, we apply item response theory, namely Rasch modeling, and confirmatory factor analysis. Our test shows favorable properties, e.g., Cronbach's alpha > .75, and may be suitable for the efficient assessment of CT across higher education programs.

Josef Guggemos, Roman Rietsche, Stephan Aier, Jannis Strecker and Simon Mayer. 2024. Measuring Computational Thinking - Developing A Short Performance Test For Higher Education. In Proceedings of the 21st International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2024), October 26-28, Zabreg, Croatia.

Social media platforms play a significant role in shaping public opinion and societal norms. Understanding this influence requires examining the diversity of content that users are exposed to. However, studying filter bubbles in social media recommender systems has proven challenging, despite extensive research in this area. In this work, we introduce SOAP (System for Observing and Analyzing Posts), a novel system designed to collect and analyze very large online platforms (VLOPs) data to study filter bubbles at scale. Our methodology aligns with established definitions and frameworks, allowing us to comprehensively explore and log filter bubbles data. From an input prompt referring to a topic, our system is capable of creating and navigating filter bubbles using a multimodal LLM. We demonstrate SOAP by creating three distinct filter bubbles in the feed of social media users, revealing a significant decline in topic diversity as fast as in 60min of scrolling. Furthermore, we validate the LLM analysis of posts through an inter-and intra-reliability testing. Finally, we open source SOAP as a robust tool for facilitating further empirical studies on filter bubbles in social media.

Luka Bekavac, Kimberly Garcia, Jannis Strecker, Simon Mayer, and Aurelia Tamò-Larrieux. 2024. From Walls to Windows: Creating Transparency to Understand Filter Bubbles in Social Media. In NORMalize 2024: The Second Workshop on the Normative Design and Evaluation of Recommender Systems. 12 pages.

Establishing rapport between authors and readers of scientific texts is essential for supporting readers in understanding texts as intended, facilitating socio-discursive practices within disciplinary communities, and helping in identifying interdisciplinary links among scientific writings.We propose a Reader-aware Congruence Assistant (RaCA), which supports writers to create texts that are adapted to target readers. Similar to user-centered design which is based on user profiles, RaCA features reader-centered writing through reader profiles that are dynamically computed from information discovered through academic search engines. Our assistant then leverages large language models to measure the congruence of a written text with a given reader profile, and provides feedback to the writer. We demonstrate our approach with an implemented prototype that illustrates how RaCA exploits information available on the Web to construct reader profiles, assesses writer-reader congruence and offers writers color-coded visual feedback accordingly. We argue that our approach to reader-oriented scientific writing paves the way towards the more personalized interaction of readers and writers with scientific content, and discuss how integration with Semantic Web technologies and Adaptive User Interface design can help materialize this vision within an ever-growing Web of scientific ideas, proof, and discourse.

Ge Li, Danai Vachtsevanou, Jérémy Lemée, Simon Mayer, and Jannis Strecker. 2024.2024. Reader-aware Writing Assistance through Reader Profiles. In 34th ACM Conference on Hypertext and Social Media (HT ’24), September 10–13, 2024, Poznań, Poland. ACM, New York, NY, USA, 7 pages.

The expanding capabilities of Mixed Reality and Ubiquitous Computing technologies enable personalization to be increasingly integrated with physical reality in all areas of people's lives. While such ubiquitous personalization promises more inclusive, efficient, pleasurable, and safer everyday interaction, it may also entail serious societal consequences such as isolated perceptions of reality or a loss of control and agency. We present this paper to initiate a discussion towards the responsible creation of ubiquitous personalization experiences that mitigate these harmful implications while retaining the benefits of personalization. To this end, we present the concept of Personalized Reality (PR) to describe a perceived reality that has been adapted in response to personal user data. We provide avenues for future work, and list open questions and challenges towards the creation of responsible PR experiences.

Jannis Strecker, Simon Mayer, and Kenan Bektaş. 2024. Personalized Reality: Challenges of Responsible Ubiquitous Personalization. In Proceedings of Mensch und Computer 2024 – Workshopband, Gesellschaft für Informatik e.V. (MuC'24). 5 pages. https://doi.org/10.18420/muc2024-mci-ws11-200

@inproceedings{strecker2024a,
title = {{Personalized Reality: Challenges of Responsible Ubiquitous Personalization}},
booktitle = {Mensch Und {{Computer}} 2024 \textendash{} {{Workshopband}}},
author = {Strecker, Jannis and Mayer, Simon and Bekta{\c s}, Kenan},
year = 2024,
publisher = {{Gesellschaft f\"ur Informatik e.V.}},
doi = {muc2024-mci-ws11-200},
langid = {english}
}

ABIS is an international workshop, organized by the SIG on Adaptivity and User Modeling in Interactive Software Systems of the German Gesellschaft für Informatik. For more than 25 years, the ABIS Workshop has been a highly interactive forum for discussing the state of the art in personalization, user modeling, and related areas. ABIS 2024’s focus will be on the topics of personalization and recommendation within the areas of Computer-Supported Cooperative Work (CSCW) (i.e., support of individuals who work organized in groups), Cross-Reality (XR) Interaction (e.g., transitions inside the reality-virtuality continuum), and/or making sense of sensory data for personalization purposes. To discuss such questions, our workshop aims to bring together researchers and practitioners who are interested in the general personalization domain, and/or in our SIG’s current focus. Our goal is to identify current issues and future directions of research and foster future development of the discipline and collaborations.

Thomas Neumayr, Enes Yigitbas, Mirjam Augstein, Eelco Herder, Laura Stojko, and Jannis Strecker. 2024. ABIS 2024 - International Workshop on Personalization and Recommendation. In Proceedings of Mensch und Computer 2024 – Workshopband, Gesellschaft für Informatik e.V. (MuC'24). 3 pages. https://doi.org/10.18420/muc2024-mci-ws11-107

We are currently in a period of upheaval, as many new technologies are emerging that open up new possibilities to shape our everyday lives. Particularly, within the field of Personalized Human-Computer Interaction we observe high potential, but also challenges. In this article,we explore how an increasing amount of online services and tools not only further facilitates our lives, but also shapes our lives and how we perceive our environments. For this purpose, we adopt the metaphor of personalized ‘online layers’ and show how these layers are and will be interwoven with the lives that we live in the ‘human layer’ of the real world.

Eelco Herder, Laura Stojko, Jannis Strecker, Thomas Neumayr, Enes Yigitbas, and Mirjam Augstein. 2024. Towards new realities: implications of personalized online layers in our daily lives. i-com (June 2024). https://doi.org/10.1515/icom-2024-0017

@article{herder2024,
title = {Towards New Realities: Implications of Personalized Online Layers in Our Daily Lives},
shorttitle = {Towards New Realities},
author = {Herder, Eelco and Stojko, Laura and Strecker, Jannis and Neumayr, Thomas and Yigitbas, Enes and Augstein, Mirjam},
year = {2024},
month = jun,
journal = {i-com},
publisher = {Oldenbourg Wissenschaftsverlag},
issn = {2196-6826},
doi = {10.1515/icom-2024-0017},
urldate = {2024-06-18},
abstract = {We are currently in a period of upheaval, as many new technologies are emerging that open up new possibilities to shape our everyday lives. Particularly, within the field of Personalized Human-Computer Interaction we observe high potential, but also challenges. In this article, we explore how an increasing amount of online services and tools not only further facilitates our lives, but also shapes our lives and how we perceive our environments. For this purpose, we adopt the metaphor of personalized `online layers' and show how these layers are and will be interwoven with the lives that we live in the `human layer' of the real world.},
langid = {english}
}

Humans only recognize a few items in a scene at once and memorize three to seven items in the short term. Such limitations can be mitigated using cognitive offloading (e.g., sticky notes, digital reminders). We studied whether a gaze-enabled Augmented Reality (AR) system could facilitate cognitive offloading and improve object retrieval performance. To this end, we developed NeighboAR, which detects objects in a user's surroundings and generates a graph that stores object proximity relationships and user's gaze dwell times for each object. In a controlled experiment, we asked N=17 participants to inspect randomly distributed objects and later recall the position of a given target object. Our results show that displaying the target together with the proximity object with the longest user gaze dwell time helps recalling the position of the target. Specifically, NeighboAR significantly reduces the retrieval time by 33%, number of errors by 71%, and perceived workload by 10%.

Aleksandar Slavuljica, Kenan Bektaş, Jannis Strecker, and Simon Mayer. 2024. NeighboAR: Efficient Object Retrieval using Proximity- and Gaze-based Object Grouping with an AR System. Proc. ACM Hum.-Comput. Interact. 8, ETRA, Article 225 (May 2024), 19 pages. https://doi.org/10.1145/3655599

@inproceedings{slavuljica2024,
author = {Slavuljica, Aleksandar and Bekta\c{s}, Kenan and Strecker, Jannis and Mayer, Simon},
title = {NeighboAR: Efficient Object Retrieval using Proximity-and Gaze-based Object Grouping with an AR System},
year = {2024},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3655599},
doi = {10.1145/3655599},
abstract = {Humans only recognize a few items in a scene at once and memorize three to seven items in the short term. Such limitations can be mitigated using cognitive offloading (e.g., sticky notes, digital reminders). We studied whether a gaze-enabled Augmented Reality (AR) system could facilitate cognitive offloading and improve object retrieval performance. To this end, we developed NeighboAR, which detects objects in a user's surroundings and generates a graph that stores object proximity relationships and user's gaze dwell times for each object. In a controlled experiment, we asked N=17 participants to inspect randomly distributed objects and later recall the position of a given target object. Our results show that displaying the target together with the proximity object with the longest user gaze dwell time helps recalling the position of the target. Specifically, NeighboAR significantly reduces the retrieval time by 33%, number of errors by 71%, and perceived workload by 10%.},
booktitle = {Proc. ACM Hum.-Comput. Interact.},
volume = {8}
issue = {ETRA},
articleno = {225},
numpages = {19},
keywords = {augmented reality, cognitive offloading, eye tracking, object detection,human augmentation, mixed reality, working memory, visual search},
}

Non-communicable diseases, such as obesity and diabetes, have a significant global impact on health outcomes. While governments worldwide focus on promoting healthy eating, individuals still struggle to follow dietary recommendations. Augmented Reality (AR) might be a useful tool to emphasize specific food products at the point of purchase. However, AR may also add visual clutter to an already complex supermarket environment. Instead, reducing the visual prevalence of unhealthy food products through Diminished Reality (DR) could be a viable alternative: We present Shopping-Coach, a DR prototype that identifies supermarket food products and visually diminishes them dependent on the deviation of the target product’s composition from dietary recommendations. In a study with 12 participants, we found that ShoppingCoach increased compliance with dietary recommendations from 75% to 100% and reduced decision time by 41%. These results demonstrate the promising potential of DR in promoting healthier food choices and thus enhancing public health.

Jannis Strecker, Jing Wu, Kenan Bektaş, Conrad Vaslin, and Simon Mayer. 2024. ShoppingCoach: Using Diminished Reality to Prevent Unhealthy Food Choices in an Offline Supermarket Scenario. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’24), May 11–16, 2024, Honolulu, HI, USA. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3613905.3650795

@inproceedings{10.1145/3613905.3650795,
author = {Strecker, Jannis and Wu, Jing and Bekta\c{s}, Kenan and Vaslin, Conrad and Mayer, Simon},
title = {ShoppingCoach: Using Diminished Reality to Prevent Unhealthy Food Choices in an Offline Supermarket Scenario},
year = {2024},
isbn = {9798400703317},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3613905.3650795},
doi = {10.1145/3613905.3650795},
abstract = {Non-communicable diseases, such as obesity and diabetes, have a significant global impact on health outcomes. While governments worldwide focus on promoting healthy eating, individuals still struggle to follow dietary recommendations. Augmented Reality (AR) might be a useful tool to emphasize specific food products at the point of purchase. However, AR may also add visual clutter to an already complex supermarket environment. Instead, reducing the visual prevalence of unhealthy food products through Diminished Reality (DR) could be a viable alternative: We present ShoppingCoach, a DR prototype that identifies supermarket food products and visually diminishes them dependent on the deviation of the target product’s composition from dietary recommendations. In a study with 12 participants, we found that ShoppingCoach increased compliance with dietary recommendations from 75\% to 100\% and reduced decision time by 41\%. These results demonstrate the promising potential of DR in promoting healthier food choices and thus enhancing public health.},
booktitle = {Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems},
articleno = {288},
numpages = {8},
keywords = {diminished reality, extended reality, food choices, health informatics, nutrition and health},
location = {Honolulu, HI, USA},
series = {CHI EA '24}
}

As QR codes become ubiquitous in various applications and places, their susceptibility to tampering, known as quishing, poses a significant threat to user security. In this paper we introduce SafeQR codes that address this challenge by introducing innovative design strategies to enhance QR code security. Leveraging visual elements and secure design principles, the project aims to make tampering more noticeable, thereby empowering users to recognize and avoid potential phishing threats. Further, we highlight the limitations of current user-education methods in combating quishing and propose different attacker models tailored to address quishing attacks. In addition, we introduce a multi-faceted defense strategy that merges design innovation with user vigilance. Through a user study, we demonstrate the efficacy of ’Integrity by Design’ QR codes. These innovatively designed QR codes significantly raise user suspicion in case of tampering and effectively reduce the likelihood of successful quishing attacks.

Luka Bekavac, Simon Mayer, and Jannis Strecker. 2024. QR Code Integrity by Design. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’24), May 11–16, 2024, Honolulu, HI, USA. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3613905.3651006

@inproceedings{10.1145/3613905.3651006,
author = {Bekavac, Luka Jure Lars and Mayer, Simon and Strecker, Jannis},
title = {QR-Code Integrity by Design},
year = {2024},
isbn = {9798400703317},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3613905.3651006},
doi = {10.1145/3613905.3651006},
abstract = {As QR codes become ubiquitous in various applications and places, their susceptibility to tampering, known as quishing, poses a significant threat to user security. In this paper we introduce SafeQR codes that address this challenge by introducing innovative design strategies to enhance QR code security. Leveraging visual elements and secure design principles, the project aims to make tampering more noticeable, thereby empowering users to recognize and avoid potential phishing threats. Further, we highlight the limitations of current user-education methods in combating quishing and propose different attacker models tailored to address quishing attacks. In addition, we introduce a multi-faceted defense strategy that merges design innovation with user vigilance. Through a user study, we demonstrate the efficacy of ’Integrity by Design’ QR codes. These innovatively designed QR codes significantly raise user suspicion in case of tampering and effectively reduce the likelihood of successful quishing attacks.},
booktitle = {Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems},
articleno = {274},
numpages = {9},
keywords = {QR code based phishing, QR codes, phishing susceptibility, privacy, quishing},
location = {Honolulu,HI,USA},
series = {CHI EA '24}
}
}

Recent research on remote collaboration focuses on improving the sense of co-presence and mutual understanding among the collaborators, whereas there is limited research on using non-verbal cues such as gaze or head direction alongside their main communication channel. Our system – GlassBoARd – permits collaborators to see each other’s gaze behavior and even make eye contact while communicating verbally and in writing. GlassBoARd features a transparent shared Augmented Reality interface that is situated in-between two users, allowing face-to-face collaboration. From the perspective of each user, the remote collaborator is represented as an avatar that is located behind the GlassBoARd and whose eye movements are contingent on the remote collaborator’s instant eye movements. In three iterations, we improved the design of GlassBoARd and tested it with two use cases. Our preliminary evaluations showed that GlassBoARd facilitates an environment for conducting future user experiments to study the effect of sharing eye gaze on the communication bandwidth.

Kenan Bektaş, Adrian Pandjaitan, Jannis Strecker, and Simon Mayer. 2024. GlassBoARd: A Gaze-Enabled AR Interface for Collaborative Work. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’24), May 11–16, 2024, Honolulu, HI, USA. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3613905.3650965

@inproceedings{10.1145/3613905.3650965,
author = {Bekta\c{s}, Kenan and Pandjaitan, Adrian and Strecker, Jannis and Mayer, Simon},
title = {GlassBoARd: A Gaze-Enabled AR Interface for Collaborative Work},
year = {2024},
isbn = {9798400703317},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3613905.3650965},
doi = {10.1145/3613905.3650965},
abstract = {Recent research on remote collaboration focuses on improving the sense of co-presence and mutual understanding among the collaborators, whereas there is limited research on using non-verbal cues such as gaze or head direction alongside their main communication channel. Our system – GlassBoARd – permits collaborators to see each other’s gaze behavior and even make eye contact while communicating verbally and in writing. GlassBoARd features a transparent shared Augmented Reality interface that is situated in-between two users, allowing face-to-face collaboration. From the perspective of each user, the remote collaborator is represented as an avatar that is located behind the GlassBoARd and whose eye movements are contingent on the remote collaborator’s instant eye movements. In three iterations, we improved the design of GlassBoARd and tested it with two use cases. Our preliminary evaluations showed that GlassBoARd facilitates an environment for conducting future user experiments to study the effect of sharing eye gaze on the communication bandwidth.},
booktitle = {Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems},
articleno = {181},
numpages = {8},
keywords = {CSCW, augmented reality, eye tracking, gaze, non-verbal cues, presence, remote collaboration},
location = {Honolulu, HI, USA},
series = {CHI EA '24}
}

This paper introduces a novel system to enhance the spatiotemporal alignment of human abilities in agent-based workflows. This optimization is realized through the application of Linked Data and Semantic Web technologies and the system makes use of gaze data and contextual information. The showcased prototype demonstrates the feasibility of implementing such a system, where we specifically emphasize the system’s ability to constrain the dissemination of privacy-relevant information.

Jan Grau, Simon Mayer, Jannis Strecker, Kimberly Garcia, and Kenan Bektaş. 2024. Gaze-based Opportunistic Privacy-preserving Human-Agent Collaboration. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’24), May 11–16, 2024, Honolulu, HI, USA. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3613905.3651066

@inproceedings{10.1145/3613905.3651066,
author = {Grau, Jan and Mayer, Simon and Strecker, Jannis and Garcia, Kimberly and Bektas, Kenan},
title = {Gaze-based Opportunistic Privacy-preserving Human-Agent Collaboration},
year = {2024},
isbn = {9798400703317},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3613905.3651066},
doi = {10.1145/3613905.3651066},
abstract = {This paper introduces a novel system to enhance the spatiotemporal alignment of human abilities in agent-based workflows. This optimization is realized through the application of Linked Data and Semantic Web technologies and the system makes use of gaze data and contextual information. The showcased prototype demonstrates the feasibility of implementing such a system, where we specifically emphasize the system’s ability to constrain the dissemination of privacy-relevant information.},
booktitle = {Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems},
articleno = {176},
numpages = {6},
keywords = {Human-Agent-Collaboration, Koreografeye, Privacy-Preserving, Solid},
location = {Honolulu, HI, USA},
series = {CHI EA '24}
}

Mixed Reality technologies are increasingly interwoven with our everyday lives. A variety of powerful Head Mounted Displays have recently entered consumer electronics markets, and more are under development, opening new dimensions for spatial computing. This development will likely not stop at the advertising industry either, as first forays into this area have already been made. We present AuctentionAR which allows users to sell off their visual attention to interested parties. It consists of a HoloLens 2, a remote server executing the auctioning logic, the YOLOv7 model for image recognition of products which may induce an advertising intent, and several bidders interested in advertising their products. As this system comes with substantial privacy implications, we discuss what needs to be considered in future implementation so as to make this system a basis for a privacy preserving MR advertising future.

Adrian Pandjaitan, Jannis Strecker, Kenan Bektaş, and Simon Mayer. 2024. AuctentionAR - Auctioning Off Visual Attention in Mixed Reality. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’24), May 11–16, 2024, Honolulu, HI, USA. ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/3613905.3650941

@inproceedings{pandjaitan2024,
author = {Pandjaitan, Adrian and Strecker, Jannis and Bekta{\c{s}}, Kenan and Mayer, Simon},
title = {{AuctentionAR} - {Auctioning} Off Visual Attention in Mixed Reality},
year = {2024},
isbn = {79-8-4007-0331-7/24/05},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3613905.3650941},
doi = {10.1145/3613905.3650941},
booktitle = {Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems},
location = {Honolulu, HI, USA},
series = {CHI EA '24}
}

Head-mounted Augmented Reality (AR) displays overlay digital information on physical objects. Through eye tracking, they provide insights into user attention, intentions, and activities, and allow novel interaction methods based on this information. However, in physical environments, the implications of using gaze-enabled AR for human activity recognition have not been explored in detail. In an experimental study with the Microsoft HoloLens 2, we collected gaze data from 20 users while they performed three activities: Reading a text, Inspecting a device, and Searching for an object. We trained machine learning models (SVM, Random Forest, Extremely Randomized Trees) with extracted features and achieved up to 89.6% activity-recognition accuracy. Based on the recognized activity, our system—GEAR—then provides users with relevant AR feedback. Due to the sensitivity of the personal (gaze) data GEAR collects, the system further incorporates a novel solution based on the Solid specification for giving users fine-grained control over the sharing of their data. The provided code and anonymized datasets may be used to reproduce and extend our findings, and as teaching material.

Kenan Bektaş, Jannis Strecker, Simon Mayer, and Kimberly Garcia. 2024. Gaze-enabled activity recognition for augmented reality feedback. Computers & Graphics (March 2024), 103909. https://doi.org/10.1016/j.cag.2024.103909

The increasing number of objects in ubiquitous computing environments creates a need for effective object detection and identification mechanisms that permit users to intuitively initiate interactions with these objects. While multiple approaches to such object detection -- including through visual object detection, fiducial markers, relative localization, or absolute spatial referencing -- are available, each of these suffers from drawbacks that limit their applicability. In this paper, we propose ODIF, an architecture that permits the fusion of object situation information from such heterogeneous sources and that remains vertically and horizontally modular to allow extending and upgrading systems that are constructed accordingly. We furthermore present BLEARVIS, a prototype system that builds on the proposed architecture and integrates computer-vision (CV) based object detection with radio-frequency (RF) angle of arrival (AoA) estimation to identify BLE-tagged objects. In our system, the front camera of a Mixed Reality (MR) head-mounted display (HMD) provides a live image stream to a vision-based object detection module, while an antenna array that is mounted on the HMD collects AoA information from ambient devices. In this way, BLEARVIS is able to differentiate between visually identical objects in the same environment and can provide an MR overlay of information (data and controls) that relates to them. We include experimental evaluations of both, the CV-based object detection and the RF-based AoA estimation, and discuss the applicability of the combined RF and CV pipelines in different ubiquitous computing scenarios. This research can form a starting point to spawn the integration of diverse object detection, identification, and interaction approaches that function across the electromagnetic spectrum, and beyond.

Jannis Strecker, Khakim Akhunov, Federico Carbone, Kimberly García, Kenan Bektaş, Andres Gomez, Simon Mayer, and Kasim Sinan Yildirim. 2023. MR Object Identification and Interaction: Fusing Object Situation Information from Heterogeneous Sources. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 7, 3, Article 124 (September 2023), 26 pages. https://doi.org/10.1145/3610879

@article{10.1145/3610879,
author = {Strecker, Jannis and Akhunov, Khakim and Carbone, Federico and Garc\'{\i}a, Kimberly and Bekta\c{s}, Kenan and Gomez, Andres and Mayer, Simon and Yildirim, Kasim Sinan},
title = {MR Object Identification and Interaction: Fusing Object Situation Information from Heterogeneous Sources},
year = {2023},
issue_date = {September 2023},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {7},
number = {3},
url = {https://doi.org/10.1145/3610879},
doi = {10.1145/3610879},
abstract = {The increasing number of objects in ubiquitous computing environments creates a need for effective object detection and identification mechanisms that permit users to intuitively initiate interactions with these objects. While multiple approaches to such object detection -- including through visual object detection, fiducial markers, relative localization, or absolute spatial referencing -- are available, each of these suffers from drawbacks that limit their applicability. In this paper, we propose ODIF, an architecture that permits the fusion of object situation information from such heterogeneous sources and that remains vertically and horizontally modular to allow extending and upgrading systems that are constructed accordingly. We furthermore present BLEARVIS, a prototype system that builds on the proposed architecture and integrates computer-vision (CV) based object detection with radio-frequency (RF) angle of arrival (AoA) estimation to identify BLE-tagged objects. In our system, the front camera of a Mixed Reality (MR) head-mounted display (HMD) provides a live image stream to a vision-based object detection module, while an antenna array that is mounted on the HMD collects AoA information from ambient devices. In this way, BLEARVIS is able to differentiate between visually identical objects in the same environment and can provide an MR overlay of information (data and controls) that relates to them. We include experimental evaluations of both, the CV-based object detection and the RF-based AoA estimation, and discuss the applicability of the combined RF and CV pipelines in different ubiquitous computing scenarios. This research can form a starting point to spawn the integration of diverse object detection, identification, and interaction approaches that function across the electromagnetic spectrum, and beyond.},
journal = {Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.},
month = {sep},
articleno = {124},
numpages = {26},
keywords = {computer vision, detection, identification, mixed reality}
}

Nowadays, people encounter personalized services predominantly on the Web using personal computers or mobile devices. The increasing capabilities and pervasiveness of Mixed Reality (MR) devices, however, prepare the ground for personalization possibilities that are increasingly interwoven with our physical reality, extending beyond these traditional devices. Such ubiquitous, personalized MR experiences bring the potential to make our lives and interactions with our environments more convenient, intuitive, and safer. However, these experiences will also be prone to amplify the known beneficial and, notably, harmful implications of personalization. For instance, the loss of shared world objects or the nourishing of "real-world filter bubbles" might have serious social and societal consequences as they could lead to increasingly isolated experienced realities. In this work, we envision different modes for the sharing of personalized MR environments to counteract these potential harms of ubiquitous personalization. We furthermore illustrate the different modes with use cases and list open questions towards this vision.

Jannis Strecker, Simon Mayer, and Kenan Bektas. (2023). Sharing Personalized Mixed Reality Experiences. In P. Fröhlich & V. Cobus (Eds.): Mensch und Computer 2023 – Workshopband. 03.-06. September 2023. Rapperswil (SG). https://doi.org/10.18420/muc2023-mci-ws12-263

@inproceedings{strecker2023a,
title = {Sharing {{Personalized Mixed Reality Experiences}}},
booktitle = {Mensch Und {{Computer}} 2023 \textendash{} {{Workshopband}}},
author = {Strecker, Jannis and Mayer, Simon and Bekta{\c s}, Kenan},
year = 2023,
publisher = {{Gesellschaft f\"ur Informatik e.V.}},
doi = {10.18420/muc2023-mci-ws12-263},
langid = {english}
}

Head-mounted Augmented Reality (AR) displays overlay digital information on physical objects. Through eye tracking, they allow novel interaction methods and provide insights into user attention, intentions, and activities. However, only few studies have used gaze-enabled AR displays for human activity recognition (HAR). In an experimental study, we collected gaze data from 10 users on a HoloLens 2 (HL2) while they performed three activities (i.e., read, inspect, search). We trained machine learning models (SVM, Random Forest, Extremely Randomized Trees) with extracted features and achieved an up to 98.7% activity-recognition accuracy. On the HL2, we provided users with an AR feedback that is relevant to their current activity. We present the components of our system (GEAR) including a novel solution to enable the controlled sharing of collected data. We provide the scripts and anonymized datasets which can be used as teaching material in graduate courses or for reproducing our findings.

Kenan Bektaş, Jannis Strecker, Simon Mayer, Kimberly Garcia, Jonas Hermann, Kay Erik Jenss, Yasmine Sheila Antille, and Marc Elias Solèr. 2023. GEAR: Gaze-enabled augmented reality for human activity recognition. In 2023 Symposium on Eye Tracking Research and Applications (ETRA ’23), May 30–June 02, 2023, Tubingen, Germany. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3588015.3588402

@inproceedings{10.1145/3588015.3588402,
author = {Bekta\c{s}, Kenan and Strecker, Jannis and Mayer, Simon and Garcia, Dr. Kimberly and Hermann, Jonas and Jen\ss{}, Kay Erik and Antille, Yasmine Sheila and Sol\`{e}r, Marc},
title = {GEAR: Gaze-enabled augmented reality for human activity recognition},
year = {2023},
isbn = {9798400701504},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3588015.3588402},
doi = {10.1145/3588015.3588402},
abstract = {Head-mounted Augmented Reality (AR) displays overlay digital information on physical objects. Through eye tracking, they allow novel interaction methods and provide insights into user attention, intentions, and activities. However, only few studies have used gaze-enabled AR displays for human activity recognition (HAR). In an experimental study, we collected gaze data from 10 users on a HoloLens 2 (HL2) while they performed three activities (i.e., read, inspect, search). We trained machine learning models (SVM, Random Forest, Extremely Randomized Trees) with extracted features and achieved an up to 98.7\% activity-recognition accuracy. On the HL2, we provided users with an AR feedback that is relevant to their current activity. We present the components of our system (GEAR) including a novel solution to enable the controlled sharing of collected data. We provide the scripts and anonymized datasets which can be used as teaching material in graduate courses or for reproducing our findings.},
booktitle = {Proceedings of the 2023 Symposium on Eye Tracking Research and Applications},
articleno = {9},
numpages = {9},
keywords = {attention, augmented reality, context-awareness, human activity recognition, pervasive eye tracking},
location = {Tubingen, Germany},
series = {ETRA '23}
}

To enable people to interact more efficiently with virtual and physical services in their surroundings, it would be beneficial if information could more fluently be passed across digital and non-digital spaces. To this end, we propose to combine semantic technologies with Optical Character Recognition on an Augmented Reality (AR) interface to enable the semantic integration of (written) information located in our everyday environments with Internet of Things devices. We hence present SOCRAR, a system that is able to detect written information from a user’s physical environment while contextualizing this data through a semantic backend. The SOCRAR system enables in-band semantic translation on an AR interface, permits semantic filtering and selection of appropriate device interfaces, and provides cognitive offloading by enabling users to store information for later use. We demonstrate the feasibility of SOCRAR through the implementation of three concrete scenarios.

Jannis Strecker, Kimberly García, Kenan Bektaş, Simon Mayer, and Ganesh Ramanathan. 2022. SOCRAR: Semantic OCR through Augmented Reality. In Proceedings of the 12th International Conference on the Internet of Things (IoT ’22), November 7–10, 2022, Delft, Netherlands. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3567445.3567453

@inproceedings{10.1145/3567445.3567453,
author = {Strecker, Jannis and Garc\\'{\\i}a, Kimberly and Bekta\\c{s}, Kenan and Mayer, Simon and Ramanathan, Ganesh},
title = {SOCRAR: Semantic OCR through Augmented Reality}, 
year = {2023}, 
isbn = {9781450396653}, 
publisher = {Association for Computing Machinery}, 
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3567445.3567453},
doi = {10.1145/3567445.3567453},
abstract = {To enable people to interact more efficiently with virtual and physical services in their surroundings, it would be beneficial if information could more fluently be passed across digital and non-digital spaces. To this end, we propose to combine semantic technologies with Optical Character Recognition on an Augmented Reality (AR) interface to enable the semantic integration of (written) information located in our everyday environments with Internet of Things devices. We hence present SOCRAR, a system that is able to detect written information from a user’s physical environment while contextualizing this data through a semantic backend. The SOCRAR system enables in-band semantic translation on an AR interface, permits semantic filtering and selection of appropriate device interfaces, and provides cognitive offloading by enabling users to store information for later use. We demonstrate the feasibility of SOCRAR through the implementation of three concrete scenarios.},
booktitle = {Proceedings of the 12th International Conference on the Internet of Things},
pages = {25–32},
numpages = {8},
keywords = {Augmented Reality, Knowledge Graph, Optical Character Recognition, Ubiquitous Computing, Web of Things},
location = {Delft, Netherlands},
series = {IoT '22}
}

Mixed Reality (MR) is becoming an integral part of many context-aware industrial applications. In maintenance and remote support operations, the individual steps of computer-supported (cooperative) work can be defined and presented to human operators through MR headsets. Tracking of eye movements can provide valuable insights into a user’s decision-making and interaction processes. Thus, our overarching goal is to better understand the visual inspection behavior of machine operators on shopfloors and to find ways to provide them with attention-aware and context-aware assistance through MR headsets that increasingly come with eye tracking (ET) as a default feature. Toward this goal, in two industrial scenarios, we used two mobile eye tracking devices and systematically compared the visual inspection behavior of novice and expert operators. In this paper we present our preliminary findings and lessons learned

Kenan Bektas, Jannis Strecker, Simon Mayer, and Markus Stolze. 2022. EToS-1: Eye Tracking on Shopfloors for User Engagement with Automation. In Proceedings of the Workshop on Engaging with Automation co-located with the ACM Conference on Human Factors in Computing Systems (CHI 2022), April 30, 2022, New Orleans, LA, USA. https://www.alexandria.unisg.ch/266339