Quantifying dwell time with location-based augmented reality: Dynamic AOI analysis on mobile eye tracking data with vision transformer

  • Julien Mercier Media Engineering Institute (MEI), School of Engineering and Management Vaud, HES-SO, Yverdon-les-Bains; Lab-STICC, UMR 6285, CNRS, Université Bretagne Sud, F-56000 Vannes, France https://orcid.org/0000-0002-5325-3824
  • Olivier Ertz Media Engineering Institute (MEI), School of Engineering and Management Vaud, HES-SO University of Applied Sciences and Arts Western Switzerland, 1400 Yverdon-les-Bains, Switzerland https://orcid.org/0000-0002-9062-8453
  • Erwan Bocher Lab-STICC, UMR 6285, CNRS, Université Bretagne Sud, F-56000 Vannes, France https://orcid.org/0000-0002-4936-7079
Keywords: Mobile Eye Tracking Methodology, Dynamic Area of Interest, Dwell Time, Frame-by-frame analysis, Vision Transformer, Location-based Augmented Reality, Educational Technology

Abstract

Mobile eye tracking captures egocentric vision and is well-suited for naturalistic studies. However, its data is noisy, especially when acquired outdoor with multiple participants over several sessions. Area of interest analysis on moving targets is difficult because A) camera and objects move nonlinearly and may disappear/reappear from the scene; and B) off-the-shelf analysis tools are limited to linearly moving objects. As a result, researchers resort to time-consuming manual annotation, which limits the use of mobile eye tracking in naturalistic studies. We introduce a method based on a fine-tuned Vision Transformer (ViT) model for classifying frames with overlaying gaze markers. After fine-tuning a model on a manually labelled training set made of 1.98% (=7845 frames) of our entire data for three epochs, our model reached 99.34% accuracy as evaluated on hold-out data. We used the method to quantify participants’ dwell time on a tablet during the outdoor user test of a mobile augmented reality application for biodiversity education. We discuss the benefits and limitations of our approach and its potential to be applied to other contexts.

Published
2024-04-29
How to Cite
Mercier, J., Ertz, O., & Bocher, E. (2024). Quantifying dwell time with location-based augmented reality: Dynamic AOI analysis on mobile eye tracking data with vision transformer. Journal of Eye Movement Research, 17(3). https://doi.org/10.16910/jemr.17.3.3
Section
Articles