The size of the human pupil regulates to the amount of light that enters the eye. An increase in luminance therefore results in a fast constriction of the pupil, a luminance decrease in a gradual ‘unconstriction’. The pupillary light reflex (PLR) ( 1 )regulates the light influx. On the other hand, there is a well-studied correlation between pupillary dilation and cognitive factors such as workload( 2 ), surprise( 3 ), attention( 4 ), and emotional arousal( 5 ).

Pupil dilation constitutes a proxy for indirect measurement of these cognitive factors, which would otherwise only be visible with costly and intrusive measurements such as EEG. Through behavioral observation, however, one can only measure the superposition of PLR and cognitive influences. Both, ‘unconstriction’ due to a luminance change and pupil dilation due to an increased arousal level result in a larger pupil. This fact effectively limits the usefulness of pupil dilation in most practical applications, as an equiluminant surrounding is only realistic for laboratory experiments.

Not only are the causes of PLR unconstriction and cognitive pupil dilation different, but also different brain regions trigger them and they manifest through different components of the eye musculature. PLR is driven mainly by the constriction and relaxation of the iris sphincter muscle; cognitive changes innervate the iris dilator muscle( 1 ),( 6 ).

With their work on the ‘index of cognitive activity’ (ICA) Marshall demonstrated that these processes are in fact so different in their manifestation (especially in the speed and acceleration of dilation and constriction) that sophisticated signal processing can separate cognitive from PLR caused pupil size changes( 7 ). Schwalm et al. later used this method to distinguish between mental workload levels of drivers in a simulator( 2 ). In the context of driving, the study of pupillary dilation has mostly focused on mental workload( 8, 9 ).

Driving is generally considered a foveated task (i.e., an object generally needs to be fixated by the driver in order to be perceived). However, drivers can perceive certain potential hazards without or shortly before an explicit fixation. On the other hand, several studies have empirically shown that the mere fixation of a specific object does not imply its perception not its interpretation as hazardous by the driver. For example, in ( 10 ) hazard fixation was found to be unreliable for predicting hazard perception, as an object can either be ‘cognitively overlooked’ or incorrectly judged as non-hazardous by the driver. Thus, if we want to infer information on hazard perception in a driving scene, the fixation-based information is not sufficient. Other physiological signals, such as electrocardiography (ECG) or Galvanic Skin Response (GSR), can help us to disambiguate. However, they usually show a variable and relatively long delay (within several seconds) so that they are not applicable to a realtime use case, e.g. to trigger assistance systems, nor to determine the exact moment in time when a hazard is perceived. Contrary to these physiological parameters, pupil response happens almost instantly and spans only about 2 seconds( 11 ). This lack of a delay allows for a timely interaction with in-vehicle systems.

In this study, we investigate the pupil dilation in immediate response to a hazard during driving. Our aim is to investigate the predictive quality of the pupillary signal to infer hazard perception. Being able to detect hazard perception of a driver reliably via a change in pupil diameter is interesting for multiple reasons: In Underwood, Ngai & Underwood, 2013 the authors perform a hazard perception task where subjects are to press the space bar once they perceive a hazard. Similar experimental setups are common in studying hazard detection, e.g., in ( 12 ) subjects were to honk upon detection of a pedestrian. We could substitute such artificial manual feedback by a noninvasive measurement of pupil dilation. Furthermore, we could disambiguate other stress signals, such as hazard fixation, heart rate changes or the galvanic skin response by use of the pupil diameter: was an object perceived and judged as hazardous?

For the purpose we can built on insights gained from the analysis of mental workload during driving, as the identification of a stress response shares the common problem of isolating a cognitive pupillary dilation from the PLR. For example in( 13 ), the authors find an increase in pupillary dilation with mental workload that is reliable even under the daylight variations of a natural environment. However, the detection of this effect is only possible through averaging over of a large amount of data and by applying statistical methods. Finding a statistically significant difference in a large collection of data does not imply that a useful classification of individual trials towards a specific mental workload state is possible.

In this context, the Index of cognitive activity is of much interest, as its authors claim that it is almost immune to illumination changes. Therefore, a wavelet transformation filters only those pupil changes that did not originate from ambient illumination changes. By analyzing only certain components of the wavelet-transformed signal, we filter for a specific dilation speed and amplitude( 7 ).

For determining a stress level, increasing mean values of the pupil diameter over time are commonly used( 14 ). This averaging has the advantage of being relatively robust towards momentary pupil diameter changes as caused by rapid illumination changes. Pedrotti et al. used a wavelet transformed pupil diameter in a simulated driving task in order to classify different stress levels of the driver( 15 ). Such a procedure is useful when a gradual change in stress level is expected. However, for our application, we are interested in spontaneous, fast stress events and an average filter would delay the detection of the expected steep and short peaks.

In the following, a filtering and classification cascade for the pupil diameter signal is introduced that can be utilized to classify the perception of hazards during driving in a simulator.

Figure 5 shows the detailed results of the classification for each subject. A white circle indicates a change in pupil diameter that the classificatory judged relevant; a black circle indicates that such a change was not detected. The surrounding square indicates whether the driving instructor judged the driving response as adequate or not. Both markers have to be considered in conjunction. For example, a black square and black circle indicate a situation that the driver did not perceive and, consequently, did not react to. A white square with a white circle would correspond to a hazard that the subject responded to adequately and that caused a pupil dilation.

Figure 6 shows the ROC curves for the classification, separately for the patient and the control group. As there were very few inadequate driving responses in the control group, we can expect the curves to differ even in the case that the visual field defect does not have any effect on the pupil diameter.

Such a prediction assumes that a hazard to which the driver reacted was perceived and via versa a hazard that was not reacted to adequately was overlooked by the driver. From previous analyses of vital parameter data we know that this was not always the case, e.g., some drivers responded inadequately to a hazard they had perceived. Therefore, we cannot expect a perfectly reliable classification result. For our analysis, we decided to predict as many of the hazardous situations from the pupil data as possible and allowed for a moderate number of false positives (so we judge in favor of hazard perception in case of doubt). The numerical classification results are provided in Table 1.

When we analyze those situations that lead to a failure of the driving test, we can now distinguish between a perceptual failure and a behavioral failure: Subject PH11 fails the first situation without perceiving the hazard. The same subject also fails the sixth situation, but this time perceived the hazard, as a pupil diameter change happens. This might be due to a general awareness of a dangerous situation without knowledge about the exact location. Just as interesting is that we can also derive that PH07 showed an adequate driving behavior to the first and seventh situation, even though the hazardous object was likely not perceived. Being able to include such events in the evaluation of driving performance will allow us to better judge driving safety also for subjects with a more defensive driving style that would require extensive testing before the perceptional deficit becomes obvious in terms of a driving test failure.

Table 2 gives some insights to the false positives that influence the ROC curves and classification results. We can observe that the classification step performs well in filtering only few events from many candidates (e.g., from 86 to 7 for subject CG66). It returns an average of 8±8 false positives, i.e. it classified pupil size peaks as a stress response to a hazard that were not associated with one of the predefined hazardous situations. Without the classifier, an average of 50 false peaks per drive would be reported. For the task at hand we aimed at predicting hazard perception at the predefined hazardous situations. It is possible that some subjects were very careful at several other situations along the route that looked like potential hazards, and therefore, showed valid additional stress responses that we are (wrongly) counting as false positives. In order to decide in favor of hazard perception we accepted a relatively high number of false positives along the complete drive..

Hazard perception involves the input of sensory information and subsequent cognitive processing. This processes result in the identification of potentially dangerous traffic situations. Only the combined process of seeing and identifying a hazard will lead to a stress response. We found that pupil dilation can be utilized to disambiguate hazard perception and adequate driving reaction in a simulated driving scenario.

We employed a filtering and classification cascade that is able to identify sudden stress responses from the pupil data. We aimed at correctly detecting as many of the hazardous situations as possible from the pupil diameter only while trying to minimize the false positives. This allows us to determine whether the subject likely perceived a hazard. Due to the number of false positives during the drive it could not be used as a stand-alone detection system for hazardous situations, e.g., to trigger assistance systems. It only indicates the perception of the driver, if such an event has occurred. We designed the hazard situations to be easily overlooked by the driver and to resemble a looming emergency. They are therefore very attention arousing and stress inducing. For less challenging scenarios where the driver can detect hazardous objects earlier and sufficient reaction time is available, no stress signals would be expected. Eye-tracking measures would then be sufficient.

Pupil dilation events were more absent in those situations that were relatively difficult (e.g. 1, 2 and 6, where subjects actually failed the driving test or had only few reaction time available). That indicates that careful, prospective driving behavior may have resulted in a less intense experience of the hazardous situation for some drivers – or that they were simply lucky to have passed the situation. We further found that there is a large individual variation in the number of predicted stress peaks per subject, likely associated with the level of engagement and emotional arousal of the driver during the test scenario.

We showed that the pupil variation events occur with the detection, recognition and reaction to potentially dangerous events while driving. It indicates the moment at which a potentially dangerous event becomes relevant to awareness. Furthermore, the pupil dynamics can resolve the ambiguity of perception and unexpected uncertainty that plays an important role in detecting and recognizing unexpected dangerous events( 25 ),( 26 ).

As brightness within a simulated world (road surface, sky, vegetation, etc.) varies only about ±5% from the average brightness( 27 ), we can currently not conclude as to whether and to what extend these findings may hold for on-road driving. Yet, the indicator may be useful for studies that require a precise distinction between hazard perception and the behavioral driving response without requiring unnatural behavior such as pressing a button upon detection. The approach may also be used to assess the design of a simulator track as to whether a timely detection of planned hazard scenarios is possible.

The author(s) declare(s) that the contents of the article are in agreement with the ethics described in http://biblio.unibe.ch/portale/elibrary/BOP/jemr/ethics.html and that there is no conflict of interest regarding the publication of this paper.

The authors would like to thank Daimler AG for the possibility to use the moving-base driving simulator for this study.

We would further like to thank Pfizer and MSD Sharp & Dohme GmbH for supporting and enabling this study. The authors declare that there is no conflict of interest regarding the publication of this paper.

Work of the authors is supported by the Institutional Strategy of the University of Tübingen (Deutsche Forschungsgemeinschaft, ZUK 63)

We acknowledge support by Deutsche Forschungsgemeinschaft and Open Access Publishing Fund of University of Tübingen.