Although many commercial eye-tracker manufacturers do not provide exact documentation of their pupil detection method, there are a number of published algorithms. In the following, we will provide a summary of the workflow for a selection of algorithms. For a more detailed overview and comparison of the state-of-the-art we refer the reader to a recent review by Fuhl, Tonsen, et al. ( 5 ). In the following we will briefly discuss details of some of these algorithms, namely Świrski, Bulling, and Dodgson ( 12 ), Else Fuhl, Santini, Kübler, and Kasneci ( 4 ), and ExCuSe Fuhl et al. ( 2 )due to their good performance in prior evaluations Fuhl, Tonsen, et al. ( 5 ) and their conceptual differences. ElSe Fuhl, Santini, Kübler, and Kasneci ( 4 )was chosen as the currently best performing state-of-the-art method Fuhl, Geisler, Santini, Rosenstiel, and Kasneci ( 1 ). We also include the Set algorithm Javadi et al. ( 7 ) as a representative of simple, threshold-based approach.

The Exclusive Curve Selector (ExCuSe)Fuhl et al.( 2 )first analyzes the image based on the intensity histogram with regard to large reflections. For images with such reflections, the algorithm tries to find the pupils outer edge, otherwise this step is skipped. To localize the pupil boundary, a Canny edge filter is applied and all orthogonally connected edges are broken at their intersection. This is done by applying different morphologic operations. All non-curvy lines are then removed. For each curved line, the average intensity of its enclosed pixels is computed. The curved line with the darkest enclosed intensity value is selected as pupil boundary candidate and an ellipse fit is applied to it. If the previous step did not yield a clear result or was skipped, a binary threshold based on the standard deviation of the image is applied. For four orientations, the Angular Integral Projection Function Mohammed, Hong, and Jarjes ( 10 ) is calculated on the binary image and an intersection of the four maximal responses is determined. This intersection location is further refined within the surrounding image region by using similar or darker intensity values as attractive force. Based on the refined position, the surrounding image region is extracted and a Canny edge filter applied. These edges are refined using the binary image obtained by applying a calculated threshold. Beginning at the estimated center location, rays are send out to select the closest edge candidates. The last step is a least squares ellipse fit on the selected edges to correct the pupil center location.

The Ellipse Selector (ElSe) Fuhl, Santini, Kübler, and Kasneci ( 4 )begins by applying a Canny edge filter to the eye image. Afterwards, all edges are filtered either morphologically or algorithmically. In this work, we used the morphological approach due to the lower computational demands. The filter removes orthogonal connections and applies a thinning and straightening with different morphologic operations than ExCuSe Fuhl et al.( 2 ). Afterwards, all straight lines are removed and each curved segment is evaluated based on the enclosed intensity value, size, ellipse parameters, and the ease of fitting an ellipse to it. The last evaluation metric is a pupil plausibility check. In case the previously described step fails to detect a pupil, a convolution based approach is applied. Therefore, a mean circle and a surface difference circle are convolved with the downscaled image. The magnitude result of both convolutions is then multiplied and the maximum is selected as pupil center estimation. This position is refined on the full sized image by calculating a intensity range from its neighborhood. All connected pixels in this range are grouped and the center of mass is calculated.

Set Javadi et al. ( 7 )can be subdivided into pupil extraction and validation. An intensity threshold is provided as a parameter and used to convert the input image into a binary image. All connected pixels below (darker than) the threshold are considered as belonging to the pupil and grouped together. Pixel groups that exceed a certain size, provided to the algorithm as a second parameter, are selected as possible pupil candidates. For each such group the convex hull is computed and an ellipse is fit to it. This ellipse fit is based on comparing the sine and cosine part of each segment to possible ellipse axis parameters. The most circular segment is chosen as the final pupil.

In a first step of the algorithm introduced by Świrski et al. ( 10 ), Haar-Cascade-like features of different sizes are used to find a coarse position for the pupil. To save computational costs this is done on the integral image. The range at which these features are searched is specified by a minimum and maximum pupil radius.

This results in a magnitude map where the strongest response is selected as coarse pupil center estimate. An intensity histogram is calculated on the surrounding region. This histogram is segmented using k-means clustering, resulting thus in an intensity threshold. This threshold converts the image into a binary pixel inside-pupil, outside-pupil image. The largest continuously connected patch is selected as pupil and its center of mass as the refined pupil center location. In the final step, an ellipse is fitted to the pupil boundary. A morphologic preprocessing by an opening operation is applied to the image to remove the eyelashes. Afterwards, the canny edge detector is used for edge extraction. Edges that surround the refined pupil location are selected and an ellipse is fitted using RANSAC and an image aware support function for edge pixel selection.

Each eye-tracking recording is associated with a quite unique mixture of noise components. Therefore, artificial eye models and image synthesis methods for eye-tracker images were created in order to produce mostly artifactfree recordings. Świrski and Dodgson ( 13 ) even model and render the complete head to generate data for remote as well as head mounted eye trackers. The model renders as physically correct as possible, including reflections, refraction, shadows and depth-of-field blur together with the facial marks like eyelashes and eyebrows.

Wood, Baltrušaitis, Morency, Robinson, and Bulling( 17 ) used rendered images for estimating the gaze of a person using a k nearest neighbor’s estimator. For fast rendering they employed the Unity game engine. Furthermore, the authors generated data for different skin colors, head poses, and pupil states. The accuracy of this concept was further improved by Zhang, Sugano, Fritz, and Bulling ( 18 ) using a convolution neuronal network trained on the complete face.

The work by Kübler, Rittig, Kasneci, Ungewiss, and Krauss ( 9 ) advances in a different direction. More specifically, the authors evaluate the effect of eyeglasses on traditional gaze estimation methods by including the optical medium into the simulation model. The authors showed that eyeglasses have a major impact on gaze direction predicted by a geometrical model, but not on that of a polynomial fit.

To get an impression of the impact of dust during realworld eye-tracking, we browsed about 30 datasets from a real-world driving experiment Kasneci et al. ( 8 ). Dust particles are almost invisible on still images, but become clearly visible in a video. This is due to the static behavior of dust while the eye is moving. Figure 1 shows some examples of dust we found in the dataset.

In 2005, Willson et al. first described a method for the simulation of dust particles on optical elements in Willson, Maimone, Johnson, and Scherr ( 16 ). They formulated a camera model, derived the influence of dust particles on the final image, and specified formulae to calculate these effects. However, their particle model and the final image synthesis were still lacking some of the occurring effects: particles were modeled as circular achromatic shapes that were distributed on a plane perpendicular to the image sensor. We extended their work by modeling particles as triangulated objects with color and positions in 3D, which allows distribution in potentially arbitrary 3D-subspaces, e.g. curved and rotated planes. By modeling particles as a set of triangles and adding color information, the formulae for intersection calculation and color blending are significantly different and more expensive to compute. To calculate the final image in reasonable time, a rtree is used to speed the location of particles close to a specified point.

As presented by Willson et al. ( 16 ), the influence of dust particles on the final image depends on the camera model and the dust particle model. These models are presented in the following subsections.

In real cameras, an aperture controls the amount of light and the directions from which light is collected. This bundle of light rays is called the collection cone. Such a camera model that respects was employed in this work to vary the depth of field, to gain naturally blurred images of objects that are out of focus and control for the amount of light and blur, Figure 2.

We model dust particles based on position, color (including an alpha channel for transparency), shape variance, and size. They are randomly distributed on a userdefined plane, not necessarily perpendicular to the image plane. As shape we modify a basic circle by smooth deviations. The final shape is triangulated and the triangulation detail level controlled by specifying the number of edges for each particle. The maximum extent of the dust plane is calculated using the maximum angle of view of the camera. Finally, by setting the number of particles for the next simulation run, the desired particles are distributed over the given plane subset.Using the center of location d_i of the dust particle d̂_i with index i, its radius r_i and the number of edges n, the edge vertices v_i of a circle-like particle can be calculated as formulated in the following equation

where 0 ≤ j ≤ n. These vertices are then appended to form a polygon. A similar approach is used for varying the particle shape. Setting a property value k ∈ [0; 2], which controls the scale of the shape variance, a new radius is calculated for each of the edge vertices by

These randomly shaped particles are then smoothed by using simple interpolation between two edge points. For each particle at location di, the left and right neighbor (d_i-1 and d_i+1) are taken into account. Finally, the edge vertices are smoothed using the equation

Every image that enters the simulation has already been recorded with a real camera. The parameters chosen for the simulation should therefore be as close as possible to the real recording camera. The appearance of the dust particles will only yield correct results if this condition holds. For each pixel p_x,y, on the output image, the following steps are performed to calculate the final pixel output color.

First, the intersection point p_w of the light ray starting at the pixel at p_x,y, and leaving through the center of the aperture towards the scene with the dust plane needs to be found. In case of perpendicular planes, this can be calculated rather easy using similar triangles Willson et al. ( 16 ). If the plane can have arbitrary geometry, it is best calculated using ray-plane-intersection.

Second, the projection of the collection cone at the point p_w needs to be calculated. This is done by projecting the triangle edge points of the aperture onto the plane, gaining a projected polygon c_w of the collection cone section with area A.

To determine the influence of the dust particles on the final output color, all surrounding particles that satisfy the condition ‖p_w − d_i‖ < α + 2 ∗ r_i are retrieved. They are referred to as the subset C of dust particles in the following. These particles potentially have an influence on the final pixel color. To calculate the magnitude of that influence, for each particle d_i ∈ C, the intersection area A_i with c_w is calculated. If we assume that the equation for the overall area holds:

The fraction α_i = A_i/A is the alpha-blending factor of d̂_i and determines the amount of its color contributing to the final pixel color. Therefore, if a particle has huge overlap with the current collection cone, the final output color of that pixel is strongly mixed with the particle color. Figure 3 visualizes this process.

For fast retrieval of the particles close to c_w, the boost implementation of an rtree is used. Further, since the blending factors are constant as long as the camera parameters remain the same, an attenuation image is calculated that can be applied to all subsequent images of a stream. The generation of the attenuation image is computationally expensive, whereas the application to an image can be done in real-time. The attenuation image contains the alpha-blending values and the color information for each pixel on the sensor and is valid as long as the camera parameters remain fixed.

We evaluated our approach on a subset of the data set by Fuhl et al. ( 2 ), namely data set X, XII, XIV, and XVII (Figure 4). Based on visual inspection, these data sets were found to be mostly free of dust particles and provided thus good baseline results for all of evaluated algorithms. A total of 2,101 images from four different subjects were extracted. These images do not contain any other challenges to the pupil detection such as make-up or contact lenses, since we wanted to investigate the iso-lated influence of dust and dirt. However, all subjects wore eyeglasses and the ambient illumination changed. Furthermore, we did not use completely synthetic images as comparable results can only be achieved within strict laboratory conditions, where dust would usually simply be removed from the recording devices.

Figure 5 shows the influence of the focal length on the final image. It should be noted here that in a realistic scenario the focal length would also have an influence on the image of the eye, not just on the particles. This effect was omitted here (visible for example at the eyelashes). For the images in Figure 5, a focal length of 5.6 puts the dust particles in focus. For real dust particles this is based on their distance to the camera and depends mainly on the design of worn glasses. Most eye cameras do not employ an autofocus mechanism but provide a possibility of adjusting the focus. However, it is rarely adjusted with dust on the eyeglasses in mind (to our experience also by the manufacturers). In the following, we will use the value of 5.6mm focus as a reference.

Another important aspect of dirt is the size of different particles. This effect is shown in Figure 6. The amount and focal length is fixed to 200 and 5.6 respec-tively. For real world recordings dust can occur in differ-ent sizes for which we used four size groups. As can be seen in figure 6(d) they are not simple dots they are vary-ing polygons.

The effect of the amount of particles rendered can be seen in Figure 7. The particles are spread uniformly over the image. This is one limitation of the current simula-tion, as realistic dust distributions would include the lens lenticular buckle of the camera and the curvature of the glasses of a subject.