This section overviews gaze tracking systems under natural light. The systems can be classified into feature-based and appearance-based methods (14).

Feature-based methods extract features such as the iris/pupil center, eye corners and iris/pupil contours. Then, model-based and regression-based methods use the features to track the gaze. Model-based methods (10, 15) use a geometric eye model to compute the gaze direction from the features. Regression-based methods (16, 17) compute a mapping function between the gaze direction and eye vectors.

The performance of model-based methods relies on the accurate detection of the iris center. In (10), the iris center was obtained by an ellipse fitting algorithm, where the ellipse of the iris in the image was described by the yaw and pitch angles. J.Li and S.Li (10) achieved 7.6° and 6.7° in horizontal and vertical directions on the public EYEDIAP database (18) with an execution speed of 3 frames per second (fps) on a 2.5-GHz Inter(R) Core(TM) i5-2400S processor. In (15), the shape of the iris was estimated by ellipse fitting. Then, an accuracy of 7° of the gaze direction was inferred by the hypothesis that the shape of the iris appears to deform from circular to elliptical when the iris orientation changes. Wood and Bulling (15) achieved an execution speed of 12 fps on a commodity tablet computer with a quad-core 2 GHz processor. Ellipse fitting has a low consistency and reliability because iris edges or points cannot be accurately extracted in low resolution images.

In addition to the iris center, the anchor point is one of the key features influencing the accuracy of the regression-based methods. In (16), the eye corner was used as the anchor point. Instead of detecting the eye corners, the anchor point in (17) was set as the center coordinate of the patch which contains the inner eye corners and eyebrow edges. The proposed system yielded a mean accuracy of 2.33° and 1.8° in the horizontal and vertical directions on their self-built database and 7.53° on the public UulmHPG database (19). However, eye corners or the center of the patch cannot be accurately detected in low resolution images with a large head rotation.

Appearance-based methods do not extract specific features and usually learn a mapping function from eye images to gaze directions. In (20), gaze estimation was learned by random regression forests with a significantly larger dataset, which reduced the error by 50% from the work in (21) with an error larger than 10°. In (9), k-nearest neighbor regression and adaptive linear regression were used to learn mapping functions between eye images and gaze directions, which achieved a mean accuracy of 7.2° (keeping the head still) and 8.9° (head movement) on the EYEDIAP database. With the development of deep learning, convolutional neural networks (CNNs) have been used to estimate the gaze with millions of eye images in (22). They proved that a largescale dataset and a large variety of data could improve the accuracy of the appearance-based model for gaze tracking, which achieved errors of 1.71 cm and 2.53 cm without calibration on mobile phones and tablets, respectively. Krafka, et al. (22) achieved a detection rate of 10–15 fps on a typical mobile device. One of the main drawbacks to appearance-based methods is that the appearance of the eyes is significantly affected by the head pose (17). In addition, compared with feature-based methods, appearance-based methods generally require larger numbers of training images.

The eye ROI should be detected before locating the iris center. Facial landmarks can provide more precise positioning of the mouth, eyes, nose, etc. Therefore, an ensemble of regression trees algorithm (23) is used to detect 68 facial landmarks. This algorithm uses intensity differences between pixels to estimate the positions of 68 facial landmarks. The locations of 68 facial landmarks are shown in Figure 2.

The rectangular eye ROIs are next extracted by the twelve points around the eyes. The boundary coordinates of the eye ROIs are computed by the equations in Table 1.

Results in Figure 3 show that accurate eye ROIs can be extracted even when large head rotations occur. Then, the eye ROIs dimensions are magnified by a factor of two. Grayscale erosion with a 1-pixel disk structure element is used in the eye ROIs to delete possible noise. Finally, the simple active contour snakuscule is used to locate the iris centers.

As shown in Figure 4, snakuscule is an area-based circular snake that contains an outer annulus and an inner disk. It performs well in detecting circular regions with the maximum gray difference of the outer annulus and the inner disk. β (Figure 4) is defined as the ratio of outer to inner radius.

Based on the sclera and the iris having the maximum gray difference, snakuscule can expand or shrink to maximize the values of the outer annulus and the inner disk. However, uncontrolled expansion or shrinkage of the snakuscule needs numerous iterations before its final convergence. To overcome the shortcoming, the snakuscule’s inner radius is initialized by the eye anatomical dimensions that the radius of the eyeball is in the range of 12-13 mm (25) and the radius of the iris is approximately equal to an anatomical constant (approximately 7 mm) (26) for most people. In addition, the method works well as the width of the eye ROI extracted by facial landmarks is close to the diameter of the eyeball in the image. Therefore, the snakuscule inner radius is initialized by

where N is the width of the eye ROI and α is a constant that involves the ratio of iris radius to the eye ROI width.

Using the initialized snakuscule, the gray difference of the outer annulus and the inner disk is calculated by use of formula (2) is suggested in (12).

where f(p) denotes the image gray value at the position p. Formula (2) is used to compute the gray difference of G(p_i), where p_i = (x_i, y_i), x_i ∈ [βr, N-βr], x_i is an integer with a minimum interval of 1, y_i = [M/2] and M is the height of the eye ROI. In other words, the snakuscule is used to compute gray differences from left to right along the horizontal centerline in the eye ROI. The location p_rc(x_rc, y_rc) with the maximum G(p_i) is the rough iris center.

As shown in Figure 5, (2δ+1)×(2δ+1) iris center candidate points are determined by the rough iris center (x_rc, y_rc) in the eye ROI. The unit of δ is the pixel. The iris center candidate points are used to accurately locate the iris center. In other words, in the range of [x_rc±δ, y_rc±δ], (2δ+1)×(2δ+1) gray differences of G were calculated by formula (2). The location p_c(x_c, y_c) with the maximum G of the (2δ+1)×(2δ+1) iris center candidate points was considered as the final iris center.

Anchor point is used as a reference point to compute the eye vector. The use of the inner or outer eye corners as the anchor points is the common approach (11, 16) among regression-based methods for gaze tracking under natural light. However, Sesma et al. (11) showed that the eye corners vibrate with eye rotations, which introduces errors in the gaze tracking. In (17), the anchor point was set as the center of the image patch tracked by the Lucas–Kanade inverse affine transform. However, sometimes eye corners or the center of the patch cannot be accurately tracked because they may be blocked or deformed in an image with a large head rotation. Therefore, a novel anchor point is designed as the reference point in this paper. The anchor point p_a(x_a, y_a) is computed by

where n is the number of facial landmarks and (x_i, y_i) is the coordinate of the i^th facial landmark.

There are two advantages to the anchor point. (1) The anchor point computed by stable facial landmarks does not vibrate with eye rotations and is not blocked with large head movements. (2) The mean of the facial landmarks can reduce the error compared with the single feature point near the eye area.

The action of looking at objects usually involves head movement towards the object, and eye rotation focusing on the object. Fridman et al. (27) used the head pose to track the gaze. However, Kennedy et al. (28) found that gaze tracking merely based on the head pose is neither accurate nor consistent in human-robot interactions. Therefore, gaze tracking should synchronize eye rotation and head movement.

For the past several years, different methods for head pose estimation have been developed. The 2D-3D point correspondence methods achieve robust performance and can address large head movements. Therefore, the OpenCV iterative (Levenberg-Marquardt optimization) algorithm is used to estimate the head pose.

After the eye vectors, head pose and screen coordinates have been obtained, the regression strategy is used to establish the mapping function between them. The linear terms, squared terms, cubic terms and interactions summarized in (29) are widely used for mapping eye vectors to screen coordinates. Unlike the head pose that was used to improve the eye vectors in (30), it is directly introduced in the mapping functions. The mapping functions of n points with a polynomial of n or fewer terms can be expressed by

where g_h and g_v are the POR of the horizontal and vertical directions, the coefficients a_k and b_k are determined by the calibration phase, e_h and e_v are the eye vectors of the horizontal and vertical directions, and h_p, h_y and h_r are the head pose angles of the pitch, yaw and roll, respectively. In this paper, six mapping functions derived by formula (4) are used to estimate the gaze. As shown in Table 2, the mapping functions of No.1 and No.2 use the linear and squared terms of eye vectors. No.3, No.4, No.5 and No.6 mix the linear and squared terms of eye vectors and head pose.

Ocular dominance theory is common and long-standing (31). In (32), a dominant eye is shown to be more accurate on SMI HiSpeed 500-Hz eye tracker systems. In addition, Quartley and Firth (33) found that observers favor the left eye for leftward targets and the right eye for rightward targets. Furthermore, for relatively small eye-in-head rotations, Cui and Hondzinski (34) proved that taking the average POR of the two eyes for gaze tracking is more accurate than using only one eye on remote eye tracker. In addition, one of the eyes may be blocked due to a large head movement. To unify these situations, a weight coefficient is used on the POR of the left and right eyes to revise the final POR of the horizontal g_fh and vertical g_fv directions.

where w is the weight coefficient and g_lh, g_rh, g_lv and g_rv are POR of the horizontal and vertical directions of the left and right eyes, respectively.

The GI4E database (35) consists of 1236 images (800×600) from 103 different participants. Each participant has 12 images in which the participant gazed at different points on the screen. A large number of participants with low resolution images make it suitable for evaluating the performance of the proposed iris center localization method.

The EYEDIAP database contains RGB (640×480), RGB-D and HD (1920×1080) video clips from 16 participants. Continuous Screen (CS), Discrete Screen (DS) and 3D Floating Target (FT) are the stimuli that were used for the participants to gaze at. As shown in Figure 6, on the computer screen, DS target was drawn every 1.1 seconds on random locations and CS target was programmed to move along a random trajectory for 2s. The participants were asked to keep an approximately Static (S) or perform head Movements (M) when they gazed at the visual target. Each participant was recorded for 2 to 3 minutes. The proposed method was implemented on the RGB video clips that contains Discrete Screen with Static (DSS) and Discrete Screen with head Movements (DSM), and Continuous Screen with Static (CSS) and Continuous Screen with head Movements (CSM).

On the EYEDIAP database, the frame-by-frame screen target coordinates, head pose tracking states and eyes tracking states including the eyeballs’ 3D coordinates have been provided in the files of "screen_coordinates.txt", "head_pose.txt" and "eyes tracking.txt", respectively. It is noted that, a total of 52 RGB video clips of 13 participants were used to estimate the gaze in this paper because the 12^th and 13^th participants only recorded the video clips for 3D FT and the 7^th participant’s facial landmarks can be tracked on a small fraction of the entire RGB video clips due to the poor contrast.

The computation of the estimated eye center normalized error by use of formula (6) is suggested in (36).

where d_left and d_right are the distances between the estimated and labelled iris centers of the left and right eyes, and d is the distance between the labeled left and right iris centers. The estimated eye centers in the range of the normalized error e ≤ 0.05 that are equivalent to locate in the pupil can be used for gaze tracking applications (37). Therefore, e ≤ 0.05 is used as the benchmark to evaluate the optimal parameters of the iris center localization method in this paper.

For iris center localization, α, β and δ with different values were used on the GI4E database. The optimal values can be obtained when the number of eyes with a normalized error e ≤ 0.05 reaches the maximum value. Therefore, values for α from 0.21 to 0.25 with the minimum interval of 0.05, β from 1.32 to 1.52 with the minimum interval of 0.04 and δ from 1 to 4 with the minimum interval of 1 were assessed in this paper

As shown in Figure 7, the maximum number of images with a normalized error e ≤ 0.05 is 1222 when α = 0.25, β = 1.4 and δ = 1 or 2.

The performance of different mapping functions was compared on the DSS, CSS, DSM and CSM RGB video clips. Iris centers were detected by the optimal parameters of α = 0.25, β = 1.4 and δ = 1 or 2. The anchor point was computed by formula (3), where the unstable facial landmarks around the mouth and eye areas were removed. Therefore, the parameter n equals 36. Then, the eye vectors for the horizontal and vertical directions were computed by the iris centers and the anchor point. Head pose was estimated by the iterative with six points (9, 31, 37, 46, 49 and 55 in Figure 2). Finally, the six mapping functions listed in Table 2 were used to estimate the gaze from the eye vectors and the head pose.

For each RGB video clip on the EYEDIAP database, the first 1000 frames that the faces could be detected were used as calibration frames, and the remaining frames were used as testing frames. The gaze tracking errors for the 13 participants were computed by averaging the results of the participants’ testing frames. The gaze tracking error of each frame is computed by the POR of the left eye, the original 3D coordinate of the eye gaze screen point and the original 3D coordinate of the left eyeball.

The average gaze tracking errors of 52 RGB video clips that were used to evaluate the optimal mapping functions are shown in Table 3. Overall, the mapping functions of No.4 and No.2 achieved the best results in the horizontal and vertical directions, respectively. In addition, the gaze tracking errors show that δ = 2 performs better than δ = 1. Therefore, the following experimental results that involve the iris centers localization are conducted with δ = 2. Meanwhile, the horizontal and vertical gaze tracking errors in the following experiments are regressed by the mapping functions of No.4 and No.2, respectively.

In this paper, the weight coefficient w ∈ [0,1] with a minimum value of 0.1 was used to compute the gaze tracking errors on the EYEDIAP database. As shown in Figure 8, w = 0.5, 0.6 and 0.5 achieve the best horizontal, vertical and combined gaze tracking errors, respectively. For simplicity, w = 0.5 was used in this paper.

To provide more comprehensive improved results by w for the 13 participants on the EYEDIAP database, the horizontal, vertical and combined gaze tracking errors on the DSS and CSS RGB video clips are shown in Figure 9. The results on the DSM and CSM RGB video clips are shown in Table 4. L (w = 1), R (w = 0) and L+R (w = 0.5) are the gaze tracking error computed by the POR of the left, right eyes and improved, the original 3D coordinate of the eye gaze screen point and the original 3D coordinate of the left and right eyeball. Meanwhile, the Total Frames (TF) and the Detected Frames (DF) are presented in the table. DF is a frame in which the face can be detected. Compared with the average face detection rate (DF/TF) of 86.9% in (10) on the CSM RGB video clips, an average of 97.2% was obtained in this paper. Considering the low quality frames and large head pose variations in the video clips, we believe the face detection rate is robust. As shown in Figure 9 and Table 4, most of the single eye gaze tracking errors are improved by averaging the POR of both eyes. The method achieved the average combined gaze tracking errors 5.5°, 4.6°, 7.2° and 7.6° on the DSS, CSS, DSM and CSM RGB video clips. Compared with the gaze tracking error of 2.9° under natural light in (17). The reason the errors are high is because that the EYEDIAP database has the lower quality eye image (the iris radius ≈4.5 pixels) compared to (17) self-built database (the iris radius ≈9 pixels).

The method was realized by using the C++ language with Microsoft Visual Studio 2017, OpenCV and the dlib (38) library on a laptop with a 2.7-GHz Intel(R) Core(TM) i7-7500 processor and 8-GB RAM. Data from the EYEDIAP database and the laptop camera were used to measure the execution time, which was computed by averaging the processing time of all testing frames. The execution times of the proposed method are shown in Table 5. It is noted that facial landmarks detection includes face detection and landmarks detection. Experiment results show that face detection consumes most of the processing time. Therefore, the original resolution (640×480) was resized to improve the efficiency of face detection. Facial landmarks are tracked on the faces from the raw frames, in which the faces are obtained by use of the rectangular face ROI detected in the resized frames. Unfortunately, the face detection rate of the EYEDIAP database decreased when the resolution is lower than 512×380 because the RGB video clips have small faces. Therefore, the execution speed is 22 fps for the EYEDIAP database. The mode of data from the laptop camera is closer to the practical system, which had an execution speed of 35 fps. Compared to the IR tracker with an execution speed in excess of 100 fps, natural light trackers still have a long way to go to be usable in practice.