Toyama et al. [ 16 ] proposed using SIFT (Scale-Invariant Feature Transform) features. Points that have high contrast characteristics, or points at the corner, are regarded as key points with highly noticeable features. These are suitable for matching because they are not affected by rotation or scaling. Jensen et al. [ 7 ] applied SIFT features to construct a 3D AOI (Area Of Interest) from eye-tracking data obtained by a head-mounted eye-tracker. Takemura et al. [ 14 ] proposed using PTAM (Parallel Tracking and Mapping) and Chanijani et al. [ 3 ] applied LLAH (Locally Likely Arrangement Hashing) to find feature points. However, these methods require a sufficient number of feature points in the image for accuracy, which may or may not be present depending on the contents of the view camera image.

NAC Image Technology [ 11 ] offers a method using AR markers to create artificial feature points, where AR markers captured in the view camera are matched with spatial coordinates. Tomi and Rambli [ 15 ] also proposed using an eye-tracker with an AR application in the calibration of a head-mounted display. Huang and Tan [ 5 ] used circular patterns as markers. However, large markers could influence eye movements due to their size and appearance. Kocejko et al. [ 9 ] proposed an algorithm to compensate for head movements with three cameras (to observe the eye, scene, and head angle) and LED markers. However, objects of view were limited in the monitor as were the movement of the subjects.

Tobii Technology offered a solution that uses infrared data communication markers. Eight such markers (approximately 30 mm3) are required for position detection, where each marker communicates with the eye-tracking device and matches the image of the view camera with the respective spatial coordinates. However, the size of such markers could have a significant impact on eye movements. Note that this device is not currently available.

Ahlstrom et al. [ 1 ] proposed compensating for head movements using the recorded gaze behaviors in actual driving scenes with a video camera. However, detection of head movement is performed manually for each frame, which is costly. Larsson et al. [ 10 ] applied a gyro, an accelerometer, and a magnetometer. Even though the accuracy has been improved, the synchronization of eye-tracking data and other sensors still remains an issue.

Figure 1 illustrates the overview of the experimental apparatus devised to measure the subjects’ gaze behavior while watching the large screen. Figure illustrates the actual experimental environment.

To record the data of the eye position, we selected NAC Image Technology’s EMR-9 as the eye-tracking device, which includes a view camera attached to the subject’s forehead for video recording. The eye position is indicated by the x–y coordinates in the area recorded by the view camera (Figure 3). Even if the eye position is fixed on a specific item, head movements will cause shifts in the view camera area and the x–y coordinates, leading to difficulties in identifying the target object, as seen in Figure 4a,Figure 4bFigure 4c.

In this paper, a new eye-tracking method is proposed via the creation of artificial feature points made of invisible NIR (near-infrared)-LED markers and image processing. NIR-LEDs are invisible to the human naked eye, therefore reducing their effect on eye tracking despite their presence. At the same time, NIR-LEDs are visible through IR filters, as seen in Figure 5aandFigure 5b. In robot technology, it is popular to use NIR-LEDs to detect locations or to follow target objects [ 13 ]. However, to the authors’ best knowledge, there have been no NIR-LED applications used for eye tracking, which has the potential to enable eye-movement detection even with head movements.

The view camera with IR filters captures the feature points of NIR-LEDs installed on the projection screen. This image can be used to verify the eye position relative to the NIR-LED feature points, which can then be used to calculate exactly what the subject is looking at on the screen by image processing. We call these invisible NIR-LED markers “IR markers” hereafter.

Image processing is another question that requires attention. SIFT features could be a potential option. However, these methods are not adequate for images of IR markers received through the IR filter, because single NIR-LED IR markers are homogeneous and less characteristic, as shown in the image on the right side of Figure 5. As a countermeasure, several patterns composed of multiple NIR-LEDs have been developed as matching templates, as described below. The overall flow is described later.

IR Marker patterns have been created taking into account the four following conditions.

1. Patterns should have a sufficient number of features.

2. Patterns should be composed of the smallest number of markers possible.

3. Patterns should be sufficiently differentiable from one another.

4. Patterns should be easily produced.

To decide on the exact patterns, the similarities between patterns of filtered IR markers (Figure 6) have been schematically calculated. Taking condition 2 into consideration, a three-point pattern was selected from a 5 × 5 dot matrix for each pattern, which was the best balance to ensure noticeable differentiation. Similarities are calculated by Hu invariant moment algorithm (Hu, 1962).

For a two-dimensional continuous function ƒ(x,y) the moment of order (p + q) is defined as Equation 1.

The image moment is the variance value of the pixel centered on the origin of the image. Here, the suffix represents the weight in the axial direction. Subsequently, the centroid is obtained by Equation 2.

The pixel point (x̅,y̅ ) are the centroid of the image ƒ(x,y). Based on the coordinates of this centroid, the moment considering the centroid is obtained by Equation 3.

Further, normalize this moment of centroid by Equation 4 to find the normalized centroid.

where Equation 5

By normalizing, the variance no longer affects the moment value, therefore it is invariant to the scale.

Seven kinds of Hu invariant moment are defined by using the normalized centroid moment, in this study, the moment is calculated by Equation 6

This is the sum of variances in the x-axis direction and the y-axis direction.

Table 1 shows the result of the similarity calculation using the Hu invariant moment.

The template images are on the top of the table and the searched images are on the side of the table; lower matching evaluation scores indicate higher similarity and are represented with red cells. The Hu invariant moment allows checks of both rotational and scale invariance; therefore, relevant combinations of patterns with high similarity scores can be calculated.

Based on the findings, several patterns were chosen and created with IR markers. Specifically, NIR-LEDs and resistors were attached to a solder-less breadboard and were mounted onto a polystyrene board. To ensure the high accuracy of the template matching, it was found that twelve patterns were required to be on the board for at least three patterns to be within the view camera at a given time for image processing. The layout of the IR markers was decided based on the similarity results seen in Table 1, and the actual implementation can be seen in Figure 7a and Figure 7b.

The operation principle and pattern creating method of NIR-LEDs are described in the previous section. In this section, we will introduce the process and the algorithm of calculating the subject’s view point on the screen, derived from the LED points on the screen and the eye positions.

1. Distortion of the image is caused by the lens of the view camera, therefore calibration is performed for each frame of the obtained movie.

2.Apply template matching on the distortion-corrected images of the view camera to detect the IDs of the IR markers and their coordinates.

3. Detect three points with high matching rates, and obtain their coordinates. In order to calculate the line-of-sight positions on the screen, apply affine transformation to the known coordinates of the markers on the screen.

4. Map the corrected eye coordinates on the image projected on the screen (Figure 8).

5. Output the image or movie with the mapped eye positions (format depends on the visual source).

Affine transformation is used to map coordinates of eye positions in the images of view camera onto the screen. Specifically, scaling is required to adjust the difference in resolution between the view camera and the image projected on the screen, rotation and translation are required to compensate the head movements. Affine transformation is a movement and deformation of a shape that preserves collinearity, including geometric contraction, expansion, dilation, reflection, rotation, shear, similarity transformations, spiral similarities, translation and compositions of them in any combination and sequence.

These transformations for point p(x,y) on a plane to be mapped to point p′(x′,y′) on another plane are expressed as Equation 7.

where Equation 8

Α represents a linear transformation, and t represents a translation. Scaling can be expressed as Equation 9.

α and β are scale factors of x-axis and y-axis direction respectively. Similarly, rotation can be expressed as Equation 8.

θ is the angle of rotation in the mapped plane. Scaling, rotation and translation are used in this research because distortion caused by the lens of the view camera is calibrated before affine transformation, and scale factor is common to x-axis and y-axis. Therefore, affine transformation matrix required to detect eye positions are obtained by Equation 9.

Figure 8 illustrates the image of affine transformation used in our method.

Verification experiments were conducted to examine the proposed method’s correlation between the eye position, as seen through the view camera, and the actual projected image. Since our method assumes covering the field camera with a filter, the image from the view camera won’t allow detection of what the subject is looking at. In order to verify the results, template matching was conducted by creating a simulated filtered image, by projecting an identical image of that seen on the view camera onto the screen through an IR filter. The image projected on the screen is shown in Figure 9.

Before conducting template matching of all gaze data, preliminary experiments were conducted to confirm template matching performance. The numbers 1 through 3 were added to the image seen in Figure 9 and projected as shown in Figure 10a and Figure 10b, where the subjects wearing the EMR9 eye tracker were requested to look at them in order. Figure 11 shows an image clipped from the view camera movie during eye-tracking measurements, and Figure 12 represents six template matching results with obtained gaze data.

Gaze behaviors of the subjects were measured with EMR-9 at 30fps, in a zigzag manner from the upper left marker to the lower right marker of the image shown in Figure 9. Subjects could move their heads freely. To verify template matching performance, Affine transformation was manually conducted based on the template shown in the view camera’s image, and eye positions were mapped onto the projected image. Figure 13 represents template matching results, including a comparison with manually mapped eye points.

Approximately 250 eye points were mapped, where data suggests a very high correlation between template matching and manually conducted mapping results, although some deviation does remain.

Let ∆d_i be the distance between the actual eye position and the corrected eye position, where i denotes the i`th eye points. The detection rate of template matching was 98.6%. Averaged ∆d was 15.9 pixels. Note that the resolution of the projected image was 1920 x 1080 pixels. Points containing detection errors can be seen in Figure 14 and the histogram of ∆d_i is represented in Figure 15. More than 90% of ∆d_i are within 30 pixels. The main cause of such errors is due to view camera image capture failures, caused by very quick head motions and camera shake, leading to image blur which prevents accurate template matching. However, for example, ∆d_i of 30 pixels falls within the range of rear combination lamp of the car shown in the top of Figure 17a, Figure 17b, Figure 17cand Figure 17d (a white circle at point A represents 30 pixels). It can therefore be assumed that our method works in practical use.

Our method can also be used for gaze measurement while watching a movie, and output the movie with eye point mapped on each frame automatically. Here, a driving video footage taken from the inside of a vehicle while driving was adopted as a visual stimulus. Figure 16a and Figure 16b shows the images clipped from the movie and Figure 17 shows the images of view camera and the corresponding corrected eye positions mapped on the source movie.