Virtual reality has the major goal of presenting a virtual world in a way that resembles reality as close as possible. Recently, head-mounted displays (HMDs) are becoming widely available, providing a suitable display technology for this purpose. To enable a visually pleasant experience without disturbing aliasing artifacts and visible pixel grids, high display resolutions are required. Early HMDs like the Forte VFX 3D (1997, 263 480 2 = 0.25 million pixels) worked at very low resolutions, while modern HMDs like the StarVR (2016, 2560 1440 2 = 7.37 million pixels) have made a huge step forward in this regard. However, the full retinal resolution including a user’s full dynamic field of view (200° horizontally, 150° vertically) would potentially require a resolution of 32k 24k = 768 million pixels [ 1 ]. Such resolutions are neither achievable by current display technology nor are they tractable by current GPUs. In addition, frames need to be displayed with a low latency and at high frame rates to meet the requirements of the display devices and at the same time reduce nausea caused by a perceptual mismatch of the self-induced motion and the visual response (simulator sickness) [ 2 ]. With the ongoing improvements of pixel densities in HMDs and the current inability to render at the required resolutions while maintaining performance, developing new rendering methods to tackle these challenges is urgently required.

Fortunately, the human visual system (HVS) has several limitations which imply that it is not necessary to provide the highest level of detail over the entire visual field. There is a drop of the eye’s visual acuity with increasing eccentricities, where the eccentricity describes the angular deviation from the central optical axis. Thus, one possible approach is to adopt techniques that adjust rendering quality based on the exploitation of a user’s current viewing direction. This process is referred to as foveated rendering.

The visual field can be divided in central and peripheral vision. We go with the definition in [ 3 ], where central vision is defined to include the following areas of the fovea (up to 5.2° from the optical axis), the parafovea (up to 9°) and the perifovea (up to 17°). Larger eccentricities are defined to belong to peripheral vision.

The tracking-based adaptation of rendering quality based on the HVS’ drop in visual acuity requires highly accurate and low-latency eye tracking to determine the point of regard (PoR), i.e., the screen space position currently focused by the user. Moreover, several models of human attention have been used in computer graphics to get a notion of the screen space position without using an active tracking mechanism. Both approaches benefit from insights regarding eye tracking data acquired in a foveated rendering system.

Eye tracking as a way to actively measure the user’s gaze directly has been used by the computer graphics community in various disciplines. A survey on rendering techniques can be found in [ 4, 5 ], while perception-driven geometric processing and mesh simplification are described in [ 6 ]. Another field that uses eye tracking is computational displays, where a survey on various techniques can be found in [ 7 ].

Early work in the field of gaze-contingent and foveated rendering techniques utilized focus assumptions [ 8 ] and visual attention models [ 9, 10 ] instead of employing eye tracking devices. Those early systems su ered from the lacking hardware capabilities for both accurate and low-latency eye tracking as well as computational power to synthesize high-quality images. One of the earliest approaches to speed up rendering using perceptual methods is described in [ 11 ], where eye tracking is used to adapt the sampling frequency on the image plane and in object space in accordance with the spatial acuity of the HVS. Another early system can be found in [ 12 ], adapting the geometric quality of a three-dimensional mesh using an anisotropic simplification system. Most notably this is one of the first publications where binocular eye tracking is used inside an HMD.

However, the eye tracking and graphics hardware was still lacking the necessary accuracy, latency and rendering performance to meet perceptual requirements. Hence, early systems focused primarily on a theoretical analysis, e.g., the general influence of the quality degradation on visual performance, especially on search performance. Watson et al. [ 13 ] demonstrated that image resolution could be reduced by half for peripheral vision without a significant influence on search time. Duchowski et al. [ 14 ] demonstrated that color precision can be reduced for peripheral vision, though not as readily as resolution.

As we take a closer look at the eye tracking data one can ask how precise eye tracking devices need to be in order to be suitable for foveated rendering. Loschky et al. [ 15, 16 ] showed that the update must be started at 5 ms to 60 ms after an eye movement for an image change to go undetected. However, an acceptable delay highly depends on the task of the application and the stimulus size and positioning in the visual field. Ways to measure latency and a discussion on di erent tasks can be found in the work by [ 17 ] and [ 18 ].

Synthesizing images from a 3D scene description is made possible by some basic methods in the field of computer graphics, the most important being rasterization and raybased approaches (ray tracing). A comprehensive description of recent work in the field of perception-driven accelerated and foveated rendering can be found in [ 5 ].

Our analysis is based on the fully adaptive foveated ray tracing technique suggested in [ 19 ]. In addition to the system’s fully adaptive sampling, improvements of temporal stability and a reduction of artifacts in the visual periphery are achieved by incorporating a reprojection technique to improve image quality and fill gaps in sparsely sampled images.

The main idea here is that samples are cached in image space, reprojected to a new view and used to aid the image quality of subsequent frames. In this regard the system most closely relates to temporal anti-aliasing [ 20 ] and the mathematical considerations on how to combine samples temporally [ 21 ]. The unique characteristics of our system come from combining a performance-focused reprojection method based on a coarse geometry approximation with foveated rendering methods, which enables us to generate visually pleasant results at high update rates. The user’s gaze is measured using a binocular SMI eye tracker built into an Oculus Rift DK2 and then used to parameterize the rendering process. The evaluation of our rendering system has shown that the subjective perceived quality is very similar to full ray tracing with benchmarks showing a clearly superior rendering performance.

This paper serves the purpose of extending the short analysis of eye tracking data from our user study which is provided in [ 22 ]. The main objective of this extension is to give better, more detailed insights into the recorded tracking data and a more extensive discussion of the results and the connections between gaze data and subjective perceived quality.

In order to provide context, we describe the basic design of the user study and the recorded eye tracking data. Based on this data we describe how the recorded data is analyzed. Amongst others, this analysis revealed an interesting relation between fixation accuracy and quality ratings for di erent fixation modes.

Although there has been related work on analyzing eye tracking data, there is no work in the context of foveated rendering that thoroughly investigates the link between the subjective perceived image quality, the eye tracking precision and the induced effects when asking users to focus or fixate a target in the image. Tracking precision is measured by the distance between the recorded PoR and the actual location that people were instructed to fixate on predefined paths on a screen. [ 23 ] represent this precision by the standard deviation of these measurements to evaluate their low-cost eye tracking system. [ 24 ] use the glint in the eye to derive a PoR. However, they evaluate the precision in terms of determining the right image quadrant. Although the book by Duchowski [25, ch. 12] contains various strategies to evaluate eye tracking data, the author focuses on di erent aspects like dwell time, saccade detection and denoising.

The main contributions of our work are: 1. An estimation of the tracking precision of an HMDmounted eye tracking device, supported by the evaluation of eccentricity-based quality ratings. 2. An analysis of fixation accuracy based on the data recorded during a quality-focused user study carried out for our foveated rendering system. 3. An analysis of the connection between subjective perceived quality and fixation accuracy, providing possible evidence of the presence of visual tunneling effects and the magnitude of their influence on the user’s perception.

The results are discussed and conclusions are drawn in the according sections at the end of this article, together with some suggestions on how to benefit from our findings in practical systems and how to further improve the suggested methods.

In this section we present the results of our analysis.

As shown above, the precision of the utilized eye tracking device drops with increasing eccentricities. Our solution at this point has been to limit the area accounted for in our considerations to only include fixation targets up to the CVA. Still, this issue could also be approached di erently. One possibility would be to take a deeper look into the calibration step of the eye tracking device. As Figure 4 illustrates, the tracking precision decreases smoothly with increasing eccentricities. It may be worthwhile to analyze di erent calibration procedures for their e ect on tracking precision. Also, our most recent tests have shown other devices to be possibly more capable of capturing accurate gaze data over a larger area. New generation eye trackers will likely improve on accuracy for greater angles. The result of our tracking precision analysis should not be interpreted as a direct measure for tracking precision, even though it seems to be quite accurate. Latency-based deviations, saccades and other possible disturbances have not been filtered from the data. The actual behaviour of the measured gaze deviation however yields a good estimate of the eccentricity-dependent precision fallo .

The lower fixation accuracy that we found for in moving target mode implies that the users PoR was often located within the border area of the foveal region for the small FRC. This exhibits reconstructed (and thus lower-quality) parts of the scene to the user in his/her central vision. Causes for the low accuracy that we measured are tracking latency and possible unpredictabilities of the target’s movement as well as tracking precision itself.

For the subjective perceived quality ratings, we have found that an increase in rendering detail did not always result in improved quality ratings. One possible cause for this is the reprojection method hiding visual artifacts by effectively putting a low-pass filter over them, as even full rendering – as all rendering methods – still is a subsampling of the rendered scene, just with a finer and more regular pixel grid. Thus, it may also contain visual artifacts. A more detailed explanation of blur effects that occur when using reprojection methods can be found in (?). Also, when rating the subjective perceived quality for the fixed targets on the one hand and the moving targets on the other hand, intuition may suggest a worse outcome for the latter because of the larger gaze deviation illustrated in Figure 3 and Figure 5. This assumption is mainly based on the fact that the visibility of image regions rendered at a lower quality is increased when the gaze deviation from the fixation target is higher. However, contrary to this, Figure 6 and Figure 7 reveal the quality ratings for moving target fixation to be better in all tested scenes. We interpret the consistent di erences in subjective perceived quality between fixation modes and their counterintuitive nature when taking tracking precision and temporal effects into account as evidence for the possible presence of visual tunneling effects. This means that visual artifacts that appear in our rendering system are effectively filtered by human perception, which makes them largely imperceptible. Also, there often was a clear tendency towards negative ratings for the small FRC. One possibility to overcome this issue is to enlarge the foveal region proportionally to the occuring eye movement. However, this approach poses a significant challenge. The achieved frame rate is already considered to be critical when it comes to head-mounted displays in general, and it becomes even more critical when incorporating (potentially rapid) eye movements. Contrariwise, increasing the rendering quality results in a performance hit, making it even more di cult to achieve the necessary refresh rate. Thus, this approach is only viable in an environment where enough computational resources are available. This may raise the question why these computational resources should not be initially put into rendering a larger FRC, but it has to be kept in mind that visualization is often only one part of an application and resources also need to be available for other components such as physics, interaction or AI, for example in computer games.

We have also shown results of the subjective perceived quality in the free focus mode (cf. Figure 8). As mentioned above, it becomes clear that subjective perceived quality in this mode degrades with increasing eccentricities. Besides a degradation on average, quality measurements also become rather unpredictable in areas further away from the image center. This may imply that the visual effects that occur through mismatches between the actual PoR and the measured PoR do not have the same e ect for all users. We suggest that this can be attributed mainly to the FRCs, as a large foveal region still presents the most important parts of the image at full detail to the user, while smaller FRCs tend to miss the user’s central vision completely due to tracking inaccuracies.

One of the challenges that are yet unsolved is the issue of HMDs getting out of place in the process of a user study, or, more generally, during the execution of an application or specific task. Even slight movements of the HMD may lead to an eye tracker’s calibration becoming invalid. However, asking the user to repeat the calibration step each time the HMD has moved too much is not a viable option. In opposition to an explicit calibration procedure, having a calibration that is embedded into the task at hand would make HMDs with eye trackers more practical for everyday applications.

In this paper we have analyzed the data recorded by an eye tracking device during the evaluation of our foveated rendering method. We described our evaluation setup as well as the rendering method itself. Tracking precision has been analyzed regarding its angular dependencies, revealing a clear drop of tracking quality for higher eccentricities. Accordingly, quality ratings for free focus mode also show a clear drop towards larger eccentricities. Properties of tracking devices such as this have to be accounted for when implementing foveated rendering methods, as the point of regard is a crucial measurement in such setups. Having measured these inaccuracies of the tracking device, it becomes clear that applications that rely on methods from this field have to adjust the specific parameterizations for the given circumstances.

We have analyzed the ability of users to focus static and moving fixation targets. While we found the PoR being scattered over larger areas for the moving target mode, the results seemed to contradict the intuitive assumption that worse fixations should result in worse quality ratings. The mean quality ratings were best for the moving target mode in all scenes, even though the match between the measured PoR and the actually focused PoR was worse than for the static fixation mode. Even though this may lead to subsampling and reprojection artifacts being exposed to the user, the ratings were still better, which we attribute to the potential presence of visual tunneling effects that are induced by the mental load of the task that has to be carried out, although the task has just been to follow a moving point. effectively, this reduces the user’s field of view.

Thus, there are circumstances which make it possible to reduce visual quality. This is the case in games, where events can be triggered that produce a change in the visuals, or taskdriven environments, where task or navigation complexity may lead to high mental workloads. Moreover, certain events may allow for deriving a hint which part of the scene attracts attention. Thus, visual quality can be reduced even further (Selective rendering). However, attentional models and gaze predictions are far from accurate [ 5 ] However, more recently the flicker observer e ect and the higher temporal resolution for peripheral vision has successfully been used to direct the user’s gaze directly [ 30 ].

Future research in the area of foveated rendering may analyze further how optimal foveal region configurations can be determined and how the point of regard can be optimally placed even with imprecise tracking. Also, it may be possible to exploit visual tunneling effects directly to improve performance or, alternatively, visual quality for central vision. In addition, it may be worth looking into the comparative behavior of tracking devices with di erent update rates for analyses such as the one we have presented here.

We would like to thank NVIDIA for providing us with two Quadro K6000 graphics cards for the user study, the Intel Visual Computing Institute, the European Union (EU) for the co-funding as part of the Dreamspace project, the German Federal Ministry for Economic A airs and Energy (BMWi) for funding the MATEDIS ZIM project (grant no KF2644109) and the Federal Ministry of Education and Research (BMBF) for funding the project OLIVE (grant no 13N13161).

The authors declare that there is no conflict of interest regarding the publication of this paper.