The main disadvantage of recurrence plots, and at the same time of the GSSP presented above, is that the upper right part of the plot is a mirror of its lower left part. To avoid such a redundancy and to provide more information on the same plot we propose the extended version of the GSSP - denoted by GSSP_VH - in which the upper right part of the plot shows horizontal distances between gazes, while the lower left part presents vertical distances. Additionally, we propose to use the directed distances to preserve information not only about the distance, but also about the direction of the distance (e.g. from left to right or from right to left).

If we consider two gazes g (a) and g(b) for which a<b, i.e. g (a) was measured before g(b) , we can calculate horizontally and vertically directed distances as:

Therefore, the general formula for GSSP_VH calculation is:

and every value may be in the range of (−1...1) .

It is worth noting that when condition i > j is fulfilled, it means that the gaze i was after the gaze j , so −dx must be taken as a directed distance.

Two ways to visualize such a matrix may be applied. One is to recalculate values to (0...1) range in the greyscale, similarly to the previous example. However, the main drawback of such an approach is that the distance equal to zero is difficult to distinguish visually, as after the recalculation it is equal to 0.5.

Therefore, we propose a colored plot and encoding each direction using a different color channel. For every point on the plot its color is defined using its three components (R, G, B): red (R), green (G) and blue (B). Every component may have a value in the range of (0...1) where 0 denotes lack of the component.

For instance, movements from left to right and from top to bottom may be characterized by a red component and from right to left and from bottom to top by a green component (but it is also possible to use any other color pattern) (see Figure 2).

For such color encoding every pixel value would be calculated as:

It is worth noting that a point may be only black, red or green and the intensity of a red or green component may change and it is not possible to have a point with both red and green components greater than 0.

Figure 3b presents both types of GSSP calculated for the gaze sequence shown in Figure 3a.

An interesting property of GSSP_VH matrix is that it may be used to reconstruct a scan-path. The only required information is an absolute position of one gaze point. Having such a gaze point g_s (x_s , y_s ) we can calculate an absolute position of any other gaze point g_i (x_i , y_i ) using the following formulas:

Analysis of the above-described plots may reveal a lot of interesting information, which will be shown in further parts of the paper. However, comparison of several of such plots and their assessment based only on visual inspection may be difficult, thus we propose several quantitative metrics for GSSPs comparison. Since the GSSP is in fact an image, the metrics stem from image analysis algorithms.

The calculation of various characteristics of GSSP images has been based on the Co-occurrence Matrix (CM). It characterizes the texture of an image by determining how often pairs of pixels with specific values and in a specified spatial relationship occur in this image [ 7 ]. The size of CM is equal to the number of distinct values derived from the image, thus the calculation of CM must start with discretization of distances encoded in GSSP. All GSSP points must be recalculated from a continuous range of 0..1 to K discrete values forming a new matrix with integer values in range 0..K .

where I(x,y) represents GSSP with recalculated values. Subsequently, CM (K + 1, K + 1) matrix is determined for every pair of values a = 0...K and b = 0...K and for a given offset d = (dx, dy) representing their spatial relationship. For the purpose of this research the value of K was arbitrarily set to 10.

In the case of GSSP_VH CM matrices are calculated separately for horizontal (upper right) and vertical (lower left) parts of the GSSP and are denoted by CM^V and CM^H respectively.

Co-occurrence matrices created in this way may serve to compute various image-related metrics.

The homogeneity of an image gives information to what extend nearby gazes are in similar locations. It is high when values in CM concentrate along the diagonal, meaning that there are a lot of pixels with the same or very similar value. The range of homogeneity is [0,1]. If an image is constant then homogeneity is equal to 1.

The contrast is a difference moment of the CM and it measures the amount of local variations in an image. If the neighboring pixels have similar values then the contrast in the image is low. Therefore, the contrast is sensitive to long jumps from one gaze point to another. The range of contrast is [0, K ² ] where contrast is 0 for a constant image. The contrast is inversely proportional to homogeneity.

Uniformity (also called energy) measures gaze pairs repetitions. It is high when the GSSP contains similar areas, which means that the same paired values with the same arrangement appear repeatedly in the image. It is low when there are no dominant pairs and the CM matrix contains a large number of small entries. The range of uniformity is [0,1], and it is 1 for a constant image.

For the purpose of the presented research all these metrics - homogeneity, contrast and uniformity - were evaluated taking into account the three offsets - vertical (0,1), horizontal (1,0) and diagonal (1,1).

Outliers are visible on the GSSP plot as a bright cross with black square on a diagonal. One look at the GSSP gives information about the overall signal quality. Figure 4 presents the plot with one obvious outlier in the center of the plot and three more possible outliers. Of course evident outliers may be removed by means of simple analytic methods based on velocity thresholds [ 8 ], however the GSSP may be useful for examining the remaining scan-path to check for some less obvious outliers.

Eye movement analysis is usually based on a fixations-saccades sequence extracted from a registered signal. It has been shown that such a sequence structure is sensitive to the fixation detection algorithm settings ( [ 9, 10 ]), and it is difficult to visually check, if the settings used are adequate. It became possible to present the detailed characteristics of fixations and saccades in 2D space on a single plot by means of the GSSP. We applied such a plot to estimate how homogeneous fixations are. On one hand, it may reveal that gaze points constituting the fixation are scattered, if there are shades on a fixation’s square. On the other hand, if two subsequent fixations appear in a similar place, they are visible as one big square. This gives the opportunity to observe a scan-path on a higher level - based on regions of interest instead of on separated fixations. An example is presented in Figure 5 on a plot with seven fixations and only two regions of interest.

One of the main aims of the recurrence plots usage is revealing repeated pattern existing in time series. In the presented solution this feature was applied for the purpose of a registered gaze points analysis. Figure 5 presents the GSSP with a recurrence of gaze points’ placements. They are represented by dark rectangles appearing out of the diagonal line, which means that two groups of gaze points are located in the same place. In contrast to a classic recurrence plot, the proposed approach reveals not only repeated gaze points positions, but also allows - due to the application of the coloring mechanism - to estimate relative positions of the remaining points set. Additionally, the applied strategy of gazes’ presentation makes the simultaneous comparison of recurring fixations durations possible.

Smooth pursuits are much slower eye movements than saccades, occurring when somebody is following with eyes a slowly moving object. Unfortunately, algorithms commonly used for the fixation detection frequently mistakenly classify smooth pursuits as fixations or saccades [ 11 ]. Smooth pursuits are also difficult to visualize. We conduced experiment showing that, based on the GSSP, it is easy to distinguish smooth pursuits and fixations, because edges of rectangles representing the former event are smoother (Figure 6).

Two modes of processing visual information are commonly known: the focal and ambient processing [ 12 ] [ 13 ] which are used for the purpose of two different tasks: exploration and inspection. Short duration fixations followed by long saccades are characteristic for the ambient processing, while longer duration fixations followed by shorter saccades are indicative of the focal processing [ 14 ]. The visualization of eye movement that takes an ambient/focal processing into account is not a simple task. One of the attempts dealing with this issue may be found in [ 1 ], where ambient and focal fixations were distinguished by the usage of different coloring.

Assuming that the GSSP is a good tool for the ambient-focal distinction we have undertaken an appropriate experiment. Figure 7 shows two examples of plots. One of them is an example of ambient processing - a person is looking for something in the scene. Another one is a typical example of a focal processing - only some interesting objects are carefully inspected.

As the GSSP reveals both spatial and temporal patterns on one plot, it may be used to analyze strategies while exploring an image. Figure 8a presents two basic strategies: horizontal and vertical which are easily distinguishable in GSSP_VH shown in Figure 8b.

For the horizontal search strategy a person exploring the scene starts eye movement from the left upper corner and moves eyes to the right, thus all subsequent gaze positions are always to the right or in the same place (i.e. near the left edge of the scene). It is represented by red and black regions in the first row of the upper part of the plot. The whole horizontal (upper right) part of the GSSP_VH consists of subsequent red and green regions of similar size, which indicates that gaze was moving left and right with the similar speed. The part of the plot below the diagonal (visualizing vertical movements) is only black and red with very sparse green components, because vertical eye movements are made only downwards.

In the case of the vertical strategy similar color layout may be found, yet it is visible in lower and upper plot’s parts.

Fixations’ patterns during reading are very specific, which makes the GSSP obtained for reading tasks also very specific. Analyzing the GSSP_VH presented in Figure 9 it may be noticed that vertical movements are only directed downwards, while horizontal ones are both to the left and to the right. Subsequent lines of text are easily distinguishable on the horizontal (upper right) part of the GSSP. It consists of squares with red upper right part and green lower left part which indicates that there were slow movements to the right and then rapid movements to the left (which makes it different from the GSSP for the horizontal search strategy presented in Figure 8b).

Another example of the text reading task is presented in Figure 10. A careful examination of the vertical (lower left) part of the GSSP_VH reveals that the same sequence repeats twice, which means that the same text was read twice. It is not so obvious when looking only at the scan-path.

The subsequent example is a backward reading task. Figure 11 presents the scan-path and the GSSP_VH for such a task. It is visible that this time the horizontal part of the plot consists of rectangles with green upper right corner and red lower left corner which indicates slow movements to the left and rapid ones to the right. However, the pattern is not so clear, as in the case of the normal text reading, because the person was not used to this kind of reading.

The same text was presented to another person and the corresponding scan-paths and GSSP_VH are presented in Figure 12. This time the person had serious problems with reading from right to left and it is visible on the GSSP_VH - the rectangles are not similar to the previous ones - movements to the right and to the left have similar velocity (such as in the case of the horizontal search strategy). It is worth noting that this information is not visible on scan-paths which look similarly in Figures 11 and 12.

Our assumption was that the metrics presented in Method section (contrast, homogeneity and uniformity) may reveal interesting information about the gaze patterns. To check it, all three metrics for [0,1] offset were calculated for the first three seconds of five GSSPs presented in the previous sections (Table 1). This way we were able to compare metrics for normal and smooth pursuit observations (first row) and for ambient and focal observations (second row). The differences are visible for all compared observations, especially in the case of contrast and uniformity. The third row of the table shows comparison between the same metrics calculated for the same gaze pattern presented in Figure 9, but separately for horizontal and vertical directions.

The next step in ascertaining the usefulness of the proposed metrics was utilizing them in distinguishing visual behavior depending on an image type.

The dataset used for this purpose consisted of gaze recordings registered for 18 participants looking at four images - two free observation images denoted as ’bus’ and ’cat’, one image with text to be read (’text’) and one image for which the participants’ task was to count the number of rabbits. All four images are presented in Figure 13. After removing two bad samples the remaining subset formed a dataset consisting of 232 observations.

For each of them the GSSP_VH was created and three metrics - contrast, homogeneity and uniformity - calculated separately (1) for every direction (horizontal - upper right triangle and vertical - lower left triangle) and (2) for three different offsets: (0,1), (1,0) and (1,1). It gave overall 18 attributes derived from one GSSP corresponding to one observation.

During the metrics analysis it occurred that values of uniformity calculated for the same direction (V or H) and for different offsets are highly correlated (Pearson correlation for every pair >.9). Therefore, it was decided to remove those metrics determined for (0,1) and (1,0) offsets from the further studies. After that step there were 14 attributes describing every GSSP_VH (and this way every observation).

The resulting GSSPs for all four images and two exemplary participants are presented in Figures 14 and 15.

The mean values of attributes calculated for each image are presented in Table 2. Because, according to Shapiro-Wilk test, none of the 14 analyzed attributes exhibited normal distribution, the nonparametric Kruskal-Wallis test was utilized to check, if there are differences in attributes values among images. The differences were significant, so the post-hoc pairwise comparison realized by means of Mann-Whitney test was also calculated (see Table 3).

The above-presented results with their statistically significant differences showed that distinguishing image types based on calculated GSSP metrics is possible. To confirm the findings a subsequent step of the analysis was undertaken, in which all 14 attributes were used to associate an observation with an image type. That classification process was performed by means of Random Forest classifier with one leave out cross validation using WEKA implementation with default parameters [ 15 ]. The resulting confusion matrix is presented in Table 4. It is visible that ’text’ and ’task’ were the easiest images to classify (17 out of 18 and 16 out of 18 correct classifications, respectively). On the other hand the ’bus’ and ’cat’ images both representing the free viewing visual pattern - were frequently mistaken with each other.

One of the intensively studied issues regarding utilizing eye tracking methods is revealing eye movement patterns of people with various levels of expertise, which is especially visible in medicine. For this reason the next test aimed to check, if the GSSP may be used to distinguish gaze patterns for laymen and specialists. The dataset utilized in the analysis consisted of eye movement recordings of 8 laymen and 8 specialists looking at 12 X-rays for 5 seconds (the duration was chosen arbitrarily). The set of images included chest X-rays with and without various diseases. Participants’ task was to explore each image and assess it based on four possibilities provided.

Similarly to the previously described case, there was the GSSP_VH created for every observation and three attributes: contrast, homogeneity and uniformity calculated separately for both directions and three different offsets.

In this case it occurred that values of all attributes calculated for the same direction (V or H) and for different offsets are highly correlated (Pearson correlation for every pair >.8). Therefore, only (1,1) offset was taken into account. The mean values of attributes for each group together with Kruskal-Wallis test results are presented in Table 5.

Similarly to the previous experiment, all six attributes of each observation were used to classify it as specialist’s or layman’s. Once more Random Forest classification algorithm using WEKA implementation with default parameters [ 15 ] was applied. The accuracy of the classification was 85% and the confusion matrix is presented in Table 6. Additionally the Detection Error Tradeoff (DET) curve for specialist-layman prediction is presented in Figure 16.

Moreover, when the classification results of the same person were summarized for 12 images observations with the usage of a classic voting algorithm - all participants were classified correctly either as a layman or specialist (8 out of 8 correct for both classes). Such results may be treated as the confirmation of the GSSP usefulness for distinguishing laymen and specialists.

Visualization techniques very often have to deal with problems of large amount of samples. Presenting big numbers of fixations and saccades makes scan-paths or heat maps difficult to analyze, especially in regard to detailed information. The problem may be overcome by analyzing data taking its smaller parts into account. Similar solution may be used in the case of the GSSP. If a gaze sequence (scan-path) is very long (e.g. during watching a movie) it is not necessary to analyze the whole GSSP - the better option is to create multiple GSSPs for successive periods. The idea is visually presented in Figure 17, where parts were selected from the whole GSSP . Such extracted GSSPs may be then compared to find characteristic moments during observation.

That GSSP feature was investigated during the next experiment aimed to check, if it was possible to find out, based on GSSP metrics, if a person was reading a text. For the sake of the experiment a cartoon movie was used. From time to time a foreground text appeared on a screen (see Figure 18).

There were six various texts displayed during the movie with different durations from 7 to 9 seconds and short breaks (2-5 seconds) between subsequent texts presentations. A participant’s task was to watch the movie, but at the same time to read all texts.

The research question was to ascertain, if it was possible to indicate whether a person was reading the text while watching the movie, based on metrics values. To answer this question at first the GSSPs were created for one-second windows with 0.16 second step. Then, all metrics were calculated separately for each GSSP and their values were ascertained in terms of the ’text visibility’-’metrics values’ correlation existence. For this purpose the function defining moments of text presentation was defined as:

where textvisible(t) indicates whether in time t a text was visible on the screen (the function value is 1) or not (function value is 0).

It occurred that Pearson correlation between horizontal contrast and the outcome of textvisible(t) function was 0.46 (see Figure 19) and between horizontal uniformity and the textvisible(t) function values was -0.54 (see Figure 20).

When a participant’s task was defined as: ’watch the movie and do not pay attention to texts’ there was no correlation between metrics and textvisible(t) function results found.

The results presented in Table 2 revealed significant effect among images for all attributes derived from GSSP. The differences were especially visible for attributes calculated for horizontal part of the GSSP.

The post-hoc pairwise comparison realized by means of Mann-Whitney test revealed that there were no significant differences between ’bus’ and ’cat’ observations (Table 3). However, all horizontal attributes values showed significant differences when comparing both free observations (’cat’ and ’bus’) with ’text’ one. On the other hand, there were no significant differences for horizontal contrast and homogeneity between free observations and ’task’ explorations, but there were some significant differences for vertical attributes. Horizontal contrast and homogeneity as well as vertical uniformity significantly distinguishes ’text’ and ’task’ observations.

The classification results presented in Table 4 show that it was possible to differentiate observation based on its purpose. The ’text’ and ’task’ observations were classified correctly with accuracies 88% and 94% respectively while ’cat’ and ’bus’ observations were frequently misclassified.

The careful analysis of the results leads to the following conclusions:

• ’text’ has significantly higher horizontal contrast and lower horizontal homogeneity than other images,

• ’task’ has significantly lower horizontal contrast than other images,

• both ’task’ and ’test’ have significantly lower uniformity and higher vertical contrast than both ’free observation’ images,

• ’free observation’ images have similar attributes values and no significant differences between them were observed,

• it is possible to distinguish the type of observation taking into account only three metrics derived from the GSSP, which was demonstrated using the Random Forest classification algorithm.

The results presented in Table 5 reveal that the uniformity and vertical contrast are significantly lower for specialists whereas the vertical homogeneity is higher. It suggests that specialists’ gaze pattern is more sophisticated - there are different jumps/saccades to different directions and there are no dominant directions, which results in lower uniformity. At the same time the jumps/saccades (especially in vertical direction) are shorter, which results in lower contrast and higher homogeneity. Additionally, the standard deviations of metrics among specialists are lower than among laymen.

Based on those outcomes it may be concluded that specialists observe the image more carefully - they focus their attention on relevant parts of the image (more or less the same for each specialist), whereas laymen just scan the image - using similar and predictable patterns for each image (but specific to each observer).

The classification part of the experiment showed that it is possible to distinguish a layman and a specialist gaze patterns taking into account only three metrics derived from the GSSP. With 12 gaze patterns available for a person the classification algorithm performed perfectly in predicting the person’s level of expertise.

The last (movie) experiment described in the previous section leads to the conclusion that the proposed technique is scalable towards long sequences of recordings. By dividing them into shorter series with the application of arbitrarily defined windows, within - as well as between - series comparison is facilitated. Additionally, the results obtained showed the usefulness of the proposed metrics, with the example of the horizontal contrast and horizontal uniformity metrics, which may be good indicators, if a person is reading a text.

The research presented in this paper was partially supported by the Silesian University of Technology grant BK/263/RAu2/2016. The authors declare that no conflicts of interest exist.