Blascheck et al(9) have categorized the various EM visualization methods into point based, AOI based, and hybrid visualizations. Summaries and issues are as follows.

Existing point based visualization methods, e.g. timeline visualization,(15) scanpath visualization,(16) attention maps,(17) space-time cubes,(17) represent the time-ordered horizontal and vertical coordinates of the EFs occurring on the display. These methods are effective on visualizing the exact EF locations to unravel important regions (in absence of predefined targets) when given static stimuli. However, due to the visual angle error of the eye trackers, it is challenging to map the EFs with small and dynamic multi-element targets making it difficult to apply the point based methods.(18) In addition, our interest is in investigating which moving targets were focused upon rather than the physically fixed area within a display.

On the other hand, existing AOI based visualization methods allows EM analysis based on either pre-defined region or target on the display. The AOI based methods have been categorized into timeline and relational AOI visualizations.(9) Time­line AOI visualizations such as parallel scanpath,(19) scarf plot,(20) and AOI river plot(21) on developing effective methods to represent the AOIs that have been fixated upon at various time intervals. However, these methods are challenging to apply for long duration tasks having large number of targets (e.g. twenty or more targets) and frequent EF transitions between them (e.g. air traffic control task).

Relational AOI visualization methods are more appropriate to handle the issues raised above through visualizations using circular heat map transition diagram,(10) transition matrix,(12) and network visualization.(11) These methods visualize the aggregated EM data by showing the relationship that exists between he AOIS in terms of the EF transitions between them, unlike the timeline approaches. Relational AOI based approaches do not represent the physical location of the AOIs on the display. In detail, in circular heat map visualization,(10) AOIs are represented as segments of a circular layout (using different colors and sizes) and the EF transitions are shown by directed arrows between the circular segments. Transition matrix visualization(12) represents the EF transitions among the AOIs in a tabular fashion. The most appropriate approach to address the issues of a dynamic task is through the network visualization that shows the AOIs as vertices and the EF transitions between AOIs as the directed edges between the vertices of a network.(11, 18, 22, 23)

However, if we try to apply the relational AOI visualization methods, we often run into possible visual clutter issues if there are large number of targets and it can be difficult to represent all the EM characteristics using the existing network approach. The following subsections 2.2. and 2.3 provide summaries of the DNet mathematical framework and how various EM network characteristics can be integrated based on time intervals.

A DNet is as a sequence of static networks (also called networks), where each constituent network is associated with a time interval.(24) If the total time duration of the collected EM data is divided into T time intervals, then a DNet representing such a data is written as DynN={N1,N2,..Nt ,..,NT}, where Nt is the network for time interval t, where t=1,2,..,T.

A network Nt is written as Nt =(Vt ,Et ,Mt ), where, Vt is the set of vertices (AOIs for the present study), Et is the set of edges (EF transitions for the present study) between the vertices, and Mt is the adjacency matrix which contains all edge weights (amount of EF transitions between AOI pairs).

The set of vertices is written as Vt =(v1,v2,…,vmt), where mt is the number of vertices for time interval t. A network can either have directed or undirected edges, although for EM visualization we only consider directed edges. The set Et consists of ordered pairs of vertices (vi,vj) showing that there exists a directed edge from the vertex vi towards vertex vj. Thus, Et ={eij(t)|vi,vj∈Vt ,i≠j}. Lastly, the adjacency matrix is written as Mt =[wij(t)]mt ×mt , where, wij(t) is the weight of the edge eij(t).(25, 26)

Beck et al(24) provided an exhaustive list of various DNet visualization approaches representing EM data. Depending on the representation of the time variable, various visualization approaches have been categorized into two groups: Animation, and timeline visualization. Animation visualization refers to representing a DNet as an animated sequence of networks. Timeline visualization refers to representing a DNet as a sequence of networks in a single image showing the complete sequence of interactions between the targets. In the present work, we have applied the node-link based timeline approach for representing the DNet, because this visualization helps to preserve the mental map and reduces the cognitive load of the observer.(27)

As noted by various researchers,(28, 29, 30) preserving the mental map (i.e. the abstract structural information layout about a network’s elements that an analyst develops in their mind as they visually scan the visualization) helps in tracing the change in vertex properties and edge paths across different time intervals. Additionally, the timeline visualization, using the node-link approach, provides an intuitive and efficient framework for analyzing the change of states of multiple vertices over time.(31, 32)

However, the existing DNet approach only uses the number and duration of EFs on the AOI to measures the AOI importance. As noted in the introduction, these two raw measures may lead to misleading results in case of dynamic targets. Therefore, it is required to consider other target importance measures to address the highlighted issue.

The next section discusses the three target importance measures (from the network science domain) that can be adapted for analysing AOI importance for dynamic scenarios.

The three most popular target importance measures in a given network are indegree, closeness, and betweenness.(14, 25, 33) Mandal et al(18) have shown some possibilities in applying the above-mentioned measures to build a basic foundation for the proposed approach in this article. It is noted that we introduce the “time” element (“t”) within the three vertex importance measures to consider the dynamicity.

Indegree of a vertex is defined as the sum of all incoming weights to it from all other vertices in the network. For the present study, incoming weights can be interpreted as the incoming EF transitions to a given AOI. Thus, indegree for the jth AOI is given as Ij=∑k=1mwkj,(25) where, wkj is the number of EF transitions from the kthAOI to the jthAOI, and m is the total number of AOIs on the display.

We should note that the indegree measure, shown above, is static in nature. As a result, we modified it to develop the dynamic analogous, where the indegree for an AOI is defined for each of the time interval considered in the DNet framework. Thus, the modified indegree measure for the jthAOI for time interval t is calculated as:

(i) Ij(t)=∑k=1k≠jmtwkj(t)

Where, wkj(t) is the number of EF transitions coming for the kthAOI to jthAOI and mt is the number of unique AOIs in the AOI fixation sequence for time interval t. Large indegree value suggests higher importance for an AOI, as it received large number of EFs. Thus, indegree can be interpreted as a measure of direct attention received by an AOI.

However, indegree measure only considers the local structure (direct EF transitions) around a vertex but neglects the global structure of the network.(33, 34) This issue is addressed by the two measures named “closeness” and “betweenness.”

Note that the network science (or graph theory) includes the concept of “outdegree” however, the indegree values and outdegree values are always the same within the EM network since there is no possibility of an inward single EM transition dividing into two or more outward transitions.

Before we understand the two measures, we need to define the concept of distance between AOIs in a network visualizing EM data. In the present case, we define the distance from one AOI (e.g. ‘A’) to another AOI (e.g. ‘B’) as the inverse of the number of EF transitions from AOI A towards B. Thus, a large number of EF transitions between AOIs results in a smaller distance between them in terms of the visual scanning strategy.

The closeness of a vertex measures its distance from all other vertices in the network. Thus, higher closeness value for a given vertex means it is easier to access any part of the network from it. In the present study, higher accessibility of an AOI can be interpreted as a greater association (both direct and indirect) with other AOIs in the network. High closeness value suggests that the AOI lies in the central location in terms of the observer’s visual scanning strategy. The closeness for the jth AOI is given as Cj=∑k=1m(1djk*),(33) where, djk* is the minimum distance from the jthAOI to the kthAOI (if multiple paths exist) and m is the total number of AOIs on the display. Like the previous indegree measure case, the static closeness measure is also modified to develop its dynamic analogous. Thus, the modified closeness measure for the jth AOI for time interval t is defined as follows:

(ii) Cj(t)=∑k=1k≠jmt1djk*(t)

Where, djk*(t) is the minimum distance from the jthAOI to the kthAOI (if multiple paths exist) and mt is the number of unique AOIs in the AOI fixation sequence for time interval t.

In a dynamic scenario, there are instances where due to visual scanning strategy of the observer, an AOI despite receiving small amount of direct attention (low indegree measure) and being present in a non-central location (low closeness value) can still play a significant role by connecting (acting as a bridge between) two groups of AOIs. Such an AOI plays a crucial role in controlling the flow of attention among the other AOIs on the display, and this aspect is measured through the concept of betweenness explained below.

Betweenness for the jth AOI is defined as Bj=∑l=1m∑k=1,j≠l≠km(SPkljSPkl),(33) where SPkl represents the total number of shortest paths (if multiple paths exist) from the kthAOI to the lthAOI, and SPklj represents the number of such shortest paths that pass through the jthAOI. Thus, the modified betweenness measure for the jth AOI for time interval t is defined as follows:

(iii) Bj(t)=∑k=1mt∑l=1j≠l≠kmtSPklj(t)SPkl(t)

Where, SPkl(t) represents the total number of shortest paths (if multiple paths exist) from the kthAOI to the lthAOI, and SPklj(t) represents the number of such shortest paths which pass through the jthAOI for the time interval t.

The input for the first step is the simulation experimental data. As output for this step, two types of data are obtained: (a) EM data, that consists of horizontal and vertical coordinates of the EF and its associated fixation duration, and (b) target location data, that consists of the pixel coordinates of the various targets on the display.

The input for step 2 is the time duration (in minutes or hours) for which the experiment has been conducted, and the output are several equal (unequal) sized time intervals which sum to the experimental time duration. For example, consider that the total experiment duration is divided into four equal time intervals (i.e. T=4, see section 2.2). Note that the time intervals can be chosen based on the task characteristics or the researcher’s judgment (i.e. fixed or event-based time intervals).

The inputs for this step are the various time intervals obtained in the previous step 2. The intervals are indexed and arranged in a time-order sequence. In this step, we start with the the first time interval to initiate developing the AOI fixation sequence.

As input, step 4 receives three things, the time interval selected in step 3, and the EM and target location data for this time interval. As output from this step, we obtain the AOI fixation sequence for the time interval considered. The size of the AOI fixation sequence is directly proportional to the number of EFs in the time interval. The AOI fixation sequence is created for the selected time interval by adapting the approach suggested by Kang et al(7). Creating the AOI fixation sequence involves mapping the EFs with the AOIs. Only those EFs falling within any of the AOI boundaries are considered for AOI fixation sequence development, else they are ignored. AOIs are coded by assigning uppercase letters followed by lowercase alphabets (i.e. A, B, …, a, b). The developed AOI fixation sequence is the collapsed form of a raw AOI fixation sequence. In the collapsed form of the sequence, multiple consecutive fixations of the same AOI is collapsed to a single fixation case (e.g. AAA is collapsed into A). Thus, a raw AOI fixation sequence AABCC is collapsed to ABC. In addition, an overlapping AOI case is shown in parenthesis with individual constituent AOIs separated by a semi-colon. For example, if an EF falls within the overlapped region of AOIs A and B, it is represented as (A; B). After mapping all the EFs to the AOIs, we obtain a time-ordered sequence of AOIs, that shows which AOIs were fixated and in which order. Table 1 shows a sample AOI fixation sequence for four time intervals in a hypothetical scenario with seven AOIs.

Step 5 receives the AOI fixation sequence developed in step 4 as input. Afterwards, the AOI fixation sequence is transformed, as per the approach suggested by Noton and Stark,(35) for developing the associated AOI transition matrix, which is the output of this step. The size of the transition matrix depends on the number of unique AOI states (single and overlapped both) in the fixation sequence. The AOI transition matrix shows, in a tabular manner, how many transitions have occurred between various AOIs pairs. For example, Table 2 represents the AOI transition matrix associated with the AOI fixation sequence for time interval 1 in Table 1. The sequence shows there are three EF transitions from AOI A to B highlighted in grey within Table 2. A different AOI transition matrix is required for each time interval; thus, before moving to the next step, we need to create the associated AOI transition matrix for each time interval.

The inputs for step 6 are the AOI transition matrix for all the time intervals obtained in step 2. As output from this step, we obtain the DNet representation of the EM data. Developing the DNet involves two steps. First, developing a static network for each of the time intervals considered in STEP 2. Second, arranging the static networks in a time-based order to visualize the DNet. The details of both the steps are given below.

Step 7 involves calculating the measures using equations (i), (ii), and (iii). As a result, we provide the DNet as the input to this step, and as output we obtain all three importance measure values for each of the AOIs with respect to all the time intervals considered in the DNet. For example, consider the DNet visualization in Figure 4, where the indegree value of AOI B changes from 4 to 3 as we move from the time interval 1 to 2 (Figure 4 (a)-(b)). For the given DNet in Figure 4, to calculate the closeness and betweenness value for AOI B, we first demonstrate how to calculate the distance between the AOIs. For example, in the first time interval, the distance from AOI B to A is given by dBA(t)=1wBA(t) and substituting t=1, we get dBA(1)=13.

To calculate the minimum distance between two AOIs consider Figure 4 (a), where there are two EF transition paths from AOI B to E: Direct path from AOI B to E, which has edge weight 1; indirect path from AOI B to A and then to E, which has edge weights 3 and 3, respectively. Thus, the shortest distance from AOI B to E is defined as the path having the minimum distance of dBE*(t)=min[dBE(t),(dBA(t)+dAE(t))]. For the first time interval ( t=1), we obtain dBE*(1)=min[1/1,(1/3+1/3)]=min[1,2/3]=2/3. Thus, for the first time interval, the closeness and betweenness value of AOI B is given by 2.375 and 8, respectively.

The last step involves normalizing the calculated measures and subsequently visualizing the normalized measure values. Thus, the obtained AOI importance measures in step 7 acts as input for this step, and as output we obtain the normalized measure values accompanied by their visualization. Equation (i), (ii), and (iii) shows that the three measures both depend on the number of AOIs and the amount of EFs. In addition, these measures also have different units, as a result, they are incommensurable. For the present dynamic scenario, both the number of AOIs and EFs are not constant for the different time intervals considered in the DNet analysis. In addition, to compare the AOI importance values across various time intervals and across multiple observers, we need to eliminate the units of importance measures, thereby bringing them to a similar scale. To address this issue, we present two normalization options.

For the experiment we used one 20 minutes long simulated enroute air traffic scenario, which was developed and provided by the FAA. This scenario had a total of thirty nine unique aircraft (named as A, B, …, Z, a, b, …, m) with an average of twenty aircraft per frame. The scenario had a minimum of seven and maximum of thirty aircraft per frame. Figure 7 represents a screen capture of the simulated scenario. The scenario update rate was 1 second. The display shows various aircraft and a weather patch in blue color. The aircraft representation shows the direction in which the aircraft is moving (shown by the white line) and data block which contains information about the aircraft’s computer code name, altitude, current speed, and destination airport. For example, in Figure 7, the aircraft N7890 (AOI U) is flying at a constant altitude of 19,000 ft (as shown by 190C) with a speed of 402 knots and it going to KGPT airport (i.e. Gulfport–Biloxi International Airport in Mississippi).

The task involved controlling the simulated low altitude enroute airspace using the ERAM system. The ATCSs are required to fulfil two objectives: (1) Route the aircraft through the sector within the display and (2) avoid any conflict scenario by preventing loss of separation (vertical and horizontal separation of 1000 ft and 5 knots respectively) between aircraft. To achieve these objectives, the ATCSs gave voice commands (e.g. change is altitudes, speed and direction of the aircraft) to the pseudo pilots, using the simulated radio, for necessary maneuvering of the aircraft.

Figure 8 represents the DNet visualization of the EM data of one ATCS for the simulated scenario shown in Figure 7. In Figure 8 (a) (i.e. time interval 0-5 minutes), AOI F (i.e. AAL68) and U (i.e. N7890) are the most important AOIs as they both have most EF numbers (circle size) and longest EF duration (circle color). In addition, there are highest EF transitions between AOI F and AOI U (based on the thickness of the link). AOI K (i.e. EJA33), despite having a small number of EFs, has substantially longer EF duration. AOI b and AOI d can also be considered as important AOIs based on how a researcher wants to set the threshold.

For the second time interval (i.e. 5-10 minutes) shown in Figure 8 (b), the important AOIs have changed to AOI

G (newly appeared aircraft not shown in Figure 8(a) and AOI K. Notice that AOI F has moved out of the display (see Figure 7(b)) and AOI U is still within the display but AOI U is not visually attended any longer.

In addition, notice that AOI d (i.e. UAL1322) has been receiving consistent visual attention throughout the two time intervals 0-5 minutes and 5-10 minutes. Similarly, the important AOIs change for the next two time intervals. In the last time interval, the overlapping AOI (i.e. AOI (J;f)) receives much visual attention. As we move across time intervals from 1 to 4, a visible trend is the increase in the complexity of the network, with increase in the number of AOIs and EF transitions among them.

Figure 9 represents the dotplot visualization of the normalized indegree measure for all the AOIs present in DNet shown in Figure 8. For example, in Figure 9 (a), AOI F and U has high importance in first time interval (i.e. 0-5), although, their importance reduces drastically in the subsequent intervals. AOI d receives consistent visual attention throughout the first two intervals. The indegree results are in accordance with the DNet results in Figure 8 since indegree measures the number of EFs received by an AOI.

The adapted dot plot better shows the evolution of important AOIs. Considering AOI K, we see that its importance initially grows as we move from the first time interval to the second, where it reaches its maximum and then starts decreasing for the last two time intervals. Another noticeable fact is that majority of the overlapped AOI cases have significant importance only in one time interval. These trends took more time to identify when observing the DNet.

Figure 10 represents the relative importance of various AOIs present in the first time interval of participant 1 (i.e. P1). Again, AOI F and U are important, but also, we can identify that AOI b (i.e. SWA340) and AOI d can also be important AOIs when we additionally consider the closeness and betweenness values. We can also observe that AOI K had small EF numbers during the first interval, however, it can be considered as an important AOI due to the long EF duration and relatively high closeness value. Furthermore, we can see that the AOI Q and AOI a might be an important AOI considering that AOI (Q;a) also has moderate indegree and closeness values.

Figure 11 shows the measures (i.e. indegree, closeness, and betweenness) visualized based on participants for the first time interval. Although there are slight variations, we can see a consistent trend among the participants. In addition, it becomes more evident that AOI Q and AOI a can be important AOIs (considering the values of AOI (Q;a)) in addition to AOI F, AOI K, AOI b and AOI d. In detail, overlapping AOI (Q;a) received substantial EF duration (as shown by the vertex color in the DNet visualization) although it received moderate number of EFs on it (as shown by the vertex size in the DNet visualization). On the other hand, comparing the three importance measurements, AOI (Q;a), in spite of having moderate indegree and closeness value, has insignificant betweenness value.

This might be due to the short lifespan of the overlapped AOI state since AOI a (i.e. SWA2920 flying at 21000 ft with speed 424 knots overtakes AOI Q (i.e. N46332 flying at 7000 ft with speed 163 knots very quickly on the display. As a result, it is highly unlikely that it will play a crucial role as a bridge for the flow of attention between other groups of aircraft.