A total of 15 surgical residents (ten surgeons and five interns) from the Department of Neurosurgery (six participants) and Otolaryngology (ENT) (four participants) from Hacettepe Medical School in Ankara, Turkey participated in this study. There were two skill level groups of participants, intermediate and novice, based on the categorization presented in Silvennoinen et al.’s study (39). In that concern, those who have operated at least one endoscopic surgery are considered as ‘intermediate’, whereas others who have only observed and assisted in endoscopic operations, but have not performed any surgeries by themselves, are considered as ‘novice’. As shown in Table 1, most of the participants were male (86.66%).

Detailed information about these participants according to their endoscopic surgical expertise (average number of operations observed, assisted and performed) is given in Table 2.

Haptic devices can be integrated into training simulations for MIS procedures (40) in order to provide similar and real-world practice as well as a realistic sense of touch. Accordingly, in this study, a mid-range professional ‘Geomagic Touch’ haptic device is used to perform the tasks in the scenarios. The simulation software recorded a hundred data points per second as the hand coordinates for both hands with the help of these haptic devices. Additionally, the eye movement data is gathered by using a 60 Hz eye-tracking device, the Eye Tribe (41), which is easy to set up, transportable and reported as appropriate for use in scientific research (42). This tool is used to track the user’s eye movements and calculate the on-screen gaze coordinates with an accuracy of 0.5°-0.7°. During the experiment, eye-gaze coordinates were gathered in a top-left oriented 2D coordinate system. The screen resolution is 1920 x 1080 pixels; in other words, the horizontal field of view (x-coordinate) is 1920 pixels, whereas the vertical field of view (y-coordinate) is 1080 pixels. The field of view (FOV) of the camera, from the left-perspective for Scenario-2 can be seen in Figure 1.

In our software, hand motion coordinates, regarded as the position of tool and camera in the scenarios, were represented as 3D vectors. Accordingly, the origins for eye O_eye and for hand O_hand coordinates have been represented in a 2D scene and appear in Figure 2-A and B.

Due to the low sampling rate, the analysis of saccades may become problematic when collecting data using the Eye Tribe tracker. It is reported that measuring saccade metrics require high frequency sampling (43). However, it has also been stated that velocity-based algorithms for saccade detection that works on high-frequency data do a relatively good job on the Eye Tribe data (43, 44 ), but still they fail to be accurate when it comes to saccadic positions (43). However, there are a number of studies in the literature using 50 Hz (38) and 60 Hz sampling rate trackers in order to classify eye events into fixations and saccades (44).

In the experimental study, there were two both-handed scenarios for the surgical training process used. The first one was prepared to practice on general skills, such as learning the use of surgical tools with the endoscope and developing depth-perception in a simulated 3D environment (Scenario-1). The other scenario closer to the operational procedures uses a simulated anatomical nose model (Scenario-2). Each scenario consists of ten repetitive tasks. The layout of the scenarios is shown in Figure 3-A and B, respectively.

A: Scenario-1: Moving the Red Ball into the Box

B: Scenario-2: Clearing the Nose

In this scenario (Scenario-1), each participant is asked to approach the red ball with the haptic device, catch it, and then move it into the green box as shown in Figure 3-A. The tool is controlled with the dominant hand of the participant, whereas the camera is controlled by his/her non-dominant hand. The position of the ball and the box changes arbitrarily in each trial. The participant must complete this process successfully, which includes 10 tasks, within the allocated time period. In case of failure to complete each task within 10 seconds, the ball and the box disappear.

In this scenario (Scenario-2), the participant must remove the green ball-like objects, which are spread through the nose model as shown in Figure 3-B. The camera is used as the light source and the cautery model as the tool to collect the objects. In case of a collision - that is, if the haptic device touches the tissue - it provides a force feedback that feels as if the device pushes back in the hands of the user.

This study is conducted as part of a research project and the content of both scenarios are prepared based on opinions of the neurosurgery and ENT domain experts. Mainly endoscopic pituitary surgery procedures are aimed to be practiced which is in the scope of both domains and performed starting from the nose holes through the pituitary area. Accordingly, the scenarios are designed for beginners of these operations.

As shown in Figure 4, in this study, the research procedure mainly consisted of five stages. These are S1: Experimental study; S2: regenerated simulation version; S3: BIT algorithm; S4: Eye and hand metrics; and S5: Eye-hand coordination analyses.

At the beginning of the experiment, the participants were asked to fill out a questionnaire including their demographic information, dominant-hand and experience level (i.e., years in the department, and the number of operations observed, assisted, and performed). Prior to the experimental study, the participant is seated 70 cm. away from the screen. First, each participant was informed about the calibration process; more specifically that they should maintain the distance and that they are not allowed to move their heads or body once calibration is completed. After this briefing, the calibration process started. Nine calibration points appeared on the screen one after the other, with a viewing time of two seconds each. At the end, a five-star rating is displayed to provide the accuracy of the calibration. If the result of the calibration is four-star (<0.7°) or five-star (<0.5°), then it is regarded as acceptable for the experimental study. After that, a brief instructional video explaining how to perform the task was shown to each participant separately about the procedure. Next, each participant was asked to perform two scenarios, Scenario 1 and 2, using both their dominant and non-dominant hands at the same time. The eye-gaze data (i.e., pupil size, fixation, raw and smoothed X and Y coordinates of both left and right eye) and hand motion data (i.e., tool and camera position, tool and camera rotation as 3D vectors) were collected and stored using a special software (Figure 4: S1). The experimental setup can be seen in Figure 5.

Afterwards, the performance of each participant in both scenarios is regenerated in the simulation environment using a software developed in the Unity environment (Figure 4: S2). This regeneration software took each participant’s eye data with the help of the tracker device and the hand data from the haptic device while they were performing each task during the experimental study. In this way, each participant’s eye and hand coordinates are aligned within the same time scale by representing the eye location through an eye icon and hand location through a small blue sphere in the regenerated simulation re-play (See Figure 6).

This regenerated simulation allows researchers to determine the extent regarding tissue-contact, left-right hand coordination and the eye-hand coordination of each individual. In the meantime, the observation data was gathered through a questionnaire to understand such behavior differences between novice and intermediate groups. Five observers (other than the researchers) have evaluated the participants’ performances in Scenario-2. As this scenario is based on an environment similar to the operational procedures (Figure 4: S3), the observations are conducted on this scenario. All of the observers are the graduate students in the field of engineering and briefed about the observation procedure before they start the evaluations.

Using this approach, the raw coordinates for both the eye and hand movement data aligned in the time scale were obtained as the output for regenerated simulated version of the scenarios. Then, these coordinates were given as an input to the BIT Algorithm (Figure 4: S4). Afterwards, the classification process of the eye and hand movements - as either fixation or saccade events- were identified by running the algorithm, separately for the eye data and the hand data. The output of the BIT algorithm performed on the eye data is the fixation and saccade metrics, whereas metrics related hand movement were obtained as the outputs from the hand data (Figure 4: S5). Finally, all collected data is analyzed to better understand the efficiency of the proposed metrics and to objectively measure the eye-hand coordination skills of the participants (Figure 4: S6).

In this study, using the BIT algorithm, three metrics are identified, namely Fixation Duration (FD) (the time from one saccade to another), Fixation Number (FN) (the number of fixations in an interval) and Saccade Number (SN) (the number of saccades in an interval).

As explained in the procedure, the eye movements and the hand movements of the participants are aligned in the same time scale. The authors believe that further insight regarding the eye-movement events can be also be applied to the hand movement events as well. Our main aim in this study is to test this assumption by comparing the results conducted in this study with the ones that are reported in earlier studies on eye-hand coordination skills of the surgeons. Accordingly, in the present work, new hand metrics are introduced to identify the hand movements of the participants as explained below:

The ‘Stand Still’ metric is proposed in this study as the period when the hand movement remains within a very small range and lower velocity for some time. In other words, it determines the ‘idle state’ of the hand movement. By running the BIT algorithm, such events can be classified into ‘Stand Still Duration’ (SSD) and ‘Stand Still Number’ (SSN) for hand movements. In this respect, the ‘Sudden Sharp Movement’ (SSM) metric is also proposed to identify very fast, sharp hand movements while performing any given task.

In Scenario-1, the descriptive statistics for the two groups of participants (intermediate and novice) for both the eye metrics (FD, FN, and SN), and the hand metrics (SSD, SSM and SSN) are depicted in Figure 7. Additionally, the mean and standard deviations for all metrics are in Scenario-1 is shown in Table 3.

A: Scenario-1: Mean FD (ms) of eye and Mean SSD (ms) of hand

B: Scenario-1: Mean FN of eye and Mean SSN of hand

C: Scenario-1: Mean SN of eye and Mean SSM of hand

A Pearson's product-moment correlation was run to assess the relationship between the eye-gaze and hand motion metrics, (FD- SSD and FN- SSN) and saccades (SN- SSM) for intermediates. For all of these three pairs, the preliminary analyses showed the relationship to be linear with both variables normally distributed, as assessed by the Shapiro-Wilk's test (p > .05), and there were no outliers. Also, there was a strong negative correlation between the FD – SSD and FN- SSN metrics among the intermediates, r = -.836 and r = -.837, respectively. On the other hand, a strong positive correlation existed between the saccade metrics SN and SSM in the same group r = .755 (Table 4).

Again, a Pearson's product-moment correlation was run to assess the relationship between the eye-gaze and hand motion metrics, and fixations (FD- SSD and FN- SSN) and saccades (SN- SSM) for novices. For the first two pairs related to fixation, preliminary analyses showed the relationship to be linear with both variables normally distributed, as assessed by the Shapiro-Wilk's test (p > .05) with no outliers. There was a moderate positive correlation for both the FD- SSD and FN- SSN metrics among the novice participants, r = .448. However, not all variables were normally distributed for the saccade metrics SN and SSM, as assessed by the Shapiro-Wilk's test (p < .05). Accordingly, a Spearman's rank-order correlation was run to assess the relationship between the saccade number of eye-gaze data and the sudden sharp movements of the hand data, with results showing a strong positive correlation for the SN and SSM measures, rs = .590 (Table 4).

Figure 8 show the descriptive statistics for the eye (FD, FN, and SN) and hand metrics (SSD, SSM, SSN) for the two groups in Scenario-2. Additionally, the mean and standard deviations for all metrics are in Scenario-2 is depicted in Table 5.

A: Scenario-2: Mean FD (ms) of eye and Mean SSD (ms) of hand

B: Scenario-2: Mean FN of eye and Mean SSN of hand

C: Scenario-2: Mean SN of eye and Mean SSM of hand

A Spearman's rank-order correlation was run to assess the relationship between the FD- SSD and FN- SSN measures since not all the variables were normally distributed, as assessed by the Shapiro-Wilk's test (p < .05). There was a strong negative correlation for the pairs related to fixation that is the FD- SSD and FN- SSN metrics. In other words, an increase in the eye fixation duration and fixation number was strongly correlated with a decrease in the hand stand still duration and stand still number among the intermediates, rs = -.900, p < .05 (Table 6). However, both variables of the saccade metrics, SN and SSM, were normally distributed, as assessed by the Shapiro-Wilk's test (p > .05). Also, a Pearson’s correlation was run to assess the relationship between the saccade number of eye-gaze data and the sudden sharp movements of the hand data, with the outcome that there was also a strong positive correlation for the SN and SSM metrics, r = .846 (Table 6).

A Pearson's product-moment correlation was run to assess the relationship between the eye-gaze and hand motion metrics, the fixation (FD- SSD and FN- SSN) and saccades (SN- SSM) for novices. For the first two pairs related to fixation, preliminary analyses showed the relationship to be linear with both variables normally distributed, as assessed by the Shapiro-Wilk's test (p > .05), and there were no outliers. There was a moderate negative correlation for both FD- SSD and FN- SSN metrics in novices, r = -.443 and -.441, respectively. However, not all variables were normally distributed for the saccade metrics SN and SSM, as assessed by the same test (p < .05). Accordingly, a Spearman's rank-order correlation was run to assess the relationship between the saccade number of the eye-gaze data and sudden sharp movements of the hand data. The results show a small positive correlation for the SN and SSM metrics for novices, rs = .06 (Table 6).

Five observers have monitored the participants’ performances in Scenario-2 in order to assess the left-right hand coordination and eye-hand coordination skills of the participants. Their evaluation is based on the items given in Table 7 to be ranked in accordance to five alternatives (1: Strongly Disagree, 5: Strongly agree) in a Likert scale-type questionnaire. 11 out of 15 participants (3 intermediates, 8 novices) were evaluated using the questionnaire data. The descriptive results appear in Table 7.

A Mann-Whitney U test was run to determine if there were any differences in the scores for the given expressions (Table 7) between the intermediate and novice groups. The distributions of the scores for intermediates and novices were found to be dissimilar among the participants once inspected visually. According to the results, considering these expressions, the 3D depth perception and eye-hand coordination skills of the intermediates (mean rank = 10.00) were significantly higher than novices (mean rank = 4.50), U=0, z= -2.461, p=.012.

This study has two major contributions. Firstly, new hand movement metrics are proposed by adapting an open-source eye-movement classification algorithm (BIT) to the data collected through a computer-based simulation software using a haptic interface and eye-tracker. Secondly, it can be stated that such metrics and eye-hand correlation analyses can be used for the objective assessment of the skill levels required for endoscopic surgery trainees. As studies on eye-hand coordination in the literature are very limited, we believe that analyzing surgeons' hand and eye data jointly is an important contribution to objective evaluations and assessment of their skills and development of some standards for surgical education programs.

In the literature, dispersion-based algorithms are recommended for eye movement event classification while using a low-frequency eye tracking device. However, in this study, an adaptive algorithm is used because the velocity of the eye and hand movements can be different. Due to the ability of automatically defining task- and individual-specific thresholds and being machine and sampling frequency independent, we believe that the BIT algorithm proposed by (38) is suitable for skill-based studies.

As it is commonly the case, the number of surgeons in the neurosurgery and ENT departments is very limited, for which reason this study was conducted with 15 participants and only intermediate and novice surgeons. In the future attempts, it may be possible to validate the results of this study with a larger number of participants and also upon a wider range of skill levels. Additionally, in the future, experimental studies should be conducted with the help of scenarios in a larger scope aiming at different tasks with different difficulty levels and under different conditions. For instance, in the present work only both-handed task performances were evaluated; whereas later, the participants’ performances may also be compared in dominant, non-dominant and both hands settings.

As we were unable to obtain the values for the exact time stamps of the provided events, in this study the average values for each metric could be analyzed because of the values provided by the BIT algorithm. Due to this drawback, further analysis needs to be carried out to examine whether the eye and hand events occur in a synchronized way using other appropriate algorithms. Also, the differences among the results of different algorithms can also be further analyzed.

Lastly, Scenario-1 in this study was designed in a more general sense and to gain endoscopic skills, while Scenario-2 was arranged as more specific to the endo-neurosurgery tasks. Accordingly, the experimental study is conducted in the same order as the scenarios (Scenario-1 is performed first and followed by Scenario-2). However, to eliminate the order effect in learning, with more scenarios this effect can be brought under control.

The authors declare that the contents of the article are in agreement with the ethics described in http://biblio.unibe.ch/portale/elibrary/BOP/jemr/ethics.html and that there is no conflict of interest regarding the publication of this paper.

This study was conducted to improve the scenario designs applied in educational materials developed for the endo-neurosurgery education project (ECE: Tubitak 1001, Project No: 112K287). The authors would like to thank the support of TÜBİTAK 1001 program for realizing this research. Our thanks also go to the ECE project team and the Hacettepe University Medical School for their valuable support throughout the research. We also wish to express our sincere thanks to Payam Danesh for his valuable comments on an earlier version of this manuscript.