Twenty-two adults (aged 22-43; median of 35) participated in the study. All had normal or corrected-to-normal vision and reported normal hearing. All participants were naïve about the purpose of the study and received payment for their participation. All the procedures were approved by the NTT Communication Science Laboratories Ethical Committee, and all participants gave informed written consent before the experiment.

Stimuli were generated and controlled by a personal computer (Dell OptiPlex 980) and presented through a headphone (Sennheiser HD 595) and on an 18.1-inch monitor (EIZO FlexScanL685Ex). Auditory stimuli were 15 excerpts of the first 90 s of selected pieces (Table 1). They were selected because their structure consisted of both several repetitions and variations of them. The sound pressure levels were fixed across the participants at a comfortable listening level. The visual stimulus was a dark gray fixation point (0.25 × 0.25°, 0.33 cd/m²) presented against light gray background (27.0 cd/m²).

Behavioral responses were collected from a transducer (TSD115) connected to a Biopac MP system (HLT100C module, BIOPAC Systems, Inc.). The transducer had a slider on the panel to allow participants to report subjective assessments from 0 to 10 continuously. The sampling rate of the transducer was 1000 Hz. Pupillary responses were recoded binocularly by an infrared eye-tracker camera (Eyelink 1000 Desktop Mount, SR Research Ltd.) with a sampling rate of 1000 Hz.

The 15 musical excerpts were presented twice in different sessions: first in a surprise-rating session and then again in a passive-listening session. The order of the excerpts in each session for each participant was randomly assigned. The inter-stimulus interval (ISI) was 5 s. The total duration of each session was around 25 min.

All participants were given written and oral explanations about the nature of the experiment and the pupillary response recording. Participants sat in front of the monitor at a viewing distance of 51 cm in a dimly lit soundproof chamber, with their chin on a chinrest. Before each session, a five-point calibration procedure was performed, after which the participants were instructed to fixate the central point throughout the experiment.

In the surprise-rating session, participants were asked to concurrently rate how they felt about changes (in any sense) compared with the portions within the excerpt they had heard so far. For example, if they felt any aspect in the music, including melody, tempo, harmony, or texture (e.g., more instruments playing), became richer in variation, they moved the slider to the right to register higher scores. If they felt the change became monotonous, they moved it to the left to register lower scores. The slider was reset in the middle (i.e., scored as 5) at the beginning of each excerpt.

In the passive-listening session, participants listened to the same musical excerpts again without any task involvement. The break between the two sessions was longer than 30 min. The order of the two sessions was fixed to avoid the influence of expectation on the surprise rating due to the repetition.

After the two sessions, participants answered a questionnaire to rate from 1 (never heard the piece) to 7 (often heard the piece) how familiar they were with each excerpt and to write down the name of the piece and/or the artist/composer if they knew it. They were allowed to replay the excerpts at their own pace when answering the questionnaire.

The mean familiarity scores for the classical, jazz, and rock music were 4.1, 2.4, and 2.6, respectively (scores for individual excerpts are listed in Table 2). The mean scores for each participant were subjected to a repeated-measures ANOVA with the music genres (classical, jazz, rock) as within-subject factors. Results showed that participants were more familiar with the classical music we selected than the other types of music [F(2,42) = 19.07, p < .001, η² = .48]. The results of the open questions are shown in Table 2 (second and third columns). Participants tended to give more answers and correct ones to questions about the classical music than to those about the other genres, which is consistent with the results of the subjective feeling of familiarity.

Figure 1A shows examples of the surprise rating over time. The average (the fourth column) and variation (the fifth column) of the surprise rating over time are listed in Table 2.

To examine whether the surprise rating varied among the music genres or excerpts (e.g., in terms of its familiarity), we conducted two different analyses. First, the means of the average, as well as the standard deviations, of the surprise rating score were subjected to a repeated-measures ANOVA with the music genres (classical, jazz, rock) as within-subject factors. Results showed that neither the average surprise rating [F(2,42) = 2.80, p > .07, η² = .12] nor the variations in the surprise rating over time [F(2,42) = 1.13, p > .3, η² = .05] differed among music genres. Second, we calculated the correlation between the familiarity rating and average surprise rating and the correlation between the familiarity rating and the variations in the surprise rating over time. Results showed a positive correlation between familiarity and the average surprise rating over time (r = .24, p < .001) but not between familiarity and variations in the surprise rating over time (r = -0.07, p > .2).

To examine the consensus among the participants on the surprise rating, we calculated Kendall’s coefficient of concordance (W). The rating data were resampled with a 10-Hz sampling rate for the analysis. The results are shown in Table 2 (sixth column). The consensuses among the participants were moderate but significant, and they varied among musical excerpts (median of 0.56, min of 0.30, max of 0.73; all ps < .001).

Figure 1B shows the pupil size change over time. Only data recorded from the right eye were analyzed since the pupillary responses from both eyes were consensual. Data during blinks were treated as missing and discarded (30.1%). The range of the average blink rate was about the same as in our previous studies (4, 12), where the task was an auditory one that allowed normal blinks.

The pupil size measurement in the video-based eye tracker system, as used in the current study, was covariant with the gaze position (21). To avoid recording errors due to unexpected gaze positions, pupil size data were screened when the gaze position deviated 1.5 deg. from the central fixation point, and 23.1% of the data were screened out.

The Eyelink system outputs arbitrary units [au] to represent the pupil size, which was not calibrated across participants or conditions. To compare the results across conditions, we computed z-score during each 90-s excerpt. To reduce high-frequency noise due to the over-fine sampling rate (i.e., 1000 Hz) for pupillary response measurements, we resampled the data with a 10-Hz sampling rate for the analysis. Specifically, the data between the resampling points (i.e., every 100 data points) were discarded without any interpolation or filtering procedure. In this work, we used an EDF converter (provided by SR Research) to convert the Eyelink EDF file to the ASC format, and we used Matlab for all the data analyses. The function for the resampling procedure described above was “downsample.” We followed the same protocol as in our previous study (4).

The pupil data recorded in the two sessions (surprise-rating and passive-listening) were time-aligned to the rating data obtained in the surprise-rating session. The surprising moments were defined arbitrarily as the period when the surprise rating score was above 7 (the red lines in Fig. 1), the unsurprising moments as a surprise rating score below 4 (the blue lines in Fig. 1), and the neutral moments as a surprise rating score between 7 and 4 (the black lines in Fig. 1). The criterion was set to obtain similar probabilities of the valid data for surprising and unsurprising moments: 24.7% and 21.7% of the total duration, respectively.

Results are shown in Fig. 2. Mean pupil diameter was subjected to a three-way repeated-measures ANOVA with the task (surprise-rating, passive-listening), music genre (classical, jazz, rock), and surprise (surprising, neutral, unsurprising) as within-subject factors. Results showed main effects of surprise [F(2,42) = 9.66, p < .001, η² = .32] and music genre [F(2,42) = 3.31, p < .05, η² = .14] but not any other effect or interaction (ps > .1). When we applied a different criterion to define the surprising moments in which the deviation of the rating score from mean was more than 1.5 times the standard deviation, the effect of surprise remained [F(2,42) = 10.82, p < .001, η² = .34]. The results suggest that the pupil dilated more strongly during the surprising moments than during the unsurprising ones regardless of the music genre or whether the on-line surprise rating was involved or not.

To further investigate whether there was systematic bias induced by a particular musical excerpt or participant, we used scatter plots to represent the surprise-related PDR for individual excerpts and participants. Results are shown in Fig. 3. The data were clustered below the diagonal line (confirming larger PDR during surprising moments than during unsurprising ones), while the distribution of the genres or participants was spread equally, indicating a consistent tendency of the surprise-related PDR among different genres or participants. There was no significant correlation between the surprise-related PDR (i.e., the difference in average pupil size between surprising moments and unsurprising ones) and the familiarity rating (r = .02, p > .8 for the surprise-rating condition, and r = .03, p > .7 for the passive-listening condition; data not shown). The overall results suggest that the surprise-related PDR did not depend on the genre, participant, or familiarity with the excerpt.

We conduced further analysis to verify the effect of the surprise-related PDRs and examine whether the effect could be explained by stimulus-driven factors coupled with the musical excerpts or response biases/tendencies associated with the participants. Specifically, we calculated the estimated PDR-surprise association using bootstrapping procedures. In the completely random procedure (as a baseline), the pupil data were aligned with rating data randomly selected from different participants/excerpts. The difference in the mean pupil diameter between surprising and unsurprising moments, derived from the ratings of different participants and excerpts, was calculated 1,000 times (by random selection between the pair of the pupil and rating data) to form a distribution, where the PDR was expected not to be associated with the surprise at all. The results are shown in Fig. 4. In both the surprise-rating and passive-listening conditions, the baseline distributions (i.e., the black distributions) were quite distant from the observed surprise-related PDR (indicated as vertical dashed lines), indicating a reliable surprise-related PDR: when the pupil data matched the rating data for the same participant and musical excerpt, pupil size was larger during surprising moments.

We further calculated the estimated PDR-surprise associations when the pupil data were paired with the rating data for the same excerpt, but randomly selected from different participants (i.e., shuffled-participant condition), and when the pupil data were paired with the rating data obtained from the same participant, but randomly selected from different excerpts (i.e., shuffled-excerpt condition). We expected that if the observed surprise-related PDR could mainly be explained by the stimulus-driven factor, the distribution of the estimated PDR-surprise association from the shuffled-participant condition would be close to the observed surprise-related PDR. Namely, as long as the pupil data were aligned with the rating data from the same excerpt, regardless of the rater/participant, the PDR-surprise association would increase. In contrast, if the surprise-related PDR could mainly be explained by the participant-related factors, such as response bias systematic tendency of rating, etc., the surprise-related PDR would be close to the estimation obtained from the shuffled-excerpt procedure. Namely, the surprise-related PDR would be due to coordination between the pupillary response and rating of a particular rater/participant, regardless of the excerpt that was to be rated.

As shown in Fig. 4, in the surprise-rating session, the observed surprise-related PDR was quite close to the distribution derived from the shuffled-participant procedure but not to the distribution derived from the shuffled-excerpt procedure, indicating that the stimulus characteristics might contribute to the surprise-related PDRs during surprise rating. In contrast, no such result was found in the passive-listening session. The distribution of the shuffled-participant or shuffled-excerpt condition overlapped the baseline distribution and was distant from the observed surprise-related PDR. The results suggest that the surprise-related PDRs observed during passive listening cannot be explained by the stimulus characteristics or response biases/tendencies associated with the participants.

Surprise-related PDRs were observed in both the surprise-rating and passive-listening sessions. It may be suspected that the participant performed the surprise rating implicitly and spontaneously even though they were not asked to, especially since the passive-listening condition was always conducted in the later session. To examine whether the participant performed the rating while listening to the music, we conducted another analysis to examine the decision-making-related PDR (e.g., 10, 11).

The pupil data were time-locked to the rating change instead of the rating moment as shown in the surprise-related PDR analysis. The rating change was defined as the moment the rating started to move in a particular direction. Figure 5 shows examples of the results of the time-locked analysis of the rating change. The timing of the surprising change was defined as the start of the period in which an increased rating score lasted longer than 0.5 s (the vertical red dotted lines). The timing of the unsurprising change was defined as the start of the period in which a decreased rating score lasted longer than 0.5 s (vertical blue dotted lines). To provide the baseline, we defined the neutral time as the median timing of the period in which an unchanged rating score lasted longer than 4 s (vertical green dotted lines).

Mean pupil diameter changes time-locked to the reference timing are shown in Fig. 6. To examine whether pupil diameter reliably increased, we conducted a pairwise t-test at each time point for all the pairs in the three conditions (Bonferroni corrected p-value). Results showed that in the surprise-rating session, pupil diameter increased around 1 s before the decision-making event, and reached statistical significance around the reference timing, as indicated by the difference between the surprising/unsurprising and neutral conditions. Note that the reference timing of the surprising and unsurprising events was when the participant started moving the rating bar. The pupil dilated before the timing, suggesting that this PDR is evoked by a decision-making processing, rather than motor behavior. This is consistent with Einhäuser et al. (10), who showed that the pupil dilated at the moment a choice was made even when the actual motor response occurred thereafter. This decision-making-related PDR had a similar pattern regardless of whether the decision was made in the surprising or unsurprising direction. In contrast, in the passive-listening condition, no such decision-making-related PDR was found. The overall results suggest that there was no spontaneous decision-making process involved in the passive-listening session.

We have previously found that the subjective salience evaluation of sounds is highly correlated with their loudness, as well as with the PDR to them (12). The sounds used in the previous study were environmental sounds presented briefly (500-ms duration) and discretely (10-s ISI). It remains unclear whether the current online surprise judgment on the long-lasting music can be explained by the instantaneous loudness change of the music, and whether the surprise-related PDR can be simply explained by the loudness change.

To investigate the issue, we conducted an analysis to examine the similarity between the surprise rating and instantaneous loudness change for each excerpt. The loudness of the musical excerpt was estimated by an excitation-pattern-based loudness model (e.g., 22). The acoustic signal was bandpass filtered with a bank of filters of equivalent rectangular bandwidth (ERB) with the center frequencies spaced 0.5 ERB from 30 to 16,000 Hz, and weighted with the middle ear transfer function. The outputs were divided into segments of 100-ms windows to compute the instantaneous loudness. The instantaneous loudness was smoothed with a 1-s window to represent the estimated loudness change over time. We then calculated the correlation between the surprise rating and loudness change over time. Results showed that among the 15 excerpts we selected, 13 showed significant correlation between the surprise rating and instantaneous loudness change (see Fig. 7).

To further investigate whether the pupil data simply reflected the instantaneous loudness change of the music, we conducted an analysis of the loudness-related PDR. The idea was to align the pupil data with the loudness change over time, as in the analysis of the surprise-related PDR, to examine whether the pupil size was larger during the loud moments than during the quiet ones. The loud and quiet moments were defined as when the deviation of the loudness change from the mean was larger and smaller than 1.5 times the standard deviation, respectively, and the middle ground between the criteria. Mean pupil diameter during loud, quiet, and middle ground moments (Fig. 8) was subjected to a three-way repeated-measures ANOVA with the task (surprise-rating, passive-listening), music genre (classical, jazz, rock), and loudness (loud, middle ground, quiet) as within-subject factors. Results showed an two-way interaction between music genre and loudness [F(4,84) = 7.36, p < .001, η² = .26] and the three-way interaction among task, music genre, and loudness [F(4,84) = 2.95, p < .03, η² = .12], but not any other main effect or interaction (ps > .3).