Supp.apa.org



Supplementary Material: The Social Perceptual Salience Effect 1

Running head: SUPPLEMENTARY MATERIAL: THE SOCIAL PERCEPTUAL SALIENCE EFFECT

Supplementary Material: The Social Perceptual Salience Effect

Martin P. Inderbitzin1, Alberto Betella1, Antonio Lanatà2, Enzo P. Scilingo2 , Ulysses Bernardet1, Paul F. M. J. Verschure1,3

1 Laboratory for Synthetic, Perceptive and Emotive Systems, Technology Department, Universitat Pompeu Fabra, Barcelona, Spain.

2 Interdepartmental Research Center E. Piaggio, Faculty of Engineering, University of

Pisa, Italy.

3 Catalan Institute for Research and Advanced Studies (ICREA), Barcelona, Spain.

March 7, 2012

Author Note

Correspondence concerning this paper should be addressed to:

Paul F.M.J.Verschure, Synthetic, Perceptive, Emotive, Cognitive Systems Group

Universitat Pompeu Fabra, Roc Boronat 138, 08018 Barcelona, Spain. E-mail: paul.verschure@upf.edu, Phone: (0034) 93 5421372

Supplementary Material: The Social Perceptual Salience Effect 2

Supplementary Material: The Social Perceptual Salience Effect

Preprocessing of Physiological Data

The raw data was collected from biosensors and pre-processed. Each signal was segmented according to the time duration of the stimulating epochs. The methods for data analysis of the physiological signals are based on previously published algorithms. Detailed descriptions can be found in (Valenza, Lanata, & Scilingo, 2011).

Heart Rate Variability (HRV)

Electrocardiogram (ECG) was pre-filtered through a Moving Average Filter (MAF) in order to extract and subtract the baseline. Since HRV refers to the change over time of

the Heart Rate (HR). We adapted an automatic algorithm to detect the Q-, R- and S-wave forms of the ECG signal (QRS complex) (Pan & Tompkins, 1985).

The time interval between two successive QRS complexes is defined as the R-wave

to R-wave (RR) interval (tR−R ). Thereafter the heart rate (HR) is defined as:

H R =

60

tR−R

(1)

Because the HR is a time series sequence of non-uniform RR intervals, we re-sampled the signal using the algorithm of Berger et al. (2007)

Respiration (RSP)

We identified the baseline and removed movement artifacts. Additionally we filtered the signal using a tenth order low-pass finite impulse response filter (FIR) with a cut-off frequency of 1 Hz approximated by Butterworth polynomial.

Supplementary Material: The Social Perceptual Salience Effect 3

Electrodermal Response (EDR)

The EDR signal was filtered using a 2.5 Hz low-pass FIR filter. Because it has been shown that the energy of the tonic component is in the frequency band from 0 to 0.05 Hz and the energy of the phasic component in the band from 0.05 to 1-2 Hz (Ishchenko & Shev’ev, 1989), we applied a twelve level decomposition wavelet filter in order to identify two main response components in these bands. Approximation at level 1 of the filter was the tonic component and subsequent details were the phasic component.

Standard Feature Set Identification

We calculated all the features for each neutral as well as for each stimulation session. We used 43 standard features and 8 features extracted using non-linear dynamic methods which are described in the next section. The standard feature set was derived from the following components of the signal: time series, statistics, frequency domain and geometric analysis.

Heart Rate Variability (HRV) HRV features were decomposed in features describing both the time and frequency domain. Defining the beat-to-beat time window (NN), we calculated the mean of the NN (MNN) and the standard deviation of the NN (SDNN). Additionally the root mean square of successive differences of intervals (RMSSD) and the number of successive differences of intervals which differ by more than 50 ms (pNN50 %) was calculated.

The triangular index, that refers to the morphological changes of the HRV, was derived from the histogram of RR intervals by performing a triangular interpolation over the NN windows (TINN).

We based all features extracted in the frequency domain on the Power Spectral

Density (PSD) of the HRV. Three main spectral components were distinguished in a

Supplementary Material: The Social Perceptual Salience Effect 4

spectrum calculated from short-term recordings: Very Low Frequency (VLF), Low Frequency (LF), and High Frequency (HF) components. We additionally calculated the LF/HF Ratio which should give information about the Sympatho-Vagal balance (Camm et al., 1996).

Respiration (RSP) By defining a time window W the respiration rate (RSPR) was calculated as the frequency corresponding to the maximum spectral magnitude. We identified the maximum (MAXRSP) and the minimum (MINRSP) value of breathing amplitude and their difference (DMMRSP) to characterize the differences between inspiratory and expiratory phase (range or greatest breath).

Electrodermal Response (EDR) We applied the same standard methods as used for the

RSP signal processing to identify both the tonic and the phasic EDR the central frequency, mean and standard deviation of the amplitude. Additionally we calculated the maximum peak and the relative latency from the beginning of the interaction phase, frequency (rate) and magnitude (max) of the maximum component of the phasic EDR.

Non-Linear Dynamic Methods for Feature Extraction

The here documented non-linear dynamic methods for feature extraction are based on the study of Valenza et al. (2011).

We based our analysis on the so-called embedding procedure. Embedding of a time series xt = (x1, x2 , ..., xN ) is realized by creating a set of vectors Xi such that

Xi = [xi , xi+4 , xi+24 , ..., xi+(m−1)4 ] (2)

where 4 is the delay in number of samples and m is the number of samples of the array

Xi . In order that the vector Xi represents the values that reveal the topological

Supplementary Material: The Social Perceptual Salience Effect 5

relationship between subsequent points in the time series, we must define the dimension of m and Xi and the delay ∆. We can represent the temporal evolution of the system by projecting the vectors Xi onto a trajectory through a multidimensional space, often referred to as phase space or state space. The Recurrence Plot (RP) visualizes all times at which a state of the dynamical system recurs (Marwan, Carmen Romano, Thiel, & Kurths, 2007). Higher dimensional phase spaces can be visualized by projecting them into two or three dimensional sub-spaces (Eckmann, Kamphorst, & Ruelle, 1987). When

a state at time i recurs also at time j, the element (i, j) of a squared matrix N xN is set to

1, 0 otherwise. This representation is called recurrence plot (RP). We can mathematically

express such an RP as

Ri,j = Θ ( i − ||xi − xj ||)

where xi Rm , i, j = 1, ...., N ; N is the number of considered states xi , εi is a threshold distance, ||.|| a norm and Θ (.) the Heaviside function which is defined as:



1, if z ≥ 0

H (z) =

0, if z < 0

(3)

We chose the optimal value of εi (Schinkel, Dimigen, & Marwan, 2008) as following:

i = 0.1 ∗ AP D (4)

where AP D is averaged phase space diameter of data xi .

To quantify the number and duration of recurrences of a dynamical system presented by its state space trajectory the Recurrence Quantification Analysis (RQA) can be applied (Zbilut & Webber Jr, 2006). In this study we calculated the following features:

Recurrence Rate (RR) is the percentage of recurrence points in an RP and it corresponds to the correlation sum:

RR =

N

XRi,j

N 2

i,j=1

(5)

Supplementary Material: The Social Perceptual Salience Effect 6

where N is the number of points on the phase space trajectory.

The determinism (DET ) is defined as the percentage of recurrence points which form diagonal lines:

DET =

N

PlP (l)

l=lmin

N

PRi,j

i,j=1

(6)

where P (l) is the histogram of the lengths l of the diagonal lines.

Laminarity (LAM ) is the percentage of recurrence points which form vertical lines:

LAM =

N

PυP (υ)

υ=υmin

N

PυP (υ)

υ=1

(7)

where P (υ) is the histogram of the lengths υ of the diagonal lines.

Trapping Time T T is the average length of the vertical lines:

T T =

N

PυP (υ)

υ=υm in

N

PP (υ)

υ=υm in

(8)

Ratio (RAT I O) is the ratio between DET and RR:

RAT I O =

DET

RR

(9)

Averaged diagonal line length (L) is the average length of the diagonal lines:

N

PlP (l)

L = l=lmin

PP (l)

l=lmin

(10)

Entropy (EN T R) is the Shannon entropy of the probability distribution of the

diagonal line lengths p(l):

N

EN T R = −Xp(l) lnp(l) (11)

l=lmin

Supplementary Material: The Social Perceptual Salience Effect 7

Longest diagonal line (Lmax ) The length of the longest diagonal line:

Lmax = max ({li ; i = 1, ..., Nl }) (12)

where Nl is the number of diagonal lines in the recurrence plot.

It has been shown that Approximate Entropy (ApEn) can be used to measure the complexity or irregularity of the signal (Fusheng, Bo, & Qingyu, 2000; Richman & Moorman, 2000). Small values of ApEn indicate a more regular signal, lager values a high irregular one.

To compute the ApEN first a set of length m vectors uj is formed:

uj = (RRj , RRj+1 , ..., RRj+m−1 ), (13)

where j = 1, 2, ..., N − m + 1, m is the embedding dimension, and N is the number of measured RR intervals. The maximum absolute difference between the corresponding elements defines the distance between these vectors:

d(uj , uk ) = max

n=0,...,m−1

{|RRj+n − RRk+n |} (14)

For each uj the relative number of vectors uk for which d(uj , uk ) ≤ r is calculated. r

is the tolerance value. The index is denoted with C m (r) and can be written in the form:

j (r) =

nbr of {uk |d(uj , uk ) ≤ r}

N − m + 1 ∀k

(15)

Due to the normalization, the value of C m (r) is smaller or equal to 1. The value of C m (r)

j j

is at least 1/(N − m + 1) since uj is also included in the count. The averaged natural

logarithm of each C m (r) yields to:

Φm (r) = 1

N − m + 1

N −m+1

X

j=1

ln C m (r). (16)

The approximate entropy finally can be calculated as:

ApEn(m, r, N ) = Φm (r) − Φm+1 (r) (17)

Supplementary Material: The Social Perceptual Salience Effect 8

Three parameters are influencing the value of ApEn: the length m of the vectors uj , the tolerance r, and the data length N. In this work we have chosen m = 2. The length N of the data also affects ApEn. As N increases the ApEn approaches its asymptotic value. The tolerance r has a strong effect on ApEn and should be selected as a fraction of the Standard Deviation of all Normal-to-Normal (SDNN) intervals, i.e. the standard deviation of the intervals between successive normal QRS complexes. A common selection for r is r = 0.2 · SDN N , which is also used here.

Feature reduction strategy

We obtained a high-dimensional feature space, that we reduced by applying the Principal Component Analysis (PCA) method (Jolliffe, 2002). We implemented this approach by means of the Singular Value Decomposition (SVD). Each training set vector can be approximated by taking only the first few k, where, k ≤ r, Principal Components. This mathematical method is based on the linear transformation of the different variables in principal components which could be assembled in clusters.

Classification

For classification a Nearest Mean Classifier (NMC)) is used. This classifier uses the similarity between patterns to decide on a good classification. The question is how to define similarity. NMC defines the features of a class as a vector and represents the class with the mean of the elements of this vector. Thus, any unlabeled vector of features will be classified as the class with the nearest mean value. Template matching uses a template for defining class labels, and tries to find the most similar template for classification.

The classification task was evaluated using the confusion matrix. The generic element rij of the confusion matrix indicates how frequently a pattern belonging to the stimulus class i was classified as belonging to the response class j. The matrix has to be read by columns. We used 80% of the feature dataset for training and the remaining 20%

Supplementary Material: The Social Perceptual Salience Effect 9

for the testing phase. In order to obtain unbiased classification results, we performed

40-fold cross-validation steps. This procedure allowed us to consider the distribution of the classification results as Gaussian. The classification is described by the mean and standard deviations among the 40 confusion matrices (See Table 1).

Supplementary Material: The Social Perceptual Salience Effect 10

Questionnaire

We would like to evaluate in more detail how you perceived the interaction with the real person or the avatar at DIFFERENT distances. ’Very close’ was the interaction distance of < 0.5 meter. ’Close’ the distance of 1.2 meters. Please mark with a cross your personal experience. Thanks.

Virtual Interaction

I perceived the close interaction with the VIRTUAL avatar as:

Pleasant 1 2 3 4 5 6 7 Unpleasant

Negative 1 2 3 4 5 6 7 Positive

I perceived the very close interaction with the VIRTUAL avatar as: Pleasant 1 2 3 4 5 6 7 Unpleasant

Negative 1 2 3 4 5 6 7 Positive

Physical Interaction

I perceived the close interaction with the PHYSICAL person as:

Pleasant 1 2 3 4 5 6 7 Unpleasant

Negative 1 2 3 4 5 6 7 Positive

I perceived the very close interaction with the PHYSICAL person as: Pleasant 1 2 3 4 5 6 7 Unpleasant

Negative 1 2 3 4 5 6 7 Positive

Supplementary Material: The Social Perceptual Salience Effect 11

References

Berger, R., Akselrod, S., Gordon, D., & Cohen, R. (2007). An efficient algorithm for spectral analysis of heart rate variability. Biomedical Engineering, IEEE Transactions on(9), 900–904.

Camm, A., Malik, M., Bigger, J., Breithardt, G., Cerutti, S., Cohen, R., et al. (1996). Heart rate variability: standards of measurement, physiological interpretation, and clinical use. Circulation, 93(5), 1043–1065.

Eckmann, J., Kamphorst, S., & Ruelle, D. (1987). Recurrence plots of dynamical systems.

EPL (Europhysics Letters), 4, 973.

Fusheng, Y., Bo, H., & Qingyu, T. (2000). Approximate Entropy and its application in biosignal analysis. Nonlinear biomedical signal processing, 72.

Ishchenko, A., & Shev’ev, P. (1989). Automated complex for multiparameter analysis of the galvanic skin response signal. Biomedical Engineering, 23(3), 113–117.

Jolliffe, I. (2002). Principal component analysis. Wiley Online Library.

Marwan, N., Carmen Romano, M., Thiel, M., & Kurths, J. (2007). Recurrence plots for the analysis of complex systems. Physics Reports, 438(5-6), 237–329.

Pan, J., & Tompkins, W. (1985). A real-time QRS detection algorithm. IEEE Transactions on Biomedical Engineering, 230–236.

Richman, J., & Moorman, J. (2000). Physiological time-series analysis using approximate entropy and sample entropy. American Journal of Physiology- Heart and Circulatory Physiology , 278(6), H2039.

Schinkel, S., Dimigen, O., & Marwan, N. (2008). Selection of recurrence threshold for signal detection. The European Physical Journal-Special Topics, 164(1), 45–53.

Valenza, G., Lanata, A., & Scilingo, E. P. (2011). The Role of Nonlinear Dynamics in Affective Valence and Arousal Recognition. IEEE Transaction on Affective Computing, 1–14.

Supplementary Material: The Social Perceptual Salience Effect 12

Zbilut, J., & Webber Jr, C. (2006). Recurrence quantification analysis. Wiley Online

Library.

Supplementary Material: The Social Perceptual Salience Effect 13

Table 1

NMC classifier - Physiological signal classification based on a 20 component feature set. The rows are "response class" while the columns are "stimulus class". The table must be read column-wise.

| | | |

|NMC |Physical Intimate |Virtual Intimate |

| | | |

|Physical Intimate |88.23 (21.5) |23.53 (25.3) |

|Virtual Intimate |11.76 (21.5) |76.47 (25.3) |

| | | |

| | | |

| | | |

| |Physical Neutral |Physical Intimate |

| | | |

|Physical Neutral |98.75 (7.9) |0.0 (0.0) |

|Physical Intimate |1.25 (7.9) |100.0 (0.0) |

| | | |

| | | |

| | | |

| |Virtual Neutral |Virtual Intimate |

| | | |

|Virtual Neutral |79.15 (25.0) |12.50 (21.9) |

|Virtual Intimate |20.84 (25.0) |87.50 (21.9) |

| | | |

| | | |

| | | |

| |Virtual Neutral |Physical Neutral |

| | | |

|Virtual Neutral |60.00 (35.7) |48.33 (33.4) |

|Physical Neutral |40.00 (35.7) |51.66 (33.4) |

-----------------------

1

N

j

C m

j

j

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download