ACKNOWLEDGEMENT - University of Missouri
ACTIVE LABELING IN DEEP LEARNING AND ITS APPLICATION TO EMOTION PREDICTION ___________________________A Dissertation Presented to the Faculty of the Graduate SchoolUniversity of Missouri-Columbia___________________________In Partial Fulfillment of the Requirements for the Degree ofPh.D. in Computer Science___________________________byDan WangAdvisor: Dr. Yi ShangNovember 2013The undersigned, appointed by the Dean of the Graduate School, have examined the thesis entitledACTIVE LABELING IN DEEP LEARNING AND ITS APPLICATION TO EMOTION PREDICTIONpresented by Dan Wanga candidate for the degree of Doctor of Philosophyand hereby certify that in their opinion it is worthy of acceptance.Dr. Yi ShangDr. Wenjun ZengDr. Dale MusserDr. Tony HanACKNOWLEDGEMENTFirst and foremost, I would like to show my appreciation to my advisor, Dr. Yi Shang, for his continuous support, guidance, and encouragement throughout my PhD study and research. His deep insights and broad knowledge constantly keep my research on the right track. I am also impressed by his patience and tireless help. The weekly individual and group meetings with him are my greatest pleasure. He is a supportive advisor not only on my study and research, but also on my life and career path. I would like to thank my committee members, Dr. Wenjun Zeng, Dr. Dale Musser, and Dr. Tony Han for reviewing my dissertation, attending my comprehensive exam and defense, and offering all the constructive suggestions, guidance, and comments.Thanks to Jodette Lenser and Sandra Moore for taking care of my academic needs. Thanks to all the faculty and staff in the Department of Computer Science at University of Missouri, for providing me with such a nice platform.I also want to thank all the people in our research group, especially Peng Zhuang, Qi Qi, and Chao Fang, for their selfless help. It was fun to exchange ideas and thoughts with these great guys. Many thanks go to Qia Wang in Dr. Zeng’s group and Liyang Rui in Dr. Ho’s group. I will never forget the good time when we worked together as a team.Finally, I would like to thank my parents for their support from the other side of the Earth.Table of Contents TOC \o "1-4" \h \z \u ACKNOWLEDGEMENT PAGEREF _Toc373367082 \h IIITable of Contents PAGEREF _Toc373367083 \h IVList of Figures PAGEREF _Toc373367084 \h VIIList of Tables PAGEREF _Toc373367085 \h XAbstract PAGEREF _Toc373367086 \h XIChapter 1 Introduction PAGEREF _Toc373367087 \h 11.1 Motivations PAGEREF _Toc373367088 \h 11.2 Contributions PAGEREF _Toc373367089 \h 41.3 Outline of the Dissertation PAGEREF _Toc373367090 \h 5Chapter 2 Deep Learning Background and General Knowledge PAGEREF _Toc373367091 \h 62.1 Background PAGEREF _Toc373367092 \h 62.2 The Building Blocks PAGEREF _Toc373367093 \h 92.2.1 Restricted Boltzmann Machine PAGEREF _Toc373367094 \h 92.2.2 Autoencoder PAGEREF _Toc373367095 \h 142.3 Unsupervised Learning Stage PAGEREF _Toc373367096 \h 162.4 Supervised Learning Stage PAGEREF _Toc373367097 \h 172.5 Deep Learning Performance Evaluation on MNIST PAGEREF _Toc373367098 \h 182.5.1 Stacked RBMs PAGEREF _Toc373367099 \h 182.5.2 Stacked Autoencoders PAGEREF _Toc373367100 \h 252.6 State of the Art on Deep Learning PAGEREF _Toc373367101 \h 27Chapter 3 Meta-parameters and Data Pre-processing in Deep Learning PAGEREF _Toc373367102 \h 293.1 Motivation PAGEREF _Toc373367103 \h 293.2 Dimensionality Reduction by PCA and Whitening PAGEREF _Toc373367104 \h 303.3 Sleep Stage Dataset and its Features PAGEREF _Toc373367105 \h 323.4 Investigation of Optimal Meta-parameters and Pre-processing Techniques PAGEREF _Toc373367106 \h 343.5 Experimental Results on MNIST Dataset PAGEREF _Toc373367107 \h 353.5.1 Data Pre-processing PAGEREF _Toc373367108 \h 353.5.1.1 On Raw Data PAGEREF _Toc373367109 \h 353.5.1.2 On PCA With or Without Whitening PAGEREF _Toc373367110 \h 363.5.2 Deep Learning Network Structure PAGEREF _Toc373367111 \h 393.5.2.1 Stacked RBMs on Raw Data PAGEREF _Toc373367112 \h 403.5.2.2 Stacked RBMs on PCA at Retention Rate 95% PAGEREF _Toc373367113 \h 413.5.2.3 Stacked Autoencoders on Raw Data PAGEREF _Toc373367114 \h 423.5.2.4 Stacked Autoencoders on PCA at Retention Rate 95% PAGEREF _Toc373367115 \h 433.6 Experimental Results on Sleep Stage Dataset PAGEREF _Toc373367116 \h 453.6.1 Data Pre-processing PAGEREF _Toc373367117 \h 453.6.1.1 On Raw Data PAGEREF _Toc373367118 \h 453.6.1.2 On Features PAGEREF _Toc373367119 \h 453.6.2 Deep Learning Network Structure PAGEREF _Toc373367120 \h 453.6.2.1 Stacked RBMs on Raw Data PAGEREF _Toc373367121 \h 453.6.2.2 Stacked RBMs on Features PAGEREF _Toc373367122 \h 473.6.2.3 Stacked Autoencoders on Raw Data PAGEREF _Toc373367123 \h 483.6.2.4 Stacked Autoencoders on Features PAGEREF _Toc373367124 \h 483.7 Summary PAGEREF _Toc373367125 \h 49Chapter 4 Active Labeling in Deep Learning PAGEREF _Toc373367126 \h 514.1 Motivation PAGEREF _Toc373367127 \h 514.2 Related Works on Active Learning PAGEREF _Toc373367128 \h 534.3 Algorithms PAGEREF _Toc373367129 \h 534.4 Experimental Results on MNIST PAGEREF _Toc373367130 \h 564.4.1 Stacked RBMs on Raw Data PAGEREF _Toc373367131 \h 574.4.2 Stacked Autoencoders on Raw Data PAGEREF _Toc373367132 \h 614.4.3 Stacked Autoencoders on PCA at Retention Rate 95% PAGEREF _Toc373367133 \h 614.5 Experimental results on sleep stage dataset PAGEREF _Toc373367134 \h 624.5.1 Stacked RBMs on Raw Data PAGEREF _Toc373367135 \h 634.5.2 Stacked RBMs on Features PAGEREF _Toc373367136 \h 634.5.3 Stacked Autoencoders on Features PAGEREF _Toc373367137 \h 644.6 Summary PAGEREF _Toc373367138 \h 65Chapter 5 Modeling Raw Physiological Data and Predicting Emotions with Deep Belief Networks PAGEREF _Toc373367139 \h 665.1 Introduction PAGEREF _Toc373367140 \h 665.2 Related Works on Modeling Physiological Data and Emotion Prediction PAGEREF _Toc373367141 \h 695.3 Train a DBN Classifier on Physiological Data PAGEREF _Toc373367142 \h 705.4 Experimental Results PAGEREF _Toc373367143 \h 725.5 Discussion PAGEREF _Toc373367144 \h 77Chapter 6 Conclusion PAGEREF _Toc373367145 \h 78Reference PAGEREF _Toc373367146 \h 80Publications PAGEREF _Toc373367147 \h 87VITA PAGEREF _Toc373367148 \h 88List of Figures TOC \h \z \c "Fig." Fig. 1 An example of shallow neutral networks PAGEREF _Toc373367149 \h 7Fig. 2 A DBN schema with three hidden layers. (a) The pre-training stage without labels involved (b) The fine-tuning stage PAGEREF _Toc373367150 \h 8Fig. 3 Graphical depiction of an RBM PAGEREF _Toc373367151 \h 10Fig. 4 Updating an RBM with contrastive divergence (k = 1) PAGEREF _Toc373367152 \h 13Fig. 5 An autoencoder PAGEREF _Toc373367153 \h 15Fig. 6 Learned features of the first hidden layer PAGEREF _Toc373367154 \h 20Fig. 7 Learned features of the second hidden layer PAGEREF _Toc373367155 \h 21Fig. 8 Learned features of the third hidden layer PAGEREF _Toc373367156 \h 21Fig. 9 Learned features of the first hidden layer of sRBM PAGEREF _Toc373367157 \h 23Fig. 10 Learned features of the second hidden layer of sRBM PAGEREF _Toc373367158 \h 23Fig. 11 Learned features of the third hidden layer of sRBM PAGEREF _Toc373367159 \h 24Fig. 12 The predication accuracy of a 784-150-150-150-10 DBN using different numbers of unlabeled and labeled samples. The legends show the numbers of labeled samples. PAGEREF _Toc373367160 \h 25Fig. 13 First hidden layer of stacked autoencoders detects edges PAGEREF _Toc373367161 \h 26Fig. 14 Second layer of stacked autoencoders detects contours PAGEREF _Toc373367162 \h 27Fig. 15 60-second raw data downsampled at 64Hz of sleep stage dataset. PAGEREF _Toc373367163 \h 33Fig. 16 28 features of a subject's data. X-axis is time in seconds and y-axis shows non-normalized values PAGEREF _Toc373367164 \h 34Fig. 17 sleep stage classes of a subject. X-axis is time in seconds and y-axis is the 5 possible sleep stages PAGEREF _Toc373367165 \h 34Fig. 18 weights learned by 10 softmax nodes on raw of MNIST PAGEREF _Toc373367166 \h 36Fig. 19 PCA with or without whitening PAGEREF _Toc373367167 \h 38Fig. 20 Classification accuracy of softmax regression on MNIST PAGEREF _Toc373367168 \h 38Fig. 21Training time of softmax regression on MNIST PAGEREF _Toc373367169 \h 39Fig. 22 Classification accuracy of stacked RBMs of various net structures on raw of MNIST PAGEREF _Toc373367170 \h 40Fig. 23 Training time of stacked RBMs of various net structures on raw of MNIST PAGEREF _Toc373367171 \h 41Fig. 24 Classification accuracy of stacked RBMs of various net structures on PCA95 whitened of MNIST PAGEREF _Toc373367172 \h 41Fig. 25 Training time of stacked RBMs of various net structures on raw of MNIST PAGEREF _Toc373367173 \h 42Fig. 26 Classification accuracy of stacked autoencoders of various net structures on raw of MNIST PAGEREF _Toc373367174 \h 43Fig. 27 Training time of stacked autoencoders of various net structures on raw of MNIST PAGEREF _Toc373367175 \h 43Fig. 28 Classification accuracy of stacked autoencoders of various net structures on PCA95 whitened of MNIST PAGEREF _Toc373367176 \h 44Fig. 29 Training time of stacked autoencoders of various net structures on PCA95 whitened of MNIST PAGEREF _Toc373367177 \h 44Fig. 30 Classification accuracy of stacked RBMs of various net structures on raw of sleep stage dataset PAGEREF _Toc373367178 \h 46Fig. 31Training time of stacked RBMs of various net structures on raw of sleep stage dataset PAGEREF _Toc373367179 \h 46Fig. 32 Classification accuracy of stacked RBMs of various net structures on features of sleep stage dataset PAGEREF _Toc373367180 \h 47Fig. 33 Training time of stacked RBMs of various net structures on features of sleep stage dataset PAGEREF _Toc373367181 \h 48Fig. 34 Classification accuracy of stacked autoencoders of various net structures on features of sleep stage dataset PAGEREF _Toc373367182 \h 49Fig. 35 Training time of stacked autoencoders of various net structures on features of sleep stage dataset PAGEREF _Toc373367183 \h 49Fig. 36 The pool-based active learning cycle (taken from [4]) PAGEREF _Toc373367184 \h 52Fig. 37 Active labeling – deep learning framework PAGEREF _Toc373367185 \h 54Fig. 38 The mean of classification accuracy of random labeling DBN and active labeling DBN. RL and AL-LC show their error bars. Iteration 0 means only the seed set is used. The performances of the three AL DBNs are close, all better than that of RL DBN. PAGEREF _Toc373367186 \h 58Fig. 39 All the 200 samples picked before the 1st iteration by RL DBN (top) and AL-LC DBN (bottom). The digits in the bottom graph look harder to recognize, indicating AL-LC DBN does pick more challenging samples than RL DBN PAGEREF _Toc373367187 \h 60Fig. 40 The comparison of classification error rates on training and validation sets with RL-DBN and AL-DBN-LC in one iteration. AL-DBN-LC works much worse on training and validation sets because it always picks the most challenging samples. PAGEREF _Toc373367188 \h 60Fig. 41 The mean of classification accuracy of random labeling stacked autoencoders and active labeling stacked autoencoders. RL and AL-LC show their error bars. Iteration 0 means only the seed set is used. PAGEREF _Toc373367189 \h 61Fig. 42 The mean of classification accuracy of random labeling stacked autoencoders and active labeling stacked autoencoders. RL and AL-LC show their error bars. Iteration 0 means only the seed set is used. PAGEREF _Toc373367190 \h 62Fig. 43 The mean of classification accuracy of random labeling stacked RBMs and active labeling stacked RBMs. RL and AL-LC show their error bars. Iteration 0 means only the seed set is used. PAGEREF _Toc373367191 \h 63Fig. 44 The mean of classification accuracy of random labeling stacked RBMs and active labeling stacked RBMs. RL and AL-LC show their error bars. Iteration 0 means only the seed set is used. PAGEREF _Toc373367192 \h 64Fig. 45 The mean of classification accuracy of random labeling stacked autoencoders and active labeling stacked autoencoders. RL and AL-LC show their error bars. Iteration 0 means only the seed set is used. PAGEREF _Toc373367193 \h 65Fig. 46 Learned EOG features. (top) hEOG (buttom) vEOG PAGEREF _Toc373367194 \h 73Fig. 47 Learned EMG features. (top) zEMG (bottom) tEMG PAGEREF _Toc373367195 \h 74Fig. 48 Accuracy of DBN classification. Error bars for DBN classification accuracy on raw data and filled dots for Gaussian na?ve Bayes on features PAGEREF _Toc373367196 \h 75Fig. 49 Histogram of accuracy distribution in 32 subjects PAGEREF _Toc373367197 \h 76List of Tables TOC \h \z \c "Table" Table 1 DBN parameters on MNIST PAGEREF _Toc373367198 \h 19Table 2 Stacked autoencoders parameters on MNIST PAGEREF _Toc373367199 \h 26Table 3 Summary of network structures including optimal settings as well as pre-processing techniques on MINIST and sleep stage dataset PAGEREF _Toc373367200 \h 50Table 4 DBN parameters for both random and AL schemes PAGEREF _Toc373367201 \h 57Table 5 DBN parameters PAGEREF _Toc373367202 \h 72AbstractRecent breakthroughs in deep learning have made possible the learning of deep layered hierarchical representations of sensory input. Stacked restricted Boltzmann machines (RBMs), also called deep belief networks (DBNs), and stacked autoencoders are two representative deep learning methods. The key idea is greedy layer-wise unsupervised pre-training followed by supervised ?ne-tuning, which can be done efficiently and overcomes the difficulty of local minima when training all layers of a deep neural network at once. Deep learning has been shown to achieve outstanding performance in a number of challenging real-world applications. Existing deep learning methods involve a large number of meta-parameters, such as the number of hidden layers, the number of hidden nodes, the sparsity target, the initial values of weights, the type of units, the learning rate, etc. Existing applications usually do not explain why the decisions were made and how changes would affect performance. Thus, it is difficult for a novice user to make good decisions for a new application in order to achieve good performance. In addition, most of the existing works are done on simple and clean datasets and assume a fixed set of labeled data, which is not necessarily true for real-world applications. The main objectives of this dissertation are to investigate the optimal meta-parameters of deep learning networks as well as the effects of various data pre-processing techniques, propose a new active labeling framework for cost-effective selection of labeled data, and apply deep learning to a real-world application – emotion prediction via physiological sensor data, based on real-world, complex, noisy, and heterogeneous sensor data. For meta-parameters and data pre-processing techniques, this study uses the benchmark MNIST digit recognition image dataset and a sleep-stage-recognition sensor dataset and empirically compares the deep network’s performance with a number of different meta-parameters and decisions, including raw data vs. pre-processed data by Principal Component Analysis (PCA) with or without whitening, various structures in terms of the number of layers and the number of nodes in each layer, stacked RBMs vs. stacked autoencoders. For active labeling, a new framework for both stacked RBMs and stacked autoencoders is proposed based on three metrics: least confidence, margin sampling, and entropy. On the MINIST dataset, the methods outperform random labeling consistently by a significant margin. On the other hand, the proposed active labeling methods perform similarly to random labeling on the sleep-stage-recognition dataset due to the noisiness and inconsistency in the data. For the application of deep learning to emotion prediction via physiological sensor data, a software pipeline has been developed. The system first extracts features from the raw data of four channels in an unsupervised fashion and then builds three classifiers to classify the levels of arousal, valence, and liking based on the learned features. The classification accuracy is 0.609, 0.512, and 0.684, respectively, which is comparable with existing methods based on expert designed features.Chapter 1Introduction1.1 MotivationsShallow neural networks with an input layer, a single hidden layer, and an output layer require more computational elements or are hard to model complex concepts and multi-level abstractions. In contrast, multi-layer neural networks provide better representational power and could derive more descriptive multi-level models due to their hierarchical structures, with each higher layer representing higher-level abstraction of the input data. Unfortunately, it is difficult to train all layers of a deep neutral network at once ADDIN EN.CITE <EndNote><Cite><Author>Bengio</Author><Year>2007</Year><RecNum>45</RecNum><DisplayText>[1]</DisplayText><record><rec-number>45</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">45</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Bengio, Y.</author><author>Lamblin, P.</author><author>Popovici, D.</author><author>Larochelle, H.</author></authors></contributors><titles><title>Greedy layer-wise training of deep networks</title><secondary-title>Advances in neural information processing systems</secondary-title></titles><periodical><full-title>Advances in neural information processing systems</full-title></periodical><pages>153</pages><volume>19</volume><dates><year>2007</year></dates><isbn>1049-5258</isbn><urls></urls><research-notes>good!
to handle real-valued data
UCI Abalone data set
compared w/ shallow networks
stacked autoencoders</research-notes></record></Cite></EndNote>[1]. With random initial weights, the learning is likely to get stuck in local minima.The breakthrough happened in 2006, when Hinton introduced a novel way called Deep Belief Networks (DBNs) to train multi-layer neural networks to learn features from unlabeled data ADDIN EN.CITE <EndNote><Cite><Author>Hinton</Author><Year>2006</Year><RecNum>31</RecNum><DisplayText>[2]</DisplayText><record><rec-number>31</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">31</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Hinton, G.E.</author><author>Salakhutdinov, R.R.</author></authors></contributors><titles><title>Reducing the dimensionality of data with neural networks</title><secondary-title>Science</secondary-title></titles><periodical><full-title>Science</full-title></periodical><pages>504-507</pages><volume>313</volume><number>5786</number><dates><year>2006</year></dates><isbn>0036-8075</isbn><urls></urls></record></Cite></EndNote>[2]. A DBN trains a multi-layer neural network in a greedy fashion, each layer being a restricted Boltzmann machine (RBM) ADDIN EN.CITE <EndNote><Cite><Author>Smolensky</Author><Year>1986</Year><RecNum>64</RecNum><DisplayText>[3]</DisplayText><record><rec-number>64</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">64</key></foreign-keys><ref-type name="Book Section">5</ref-type><contributors><authors><author>Smolensky, Paul</author></authors></contributors><titles><title>Information processing in dynamical systems: Foundations of harmony theory</title><secondary-title>Parallel distributed processing: explorations in the microstructure of cognition</secondary-title></titles><pages>194-281</pages><dates><year>1986</year></dates><pub-location>Cambridge, MA, USA</pub-location><publisher>MIT Press</publisher><isbn>0-262-68053-X</isbn><urls></urls></record></Cite></EndNote>[3]. The trained weights and biases in each layer can be thought of as the features or filters learned from the input data. Then the weights and biases act as the initial values for the supervised fine-tuning using backpropogation. In short, a DBN discovers features on its own and does semi-supervised learning by modeling unlabeled data first in an unsupervised way and then incorporates labels in a supervised fashion.A similar approach to the above mentioned stacked RBMs was introduced by Bengio ADDIN EN.CITE <EndNote><Cite><Author>Bengio</Author><Year>2007</Year><RecNum>45</RecNum><DisplayText>[1]</DisplayText><record><rec-number>45</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">45</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Bengio, Y.</author><author>Lamblin, P.</author><author>Popovici, D.</author><author>Larochelle, H.</author></authors></contributors><titles><title>Greedy layer-wise training of deep networks</title><secondary-title>Advances in neural information processing systems</secondary-title></titles><periodical><full-title>Advances in neural information processing systems</full-title></periodical><pages>153</pages><volume>19</volume><dates><year>2007</year></dates><isbn>1049-5258</isbn><urls></urls><research-notes>good!
to handle real-valued data
UCI Abalone data set
compared w/ shallow networks
stacked autoencoders</research-notes></record></Cite></EndNote>[1] using stacked autoencoders instead of stacked RBMs. Each autoencoder is a one-hidden-layer neural network with the same number of nodes in the input and output layers, which tries to learn an approximation to the identity function by applying regular backpropagation. The hidden layer with fewer units (or with a sparsity term) is a compressed representation of the input data, thus discovering interesting structure about the data. An autoencoder serves as a building block for deep neural networks similar to an RBM. The supervised fine-tuning stage could also be applied to stacked autoencoders by incorporating a softmax on top to train a classifier. One issue with current applications on deep learning is the lack of explanations about how to achieve a large number of meta-parameters that yield good results. Firstly, the authors usually fail to present the meta-parameters tuning process in their literature, particularly, the number of hidden layers and the number of hidden nodes. Thus, it is difficult for novice users to replicate the similar results on different problems. Secondly, although deep learning has the ability to learn features automatically from raw data, it may be interesting to investigate the effects of various data pre-processing techniques, either hand engineered features or commonly used dimension reduction algorithms such as principle component analysis (PCA) with or without whitening. Thirdly, RBMs and autoencoders are both designed to represent input data in a compressed fashion, but whether they perform the same in different problems remains questionable.Another practical problem with deep learning framework is how to choose samples to be labeled. Since the labeled data are scarce and expensive, it makes sense to choose the most informative samples to be labeled and then deep learning fine-tunes the classification models using these labeled data. The existing researches on deep learning assume the labeled data are passive, either available there already or obtained from the samples randomly chosen to be labeled by human experts. The former may not be practical; the latter does not yield the best results. Suppose there is a budget on the number of samples to be labeled. It is expected to produce better classification performance in most cases (but the same or even worse performance on some datasets) for an active labeling algorithm to always select the most challenging samples to be labeled. This falls into the well-defined active learning framework ADDIN EN.CITE <EndNote><Cite><Author>Settles</Author><Year>2010</Year><RecNum>69</RecNum><DisplayText>[4]</DisplayText><record><rec-number>69</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">69</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Settles, Burr</author></authors></contributors><titles><title>Active learning literature survey</title><secondary-title>University of Wisconsin, Madison</secondary-title></titles><periodical><full-title>University of Wisconsin, Madison</full-title></periodical><dates><year>2010</year></dates><urls></urls><research-notes>active learning</research-notes></record></Cite></EndNote>[4]. However, how to estimate which samples are more informative or more challenging remains unexplored with deep learning. Last, despite the power of deep learning, blindly applying it to real scenarios does not yield satisfactory results, except on toy datasets. A pipeline of applying deep learning to actual problems is desired, which includes raw data pre-processing, raw data selection and division, normalization, randomization, and deep learning training and classification. To map physiological data to emotions using deep learning could serve as a good example to articulate the process. Physiological data is collected by sensors as a means of human-computer interaction by monitoring, analyzing and responding to psychophysiological activities ADDIN EN.CITE <EndNote><Cite><Author>Fairclough</Author><Year>2009</Year><RecNum>7</RecNum><DisplayText>[5]</DisplayText><record><rec-number>7</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">7</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Stephen H. Fairclough</author></authors></contributors><titles><title>Fundamentals of physiological computing</title><secondary-title>Interact. Comput.</secondary-title></titles><periodical><full-title>Interact. Comput.</full-title></periodical><pages>133-145</pages><volume>21</volume><number>1-2</number><dates><year>2009</year></dates><isbn>0953-5438</isbn><urls></urls><custom1>1507839</custom1><electronic-resource-num>10.1016/j.intcom.2008.10.011</electronic-resource-num></record></Cite></EndNote>[5]. The data types include a lot of channels, useful to predict human’s physical activities, emotions, and even potential diseases. Since the goals vary a lot, it is difficult to know useful features without expert’s knowledge. Deep learning is promising to overcome the barrier by extracting useful features automatically. Moreover, deep learning, acting as a semi-supervised machine learning algorithm, takes advantage of scarce labeled data and abundant unlabeled data in this scenario.1.2 ContributionsThis dissertation makes three major contributions to the area of deep learning that are summarized as follows.First, I investigate the optimal meta-parameters of deep learning networks as well as the effects of various data pre-processing techniques. This study uses the benchmark MNIST digit recognition image dataset and a sleep-stage-recognition sensor dataset and empirically compares the deep learning network’s performance with quite a few combinations of settings, including raw data vs. pre-processed data by Principal Component Analysis (PCA) with or without whitening for MNIST and hand extracted features for the sleep stage dataset, various structures in terms of the number of layers and the number of nodes in each layer, different building blocks including stacked RBMs vs. stacked autoencoders. The process is presented as a guideline for future deep learning applications to tune meta-parameters and data pre-processing.Second, I propose a new active labeling framework for deep learning including both stacked RBMs and stacked autoencoders based on three metrics: least confidence, margin sampling, and entropy, for cost-effective selection of labeled data. Then I investigate the performance of the active labeling deep learning technique in all the three metrics, compared to a random labeling strategy, on the raw data and features of the MNIST dataset and the sleep stage dataset.Last, I develop a pipeline to apply deep learning to emotion prediction via physiological data, based on real-world, complex, noisy, and heterogeneous sensor data. The system first extracts features from the raw data of four channels in an unsupervised fashion and then builds three classifiers to classify the levels of arousal, valence, and liking based on the learned features.1.3 Outline of the DissertationThis dissertation is organized into the following chapters.In Chapter 1, the motivations and the scope of the proposed research are introduced.Chapter 2 presents the background knowledge and state-of-the-art techniques of deep learning, and introduces two types of building blocks and their performance on a benchmark hand written digit dataset MNIST.In Chapter 3, deep learning is revisited to explore the optimal settings of the input pre-processing, neural network structure, and the learning unit types.In Chapter 4, an active labeling deep learning framework is proposed to choose the most informative samples to be labeled. Three criteria are introduced in the active labeling framework. In Chapter 5, the application of DBNs on the physiological dataset DEAP ADDIN EN.CITE <EndNote><Cite><Author>Koelstra</Author><Year>2011</Year><RecNum>13</RecNum><DisplayText>[6]</DisplayText><record><rec-number>13</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">13</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Koelstra, S.</author><author>Muhl, C.</author><author>Soleymani, M.</author><author>Lee, J.</author><author>Yazdani, A.</author><author>Ebrahimi, T.</author><author>Pun, T.</author><author>Nijholt, A.</author><author>Patras, I.</author></authors></contributors><titles><title>Deap: A database for emotion analysis using physiological signals</title><secondary-title>Affective Computing, IEEE Transactions on</secondary-title></titles><periodical><full-title>Affective Computing, IEEE Transactions on</full-title></periodical><pages>1-1</pages><number>99</number><dates><year>2011</year></dates><isbn>1949-3045</isbn><urls></urls></record></Cite></EndNote>[6] is developed and its performance is evaluated.Chapter 6 concludes the dissertation.Chapter 2Deep Learning Background and General Knowledge2.1 BackgroundAn example of shallow neural networks with an input layer, a single hidden layer, and an output layer, as shown in REF _Ref354949783 \h \* MERGEFORMAT Fig. 1, can be trained with backpropagation for classification or regression. Theoretically a shallow net can approximate any functions as long as it has enough units in the hidden layer ADDIN EN.CITE <EndNote><Cite><Author>Leshno</Author><Year>1993</Year><RecNum>72</RecNum><DisplayText>[7, 8]</DisplayText><record><rec-number>72</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">72</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Leshno, Moshe</author><author>Lin, Vladimir Ya</author><author>Pinkus, Allan</author><author>Schocken, Shimon</author></authors></contributors><titles><title>Multilayer feedforward networks with a nonpolynomial activation function can approximate any function</title><secondary-title>Neural networks</secondary-title></titles><periodical><full-title>Neural networks</full-title></periodical><pages>861-867</pages><volume>6</volume><number>6</number><dates><year>1993</year></dates><isbn>0893-6080</isbn><urls></urls></record></Cite><Cite><Author>Funahashi</Author><Year>1989</Year><RecNum>73</RecNum><record><rec-number>73</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">73</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Funahashi, Ken-Ichi</author></authors></contributors><titles><title>On the approximate realization of continuous mappings by neural networks</title><secondary-title>Neural networks</secondary-title></titles><periodical><full-title>Neural networks</full-title></periodical><pages>183-192</pages><volume>2</volume><number>3</number><dates><year>1989</year></dates><isbn>0893-6080</isbn><urls></urls></record></Cite></EndNote>[7, 8], but the size of hidden units grows exponentially with the input layer size ADDIN EN.CITE <EndNote><Cite><Author>Hastad</Author><Year>1986</Year><RecNum>74</RecNum><DisplayText>[9]</DisplayText><record><rec-number>74</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">74</key></foreign-keys><ref-type name="Conference Proceedings">10</ref-type><contributors><authors><author>Hastad, Johan</author></authors></contributors><titles><title>Almost optimal lower bounds for small depth circuits</title><secondary-title>Proceedings of the eighteenth annual ACM symposium on Theory of computing</secondary-title></titles><pages>6-20</pages><dates><year>1986</year></dates><publisher>ACM</publisher><isbn>0897911938</isbn><urls></urls></record></Cite></EndNote>[9]. So many units are needed in the hidden layer of a shallow neural network for a representative model is because the single hidden layer is just one step away from the input layer and the output layer, which is forced to translate the raw data from the input layer to complex features that can be used for classification in the output layer. Too many hidden units increase computational complexity, and even worse, easily result in overfitting, especially when the training set size is relatively small. In contrast, a deep network with two or more hidden layers provides better representational power and thus obtains more descriptive models thanks to feature sharing and abstraction. A lower layer’s features are reused by the layer above it, whereas a higher layer represents higher-level abstraction of data. In deep nets lower layers are relaxed to learn simple or concrete features whereas higher layers tend to represent complex or abstract features. For example, to transform the raw input images of handwritten digits into three gradually higher levels of representations, the first layer could feature key dots, the second layer could represent lines and curves, and the features learned by the third layer are closer to more meaningful digit parts. EnhancedMetaFilefalse\f 0 \* MERGEFORMATFig. SEQ Fig. \* ARABIC 1 An example of shallow neutral networksUnfortunately, it is difficult to train deep neutral networks all layers at once ADDIN EN.CITE <EndNote><Cite><Author>Bengio</Author><Year>2007</Year><RecNum>45</RecNum><DisplayText>[1]</DisplayText><record><rec-number>45</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">45</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Bengio, Y.</author><author>Lamblin, P.</author><author>Popovici, D.</author><author>Larochelle, H.</author></authors></contributors><titles><title>Greedy layer-wise training of deep networks</title><secondary-title>Advances in neural information processing systems</secondary-title></titles><periodical><full-title>Advances in neural information processing systems</full-title></periodical><pages>153</pages><volume>19</volume><dates><year>2007</year></dates><isbn>1049-5258</isbn><urls></urls><research-notes>good!
to handle real-valued data
UCI Abalone data set
compared w/ shallow networks
stacked autoencoders</research-notes></record></Cite></EndNote>[1]. With large initial weights, the learning is likely to get stuck in local minima. With small initial weights, the gradients are tiny, so the training takes forever to converge.Hinton ADDIN EN.CITE <EndNote><Cite><Author>Hinton</Author><Year>2006</Year><RecNum>31</RecNum><DisplayText>[2, 10]</DisplayText><record><rec-number>31</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">31</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Hinton, G.E.</author><author>Salakhutdinov, R.R.</author></authors></contributors><titles><title>Reducing the dimensionality of data with neural networks</title><secondary-title>Science</secondary-title></titles><periodical><full-title>Science</full-title></periodical><pages>504-507</pages><volume>313</volume><number>5786</number><dates><year>2006</year></dates><isbn>0036-8075</isbn><urls></urls></record></Cite><Cite><Author>Hinton</Author><Year>2006</Year><RecNum>28</RecNum><record><rec-number>28</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">28</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Hinton, G.E.</author><author>Osindero, S.</author><author>Teh, Y.W.</author></authors></contributors><titles><title>A fast learning algorithm for deep belief nets</title><secondary-title>Neural computation</secondary-title></titles><periodical><full-title>Neural computation</full-title></periodical><pages>1527-1554</pages><volume>18</volume><number>7</number><dates><year>2006</year></dates><isbn>0899-7667</isbn><urls></urls></record></Cite></EndNote>[2, 10] proposed deep belief networks (DBNs) to overcome the difficulties by constructing multilayer restricted Boltzmann machines (RBMs) ADDIN EN.CITE <EndNote><Cite><Author>Smolensky</Author><Year>1986</Year><RecNum>64</RecNum><DisplayText>[3]</DisplayText><record><rec-number>64</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">64</key></foreign-keys><ref-type name="Book Section">5</ref-type><contributors><authors><author>Smolensky, Paul</author></authors></contributors><titles><title>Information processing in dynamical systems: Foundations of harmony theory</title><secondary-title>Parallel distributed processing: explorations in the microstructure of cognition</secondary-title></titles><pages>194-281</pages><dates><year>1986</year></dates><pub-location>Cambridge, MA, USA</pub-location><publisher>MIT Press</publisher><isbn>0-262-68053-X</isbn><urls></urls></record></Cite></EndNote>[3] and training them layer-by-layer in a greedy fashion. Since the network consists of a stack of RBMs, it is also called stacked RBMs. We use DBN and stacked RBMs interchangeably in this dissertation, as opposed to stacked autoencoders to be introduced later.The training process has two stages. The first is the pre-training stage, in which no labels are involved and the training is done in an unsupervised way. The training starts off with the bottom two layers to obtain features in hidden layer 1 from the input layer. Then the training moves up to hidden layer 1 and layer 2, treating layer 1 as the new input to get its features in layer 2. The greedy layer-wise training is performed until reaching the highest hidden layer. The first stage trains a generative model as weights between layers to capture the raw input’s features, resulting in better starting point for the second stage than randomly assigned initial weights. The second is the fine-tuning stage. In this stage, a new layer is put on top of the stacked RBMs of the first stage to construct a discriminative model. The overall schema of a DBN with three hidden layers is shown in REF _Ref353289805 \h \* MERGEFORMAT Fig. 2.EnhancedMetaFilefalse\f 0 \* MERGEFORMATFig. SEQ Fig. \* ARABIC 2 A DBN schema with three hidden layers. (a) The pre-training stage without labels involved (b) The fine-tuning stageA variation to the above mentioned deep learning algorithm was introduced by Bengio ADDIN EN.CITE <EndNote><Cite><Author>Bengio</Author><Year>2007</Year><RecNum>45</RecNum><DisplayText>[1]</DisplayText><record><rec-number>45</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">45</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Bengio, Y.</author><author>Lamblin, P.</author><author>Popovici, D.</author><author>Larochelle, H.</author></authors></contributors><titles><title>Greedy layer-wise training of deep networks</title><secondary-title>Advances in neural information processing systems</secondary-title></titles><periodical><full-title>Advances in neural information processing systems</full-title></periodical><pages>153</pages><volume>19</volume><dates><year>2007</year></dates><isbn>1049-5258</isbn><urls></urls><research-notes>good!
to handle real-valued data
UCI Abalone data set
compared w/ shallow networks
stacked autoencoders</research-notes></record></Cite></EndNote>[1] using stacked autoencoders. An autoencoder neural network is an unsupervised learning algorithm trying to learn an approximation to the identity function by applying regular backpropagation. The hidden layer with fewer nodes (or with a sparsity term) than the input layer learns a compressed representation of the input data, aiming at the same goal as a hidden layer does in stacked RBMs. To exploit stacked autoencoders to do deep learning for a classification task, it also involves an unsupervised pre-training stage and a supervised fine-tuning stage. In the pre-training stage, the output of one hidden layer serves as the input for the higher hidden layer, resulting in a stacked hierarchical structure to learn more and more abstract features. In the fine-tuning stage, a softmax layer is added on top of the stacked autoencoders and a regular backpropagation is applied using the learned weights and biases in the first stage as a starting point. All in all, stacked RBMs and stacked autoencoders only differ in the unsupervised pre-training stage, where different building blocks are used, to attempt to achieve the same functionality.For simplicity, this dissertation refers stacked RBMs with a softmax on top to “stacked RBMs” or a “DBN”. The same rule applies to “stacked autoencoders”.2.2 The Building Blocks2.2.1 Restricted Boltzmann MachineAs the building blocks of DBNs, a restricted Boltzmann machine (RBM) ADDIN EN.CITE <EndNote><Cite><Author>Smolensky</Author><Year>1986</Year><RecNum>64</RecNum><DisplayText>[3]</DisplayText><record><rec-number>64</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">64</key></foreign-keys><ref-type name="Book Section">5</ref-type><contributors><authors><author>Smolensky, Paul</author></authors></contributors><titles><title>Information processing in dynamical systems: Foundations of harmony theory</title><secondary-title>Parallel distributed processing: explorations in the microstructure of cognition</secondary-title></titles><pages>194-281</pages><dates><year>1986</year></dates><pub-location>Cambridge, MA, USA</pub-location><publisher>MIT Press</publisher><isbn>0-262-68053-X</isbn><urls></urls></record></Cite></EndNote>[3] has a visible layer consisting of stochastic, binary nodes as the input and a hidden output layer consisting of stochastic, binary feature detectors as the output, connected by symmetrical weights between nodes in different layers. RBMs have two key features. Firstly, there are no connections between the nodes in the same layer, making tractable learning possible. Second, the hidden units are conditionally independent given the visible layer thanks to the undirected connections, so it is fast to get an unbiased sample from the posterior distribution. A graphical depiction of an RBM is shown in REF _Ref353273793 \h \* MERGEFORMAT Fig. 3.EnhancedMetaFilefalse\f 0 \* MERGEFORMATFig. SEQ Fig. \* ARABIC 3 Graphical depiction of an RBMA joint configuration (v,h) of the visible and hidden nodes can be represented by an energy function given byEv,h= -i,jvihjwij-ibivi-jbjhj MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 1)where E(v,h) is the energy with configuration v on the visible nodes and h on the hidden nodes, vi is the binary state of visible node i, hj is the binary state of hidden node j, wij is the weight between node i and j, and the bias terms are bi for the visible nodes and bj for the hidden nodes (biases not shown in REF _Ref353273793 \h Fig. 3).The probability of a given joint configuration depends on the energy of that joint configuration compared to the energy of all joint configurations, specified bypv,h∝e-E(v,h) MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 2) Concretely its probability is determined by normalizing it by a partition function Z, as shown inpv,h= e-E(v,h)Z MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 3)Z=u,ge-E(u,g) MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 4)The formula implies that the lower energy a configuration has, the higher probability it would occur.The summation along all the hidden units produces the probability of a configuration of the visible units.pv= he-E(v,h)Z MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 5)The binary nodes of the hidden layer are Bernoulli random variables. The probability that node hj is activated, given visible layer v, can be derived from REF NumberRef7055475116 \h \* MERGEFORMAT (4) asPhj=1 v)= σ(bj+ iwijvi) MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 6)σx= 11+e-x MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 7)where σ? is called the sigmoid logistic function.The probability that node vi is activated, given hidden layer h, can be calculated in a similar way as followsPvi=1 h)= σ(bi+ jwijhj) MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 8)It is intractable to compute the gradient of the log likelihood of v directly. Therefore, ADDIN EN.CITE <EndNote><Cite><Author>Hinton</Author><Year>2002</Year><RecNum>66</RecNum><DisplayText>[11]</DisplayText><record><rec-number>66</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">66</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Hinton, Geoffrey E</author></authors></contributors><titles><title>Training products of experts by minimizing contrastive divergence</title><secondary-title>Neural computation</secondary-title></titles><periodical><full-title>Neural computation</full-title></periodical><pages>1771-1800</pages><volume>14</volume><number>8</number><dates><year>2002</year></dates><isbn>0899-7667</isbn><urls></urls></record></Cite></EndNote>[11] proposed contrastive divergence by doing k iterations of Gibbs sampling to approximate it. Note from REF NumberRef2895624638 \h \* MERGEFORMAT (8) it is easy to get an unbiased sample of the visible layer given a hidden vector because there are no direct connections between visible units in an RBM. The algorithm starts with a training vector on the units in the visible layer, then uses the vector to update all the hidden units in parallel, samples from the hidden units, and uses these samples to update all the visible units in parallel to get the reconstruction. This process is applied k (often k=1) iterations to obtain the change to wij as below ?logp(v)?wij≈?wij= ?(<vihj>0- <vihj>k) MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 9)where <?>t is the average over a given size of samples when working with minibatches (described later) at a contrastive divergence iteration t and ? is the learning rate. The update rule to the biases takes the similar form. REF _Ref355565419 \h Fig. 4 illustrates how to update an RBM with the contrastive divergence with k = 1 (CD1).The above-mentioned rule works, but a few tricks are used to accelerate the learning process and/or prevent overfitting. The three mostly commonly used techniques are minibatch, momentum, and weight decay ADDIN EN.CITE <EndNote><Cite><Author>Hinton</Author><Year>2010</Year><RecNum>24</RecNum><DisplayText>[12]</DisplayText><record><rec-number>24</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">24</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Hinton, G.</author></authors></contributors><titles><title>A practical guide to training restricted Boltzmann machines</title><secondary-title>Momentum</secondary-title></titles><periodical><full-title>Momentum</full-title></periodical><pages>1</pages><volume>9</volume><dates><year>2010</year></dates><urls></urls></record></Cite></EndNote>[12].Fig. SEQ Fig. \* ARABIC 4 Updating an RBM with contrastive divergence (k = 1)Minibatch is a minor variation of REF NumberRef3019480109 \h \* MERGEFORMAT (9) in which wij is updated by taking the average over a small batch instead of a single training vector. This produces two advantages. Firstly minibatch works with a less noisy estimate of the gradient since it takes the average and the outliers in the training vectors does not impact much. Secondly it allows a matrix by matrix product instead of a vector by matrix product, which can be taken advantage by modern GPUs or Matlab to speed up the computation. However, it is a bad idea to make the minibatch size too large because the number of updates will decrease accordingly, eventually resulting in inefficiency. Momentum is used to speed up learning by simulating a ball moving on a surface. It is an analogy to the acceleration as if wij were the distance and ?wij were the velocity. Instead of using the estimated gradient to change weights directly as shown in REF NumberRef7747400999 \h \* MERGEFORMAT (9), the momentum method uses it to change the velocity of weights change.?θik=vik=αvik-1-?dEdθi(k) MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 10)where α is a hyper-parameter to control the weight given to the previous velocity.Weight decay is a standard L2 regularization to prevent the weights from getting too large. The updated rule is changed to?θik=vik=αvik-1-?(dEdθik+λθik) MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 11)where λ is the weight cost which controls how much penalty should be applied to weight θik.Moreover, a technique called early stopping is often exploited to prevent the model from overfitting. The root mean squared error (RMSE) between the input and its reconstruction on validation set (if available) or training set often acts as the loss function. Then the constant increase of the RMSE indicates the model is overfitting so the training should stop.2.2.2 AutoencoderAn autoencoder ADDIN EN.CITE <EndNote><Cite><Author>Le Cun</Author><Year>1987</Year><RecNum>98</RecNum><DisplayText>[13-15]</DisplayText><record><rec-number>98</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">98</key></foreign-keys><ref-type name="Thesis">32</ref-type><contributors><authors><author>Le Cun, Yann</author></authors></contributors><titles><title>Modèles connexionnistes de l'apprentissage</title></titles><dates><year>1987</year></dates><urls></urls></record></Cite><Cite><Author>Bourlard</Author><Year>1988</Year><RecNum>99</RecNum><record><rec-number>99</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">99</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Bourlard, Hervé</author><author>Kamp, Yves</author></authors></contributors><titles><title>Auto-association by multilayer perceptrons and singular value decomposition</title><secondary-title>Biological cybernetics</secondary-title></titles><periodical><full-title>Biological cybernetics</full-title></periodical><pages>291-294</pages><volume>59</volume><number>4-5</number><dates><year>1988</year></dates><isbn>0340-1200</isbn><urls></urls></record></Cite><Cite><Author>Hinton</Author><Year>1994</Year><RecNum>100</RecNum><record><rec-number>100</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">100</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Hinton, Geoffrey E</author><author>Zemel, Richard S</author></authors></contributors><titles><title>Autoencoders, minimum description length, and Helmholtz free energy</title><secondary-title>Advances in neural information processing systems</secondary-title></titles><periodical><full-title>Advances in neural information processing systems</full-title></periodical><pages>3-3</pages><dates><year>1994</year></dates><isbn>1049-5258</isbn><urls></urls></record></Cite></EndNote>[13-15] is an unsupervised learning algorithm that attempts to learn an identity function by setting the outputs to be equal to the inputs (or at least minimizing the reconstruction error), shown in REF _Ref371722095 \h Fig. 5. When the number of nodes in the hidden layer is larger than or equal to the number of nodes in the input/output layers, it is trivial to learn an identity function. By placing some restrictions on the network to make it as a regularized autoencoder, we can learn compressed representation of the input data. The easiest way to do so is limit the number of nodes in the hidden layer to force fewer nodes to represent features. Actually the discovered low-dimensional features will be very similar to PCA. In this sense, the mapping from the input layer to the hidden layer is called encoding and the mapping from the hidden layer to the output layer is called decoding.In summary, the basic autoencoder tries to find W, b argminJW,b=(hW,bx-x) MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 12)where x is the input, W is the weights, b is the biases, and h is the function mapping input to output.EnhancedMetaFilefalse\f 0 \* MERGEFORMATFig. SEQ Fig. \* ARABIC 5 An autoencoderThe argument above does not hold when the number of hidden nodes is large. But even when it is large, we can still apply a different kind of regularization called sparsity on the hidden nodes, to force them to learn compressed representations. Specifically, letpj=1mi=1maj2(x(i)) MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 13)be the average activation of hidden unit j over the training set of size m. The objective is to approximate the sparsity parameter p to p. The extra penalty term to REF NumberRef7090379000 \h \* MERGEFORMAT (12) to measure the difference between p and p could bej=1s2plogppj+1-plog1-p1-pj MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 14)where j is a hidden node, s2 is the number of nodes in the hidden layer. The value reaches its minimum of 0 at pj=p and blows up as pjapproaches 0 or 1.The overall cost function now becomesJsparseW,b=JW,b+β(j=1s2plogppj+1-plog1-p1-pj) MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 15)Now we need to do a backpropagation to update W and b. The full derivation is similar to that on an RBM.2.3 Unsupervised Learning StageA single RBM or autoencoder may not be good enough to model features in real data. Iteratively we could build another layer on top of the trained RBM or autoencoder by treating the learned feature detectors in the hidden layer as visible input layer for the new layer, as shown in REF _Ref353289805 \h \* MERGEFORMAT Fig. 2(a). The unsupervised learning stage has no labels involved and solely relies on unlabeled data itself. The learned weights and biases reflect the features of the data and then will be used as the starting point for the fine-tuning supervised learning stage.2.4 Supervised Learning StageTo train a deep learning network as a discriminative model, the supervised learning stage adds a label layer and removes the links in the top to down direction, or decoding layers called by ADDIN EN.CITE <EndNote><Cite><Author>Hinton</Author><Year>2006</Year><RecNum>31</RecNum><DisplayText>[2]</DisplayText><record><rec-number>31</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">31</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Hinton, G.E.</author><author>Salakhutdinov, R.R.</author></authors></contributors><titles><title>Reducing the dimensionality of data with neural networks</title><secondary-title>Science</secondary-title></titles><periodical><full-title>Science</full-title></periodical><pages>504-507</pages><volume>313</volume><number>5786</number><dates><year>2006</year></dates><isbn>0036-8075</isbn><urls></urls></record></Cite></EndNote>[2], as shown in REF _Ref353289805 \h \* MERGEFORMAT Fig. 2(b). Then the standard backpropogation is executed. The goal is to minimize the classification errors given the labels of all or partial samples. Since the newly added top layer has one and only one unit that can be activated at a time and the probabilities of turning each unit on must add up to 1, REF NumberRef140176415 \h \* MERGEFORMAT (6) for binary units does not apply any more but it can be generalized to K alternative states by a softmax function.pj=exji=1Kexi MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 16)where xj= Phj=1 v).The weights and biases are initialized as the values learned in the pre-training stage, except for those between the original top layer and the newly added top layer, which are randomly initialized. In the first few iterations, the training tackles the randomly initialized weights and biases between the top two layers by keeping other weights and biased fixed. The reason to do this is because the initial values learned from the pre-training are quite close to the representative features of the training data but the randomly initialized values are far from optimal. After the first few iterations, all layers are trained together treating the raw training data as the input and the labels as the output. If fewer labeled samples are used in the supervised learning stage than the unlabeled samples used in the unsupervised pre-training stage, it is a paradigm of semi-supervised learning ADDIN EN.CITE <EndNote><Cite><Author>Zhu</Author><Year>2005</Year><RecNum>44</RecNum><DisplayText>[16]</DisplayText><record><rec-number>44</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">44</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Zhu, X.</author></authors></contributors><titles><title>Semi-supervised learning literature survey</title></titles><dates><year>2005</year></dates><urls></urls></record></Cite></EndNote>[16].A validation set should be used whenever possible to avoid overfitting the model, as done in the pre-training stage. The difference is now the validation set has labeled samples rather than unlabeled ones.2.5 Deep Learning Performance Evaluation on MNIST2.5.1 Stacked RBMsTo evaluate the performance of DBNs and explore how the hyper-parameters settings impact the performance, a few experiments have been carried out on a widely used dataset MNIST.The MNIST handwritten digits dataset ADDIN EN.CITE <EndNote><Cite><Author>LeCun</Author><Year>1998</Year><RecNum>75</RecNum><DisplayText>[17]</DisplayText><record><rec-number>75</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">75</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>LeCun, Yann</author><author>Bottou, Léon</author><author>Bengio, Yoshua</author><author>Haffner, Patrick</author></authors></contributors><titles><title>Gradient-based learning applied to document recognition</title><secondary-title>Proceedings of the IEEE</secondary-title></titles><periodical><full-title>Proceedings of the IEEE</full-title></periodical><pages>2278-2324</pages><volume>86</volume><number>11</number><dates><year>1998</year></dates><isbn>0018-9219</isbn><urls></urls><research-notes>MNIST</research-notes></record></Cite></EndNote>[17] has 70,000 samples in total, conventionally divided into a training set of 50,000 samples, a validation set of 10,000 samples, and a test set of 10,000 samples. The digits have been size-normalized and centered in a 28 by 28 pixels image. The classes are 0 through 9. The MNIST dataset has been broadly used to evaluate the performance of machine learning algorithms. ADDIN EN.CITE <EndNote><Cite><Author>Wulsin</Author><Year>2011</Year><RecNum>22</RecNum><DisplayText>[18]</DisplayText><record><rec-number>22</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">22</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>D F Wulsin</author><author>J R Gupta</author><author>R Mani</author><author>J A Blanco</author><author>B Litt</author></authors></contributors><titles><title>Modeling electroencephalography waveforms with semi-supervised deep belief nets: fast classification and anomaly measurement</title><secondary-title>Journal of Neural Engineering</secondary-title></titles><periodical><full-title>Journal of Neural Engineering</full-title></periodical><pages>036015</pages><volume>8</volume><number>3</number><dates><year>2011</year></dates><isbn>1741-2552</isbn><urls><related-urls><url>;[18] provided an object-oriented matlab toolbox named DBNToolbox for working with RBMs and DBNs. The toolbox has an abstract class NNLayer and its imeplemtation class RBM to train a single RBM, a class DeepNN to train all layer together in the fine-tuning stage, and a few helper functions. The experiments in this section employed the DBNToolbox and most other experiments of my own proposed algorithms on top of the original DBNs were written based on the toolbox.The main metric to evaluate the performance in this work is the classification accuracy defined belowclassification accuracy= number of correctly classified samplestotal number of samples MACROBUTTON NumberReference \* MERGEFORMAT (17)The same network structure 784-500-500-2000-10 as in ADDIN EN.CITE <EndNote><Cite><Author>Hinton</Author><Year>2006</Year><RecNum>31</RecNum><DisplayText>[2]</DisplayText><record><rec-number>31</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">31</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Hinton, G.E.</author><author>Salakhutdinov, R.R.</author></authors></contributors><titles><title>Reducing the dimensionality of data with neural networks</title><secondary-title>Science</secondary-title></titles><periodical><full-title>Science</full-title></periodical><pages>504-507</pages><volume>313</volume><number>5786</number><dates><year>2006</year></dates><isbn>0036-8075</isbn><urls></urls></record></Cite></EndNote>[2] results in 0.9849 accuracy by the basic experiment, whose parameters are listed in REF _Ref355129403 \h Table 1.Table SEQ Table \* ARABIC 1 DBN parameters on MNISTUnsupervised pre-training stagelearning rate 0.05number of epochs50minibatch size100momentum0.5 for the first 5 epochs, 0.9 thereafter weight cost0.0002Supervised fine-tuning stagelearning rate 0.05number of epochs50minibatch size1000number of initial epochs5DBNs have the ability to learn features automatically, so it would be helpful to visualize the features learned in the example. Although ADDIN EN.CITE <EndNote><Cite><Author>Erhan</Author><Year>2009</Year><RecNum>36</RecNum><DisplayText>[19]</DisplayText><record><rec-number>36</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">36</key></foreign-keys><ref-type name="Report">27</ref-type><contributors><authors><author>Erhan, D.</author><author>Bengio, Y.</author><author>Courville, A.</author><author>Vincent, P.</author></authors></contributors><titles><title>Visualizing higher-layer features of a deep network</title></titles><dates><year>2009</year></dates><publisher>Technical report, University of Montreal</publisher><urls></urls><research-notes>good!
how to visualize MNIST learned features</research-notes></record></Cite></EndNote>[19] proposed two techniques called activation maximization and sampling from a unit to show clearer patterns in higher layers, they need to clamp input or somehow use the training set’s information. In contrast, it is more likely to show the features of the model itself by simply doing a weighted linear summation over a visible layer to obtain a hidden layer’s feature as done in ADDIN EN.CITE <EndNote><Cite><Author>Lee</Author><Year>2008</Year><RecNum>37</RecNum><DisplayText>[20]</DisplayText><record><rec-number>37</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">37</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Lee, H.</author><author>Ekanadham, C.</author><author>Ng, A.</author></authors></contributors><titles><title>Sparse deep belief net model for visual area V2</title><secondary-title>Advances in neural information processing systems</secondary-title></titles><periodical><full-title>Advances in neural information processing systems</full-title></periodical><pages>873-880</pages><volume>20</volume><dates><year>2008</year></dates><urls></urls><research-notes>how to visualize learned features</research-notes></record></Cite></EndNote>[20].The figure below shows how each pixel of the input images weighs on each unit in the first layer as an image of the same size as the input images. The weights are scaled to [-1 1]. The unsupervised learned features of the units in the first hidden layer are depicted in REF _Ref355132641 \h Fig. 6.Fig. SEQ Fig. \* ARABIC 6 Learned features of the first hidden layerThe first layer’s weights multiplied by the second layer’s produce the features learned in the second layer in the input space, as shown in REF _Ref355133207 \h Fig. 7. The weights are scaled to [-1 1], too.The third layer’s features are shown in REF _Ref373367693 \h Fig. 8 by applying the same trick.Fig. SEQ Fig. \* ARABIC 7 Learned features of the second hidden layerFig. SEQ Fig. \* ARABIC 8 Learned features of the third hidden layerIt is quite hard to tell if each layer is more abstract than the layer below. Even the first layer does not have clearly human readable patterns. The reason is because the regular RBM is a distributed model, so the features learned are not local or sparse. ADDIN EN.CITE <EndNote><Cite><Author>Lee</Author><Year>2009</Year><RecNum>39</RecNum><DisplayText>[21]</DisplayText><record><rec-number>39</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">39</key></foreign-keys><ref-type name="Conference Proceedings">10</ref-type><contributors><authors><author>Lee, H.</author><author>Grosse, R.</author><author>Ranganath, R.</author><author>Ng, A.Y.</author></authors></contributors><titles><title>Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations</title><secondary-title>Proceedings of the 26th Annual International Conference on Machine Learning</secondary-title></titles><pages>609-616</pages><dates><year>2009</year></dates><publisher>ACM</publisher><isbn>1605585165</isbn><urls></urls></record></Cite></EndNote>[21] states that the sparsity regularization is important for the first layer to learn oriented edge filters. A sparse RBM (sRBM) proposed by ADDIN EN.CITE <EndNote><Cite><Author>Lee</Author><Year>2008</Year><RecNum>37</RecNum><DisplayText>[20]</DisplayText><record><rec-number>37</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">37</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Lee, H.</author><author>Ekanadham, C.</author><author>Ng, A.</author></authors></contributors><titles><title>Sparse deep belief net model for visual area V2</title><secondary-title>Advances in neural information processing systems</secondary-title></titles><periodical><full-title>Advances in neural information processing systems</full-title></periodical><pages>873-880</pages><volume>20</volume><dates><year>2008</year></dates><urls></urls><research-notes>how to visualize learned features</research-notes></record></Cite></EndNote>[20] is an algorithm to make the features learned by RBMs sparse. This is done by adding a regularization term that penalizes a deviation of the expected activation of the hidden units from a fixed level. The update rule for hidden layer’s biases becomesθik=θik-1+αvik-1-?dEdθik+λθik+2p*(p-<Phi=1 v)>) MACROBUTTON NumberReference \* MERGEFORMAT ( SEQ EquationNumber \n \* Arabic \* MERGEFORMAT 18)where p is a constant controlling the sparseness of the hidden units. The last term is introduced by sRBM. Since the original DBNToolbox does not have a sparse implementation of RBMs, I added a protected abstract method UpdateSparcity(obj, visSamples) to NNLayer.m. RBM.m implements the function, which is called at the end of each epoch by NNLayer.m’s Train method. The value of p can be set in RBM.m’s UpdateSparcity function.When p=0.1, the features learned in 3 layers of sRBMs are shown as follows. The classification accuracy is 0.9731, slightly worse than the DBN composed of regular RBMs.Fig. SEQ Fig. \* ARABIC 9 Learned features of the first hidden layer of sRBMFig. SEQ Fig. \* ARABIC 10 Learned features of the second hidden layer of sRBMFig. SEQ Fig. \* ARABIC 11 Learned features of the third hidden layer of sRBMThe features learned by sRBM are sparser and they are more human-readable. The first layer seems like strokes; the second layer looks like digits parts; digits and digit-like shapes are shown in the third layer. To demonstrate how DBNs work as a semi-supervised learning algorithm, experiments with different numbers of unlabeled samples for pre-training and different numbers of labeled samples for fine-tuning were performed.Because a small number of labeled samples are used in the experiment, DBNToolbox has a minor bug in this scenario. DeepNN’s default value for the parameter miniBatchSize is 1000. When calculating the number of mini-batches, the program floors it to 0 if the sampling size is less than 1000. To overcome the bug, DeepNN’s miniBatchSize is set to a number less than or equal to the size of training samples.Another caveat is when the size of samples is small the distribution of digits could be highly unbalanced. A function was implemented to randomly sample indices for training and validation sets and guarantee each digit has the same size in both sets. DBNToolbox also requires in both sets there is at least one sample from each class.A 784-150-150-150-10 DBN with 0 to 6400 unlabeled samples and 50 to 250 labeled samples produces the result below. Each point is the average of 10 trials.Fig. SEQ Fig. \* ARABIC 12 The predication accuracy of a 784-150-150-150-10 DBN using different numbers of unlabeled and labeled samples. The legends show the numbers of labeled samples.The unlabeled samples help, especially when the labeled samples are scarce.2.5.2 Stacked AutoencodersThe similar experiments have been conducted on the same MNIST dataset using stacked autoencoders. The Matlab code from ADDIN EN.CITE <EndNote><Cite><Author>Ng</Author><Year>2013</Year><RecNum>96</RecNum><DisplayText>[22]</DisplayText><record><rec-number>96</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">96</key></foreign-keys><ref-type name="Web Page">12</ref-type><contributors><authors><author>Andrew Ng</author></authors></contributors><titles><title>UFLDL Tutorial</title></titles><dates><year>2013</year></dates><urls><related-urls><url>;[22] is used as a starting point for all the staked autoencoders implementation. A 784-200-200-10 network with parameters listed in REF _Ref371623354 \h Table 2 yields 0.9781 classification accuracy.Table SEQ Table \* ARABIC 2 Stacked autoencoders parameters on MNISTUnsupervised pre-training stagesparsity object0.1sparsity penalty term3weight decay0.003maximum iteration200Supervised fine-tuning stageweight decay0.003maximum iteration200The learned features in the first and second hidden layers are depicted in REF _Ref371779107 \h Fig. 13 and REF _Ref371779111 \h Fig. 14. The first layer seems to detect edges and the second layer seems to detect contours, which are similar to the features learned by the first and third layers of stacked sRBMs as shown in REF _Ref371632800 \h Fig. 9 and REF _Ref371632804 \h Fig. 11.Fig. SEQ Fig. \* ARABIC 13 First hidden layer of stacked autoencoders detects edgesFig. SEQ Fig. \* ARABIC 14 Second layer of stacked autoencoders detects contoursIn conclusion, stacked RBMs and stacked autoencoders tend to learn similar features and achieve similar classification accuracy on the MNIST dataset.2.6 State of the Art on Deep LearningAs the first breakthrough, a deep belief network (DBN) by stacking pre-trained RBMs without the decoding parts was proposed by Hinton PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5IaW50b248L0F1dGhvcj48WWVhcj4yMDA2PC9ZZWFyPjxS
ZWNOdW0+MzE8L1JlY051bT48RGlzcGxheVRleHQ+WzIsIDEwLCAyM108L0Rpc3BsYXlUZXh0Pjxy
ZWNvcmQ+PHJlYy1udW1iZXI+MzE8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0i
RU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjMxPC9rZXk+
PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10
eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5IaW50b24sIEcuRS48L2F1dGhvcj48
YXV0aG9yPlNhbGFraHV0ZGlub3YsIFIuUi48L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRv
cnM+PHRpdGxlcz48dGl0bGU+UmVkdWNpbmcgdGhlIGRpbWVuc2lvbmFsaXR5IG9mIGRhdGEgd2l0
aCBuZXVyYWwgbmV0d29ya3M8L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+U2NpZW5jZTwvc2Vjb25k
YXJ5LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPlNjaWVuY2U8L2Z1bGwt
dGl0bGU+PC9wZXJpb2RpY2FsPjxwYWdlcz41MDQtNTA3PC9wYWdlcz48dm9sdW1lPjMxMzwvdm9s
dW1lPjxudW1iZXI+NTc4NjwvbnVtYmVyPjxkYXRlcz48eWVhcj4yMDA2PC95ZWFyPjwvZGF0ZXM+
PGlzYm4+MDAzNi04MDc1PC9pc2JuPjx1cmxzPjwvdXJscz48L3JlY29yZD48L0NpdGU+PENpdGU+
PEF1dGhvcj5IaW50b248L0F1dGhvcj48WWVhcj4yMDA2PC9ZZWFyPjxSZWNOdW0+Mjg8L1JlY051
bT48cmVjb3JkPjxyZWMtbnVtYmVyPjI4PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+PGtleSBh
cHA9IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij4yODwv
a2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9y
ZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+SGludG9uLCBHLkUuPC9hdXRo
b3I+PGF1dGhvcj5Pc2luZGVybywgUy48L2F1dGhvcj48YXV0aG9yPlRlaCwgWS5XLjwvYXV0aG9y
PjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0bGVzPjx0aXRsZT5BIGZhc3QgbGVhcm5pbmcg
YWxnb3JpdGhtIGZvciBkZWVwIGJlbGllZiBuZXRzPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPk5l
dXJhbCBjb21wdXRhdGlvbjwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxm
dWxsLXRpdGxlPk5ldXJhbCBjb21wdXRhdGlvbjwvZnVsbC10aXRsZT48L3BlcmlvZGljYWw+PHBh
Z2VzPjE1MjctMTU1NDwvcGFnZXM+PHZvbHVtZT4xODwvdm9sdW1lPjxudW1iZXI+NzwvbnVtYmVy
PjxkYXRlcz48eWVhcj4yMDA2PC95ZWFyPjwvZGF0ZXM+PGlzYm4+MDg5OS03NjY3PC9pc2JuPjx1
cmxzPjwvdXJscz48L3JlY29yZD48L0NpdGU+PENpdGU+PEF1dGhvcj5IaW50b248L0F1dGhvcj48
WWVhcj4yMDA3PC9ZZWFyPjxSZWNOdW0+Mjc8L1JlY051bT48cmVjb3JkPjxyZWMtbnVtYmVyPjI3
PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+PGtleSBhcHA9IkVOIiBkYi1pZD0icGFzdGQ1Mnpy
YWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij4yNzwva2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYt
dHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9yZWYtdHlwZT48Y29udHJpYnV0b3JzPjxh
dXRob3JzPjxhdXRob3I+SGludG9uLCBHLkUuPC9hdXRob3I+PC9hdXRob3JzPjwvY29udHJpYnV0
b3JzPjx0aXRsZXM+PHRpdGxlPlRvIHJlY29nbml6ZSBzaGFwZXMsIGZpcnN0IGxlYXJuIHRvIGdl
bmVyYXRlIGltYWdlczwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5Qcm9ncmVzcyBpbiBicmFpbiBy
ZXNlYXJjaDwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxl
PlByb2dyZXNzIGluIGJyYWluIHJlc2VhcmNoPC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48cGFn
ZXM+NTM1LTU0NzwvcGFnZXM+PHZvbHVtZT4xNjU8L3ZvbHVtZT48ZGF0ZXM+PHllYXI+MjAwNzwv
eWVhcj48L2RhdGVzPjxpc2JuPjAwNzktNjEyMzwvaXNibj48dXJscz48L3VybHM+PC9yZWNvcmQ+
PC9DaXRlPjwvRW5kTm90ZT5=
ADDIN EN.CITE PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5IaW50b248L0F1dGhvcj48WWVhcj4yMDA2PC9ZZWFyPjxS
ZWNOdW0+MzE8L1JlY051bT48RGlzcGxheVRleHQ+WzIsIDEwLCAyM108L0Rpc3BsYXlUZXh0Pjxy
ZWNvcmQ+PHJlYy1udW1iZXI+MzE8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0i
RU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjMxPC9rZXk+
PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10
eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5IaW50b24sIEcuRS48L2F1dGhvcj48
YXV0aG9yPlNhbGFraHV0ZGlub3YsIFIuUi48L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRv
cnM+PHRpdGxlcz48dGl0bGU+UmVkdWNpbmcgdGhlIGRpbWVuc2lvbmFsaXR5IG9mIGRhdGEgd2l0
aCBuZXVyYWwgbmV0d29ya3M8L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+U2NpZW5jZTwvc2Vjb25k
YXJ5LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPlNjaWVuY2U8L2Z1bGwt
dGl0bGU+PC9wZXJpb2RpY2FsPjxwYWdlcz41MDQtNTA3PC9wYWdlcz48dm9sdW1lPjMxMzwvdm9s
dW1lPjxudW1iZXI+NTc4NjwvbnVtYmVyPjxkYXRlcz48eWVhcj4yMDA2PC95ZWFyPjwvZGF0ZXM+
PGlzYm4+MDAzNi04MDc1PC9pc2JuPjx1cmxzPjwvdXJscz48L3JlY29yZD48L0NpdGU+PENpdGU+
PEF1dGhvcj5IaW50b248L0F1dGhvcj48WWVhcj4yMDA2PC9ZZWFyPjxSZWNOdW0+Mjg8L1JlY051
bT48cmVjb3JkPjxyZWMtbnVtYmVyPjI4PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+PGtleSBh
cHA9IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij4yODwv
a2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9y
ZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+SGludG9uLCBHLkUuPC9hdXRo
b3I+PGF1dGhvcj5Pc2luZGVybywgUy48L2F1dGhvcj48YXV0aG9yPlRlaCwgWS5XLjwvYXV0aG9y
PjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0bGVzPjx0aXRsZT5BIGZhc3QgbGVhcm5pbmcg
YWxnb3JpdGhtIGZvciBkZWVwIGJlbGllZiBuZXRzPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPk5l
dXJhbCBjb21wdXRhdGlvbjwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxm
dWxsLXRpdGxlPk5ldXJhbCBjb21wdXRhdGlvbjwvZnVsbC10aXRsZT48L3BlcmlvZGljYWw+PHBh
Z2VzPjE1MjctMTU1NDwvcGFnZXM+PHZvbHVtZT4xODwvdm9sdW1lPjxudW1iZXI+NzwvbnVtYmVy
PjxkYXRlcz48eWVhcj4yMDA2PC95ZWFyPjwvZGF0ZXM+PGlzYm4+MDg5OS03NjY3PC9pc2JuPjx1
cmxzPjwvdXJscz48L3JlY29yZD48L0NpdGU+PENpdGU+PEF1dGhvcj5IaW50b248L0F1dGhvcj48
WWVhcj4yMDA3PC9ZZWFyPjxSZWNOdW0+Mjc8L1JlY051bT48cmVjb3JkPjxyZWMtbnVtYmVyPjI3
PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+PGtleSBhcHA9IkVOIiBkYi1pZD0icGFzdGQ1Mnpy
YWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij4yNzwva2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYt
dHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9yZWYtdHlwZT48Y29udHJpYnV0b3JzPjxh
dXRob3JzPjxhdXRob3I+SGludG9uLCBHLkUuPC9hdXRob3I+PC9hdXRob3JzPjwvY29udHJpYnV0
b3JzPjx0aXRsZXM+PHRpdGxlPlRvIHJlY29nbml6ZSBzaGFwZXMsIGZpcnN0IGxlYXJuIHRvIGdl
bmVyYXRlIGltYWdlczwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5Qcm9ncmVzcyBpbiBicmFpbiBy
ZXNlYXJjaDwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxl
PlByb2dyZXNzIGluIGJyYWluIHJlc2VhcmNoPC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48cGFn
ZXM+NTM1LTU0NzwvcGFnZXM+PHZvbHVtZT4xNjU8L3ZvbHVtZT48ZGF0ZXM+PHllYXI+MjAwNzwv
eWVhcj48L2RhdGVzPjxpc2JuPjAwNzktNjEyMzwvaXNibj48dXJscz48L3VybHM+PC9yZWNvcmQ+
PC9DaXRlPjwvRW5kTm90ZT5=
ADDIN EN.CITE.DATA [2, 10, 23]. The pre-training idea was then adopted by many researchers to come up with new algorithms. ADDIN EN.CITE <EndNote><Cite><Author>Lee</Author><Year>2008</Year><RecNum>37</RecNum><DisplayText>[20]</DisplayText><record><rec-number>37</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">37</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Lee, H.</author><author>Ekanadham, C.</author><author>Ng, A.</author></authors></contributors><titles><title>Sparse deep belief net model for visual area V2</title><secondary-title>Advances in neural information processing systems</secondary-title></titles><periodical><full-title>Advances in neural information processing systems</full-title></periodical><pages>873-880</pages><volume>20</volume><dates><year>2008</year></dates><urls></urls><research-notes>how to visualize learned features</research-notes></record></Cite></EndNote>[20] used sparse RBMs as learning units. PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5CZW5naW88L0F1dGhvcj48WWVhcj4yMDA3PC9ZZWFyPjxS
ZWNOdW0+NDU8L1JlY051bT48RGlzcGxheVRleHQ+WzEsIDI0LCAyNV08L0Rpc3BsYXlUZXh0Pjxy
ZWNvcmQ+PHJlYy1udW1iZXI+NDU8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0i
RU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjQ1PC9rZXk+
PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10
eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5CZW5naW8sIFkuPC9hdXRob3I+PGF1
dGhvcj5MYW1ibGluLCBQLjwvYXV0aG9yPjxhdXRob3I+UG9wb3ZpY2ksIEQuPC9hdXRob3I+PGF1
dGhvcj5MYXJvY2hlbGxlLCBILjwvYXV0aG9yPjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0
bGVzPjx0aXRsZT5HcmVlZHkgbGF5ZXItd2lzZSB0cmFpbmluZyBvZiBkZWVwIG5ldHdvcmtzPC90
aXRsZT48c2Vjb25kYXJ5LXRpdGxlPkFkdmFuY2VzIGluIG5ldXJhbCBpbmZvcm1hdGlvbiBwcm9j
ZXNzaW5nIHN5c3RlbXM8L3NlY29uZGFyeS10aXRsZT48L3RpdGxlcz48cGVyaW9kaWNhbD48ZnVs
bC10aXRsZT5BZHZhbmNlcyBpbiBuZXVyYWwgaW5mb3JtYXRpb24gcHJvY2Vzc2luZyBzeXN0ZW1z
PC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+MTUzPC9wYWdlcz48dm9sdW1lPjE5PC92
b2x1bWU+PGRhdGVzPjx5ZWFyPjIwMDc8L3llYXI+PC9kYXRlcz48aXNibj4xMDQ5LTUyNTg8L2lz
Ym4+PHVybHM+PC91cmxzPjxyZXNlYXJjaC1ub3Rlcz5nb29kISYjeEQ7dG8gaGFuZGxlIHJlYWwt
dmFsdWVkIGRhdGEmI3hEO1VDSSBBYmFsb25lIGRhdGEgc2V0JiN4RDtjb21wYXJlZCB3LyBzaGFs
bG93IG5ldHdvcmtzJiN4RDtzdGFja2VkIGF1dG9lbmNvZGVyczwvcmVzZWFyY2gtbm90ZXM+PC9y
ZWNvcmQ+PC9DaXRlPjxDaXRlPjxBdXRob3I+VmluY2VudDwvQXV0aG9yPjxZZWFyPjIwMDg8L1ll
YXI+PFJlY051bT4xMDU8L1JlY051bT48cmVjb3JkPjxyZWMtbnVtYmVyPjEwNTwvcmVjLW51bWJl
cj48Zm9yZWlnbi1rZXlzPjxrZXkgYXBwPSJFTiIgZGItaWQ9InBhc3RkNTJ6cmFmZGF1ZWVyZXF2
ZHBzOWVhZWZ4enRyZGZzeiI+MTA1PC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9
IkNvbmZlcmVuY2UgUHJvY2VlZGluZ3MiPjEwPC9yZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRo
b3JzPjxhdXRob3I+VmluY2VudCwgUGFzY2FsPC9hdXRob3I+PGF1dGhvcj5MYXJvY2hlbGxlLCBI
dWdvPC9hdXRob3I+PGF1dGhvcj5CZW5naW8sIFlvc2h1YTwvYXV0aG9yPjxhdXRob3I+TWFuemFn
b2wsIFBpZXJyZS1BbnRvaW5lPC9hdXRob3I+PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRs
ZXM+PHRpdGxlPkV4dHJhY3RpbmcgYW5kIGNvbXBvc2luZyByb2J1c3QgZmVhdHVyZXMgd2l0aCBk
ZW5vaXNpbmcgYXV0b2VuY29kZXJzPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPlByb2NlZWRpbmdz
IG9mIHRoZSAyNXRoIGludGVybmF0aW9uYWwgY29uZmVyZW5jZSBvbiBNYWNoaW5lIGxlYXJuaW5n
PC9zZWNvbmRhcnktdGl0bGU+PC90aXRsZXM+PHBhZ2VzPjEwOTYtMTEwMzwvcGFnZXM+PGRhdGVz
Pjx5ZWFyPjIwMDg8L3llYXI+PC9kYXRlcz48cHVibGlzaGVyPkFDTTwvcHVibGlzaGVyPjxpc2Ju
PjE2MDU1ODIwNTA8L2lzYm4+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0
aG9yPlZpbmNlbnQ8L0F1dGhvcj48WWVhcj4yMDEwPC9ZZWFyPjxSZWNOdW0+MTA2PC9SZWNOdW0+
PHJlY29yZD48cmVjLW51bWJlcj4xMDY8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFw
cD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjEwNjwv
a2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9y
ZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+VmluY2VudCwgUGFzY2FsPC9h
dXRob3I+PGF1dGhvcj5MYXJvY2hlbGxlLCBIdWdvPC9hdXRob3I+PGF1dGhvcj5MYWpvaWUsIElz
YWJlbGxlPC9hdXRob3I+PGF1dGhvcj5CZW5naW8sIFlvc2h1YTwvYXV0aG9yPjxhdXRob3I+TWFu
emFnb2wsIFBpZXJyZS1BbnRvaW5lPC9hdXRob3I+PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0
aXRsZXM+PHRpdGxlPlN0YWNrZWQgZGVub2lzaW5nIGF1dG9lbmNvZGVyczogTGVhcm5pbmcgdXNl
ZnVsIHJlcHJlc2VudGF0aW9ucyBpbiBhIGRlZXAgbmV0d29yayB3aXRoIGEgbG9jYWwgZGVub2lz
aW5nIGNyaXRlcmlvbjwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5UaGUgSm91cm5hbCBvZiBNYWNo
aW5lIExlYXJuaW5nIFJlc2VhcmNoPC9zZWNvbmRhcnktdGl0bGU+PC90aXRsZXM+PHBlcmlvZGlj
YWw+PGZ1bGwtdGl0bGU+VGhlIEpvdXJuYWwgb2YgTWFjaGluZSBMZWFybmluZyBSZXNlYXJjaDwv
ZnVsbC10aXRsZT48L3BlcmlvZGljYWw+PHBhZ2VzPjMzNzEtMzQwODwvcGFnZXM+PHZvbHVtZT45
OTk5PC92b2x1bWU+PGRhdGVzPjx5ZWFyPjIwMTA8L3llYXI+PC9kYXRlcz48aXNibj4xNTMyLTQ0
MzU8L2lzYm4+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48L0VuZE5vdGU+
ADDIN EN.CITE PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5CZW5naW88L0F1dGhvcj48WWVhcj4yMDA3PC9ZZWFyPjxS
ZWNOdW0+NDU8L1JlY051bT48RGlzcGxheVRleHQ+WzEsIDI0LCAyNV08L0Rpc3BsYXlUZXh0Pjxy
ZWNvcmQ+PHJlYy1udW1iZXI+NDU8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0i
RU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjQ1PC9rZXk+
PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10
eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5CZW5naW8sIFkuPC9hdXRob3I+PGF1
dGhvcj5MYW1ibGluLCBQLjwvYXV0aG9yPjxhdXRob3I+UG9wb3ZpY2ksIEQuPC9hdXRob3I+PGF1
dGhvcj5MYXJvY2hlbGxlLCBILjwvYXV0aG9yPjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0
bGVzPjx0aXRsZT5HcmVlZHkgbGF5ZXItd2lzZSB0cmFpbmluZyBvZiBkZWVwIG5ldHdvcmtzPC90
aXRsZT48c2Vjb25kYXJ5LXRpdGxlPkFkdmFuY2VzIGluIG5ldXJhbCBpbmZvcm1hdGlvbiBwcm9j
ZXNzaW5nIHN5c3RlbXM8L3NlY29uZGFyeS10aXRsZT48L3RpdGxlcz48cGVyaW9kaWNhbD48ZnVs
bC10aXRsZT5BZHZhbmNlcyBpbiBuZXVyYWwgaW5mb3JtYXRpb24gcHJvY2Vzc2luZyBzeXN0ZW1z
PC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+MTUzPC9wYWdlcz48dm9sdW1lPjE5PC92
b2x1bWU+PGRhdGVzPjx5ZWFyPjIwMDc8L3llYXI+PC9kYXRlcz48aXNibj4xMDQ5LTUyNTg8L2lz
Ym4+PHVybHM+PC91cmxzPjxyZXNlYXJjaC1ub3Rlcz5nb29kISYjeEQ7dG8gaGFuZGxlIHJlYWwt
dmFsdWVkIGRhdGEmI3hEO1VDSSBBYmFsb25lIGRhdGEgc2V0JiN4RDtjb21wYXJlZCB3LyBzaGFs
bG93IG5ldHdvcmtzJiN4RDtzdGFja2VkIGF1dG9lbmNvZGVyczwvcmVzZWFyY2gtbm90ZXM+PC9y
ZWNvcmQ+PC9DaXRlPjxDaXRlPjxBdXRob3I+VmluY2VudDwvQXV0aG9yPjxZZWFyPjIwMDg8L1ll
YXI+PFJlY051bT4xMDU8L1JlY051bT48cmVjb3JkPjxyZWMtbnVtYmVyPjEwNTwvcmVjLW51bWJl
cj48Zm9yZWlnbi1rZXlzPjxrZXkgYXBwPSJFTiIgZGItaWQ9InBhc3RkNTJ6cmFmZGF1ZWVyZXF2
ZHBzOWVhZWZ4enRyZGZzeiI+MTA1PC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9
IkNvbmZlcmVuY2UgUHJvY2VlZGluZ3MiPjEwPC9yZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRo
b3JzPjxhdXRob3I+VmluY2VudCwgUGFzY2FsPC9hdXRob3I+PGF1dGhvcj5MYXJvY2hlbGxlLCBI
dWdvPC9hdXRob3I+PGF1dGhvcj5CZW5naW8sIFlvc2h1YTwvYXV0aG9yPjxhdXRob3I+TWFuemFn
b2wsIFBpZXJyZS1BbnRvaW5lPC9hdXRob3I+PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRs
ZXM+PHRpdGxlPkV4dHJhY3RpbmcgYW5kIGNvbXBvc2luZyByb2J1c3QgZmVhdHVyZXMgd2l0aCBk
ZW5vaXNpbmcgYXV0b2VuY29kZXJzPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPlByb2NlZWRpbmdz
IG9mIHRoZSAyNXRoIGludGVybmF0aW9uYWwgY29uZmVyZW5jZSBvbiBNYWNoaW5lIGxlYXJuaW5n
PC9zZWNvbmRhcnktdGl0bGU+PC90aXRsZXM+PHBhZ2VzPjEwOTYtMTEwMzwvcGFnZXM+PGRhdGVz
Pjx5ZWFyPjIwMDg8L3llYXI+PC9kYXRlcz48cHVibGlzaGVyPkFDTTwvcHVibGlzaGVyPjxpc2Ju
PjE2MDU1ODIwNTA8L2lzYm4+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0
aG9yPlZpbmNlbnQ8L0F1dGhvcj48WWVhcj4yMDEwPC9ZZWFyPjxSZWNOdW0+MTA2PC9SZWNOdW0+
PHJlY29yZD48cmVjLW51bWJlcj4xMDY8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFw
cD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjEwNjwv
a2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9y
ZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+VmluY2VudCwgUGFzY2FsPC9h
dXRob3I+PGF1dGhvcj5MYXJvY2hlbGxlLCBIdWdvPC9hdXRob3I+PGF1dGhvcj5MYWpvaWUsIElz
YWJlbGxlPC9hdXRob3I+PGF1dGhvcj5CZW5naW8sIFlvc2h1YTwvYXV0aG9yPjxhdXRob3I+TWFu
emFnb2wsIFBpZXJyZS1BbnRvaW5lPC9hdXRob3I+PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0
aXRsZXM+PHRpdGxlPlN0YWNrZWQgZGVub2lzaW5nIGF1dG9lbmNvZGVyczogTGVhcm5pbmcgdXNl
ZnVsIHJlcHJlc2VudGF0aW9ucyBpbiBhIGRlZXAgbmV0d29yayB3aXRoIGEgbG9jYWwgZGVub2lz
aW5nIGNyaXRlcmlvbjwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5UaGUgSm91cm5hbCBvZiBNYWNo
aW5lIExlYXJuaW5nIFJlc2VhcmNoPC9zZWNvbmRhcnktdGl0bGU+PC90aXRsZXM+PHBlcmlvZGlj
YWw+PGZ1bGwtdGl0bGU+VGhlIEpvdXJuYWwgb2YgTWFjaGluZSBMZWFybmluZyBSZXNlYXJjaDwv
ZnVsbC10aXRsZT48L3BlcmlvZGljYWw+PHBhZ2VzPjMzNzEtMzQwODwvcGFnZXM+PHZvbHVtZT45
OTk5PC92b2x1bWU+PGRhdGVzPjx5ZWFyPjIwMTA8L3llYXI+PC9kYXRlcz48aXNibj4xNTMyLTQ0
MzU8L2lzYm4+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48L0VuZE5vdGU+
ADDIN EN.CITE.DATA [1, 24, 25] used an autoencoder and its variants such as denoising or sparse version as the building blocks instead of RBM families. Deep learning shows it power mainly in two areas: speech recognition / language processing PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5TZWlkZTwvQXV0aG9yPjxZZWFyPjIwMTE8L1llYXI+PFJl
Y051bT4xMDI8L1JlY051bT48RGlzcGxheVRleHQ+WzI2LTI5XTwvRGlzcGxheVRleHQ+PHJlY29y
ZD48cmVjLW51bWJlcj4xMDI8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0iRU4i
IGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjEwMjwva2V5Pjwv
Zm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJDb25mZXJlbmNlIFByb2NlZWRpbmdzIj4xMDwv
cmVmLXR5cGU+PGNvbnRyaWJ1dG9ycz48YXV0aG9ycz48YXV0aG9yPlNlaWRlLCBGcmFuazwvYXV0
aG9yPjxhdXRob3I+TGksIEdhbmc8L2F1dGhvcj48YXV0aG9yPll1LCBEb25nPC9hdXRob3I+PC9h
dXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRsZXM+PHRpdGxlPkNvbnZlcnNhdGlvbmFsIFNwZWVj
aCBUcmFuc2NyaXB0aW9uIFVzaW5nIENvbnRleHQtRGVwZW5kZW50IERlZXAgTmV1cmFsIE5ldHdv
cmtzPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPklOVEVSU1BFRUNIPC9zZWNvbmRhcnktdGl0bGU+
PC90aXRsZXM+PHBhZ2VzPjQzNy00NDA8L3BhZ2VzPjxkYXRlcz48eWVhcj4yMDExPC95ZWFyPjwv
ZGF0ZXM+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPkhpbnRvbjwv
QXV0aG9yPjxZZWFyPjIwMTI8L1llYXI+PFJlY051bT4xMDM8L1JlY051bT48cmVjb3JkPjxyZWMt
bnVtYmVyPjEwMzwvcmVjLW51bWJlcj48Zm9yZWlnbi1rZXlzPjxrZXkgYXBwPSJFTiIgZGItaWQ9
InBhc3RkNTJ6cmFmZGF1ZWVyZXF2ZHBzOWVhZWZ4enRyZGZzeiI+MTAzPC9rZXk+PC9mb3JlaWdu
LWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10eXBlPjxjb250
cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5IaW50b24sIEdlb2ZmcmV5PC9hdXRob3I+PGF1dGhv
cj5EZW5nLCBMaTwvYXV0aG9yPjxhdXRob3I+WXUsIERvbmc8L2F1dGhvcj48YXV0aG9yPkRhaGws
IEdlb3JnZSBFPC9hdXRob3I+PGF1dGhvcj5Nb2hhbWVkLCBBYmRlbC1yYWhtYW48L2F1dGhvcj48
YXV0aG9yPkphaXRseSwgTmF2ZGVlcDwvYXV0aG9yPjxhdXRob3I+U2VuaW9yLCBBbmRyZXc8L2F1
dGhvcj48YXV0aG9yPlZhbmhvdWNrZSwgVmluY2VudDwvYXV0aG9yPjxhdXRob3I+Tmd1eWVuLCBQ
YXRyaWNrPC9hdXRob3I+PGF1dGhvcj5TYWluYXRoLCBUYXJhIE48L2F1dGhvcj48L2F1dGhvcnM+
PC9jb250cmlidXRvcnM+PHRpdGxlcz48dGl0bGU+RGVlcCBuZXVyYWwgbmV0d29ya3MgZm9yIGFj
b3VzdGljIG1vZGVsaW5nIGluIHNwZWVjaCByZWNvZ25pdGlvbjogVGhlIHNoYXJlZCB2aWV3cyBv
ZiBmb3VyIHJlc2VhcmNoIGdyb3VwczwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5TaWduYWwgUHJv
Y2Vzc2luZyBNYWdhemluZSwgSUVFRTwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2Rp
Y2FsPjxmdWxsLXRpdGxlPlNpZ25hbCBQcm9jZXNzaW5nIE1hZ2F6aW5lLCBJRUVFPC9mdWxsLXRp
dGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+ODItOTc8L3BhZ2VzPjx2b2x1bWU+Mjk8L3ZvbHVtZT48
bnVtYmVyPjY8L251bWJlcj48ZGF0ZXM+PHllYXI+MjAxMjwveWVhcj48L2RhdGVzPjxpc2JuPjEw
NTMtNTg4ODwvaXNibj48dXJscz48L3VybHM+PC9yZWNvcmQ+PC9DaXRlPjxDaXRlPjxBdXRob3I+
QmVuZ2lvPC9BdXRob3I+PFllYXI+MjAwODwvWWVhcj48UmVjTnVtPjEwNzwvUmVjTnVtPjxyZWNv
cmQ+PHJlYy1udW1iZXI+MTA3PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+PGtleSBhcHA9IkVO
IiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij4xMDc8L2tleT48
L2ZvcmVpZ24ta2V5cz48cmVmLXR5cGUgbmFtZT0iSm91cm5hbCBBcnRpY2xlIj4xNzwvcmVmLXR5
cGU+PGNvbnRyaWJ1dG9ycz48YXV0aG9ycz48YXV0aG9yPkJlbmdpbywgWW9zaHVhPC9hdXRob3I+
PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRsZXM+PHRpdGxlPk5ldXJhbCBuZXQgbGFuZ3Vh
Z2UgbW9kZWxzPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPlNjaG9sYXJwZWRpYTwvc2Vjb25kYXJ5
LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPlNjaG9sYXJwZWRpYTwvZnVs
bC10aXRsZT48L3BlcmlvZGljYWw+PHBhZ2VzPjM4ODE8L3BhZ2VzPjx2b2x1bWU+Mzwvdm9sdW1l
PjxudW1iZXI+MTwvbnVtYmVyPjxkYXRlcz48eWVhcj4yMDA4PC95ZWFyPjwvZGF0ZXM+PGlzYm4+
MTk0MS02MDE2PC9pc2JuPjx1cmxzPjwvdXJscz48L3JlY29yZD48L0NpdGU+PENpdGU+PEF1dGhv
cj5Db2xsb2JlcnQ8L0F1dGhvcj48WWVhcj4yMDExPC9ZZWFyPjxSZWNOdW0+MTA4PC9SZWNOdW0+
PHJlY29yZD48cmVjLW51bWJlcj4xMDg8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFw
cD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjEwODwv
a2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9y
ZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+Q29sbG9iZXJ0LCBSb25hbjwv
YXV0aG9yPjxhdXRob3I+V2VzdG9uLCBKYXNvbjwvYXV0aG9yPjxhdXRob3I+Qm90dG91LCBMw6lv
bjwvYXV0aG9yPjxhdXRob3I+S2FybGVuLCBNaWNoYWVsPC9hdXRob3I+PGF1dGhvcj5LYXZ1a2N1
b2dsdSwgS29yYXk8L2F1dGhvcj48YXV0aG9yPkt1a3NhLCBQYXZlbDwvYXV0aG9yPjwvYXV0aG9y
cz48L2NvbnRyaWJ1dG9ycz48dGl0bGVzPjx0aXRsZT5OYXR1cmFsIGxhbmd1YWdlIHByb2Nlc3Np
bmcgKGFsbW9zdCkgZnJvbSBzY3JhdGNoPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPlRoZSBKb3Vy
bmFsIG9mIE1hY2hpbmUgTGVhcm5pbmcgUmVzZWFyY2g8L3NlY29uZGFyeS10aXRsZT48L3RpdGxl
cz48cGVyaW9kaWNhbD48ZnVsbC10aXRsZT5UaGUgSm91cm5hbCBvZiBNYWNoaW5lIExlYXJuaW5n
IFJlc2VhcmNoPC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+MjQ5My0yNTM3PC9wYWdl
cz48dm9sdW1lPjEyPC92b2x1bWU+PGRhdGVzPjx5ZWFyPjIwMTE8L3llYXI+PC9kYXRlcz48aXNi
bj4xNTMyLTQ0MzU8L2lzYm4+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48L0VuZE5vdGU+
ADDIN EN.CITE PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5TZWlkZTwvQXV0aG9yPjxZZWFyPjIwMTE8L1llYXI+PFJl
Y051bT4xMDI8L1JlY051bT48RGlzcGxheVRleHQ+WzI2LTI5XTwvRGlzcGxheVRleHQ+PHJlY29y
ZD48cmVjLW51bWJlcj4xMDI8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0iRU4i
IGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjEwMjwva2V5Pjwv
Zm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJDb25mZXJlbmNlIFByb2NlZWRpbmdzIj4xMDwv
cmVmLXR5cGU+PGNvbnRyaWJ1dG9ycz48YXV0aG9ycz48YXV0aG9yPlNlaWRlLCBGcmFuazwvYXV0
aG9yPjxhdXRob3I+TGksIEdhbmc8L2F1dGhvcj48YXV0aG9yPll1LCBEb25nPC9hdXRob3I+PC9h
dXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRsZXM+PHRpdGxlPkNvbnZlcnNhdGlvbmFsIFNwZWVj
aCBUcmFuc2NyaXB0aW9uIFVzaW5nIENvbnRleHQtRGVwZW5kZW50IERlZXAgTmV1cmFsIE5ldHdv
cmtzPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPklOVEVSU1BFRUNIPC9zZWNvbmRhcnktdGl0bGU+
PC90aXRsZXM+PHBhZ2VzPjQzNy00NDA8L3BhZ2VzPjxkYXRlcz48eWVhcj4yMDExPC95ZWFyPjwv
ZGF0ZXM+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPkhpbnRvbjwv
QXV0aG9yPjxZZWFyPjIwMTI8L1llYXI+PFJlY051bT4xMDM8L1JlY051bT48cmVjb3JkPjxyZWMt
bnVtYmVyPjEwMzwvcmVjLW51bWJlcj48Zm9yZWlnbi1rZXlzPjxrZXkgYXBwPSJFTiIgZGItaWQ9
InBhc3RkNTJ6cmFmZGF1ZWVyZXF2ZHBzOWVhZWZ4enRyZGZzeiI+MTAzPC9rZXk+PC9mb3JlaWdu
LWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10eXBlPjxjb250
cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5IaW50b24sIEdlb2ZmcmV5PC9hdXRob3I+PGF1dGhv
cj5EZW5nLCBMaTwvYXV0aG9yPjxhdXRob3I+WXUsIERvbmc8L2F1dGhvcj48YXV0aG9yPkRhaGws
IEdlb3JnZSBFPC9hdXRob3I+PGF1dGhvcj5Nb2hhbWVkLCBBYmRlbC1yYWhtYW48L2F1dGhvcj48
YXV0aG9yPkphaXRseSwgTmF2ZGVlcDwvYXV0aG9yPjxhdXRob3I+U2VuaW9yLCBBbmRyZXc8L2F1
dGhvcj48YXV0aG9yPlZhbmhvdWNrZSwgVmluY2VudDwvYXV0aG9yPjxhdXRob3I+Tmd1eWVuLCBQ
YXRyaWNrPC9hdXRob3I+PGF1dGhvcj5TYWluYXRoLCBUYXJhIE48L2F1dGhvcj48L2F1dGhvcnM+
PC9jb250cmlidXRvcnM+PHRpdGxlcz48dGl0bGU+RGVlcCBuZXVyYWwgbmV0d29ya3MgZm9yIGFj
b3VzdGljIG1vZGVsaW5nIGluIHNwZWVjaCByZWNvZ25pdGlvbjogVGhlIHNoYXJlZCB2aWV3cyBv
ZiBmb3VyIHJlc2VhcmNoIGdyb3VwczwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5TaWduYWwgUHJv
Y2Vzc2luZyBNYWdhemluZSwgSUVFRTwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2Rp
Y2FsPjxmdWxsLXRpdGxlPlNpZ25hbCBQcm9jZXNzaW5nIE1hZ2F6aW5lLCBJRUVFPC9mdWxsLXRp
dGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+ODItOTc8L3BhZ2VzPjx2b2x1bWU+Mjk8L3ZvbHVtZT48
bnVtYmVyPjY8L251bWJlcj48ZGF0ZXM+PHllYXI+MjAxMjwveWVhcj48L2RhdGVzPjxpc2JuPjEw
NTMtNTg4ODwvaXNibj48dXJscz48L3VybHM+PC9yZWNvcmQ+PC9DaXRlPjxDaXRlPjxBdXRob3I+
QmVuZ2lvPC9BdXRob3I+PFllYXI+MjAwODwvWWVhcj48UmVjTnVtPjEwNzwvUmVjTnVtPjxyZWNv
cmQ+PHJlYy1udW1iZXI+MTA3PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+PGtleSBhcHA9IkVO
IiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij4xMDc8L2tleT48
L2ZvcmVpZ24ta2V5cz48cmVmLXR5cGUgbmFtZT0iSm91cm5hbCBBcnRpY2xlIj4xNzwvcmVmLXR5
cGU+PGNvbnRyaWJ1dG9ycz48YXV0aG9ycz48YXV0aG9yPkJlbmdpbywgWW9zaHVhPC9hdXRob3I+
PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRsZXM+PHRpdGxlPk5ldXJhbCBuZXQgbGFuZ3Vh
Z2UgbW9kZWxzPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPlNjaG9sYXJwZWRpYTwvc2Vjb25kYXJ5
LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPlNjaG9sYXJwZWRpYTwvZnVs
bC10aXRsZT48L3BlcmlvZGljYWw+PHBhZ2VzPjM4ODE8L3BhZ2VzPjx2b2x1bWU+Mzwvdm9sdW1l
PjxudW1iZXI+MTwvbnVtYmVyPjxkYXRlcz48eWVhcj4yMDA4PC95ZWFyPjwvZGF0ZXM+PGlzYm4+
MTk0MS02MDE2PC9pc2JuPjx1cmxzPjwvdXJscz48L3JlY29yZD48L0NpdGU+PENpdGU+PEF1dGhv
cj5Db2xsb2JlcnQ8L0F1dGhvcj48WWVhcj4yMDExPC9ZZWFyPjxSZWNOdW0+MTA4PC9SZWNOdW0+
PHJlY29yZD48cmVjLW51bWJlcj4xMDg8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFw
cD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjEwODwv
a2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9y
ZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+Q29sbG9iZXJ0LCBSb25hbjwv
YXV0aG9yPjxhdXRob3I+V2VzdG9uLCBKYXNvbjwvYXV0aG9yPjxhdXRob3I+Qm90dG91LCBMw6lv
bjwvYXV0aG9yPjxhdXRob3I+S2FybGVuLCBNaWNoYWVsPC9hdXRob3I+PGF1dGhvcj5LYXZ1a2N1
b2dsdSwgS29yYXk8L2F1dGhvcj48YXV0aG9yPkt1a3NhLCBQYXZlbDwvYXV0aG9yPjwvYXV0aG9y
cz48L2NvbnRyaWJ1dG9ycz48dGl0bGVzPjx0aXRsZT5OYXR1cmFsIGxhbmd1YWdlIHByb2Nlc3Np
bmcgKGFsbW9zdCkgZnJvbSBzY3JhdGNoPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPlRoZSBKb3Vy
bmFsIG9mIE1hY2hpbmUgTGVhcm5pbmcgUmVzZWFyY2g8L3NlY29uZGFyeS10aXRsZT48L3RpdGxl
cz48cGVyaW9kaWNhbD48ZnVsbC10aXRsZT5UaGUgSm91cm5hbCBvZiBNYWNoaW5lIExlYXJuaW5n
IFJlc2VhcmNoPC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+MjQ5My0yNTM3PC9wYWdl
cz48dm9sdW1lPjEyPC92b2x1bWU+PGRhdGVzPjx5ZWFyPjIwMTE8L3llYXI+PC9kYXRlcz48aXNi
bj4xNTMyLTQ0MzU8L2lzYm4+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48L0VuZE5vdGU+
ADDIN EN.CITE.DATA [26-29] and object recognition ADDIN EN.CITE <EndNote><Cite><Author>Krizhevsky</Author><Year>2012</Year><RecNum>104</RecNum><DisplayText>[30]</DisplayText><record><rec-number>104</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">104</key></foreign-keys><ref-type name="Conference Proceedings">10</ref-type><contributors><authors><author>Krizhevsky, Alex</author><author>Sutskever, Ilya</author><author>Hinton, Geoff</author></authors></contributors><titles><title>Imagenet classification with deep convolutional neural networks</title><secondary-title>Advances in Neural Information Processing Systems 25</secondary-title></titles><pages>1106-1114</pages><dates><year>2012</year></dates><urls></urls></record></Cite></EndNote>[30]. Recent success in these areas features two key ingredients: convolutional architecturesPEVuZE5vdGU+PENpdGU+PEF1dGhvcj5MZUN1bjwvQXV0aG9yPjxZZWFyPjE5OTg8L1llYXI+PFJl
Y051bT43NTwvUmVjTnVtPjxEaXNwbGF5VGV4dD5bMTcsIDIxLCAzMF08L0Rpc3BsYXlUZXh0Pjxy
ZWNvcmQ+PHJlYy1udW1iZXI+NzU8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0i
RU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjc1PC9rZXk+
PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10
eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5MZUN1biwgWWFubjwvYXV0aG9yPjxh
dXRob3I+Qm90dG91LCBMw6lvbjwvYXV0aG9yPjxhdXRob3I+QmVuZ2lvLCBZb3NodWE8L2F1dGhv
cj48YXV0aG9yPkhhZmZuZXIsIFBhdHJpY2s8L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRv
cnM+PHRpdGxlcz48dGl0bGU+R3JhZGllbnQtYmFzZWQgbGVhcm5pbmcgYXBwbGllZCB0byBkb2N1
bWVudCByZWNvZ25pdGlvbjwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5Qcm9jZWVkaW5ncyBvZiB0
aGUgSUVFRTwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxl
PlByb2NlZWRpbmdzIG9mIHRoZSBJRUVFPC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+
MjI3OC0yMzI0PC9wYWdlcz48dm9sdW1lPjg2PC92b2x1bWU+PG51bWJlcj4xMTwvbnVtYmVyPjxk
YXRlcz48eWVhcj4xOTk4PC95ZWFyPjwvZGF0ZXM+PGlzYm4+MDAxOC05MjE5PC9pc2JuPjx1cmxz
PjwvdXJscz48cmVzZWFyY2gtbm90ZXM+TU5JU1Q8L3Jlc2VhcmNoLW5vdGVzPjwvcmVjb3JkPjwv
Q2l0ZT48Q2l0ZT48QXV0aG9yPktyaXpoZXZza3k8L0F1dGhvcj48WWVhcj4yMDEyPC9ZZWFyPjxS
ZWNOdW0+MTA0PC9SZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj4xMDQ8L3JlYy1udW1iZXI+PGZv
cmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczll
YWVmeHp0cmRmc3oiPjEwNDwva2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJDb25m
ZXJlbmNlIFByb2NlZWRpbmdzIj4xMDwvcmVmLXR5cGU+PGNvbnRyaWJ1dG9ycz48YXV0aG9ycz48
YXV0aG9yPktyaXpoZXZza3ksIEFsZXg8L2F1dGhvcj48YXV0aG9yPlN1dHNrZXZlciwgSWx5YTwv
YXV0aG9yPjxhdXRob3I+SGludG9uLCBHZW9mZjwvYXV0aG9yPjwvYXV0aG9ycz48L2NvbnRyaWJ1
dG9ycz48dGl0bGVzPjx0aXRsZT5JbWFnZW5ldCBjbGFzc2lmaWNhdGlvbiB3aXRoIGRlZXAgY29u
dm9sdXRpb25hbCBuZXVyYWwgbmV0d29ya3M8L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+QWR2YW5j
ZXMgaW4gTmV1cmFsIEluZm9ybWF0aW9uIFByb2Nlc3NpbmcgU3lzdGVtcyAyNTwvc2Vjb25kYXJ5
LXRpdGxlPjwvdGl0bGVzPjxwYWdlcz4xMTA2LTExMTQ8L3BhZ2VzPjxkYXRlcz48eWVhcj4yMDEy
PC95ZWFyPjwvZGF0ZXM+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9y
PkxlZTwvQXV0aG9yPjxZZWFyPjIwMDk8L1llYXI+PFJlY051bT4zOTwvUmVjTnVtPjxyZWNvcmQ+
PHJlYy1udW1iZXI+Mzk8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRi
LWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjM5PC9rZXk+PC9mb3Jl
aWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkNvbmZlcmVuY2UgUHJvY2VlZGluZ3MiPjEwPC9yZWYt
dHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+TGVlLCBILjwvYXV0aG9yPjxhdXRo
b3I+R3Jvc3NlLCBSLjwvYXV0aG9yPjxhdXRob3I+UmFuZ2FuYXRoLCBSLjwvYXV0aG9yPjxhdXRo
b3I+TmcsIEEuWS48L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRpdGxlcz48dGl0
bGU+Q29udm9sdXRpb25hbCBkZWVwIGJlbGllZiBuZXR3b3JrcyBmb3Igc2NhbGFibGUgdW5zdXBl
cnZpc2VkIGxlYXJuaW5nIG9mIGhpZXJhcmNoaWNhbCByZXByZXNlbnRhdGlvbnM8L3RpdGxlPjxz
ZWNvbmRhcnktdGl0bGU+UHJvY2VlZGluZ3Mgb2YgdGhlIDI2dGggQW5udWFsIEludGVybmF0aW9u
YWwgQ29uZmVyZW5jZSBvbiBNYWNoaW5lIExlYXJuaW5nPC9zZWNvbmRhcnktdGl0bGU+PC90aXRs
ZXM+PHBhZ2VzPjYwOS02MTY8L3BhZ2VzPjxkYXRlcz48eWVhcj4yMDA5PC95ZWFyPjwvZGF0ZXM+
PHB1Ymxpc2hlcj5BQ008L3B1Ymxpc2hlcj48aXNibj4xNjA1NTg1MTY1PC9pc2JuPjx1cmxzPjwv
dXJscz48L3JlY29yZD48L0NpdGU+PC9FbmROb3RlPgB=
ADDIN EN.CITE PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5MZUN1bjwvQXV0aG9yPjxZZWFyPjE5OTg8L1llYXI+PFJl
Y051bT43NTwvUmVjTnVtPjxEaXNwbGF5VGV4dD5bMTcsIDIxLCAzMF08L0Rpc3BsYXlUZXh0Pjxy
ZWNvcmQ+PHJlYy1udW1iZXI+NzU8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0i
RU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjc1PC9rZXk+
PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10
eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5MZUN1biwgWWFubjwvYXV0aG9yPjxh
dXRob3I+Qm90dG91LCBMw6lvbjwvYXV0aG9yPjxhdXRob3I+QmVuZ2lvLCBZb3NodWE8L2F1dGhv
cj48YXV0aG9yPkhhZmZuZXIsIFBhdHJpY2s8L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRv
cnM+PHRpdGxlcz48dGl0bGU+R3JhZGllbnQtYmFzZWQgbGVhcm5pbmcgYXBwbGllZCB0byBkb2N1
bWVudCByZWNvZ25pdGlvbjwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5Qcm9jZWVkaW5ncyBvZiB0
aGUgSUVFRTwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxl
PlByb2NlZWRpbmdzIG9mIHRoZSBJRUVFPC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+
MjI3OC0yMzI0PC9wYWdlcz48dm9sdW1lPjg2PC92b2x1bWU+PG51bWJlcj4xMTwvbnVtYmVyPjxk
YXRlcz48eWVhcj4xOTk4PC95ZWFyPjwvZGF0ZXM+PGlzYm4+MDAxOC05MjE5PC9pc2JuPjx1cmxz
PjwvdXJscz48cmVzZWFyY2gtbm90ZXM+TU5JU1Q8L3Jlc2VhcmNoLW5vdGVzPjwvcmVjb3JkPjwv
Q2l0ZT48Q2l0ZT48QXV0aG9yPktyaXpoZXZza3k8L0F1dGhvcj48WWVhcj4yMDEyPC9ZZWFyPjxS
ZWNOdW0+MTA0PC9SZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj4xMDQ8L3JlYy1udW1iZXI+PGZv
cmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczll
YWVmeHp0cmRmc3oiPjEwNDwva2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJDb25m
ZXJlbmNlIFByb2NlZWRpbmdzIj4xMDwvcmVmLXR5cGU+PGNvbnRyaWJ1dG9ycz48YXV0aG9ycz48
YXV0aG9yPktyaXpoZXZza3ksIEFsZXg8L2F1dGhvcj48YXV0aG9yPlN1dHNrZXZlciwgSWx5YTwv
YXV0aG9yPjxhdXRob3I+SGludG9uLCBHZW9mZjwvYXV0aG9yPjwvYXV0aG9ycz48L2NvbnRyaWJ1
dG9ycz48dGl0bGVzPjx0aXRsZT5JbWFnZW5ldCBjbGFzc2lmaWNhdGlvbiB3aXRoIGRlZXAgY29u
dm9sdXRpb25hbCBuZXVyYWwgbmV0d29ya3M8L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+QWR2YW5j
ZXMgaW4gTmV1cmFsIEluZm9ybWF0aW9uIFByb2Nlc3NpbmcgU3lzdGVtcyAyNTwvc2Vjb25kYXJ5
LXRpdGxlPjwvdGl0bGVzPjxwYWdlcz4xMTA2LTExMTQ8L3BhZ2VzPjxkYXRlcz48eWVhcj4yMDEy
PC95ZWFyPjwvZGF0ZXM+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9y
PkxlZTwvQXV0aG9yPjxZZWFyPjIwMDk8L1llYXI+PFJlY051bT4zOTwvUmVjTnVtPjxyZWNvcmQ+
PHJlYy1udW1iZXI+Mzk8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRi
LWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjM5PC9rZXk+PC9mb3Jl
aWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkNvbmZlcmVuY2UgUHJvY2VlZGluZ3MiPjEwPC9yZWYt
dHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+TGVlLCBILjwvYXV0aG9yPjxhdXRo
b3I+R3Jvc3NlLCBSLjwvYXV0aG9yPjxhdXRob3I+UmFuZ2FuYXRoLCBSLjwvYXV0aG9yPjxhdXRo
b3I+TmcsIEEuWS48L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRpdGxlcz48dGl0
bGU+Q29udm9sdXRpb25hbCBkZWVwIGJlbGllZiBuZXR3b3JrcyBmb3Igc2NhbGFibGUgdW5zdXBl
cnZpc2VkIGxlYXJuaW5nIG9mIGhpZXJhcmNoaWNhbCByZXByZXNlbnRhdGlvbnM8L3RpdGxlPjxz
ZWNvbmRhcnktdGl0bGU+UHJvY2VlZGluZ3Mgb2YgdGhlIDI2dGggQW5udWFsIEludGVybmF0aW9u
YWwgQ29uZmVyZW5jZSBvbiBNYWNoaW5lIExlYXJuaW5nPC9zZWNvbmRhcnktdGl0bGU+PC90aXRs
ZXM+PHBhZ2VzPjYwOS02MTY8L3BhZ2VzPjxkYXRlcz48eWVhcj4yMDA5PC95ZWFyPjwvZGF0ZXM+
PHB1Ymxpc2hlcj5BQ008L3B1Ymxpc2hlcj48aXNibj4xNjA1NTg1MTY1PC9pc2JuPjx1cmxzPjwv
dXJscz48L3JlY29yZD48L0NpdGU+PC9FbmROb3RlPgB=
ADDIN EN.CITE.DATA [17, 21, 30] and dropouts ADDIN EN.CITE <EndNote><Cite><Author>Hinton</Author><Year>2012</Year><RecNum>90</RecNum><DisplayText>[31, 32]</DisplayText><record><rec-number>90</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">90</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Hinton, Geoffrey E</author><author>Srivastava, Nitish</author><author>Krizhevsky, Alex</author><author>Sutskever, Ilya</author><author>Salakhutdinov, Ruslan R</author></authors></contributors><titles><title>Improving neural networks by preventing co-adaptation of feature detectors</title><secondary-title>arXiv preprint arXiv:1207.0580</secondary-title></titles><periodical><full-title>arXiv preprint arXiv:1207.0580</full-title></periodical><dates><year>2012</year></dates><urls></urls><research-notes>dropout</research-notes></record></Cite><Cite><Author>Goodfellow</Author><Year>2013</Year><RecNum>91</RecNum><record><rec-number>91</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">91</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Goodfellow, Ian J</author><author>Warde-Farley, David</author><author>Mirza, Mehdi</author><author>Courville, Aaron</author><author>Bengio, Yoshua</author></authors></contributors><titles><title>Maxout networks</title><secondary-title>arXiv preprint arXiv:1302.4389</secondary-title></titles><periodical><full-title>arXiv preprint arXiv:1302.4389</full-title></periodical><dates><year>2013</year></dates><urls></urls></record></Cite></EndNote>[31, 32].The convolutional architectures alternate convolutional layers and pooling layers. Units on a convolutional layer only deal with a small window which corresponds to a spatial or temporal position. Units on a pooling layer aggregate the outputs on units at a lower convolutional layer.Dropouts intentionally ignore random nodes in hidden layers and input layers when training. This process mimics averaging multiple models. It also improves neural networks by preventing co-adaptation of feature detectors by forcing different nodes to learn different features.Sparsity, denoising, and dropouts all aim to reduce the capacity of neural networks to prevent overfitting.Chapter 3Meta-parameters and Data Pre-processing in Deep Learning3.1 MotivationExisting deep learning methods involve a large number of meta-parameters, such as the number of hidden layers, the number of hidden nodes, the sparsity target, the initial values of weights, the type of units, the learning rate, etc. Existing applications usually do not explain why the decisions were made and how changes would affect performance. Firstly, how to determine the structure of deep learning networks is unknown. For example, the first DBN paper ADDIN EN.CITE <EndNote><Cite><Author>Hinton</Author><Year>2006</Year><RecNum>31</RecNum><DisplayText>[2]</DisplayText><record><rec-number>31</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">31</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Hinton, G.E.</author><author>Salakhutdinov, R.R.</author></authors></contributors><titles><title>Reducing the dimensionality of data with neural networks</title><secondary-title>Science</secondary-title></titles><periodical><full-title>Science</full-title></periodical><pages>504-507</pages><volume>313</volume><number>5786</number><dates><year>2006</year></dates><isbn>0036-8075</isbn><urls></urls></record></Cite></EndNote>[2] used a 784-500-500-2000-10 DBN to achieve 98.8% classification accuracy on MNIST. It would be interesting to know why such a network configuration was chosen and if other choices could yield similar results. Thus, it is difficult for a novice user to make good decisions for a new application in order to achieve good performance. In addition, deep learning is promising to extract features on its own, therefore the raw input should work well. But what if we feed in hand-engineered features? Will it work better? Despite the same or slightly worse performance, is it worthwhile to apply dimension reduction beforehand to speed up the training process?Last, even if the two types of building blocks in deep learning, namely RBMs and autoencoders, are expected to function similarly, it is still necessary to compare their performance in different settings on different datasets.This chapter will address these three important questions.3.2 Dimensionality Reduction by PCA and WhiteningWhen the input space is too large for neural networks to handle, we often want to reduce the dimensionality of the input data to significantly speed up the training process. This could be done by a general dimensionality reduction algorithm, or by an expert designed feature extractor. The latter has the benefit of obtaining meaningful representation but it also loses some information of the raw data.For problems that do not have human understandable features, principal component analysis (PCA) is a popular choice as a linear dimensionality reduction technique. PCA is used to find a lower-dimensional subspace onto which the original data is projected. Usually we need data x that has zero mean. If the data does not have the property, we zero mean it. First we compute the covariance matrix of x as= 1mi=1m(xi)(xi)T MACROBUTTON NumberReference \* MERGEFORMAT (19)where xi is one data point and m is the number of data points.Then we apply standard singular value decomposition to the covariance matrix to get the matrix U of eigenvectors and the vector λ of eigenvalues. We could preserve all the input data’s information but present it in a different basis by computingxrot= UTx MACROBUTTON NumberReference \* MERGEFORMAT (20)The reduced dimension representation of the data can be obtained by computingx=U1…kTx1…k MACROBUTTON NumberReference \* MERGEFORMAT (21)where k is the dimension to keep.To set k, we usually determine it by the percentage of variance retained, which is given byj=1kλjj=1nλj MACROBUTTON NumberReference \* MERGEFORMAT (22)To retain 99% of the variance, we need to pick the smallest value of k such at the above value is no less than 0.99. The special case is k = n, when the retaining rate is 1, which means the PCA processed data containing all the variance.The goal of whitening is to make features less correlated and have the same variance. We can simply get it done by rescaling features byxPCAwhite,i=xiλi+ε MACROBUTTON NumberReference \* MERGEFORMAT (23)where ε is a regularization term to prevent the result from blowing up.3.3 Sleep Stage Dataset and its FeaturesThe MNIST dataset and a sleep stage dataset will be used in this chapter. This section is the description of the benchmark sleep stage dataset provided by PhysioNet ADDIN EN.CITE <EndNote><Cite><Author>Goldberger</Author><Year>2000</Year><RecNum>111</RecNum><DisplayText>[33]</DisplayText><record><rec-number>111</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">111</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Goldberger, Ary L</author><author>Amaral, Luis AN</author><author>Glass, Leon</author><author>Hausdorff, Jeffrey M</author><author>Ivanov, Plamen Ch</author><author>Mark, Roger G</author><author>Mietus, Joseph E</author><author>Moody, George B</author><author>Peng, Chung-Kang</author><author>Stanley, H Eugene</author></authors></contributors><titles><title>Physiobank, physiotoolkit, and physionet components of a new research resource for complex physiologic signals</title><secondary-title>Circulation</secondary-title></titles><periodical><full-title>Circulation</full-title></periodical><pages>e215-e220</pages><volume>101</volume><number>23</number><dates><year>2000</year></dates><isbn>0009-7322</isbn><urls></urls></record></Cite></EndNote>[33]. This study will use the first 5 acquisitions out of 25 available in the dataset (21 males and 4 females with average age 50), each consisting of 1 EEG channel (C3-A2), 2 EOG channels, and 1 EMG channel downsampled at 64Hz. Each acquisition lasts about 7 hours on average. Each sample is taken from 1-second window with 256 dimensions, normalized to [0, 1]. The labels are 5 sleep stages: awake, stage 1 (S1), stage 2 (S2), slow wave sleep (SWS) and rapid eye-movement sleep (REM).A band-pass filter and a notch filter at 50Hz are applied to all channels. Then the 30-second data before and after a sleep stage switch are removed. Finally all classes are balanced determined by the class with the fewest samples. 1/7 data is reserved for testing and the rest is for training, which results in 8508 samples for testing and 51042 for training. A 60-second raw data of each channel is depicted in REF _Ref371809860 \h Fig. 15. 28 hand-made features are extracted from 1-second long samples. The features of a subject’s data are shown in REF _Ref371810722 \h Fig. 16. The details about how to calculate these features can be found in PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5Mw6RuZ2t2aXN0PC9BdXRob3I+PFllYXI+MjAxMjwvWWVh
cj48UmVjTnVtPjIzPC9SZWNOdW0+PERpc3BsYXlUZXh0PlszNC0zOF08L0Rpc3BsYXlUZXh0Pjxy
ZWNvcmQ+PHJlYy1udW1iZXI+MjM8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0i
RU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjIzPC9rZXk+
PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10
eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5Mw6RuZ2t2aXN0LCBNLjwvYXV0aG9y
PjxhdXRob3I+S2FybHNzb24sIEwuPC9hdXRob3I+PGF1dGhvcj5Mb3V0ZmksIEEuPC9hdXRob3I+
PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRsZXM+PHRpdGxlPlNsZWVwIFN0YWdlIENsYXNz
aWZpY2F0aW9uIFVzaW5nIFVuc3VwZXJ2aXNlZCBGZWF0dXJlIExlYXJuaW5nPC90aXRsZT48c2Vj
b25kYXJ5LXRpdGxlPkFkdmFuY2VzIGluIEFydGlmaWNpYWwgTmV1cmFsIFN5c3RlbXM8L3NlY29u
ZGFyeS10aXRsZT48L3RpdGxlcz48cGVyaW9kaWNhbD48ZnVsbC10aXRsZT5BZHZhbmNlcyBpbiBB
cnRpZmljaWFsIE5ldXJhbCBTeXN0ZW1zPC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48dm9sdW1l
PjIwMTI8L3ZvbHVtZT48ZGF0ZXM+PHllYXI+MjAxMjwveWVhcj48L2RhdGVzPjxpc2JuPjE2ODct
NzU5NDwvaXNibj48dXJscz48L3VybHM+PHJlc2VhcmNoLW5vdGVzPkRCTlRvb2xib3g8L3Jlc2Vh
cmNoLW5vdGVzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPk9zYm9ybmU8L0F1dGhvcj48
WWVhcj4xOTg5PC9ZZWFyPjxSZWNOdW0+MTEzPC9SZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj4x
MTM8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUy
enJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjExMzwva2V5PjwvZm9yZWlnbi1rZXlzPjxy
ZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9yZWYtdHlwZT48Y29udHJpYnV0b3Jz
PjxhdXRob3JzPjxhdXRob3I+T3Nib3JuZSwgQSBSbzwvYXV0aG9yPjxhdXRob3I+UHJvdmVuemFs
ZSwgQTwvYXV0aG9yPjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0bGVzPjx0aXRsZT5GaW5p
dGUgY29ycmVsYXRpb24gZGltZW5zaW9uIGZvciBzdG9jaGFzdGljIHN5c3RlbXMgd2l0aCBwb3dl
ci1sYXcgc3BlY3RyYTwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5QaHlzaWNhIEQ6IE5vbmxpbmVh
ciBQaGVub21lbmE8L3NlY29uZGFyeS10aXRsZT48L3RpdGxlcz48cGVyaW9kaWNhbD48ZnVsbC10
aXRsZT5QaHlzaWNhIEQ6IE5vbmxpbmVhciBQaGVub21lbmE8L2Z1bGwtdGl0bGU+PC9wZXJpb2Rp
Y2FsPjxwYWdlcz4zNTctMzgxPC9wYWdlcz48dm9sdW1lPjM1PC92b2x1bWU+PG51bWJlcj4zPC9u
dW1iZXI+PGRhdGVzPjx5ZWFyPjE5ODk8L3llYXI+PC9kYXRlcz48aXNibj4wMTY3LTI3ODk8L2lz
Ym4+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPlpvdWJlazwvQXV0
aG9yPjxZZWFyPjIwMDc8L1llYXI+PFJlY051bT4xMTI8L1JlY051bT48cmVjb3JkPjxyZWMtbnVt
YmVyPjExMjwvcmVjLW51bWJlcj48Zm9yZWlnbi1rZXlzPjxrZXkgYXBwPSJFTiIgZGItaWQ9InBh
c3RkNTJ6cmFmZGF1ZWVyZXF2ZHBzOWVhZWZ4enRyZGZzeiI+MTEyPC9rZXk+PC9mb3JlaWduLWtl
eXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10eXBlPjxjb250cmli
dXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5ab3ViZWssIEx1a8OhxaE8L2F1dGhvcj48YXV0aG9yPkNo
YXJib25uaWVyLCBTeWx2aWU8L2F1dGhvcj48YXV0aG9yPkxlc2VjcSwgU3V6YW5uZTwvYXV0aG9y
PjxhdXRob3I+QnVndWV0LCBBbGFpbjwvYXV0aG9yPjxhdXRob3I+Q2hhcG90b3QsIEZsb3JpYW48
L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRpdGxlcz48dGl0bGU+RmVhdHVyZSBz
ZWxlY3Rpb24gZm9yIHNsZWVwL3dha2Ugc3RhZ2VzIGNsYXNzaWZpY2F0aW9uIHVzaW5nIGRhdGEg
ZHJpdmVuIG1ldGhvZHM8L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+QmlvbWVkaWNhbCBTaWduYWwg
UHJvY2Vzc2luZyBhbmQgQ29udHJvbDwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2Rp
Y2FsPjxmdWxsLXRpdGxlPkJpb21lZGljYWwgU2lnbmFsIFByb2Nlc3NpbmcgYW5kIENvbnRyb2w8
L2Z1bGwtdGl0bGU+PC9wZXJpb2RpY2FsPjxwYWdlcz4xNzEtMTc5PC9wYWdlcz48dm9sdW1lPjI8
L3ZvbHVtZT48bnVtYmVyPjM8L251bWJlcj48ZGF0ZXM+PHllYXI+MjAwNzwveWVhcj48L2RhdGVz
Pjxpc2JuPjE3NDYtODA5NDwvaXNibj48dXJscz48L3VybHM+PC9yZWNvcmQ+PC9DaXRlPjxDaXRl
PjxBdXRob3I+UGVyZWRhPC9BdXRob3I+PFllYXI+MTk5ODwvWWVhcj48UmVjTnVtPjExNDwvUmVj
TnVtPjxyZWNvcmQ+PHJlYy1udW1iZXI+MTE0PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+PGtl
eSBhcHA9IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij4x
MTQ8L2tleT48L2ZvcmVpZ24ta2V5cz48cmVmLXR5cGUgbmFtZT0iSm91cm5hbCBBcnRpY2xlIj4x
NzwvcmVmLXR5cGU+PGNvbnRyaWJ1dG9ycz48YXV0aG9ycz48YXV0aG9yPlBlcmVkYSwgRTwvYXV0
aG9yPjxhdXRob3I+R2FtdW5kaSwgQTwvYXV0aG9yPjxhdXRob3I+UmlhbCwgUjwvYXV0aG9yPjxh
dXRob3I+R29uemFsZXosIEo8L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRpdGxl
cz48dGl0bGU+Tm9uLWxpbmVhciBiZWhhdmlvdXIgb2YgaHVtYW4gRUVHOiBmcmFjdGFsIGV4cG9u
ZW50IHZlcnN1cyBjb3JyZWxhdGlvbiBkaW1lbnNpb24gaW4gYXdha2UgYW5kIHNsZWVwIHN0YWdl
czwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5OZXVyb3NjaWVuY2UgbGV0dGVyczwvc2Vjb25kYXJ5
LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPk5ldXJvc2NpZW5jZSBsZXR0
ZXJzPC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+OTEtOTQ8L3BhZ2VzPjx2b2x1bWU+
MjUwPC92b2x1bWU+PG51bWJlcj4yPC9udW1iZXI+PGRhdGVzPjx5ZWFyPjE5OTg8L3llYXI+PC9k
YXRlcz48aXNibj4wMzA0LTM5NDA8L2lzYm4+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48
Q2l0ZT48QXV0aG9yPkdhc3NlcjwvQXV0aG9yPjxZZWFyPjE5ODI8L1llYXI+PFJlY051bT4xMTU8
L1JlY051bT48cmVjb3JkPjxyZWMtbnVtYmVyPjExNTwvcmVjLW51bWJlcj48Zm9yZWlnbi1rZXlz
PjxrZXkgYXBwPSJFTiIgZGItaWQ9InBhc3RkNTJ6cmFmZGF1ZWVyZXF2ZHBzOWVhZWZ4enRyZGZz
eiI+MTE1PC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNs
ZSI+MTc8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5HYXNzZXIsIFRo
ZW88L2F1dGhvcj48YXV0aG9yPkLDpGNoZXIsIFBldHJhPC9hdXRob3I+PGF1dGhvcj5Nw7Zja3Ms
IEpvYWNoaW08L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRpdGxlcz48dGl0bGU+
VHJhbnNmb3JtYXRpb25zIHRvd2FyZHMgdGhlIG5vcm1hbCBkaXN0cmlidXRpb24gb2YgYnJvYWQg
YmFuZCBzcGVjdHJhbCBwYXJhbWV0ZXJzIG9mIHRoZSBFRUc8L3RpdGxlPjxzZWNvbmRhcnktdGl0
bGU+RWxlY3Ryb2VuY2VwaGFsb2dyYXBoeSBhbmQgY2xpbmljYWwgbmV1cm9waHlzaW9sb2d5PC9z
ZWNvbmRhcnktdGl0bGU+PC90aXRsZXM+PHBlcmlvZGljYWw+PGZ1bGwtdGl0bGU+RWxlY3Ryb2Vu
Y2VwaGFsb2dyYXBoeSBhbmQgY2xpbmljYWwgbmV1cm9waHlzaW9sb2d5PC9mdWxsLXRpdGxlPjwv
cGVyaW9kaWNhbD48cGFnZXM+MTE5LTEyNDwvcGFnZXM+PHZvbHVtZT41Mzwvdm9sdW1lPjxudW1i
ZXI+MTwvbnVtYmVyPjxkYXRlcz48eWVhcj4xOTgyPC95ZWFyPjwvZGF0ZXM+PGlzYm4+MDAxMy00
Njk0PC9pc2JuPjx1cmxzPjwvdXJscz48L3JlY29yZD48L0NpdGU+PC9FbmROb3RlPn==
ADDIN EN.CITE PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5Mw6RuZ2t2aXN0PC9BdXRob3I+PFllYXI+MjAxMjwvWWVh
cj48UmVjTnVtPjIzPC9SZWNOdW0+PERpc3BsYXlUZXh0PlszNC0zOF08L0Rpc3BsYXlUZXh0Pjxy
ZWNvcmQ+PHJlYy1udW1iZXI+MjM8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0i
RU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjIzPC9rZXk+
PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10
eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5Mw6RuZ2t2aXN0LCBNLjwvYXV0aG9y
PjxhdXRob3I+S2FybHNzb24sIEwuPC9hdXRob3I+PGF1dGhvcj5Mb3V0ZmksIEEuPC9hdXRob3I+
PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRsZXM+PHRpdGxlPlNsZWVwIFN0YWdlIENsYXNz
aWZpY2F0aW9uIFVzaW5nIFVuc3VwZXJ2aXNlZCBGZWF0dXJlIExlYXJuaW5nPC90aXRsZT48c2Vj
b25kYXJ5LXRpdGxlPkFkdmFuY2VzIGluIEFydGlmaWNpYWwgTmV1cmFsIFN5c3RlbXM8L3NlY29u
ZGFyeS10aXRsZT48L3RpdGxlcz48cGVyaW9kaWNhbD48ZnVsbC10aXRsZT5BZHZhbmNlcyBpbiBB
cnRpZmljaWFsIE5ldXJhbCBTeXN0ZW1zPC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48dm9sdW1l
PjIwMTI8L3ZvbHVtZT48ZGF0ZXM+PHllYXI+MjAxMjwveWVhcj48L2RhdGVzPjxpc2JuPjE2ODct
NzU5NDwvaXNibj48dXJscz48L3VybHM+PHJlc2VhcmNoLW5vdGVzPkRCTlRvb2xib3g8L3Jlc2Vh
cmNoLW5vdGVzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPk9zYm9ybmU8L0F1dGhvcj48
WWVhcj4xOTg5PC9ZZWFyPjxSZWNOdW0+MTEzPC9SZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj4x
MTM8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUy
enJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjExMzwva2V5PjwvZm9yZWlnbi1rZXlzPjxy
ZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9yZWYtdHlwZT48Y29udHJpYnV0b3Jz
PjxhdXRob3JzPjxhdXRob3I+T3Nib3JuZSwgQSBSbzwvYXV0aG9yPjxhdXRob3I+UHJvdmVuemFs
ZSwgQTwvYXV0aG9yPjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0bGVzPjx0aXRsZT5GaW5p
dGUgY29ycmVsYXRpb24gZGltZW5zaW9uIGZvciBzdG9jaGFzdGljIHN5c3RlbXMgd2l0aCBwb3dl
ci1sYXcgc3BlY3RyYTwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5QaHlzaWNhIEQ6IE5vbmxpbmVh
ciBQaGVub21lbmE8L3NlY29uZGFyeS10aXRsZT48L3RpdGxlcz48cGVyaW9kaWNhbD48ZnVsbC10
aXRsZT5QaHlzaWNhIEQ6IE5vbmxpbmVhciBQaGVub21lbmE8L2Z1bGwtdGl0bGU+PC9wZXJpb2Rp
Y2FsPjxwYWdlcz4zNTctMzgxPC9wYWdlcz48dm9sdW1lPjM1PC92b2x1bWU+PG51bWJlcj4zPC9u
dW1iZXI+PGRhdGVzPjx5ZWFyPjE5ODk8L3llYXI+PC9kYXRlcz48aXNibj4wMTY3LTI3ODk8L2lz
Ym4+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPlpvdWJlazwvQXV0
aG9yPjxZZWFyPjIwMDc8L1llYXI+PFJlY051bT4xMTI8L1JlY051bT48cmVjb3JkPjxyZWMtbnVt
YmVyPjExMjwvcmVjLW51bWJlcj48Zm9yZWlnbi1rZXlzPjxrZXkgYXBwPSJFTiIgZGItaWQ9InBh
c3RkNTJ6cmFmZGF1ZWVyZXF2ZHBzOWVhZWZ4enRyZGZzeiI+MTEyPC9rZXk+PC9mb3JlaWduLWtl
eXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10eXBlPjxjb250cmli
dXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5ab3ViZWssIEx1a8OhxaE8L2F1dGhvcj48YXV0aG9yPkNo
YXJib25uaWVyLCBTeWx2aWU8L2F1dGhvcj48YXV0aG9yPkxlc2VjcSwgU3V6YW5uZTwvYXV0aG9y
PjxhdXRob3I+QnVndWV0LCBBbGFpbjwvYXV0aG9yPjxhdXRob3I+Q2hhcG90b3QsIEZsb3JpYW48
L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRpdGxlcz48dGl0bGU+RmVhdHVyZSBz
ZWxlY3Rpb24gZm9yIHNsZWVwL3dha2Ugc3RhZ2VzIGNsYXNzaWZpY2F0aW9uIHVzaW5nIGRhdGEg
ZHJpdmVuIG1ldGhvZHM8L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+QmlvbWVkaWNhbCBTaWduYWwg
UHJvY2Vzc2luZyBhbmQgQ29udHJvbDwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2Rp
Y2FsPjxmdWxsLXRpdGxlPkJpb21lZGljYWwgU2lnbmFsIFByb2Nlc3NpbmcgYW5kIENvbnRyb2w8
L2Z1bGwtdGl0bGU+PC9wZXJpb2RpY2FsPjxwYWdlcz4xNzEtMTc5PC9wYWdlcz48dm9sdW1lPjI8
L3ZvbHVtZT48bnVtYmVyPjM8L251bWJlcj48ZGF0ZXM+PHllYXI+MjAwNzwveWVhcj48L2RhdGVz
Pjxpc2JuPjE3NDYtODA5NDwvaXNibj48dXJscz48L3VybHM+PC9yZWNvcmQ+PC9DaXRlPjxDaXRl
PjxBdXRob3I+UGVyZWRhPC9BdXRob3I+PFllYXI+MTk5ODwvWWVhcj48UmVjTnVtPjExNDwvUmVj
TnVtPjxyZWNvcmQ+PHJlYy1udW1iZXI+MTE0PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+PGtl
eSBhcHA9IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij4x
MTQ8L2tleT48L2ZvcmVpZ24ta2V5cz48cmVmLXR5cGUgbmFtZT0iSm91cm5hbCBBcnRpY2xlIj4x
NzwvcmVmLXR5cGU+PGNvbnRyaWJ1dG9ycz48YXV0aG9ycz48YXV0aG9yPlBlcmVkYSwgRTwvYXV0
aG9yPjxhdXRob3I+R2FtdW5kaSwgQTwvYXV0aG9yPjxhdXRob3I+UmlhbCwgUjwvYXV0aG9yPjxh
dXRob3I+R29uemFsZXosIEo8L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRpdGxl
cz48dGl0bGU+Tm9uLWxpbmVhciBiZWhhdmlvdXIgb2YgaHVtYW4gRUVHOiBmcmFjdGFsIGV4cG9u
ZW50IHZlcnN1cyBjb3JyZWxhdGlvbiBkaW1lbnNpb24gaW4gYXdha2UgYW5kIHNsZWVwIHN0YWdl
czwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5OZXVyb3NjaWVuY2UgbGV0dGVyczwvc2Vjb25kYXJ5
LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPk5ldXJvc2NpZW5jZSBsZXR0
ZXJzPC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+OTEtOTQ8L3BhZ2VzPjx2b2x1bWU+
MjUwPC92b2x1bWU+PG51bWJlcj4yPC9udW1iZXI+PGRhdGVzPjx5ZWFyPjE5OTg8L3llYXI+PC9k
YXRlcz48aXNibj4wMzA0LTM5NDA8L2lzYm4+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48
Q2l0ZT48QXV0aG9yPkdhc3NlcjwvQXV0aG9yPjxZZWFyPjE5ODI8L1llYXI+PFJlY051bT4xMTU8
L1JlY051bT48cmVjb3JkPjxyZWMtbnVtYmVyPjExNTwvcmVjLW51bWJlcj48Zm9yZWlnbi1rZXlz
PjxrZXkgYXBwPSJFTiIgZGItaWQ9InBhc3RkNTJ6cmFmZGF1ZWVyZXF2ZHBzOWVhZWZ4enRyZGZz
eiI+MTE1PC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNs
ZSI+MTc8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5HYXNzZXIsIFRo
ZW88L2F1dGhvcj48YXV0aG9yPkLDpGNoZXIsIFBldHJhPC9hdXRob3I+PGF1dGhvcj5Nw7Zja3Ms
IEpvYWNoaW08L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRpdGxlcz48dGl0bGU+
VHJhbnNmb3JtYXRpb25zIHRvd2FyZHMgdGhlIG5vcm1hbCBkaXN0cmlidXRpb24gb2YgYnJvYWQg
YmFuZCBzcGVjdHJhbCBwYXJhbWV0ZXJzIG9mIHRoZSBFRUc8L3RpdGxlPjxzZWNvbmRhcnktdGl0
bGU+RWxlY3Ryb2VuY2VwaGFsb2dyYXBoeSBhbmQgY2xpbmljYWwgbmV1cm9waHlzaW9sb2d5PC9z
ZWNvbmRhcnktdGl0bGU+PC90aXRsZXM+PHBlcmlvZGljYWw+PGZ1bGwtdGl0bGU+RWxlY3Ryb2Vu
Y2VwaGFsb2dyYXBoeSBhbmQgY2xpbmljYWwgbmV1cm9waHlzaW9sb2d5PC9mdWxsLXRpdGxlPjwv
cGVyaW9kaWNhbD48cGFnZXM+MTE5LTEyNDwvcGFnZXM+PHZvbHVtZT41Mzwvdm9sdW1lPjxudW1i
ZXI+MTwvbnVtYmVyPjxkYXRlcz48eWVhcj4xOTgyPC95ZWFyPjwvZGF0ZXM+PGlzYm4+MDAxMy00
Njk0PC9pc2JuPjx1cmxzPjwvdXJscz48L3JlY29yZD48L0NpdGU+PC9FbmROb3RlPn==
ADDIN EN.CITE.DATA [34-38].The corresponding classes are shown in REF _Ref371811120 \h Fig. 17.Fig. SEQ Fig. \* ARABIC 15 60-second raw data downsampled at 64Hz of sleep stage dataset. Fig. SEQ Fig. \* ARABIC 16 28 features of a subject's data. X-axis is time in seconds and y-axis shows non-normalized valuesFig. SEQ Fig. \* ARABIC 17 sleep stage classes of a subject. X-axis is time in seconds and y-axis is the 5 possible sleep stages3.4 Investigation of Optimal Meta-parameters and Pre-processing TechniquesThis study will conduct a search on 2 datasets (MNIST and sleep stage dataset) along 3 dimensions: raw vs. PCA95% whitened data as input, net structures (different numbers of layers and different numbers of nodes), and RBM vs. autoencoder as learning units. There will be 8 possible combinations.Softmax regression serves as a baseline compared to more advanced deep learning algorithms (i.e., stacked RBMs and stacked autoencoders).The following two sections will be experiments carried out on MNIST and the sleep stage dataset, respectively. In the first subsection, softmax regression only will be used to evaluate the performance of raw data vs. PCA processed data on MNIST or features on the sleep stage dataset. In the second subsection, different net structures will be tried on raw data and picked PCA processed data (as for retaining rate and with or without whitening) or features.3.5 Experimental Results on MNIST Dataset3.5.1 Data Pre-processingTo evaluate how PCA processing with different retention rates and whitening affects the performance compared to the raw data, softmax regression is applied to the MNIST dataset.3.5.1.1 On Raw DataWeight decay of the softmax regression is set to be 3e-3. The training on raw data takes 17 seconds and achieves 0.9150 classification accuracy. The learned weights by the 10 softmax nodes in the output layer are shown in REF _Ref371813199 \h Fig. 18.Fig. SEQ Fig. \* ARABIC 18 weights learned by 10 softmax nodes on raw of MNIST3.5.1.2 On PCA With or Without Whitening90%, 95%, 99%, and 100% retention rates for PCA with or without whitening (regularization ε = 0.1) are evaluated. The quality of recovered data from the lower dimensional space is shown in REF _Ref371815923 \h Fig. 19. Even if we reduce the data dimension to 64-d, it still retains 90% of the variance, and the digits are highly recognizable. rawraw with zero meanPCA90% 64-d PCA95% 120-dPCA99% 300-d PCA100% 784-dPCA90% with whitening PCA95% with whiteningPCA99% with whitening PCA100% with whiteningFig. SEQ Fig. \* ARABIC 19 PCA with or without whiteningThe classification accuracy of softmax regression on raw and pre-processed data of MNIST is shown in REF _Ref371857854 \h Fig. 20. The time spent on training is depicted in REF _Ref371857957 \h Fig. 21.Fig. SEQ Fig. \* ARABIC 20 Classification accuracy of softmax regression on MNISTFig. SEQ Fig. \* ARABIC 21Training time of softmax regression on MNISTFor softmax, using whitening or not does not affect the classification accuracy much, but PCA especially with whitening reduces training time a lot. Whitened data has smaller variations, which could be the reason it converges faster. For example, PCA95 with whitening saves ? of training time with less than 0.3% accuracy loss. However, PCA process alone takes about 10 seconds and whitening takes additional 10 seconds. The pre-processing overhead makes it less attractive for simple classifier such as softmax regression.3.5.2 Deep Learning Network StructureClassifications on the raw and PCA95% whitened data using different network structures in terms of number of layers and number of nodes and different learning units will be carried out to compare their performance.3.5.2.1 Stacked RBMs on Raw Data REF _Ref371863068 \h Fig. 22 shows classification accuracy of stacked RBMs of various network structures on raw of MNIST. More layers and more nodes seem to achieve better performance, but the performance gain is not very significant when the number of nodes grows more than 200. The corresponding training time is shown in REF _Ref371863367 \h Fig. 23. The training time roughly has a linear relationship with the number of layers and number of nodes in each layer.Fig. SEQ Fig. \* ARABIC 22 Classification accuracy of stacked RBMs of various net structures on raw of MNISTFig. SEQ Fig. \* ARABIC 23 Training time of stacked RBMs of various net structures on raw of MNIST3.5.2.2 Stacked RBMs on PCA at Retention Rate 95% REF _Ref371869391 \h Fig. 24 and REF _Ref371869394 \h Fig. 25 show classification accuracy and training time of stacked RBMs of various network structures on PCA95% whitened data of MNIST. The classification accuracy is no better than that of softmax. Even worse, the 3-layer case can only achieve less than 0.8 classification accuracy. The reason is PCA itself extracts uncorrelated features and the stacked RBMs try to extract some correlations from uncorrelated features and of course fails. Thanks to the topmost softmax layer, stacked RBMs with 1 and 2 layers still work but 3-layer network does a terrible job in the pre-training stage which cannot be offset by the fine-tuning stage.Fig. SEQ Fig. \* ARABIC 24 Classification accuracy of stacked RBMs of various net structures on PCA95 whitened of MNISTFig. SEQ Fig. \* ARABIC 25 Training time of stacked RBMs of various net structures on raw of MNIST3.5.2.3 Stacked Autoencoders on Raw Data REF _Ref371869416 \h Fig. 26 and REF _Ref371869418 \h Fig. 27 show classification accuracy and training time of stacked autoencoders of various network structures on raw of MNIST. When the number of nodes in each layer is small (e.g., 50), the network with 3 layers captures less and less useful information from bottom to up due to too few nodes with sparsity regularization, so the classification accuracy is as low as less than 0.91. As the number of nodes grows, the performance becomes better, and eventually beats 1-layer network.Fig. SEQ Fig. \* ARABIC 26 Classification accuracy of stacked autoencoders of various net structures on raw of MNISTFig. SEQ Fig. \* ARABIC 27 Training time of stacked autoencoders of various net structures on raw of MNIST3.5.2.4 Stacked Autoencoders on PCA at Retention Rate 95% REF _Ref371870323 \h Fig. 28 and REF _Ref371870325 \h Fig. 29 show classification accuracy and training time of stacked autoencoders of various network structures on PCA95% whitened data of MNIST. The 1-layer network achieves comparable results with that on raw data with nearly half of the training time. However, when the number of nodes in each layer is small, the networks with 2 or 3 layers capture less and less useful information from bottom to up due to too few nodes with sparsity regularization, so the classification accuracy is as low as like random guessing. As the number of nodes grows, the performance becomes better.Fig. SEQ Fig. \* ARABIC 28 Classification accuracy of stacked autoencoders of various net structures on PCA95 whitened of MNISTFig. SEQ Fig. \* ARABIC 29 Training time of stacked autoencoders of various net structures on PCA95 whitened of MNIST3.6 Experimental Results on Sleep Stage Dataset3.6.1 Data Pre-processingTo compare the performance of raw data and features of the sleep stage dataset, softmax regression is applied.3.6.1.1 On Raw DataWeight decay of the softmax regression is set to be 3e-3. The training on raw data takes 47 seconds and achieves 0.2056 classification accuracy. Softmax regression alone is too weak to capture useful information of the raw data. 3.6.1.2 On FeaturesTraining on 28 features takes only 2.5 seconds and results in 0.5678 classification accuracy. The hand crafted features are very useful for the softmax classifier.3.6.2 Deep Learning Network StructureClassifications on the raw data and features using different network structures in terms of number of layers and number of nodes and different learning units will be carried out to compare their performance.3.6.2.1 Stacked RBMs on Raw Data REF _Ref371875812 \h \* MERGEFORMAT Fig. 30 and REF _Ref371875814 \h \* MERGEFORMAT Fig. 31show classification accuracy and training time of stacked RBMs of various network structures on raw of the sleep stage dataset. Stacked RBMs capture the features well from the raw data and the classification performance is not sensitive to the network structures.Fig. SEQ Fig. \* ARABIC 30 Classification accuracy of stacked RBMs of various net structures on raw of sleep stage datasetFig. SEQ Fig. \* ARABIC 31Training time of stacked RBMs of various net structures on raw of sleep stage dataset3.6.2.2 Stacked RBMs on Features REF _Ref371877179 \h Fig. 32 and REF _Ref371877181 \h Fig. 33 show classification accuracy and training time of stacked RBMs of various network structures on features of the sleep stage dataset. The classification accuracy trained by features in a network with small number (e.g., 50) of nodes beats trained by raw data, suggesting deep learning fails to capture some of the useful information that a hand-crafted feature extractor can do. The training is fast, too, due to the low dimensionality of the feature space. However, the performance is sensitive to the number of nodes in each layer. Too large number with more network capacities tends to deteriorate the classification performance.Fig. SEQ Fig. \* ARABIC 32 Classification accuracy of stacked RBMs of various net structures on features of sleep stage datasetFig. SEQ Fig. \* ARABIC 33 Training time of stacked RBMs of various net structures on features of sleep stage dataset 3.6.2.3 Stacked Autoencoders on Raw DataThis experiment does not work due to an error in the training in which process matrix is close to singular or badly scaled.3.6.2.4 Stacked Autoencoders on Features REF _Ref371878427 \h Fig. 34 and REF _Ref371878429 \h Fig. 35 show classification accuracy and training time of stacked autoencoders of various network structures on features of the sleep stage dataset. The classification accuracy trained by features in a network of 1-layer beats trained by raw data. The training is fast, too, due to the low dimensionality of the feature space. However, the performance is sensitive to the number of layers. 2-layer and 3-layer networks do not work better than random guess.Fig. SEQ Fig. \* ARABIC 34 Classification accuracy of stacked autoencoders of various net structures on features of sleep stage datasetFig. SEQ Fig. \* ARABIC 35 Training time of stacked autoencoders of various net structures on features of sleep stage dataset3.7 SummaryFollowed is the summary of classification accuracy of various network structures of stacked RBMs and stacked autoencoders on both MNIST and the sleep stage datasets with or without pre-processing. The listed optimal net structures will be used in the later studies. In conclusion, the optimal net structures are highly application-dependent.Table SEQ Table \* ARABIC 3 Summary of network structures including optimal settings as well as pre-processing techniques on MINIST and sleep stage datasetDatasetPre-processingstacked RBMsstacked autoencodersMNISTrawGood in any net structures[500 500 500]Good, except when hidden nodes are too few[200 200]PCA95% whitenedSensitive to number of layers[200 200]Sensitive to number of nodes[300]sleep stage datasetrawGood in any net structures[200]NA28 featuressensitive to number of nodes[50]very sensitive to number of layers[50]Chapter 4Active Labeling in Deep Learning4.1 MotivationIn a semi-supervised learning paradigm such as deep learning, unlabeled data are easy to get at low or no extra cost, but labeled data are expensive. Due to limited resources, only very few labeled data can be obtained given a certain budget. For example, in a classification problem on physiological data from biosensors, the unlabeled data can be obtained by simply asking subjects to wear sensors day and night, but labeled data may not be available until human experts manually make annotations on selected unlabeled samples. Therefore, to make the best use of the budget for a discriminative learning task, it would be useful to propose an algorithm to carefully choose unlabeled samples to be labeled.Active learning (AL) ADDIN EN.CITE <EndNote><Cite><Author>Settles</Author><Year>2010</Year><RecNum>69</RecNum><DisplayText>[4]</DisplayText><record><rec-number>69</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">69</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Settles, Burr</author></authors></contributors><titles><title>Active learning literature survey</title><secondary-title>University of Wisconsin, Madison</secondary-title></titles><periodical><full-title>University of Wisconsin, Madison</full-title></periodical><dates><year>2010</year></dates><urls></urls><research-notes>active learning</research-notes></record></Cite></EndNote>[4] asks queries in the form of unlabeled instances to be labeled by an oracle (a human annotator). The goal is to achieve high classification accuracy using as few labeled samples as possible. A pool-based active learning queries the samples in the unlabeled pool that are most ambiguous for the current model. The newly labeled samples are added to the labeled pool and used to retrain the model. The two processes form a cycle as shown in REF _Ref355220649 \h Fig. 36. The problem to be solved in this chapter falls into the field of active learning.To the best of my knowledge, there are no active learning algorithms applied in DBNs except for Active Deep Networks (ADNs) proposed by ADDIN EN.CITE <EndNote><Cite><Author>Zhou</Author><Year>2010</Year><RecNum>71</RecNum><DisplayText>[39]</DisplayText><record><rec-number>71</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">71</key></foreign-keys><ref-type name="Conference Proceedings">10</ref-type><contributors><authors><author>Zhou, Shusen</author><author>Chen, Qingcai</author><author>Wang, Xiaolong</author></authors></contributors><titles><title>Active deep networks for semi-supervised sentiment classification</title><secondary-title>Proceedings of the 23rd International Conference on Computational Linguistics: Posters</secondary-title></titles><pages>1515-1523</pages><dates><year>2010</year></dates><publisher>Association for Computational Linguistics</publisher><urls></urls><research-notes>active learning and DBN</research-notes></record></Cite></EndNote>[39]. ADNs follow previous work on active learning for SVMs by defining the uncertainty of an unlabeled sample as its distance from the separating hyperplane and it can only work on binary classification problems.Since the top layer of both stacked RBMs and stacked autoencoders outputs the probabilities of each label to be chosen, it is promising to exploit the probabilities as indicators of uncertainty. My work aims to propose a few methods to effectively use the built-in classification uncertainty in deep learning networks to select unlabeled samples to be labeled.Fig. SEQ Fig. \* ARABIC 36 The pool-based active learning cycle (taken from ADDIN EN.CITE <EndNote><Cite><Author>Settles</Author><Year>2010</Year><RecNum>69</RecNum><DisplayText>[4]</DisplayText><record><rec-number>69</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">69</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Settles, Burr</author></authors></contributors><titles><title>Active learning literature survey</title><secondary-title>University of Wisconsin, Madison</secondary-title></titles><periodical><full-title>University of Wisconsin, Madison</full-title></periodical><dates><year>2010</year></dates><urls></urls><research-notes>active learning</research-notes></record></Cite></EndNote>[4])4.2 Related Works on Active LearningSuppose there is a large pool of unlabeled data available at cheap cost and we want to sample from them to get labeled data. A pool-based sampling ADDIN EN.CITE <EndNote><Cite><Author>Lewis</Author><Year>1994</Year><RecNum>116</RecNum><DisplayText>[40]</DisplayText><record><rec-number>116</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">116</key></foreign-keys><ref-type name="Conference Proceedings">10</ref-type><contributors><authors><author>Lewis, David D</author><author>Gale, William A</author></authors></contributors><titles><title>A sequential algorithm for training text classifiers</title><secondary-title>Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval</secondary-title></titles><pages>3-12</pages><dates><year>1994</year></dates><publisher>Springer-Verlag New York, Inc.</publisher><isbn>038719889X</isbn><urls></urls></record></Cite></EndNote>[40] was proposed to draw queries from the pool in a greedy fashion.Active learning is adopted for a range of base learners, such as support vector machines and Bayesian networks ADDIN EN.CITE <EndNote><Cite><Author>Tong</Author><Year>2001</Year><RecNum>128</RecNum><DisplayText>[41]</DisplayText><record><rec-number>128</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">128</key></foreign-keys><ref-type name="Thesis">32</ref-type><contributors><authors><author>Tong, Simon</author></authors></contributors><titles><title>Active learning: theory and applications</title></titles><dates><year>2001</year></dates><publisher>Citeseer</publisher><urls></urls></record></Cite></EndNote>[41], logistic regression ADDIN EN.CITE <EndNote><Cite><Author>Lewis</Author><Year>1994</Year><RecNum>116</RecNum><DisplayText>[40]</DisplayText><record><rec-number>116</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">116</key></foreign-keys><ref-type name="Conference Proceedings">10</ref-type><contributors><authors><author>Lewis, David D</author><author>Gale, William A</author></authors></contributors><titles><title>A sequential algorithm for training text classifiers</title><secondary-title>Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval</secondary-title></titles><pages>3-12</pages><dates><year>1994</year></dates><publisher>Springer-Verlag New York, Inc.</publisher><isbn>038719889X</isbn><urls></urls></record></Cite></EndNote>[40], and Markov Models ADDIN EN.CITE <EndNote><Cite><Author>Scheffer</Author><Year>2001</Year><RecNum>129</RecNum><DisplayText>[42]</DisplayText><record><rec-number>129</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">129</key></foreign-keys><ref-type name="Book Section">5</ref-type><contributors><authors><author>Scheffer, Tobias</author><author>Decomain, Christian</author><author>Wrobel, Stefan</author></authors></contributors><titles><title>Active hidden markov models for information extraction</title><secondary-title>Advances in Intelligent Data Analysis</secondary-title></titles><pages>309-318</pages><dates><year>2001</year></dates><publisher>Springer</publisher><isbn>3540425810</isbn><urls></urls></record></Cite></EndNote>[42].The pool-based active learning algorithm has been studied for many real-world problems, surveyed by ADDIN EN.CITE <EndNote><Cite><Author>Settles</Author><Year>2010</Year><RecNum>69</RecNum><DisplayText>[4]</DisplayText><record><rec-number>69</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">69</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Settles, Burr</author></authors></contributors><titles><title>Active learning literature survey</title><secondary-title>University of Wisconsin, Madison</secondary-title></titles><periodical><full-title>University of Wisconsin, Madison</full-title></periodical><dates><year>2010</year></dates><urls></urls><research-notes>active learning</research-notes></record></Cite></EndNote>[4], such as text classificationPEVuZE5vdGU+PENpdGU+PEF1dGhvcj5MZXdpczwvQXV0aG9yPjxZZWFyPjE5OTQ8L1llYXI+PFJl
Y051bT4xMTY8L1JlY051bT48RGlzcGxheVRleHQ+WzQwLCA0My00NV08L0Rpc3BsYXlUZXh0Pjxy
ZWNvcmQ+PHJlYy1udW1iZXI+MTE2PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+PGtleSBhcHA9
IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij4xMTY8L2tl
eT48L2ZvcmVpZ24ta2V5cz48cmVmLXR5cGUgbmFtZT0iQ29uZmVyZW5jZSBQcm9jZWVkaW5ncyI+
MTA8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5MZXdpcywgRGF2aWQg
RDwvYXV0aG9yPjxhdXRob3I+R2FsZSwgV2lsbGlhbSBBPC9hdXRob3I+PC9hdXRob3JzPjwvY29u
dHJpYnV0b3JzPjx0aXRsZXM+PHRpdGxlPkEgc2VxdWVudGlhbCBhbGdvcml0aG0gZm9yIHRyYWlu
aW5nIHRleHQgY2xhc3NpZmllcnM8L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+UHJvY2VlZGluZ3Mg
b2YgdGhlIDE3dGggYW5udWFsIGludGVybmF0aW9uYWwgQUNNIFNJR0lSIGNvbmZlcmVuY2Ugb24g
UmVzZWFyY2ggYW5kIGRldmVsb3BtZW50IGluIGluZm9ybWF0aW9uIHJldHJpZXZhbDwvc2Vjb25k
YXJ5LXRpdGxlPjwvdGl0bGVzPjxwYWdlcz4zLTEyPC9wYWdlcz48ZGF0ZXM+PHllYXI+MTk5NDwv
eWVhcj48L2RhdGVzPjxwdWJsaXNoZXI+U3ByaW5nZXItVmVybGFnIE5ldyBZb3JrLCBJbmMuPC9w
dWJsaXNoZXI+PGlzYm4+MDM4NzE5ODg5WDwvaXNibj48dXJscz48L3VybHM+PC9yZWNvcmQ+PC9D
aXRlPjxDaXRlPjxBdXRob3I+TWNDYWxsdW08L0F1dGhvcj48WWVhcj4xOTk4PC9ZZWFyPjxSZWNO
dW0+MTE4PC9SZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj4xMTg8L3JlYy1udW1iZXI+PGZvcmVp
Z24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVm
eHp0cmRmc3oiPjExODwva2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJDb25mZXJl
bmNlIFByb2NlZWRpbmdzIj4xMDwvcmVmLXR5cGU+PGNvbnRyaWJ1dG9ycz48YXV0aG9ycz48YXV0
aG9yPk1jQ2FsbHVtLCBBbmRyZXc8L2F1dGhvcj48YXV0aG9yPk5pZ2FtLCBLYW1hbDwvYXV0aG9y
PjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0bGVzPjx0aXRsZT5FbXBsb3lpbmcgRU0gYW5k
IFBvb2wtQmFzZWQgQWN0aXZlIExlYXJuaW5nIGZvciBUZXh0IENsYXNzaWZpY2F0aW9uPC90aXRs
ZT48c2Vjb25kYXJ5LXRpdGxlPklDTUw8L3NlY29uZGFyeS10aXRsZT48L3RpdGxlcz48cGFnZXM+
MzUwLTM1ODwvcGFnZXM+PHZvbHVtZT45ODwvdm9sdW1lPjxkYXRlcz48eWVhcj4xOTk4PC95ZWFy
PjwvZGF0ZXM+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPlRvbmc8
L0F1dGhvcj48WWVhcj4yMDAyPC9ZZWFyPjxSZWNOdW0+MTE5PC9SZWNOdW0+PHJlY29yZD48cmVj
LW51bWJlcj4xMTk8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlk
PSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjExOTwva2V5PjwvZm9yZWln
bi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9yZWYtdHlwZT48Y29u
dHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+VG9uZywgU2ltb248L2F1dGhvcj48YXV0aG9yPktv
bGxlciwgRGFwaG5lPC9hdXRob3I+PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRsZXM+PHRp
dGxlPlN1cHBvcnQgdmVjdG9yIG1hY2hpbmUgYWN0aXZlIGxlYXJuaW5nIHdpdGggYXBwbGljYXRp
b25zIHRvIHRleHQgY2xhc3NpZmljYXRpb248L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+VGhlIEpv
dXJuYWwgb2YgTWFjaGluZSBMZWFybmluZyBSZXNlYXJjaDwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0
bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPlRoZSBKb3VybmFsIG9mIE1hY2hpbmUgTGVhcm5p
bmcgUmVzZWFyY2g8L2Z1bGwtdGl0bGU+PC9wZXJpb2RpY2FsPjxwYWdlcz40NS02NjwvcGFnZXM+
PHZvbHVtZT4yPC92b2x1bWU+PGRhdGVzPjx5ZWFyPjIwMDI8L3llYXI+PC9kYXRlcz48aXNibj4x
NTMyLTQ0MzU8L2lzYm4+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9y
PkhvaTwvQXV0aG9yPjxZZWFyPjIwMDY8L1llYXI+PFJlY051bT4xMjA8L1JlY051bT48cmVjb3Jk
PjxyZWMtbnVtYmVyPjEyMDwvcmVjLW51bWJlcj48Zm9yZWlnbi1rZXlzPjxrZXkgYXBwPSJFTiIg
ZGItaWQ9InBhc3RkNTJ6cmFmZGF1ZWVyZXF2ZHBzOWVhZWZ4enRyZGZzeiI+MTIwPC9rZXk+PC9m
b3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkNvbmZlcmVuY2UgUHJvY2VlZGluZ3MiPjEwPC9y
ZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+SG9pLCBTdGV2ZW4gQ0g8L2F1
dGhvcj48YXV0aG9yPkppbiwgUm9uZzwvYXV0aG9yPjxhdXRob3I+THl1LCBNaWNoYWVsIFI8L2F1
dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRpdGxlcz48dGl0bGU+TGFyZ2Utc2NhbGUg
dGV4dCBjYXRlZ29yaXphdGlvbiBieSBiYXRjaCBtb2RlIGFjdGl2ZSBsZWFybmluZzwvdGl0bGU+
PHNlY29uZGFyeS10aXRsZT5Qcm9jZWVkaW5ncyBvZiB0aGUgMTV0aCBpbnRlcm5hdGlvbmFsIGNv
bmZlcmVuY2Ugb24gV29ybGQgV2lkZSBXZWI8L3NlY29uZGFyeS10aXRsZT48L3RpdGxlcz48cGFn
ZXM+NjMzLTY0MjwvcGFnZXM+PGRhdGVzPjx5ZWFyPjIwMDY8L3llYXI+PC9kYXRlcz48cHVibGlz
aGVyPkFDTTwvcHVibGlzaGVyPjxpc2JuPjE1OTU5MzMyMzk8L2lzYm4+PHVybHM+PC91cmxzPjwv
cmVjb3JkPjwvQ2l0ZT48L0VuZE5vdGU+
ADDIN EN.CITE PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5MZXdpczwvQXV0aG9yPjxZZWFyPjE5OTQ8L1llYXI+PFJl
Y051bT4xMTY8L1JlY051bT48RGlzcGxheVRleHQ+WzQwLCA0My00NV08L0Rpc3BsYXlUZXh0Pjxy
ZWNvcmQ+PHJlYy1udW1iZXI+MTE2PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+PGtleSBhcHA9
IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij4xMTY8L2tl
eT48L2ZvcmVpZ24ta2V5cz48cmVmLXR5cGUgbmFtZT0iQ29uZmVyZW5jZSBQcm9jZWVkaW5ncyI+
MTA8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5MZXdpcywgRGF2aWQg
RDwvYXV0aG9yPjxhdXRob3I+R2FsZSwgV2lsbGlhbSBBPC9hdXRob3I+PC9hdXRob3JzPjwvY29u
dHJpYnV0b3JzPjx0aXRsZXM+PHRpdGxlPkEgc2VxdWVudGlhbCBhbGdvcml0aG0gZm9yIHRyYWlu
aW5nIHRleHQgY2xhc3NpZmllcnM8L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+UHJvY2VlZGluZ3Mg
b2YgdGhlIDE3dGggYW5udWFsIGludGVybmF0aW9uYWwgQUNNIFNJR0lSIGNvbmZlcmVuY2Ugb24g
UmVzZWFyY2ggYW5kIGRldmVsb3BtZW50IGluIGluZm9ybWF0aW9uIHJldHJpZXZhbDwvc2Vjb25k
YXJ5LXRpdGxlPjwvdGl0bGVzPjxwYWdlcz4zLTEyPC9wYWdlcz48ZGF0ZXM+PHllYXI+MTk5NDwv
eWVhcj48L2RhdGVzPjxwdWJsaXNoZXI+U3ByaW5nZXItVmVybGFnIE5ldyBZb3JrLCBJbmMuPC9w
dWJsaXNoZXI+PGlzYm4+MDM4NzE5ODg5WDwvaXNibj48dXJscz48L3VybHM+PC9yZWNvcmQ+PC9D
aXRlPjxDaXRlPjxBdXRob3I+TWNDYWxsdW08L0F1dGhvcj48WWVhcj4xOTk4PC9ZZWFyPjxSZWNO
dW0+MTE4PC9SZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj4xMTg8L3JlYy1udW1iZXI+PGZvcmVp
Z24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVm
eHp0cmRmc3oiPjExODwva2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJDb25mZXJl
bmNlIFByb2NlZWRpbmdzIj4xMDwvcmVmLXR5cGU+PGNvbnRyaWJ1dG9ycz48YXV0aG9ycz48YXV0
aG9yPk1jQ2FsbHVtLCBBbmRyZXc8L2F1dGhvcj48YXV0aG9yPk5pZ2FtLCBLYW1hbDwvYXV0aG9y
PjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0bGVzPjx0aXRsZT5FbXBsb3lpbmcgRU0gYW5k
IFBvb2wtQmFzZWQgQWN0aXZlIExlYXJuaW5nIGZvciBUZXh0IENsYXNzaWZpY2F0aW9uPC90aXRs
ZT48c2Vjb25kYXJ5LXRpdGxlPklDTUw8L3NlY29uZGFyeS10aXRsZT48L3RpdGxlcz48cGFnZXM+
MzUwLTM1ODwvcGFnZXM+PHZvbHVtZT45ODwvdm9sdW1lPjxkYXRlcz48eWVhcj4xOTk4PC95ZWFy
PjwvZGF0ZXM+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPlRvbmc8
L0F1dGhvcj48WWVhcj4yMDAyPC9ZZWFyPjxSZWNOdW0+MTE5PC9SZWNOdW0+PHJlY29yZD48cmVj
LW51bWJlcj4xMTk8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlk
PSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjExOTwva2V5PjwvZm9yZWln
bi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9yZWYtdHlwZT48Y29u
dHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+VG9uZywgU2ltb248L2F1dGhvcj48YXV0aG9yPktv
bGxlciwgRGFwaG5lPC9hdXRob3I+PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRsZXM+PHRp
dGxlPlN1cHBvcnQgdmVjdG9yIG1hY2hpbmUgYWN0aXZlIGxlYXJuaW5nIHdpdGggYXBwbGljYXRp
b25zIHRvIHRleHQgY2xhc3NpZmljYXRpb248L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+VGhlIEpv
dXJuYWwgb2YgTWFjaGluZSBMZWFybmluZyBSZXNlYXJjaDwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0
bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPlRoZSBKb3VybmFsIG9mIE1hY2hpbmUgTGVhcm5p
bmcgUmVzZWFyY2g8L2Z1bGwtdGl0bGU+PC9wZXJpb2RpY2FsPjxwYWdlcz40NS02NjwvcGFnZXM+
PHZvbHVtZT4yPC92b2x1bWU+PGRhdGVzPjx5ZWFyPjIwMDI8L3llYXI+PC9kYXRlcz48aXNibj4x
NTMyLTQ0MzU8L2lzYm4+PHVybHM+PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9y
PkhvaTwvQXV0aG9yPjxZZWFyPjIwMDY8L1llYXI+PFJlY051bT4xMjA8L1JlY051bT48cmVjb3Jk
PjxyZWMtbnVtYmVyPjEyMDwvcmVjLW51bWJlcj48Zm9yZWlnbi1rZXlzPjxrZXkgYXBwPSJFTiIg
ZGItaWQ9InBhc3RkNTJ6cmFmZGF1ZWVyZXF2ZHBzOWVhZWZ4enRyZGZzeiI+MTIwPC9rZXk+PC9m
b3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkNvbmZlcmVuY2UgUHJvY2VlZGluZ3MiPjEwPC9y
ZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+SG9pLCBTdGV2ZW4gQ0g8L2F1
dGhvcj48YXV0aG9yPkppbiwgUm9uZzwvYXV0aG9yPjxhdXRob3I+THl1LCBNaWNoYWVsIFI8L2F1
dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRpdGxlcz48dGl0bGU+TGFyZ2Utc2NhbGUg
dGV4dCBjYXRlZ29yaXphdGlvbiBieSBiYXRjaCBtb2RlIGFjdGl2ZSBsZWFybmluZzwvdGl0bGU+
PHNlY29uZGFyeS10aXRsZT5Qcm9jZWVkaW5ncyBvZiB0aGUgMTV0aCBpbnRlcm5hdGlvbmFsIGNv
bmZlcmVuY2Ugb24gV29ybGQgV2lkZSBXZWI8L3NlY29uZGFyeS10aXRsZT48L3RpdGxlcz48cGFn
ZXM+NjMzLTY0MjwvcGFnZXM+PGRhdGVzPjx5ZWFyPjIwMDY8L3llYXI+PC9kYXRlcz48cHVibGlz
aGVyPkFDTTwvcHVibGlzaGVyPjxpc2JuPjE1OTU5MzMyMzk8L2lzYm4+PHVybHM+PC91cmxzPjwv
cmVjb3JkPjwvQ2l0ZT48L0VuZE5vdGU+
ADDIN EN.CITE.DATA [40, 43-45], information extraction ADDIN EN.CITE <EndNote><Cite><Author>Thompson</Author><Year>1999</Year><RecNum>121</RecNum><DisplayText>[46, 47]</DisplayText><record><rec-number>121</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">121</key></foreign-keys><ref-type name="Conference Proceedings">10</ref-type><contributors><authors><author>Thompson, Cynthia A</author><author>Califf, Mary Elaine</author><author>Mooney, Raymond J</author></authors></contributors><titles><title>Active learning for natural language parsing and information extraction</title><secondary-title>ICML</secondary-title></titles><pages>406-414</pages><dates><year>1999</year></dates><publisher>Citeseer</publisher><urls></urls></record></Cite><Cite><Author>Settles</Author><Year>2008</Year><RecNum>122</RecNum><record><rec-number>122</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">122</key></foreign-keys><ref-type name="Conference Proceedings">10</ref-type><contributors><authors><author>Settles, Burr</author><author>Craven, Mark</author></authors></contributors><titles><title>An analysis of active learning strategies for sequence labeling tasks</title><secondary-title>Proceedings of the Conference on Empirical Methods in Natural Language Processing</secondary-title></titles><pages>1070-1079</pages><dates><year>2008</year></dates><publisher>Association for Computational Linguistics</publisher><urls></urls></record></Cite></EndNote>[46, 47], image classification and retrieval ADDIN EN.CITE <EndNote><Cite><Author>Tong</Author><Year>2001</Year><RecNum>123</RecNum><DisplayText>[48, 49]</DisplayText><record><rec-number>123</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">123</key></foreign-keys><ref-type name="Conference Proceedings">10</ref-type><contributors><authors><author>Tong, Simon</author><author>Chang, Edward</author></authors></contributors><titles><title>Support vector machine active learning for image retrieval</title><secondary-title>Proceedings of the ninth ACM international conference on Multimedia</secondary-title></titles><pages>107-118</pages><dates><year>2001</year></dates><publisher>ACM</publisher><isbn>1581133944</isbn><urls></urls></record></Cite><Cite><Author>Zhang</Author><Year>2002</Year><RecNum>124</RecNum><record><rec-number>124</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">124</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Zhang, Cha</author><author>Chen, Tsuhan</author></authors></contributors><titles><title>An active learning framework for content-based information retrieval</title><secondary-title>Multimedia, IEEE Transactions on</secondary-title></titles><periodical><full-title>Multimedia, IEEE Transactions on</full-title></periodical><pages>260-268</pages><volume>4</volume><number>2</number><dates><year>2002</year></dates><isbn>1520-9210</isbn><urls></urls></record></Cite></EndNote>[48, 49], video classification and retrieval ADDIN EN.CITE <EndNote><Cite><Author>Yan</Author><Year>2003</Year><RecNum>125</RecNum><DisplayText>[50, 51]</DisplayText><record><rec-number>125</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">125</key></foreign-keys><ref-type name="Conference Proceedings">10</ref-type><contributors><authors><author>Yan, Rong</author><author>Yang, Jie</author><author>Hauptmann, Alexander</author></authors></contributors><titles><title>Automatically labeling video data using multi-class active learning</title><secondary-title>Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on</secondary-title></titles><pages>516-523</pages><dates><year>2003</year></dates><publisher>IEEE</publisher><isbn>0769519504</isbn><urls></urls></record></Cite><Cite><Author>Hauptmann</Author><Year>2006</Year><RecNum>126</RecNum><record><rec-number>126</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">126</key></foreign-keys><ref-type name="Conference Proceedings">10</ref-type><contributors><authors><author>Hauptmann, Alexander G</author><author>Lin, Wei-Hao</author><author>Yan, Rong</author><author>Yang, Jun</author><author>Chen, Ming-Yu</author></authors></contributors><titles><title>Extreme video retrieval: joint maximization of human and computer performance</title><secondary-title>Proceedings of the 14th annual ACM international conference on Multimedia</secondary-title></titles><pages>385-394</pages><dates><year>2006</year></dates><publisher>ACM</publisher><isbn>1595934472</isbn><urls></urls></record></Cite></EndNote>[50, 51], and speech recognition ADDIN EN.CITE <EndNote><Cite><Author>Tur</Author><Year>2005</Year><RecNum>127</RecNum><DisplayText>[52]</DisplayText><record><rec-number>127</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">127</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Tur, Gokhan</author><author>Hakkani-Tür, Dilek</author><author>Schapire, Robert E</author></authors></contributors><titles><title>Combining active and semi-supervised learning for spoken language understanding</title><secondary-title>Speech Communication</secondary-title></titles><periodical><full-title>Speech Communication</full-title></periodical><pages>171-186</pages><volume>45</volume><number>2</number><dates><year>2005</year></dates><isbn>0167-6393</isbn><urls></urls></record></Cite></EndNote>[52].4.3 AlgorithmsThe problem to be solved is formulized as follows. Given an unlabeled sample set XU and a labeled sample set XL, the algorithm needs to take B samples from XU, have them labeled, and add them to XL, in order to minimize the classification error of a deep learning model fine-tuned by XL, where B is a constant as the budget in each iteration.The basic idea of my algorithm is simple and not much different from other active learning algorithms – greedily select those samples that are most difficult to classify. The framework is depicted in REF _Ref355229204 \h Fig. 37. Then the problem remains how to find heuristic methods to define difficulty, uncertainty, or ambiguity. Three criteria are proposed under the active labeling deep learning (AL-DL) framework.AL-DL frameworkInput: an unlabeled sample set XU1. Use all the unlabeled data XU to train a DL network (stacked RBMs or stacked autoencoders) layer by layer. The weights and biases will be used as the initial values for fine-tuning a DL network in the following steps.2. Randomly take B0 samples from XU, have them labeled, and add them to the empty set XL as a “seed” set. For simplicity, assume the classes are balanced in the set XL. 3. Use XL to fine-tune a N-layer DL network classifier ModelN.4. Use the classifier ModelN to classify all the unlabeled samples in XU. Take B the most uncertain samples from XU, have them labeled, and add them to XL. (see the three detailed criteria to measure uncertainty in text). Go to step 3.Output: a N-layer DL network classifier ModelNFig. SEQ Fig. \* ARABIC 37 Active labeling – deep learning frameworkThe classification in step 4 applies REF NumberRef5334240198 \h \* MERGEFORMAT (6) iteratively starting from an unlabeled sample fed into the lowest input layer until the top layer N and then uses REF NumberRef5795186162 \h \* MERGEFORMAT (16) to softmax the activations in the top layer. If j is predicted as the class of the sample xi is determined by if the unit the class associated with has the largest probability among all the units phjN|xi in the top layer as belowyj=1, if j=argmaxj?(phjN|xi) 0, otherwise MACROBUTTON NumberReference \* MERGEFORMAT (24) Since the true class of the sample xi is unknown yet, the predicted label yj is not much interesting because we do not know it is correct or misclassified. However, phjN|xi suggests the confidence of the prediction. Least confidence (LC), margin sampling (MS) ADDIN EN.CITE <EndNote><Cite><Author>Roy</Author><Year>2001</Year><RecNum>77</RecNum><DisplayText>[53]</DisplayText><record><rec-number>77</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">77</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Roy, Nicholas</author><author>McCallum, Andrew</author></authors></contributors><titles><title>Toward optimal active learning through monte carlo estimation of error reduction</title><secondary-title>ICML, Williamstown</secondary-title></titles><periodical><full-title>ICML, Williamstown</full-title></periodical><dates><year>2001</year></dates><urls></urls></record></Cite></EndNote>[53], and entropy ADDIN EN.CITE <EndNote><Cite><Author>Shannon</Author><Year>1948</Year><RecNum>76</RecNum><DisplayText>[54]</DisplayText><record><rec-number>76</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">76</key></foreign-keys><ref-type name="Generic">13</ref-type><contributors><authors><author>Shannon, Claude Elwood</author><author>Weaver, Warren</author></authors></contributors><titles><title>A mathematical theory of communication</title></titles><dates><year>1948</year></dates><publisher>American Telephone and Telegraph Company</publisher><urls></urls></record></Cite></EndNote>[54] can act as the criteria to pick the most uncertain samples xi.Active labeling deep learning with least confidence (AL-DL-LC) picks the samples with the minimum of the maximum of activations as followsxiLC=argminximaxj?(phjN|xi) MACROBUTTON NumberReference \* MERGEFORMAT (25)where xi is an input vector, hjN is the activation of the unit j in the top layer. If more than one sample needs to be selected, the process could be performed in a greedy fashion.Least confidence works in the assumption that if the probability of the most probable label for a sample is low then the classification of the sample is uncertain. Thus it discards the information about the remaining label distributions. Active labeling deep learning with margin sampling (AL-DL-MS) partially corrects the shortcoming by incorporating the posterior of the second most likely label as followsxiMS=argminxi(py1xi-py2xi) MACROBUTTON NumberReference \* MERGEFORMAT (26)where y1 and y2 are the first and second most probable class labels under the model. Intuitively if the probabilities of predicting a sample to its most likely class and to its second most likely class are too close, the classifier is quite confused about the sample. Therefore some information is needed from the oracle to help the classifier discriminate these two classes. Active labeling deep learning with entropy (AL-DL-Entropy) aims to take all labels probabilities into consideration byxiEntropy=argmaxxi-jphjN|xilogphjN|xi MACROBUTTON NumberReference \* MERGEFORMAT (27)Entropy represents the amount of information or least number of bits to encode a distribution. It could act as a good measure of uncertainty.4.4 Experimental Results on MNISTTo evaluate the performance of AL-DL in three criteria, a few experiments on the MNIST dataset have been performed. For comparison, a random selection strategy is employed as the baseline. 4.4.1 Stacked RBMs on Raw DataThe hidden structure of DBN for pre-training is 500-500-500, with 50,000 for training and 10,000 samples for testing. A base set with 1,000 labeled samples (80% for training and 20% for validation) acts as the seed for both the random and active labeling schemes. These samples are balanced among all the classes.In each iteration the random scheme randomly picks 200 samples (80% for training and 20% for validation) and adds them to the training set and validation set. These two sets are used for the original classifier to train the model and test on the test set to get the classification accuracy.The AL scheme uses the model trained in the previous iteration to classify unlabeled samples. During the process the 200 (80% for training and 20% for validation) most uncertain samples are labeled by the oracle and added into the training set and validation set used for the new classifier. Then the AL scheme moves to the next iteration. Note the samples picked in this step are not necessarily balanced.Other DBN parameters are listed below.Table SEQ Table \* ARABIC 4 DBN parameters for both random and AL schemesUnsupervised pre-training stagelearning rate 0.05number of epochs100minibatch size100momentum0.5 for the first 5 epochs, 0.9 thereafter weight cost0.0002Supervised fine-tuning stagelearning rate 0.05number of epochs50minibatch size100number of initial epochs510 trails are done for the random scheme and AL schemes with least confidence, margin sampling, and entropy.The mean and standard deviation of the accuracy of these four methods are depicted below.Fig. SEQ Fig. \* ARABIC 38 The mean of classification accuracy of random labeling DBN and active labeling DBN. RL and AL-LC show their error bars. Iteration 0 means only the seed set is used. The performances of the three AL DBNs are close, all better than that of RL DBN.The means of the accuracy of the three methods AL-LC DBN, AL-MS DBN, and AL –Entropy DBN are close, which all outperform the mean accuracy of RL DBN. The standard deviations of the three active labeling schemes are smaller than that of RL DBN, suggesting more consistent performance of AL DBN.To confirm the active labeling does pick the most challenging samples, all the 200 samples picked before the 1st iteration by RL DBN (top) and AL –LC DBN (bottom) are shown below.Fig. SEQ Fig. \* ARABIC 39 All the 200 samples picked before the 1st iteration by RL DBN (top) and AL-LC DBN (bottom). The digits in the bottom graph look harder to recognize, indicating AL-LC DBN does pick more challenging samples than RL DBNFig. SEQ Fig. \* ARABIC 40 The comparison of classification error rates on training and validation sets with RL-DBN and AL-DBN-LC in one iteration. AL-DBN-LC works much worse on training and validation sets because it always picks the most challenging samples. REF _Ref355709303 \h Fig. 40 shows another interesting observation. In each iteration, the RMSE and accuracy using the active labeling strategy on both training set and validation set are much worse than using the random labeling method, but active labeling method’s performance on the test set is better. This is because active labeling always puts the most challenging samples to the training set and validation set, it’s understandable that the classification accuracy on them is not as high as on randomly picked samples. However, since active labeling always learns from those challenging cases, it should actually outperform the random labeling method that learns from less informative samples. 4.4.2 Stacked Autoencoders on Raw DataA similar experiment is carried out using stacked autoencoders of 2 hidden layers, each having 200 nodes. REF _Ref371899423 \h Fig. 41 shows active labeling stacked autoencoders beats its random labeling counterpart by about 2% in iteration 5, in which AL-MS works the best.Fig. SEQ Fig. \* ARABIC 41 The mean of classification accuracy of random labeling stacked autoencoders and active labeling stacked autoencoders. RL and AL-LC show their error bars. Iteration 0 means only the seed set is used.4.4.3 Stacked Autoencoders on PCA at Retention Rate 95%The experimental result using stacked autoencoders of 1 hidden layer, each having 300 nodes, on PCA 95% whitened data of MNIST, is shown in REF _Ref371899716 \h Fig. 42. Active labeling stacked autoencoders outperforms the random labeling version by up to 4% in iteration 5, in which AL-MS works the best.Fig. SEQ Fig. \* ARABIC 42 The mean of classification accuracy of random labeling stacked autoencoders and active labeling stacked autoencoders. RL and AL-LC show their error bars. Iteration 0 means only the seed set is used.4.5 Experimental results on sleep stage datasetThe same experiment is carried out on the sleep stage dataset using the optimal settings to evaluate the performance of active labeling deep learning in a complex and noisy dataset. A base set of 1000 labeled samples acts as the seed for both the random and active labeling schemes. These samples are balanced among all 5 classes. There will be 10 (for stacked RBMs) or 5 (for stacked autoencoders) iterations, in each of which 200 samples are picked, labeled, and added to the training set, by random or active labeling. 10 trials are done for the random scheme and AL schemes with least confidence, margin sampling, and entropy.4.5.1 Stacked RBMs on Raw DataA 1-layer with 200 hidden nodes stacked RBMs is trained on raw data of the sleep stage dataset with random labeling and 3 different active labeling schemes. Their classification accuracies do not show significant difference in REF _Ref371904325 \h Fig. 43. Fig. SEQ Fig. \* ARABIC 43 The mean of classification accuracy of random labeling stacked RBMs and active labeling stacked RBMs. RL and AL-LC show their error bars. Iteration 0 means only the seed set is used.4.5.2 Stacked RBMs on FeaturesA 1-layer with 50 hidden nodes stacked RBMs is trained on features of the sleep stage dataset with random labeling and 3 different active labeling schemes. No significant difference is found in them according to REF _Ref371904325 \h Fig. 43. Fig. SEQ Fig. \* ARABIC 44 The mean of classification accuracy of random labeling stacked RBMs and active labeling stacked RBMs. RL and AL-LC show their error bars. Iteration 0 means only the seed set is used.4.5.3 Stacked Autoencoders on FeaturesThe similar experiment is carried out on a 1-layer with 50 nodes stacked autoencoders on features of the sleep stage dataset. REF _Ref371904524 \h Fig. 45 suggests all 4 algorithms perform almost the same.Fig. SEQ Fig. \* ARABIC 45 The mean of classification accuracy of random labeling stacked autoencoders and active labeling stacked autoencoders. RL and AL-LC show their error bars. Iteration 0 means only the seed set is used.4.6 SummaryOn MNIST, active labeling deep learning works better than its random counterpart, no matter what uncertainty measurements (least confidence, marginal sampling, or entropy), what learning units (RBMs or autoencoders), or what data pre-processing techniques (raw or PCA processed data) are used. However, active labeling strategy does not outperform random labeling deep learning on the sleep stage dataset. The noise nature of the data seems to be blamed on. Since active labeling proactively picks the most uncertain samples to be labels, these samples are more likely to be mislabeled, introducing false information to the model.Chapter 5 Modeling Raw Physiological Data and Predicting Emotions with Deep Belief Networks5.1 IntroductionInspired by the relationship between emotional states and physiological signals ADDIN EN.CITE <EndNote><Cite><Author>Cornelius</Author><Year>1996</Year><RecNum>59</RecNum><DisplayText>[55, 56]</DisplayText><record><rec-number>59</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">59</key></foreign-keys><ref-type name="Book">6</ref-type><contributors><authors><author>Cornelius, Randolph R</author></authors></contributors><titles><title>The science of emotion: Research and tradition in the psychology of emotions</title></titles><volume>133001539</volume><dates><year>1996</year></dates><publisher>Prentice Hall Upper Saddle River, NJ</publisher><urls></urls></record></Cite><Cite><Author>Sander</Author><Year>2005</Year><RecNum>60</RecNum><record><rec-number>60</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">60</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Sander, David</author><author>Grandjean, Didier</author><author>Scherer, Klaus R</author></authors></contributors><titles><title>2005 Special Issue: A systems approach to appraisal mechanisms in emotion</title><secondary-title>Neural networks</secondary-title></titles><periodical><full-title>Neural networks</full-title></periodical><pages>317-352</pages><volume>18</volume><number>4</number><dates><year>2005</year></dates><isbn>0893-6080</isbn><urls></urls></record></Cite></EndNote>[55, 56], researchers have developed a large amount of methods to predict emotions based on physiological data PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5Ob3ZhazwvQXV0aG9yPjxZZWFyPjIwMTI8L1llYXI+PFJl
Y051bT42PC9SZWNOdW0+PERpc3BsYXlUZXh0Pls1Ny02NV08L0Rpc3BsYXlUZXh0PjxyZWNvcmQ+
PHJlYy1udW1iZXI+NjwvcmVjLW51bWJlcj48Zm9yZWlnbi1rZXlzPjxrZXkgYXBwPSJFTiIgZGIt
aWQ9InBhc3RkNTJ6cmFmZGF1ZWVyZXF2ZHBzOWVhZWZ4enRyZGZzeiI+Njwva2V5PjwvZm9yZWln
bi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9yZWYtdHlwZT48Y29u
dHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+Tm92YWssIERvbWVuPC9hdXRob3I+PGF1dGhvcj5N
aWhlbGosIE1hdGphxb48L2F1dGhvcj48YXV0aG9yPk11bmloLCBNYXJrbzwvYXV0aG9yPjwvYXV0
aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0bGVzPjx0aXRsZT5BIHN1cnZleSBvZiBtZXRob2RzIGZv
ciBkYXRhIGZ1c2lvbiBhbmQgc3lzdGVtIGFkYXB0YXRpb24gdXNpbmcgYXV0b25vbWljIG5lcnZv
dXMgc3lzdGVtIHJlc3BvbnNlcyBpbiBwaHlzaW9sb2dpY2FsIGNvbXB1dGluZzwvdGl0bGU+PHNl
Y29uZGFyeS10aXRsZT5JbnRlcmFjdGluZyB3aXRoIENvbXB1dGVyczwvc2Vjb25kYXJ5LXRpdGxl
PjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPkludGVyYWN0aW5nIHdpdGggQ29tcHV0
ZXJzPC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+MTU0LTE3MjwvcGFnZXM+PHZvbHVt
ZT4yNDwvdm9sdW1lPjxudW1iZXI+MzwvbnVtYmVyPjxrZXl3b3Jkcz48a2V5d29yZD5BZmZlY3Rp
dmUgY29tcHV0aW5nPC9rZXl3b3JkPjxrZXl3b3JkPlBoeXNpb2xvZ2ljYWwgY29tcHV0aW5nPC9r
ZXl3b3JkPjxrZXl3b3JkPlBzeWNob3BoeXNpb2xvZ3k8L2tleXdvcmQ+PGtleXdvcmQ+RGF0YSBm
dXNpb248L2tleXdvcmQ+PGtleXdvcmQ+QXV0b25vbWljIG5lcnZvdXMgc3lzdGVtPC9rZXl3b3Jk
Pjwva2V5d29yZHM+PGRhdGVzPjx5ZWFyPjIwMTI8L3llYXI+PC9kYXRlcz48aXNibj4wOTUzLTU0
Mzg8L2lzYm4+PHVybHM+PHJlbGF0ZWQtdXJscz48dXJsPmh0dHA6Ly93d3cuc2NpZW5jZWRpcmVj
dC5jb20vc2NpZW5jZS9hcnRpY2xlL3BpaS9TMDk1MzU0MzgxMjAwMDM3OTwvdXJsPjwvcmVsYXRl
ZC11cmxzPjwvdXJscz48ZWxlY3Ryb25pYy1yZXNvdXJjZS1udW0+MTAuMTAxNi9qLmludGNvbS4y
MDEyLjA0LjAwMzwvZWxlY3Ryb25pYy1yZXNvdXJjZS1udW0+PHJlc2VhcmNoLW5vdGVzPmR3MDE8
L3Jlc2VhcmNoLW5vdGVzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPk1jRHVmZjwvQXV0
aG9yPjxZZWFyPjIwMTI8L1llYXI+PFJlY051bT4xNTwvUmVjTnVtPjxyZWNvcmQ+PHJlYy1udW1i
ZXI+MTU8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0
ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjE1PC9rZXk+PC9mb3JlaWduLWtleXM+
PHJlZi10eXBlIG5hbWU9IkNvbmZlcmVuY2UgUHJvY2VlZGluZ3MiPjEwPC9yZWYtdHlwZT48Y29u
dHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+TWNEdWZmLCBELjwvYXV0aG9yPjxhdXRob3I+S2Fy
bHNvbiwgQS48L2F1dGhvcj48YXV0aG9yPkthcG9vciwgQS48L2F1dGhvcj48YXV0aG9yPlJvc2V3
YXksIEEuPC9hdXRob3I+PGF1dGhvcj5DemVyd2luc2tpLCBNLjwvYXV0aG9yPjwvYXV0aG9ycz48
L2NvbnRyaWJ1dG9ycz48dGl0bGVzPjx0aXRsZT5BZmZlY3RBdXJhOiBhbiBpbnRlbGxpZ2VudCBz
eXN0ZW0gZm9yIGVtb3Rpb25hbCBtZW1vcnk8L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+VGhlIEFD
TSBTSUdDSEkgQ29uZmVyZW5jZSBvbiBIdW1hbiBGYWN0b3JzIGluIENvbXB1dGluZyBTeXN0ZW1z
PC9zZWNvbmRhcnktdGl0bGU+PC90aXRsZXM+PHBhZ2VzPjg0OS04NTg8L3BhZ2VzPjxkYXRlcz48
eWVhcj4yMDEyPC95ZWFyPjwvZGF0ZXM+PHB1Yi1sb2NhdGlvbj5BdXN0aW4sIFRYPC9wdWItbG9j
YXRpb24+PHB1Ymxpc2hlcj5BQ008L3B1Ymxpc2hlcj48aXNibj4xNDUwMzEwMTVYPC9pc2JuPjx1
cmxzPjwvdXJscz48cmVzZWFyY2gtbm90ZXM+ZHcwMiYjeEQ7QWZmZWN0aXZhIFEgc2Vuc29yPC9y
ZXNlYXJjaC1ub3Rlcz48L3JlY29yZD48L0NpdGU+PENpdGU+PEF1dGhvcj5Cb3llcjwvQXV0aG9y
PjxZZWFyPjIwMTI8L1llYXI+PFJlY051bT4zPC9SZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj4z
PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+PGtleSBhcHA9IkVOIiBkYi1pZD0icGFzdGQ1Mnpy
YWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij4zPC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10
eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1
dGhvcnM+PGF1dGhvcj5Cb3llciwgRS5XLjwvYXV0aG9yPjxhdXRob3I+RmxldGNoZXIsIFIuPC9h
dXRob3I+PGF1dGhvcj5GYXksIFIuSi48L2F1dGhvcj48YXV0aG9yPlNtZWxzb24sIEQuPC9hdXRo
b3I+PGF1dGhvcj5aaWVkb25pcywgRC48L2F1dGhvcj48YXV0aG9yPlBpY2FyZCwgUi5XLjwvYXV0
aG9yPjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0bGVzPjx0aXRsZT5QcmVsaW1pbmFyeSBF
ZmZvcnRzIERpcmVjdGVkIFRvd2FyZCB0aGUgRGV0ZWN0aW9uIG9mIENyYXZpbmcgb2YgSWxsaWNp
dCBTdWJzdGFuY2VzOiBUaGUgaUhlYWwgUHJvamVjdDwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5K
b3VybmFsIG9mIE1lZGljYWwgVG94aWNvbG9neTwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVzPjxw
ZXJpb2RpY2FsPjxmdWxsLXRpdGxlPkpvdXJuYWwgb2YgTWVkaWNhbCBUb3hpY29sb2d5PC9mdWxs
LXRpdGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+MS01PC9wYWdlcz48ZGF0ZXM+PHllYXI+MjAxMjwv
eWVhcj48L2RhdGVzPjxpc2JuPjE1NTYtOTAzOTwvaXNibj48dXJscz48L3VybHM+PC9yZWNvcmQ+
PC9DaXRlPjxDaXRlPjxBdXRob3I+UGxhcnJlPC9BdXRob3I+PFllYXI+MjAxMTwvWWVhcj48UmVj
TnVtPjI8L1JlY051bT48cmVjb3JkPjxyZWMtbnVtYmVyPjI8L3JlYy1udW1iZXI+PGZvcmVpZ24t
a2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0
cmRmc3oiPjI8L2tleT48L2ZvcmVpZ24ta2V5cz48cmVmLXR5cGUgbmFtZT0iQ29uZmVyZW5jZSBQ
cm9jZWVkaW5ncyI+MTA8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5Q
bGFycmUsIEsuPC9hdXRob3I+PGF1dGhvcj5SYWlqLCBBLjwvYXV0aG9yPjxhdXRob3I+SG9zc2Fp
biwgUy5NLjwvYXV0aG9yPjxhdXRob3I+QWxpLCBBLkEuPC9hdXRob3I+PGF1dGhvcj5OYWthamlt
YSwgTS48L2F1dGhvcj48YXV0aG9yPkFsJmFwb3M7YWJzaSwgTS48L2F1dGhvcj48YXV0aG9yPkVy
dGluLCBFLjwvYXV0aG9yPjxhdXRob3I+S2FtYXJjaywgVC48L2F1dGhvcj48YXV0aG9yPkt1bWFy
LCBTLjwvYXV0aG9yPjxhdXRob3I+U2NvdHQsIE0uPC9hdXRob3I+PC9hdXRob3JzPjwvY29udHJp
YnV0b3JzPjx0aXRsZXM+PHRpdGxlPkNvbnRpbnVvdXMgaW5mZXJlbmNlIG9mIHBzeWNob2xvZ2lj
YWwgc3RyZXNzIGZyb20gc2Vuc29yeSBtZWFzdXJlbWVudHMgY29sbGVjdGVkIGluIHRoZSBuYXR1
cmFsIGVudmlyb25tZW50PC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPkluZm9ybWF0aW9uIFByb2Nl
c3NpbmcgaW4gU2Vuc29yIE5ldHdvcmtzIChJUFNOKSwgMjAxMSAxMHRoIEludGVybmF0aW9uYWwg
Q29uZmVyZW5jZSBvbjwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVzPjxwYWdlcz45Ny0xMDg8L3Bh
Z2VzPjxkYXRlcz48eWVhcj4yMDExPC95ZWFyPjwvZGF0ZXM+PHB1Yi1sb2NhdGlvbj5DaGljYWdv
LCBJTCwgVVNBPC9wdWItbG9jYXRpb24+PHB1Ymxpc2hlcj5JRUVFPC9wdWJsaXNoZXI+PGlzYm4+
MTYxMjg0ODU0MDwvaXNibj48dXJscz48L3VybHM+PC9yZWNvcmQ+PC9DaXRlPjxDaXRlPjxBdXRo
b3I+S3JlaWJpZzwvQXV0aG9yPjxZZWFyPjIwMTA8L1llYXI+PFJlY051bT4xMTwvUmVjTnVtPjxy
ZWNvcmQ+PHJlYy1udW1iZXI+MTE8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0i
RU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjExPC9rZXk+
PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10
eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5LcmVpYmlnLCBTeWx2aWEgRC48L2F1
dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRpdGxlcz48dGl0bGU+QXV0b25vbWljIG5l
cnZvdXMgc3lzdGVtIGFjdGl2aXR5IGluIGVtb3Rpb246IEEgcmV2aWV3PC90aXRsZT48c2Vjb25k
YXJ5LXRpdGxlPkJpb2xvZ2ljYWwgUHN5Y2hvbG9neTwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVz
PjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPkJpb2xvZ2ljYWwgUHN5Y2hvbG9neTwvZnVsbC10aXRs
ZT48L3BlcmlvZGljYWw+PHBhZ2VzPjM5NC00MjE8L3BhZ2VzPjx2b2x1bWU+ODQ8L3ZvbHVtZT48
bnVtYmVyPjM8L251bWJlcj48a2V5d29yZHM+PGtleXdvcmQ+RW1vdGlvbjwva2V5d29yZD48a2V5
d29yZD5BdXRvbm9taWMgbmVydm91cyBzeXN0ZW08L2tleXdvcmQ+PGtleXdvcmQ+RW1vdGlvbmFs
IHJlc3BvbnNlIHNwZWNpZmljaXR5PC9rZXl3b3JkPjxrZXl3b3JkPkF1dG9ub21pYyByZXNwb25z
ZSBvcmdhbml6YXRpb248L2tleXdvcmQ+PGtleXdvcmQ+Q2FyZGlvdmFzY3VsYXIgc3lzdGVtPC9r
ZXl3b3JkPjxrZXl3b3JkPlJlc3BpcmF0b3J5IHN5c3RlbTwva2V5d29yZD48a2V5d29yZD5FbGVj
dHJvZGVybWFsIHN5c3RlbTwva2V5d29yZD48L2tleXdvcmRzPjxkYXRlcz48eWVhcj4yMDEwPC95
ZWFyPjwvZGF0ZXM+PGlzYm4+MDMwMS0wNTExPC9pc2JuPjx1cmxzPjxyZWxhdGVkLXVybHM+PHVy
bD5odHRwOi8vd3d3LnNjaWVuY2VkaXJlY3QuY29tL3NjaWVuY2UvYXJ0aWNsZS9waWkvUzAzMDEw
NTExMTAwMDA4Mjc8L3VybD48L3JlbGF0ZWQtdXJscz48L3VybHM+PGVsZWN0cm9uaWMtcmVzb3Vy
Y2UtbnVtPjEwLjEwMTYvai5iaW9wc3ljaG8uMjAxMC4wMy4wMTA8L2VsZWN0cm9uaWMtcmVzb3Vy
Y2UtbnVtPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPldhZ25lcjwvQXV0aG9yPjxZZWFy
PjIwMDk8L1llYXI+PFJlY051bT40PC9SZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj40PC9yZWMt
bnVtYmVyPjxmb3JlaWduLWtleXM+PGtleSBhcHA9IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVl
ZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij40PC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5h
bWU9IkNvbmZlcmVuY2UgUHJvY2VlZGluZ3MiPjEwPC9yZWYtdHlwZT48Y29udHJpYnV0b3JzPjxh
dXRob3JzPjxhdXRob3I+V2FnbmVyLCBKLjwvYXV0aG9yPjxhdXRob3I+QW5kcmUsIEUuPC9hdXRo
b3I+PGF1dGhvcj5KdW5nLCBGLjwvYXV0aG9yPjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0
bGVzPjx0aXRsZT5TbWFydCBzZW5zb3IgaW50ZWdyYXRpb246IEEgZnJhbWV3b3JrIGZvciBtdWx0
aW1vZGFsIGVtb3Rpb24gcmVjb2duaXRpb24gaW4gcmVhbC10aW1lPC90aXRsZT48c2Vjb25kYXJ5
LXRpdGxlPkFmZmVjdGl2ZSBDb21wdXRpbmcgYW5kIEludGVsbGlnZW50IEludGVyYWN0aW9uIGFu
ZCBXb3Jrc2hvcHMsIDIwMDkuIEFDSUkgMjAwOS4gM3JkIEludGVybmF0aW9uYWwgQ29uZmVyZW5j
ZSBvbjwvc2Vjb25kYXJ5LXRpdGxlPjxhbHQtdGl0bGU+QWZmZWN0aXZlIENvbXB1dGluZyBhbmQg
SW50ZWxsaWdlbnQgSW50ZXJhY3Rpb24gYW5kIFdvcmtzaG9wcywgMjAwOS4gQUNJSSAyMDA5LiAz
cmQgSW50ZXJuYXRpb25hbCBDb25mZXJlbmNlIG9uPC9hbHQtdGl0bGU+PC90aXRsZXM+PHBhZ2Vz
PjEtODwvcGFnZXM+PGtleXdvcmRzPjxrZXl3b3JkPmVtb3Rpb24gcmVjb2duaXRpb248L2tleXdv
cmQ+PGtleXdvcmQ+aHVtYW4gY29tcHV0ZXIgaW50ZXJhY3Rpb248L2tleXdvcmQ+PGtleXdvcmQ+
c2Vuc29yIGZ1c2lvbjwva2V5d29yZD48a2V5d29yZD5kYXRhIHNlZ21lbnRhdGlvbjwva2V5d29y
ZD48a2V5d29yZD5mZWF0dXJlIGV4dHJhY3Rpb248L2tleXdvcmQ+PGtleXdvcmQ+bXVsdGltb2Rh
bCBlbW90aW9uIHJlY29nbml0aW9uPC9rZXl3b3JkPjxrZXl3b3JkPm9ubGluZSBlbW90aW9uIHJl
Y29nbml0aW9uIHN5c3RlbTwva2V5d29yZD48a2V5d29yZD5wYXR0ZXJuIHJlY29nbml0aW9uPC9r
ZXl3b3JkPjxrZXl3b3JkPnNtYXJ0IHNlbnNvciBpbnRlZ3JhdGlvbjwva2V5d29yZD48L2tleXdv
cmRzPjxkYXRlcz48eWVhcj4yMDA5PC95ZWFyPjxwdWItZGF0ZXM+PGRhdGU+MTAtMTIgU2VwdC4g
MjAwOTwvZGF0ZT48L3B1Yi1kYXRlcz48L2RhdGVzPjx1cmxzPjwvdXJscz48ZWxlY3Ryb25pYy1y
ZXNvdXJjZS1udW0+MTAuMTEwOS9hY2lpLjIwMDkuNTM0OTU3MTwvZWxlY3Ryb25pYy1yZXNvdXJj
ZS1udW0+PC9yZWNvcmQ+PC9DaXRlPjxDaXRlPjxBdXRob3I+TWlpbGxlcjwvQXV0aG9yPjxZZWFy
PjIwMDY8L1llYXI+PFJlY051bT4xNDwvUmVjTnVtPjxyZWNvcmQ+PHJlYy1udW1iZXI+MTQ8L3Jl
Yy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRh
dWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjE0PC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10eXBl
IG5hbWU9IkNvbmZlcmVuY2UgUHJvY2VlZGluZ3MiPjEwPC9yZWYtdHlwZT48Y29udHJpYnV0b3Jz
PjxhdXRob3JzPjxhdXRob3I+TWlpbGxlciwgTS4gRS48L2F1dGhvcj48L2F1dGhvcnM+PC9jb250
cmlidXRvcnM+PHRpdGxlcz48dGl0bGU+V2h5IFNvbWUgRW1vdGlvbmFsIFN0YXRlcyBBcmUgRWFz
aWVyIHRvIGJlIFJlY29nbml6ZWQgVGhhbiBPdGhlcnM6IEEgdGhvcm91Z2ggZGF0YSBhbmFseXNp
cyBhbmQgYSB2ZXJ5IGFjY3VyYXRlIHJvdWdoIHNldCBjbGFzc2lmaWVyPC90aXRsZT48c2Vjb25k
YXJ5LXRpdGxlPlN5c3RlbXMsIE1hbiBhbmQgQ3liZXJuZXRpY3MsIDIwMDYuIFNNQyAmYXBvczsw
Ni4gSUVFRSBJbnRlcm5hdGlvbmFsIENvbmZlcmVuY2Ugb248L3NlY29uZGFyeS10aXRsZT48YWx0
LXRpdGxlPlN5c3RlbXMsIE1hbiBhbmQgQ3liZXJuZXRpY3MsIDIwMDYuIFNNQyAmYXBvczswNi4g
SUVFRSBJbnRlcm5hdGlvbmFsIENvbmZlcmVuY2Ugb248L2FsdC10aXRsZT48L3RpdGxlcz48cGFn
ZXM+MTYyNC0xNjI5PC9wYWdlcz48dm9sdW1lPjI8L3ZvbHVtZT48a2V5d29yZHM+PGtleXdvcmQ+
ZW1vdGlvbiByZWNvZ25pdGlvbjwva2V5d29yZD48a2V5d29yZD5mYWNlIHJlY29nbml0aW9uPC9r
ZXl3b3JkPjxrZXl3b3JkPmh1bWFuIGNvbXB1dGVyIGludGVyYWN0aW9uPC9rZXl3b3JkPjxrZXl3
b3JkPnJvdWdoIHNldCB0aGVvcnk8L2tleXdvcmQ+PGtleXdvcmQ+c3BlZWNoIHJlY29nbml0aW9u
PC9rZXl3b3JkPjxrZXl3b3JkPmRhdGEgYW5hbHlzaXM8L2tleXdvcmQ+PGtleXdvcmQ+ZW1vdGlv
bmFsIHN0YXRlczwva2V5d29yZD48a2V5d29yZD5mYWNpYWwgZXhwcmVzc2lvbnM8L2tleXdvcmQ+
PGtleXdvcmQ+aHVtYW4tY29tcHV0ZXIgaW50ZXJhY3Rpb248L2tleXdvcmQ+PGtleXdvcmQ+bGVh
cm5pbmcgcHJvYmxlbTwva2V5d29yZD48a2V5d29yZD5waHlzaW9sb2dpY2FsIGRhdGE8L2tleXdv
cmQ+PGtleXdvcmQ+cmVjb2duaXplIGVtb3Rpb248L2tleXdvcmQ+PGtleXdvcmQ+cm91Z2ggc2V0
IGNsYXNzaWZpZXI8L2tleXdvcmQ+PGtleXdvcmQ+c2lnbmFsIHByb2Nlc3Npbmc8L2tleXdvcmQ+
PGtleXdvcmQ+c3BlZWNoIHNpZ25hbHM8L2tleXdvcmQ+PC9rZXl3b3Jkcz48ZGF0ZXM+PHllYXI+
MjAwNjwveWVhcj48cHViLWRhdGVzPjxkYXRlPjgtMTEgT2N0LiAyMDA2PC9kYXRlPjwvcHViLWRh
dGVzPjwvZGF0ZXM+PHVybHM+PC91cmxzPjxlbGVjdHJvbmljLXJlc291cmNlLW51bT4xMC4xMTA5
L2ljc21jLjIwMDYuMzg0OTUxPC9lbGVjdHJvbmljLXJlc291cmNlLW51bT48L3JlY29yZD48L0Np
dGU+PENpdGU+PEF1dGhvcj5QZXRlcjwvQXV0aG9yPjxZZWFyPjIwMDU8L1llYXI+PFJlY051bT41
PC9SZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj41PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+
PGtleSBhcHA9IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6
Ij41PC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+
MTc8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5QZXRlciwgQy48L2F1
dGhvcj48YXV0aG9yPkViZXJ0LCBFLjwvYXV0aG9yPjxhdXRob3I+QmVpa2lyY2gsIEguPC9hdXRo
b3I+PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRsZXM+PHRpdGxlPkEgd2VhcmFibGUgbXVs
dGktc2Vuc29yIHN5c3RlbSBmb3IgbW9iaWxlIGFjcXVpc2l0aW9uIG9mIGVtb3Rpb24tcmVsYXRl
ZCBwaHlzaW9sb2dpY2FsIGRhdGE8L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+QWZmZWN0aXZlIENv
bXB1dGluZyBhbmQgSW50ZWxsaWdlbnQgSW50ZXJhY3Rpb248L3NlY29uZGFyeS10aXRsZT48L3Rp
dGxlcz48cGVyaW9kaWNhbD48ZnVsbC10aXRsZT5BZmZlY3RpdmUgQ29tcHV0aW5nIGFuZCBJbnRl
bGxpZ2VudCBJbnRlcmFjdGlvbjwvZnVsbC10aXRsZT48L3BlcmlvZGljYWw+PHBhZ2VzPjY5MS02
OTg8L3BhZ2VzPjxkYXRlcz48eWVhcj4yMDA1PC95ZWFyPjwvZGF0ZXM+PHVybHM+PC91cmxzPjwv
cmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPlBpY2FyZDwvQXV0aG9yPjxZZWFyPjIwMDM8L1ll
YXI+PFJlY051bT44PC9SZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj44PC9yZWMtbnVtYmVyPjxm
b3JlaWduLWtleXM+PGtleSBhcHA9IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJlcXZkcHM5
ZWFlZnh6dHJkZnN6Ij44PC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJu
YWwgQXJ0aWNsZSI+MTc8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5Q
aWNhcmQsIFJvc2FsaW5kIFcuPC9hdXRob3I+PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRs
ZXM+PHRpdGxlPkFmZmVjdGl2ZSBjb21wdXRpbmc6IGNoYWxsZW5nZXM8L3RpdGxlPjxzZWNvbmRh
cnktdGl0bGU+SW50ZXJuYXRpb25hbCBKb3VybmFsIG9mIEh1bWFuLUNvbXB1dGVyIFN0dWRpZXM8
L3NlY29uZGFyeS10aXRsZT48L3RpdGxlcz48cGVyaW9kaWNhbD48ZnVsbC10aXRsZT5JbnRlcm5h
dGlvbmFsIEpvdXJuYWwgb2YgSHVtYW4tQ29tcHV0ZXIgU3R1ZGllczwvZnVsbC10aXRsZT48L3Bl
cmlvZGljYWw+PHBhZ2VzPjU1LTY0PC9wYWdlcz48dm9sdW1lPjU5PC92b2x1bWU+PG51bWJlcj4x
4oCTMjwvbnVtYmVyPjxkYXRlcz48eWVhcj4yMDAzPC95ZWFyPjwvZGF0ZXM+PGlzYm4+MTA3MS01
ODE5PC9pc2JuPjx1cmxzPjxyZWxhdGVkLXVybHM+PHVybD5odHRwOi8vd3d3LnNjaWVuY2VkaXJl
Y3QuY29tL3NjaWVuY2UvYXJ0aWNsZS9waWkvUzEwNzE1ODE5MDMwMDA1MjE8L3VybD48L3JlbGF0
ZWQtdXJscz48L3VybHM+PGVsZWN0cm9uaWMtcmVzb3VyY2UtbnVtPjEwLjEwMTYvczEwNzEtNTgx
OSgwMykwMDA1Mi0xPC9lbGVjdHJvbmljLXJlc291cmNlLW51bT48L3JlY29yZD48L0NpdGU+PC9F
bmROb3RlPgB=
ADDIN EN.CITE PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5Ob3ZhazwvQXV0aG9yPjxZZWFyPjIwMTI8L1llYXI+PFJl
Y051bT42PC9SZWNOdW0+PERpc3BsYXlUZXh0Pls1Ny02NV08L0Rpc3BsYXlUZXh0PjxyZWNvcmQ+
PHJlYy1udW1iZXI+NjwvcmVjLW51bWJlcj48Zm9yZWlnbi1rZXlzPjxrZXkgYXBwPSJFTiIgZGIt
aWQ9InBhc3RkNTJ6cmFmZGF1ZWVyZXF2ZHBzOWVhZWZ4enRyZGZzeiI+Njwva2V5PjwvZm9yZWln
bi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9yZWYtdHlwZT48Y29u
dHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+Tm92YWssIERvbWVuPC9hdXRob3I+PGF1dGhvcj5N
aWhlbGosIE1hdGphxb48L2F1dGhvcj48YXV0aG9yPk11bmloLCBNYXJrbzwvYXV0aG9yPjwvYXV0
aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0bGVzPjx0aXRsZT5BIHN1cnZleSBvZiBtZXRob2RzIGZv
ciBkYXRhIGZ1c2lvbiBhbmQgc3lzdGVtIGFkYXB0YXRpb24gdXNpbmcgYXV0b25vbWljIG5lcnZv
dXMgc3lzdGVtIHJlc3BvbnNlcyBpbiBwaHlzaW9sb2dpY2FsIGNvbXB1dGluZzwvdGl0bGU+PHNl
Y29uZGFyeS10aXRsZT5JbnRlcmFjdGluZyB3aXRoIENvbXB1dGVyczwvc2Vjb25kYXJ5LXRpdGxl
PjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPkludGVyYWN0aW5nIHdpdGggQ29tcHV0
ZXJzPC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+MTU0LTE3MjwvcGFnZXM+PHZvbHVt
ZT4yNDwvdm9sdW1lPjxudW1iZXI+MzwvbnVtYmVyPjxrZXl3b3Jkcz48a2V5d29yZD5BZmZlY3Rp
dmUgY29tcHV0aW5nPC9rZXl3b3JkPjxrZXl3b3JkPlBoeXNpb2xvZ2ljYWwgY29tcHV0aW5nPC9r
ZXl3b3JkPjxrZXl3b3JkPlBzeWNob3BoeXNpb2xvZ3k8L2tleXdvcmQ+PGtleXdvcmQ+RGF0YSBm
dXNpb248L2tleXdvcmQ+PGtleXdvcmQ+QXV0b25vbWljIG5lcnZvdXMgc3lzdGVtPC9rZXl3b3Jk
Pjwva2V5d29yZHM+PGRhdGVzPjx5ZWFyPjIwMTI8L3llYXI+PC9kYXRlcz48aXNibj4wOTUzLTU0
Mzg8L2lzYm4+PHVybHM+PHJlbGF0ZWQtdXJscz48dXJsPmh0dHA6Ly93d3cuc2NpZW5jZWRpcmVj
dC5jb20vc2NpZW5jZS9hcnRpY2xlL3BpaS9TMDk1MzU0MzgxMjAwMDM3OTwvdXJsPjwvcmVsYXRl
ZC11cmxzPjwvdXJscz48ZWxlY3Ryb25pYy1yZXNvdXJjZS1udW0+MTAuMTAxNi9qLmludGNvbS4y
MDEyLjA0LjAwMzwvZWxlY3Ryb25pYy1yZXNvdXJjZS1udW0+PHJlc2VhcmNoLW5vdGVzPmR3MDE8
L3Jlc2VhcmNoLW5vdGVzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPk1jRHVmZjwvQXV0
aG9yPjxZZWFyPjIwMTI8L1llYXI+PFJlY051bT4xNTwvUmVjTnVtPjxyZWNvcmQ+PHJlYy1udW1i
ZXI+MTU8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0
ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjE1PC9rZXk+PC9mb3JlaWduLWtleXM+
PHJlZi10eXBlIG5hbWU9IkNvbmZlcmVuY2UgUHJvY2VlZGluZ3MiPjEwPC9yZWYtdHlwZT48Y29u
dHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+TWNEdWZmLCBELjwvYXV0aG9yPjxhdXRob3I+S2Fy
bHNvbiwgQS48L2F1dGhvcj48YXV0aG9yPkthcG9vciwgQS48L2F1dGhvcj48YXV0aG9yPlJvc2V3
YXksIEEuPC9hdXRob3I+PGF1dGhvcj5DemVyd2luc2tpLCBNLjwvYXV0aG9yPjwvYXV0aG9ycz48
L2NvbnRyaWJ1dG9ycz48dGl0bGVzPjx0aXRsZT5BZmZlY3RBdXJhOiBhbiBpbnRlbGxpZ2VudCBz
eXN0ZW0gZm9yIGVtb3Rpb25hbCBtZW1vcnk8L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+VGhlIEFD
TSBTSUdDSEkgQ29uZmVyZW5jZSBvbiBIdW1hbiBGYWN0b3JzIGluIENvbXB1dGluZyBTeXN0ZW1z
PC9zZWNvbmRhcnktdGl0bGU+PC90aXRsZXM+PHBhZ2VzPjg0OS04NTg8L3BhZ2VzPjxkYXRlcz48
eWVhcj4yMDEyPC95ZWFyPjwvZGF0ZXM+PHB1Yi1sb2NhdGlvbj5BdXN0aW4sIFRYPC9wdWItbG9j
YXRpb24+PHB1Ymxpc2hlcj5BQ008L3B1Ymxpc2hlcj48aXNibj4xNDUwMzEwMTVYPC9pc2JuPjx1
cmxzPjwvdXJscz48cmVzZWFyY2gtbm90ZXM+ZHcwMiYjeEQ7QWZmZWN0aXZhIFEgc2Vuc29yPC9y
ZXNlYXJjaC1ub3Rlcz48L3JlY29yZD48L0NpdGU+PENpdGU+PEF1dGhvcj5Cb3llcjwvQXV0aG9y
PjxZZWFyPjIwMTI8L1llYXI+PFJlY051bT4zPC9SZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj4z
PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+PGtleSBhcHA9IkVOIiBkYi1pZD0icGFzdGQ1Mnpy
YWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij4zPC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10
eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1
dGhvcnM+PGF1dGhvcj5Cb3llciwgRS5XLjwvYXV0aG9yPjxhdXRob3I+RmxldGNoZXIsIFIuPC9h
dXRob3I+PGF1dGhvcj5GYXksIFIuSi48L2F1dGhvcj48YXV0aG9yPlNtZWxzb24sIEQuPC9hdXRo
b3I+PGF1dGhvcj5aaWVkb25pcywgRC48L2F1dGhvcj48YXV0aG9yPlBpY2FyZCwgUi5XLjwvYXV0
aG9yPjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0bGVzPjx0aXRsZT5QcmVsaW1pbmFyeSBF
ZmZvcnRzIERpcmVjdGVkIFRvd2FyZCB0aGUgRGV0ZWN0aW9uIG9mIENyYXZpbmcgb2YgSWxsaWNp
dCBTdWJzdGFuY2VzOiBUaGUgaUhlYWwgUHJvamVjdDwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5K
b3VybmFsIG9mIE1lZGljYWwgVG94aWNvbG9neTwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVzPjxw
ZXJpb2RpY2FsPjxmdWxsLXRpdGxlPkpvdXJuYWwgb2YgTWVkaWNhbCBUb3hpY29sb2d5PC9mdWxs
LXRpdGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+MS01PC9wYWdlcz48ZGF0ZXM+PHllYXI+MjAxMjwv
eWVhcj48L2RhdGVzPjxpc2JuPjE1NTYtOTAzOTwvaXNibj48dXJscz48L3VybHM+PC9yZWNvcmQ+
PC9DaXRlPjxDaXRlPjxBdXRob3I+UGxhcnJlPC9BdXRob3I+PFllYXI+MjAxMTwvWWVhcj48UmVj
TnVtPjI8L1JlY051bT48cmVjb3JkPjxyZWMtbnVtYmVyPjI8L3JlYy1udW1iZXI+PGZvcmVpZ24t
a2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0
cmRmc3oiPjI8L2tleT48L2ZvcmVpZ24ta2V5cz48cmVmLXR5cGUgbmFtZT0iQ29uZmVyZW5jZSBQ
cm9jZWVkaW5ncyI+MTA8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5Q
bGFycmUsIEsuPC9hdXRob3I+PGF1dGhvcj5SYWlqLCBBLjwvYXV0aG9yPjxhdXRob3I+SG9zc2Fp
biwgUy5NLjwvYXV0aG9yPjxhdXRob3I+QWxpLCBBLkEuPC9hdXRob3I+PGF1dGhvcj5OYWthamlt
YSwgTS48L2F1dGhvcj48YXV0aG9yPkFsJmFwb3M7YWJzaSwgTS48L2F1dGhvcj48YXV0aG9yPkVy
dGluLCBFLjwvYXV0aG9yPjxhdXRob3I+S2FtYXJjaywgVC48L2F1dGhvcj48YXV0aG9yPkt1bWFy
LCBTLjwvYXV0aG9yPjxhdXRob3I+U2NvdHQsIE0uPC9hdXRob3I+PC9hdXRob3JzPjwvY29udHJp
YnV0b3JzPjx0aXRsZXM+PHRpdGxlPkNvbnRpbnVvdXMgaW5mZXJlbmNlIG9mIHBzeWNob2xvZ2lj
YWwgc3RyZXNzIGZyb20gc2Vuc29yeSBtZWFzdXJlbWVudHMgY29sbGVjdGVkIGluIHRoZSBuYXR1
cmFsIGVudmlyb25tZW50PC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPkluZm9ybWF0aW9uIFByb2Nl
c3NpbmcgaW4gU2Vuc29yIE5ldHdvcmtzIChJUFNOKSwgMjAxMSAxMHRoIEludGVybmF0aW9uYWwg
Q29uZmVyZW5jZSBvbjwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVzPjxwYWdlcz45Ny0xMDg8L3Bh
Z2VzPjxkYXRlcz48eWVhcj4yMDExPC95ZWFyPjwvZGF0ZXM+PHB1Yi1sb2NhdGlvbj5DaGljYWdv
LCBJTCwgVVNBPC9wdWItbG9jYXRpb24+PHB1Ymxpc2hlcj5JRUVFPC9wdWJsaXNoZXI+PGlzYm4+
MTYxMjg0ODU0MDwvaXNibj48dXJscz48L3VybHM+PC9yZWNvcmQ+PC9DaXRlPjxDaXRlPjxBdXRo
b3I+S3JlaWJpZzwvQXV0aG9yPjxZZWFyPjIwMTA8L1llYXI+PFJlY051bT4xMTwvUmVjTnVtPjxy
ZWNvcmQ+PHJlYy1udW1iZXI+MTE8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0i
RU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjExPC9rZXk+
PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10
eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5LcmVpYmlnLCBTeWx2aWEgRC48L2F1
dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRpdGxlcz48dGl0bGU+QXV0b25vbWljIG5l
cnZvdXMgc3lzdGVtIGFjdGl2aXR5IGluIGVtb3Rpb246IEEgcmV2aWV3PC90aXRsZT48c2Vjb25k
YXJ5LXRpdGxlPkJpb2xvZ2ljYWwgUHN5Y2hvbG9neTwvc2Vjb25kYXJ5LXRpdGxlPjwvdGl0bGVz
PjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPkJpb2xvZ2ljYWwgUHN5Y2hvbG9neTwvZnVsbC10aXRs
ZT48L3BlcmlvZGljYWw+PHBhZ2VzPjM5NC00MjE8L3BhZ2VzPjx2b2x1bWU+ODQ8L3ZvbHVtZT48
bnVtYmVyPjM8L251bWJlcj48a2V5d29yZHM+PGtleXdvcmQ+RW1vdGlvbjwva2V5d29yZD48a2V5
d29yZD5BdXRvbm9taWMgbmVydm91cyBzeXN0ZW08L2tleXdvcmQ+PGtleXdvcmQ+RW1vdGlvbmFs
IHJlc3BvbnNlIHNwZWNpZmljaXR5PC9rZXl3b3JkPjxrZXl3b3JkPkF1dG9ub21pYyByZXNwb25z
ZSBvcmdhbml6YXRpb248L2tleXdvcmQ+PGtleXdvcmQ+Q2FyZGlvdmFzY3VsYXIgc3lzdGVtPC9r
ZXl3b3JkPjxrZXl3b3JkPlJlc3BpcmF0b3J5IHN5c3RlbTwva2V5d29yZD48a2V5d29yZD5FbGVj
dHJvZGVybWFsIHN5c3RlbTwva2V5d29yZD48L2tleXdvcmRzPjxkYXRlcz48eWVhcj4yMDEwPC95
ZWFyPjwvZGF0ZXM+PGlzYm4+MDMwMS0wNTExPC9pc2JuPjx1cmxzPjxyZWxhdGVkLXVybHM+PHVy
bD5odHRwOi8vd3d3LnNjaWVuY2VkaXJlY3QuY29tL3NjaWVuY2UvYXJ0aWNsZS9waWkvUzAzMDEw
NTExMTAwMDA4Mjc8L3VybD48L3JlbGF0ZWQtdXJscz48L3VybHM+PGVsZWN0cm9uaWMtcmVzb3Vy
Y2UtbnVtPjEwLjEwMTYvai5iaW9wc3ljaG8uMjAxMC4wMy4wMTA8L2VsZWN0cm9uaWMtcmVzb3Vy
Y2UtbnVtPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPldhZ25lcjwvQXV0aG9yPjxZZWFy
PjIwMDk8L1llYXI+PFJlY051bT40PC9SZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj40PC9yZWMt
bnVtYmVyPjxmb3JlaWduLWtleXM+PGtleSBhcHA9IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVl
ZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij40PC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5h
bWU9IkNvbmZlcmVuY2UgUHJvY2VlZGluZ3MiPjEwPC9yZWYtdHlwZT48Y29udHJpYnV0b3JzPjxh
dXRob3JzPjxhdXRob3I+V2FnbmVyLCBKLjwvYXV0aG9yPjxhdXRob3I+QW5kcmUsIEUuPC9hdXRo
b3I+PGF1dGhvcj5KdW5nLCBGLjwvYXV0aG9yPjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0
bGVzPjx0aXRsZT5TbWFydCBzZW5zb3IgaW50ZWdyYXRpb246IEEgZnJhbWV3b3JrIGZvciBtdWx0
aW1vZGFsIGVtb3Rpb24gcmVjb2duaXRpb24gaW4gcmVhbC10aW1lPC90aXRsZT48c2Vjb25kYXJ5
LXRpdGxlPkFmZmVjdGl2ZSBDb21wdXRpbmcgYW5kIEludGVsbGlnZW50IEludGVyYWN0aW9uIGFu
ZCBXb3Jrc2hvcHMsIDIwMDkuIEFDSUkgMjAwOS4gM3JkIEludGVybmF0aW9uYWwgQ29uZmVyZW5j
ZSBvbjwvc2Vjb25kYXJ5LXRpdGxlPjxhbHQtdGl0bGU+QWZmZWN0aXZlIENvbXB1dGluZyBhbmQg
SW50ZWxsaWdlbnQgSW50ZXJhY3Rpb24gYW5kIFdvcmtzaG9wcywgMjAwOS4gQUNJSSAyMDA5LiAz
cmQgSW50ZXJuYXRpb25hbCBDb25mZXJlbmNlIG9uPC9hbHQtdGl0bGU+PC90aXRsZXM+PHBhZ2Vz
PjEtODwvcGFnZXM+PGtleXdvcmRzPjxrZXl3b3JkPmVtb3Rpb24gcmVjb2duaXRpb248L2tleXdv
cmQ+PGtleXdvcmQ+aHVtYW4gY29tcHV0ZXIgaW50ZXJhY3Rpb248L2tleXdvcmQ+PGtleXdvcmQ+
c2Vuc29yIGZ1c2lvbjwva2V5d29yZD48a2V5d29yZD5kYXRhIHNlZ21lbnRhdGlvbjwva2V5d29y
ZD48a2V5d29yZD5mZWF0dXJlIGV4dHJhY3Rpb248L2tleXdvcmQ+PGtleXdvcmQ+bXVsdGltb2Rh
bCBlbW90aW9uIHJlY29nbml0aW9uPC9rZXl3b3JkPjxrZXl3b3JkPm9ubGluZSBlbW90aW9uIHJl
Y29nbml0aW9uIHN5c3RlbTwva2V5d29yZD48a2V5d29yZD5wYXR0ZXJuIHJlY29nbml0aW9uPC9r
ZXl3b3JkPjxrZXl3b3JkPnNtYXJ0IHNlbnNvciBpbnRlZ3JhdGlvbjwva2V5d29yZD48L2tleXdv
cmRzPjxkYXRlcz48eWVhcj4yMDA5PC95ZWFyPjxwdWItZGF0ZXM+PGRhdGU+MTAtMTIgU2VwdC4g
MjAwOTwvZGF0ZT48L3B1Yi1kYXRlcz48L2RhdGVzPjx1cmxzPjwvdXJscz48ZWxlY3Ryb25pYy1y
ZXNvdXJjZS1udW0+MTAuMTEwOS9hY2lpLjIwMDkuNTM0OTU3MTwvZWxlY3Ryb25pYy1yZXNvdXJj
ZS1udW0+PC9yZWNvcmQ+PC9DaXRlPjxDaXRlPjxBdXRob3I+TWlpbGxlcjwvQXV0aG9yPjxZZWFy
PjIwMDY8L1llYXI+PFJlY051bT4xNDwvUmVjTnVtPjxyZWNvcmQ+PHJlYy1udW1iZXI+MTQ8L3Jl
Yy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRh
dWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjE0PC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10eXBl
IG5hbWU9IkNvbmZlcmVuY2UgUHJvY2VlZGluZ3MiPjEwPC9yZWYtdHlwZT48Y29udHJpYnV0b3Jz
PjxhdXRob3JzPjxhdXRob3I+TWlpbGxlciwgTS4gRS48L2F1dGhvcj48L2F1dGhvcnM+PC9jb250
cmlidXRvcnM+PHRpdGxlcz48dGl0bGU+V2h5IFNvbWUgRW1vdGlvbmFsIFN0YXRlcyBBcmUgRWFz
aWVyIHRvIGJlIFJlY29nbml6ZWQgVGhhbiBPdGhlcnM6IEEgdGhvcm91Z2ggZGF0YSBhbmFseXNp
cyBhbmQgYSB2ZXJ5IGFjY3VyYXRlIHJvdWdoIHNldCBjbGFzc2lmaWVyPC90aXRsZT48c2Vjb25k
YXJ5LXRpdGxlPlN5c3RlbXMsIE1hbiBhbmQgQ3liZXJuZXRpY3MsIDIwMDYuIFNNQyAmYXBvczsw
Ni4gSUVFRSBJbnRlcm5hdGlvbmFsIENvbmZlcmVuY2Ugb248L3NlY29uZGFyeS10aXRsZT48YWx0
LXRpdGxlPlN5c3RlbXMsIE1hbiBhbmQgQ3liZXJuZXRpY3MsIDIwMDYuIFNNQyAmYXBvczswNi4g
SUVFRSBJbnRlcm5hdGlvbmFsIENvbmZlcmVuY2Ugb248L2FsdC10aXRsZT48L3RpdGxlcz48cGFn
ZXM+MTYyNC0xNjI5PC9wYWdlcz48dm9sdW1lPjI8L3ZvbHVtZT48a2V5d29yZHM+PGtleXdvcmQ+
ZW1vdGlvbiByZWNvZ25pdGlvbjwva2V5d29yZD48a2V5d29yZD5mYWNlIHJlY29nbml0aW9uPC9r
ZXl3b3JkPjxrZXl3b3JkPmh1bWFuIGNvbXB1dGVyIGludGVyYWN0aW9uPC9rZXl3b3JkPjxrZXl3
b3JkPnJvdWdoIHNldCB0aGVvcnk8L2tleXdvcmQ+PGtleXdvcmQ+c3BlZWNoIHJlY29nbml0aW9u
PC9rZXl3b3JkPjxrZXl3b3JkPmRhdGEgYW5hbHlzaXM8L2tleXdvcmQ+PGtleXdvcmQ+ZW1vdGlv
bmFsIHN0YXRlczwva2V5d29yZD48a2V5d29yZD5mYWNpYWwgZXhwcmVzc2lvbnM8L2tleXdvcmQ+
PGtleXdvcmQ+aHVtYW4tY29tcHV0ZXIgaW50ZXJhY3Rpb248L2tleXdvcmQ+PGtleXdvcmQ+bGVh
cm5pbmcgcHJvYmxlbTwva2V5d29yZD48a2V5d29yZD5waHlzaW9sb2dpY2FsIGRhdGE8L2tleXdv
cmQ+PGtleXdvcmQ+cmVjb2duaXplIGVtb3Rpb248L2tleXdvcmQ+PGtleXdvcmQ+cm91Z2ggc2V0
IGNsYXNzaWZpZXI8L2tleXdvcmQ+PGtleXdvcmQ+c2lnbmFsIHByb2Nlc3Npbmc8L2tleXdvcmQ+
PGtleXdvcmQ+c3BlZWNoIHNpZ25hbHM8L2tleXdvcmQ+PC9rZXl3b3Jkcz48ZGF0ZXM+PHllYXI+
MjAwNjwveWVhcj48cHViLWRhdGVzPjxkYXRlPjgtMTEgT2N0LiAyMDA2PC9kYXRlPjwvcHViLWRh
dGVzPjwvZGF0ZXM+PHVybHM+PC91cmxzPjxlbGVjdHJvbmljLXJlc291cmNlLW51bT4xMC4xMTA5
L2ljc21jLjIwMDYuMzg0OTUxPC9lbGVjdHJvbmljLXJlc291cmNlLW51bT48L3JlY29yZD48L0Np
dGU+PENpdGU+PEF1dGhvcj5QZXRlcjwvQXV0aG9yPjxZZWFyPjIwMDU8L1llYXI+PFJlY051bT41
PC9SZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj41PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+
PGtleSBhcHA9IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6
Ij41PC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+
MTc8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5QZXRlciwgQy48L2F1
dGhvcj48YXV0aG9yPkViZXJ0LCBFLjwvYXV0aG9yPjxhdXRob3I+QmVpa2lyY2gsIEguPC9hdXRo
b3I+PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRsZXM+PHRpdGxlPkEgd2VhcmFibGUgbXVs
dGktc2Vuc29yIHN5c3RlbSBmb3IgbW9iaWxlIGFjcXVpc2l0aW9uIG9mIGVtb3Rpb24tcmVsYXRl
ZCBwaHlzaW9sb2dpY2FsIGRhdGE8L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+QWZmZWN0aXZlIENv
bXB1dGluZyBhbmQgSW50ZWxsaWdlbnQgSW50ZXJhY3Rpb248L3NlY29uZGFyeS10aXRsZT48L3Rp
dGxlcz48cGVyaW9kaWNhbD48ZnVsbC10aXRsZT5BZmZlY3RpdmUgQ29tcHV0aW5nIGFuZCBJbnRl
bGxpZ2VudCBJbnRlcmFjdGlvbjwvZnVsbC10aXRsZT48L3BlcmlvZGljYWw+PHBhZ2VzPjY5MS02
OTg8L3BhZ2VzPjxkYXRlcz48eWVhcj4yMDA1PC95ZWFyPjwvZGF0ZXM+PHVybHM+PC91cmxzPjwv
cmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPlBpY2FyZDwvQXV0aG9yPjxZZWFyPjIwMDM8L1ll
YXI+PFJlY051bT44PC9SZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj44PC9yZWMtbnVtYmVyPjxm
b3JlaWduLWtleXM+PGtleSBhcHA9IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJlcXZkcHM5
ZWFlZnh6dHJkZnN6Ij44PC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJu
YWwgQXJ0aWNsZSI+MTc8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5Q
aWNhcmQsIFJvc2FsaW5kIFcuPC9hdXRob3I+PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRs
ZXM+PHRpdGxlPkFmZmVjdGl2ZSBjb21wdXRpbmc6IGNoYWxsZW5nZXM8L3RpdGxlPjxzZWNvbmRh
cnktdGl0bGU+SW50ZXJuYXRpb25hbCBKb3VybmFsIG9mIEh1bWFuLUNvbXB1dGVyIFN0dWRpZXM8
L3NlY29uZGFyeS10aXRsZT48L3RpdGxlcz48cGVyaW9kaWNhbD48ZnVsbC10aXRsZT5JbnRlcm5h
dGlvbmFsIEpvdXJuYWwgb2YgSHVtYW4tQ29tcHV0ZXIgU3R1ZGllczwvZnVsbC10aXRsZT48L3Bl
cmlvZGljYWw+PHBhZ2VzPjU1LTY0PC9wYWdlcz48dm9sdW1lPjU5PC92b2x1bWU+PG51bWJlcj4x
4oCTMjwvbnVtYmVyPjxkYXRlcz48eWVhcj4yMDAzPC95ZWFyPjwvZGF0ZXM+PGlzYm4+MTA3MS01
ODE5PC9pc2JuPjx1cmxzPjxyZWxhdGVkLXVybHM+PHVybD5odHRwOi8vd3d3LnNjaWVuY2VkaXJl
Y3QuY29tL3NjaWVuY2UvYXJ0aWNsZS9waWkvUzEwNzE1ODE5MDMwMDA1MjE8L3VybD48L3JlbGF0
ZWQtdXJscz48L3VybHM+PGVsZWN0cm9uaWMtcmVzb3VyY2UtbnVtPjEwLjEwMTYvczEwNzEtNTgx
OSgwMykwMDA1Mi0xPC9lbGVjdHJvbmljLXJlc291cmNlLW51bT48L3JlY29yZD48L0NpdGU+PC9F
bmROb3RlPgB=
ADDIN EN.CITE.DATA [57-65]. The emotions could be empirically modeled as classes ADDIN EN.CITE <EndNote><Cite><Author>Kreibig</Author><Year>2010</Year><RecNum>11</RecNum><DisplayText>[61]</DisplayText><record><rec-number>11</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">11</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Kreibig, Sylvia D.</author></authors></contributors><titles><title>Autonomic nervous system activity in emotion: A review</title><secondary-title>Biological Psychology</secondary-title></titles><periodical><full-title>Biological Psychology</full-title></periodical><pages>394-421</pages><volume>84</volume><number>3</number><keywords><keyword>Emotion</keyword><keyword>Autonomic nervous system</keyword><keyword>Emotional response specificity</keyword><keyword>Autonomic response organization</keyword><keyword>Cardiovascular system</keyword><keyword>Respiratory system</keyword><keyword>Electrodermal system</keyword></keywords><dates><year>2010</year></dates><isbn>0301-0511</isbn><urls><related-urls><url>;[61]. The negative emotions include anger, anxiety, disgust, embarrassment, fear, and sadness; whereas the positive emotions have affection, amusement, contentment, happiness, joy, pleasure, pride, and relief. Arousal-valence space ADDIN EN.CITE <EndNote><Cite><Author>Bradley</Author><Year>2000</Year><RecNum>58</RecNum><DisplayText>[66]</DisplayText><record><rec-number>58</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">58</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Bradley, Margaret M</author><author>Lang, Peter J</author></authors></contributors><titles><title>Measuring emotion: Behavior, feeling, and physiology</title><secondary-title>Cognitive neuroscience of emotion</secondary-title></titles><periodical><full-title>Cognitive neuroscience of emotion</full-title></periodical><pages>49-59</pages><volume>25</volume><dates><year>2000</year></dates><urls></urls><research-notes>arousal valence definition</research-notes></record></Cite></EndNote>[66] is an alternative way to define emotions with continuous values. The dimension of arousal represents calmness or excitement, whereas the dimension of valence ranges from highly positive to highly negative. The physiological data come from biosensors in the following channels: Electrodermal Activity (EDA) that measures electrical conductivity or skin conductance, Electrocardiogram (ECG) measuring heart beats, Electroencephalography (EEG) measuring brain activities, Electrooculogram (EOG) measuring eye movements ADDIN EN.CITE <EndNote><Cite><Author>Calvo</Author><Year>2010</Year><RecNum>54</RecNum><DisplayText>[67]</DisplayText><record><rec-number>54</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">54</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Calvo, Rafael A</author><author>D'Mello, Sidney</author></authors></contributors><titles><title>Affect detection: An interdisciplinary review of models, methods, and their applications</title><secondary-title>Affective Computing, IEEE Transactions on</secondary-title></titles><periodical><full-title>Affective Computing, IEEE Transactions on</full-title></periodical><pages>18-37</pages><volume>1</volume><number>1</number><dates><year>2010</year></dates><isbn>1949-3045</isbn><urls></urls></record></Cite></EndNote>[67], and, in a broader sense, accelerometer data, voice, GPS trajectories, etc. Traditionally, no matter to map what biological signals to what emotions, the first step is to retrieve features from the raw data. For example, the R-R intervals extracted from ECG represent a person’s heart beat periods, whose changes may be resulted from emotional changes between calmness and excitement. These features are usually hand-engineered using task dependent techniques developed by domain experts PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5DaGFuZWw8L0F1dGhvcj48WWVhcj4yMDA5PC9ZZWFyPjxS
ZWNOdW0+NjE8L1JlY051bT48RGlzcGxheVRleHQ+WzY4LTcwXTwvRGlzcGxheVRleHQ+PHJlY29y
ZD48cmVjLW51bWJlcj42MTwvcmVjLW51bWJlcj48Zm9yZWlnbi1rZXlzPjxrZXkgYXBwPSJFTiIg
ZGItaWQ9InBhc3RkNTJ6cmFmZGF1ZWVyZXF2ZHBzOWVhZWZ4enRyZGZzeiI+NjE8L2tleT48L2Zv
cmVpZ24ta2V5cz48cmVmLXR5cGUgbmFtZT0iSm91cm5hbCBBcnRpY2xlIj4xNzwvcmVmLXR5cGU+
PGNvbnRyaWJ1dG9ycz48YXV0aG9ycz48YXV0aG9yPkNoYW5lbCwgR3VpbGxhdW1lPC9hdXRob3I+
PGF1dGhvcj5LaWVya2VscywgSm9lcCBKTTwvYXV0aG9yPjxhdXRob3I+U29sZXltYW5pLCBNb2hh
bW1hZDwvYXV0aG9yPjxhdXRob3I+UHVuLCBUaGllcnJ5PC9hdXRob3I+PC9hdXRob3JzPjwvY29u
dHJpYnV0b3JzPjx0aXRsZXM+PHRpdGxlPlNob3J0LXRlcm0gZW1vdGlvbiBhc3Nlc3NtZW50IGlu
IGEgcmVjYWxsIHBhcmFkaWdtPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPkludGVybmF0aW9uYWwg
Sm91cm5hbCBvZiBIdW1hbi1Db21wdXRlciBTdHVkaWVzPC9zZWNvbmRhcnktdGl0bGU+PC90aXRs
ZXM+PHBlcmlvZGljYWw+PGZ1bGwtdGl0bGU+SW50ZXJuYXRpb25hbCBKb3VybmFsIG9mIEh1bWFu
LUNvbXB1dGVyIFN0dWRpZXM8L2Z1bGwtdGl0bGU+PC9wZXJpb2RpY2FsPjxwYWdlcz42MDctNjI3
PC9wYWdlcz48dm9sdW1lPjY3PC92b2x1bWU+PG51bWJlcj44PC9udW1iZXI+PGRhdGVzPjx5ZWFy
PjIwMDk8L3llYXI+PC9kYXRlcz48aXNibj4xMDcxLTU4MTk8L2lzYm4+PHVybHM+PC91cmxzPjwv
cmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPktpbTwvQXV0aG9yPjxZZWFyPjIwMDg8L1llYXI+
PFJlY051bT42MjwvUmVjTnVtPjxyZWNvcmQ+PHJlYy1udW1iZXI+NjI8L3JlYy1udW1iZXI+PGZv
cmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczll
YWVmeHp0cmRmc3oiPjYyPC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJu
YWwgQXJ0aWNsZSI+MTc8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5L
aW0sIEpvbmdod2E8L2F1dGhvcj48YXV0aG9yPkFuZHLDqSwgRWxpc2FiZXRoPC9hdXRob3I+PC9h
dXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRsZXM+PHRpdGxlPkVtb3Rpb24gcmVjb2duaXRpb24g
YmFzZWQgb24gcGh5c2lvbG9naWNhbCBjaGFuZ2VzIGluIG11c2ljIGxpc3RlbmluZzwvdGl0bGU+
PHNlY29uZGFyeS10aXRsZT5QYXR0ZXJuIEFuYWx5c2lzIGFuZCBNYWNoaW5lIEludGVsbGlnZW5j
ZSwgSUVFRSBUcmFuc2FjdGlvbnMgb248L3NlY29uZGFyeS10aXRsZT48L3RpdGxlcz48cGVyaW9k
aWNhbD48ZnVsbC10aXRsZT5QYXR0ZXJuIEFuYWx5c2lzIGFuZCBNYWNoaW5lIEludGVsbGlnZW5j
ZSwgSUVFRSBUcmFuc2FjdGlvbnMgb248L2Z1bGwtdGl0bGU+PC9wZXJpb2RpY2FsPjxwYWdlcz4y
MDY3LTIwODM8L3BhZ2VzPjx2b2x1bWU+MzA8L3ZvbHVtZT48bnVtYmVyPjEyPC9udW1iZXI+PGRh
dGVzPjx5ZWFyPjIwMDg8L3llYXI+PC9kYXRlcz48aXNibj4wMTYyLTg4Mjg8L2lzYm4+PHVybHM+
PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPlJhaW52aWxsZTwvQXV0aG9yPjxZ
ZWFyPjIwMDY8L1llYXI+PFJlY051bT42MzwvUmVjTnVtPjxyZWNvcmQ+PHJlYy1udW1iZXI+NjM8
L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJh
ZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjYzPC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10
eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1
dGhvcnM+PGF1dGhvcj5SYWludmlsbGUsIFBpZXJyZTwvYXV0aG9yPjxhdXRob3I+QmVjaGFyYSwg
QW50b2luZTwvYXV0aG9yPjxhdXRob3I+TmFxdmksIE5hc2lyPC9hdXRob3I+PGF1dGhvcj5EYW1h
c2lvLCBBbnRvbmlvIFI8L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRpdGxlcz48
dGl0bGU+QmFzaWMgZW1vdGlvbnMgYXJlIGFzc29jaWF0ZWQgd2l0aCBkaXN0aW5jdCBwYXR0ZXJu
cyBvZiBjYXJkaW9yZXNwaXJhdG9yeSBhY3Rpdml0eTwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5J
bnRlcm5hdGlvbmFsIGpvdXJuYWwgb2YgcHN5Y2hvcGh5c2lvbG9neTwvc2Vjb25kYXJ5LXRpdGxl
PjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPkludGVybmF0aW9uYWwgam91cm5hbCBv
ZiBwc3ljaG9waHlzaW9sb2d5PC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+NS0xODwv
cGFnZXM+PHZvbHVtZT42MTwvdm9sdW1lPjxudW1iZXI+MTwvbnVtYmVyPjxkYXRlcz48eWVhcj4y
MDA2PC95ZWFyPjwvZGF0ZXM+PGlzYm4+MDE2Ny04NzYwPC9pc2JuPjx1cmxzPjwvdXJscz48L3Jl
Y29yZD48L0NpdGU+PC9FbmROb3RlPn==
ADDIN EN.CITE PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5DaGFuZWw8L0F1dGhvcj48WWVhcj4yMDA5PC9ZZWFyPjxS
ZWNOdW0+NjE8L1JlY051bT48RGlzcGxheVRleHQ+WzY4LTcwXTwvRGlzcGxheVRleHQ+PHJlY29y
ZD48cmVjLW51bWJlcj42MTwvcmVjLW51bWJlcj48Zm9yZWlnbi1rZXlzPjxrZXkgYXBwPSJFTiIg
ZGItaWQ9InBhc3RkNTJ6cmFmZGF1ZWVyZXF2ZHBzOWVhZWZ4enRyZGZzeiI+NjE8L2tleT48L2Zv
cmVpZ24ta2V5cz48cmVmLXR5cGUgbmFtZT0iSm91cm5hbCBBcnRpY2xlIj4xNzwvcmVmLXR5cGU+
PGNvbnRyaWJ1dG9ycz48YXV0aG9ycz48YXV0aG9yPkNoYW5lbCwgR3VpbGxhdW1lPC9hdXRob3I+
PGF1dGhvcj5LaWVya2VscywgSm9lcCBKTTwvYXV0aG9yPjxhdXRob3I+U29sZXltYW5pLCBNb2hh
bW1hZDwvYXV0aG9yPjxhdXRob3I+UHVuLCBUaGllcnJ5PC9hdXRob3I+PC9hdXRob3JzPjwvY29u
dHJpYnV0b3JzPjx0aXRsZXM+PHRpdGxlPlNob3J0LXRlcm0gZW1vdGlvbiBhc3Nlc3NtZW50IGlu
IGEgcmVjYWxsIHBhcmFkaWdtPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPkludGVybmF0aW9uYWwg
Sm91cm5hbCBvZiBIdW1hbi1Db21wdXRlciBTdHVkaWVzPC9zZWNvbmRhcnktdGl0bGU+PC90aXRs
ZXM+PHBlcmlvZGljYWw+PGZ1bGwtdGl0bGU+SW50ZXJuYXRpb25hbCBKb3VybmFsIG9mIEh1bWFu
LUNvbXB1dGVyIFN0dWRpZXM8L2Z1bGwtdGl0bGU+PC9wZXJpb2RpY2FsPjxwYWdlcz42MDctNjI3
PC9wYWdlcz48dm9sdW1lPjY3PC92b2x1bWU+PG51bWJlcj44PC9udW1iZXI+PGRhdGVzPjx5ZWFy
PjIwMDk8L3llYXI+PC9kYXRlcz48aXNibj4xMDcxLTU4MTk8L2lzYm4+PHVybHM+PC91cmxzPjwv
cmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPktpbTwvQXV0aG9yPjxZZWFyPjIwMDg8L1llYXI+
PFJlY051bT42MjwvUmVjTnVtPjxyZWNvcmQ+PHJlYy1udW1iZXI+NjI8L3JlYy1udW1iZXI+PGZv
cmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczll
YWVmeHp0cmRmc3oiPjYyPC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10eXBlIG5hbWU9IkpvdXJu
YWwgQXJ0aWNsZSI+MTc8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1dGhvcnM+PGF1dGhvcj5L
aW0sIEpvbmdod2E8L2F1dGhvcj48YXV0aG9yPkFuZHLDqSwgRWxpc2FiZXRoPC9hdXRob3I+PC9h
dXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRsZXM+PHRpdGxlPkVtb3Rpb24gcmVjb2duaXRpb24g
YmFzZWQgb24gcGh5c2lvbG9naWNhbCBjaGFuZ2VzIGluIG11c2ljIGxpc3RlbmluZzwvdGl0bGU+
PHNlY29uZGFyeS10aXRsZT5QYXR0ZXJuIEFuYWx5c2lzIGFuZCBNYWNoaW5lIEludGVsbGlnZW5j
ZSwgSUVFRSBUcmFuc2FjdGlvbnMgb248L3NlY29uZGFyeS10aXRsZT48L3RpdGxlcz48cGVyaW9k
aWNhbD48ZnVsbC10aXRsZT5QYXR0ZXJuIEFuYWx5c2lzIGFuZCBNYWNoaW5lIEludGVsbGlnZW5j
ZSwgSUVFRSBUcmFuc2FjdGlvbnMgb248L2Z1bGwtdGl0bGU+PC9wZXJpb2RpY2FsPjxwYWdlcz4y
MDY3LTIwODM8L3BhZ2VzPjx2b2x1bWU+MzA8L3ZvbHVtZT48bnVtYmVyPjEyPC9udW1iZXI+PGRh
dGVzPjx5ZWFyPjIwMDg8L3llYXI+PC9kYXRlcz48aXNibj4wMTYyLTg4Mjg8L2lzYm4+PHVybHM+
PC91cmxzPjwvcmVjb3JkPjwvQ2l0ZT48Q2l0ZT48QXV0aG9yPlJhaW52aWxsZTwvQXV0aG9yPjxZ
ZWFyPjIwMDY8L1llYXI+PFJlY051bT42MzwvUmVjTnVtPjxyZWNvcmQ+PHJlYy1udW1iZXI+NjM8
L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJh
ZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjYzPC9rZXk+PC9mb3JlaWduLWtleXM+PHJlZi10
eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10eXBlPjxjb250cmlidXRvcnM+PGF1
dGhvcnM+PGF1dGhvcj5SYWludmlsbGUsIFBpZXJyZTwvYXV0aG9yPjxhdXRob3I+QmVjaGFyYSwg
QW50b2luZTwvYXV0aG9yPjxhdXRob3I+TmFxdmksIE5hc2lyPC9hdXRob3I+PGF1dGhvcj5EYW1h
c2lvLCBBbnRvbmlvIFI8L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRpdGxlcz48
dGl0bGU+QmFzaWMgZW1vdGlvbnMgYXJlIGFzc29jaWF0ZWQgd2l0aCBkaXN0aW5jdCBwYXR0ZXJu
cyBvZiBjYXJkaW9yZXNwaXJhdG9yeSBhY3Rpdml0eTwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5J
bnRlcm5hdGlvbmFsIGpvdXJuYWwgb2YgcHN5Y2hvcGh5c2lvbG9neTwvc2Vjb25kYXJ5LXRpdGxl
PjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPkludGVybmF0aW9uYWwgam91cm5hbCBv
ZiBwc3ljaG9waHlzaW9sb2d5PC9mdWxsLXRpdGxlPjwvcGVyaW9kaWNhbD48cGFnZXM+NS0xODwv
cGFnZXM+PHZvbHVtZT42MTwvdm9sdW1lPjxudW1iZXI+MTwvbnVtYmVyPjxkYXRlcz48eWVhcj4y
MDA2PC95ZWFyPjwvZGF0ZXM+PGlzYm4+MDE2Ny04NzYwPC9pc2JuPjx1cmxzPjwvdXJscz48L3Jl
Y29yZD48L0NpdGU+PC9FbmROb3RlPn==
ADDIN EN.CITE.DATA [68-70] and then selected either by experts or feature selection algorithms like principal components analysis (PCA). The process works fine where what features are more relevant to a specific task look obvious but it is still labor-intensive and time-consuming. When the incoming physiological data types and/or prediction tasks change, we have to redo the whole process to tailor new features. It would be useful to have a universal system that can automatically extract features from the raw physiological data without the help of expert knowledge.Moreover, multi-task learning ADDIN EN.CITE <EndNote><Cite><Author>Thrun</Author><Year>1996</Year><RecNum>78</RecNum><DisplayText>[71, 72]</DisplayText><record><rec-number>78</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">78</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Thrun, Sebastian</author></authors></contributors><titles><title>Is learning the n-th thing any easier than learning the first?</title><secondary-title>Advances in neural information processing systems</secondary-title></titles><periodical><full-title>Advances in neural information processing systems</full-title></periodical><pages>640-646</pages><dates><year>1996</year></dates><isbn>1049-5258</isbn><urls></urls><research-notes>multi-task learning</research-notes></record></Cite><Cite><Author>Thrun</Author><Year>1996</Year><RecNum>79</RecNum><record><rec-number>79</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">79</key></foreign-keys><ref-type name="Conference Proceedings">10</ref-type><contributors><authors><author>Thrun, Sebastian</author></authors></contributors><titles><title>Learning to learn: Introduction</title><secondary-title>In Learning To Learn</secondary-title></titles><dates><year>1996</year></dates><publisher>Citeseer</publisher><urls></urls><research-notes>multi-task learning</research-notes></record></Cite></EndNote>[71, 72], learning more than one problem at the same time using a shared representation, is desired when learning physiological data. These problems even include unknown ones. Multi-task learning aims to improve the performance of learning algorithms by learning the commonality among multiple tasks. When a subject wears a biosensor with several channels, the task could be to classify her activities, or to determine her emotions. Even if the tasks may not yet be predefined we still want the machine learning algorithm to acquire knowledge of the data. The learned knowledge then can be used for classification problems as long as a task is ready.Deep belief networks (DBNs) ADDIN EN.CITE <EndNote><Cite><Author>Hinton</Author><Year>2006</Year><RecNum>31</RecNum><DisplayText>[2]</DisplayText><record><rec-number>31</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">31</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Hinton, G.E.</author><author>Salakhutdinov, R.R.</author></authors></contributors><titles><title>Reducing the dimensionality of data with neural networks</title><secondary-title>Science</secondary-title></titles><periodical><full-title>Science</full-title></periodical><pages>504-507</pages><volume>313</volume><number>5786</number><dates><year>2006</year></dates><isbn>0036-8075</isbn><urls></urls></record></Cite></EndNote>[2], as a semi-supervised learning algorithm, is promising to tackle the above-mentioned problems. It trains a multilayer neural network in a greedy fashion, each layer being a restricted Boltzmann machine (RBM) ADDIN EN.CITE <EndNote><Cite><Author>Smolensky</Author><Year>1986</Year><RecNum>64</RecNum><DisplayText>[3]</DisplayText><record><rec-number>64</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">64</key></foreign-keys><ref-type name="Book Section">5</ref-type><contributors><authors><author>Smolensky, Paul</author></authors></contributors><titles><title>Information processing in dynamical systems: Foundations of harmony theory</title><secondary-title>Parallel distributed processing: explorations in the microstructure of cognition</secondary-title></titles><pages>194-281</pages><dates><year>1986</year></dates><pub-location>Cambridge, MA, USA</pub-location><publisher>MIT Press</publisher><isbn>0-262-68053-X</isbn><urls></urls></record></Cite></EndNote>[3]. The trained weights and biases in each layer can be thought of as features or filters learned from the input data. The learned features are task-independent, so any tasks can take advantage of them as a starting point for classification problems.DBNs have been mostly applied in handwriting recognition ADDIN EN.CITE <EndNote><Cite><Author>Hinton</Author><Year>2006</Year><RecNum>31</RecNum><DisplayText>[2, 20]</DisplayText><record><rec-number>31</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">31</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Hinton, G.E.</author><author>Salakhutdinov, R.R.</author></authors></contributors><titles><title>Reducing the dimensionality of data with neural networks</title><secondary-title>Science</secondary-title></titles><periodical><full-title>Science</full-title></periodical><pages>504-507</pages><volume>313</volume><number>5786</number><dates><year>2006</year></dates><isbn>0036-8075</isbn><urls></urls></record></Cite><Cite><Author>Lee</Author><Year>2008</Year><RecNum>37</RecNum><record><rec-number>37</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">37</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Lee, H.</author><author>Ekanadham, C.</author><author>Ng, A.</author></authors></contributors><titles><title>Sparse deep belief net model for visual area V2</title><secondary-title>Advances in neural information processing systems</secondary-title></titles><periodical><full-title>Advances in neural information processing systems</full-title></periodical><pages>873-880</pages><volume>20</volume><dates><year>2008</year></dates><urls></urls><research-notes>how to visualize learned features</research-notes></record></Cite></EndNote>[2, 20], natural image recognition ADDIN EN.CITE <EndNote><Cite><Author>Krizhevsky</Author><Year>2009</Year><RecNum>48</RecNum><DisplayText>[73, 74]</DisplayText><record><rec-number>48</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">48</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Krizhevsky, A.</author><author>Hinton, G.</author></authors></contributors><titles><title>Learning multiple layers of features from tiny images</title><secondary-title>Master's thesis, Department of Computer Science, University of Toronto</secondary-title></titles><periodical><full-title>Master's thesis, Department of Computer Science, University of Toronto</full-title></periodical><dates><year>2009</year></dates><urls></urls><research-notes>DBN on other image dataset</research-notes></record></Cite><Cite><Author>Vincent</Author><Year>2008</Year><RecNum>47</RecNum><record><rec-number>47</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">47</key></foreign-keys><ref-type name="Conference Proceedings">10</ref-type><contributors><authors><author>Vincent, P.</author><author>Larochelle, H.</author><author>Bengio, Y.</author><author>Manzagol, P.A.</author></authors></contributors><titles><title>Extracting and composing robust features with denoising autoencoders</title><secondary-title>Proceedings of the 25th international conference on Machine learning</secondary-title></titles><pages>1096-1103</pages><dates><year>2008</year></dates><publisher>ACM</publisher><isbn>1605582050</isbn><urls></urls><research-notes>learned feature representation</research-notes></record></Cite></EndNote>[73, 74], and modeling human motions ADDIN EN.CITE <EndNote><Cite><Author>Taylor</Author><Year>2007</Year><RecNum>50</RecNum><DisplayText>[75]</DisplayText><record><rec-number>50</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">50</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Taylor, Graham W</author><author>Hinton, Geoffrey E</author><author>Roweis, Sam T</author></authors></contributors><titles><title>Modeling human motion using binary latent variables</title><secondary-title>Advances in neural information processing systems</secondary-title></titles><periodical><full-title>Advances in neural information processing systems</full-title></periodical><pages>1345</pages><volume>19</volume><dates><year>2007</year></dates><isbn>1049-5258</isbn><urls></urls></record></Cite></EndNote>[75]. When it comes to physiological data as input, ADDIN EN.CITE <EndNote><Cite><Author>Wulsin</Author><Year>2011</Year><RecNum>22</RecNum><DisplayText>[18]</DisplayText><record><rec-number>22</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">22</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>D F Wulsin</author><author>J R Gupta</author><author>R Mani</author><author>J A Blanco</author><author>B Litt</author></authors></contributors><titles><title>Modeling electroencephalography waveforms with semi-supervised deep belief nets: fast classification and anomaly measurement</title><secondary-title>Journal of Neural Engineering</secondary-title></titles><periodical><full-title>Journal of Neural Engineering</full-title></periodical><pages>036015</pages><volume>8</volume><number>3</number><dates><year>2011</year></dates><isbn>1741-2552</isbn><urls><related-urls><url>;[18] used DBNs to classify EEG signals to five clinically significant waves; ADDIN EN.CITE <EndNote><Cite><Author>L?ngkvist</Author><Year>2012</Year><RecNum>23</RecNum><DisplayText>[34]</DisplayText><record><rec-number>23</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">23</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>L?ngkvist, M.</author><author>Karlsson, L.</author><author>Loutfi, A.</author></authors></contributors><titles><title>Sleep Stage Classification Using Unsupervised Feature Learning</title><secondary-title>Advances in Artificial Neural Systems</secondary-title></titles><periodical><full-title>Advances in Artificial Neural Systems</full-title></periodical><volume>2012</volume><dates><year>2012</year></dates><isbn>1687-7594</isbn><urls></urls><research-notes>DBNToolbox</research-notes></record></Cite></EndNote>[34] developed DBNs classifiers to determine sleep stages from EEG, EOG, and EMG. However, to the best of my knowledge, there is no existing work to predict emotional states using physiological data from biosensors using DBNs. This study explores such possibilities.This work uses DEAP dataset ADDIN EN.CITE <EndNote><Cite><Author>Koelstra</Author><Year>2011</Year><RecNum>13</RecNum><DisplayText>[6]</DisplayText><record><rec-number>13</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">13</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Koelstra, S.</author><author>Muhl, C.</author><author>Soleymani, M.</author><author>Lee, J.</author><author>Yazdani, A.</author><author>Ebrahimi, T.</author><author>Pun, T.</author><author>Nijholt, A.</author><author>Patras, I.</author></authors></contributors><titles><title>Deap: A database for emotion analysis using physiological signals</title><secondary-title>Affective Computing, IEEE Transactions on</secondary-title></titles><periodical><full-title>Affective Computing, IEEE Transactions on</full-title></periodical><pages>1-1</pages><number>99</number><dates><year>2011</year></dates><isbn>1949-3045</isbn><urls></urls></record></Cite></EndNote>[6] to show how DBNs learn features from raw physiological signals and predict emotion states.DEAP is a multimodal dataset for the analysis of human affective states. The EEG and peripheral physiological signals (downsampled to 128Hz) of 32 subjects were recorded as each watched 40 one-minute long videos. The subjects rated the levels as continuous values of arousal, valence, liking, dominance, and familiarity. This study predicts the levels of arousal, valence, and liking. The arousal scale ranges from calm or bored (1) to stimulated or excited (9). The valence scale ranges from unhappy or sad (1) to happy or joyful (9). Liking also has values from 1 to 9. All three ratings are float numbers. 32 EEG channels and totally 8 peripheral nervous system channels were recorded including hEOG (horizontal EOG), vEOG (vertical EOG), zEMG (Zygomaticus Major EMG), tEMG (Trapezius Major EMG), GSR (values from Twente converted to Geneva format in Ohm), respiration belt, plethysmograph, and body temperature.This experiment uses 4 peripheral channels to do the predication, which are the two EOG channels and the two EMG channels. The EOG channels record eye movements. The activity of the Zygomaticus major is monitored in zEMG to capture a subject’s laughs or smiles, whereas the Trapezius muscle (neck) is recorded by tEMG to reflect possible head movements. ADDIN EN.CITE <EndNote><Cite><Author>Koelstra</Author><Year>2011</Year><RecNum>13</RecNum><DisplayText>[6]</DisplayText><record><rec-number>13</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">13</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Koelstra, S.</author><author>Muhl, C.</author><author>Soleymani, M.</author><author>Lee, J.</author><author>Yazdani, A.</author><author>Ebrahimi, T.</author><author>Pun, T.</author><author>Nijholt, A.</author><author>Patras, I.</author></authors></contributors><titles><title>Deap: A database for emotion analysis using physiological signals</title><secondary-title>Affective Computing, IEEE Transactions on</secondary-title></titles><periodical><full-title>Affective Computing, IEEE Transactions on</full-title></periodical><pages>1-1</pages><number>99</number><dates><year>2011</year></dates><isbn>1949-3045</isbn><urls></urls></record></Cite></EndNote>[6] trained a Gaussian na?ve Bayes classifier for each single subject due to the high inter-subject variability. For each subject, three different binary classifiers were trained and investigated to map the 8 peripheral channels to low (1-5) / high (5-9) arousal, valence, and liking, respectively. A leave-one-video-out cross validation was performed. In other words, in each trial a video was taken out for testing and the remaining 39 videos were used for training. As was done in most machine learning researches on physiological data, hand-engineered features such as eye blinking rate, energy of the signal, mean and variance of EMG and EOG were extracted. All the extracted features were fed into the classifier to train the model. Then the model was used to predict the test cases. The average accuracy over all subjects was 0.570, 0.627, and 0.591 for arousal, valence, and liking, respectively.5.2 Related Works on Modeling Physiological Data and Emotion PredictionTo mapping physiological data to emotions, we need to train a classification model. The advance in machine learning provides many algorithms to get the job done ADDIN EN.CITE <EndNote><Cite><Author>Novak</Author><Year>2012</Year><RecNum>6</RecNum><DisplayText>[57]</DisplayText><record><rec-number>6</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">6</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Novak, Domen</author><author>Mihelj, Matja?</author><author>Munih, Marko</author></authors></contributors><titles><title>A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing</title><secondary-title>Interacting with Computers</secondary-title></titles><periodical><full-title>Interacting with Computers</full-title></periodical><pages>154-172</pages><volume>24</volume><number>3</number><keywords><keyword>Affective computing</keyword><keyword>Physiological computing</keyword><keyword>Psychophysiology</keyword><keyword>Data fusion</keyword><keyword>Autonomic nervous system</keyword></keywords><dates><year>2012</year></dates><isbn>0953-5438</isbn><urls><related-urls><url>;[57]. PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5GcmFudHppZGlzPC9BdXRob3I+PFllYXI+MjAxMDwvWWVh
cj48UmVjTnVtPjEzMDwvUmVjTnVtPjxEaXNwbGF5VGV4dD5bNzYsIDc3XTwvRGlzcGxheVRleHQ+
PHJlY29yZD48cmVjLW51bWJlcj4xMzA8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFw
cD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjEzMDwv
a2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9y
ZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+RnJhbnR6aWRpcywgQ2hyaXN0
b3MgQTwvYXV0aG9yPjxhdXRob3I+QnJhdHNhcywgQ2hhcmFsYW1wb3M8L2F1dGhvcj48YXV0aG9y
PktsYWRvcywgTWFub3Vzb3MgQTwvYXV0aG9yPjxhdXRob3I+S29uc3RhbnRpbmlkaXMsIEV2ZG9r
aW1vczwvYXV0aG9yPjxhdXRob3I+TGl0aGFyaSwgQ2hyeXNhIEQ8L2F1dGhvcj48YXV0aG9yPlZp
dmFzLCBBbmEgQjwvYXV0aG9yPjxhdXRob3I+UGFwYWRlbGlzLCBDaHJpc3RvcyBMPC9hdXRob3I+
PGF1dGhvcj5LYWxkb3VkaSwgRWxlbmk8L2F1dGhvcj48YXV0aG9yPlBhcHBhcywgQ29zdGFzPC9h
dXRob3I+PGF1dGhvcj5CYW1pZGlzLCBQYW5hZ2lvdGlzIEQ8L2F1dGhvcj48L2F1dGhvcnM+PC9j
b250cmlidXRvcnM+PHRpdGxlcz48dGl0bGU+T24gdGhlIGNsYXNzaWZpY2F0aW9uIG9mIGVtb3Rp
b25hbCBiaW9zaWduYWxzIGV2b2tlZCB3aGlsZSB2aWV3aW5nIGFmZmVjdGl2ZSBwaWN0dXJlczog
YW4gaW50ZWdyYXRlZCBkYXRhLW1pbmluZy1iYXNlZCBhcHByb2FjaCBmb3IgaGVhbHRoY2FyZSBh
cHBsaWNhdGlvbnM8L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+SW5mb3JtYXRpb24gVGVjaG5vbG9n
eSBpbiBCaW9tZWRpY2luZSwgSUVFRSBUcmFuc2FjdGlvbnMgb248L3NlY29uZGFyeS10aXRsZT48
L3RpdGxlcz48cGVyaW9kaWNhbD48ZnVsbC10aXRsZT5JbmZvcm1hdGlvbiBUZWNobm9sb2d5IGlu
IEJpb21lZGljaW5lLCBJRUVFIFRyYW5zYWN0aW9ucyBvbjwvZnVsbC10aXRsZT48L3BlcmlvZGlj
YWw+PHBhZ2VzPjMwOS0zMTg8L3BhZ2VzPjx2b2x1bWU+MTQ8L3ZvbHVtZT48bnVtYmVyPjI8L251
bWJlcj48ZGF0ZXM+PHllYXI+MjAxMDwveWVhcj48L2RhdGVzPjxpc2JuPjEwODktNzc3MTwvaXNi
bj48dXJscz48L3VybHM+PC9yZWNvcmQ+PC9DaXRlPjxDaXRlPjxBdXRob3I+U2V0ejwvQXV0aG9y
PjxZZWFyPjIwMTA8L1llYXI+PFJlY051bT4xMzE8L1JlY051bT48cmVjb3JkPjxyZWMtbnVtYmVy
PjEzMTwvcmVjLW51bWJlcj48Zm9yZWlnbi1rZXlzPjxrZXkgYXBwPSJFTiIgZGItaWQ9InBhc3Rk
NTJ6cmFmZGF1ZWVyZXF2ZHBzOWVhZWZ4enRyZGZzeiI+MTMxPC9rZXk+PC9mb3JlaWduLWtleXM+
PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10eXBlPjxjb250cmlidXRv
cnM+PGF1dGhvcnM+PGF1dGhvcj5TZXR6LCBDb3JuZWxpYTwvYXV0aG9yPjxhdXRob3I+QXJucmlj
aCwgQmVydDwvYXV0aG9yPjxhdXRob3I+U2NodW1tLCBKb2hhbm5lczwvYXV0aG9yPjxhdXRob3I+
TGEgTWFyY2EsIFJvYmVydG88L2F1dGhvcj48YXV0aG9yPlRyb3N0ZXIsIEc8L2F1dGhvcj48YXV0
aG9yPkVobGVydCwgVWxyaWtlPC9hdXRob3I+PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRs
ZXM+PHRpdGxlPkRpc2NyaW1pbmF0aW5nIHN0cmVzcyBmcm9tIGNvZ25pdGl2ZSBsb2FkIHVzaW5n
IGEgd2VhcmFibGUgRURBIGRldmljZTwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5JbmZvcm1hdGlv
biBUZWNobm9sb2d5IGluIEJpb21lZGljaW5lLCBJRUVFIFRyYW5zYWN0aW9ucyBvbjwvc2Vjb25k
YXJ5LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPkluZm9ybWF0aW9uIFRl
Y2hub2xvZ3kgaW4gQmlvbWVkaWNpbmUsIElFRUUgVHJhbnNhY3Rpb25zIG9uPC9mdWxsLXRpdGxl
PjwvcGVyaW9kaWNhbD48cGFnZXM+NDEwLTQxNzwvcGFnZXM+PHZvbHVtZT4xNDwvdm9sdW1lPjxu
dW1iZXI+MjwvbnVtYmVyPjxkYXRlcz48eWVhcj4yMDEwPC95ZWFyPjwvZGF0ZXM+PGlzYm4+MTA4
OS03NzcxPC9pc2JuPjx1cmxzPjwvdXJscz48L3JlY29yZD48L0NpdGU+PC9FbmROb3RlPgB=
ADDIN EN.CITE PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5GcmFudHppZGlzPC9BdXRob3I+PFllYXI+MjAxMDwvWWVh
cj48UmVjTnVtPjEzMDwvUmVjTnVtPjxEaXNwbGF5VGV4dD5bNzYsIDc3XTwvRGlzcGxheVRleHQ+
PHJlY29yZD48cmVjLW51bWJlcj4xMzA8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48a2V5IGFw
cD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oiPjEzMDwv
a2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUiPjE3PC9y
ZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+RnJhbnR6aWRpcywgQ2hyaXN0
b3MgQTwvYXV0aG9yPjxhdXRob3I+QnJhdHNhcywgQ2hhcmFsYW1wb3M8L2F1dGhvcj48YXV0aG9y
PktsYWRvcywgTWFub3Vzb3MgQTwvYXV0aG9yPjxhdXRob3I+S29uc3RhbnRpbmlkaXMsIEV2ZG9r
aW1vczwvYXV0aG9yPjxhdXRob3I+TGl0aGFyaSwgQ2hyeXNhIEQ8L2F1dGhvcj48YXV0aG9yPlZp
dmFzLCBBbmEgQjwvYXV0aG9yPjxhdXRob3I+UGFwYWRlbGlzLCBDaHJpc3RvcyBMPC9hdXRob3I+
PGF1dGhvcj5LYWxkb3VkaSwgRWxlbmk8L2F1dGhvcj48YXV0aG9yPlBhcHBhcywgQ29zdGFzPC9h
dXRob3I+PGF1dGhvcj5CYW1pZGlzLCBQYW5hZ2lvdGlzIEQ8L2F1dGhvcj48L2F1dGhvcnM+PC9j
b250cmlidXRvcnM+PHRpdGxlcz48dGl0bGU+T24gdGhlIGNsYXNzaWZpY2F0aW9uIG9mIGVtb3Rp
b25hbCBiaW9zaWduYWxzIGV2b2tlZCB3aGlsZSB2aWV3aW5nIGFmZmVjdGl2ZSBwaWN0dXJlczog
YW4gaW50ZWdyYXRlZCBkYXRhLW1pbmluZy1iYXNlZCBhcHByb2FjaCBmb3IgaGVhbHRoY2FyZSBh
cHBsaWNhdGlvbnM8L3RpdGxlPjxzZWNvbmRhcnktdGl0bGU+SW5mb3JtYXRpb24gVGVjaG5vbG9n
eSBpbiBCaW9tZWRpY2luZSwgSUVFRSBUcmFuc2FjdGlvbnMgb248L3NlY29uZGFyeS10aXRsZT48
L3RpdGxlcz48cGVyaW9kaWNhbD48ZnVsbC10aXRsZT5JbmZvcm1hdGlvbiBUZWNobm9sb2d5IGlu
IEJpb21lZGljaW5lLCBJRUVFIFRyYW5zYWN0aW9ucyBvbjwvZnVsbC10aXRsZT48L3BlcmlvZGlj
YWw+PHBhZ2VzPjMwOS0zMTg8L3BhZ2VzPjx2b2x1bWU+MTQ8L3ZvbHVtZT48bnVtYmVyPjI8L251
bWJlcj48ZGF0ZXM+PHllYXI+MjAxMDwveWVhcj48L2RhdGVzPjxpc2JuPjEwODktNzc3MTwvaXNi
bj48dXJscz48L3VybHM+PC9yZWNvcmQ+PC9DaXRlPjxDaXRlPjxBdXRob3I+U2V0ejwvQXV0aG9y
PjxZZWFyPjIwMTA8L1llYXI+PFJlY051bT4xMzE8L1JlY051bT48cmVjb3JkPjxyZWMtbnVtYmVy
PjEzMTwvcmVjLW51bWJlcj48Zm9yZWlnbi1rZXlzPjxrZXkgYXBwPSJFTiIgZGItaWQ9InBhc3Rk
NTJ6cmFmZGF1ZWVyZXF2ZHBzOWVhZWZ4enRyZGZzeiI+MTMxPC9rZXk+PC9mb3JlaWduLWtleXM+
PHJlZi10eXBlIG5hbWU9IkpvdXJuYWwgQXJ0aWNsZSI+MTc8L3JlZi10eXBlPjxjb250cmlidXRv
cnM+PGF1dGhvcnM+PGF1dGhvcj5TZXR6LCBDb3JuZWxpYTwvYXV0aG9yPjxhdXRob3I+QXJucmlj
aCwgQmVydDwvYXV0aG9yPjxhdXRob3I+U2NodW1tLCBKb2hhbm5lczwvYXV0aG9yPjxhdXRob3I+
TGEgTWFyY2EsIFJvYmVydG88L2F1dGhvcj48YXV0aG9yPlRyb3N0ZXIsIEc8L2F1dGhvcj48YXV0
aG9yPkVobGVydCwgVWxyaWtlPC9hdXRob3I+PC9hdXRob3JzPjwvY29udHJpYnV0b3JzPjx0aXRs
ZXM+PHRpdGxlPkRpc2NyaW1pbmF0aW5nIHN0cmVzcyBmcm9tIGNvZ25pdGl2ZSBsb2FkIHVzaW5n
IGEgd2VhcmFibGUgRURBIGRldmljZTwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5JbmZvcm1hdGlv
biBUZWNobm9sb2d5IGluIEJpb21lZGljaW5lLCBJRUVFIFRyYW5zYWN0aW9ucyBvbjwvc2Vjb25k
YXJ5LXRpdGxlPjwvdGl0bGVzPjxwZXJpb2RpY2FsPjxmdWxsLXRpdGxlPkluZm9ybWF0aW9uIFRl
Y2hub2xvZ3kgaW4gQmlvbWVkaWNpbmUsIElFRUUgVHJhbnNhY3Rpb25zIG9uPC9mdWxsLXRpdGxl
PjwvcGVyaW9kaWNhbD48cGFnZXM+NDEwLTQxNzwvcGFnZXM+PHZvbHVtZT4xNDwvdm9sdW1lPjxu
dW1iZXI+MjwvbnVtYmVyPjxkYXRlcz48eWVhcj4yMDEwPC95ZWFyPjwvZGF0ZXM+PGlzYm4+MTA4
OS03NzcxPC9pc2JuPjx1cmxzPjwvdXJscz48L3JlY29yZD48L0NpdGU+PC9FbmROb3RlPgB=
ADDIN EN.CITE.DATA [76, 77] used k-nearest neighbor. Na?ve Bayes classifier and Bayesian networks were employed in ADDIN EN.CITE <EndNote><Cite><Author>Hand</Author><Year>2001</Year><RecNum>132</RecNum><DisplayText>[78]</DisplayText><record><rec-number>132</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">132</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Hand, David J</author><author>Yu, Keming</author></authors></contributors><titles><title>Idiot's Bayes—not so stupid after all?</title><secondary-title>International Statistical Review</secondary-title></titles><periodical><full-title>International Statistical Review</full-title></periodical><pages>385-398</pages><volume>69</volume><number>3</number><dates><year>2001</year></dates><isbn>1751-5823</isbn><urls></urls></record></Cite></EndNote>[78]. Support vector machines have proved its success in ADDIN EN.CITE <EndNote><Cite><Author>Sakr</Author><Year>2010</Year><RecNum>133</RecNum><DisplayText>[79, 80]</DisplayText><record><rec-number>133</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">133</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Sakr, George E</author><author>Elhajj, Imad H</author><author>Huijer, HA-S</author></authors></contributors><titles><title>Support vector machines to define and detect agitation transition</title><secondary-title>Affective Computing, IEEE Transactions on</secondary-title></titles><periodical><full-title>Affective Computing, IEEE Transactions on</full-title></periodical><pages>98-108</pages><volume>1</volume><number>2</number><dates><year>2010</year></dates><isbn>1949-3045</isbn><urls></urls></record></Cite><Cite><Author>Wu</Author><Year>2010</Year><RecNum>134</RecNum><record><rec-number>134</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">134</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Wu, Dongrui</author><author>Courtney, Christopher G</author><author>Lance, Brent J</author><author>Narayanan, Shrikanth S</author><author>Dawson, Michael E</author><author>Oie, Kelvin S</author><author>Parsons, Thomas D</author></authors></contributors><titles><title>Optimal arousal identification and classification for affective computing using physiological signals: virtual reality Stroop task</title><secondary-title>Affective Computing, IEEE Transactions on</secondary-title></titles><periodical><full-title>Affective Computing, IEEE Transactions on</full-title></periodical><pages>109-118</pages><volume>1</volume><number>2</number><dates><year>2010</year></dates><isbn>1949-3045</isbn><urls></urls></record></Cite></EndNote>[79, 80]. Classification tree performed well in ADDIN EN.CITE <EndNote><Cite><Author>Bailenson</Author><Year>2008</Year><RecNum>135</RecNum><DisplayText>[81]</DisplayText><record><rec-number>135</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">135</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Bailenson, Jeremy N</author><author>Pontikakis, Emmanuel D</author><author>Mauss, Iris B</author><author>Gross, James J</author><author>Jabon, Maria E</author><author>Hutcherson, Cendri AC</author><author>Nass, Clifford</author><author>John, Oliver</author></authors></contributors><titles><title>Real-time classification of evoked emotions using facial feature tracking and physiological responses</title><secondary-title>International journal of human-computer studies</secondary-title></titles><periodical><full-title>International Journal of Human-Computer Studies</full-title></periodical><pages>303-317</pages><volume>66</volume><number>5</number><dates><year>2008</year></dates><isbn>1071-5819</isbn><urls></urls></record></Cite></EndNote>[81]. Artificial neural networks were used in PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5Lb2xvZHlhemhuaXk8L0F1dGhvcj48WWVhcj4yMDExPC9Z
ZWFyPjxSZWNOdW0+MTM2PC9SZWNOdW0+PERpc3BsYXlUZXh0Pls4Mi04NF08L0Rpc3BsYXlUZXh0
PjxyZWNvcmQ+PHJlYy1udW1iZXI+MTM2PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+PGtleSBh
cHA9IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij4xMzY8
L2tleT48L2ZvcmVpZ24ta2V5cz48cmVmLXR5cGUgbmFtZT0iSm91cm5hbCBBcnRpY2xlIj4xNzwv
cmVmLXR5cGU+PGNvbnRyaWJ1dG9ycz48YXV0aG9ycz48YXV0aG9yPktvbG9keWF6aG5peSwgVml0
YWxpeTwvYXV0aG9yPjxhdXRob3I+S3JlaWJpZywgU3lsdmlhIEQ8L2F1dGhvcj48YXV0aG9yPkdy
b3NzLCBKYW1lcyBKPC9hdXRob3I+PGF1dGhvcj5Sb3RoLCBXYWx0b24gVDwvYXV0aG9yPjxhdXRo
b3I+V2lsaGVsbSwgRnJhbmsgSDwvYXV0aG9yPjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0
bGVzPjx0aXRsZT5BbiBhZmZlY3RpdmUgY29tcHV0aW5nIGFwcHJvYWNoIHRvIHBoeXNpb2xvZ2lj
YWwgZW1vdGlvbiBzcGVjaWZpY2l0eTogVG93YXJkIHN1YmplY3TigJBpbmRlcGVuZGVudCBhbmQg
c3RpbXVsdXPigJBpbmRlcGVuZGVudCBjbGFzc2lmaWNhdGlvbiBvZiBmaWxt4oCQaW5kdWNlZCBl
bW90aW9uczwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5Qc3ljaG9waHlzaW9sb2d5PC9zZWNvbmRh
cnktdGl0bGU+PC90aXRsZXM+PHBlcmlvZGljYWw+PGZ1bGwtdGl0bGU+UHN5Y2hvcGh5c2lvbG9n
eTwvZnVsbC10aXRsZT48L3BlcmlvZGljYWw+PHBhZ2VzPjkwOC05MjI8L3BhZ2VzPjx2b2x1bWU+
NDg8L3ZvbHVtZT48bnVtYmVyPjc8L251bWJlcj48ZGF0ZXM+PHllYXI+MjAxMTwveWVhcj48L2Rh
dGVzPjxpc2JuPjE0NjktODk4NjwvaXNibj48dXJscz48L3VybHM+PC9yZWNvcmQ+PC9DaXRlPjxD
aXRlPjxBdXRob3I+TmFzb3o8L0F1dGhvcj48WWVhcj4yMDEwPC9ZZWFyPjxSZWNOdW0+MTM3PC9S
ZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj4xMzc8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48
a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oi
PjEzNzwva2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUi
PjE3PC9yZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+TmFzb3osIEZhdG1h
PC9hdXRob3I+PGF1dGhvcj5MaXNldHRpLCBDaHJpc3RpbmUgTDwvYXV0aG9yPjxhdXRob3I+VmFz
aWxha29zLCBBdGhhbmFzaW9zIFY8L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRp
dGxlcz48dGl0bGU+QWZmZWN0aXZlbHkgaW50ZWxsaWdlbnQgYW5kIGFkYXB0aXZlIGNhciBpbnRl
cmZhY2VzPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPkluZm9ybWF0aW9uIFNjaWVuY2VzPC9zZWNv
bmRhcnktdGl0bGU+PC90aXRsZXM+PHBlcmlvZGljYWw+PGZ1bGwtdGl0bGU+SW5mb3JtYXRpb24g
U2NpZW5jZXM8L2Z1bGwtdGl0bGU+PC9wZXJpb2RpY2FsPjxwYWdlcz4zODE3LTM4MzY8L3BhZ2Vz
Pjx2b2x1bWU+MTgwPC92b2x1bWU+PG51bWJlcj4yMDwvbnVtYmVyPjxkYXRlcz48eWVhcj4yMDEw
PC95ZWFyPjwvZGF0ZXM+PGlzYm4+MDAyMC0wMjU1PC9pc2JuPjx1cmxzPjwvdXJscz48L3JlY29y
ZD48L0NpdGU+PENpdGU+PEF1dGhvcj52YW4gZGVuIEJyb2VrPC9BdXRob3I+PFllYXI+MjAxMDwv
WWVhcj48UmVjTnVtPjEzODwvUmVjTnVtPjxyZWNvcmQ+PHJlYy1udW1iZXI+MTM4PC9yZWMtbnVt
YmVyPjxmb3JlaWduLWtleXM+PGtleSBhcHA9IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJl
cXZkcHM5ZWFlZnh6dHJkZnN6Ij4xMzg8L2tleT48L2ZvcmVpZ24ta2V5cz48cmVmLXR5cGUgbmFt
ZT0iQm9vayBTZWN0aW9uIj41PC9yZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRo
b3I+dmFuIGRlbiBCcm9laywgRWdvbiBMPC9hdXRob3I+PGF1dGhvcj5MaXPDvSwgVmlsaWFtPC9h
dXRob3I+PGF1dGhvcj5KYW5zc2VuLCBKb3JpcyBIPC9hdXRob3I+PGF1dGhvcj5XZXN0ZXJpbmss
IEpveWNlIEhETTwvYXV0aG9yPjxhdXRob3I+U2NodXQsIE1hcmxlZW4gSDwvYXV0aG9yPjxhdXRo
b3I+VHVpbmVuYnJlaWplciwgS2VlczwvYXV0aG9yPjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48
dGl0bGVzPjx0aXRsZT5BZmZlY3RpdmUgbWFuLW1hY2hpbmUgaW50ZXJmYWNlOiB1bnZlaWxpbmcg
aHVtYW4gZW1vdGlvbnMgdGhyb3VnaCBiaW9zaWduYWxzPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxl
PkJpb21lZGljYWwgRW5naW5lZXJpbmcgU3lzdGVtcyBhbmQgVGVjaG5vbG9naWVzPC9zZWNvbmRh
cnktdGl0bGU+PC90aXRsZXM+PHBhZ2VzPjIxLTQ3PC9wYWdlcz48ZGF0ZXM+PHllYXI+MjAxMDwv
eWVhcj48L2RhdGVzPjxwdWJsaXNoZXI+U3ByaW5nZXI8L3B1Ymxpc2hlcj48aXNibj4zNjQyMTE3
MjAxPC9pc2JuPjx1cmxzPjwvdXJscz48L3JlY29yZD48L0NpdGU+PC9FbmROb3RlPgB=
ADDIN EN.CITE PEVuZE5vdGU+PENpdGU+PEF1dGhvcj5Lb2xvZHlhemhuaXk8L0F1dGhvcj48WWVhcj4yMDExPC9Z
ZWFyPjxSZWNOdW0+MTM2PC9SZWNOdW0+PERpc3BsYXlUZXh0Pls4Mi04NF08L0Rpc3BsYXlUZXh0
PjxyZWNvcmQ+PHJlYy1udW1iZXI+MTM2PC9yZWMtbnVtYmVyPjxmb3JlaWduLWtleXM+PGtleSBh
cHA9IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJlcXZkcHM5ZWFlZnh6dHJkZnN6Ij4xMzY8
L2tleT48L2ZvcmVpZ24ta2V5cz48cmVmLXR5cGUgbmFtZT0iSm91cm5hbCBBcnRpY2xlIj4xNzwv
cmVmLXR5cGU+PGNvbnRyaWJ1dG9ycz48YXV0aG9ycz48YXV0aG9yPktvbG9keWF6aG5peSwgVml0
YWxpeTwvYXV0aG9yPjxhdXRob3I+S3JlaWJpZywgU3lsdmlhIEQ8L2F1dGhvcj48YXV0aG9yPkdy
b3NzLCBKYW1lcyBKPC9hdXRob3I+PGF1dGhvcj5Sb3RoLCBXYWx0b24gVDwvYXV0aG9yPjxhdXRo
b3I+V2lsaGVsbSwgRnJhbmsgSDwvYXV0aG9yPjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48dGl0
bGVzPjx0aXRsZT5BbiBhZmZlY3RpdmUgY29tcHV0aW5nIGFwcHJvYWNoIHRvIHBoeXNpb2xvZ2lj
YWwgZW1vdGlvbiBzcGVjaWZpY2l0eTogVG93YXJkIHN1YmplY3TigJBpbmRlcGVuZGVudCBhbmQg
c3RpbXVsdXPigJBpbmRlcGVuZGVudCBjbGFzc2lmaWNhdGlvbiBvZiBmaWxt4oCQaW5kdWNlZCBl
bW90aW9uczwvdGl0bGU+PHNlY29uZGFyeS10aXRsZT5Qc3ljaG9waHlzaW9sb2d5PC9zZWNvbmRh
cnktdGl0bGU+PC90aXRsZXM+PHBlcmlvZGljYWw+PGZ1bGwtdGl0bGU+UHN5Y2hvcGh5c2lvbG9n
eTwvZnVsbC10aXRsZT48L3BlcmlvZGljYWw+PHBhZ2VzPjkwOC05MjI8L3BhZ2VzPjx2b2x1bWU+
NDg8L3ZvbHVtZT48bnVtYmVyPjc8L251bWJlcj48ZGF0ZXM+PHllYXI+MjAxMTwveWVhcj48L2Rh
dGVzPjxpc2JuPjE0NjktODk4NjwvaXNibj48dXJscz48L3VybHM+PC9yZWNvcmQ+PC9DaXRlPjxD
aXRlPjxBdXRob3I+TmFzb3o8L0F1dGhvcj48WWVhcj4yMDEwPC9ZZWFyPjxSZWNOdW0+MTM3PC9S
ZWNOdW0+PHJlY29yZD48cmVjLW51bWJlcj4xMzc8L3JlYy1udW1iZXI+PGZvcmVpZ24ta2V5cz48
a2V5IGFwcD0iRU4iIGRiLWlkPSJwYXN0ZDUyenJhZmRhdWVlcmVxdmRwczllYWVmeHp0cmRmc3oi
PjEzNzwva2V5PjwvZm9yZWlnbi1rZXlzPjxyZWYtdHlwZSBuYW1lPSJKb3VybmFsIEFydGljbGUi
PjE3PC9yZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRob3I+TmFzb3osIEZhdG1h
PC9hdXRob3I+PGF1dGhvcj5MaXNldHRpLCBDaHJpc3RpbmUgTDwvYXV0aG9yPjxhdXRob3I+VmFz
aWxha29zLCBBdGhhbmFzaW9zIFY8L2F1dGhvcj48L2F1dGhvcnM+PC9jb250cmlidXRvcnM+PHRp
dGxlcz48dGl0bGU+QWZmZWN0aXZlbHkgaW50ZWxsaWdlbnQgYW5kIGFkYXB0aXZlIGNhciBpbnRl
cmZhY2VzPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxlPkluZm9ybWF0aW9uIFNjaWVuY2VzPC9zZWNv
bmRhcnktdGl0bGU+PC90aXRsZXM+PHBlcmlvZGljYWw+PGZ1bGwtdGl0bGU+SW5mb3JtYXRpb24g
U2NpZW5jZXM8L2Z1bGwtdGl0bGU+PC9wZXJpb2RpY2FsPjxwYWdlcz4zODE3LTM4MzY8L3BhZ2Vz
Pjx2b2x1bWU+MTgwPC92b2x1bWU+PG51bWJlcj4yMDwvbnVtYmVyPjxkYXRlcz48eWVhcj4yMDEw
PC95ZWFyPjwvZGF0ZXM+PGlzYm4+MDAyMC0wMjU1PC9pc2JuPjx1cmxzPjwvdXJscz48L3JlY29y
ZD48L0NpdGU+PENpdGU+PEF1dGhvcj52YW4gZGVuIEJyb2VrPC9BdXRob3I+PFllYXI+MjAxMDwv
WWVhcj48UmVjTnVtPjEzODwvUmVjTnVtPjxyZWNvcmQ+PHJlYy1udW1iZXI+MTM4PC9yZWMtbnVt
YmVyPjxmb3JlaWduLWtleXM+PGtleSBhcHA9IkVOIiBkYi1pZD0icGFzdGQ1MnpyYWZkYXVlZXJl
cXZkcHM5ZWFlZnh6dHJkZnN6Ij4xMzg8L2tleT48L2ZvcmVpZ24ta2V5cz48cmVmLXR5cGUgbmFt
ZT0iQm9vayBTZWN0aW9uIj41PC9yZWYtdHlwZT48Y29udHJpYnV0b3JzPjxhdXRob3JzPjxhdXRo
b3I+dmFuIGRlbiBCcm9laywgRWdvbiBMPC9hdXRob3I+PGF1dGhvcj5MaXPDvSwgVmlsaWFtPC9h
dXRob3I+PGF1dGhvcj5KYW5zc2VuLCBKb3JpcyBIPC9hdXRob3I+PGF1dGhvcj5XZXN0ZXJpbmss
IEpveWNlIEhETTwvYXV0aG9yPjxhdXRob3I+U2NodXQsIE1hcmxlZW4gSDwvYXV0aG9yPjxhdXRo
b3I+VHVpbmVuYnJlaWplciwgS2VlczwvYXV0aG9yPjwvYXV0aG9ycz48L2NvbnRyaWJ1dG9ycz48
dGl0bGVzPjx0aXRsZT5BZmZlY3RpdmUgbWFuLW1hY2hpbmUgaW50ZXJmYWNlOiB1bnZlaWxpbmcg
aHVtYW4gZW1vdGlvbnMgdGhyb3VnaCBiaW9zaWduYWxzPC90aXRsZT48c2Vjb25kYXJ5LXRpdGxl
PkJpb21lZGljYWwgRW5naW5lZXJpbmcgU3lzdGVtcyBhbmQgVGVjaG5vbG9naWVzPC9zZWNvbmRh
cnktdGl0bGU+PC90aXRsZXM+PHBhZ2VzPjIxLTQ3PC9wYWdlcz48ZGF0ZXM+PHllYXI+MjAxMDwv
eWVhcj48L2RhdGVzPjxwdWJsaXNoZXI+U3ByaW5nZXI8L3B1Ymxpc2hlcj48aXNibj4zNjQyMTE3
MjAxPC9pc2JuPjx1cmxzPjwvdXJscz48L3JlY29yZD48L0NpdGU+PC9FbmROb3RlPgB=
ADDIN EN.CITE.DATA [82-84].5.3 Train a DBN Classifier on Physiological DataThe overall goal of this study is to train a single DBN classifier for all the subjects on the raw data from two EEG and two EOG channels to predict arousal, valence, and liking as binary classes (low or high). Classification accuracy defined in REF NumberRef7607235909 \h \* MERGEFORMAT (17) evaluates the performance. There are five steps in the experiment: raw data pre-processing, raw data selection and division, normalization, randomization, and DBN training and classification.In the first step, all signals are pre-processed by notch filtering at 50 Hz in order to remove power line disturbances and bandpass filters of 0.3 to 32 Hz and 10 to 32 Hz are also applied to EOG and EMG, respectively, as suggested by ADDIN EN.CITE <EndNote><Cite><Author>L?ngkvist</Author><Year>2012</Year><RecNum>23</RecNum><DisplayText>[34]</DisplayText><record><rec-number>23</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">23</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>L?ngkvist, M.</author><author>Karlsson, L.</author><author>Loutfi, A.</author></authors></contributors><titles><title>Sleep Stage Classification Using Unsupervised Feature Learning</title><secondary-title>Advances in Artificial Neural Systems</secondary-title></titles><periodical><full-title>Advances in Artificial Neural Systems</full-title></periodical><volume>2012</volume><dates><year>2012</year></dates><isbn>1687-7594</isbn><urls></urls><research-notes>DBNToolbox</research-notes></record></Cite></EndNote>[34].The second step selects raw data and divides them into training set and test set. A subject’s physiological signals are more likely to be elicited by a video at the end of the one-minute watching period as the plot develops. Therefore, it sounds reasonable to discard the first 50 second and use only the last 10 seconds’ data in each one-minute record. Then the 10 seconds’ data are broke down into 10 one-second segments. In other words, the learning process trains and classifies one-second samples, each in 512 dimensions (128Hz * 1s * 4channel). There are totally 12800 samples (32subject * 40video * 10sample). In each trial, the 10 samples of one randomly chosen video from each subject are left out for testing, resulting in the size of test set being 320 (32 subject * 1video * 10sample). The remaining 12480 (32subject * 39video * 10sample) samples go to the training set. The third step applies a channel-wise normalization to scale all the values to [0 1] according tochij= chij-min?(chi)maxchi-min?(chi) MACROBUTTON NumberReference \* MERGEFORMAT (28)where chi represents all the data in the channel i and chij is a data point in the channel i. The reason to normalize data this way is two-folded: a) the ranges of different channels may vary, so normalization makes them comparable when concatenating all channels as input, and b) a DBN’s node in the input layer has to have values between 0 and 1, to be treated as probabilities of activation of this node. Alternatively, the normalization could be done by saturating a signal at saturation constant max and min. Any values larger than max are set to max; any values smaller than min are set to min. Then apply REF NumberRef8144900203 \h \* MERGEFORMAT (28). This trick should be able to remove outliers, too, but it requires some knowledge about the data’s reasonable ranges. Since the DEAP dataset is quite clean, I did not find performance difference between these two normalization methods.Randomizing training samples is necessary because the mini-batch technique in training DBNs requires samples of each class are (at least roughly) evenly distributed.The last step firstly pre-trains a DBN without any labels, which means the same features learned in the pre-train stage can be used for the three different classification problems in the fine-tuning stage. Since the pre-trained model captures the properties of the data themselves, it can even be saved for unknown classification problems, as long as a new fine-tuning is applied to it using the new labeled data when they are available. After fine-tuning features based on labels and backpropagating to train a model for arousal classification, the model is used to predict arousal labels on the test set and compute the classification accuracy. The same process is applied on valence and liking classifiers.The same experiment runs 10 times to get the mean accuracy, as well as the standard deviation.The DBNToolbox matlab code published by ADDIN EN.CITE <EndNote><Cite><Author>L?ngkvist</Author><Year>2012</Year><RecNum>23</RecNum><DisplayText>[34]</DisplayText><record><rec-number>23</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">23</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>L?ngkvist, M.</author><author>Karlsson, L.</author><author>Loutfi, A.</author></authors></contributors><titles><title>Sleep Stage Classification Using Unsupervised Feature Learning</title><secondary-title>Advances in Artificial Neural Systems</secondary-title></titles><periodical><full-title>Advances in Artificial Neural Systems</full-title></periodical><volume>2012</volume><dates><year>2012</year></dates><isbn>1687-7594</isbn><urls></urls><research-notes>DBNToolbox</research-notes></record></Cite></EndNote>[34] is provided to perform the experiment. A DBN with two hidden layers, each layer with 50 nodes is constructed. Therefore the DBN structure for the pre-training stage is 512-50-50 and for the fine-tuning stage is 512-50-50-2. DBN parameters are listed below.Table SEQ Table \* ARABIC 5 DBN parametersUnsupervised learning rate 0.05Supervised learning rate0.05Number of epochs in pre-training50Number of epochs in fine-tuning20Mini-batch size in both stages1005.4 Experimental ResultsTo show the features learned in the first hidden layer, I simply take wij, where i is these nodes corresponding to one channel in the visible layer and j is a fixed node in the hidden layer, and draw a one-dimensional graph for node j on one channel. In this experiment, 200 such feature graphs (4channel * 50node) are grouped into 4 channels each of which is shown as a big graph matrix shown below.Fig. SEQ Fig. \* ARABIC 46 Learned EOG features. (top) hEOG (buttom) vEOGFig. SEQ Fig. \* ARABIC 47 Learned EMG features. (top) zEMG (bottom) tEMGThe whole experiment with 10 trials took about 1 hour on a Windows 7 machine with 3.0 G dual-core CPU and 4G memory. The means and standard deviations of the accuracy of arousal, valence, and liking are 0.609/0.074, 0.512/0.097, and 0.684/0.093, respectively, which are drawn in REF _Ref355733731 \h Fig. 48. The classification accuracy in ADDIN EN.CITE <EndNote><Cite><Author>Koelstra</Author><Year>2011</Year><RecNum>13</RecNum><DisplayText>[6]</DisplayText><record><rec-number>13</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">13</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Koelstra, S.</author><author>Muhl, C.</author><author>Soleymani, M.</author><author>Lee, J.</author><author>Yazdani, A.</author><author>Ebrahimi, T.</author><author>Pun, T.</author><author>Nijholt, A.</author><author>Patras, I.</author></authors></contributors><titles><title>Deap: A database for emotion analysis using physiological signals</title><secondary-title>Affective Computing, IEEE Transactions on</secondary-title></titles><periodical><full-title>Affective Computing, IEEE Transactions on</full-title></periodical><pages>1-1</pages><number>99</number><dates><year>2011</year></dates><isbn>1949-3045</isbn><urls></urls></record></Cite></EndNote>[6] is also depicted as filled dots for comparison purposes. Fig. SEQ Fig. \* ARABIC 48 Accuracy of DBN classification. Error bars for DBN classification accuracy on raw data and filled dots for Gaussian na?ve Bayes on featuresNote that the three major differences when comparing the two results. Firstly, the original paper employs a subject specific classification approach, but this study trains a universal model for all subjects. Secondly, the original paper uses all the 8 peripheral channels, compared to only 4 channels used in this work, which contain less information. Thirdly, the original work takes hand-crafted features; in contrast, this work simply feeds the raw data into the DBN. In this sense, this study solves a harder problem using DBNs.The liking accuracy of this work is higher than that of the original work; the arousal accuracy is slightly higher but not significant; however the valence accuracy is lower. There may be three reasons to explain the poorer valence classification performance: a) the 4 channels used in this work do not contain much useful information to discriminate different valences; b) the features of valence are too complex for the DBN in our settings to learn from the raw data; c) the inter-subject variability as claimed by ADDIN EN.CITE <EndNote><Cite><Author>Koelstra</Author><Year>2011</Year><RecNum>13</RecNum><DisplayText>[6]</DisplayText><record><rec-number>13</rec-number><foreign-keys><key app="EN" db-id="pastd52zrafdaueereqvdps9eaefxztrdfsz">13</key></foreign-keys><ref-type name="Journal Article">17</ref-type><contributors><authors><author>Koelstra, S.</author><author>Muhl, C.</author><author>Soleymani, M.</author><author>Lee, J.</author><author>Yazdani, A.</author><author>Ebrahimi, T.</author><author>Pun, T.</author><author>Nijholt, A.</author><author>Patras, I.</author></authors></contributors><titles><title>Deap: A database for emotion analysis using physiological signals</title><secondary-title>Affective Computing, IEEE Transactions on</secondary-title></titles><periodical><full-title>Affective Computing, IEEE Transactions on</full-title></periodical><pages>1-1</pages><number>99</number><dates><year>2011</year></dates><isbn>1949-3045</isbn><urls></urls></record></Cite></EndNote>[6] is too large for a single model to capture. The distribution of DBN’s classification performance in each subject is depicted below.Fig. SEQ Fig. \* ARABIC 49 Histogram of accuracy distribution in 32 subjects5.5 DiscussionThis work presents a system to apply DBNs to learn features from raw physiological data and predict emotions. The trained universal model for all subjects in the DEAP dataset shows that DBNs are capable to learn useful features in an unsupervised fashion and have comparable classification performance with Gaussian na?ve Bayes on hand-crafted features. This result suggests the possibilities of further applications of DBNs on physiological signal analysis and even other fields of similar complexity. Chapter 6 ConclusionLeveraging the recent breakthrough in deep neural networks provides means to learn deep layered hierarchical representation of data. This brings us new possibilities to be explored and new problems to be solved. My research focuses on developing practical pipelines, frameworks, and systems on real-world deep learning applications. Particularly, I have studied a proper way to search for the optimal deep learning structures and pre-processing techniques, a new active labeling framework for cost-effective selection of labeled data, and a pipeline to apply deep learning to emotion prediction via physiological sensor data.I empirically examine the optimal meta-parameters of deep learning networks in terms of number of layers, number of nodes, and learning unit types, as well as the effects of various data pre-processing techniques on the benchmark MNIST hand-written digit dataset and a sleep stage dataset side by side. This is the first such kind of comprehensive investigation. The experimental results show some settings are sensitive to the number of hidden layers, some are sensitive to the number of nodes in each hidden layer, and the others are not sensitive to the network structure at all. This suggests the optimal meta-parameters are highly application dependent. Inspired by the general active learning framework, I propose active labeling deep learning based on three metrics: least confidence, margin sampling, and entropy. This is the first work that fits deep learning into the active learning framework, to address the practical needs for cost-effective selection of labeled data in the paradigm of semi-supervised learning. On the MNIST dataset, the proposed methods outperform random labeling by 2%-4%, suggesting the usefulness of active labeling deep learning on clean datasets. On the other hand, the new method performs similarly to random labeling on the sleep stage dataset due to the noisiness and inconsistency in the data.I also propose a pipeline to apply deep learning to emotion prediction via physiological sensor data. This is the first system for modeling physiological data by extracting features automatically and using extracted features to predict emotions. The developed system has three advantages: 1) it does not require expert knowledge for extracting features; 2) the features learned in the unsupervised learning stage can be used by multiple tasks; and 3) the classification accuracy is 0.690, 0.512, and 0.684, for the levels of arousal, valence, and liking, respectively, which is comparable with existing methods based on expert designed features.Reference ADDIN EN.REFLIST [1]Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, "Greedy layer-wise training of deep networks," Advances in neural information processing systems, vol. 19, p. 153, 2007.[2]G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, pp. 504-507, 2006.[3]P. Smolensky, "Information processing in dynamical systems: Foundations of harmony theory," in Parallel distributed processing: explorations in the microstructure of cognition, ed Cambridge, MA, USA: MIT Press, 1986, pp. 194-281.[4]B. Settles, "Active learning literature survey," University of Wisconsin, Madison, 2010.[5]S. H. Fairclough, "Fundamentals of physiological computing," Interact. Comput., vol. 21, pp. 133-145, 2009.[6]S. Koelstra, C. Muhl, M. Soleymani, J. Lee, A. Yazdani, T. Ebrahimi, et al., "Deap: A database for emotion analysis using physiological signals," Affective Computing, IEEE Transactions on, pp. 1-1, 2011.[7]M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken, "Multilayer feedforward networks with a nonpolynomial activation function can approximate any function," Neural networks, vol. 6, pp. 861-867, 1993.[8]K.-I. Funahashi, "On the approximate realization of continuous mappings by neural networks," Neural networks, vol. 2, pp. 183-192, 1989.[9]J. Hastad, "Almost optimal lower bounds for small depth circuits," in Proceedings of the eighteenth annual ACM symposium on Theory of computing, 1986, pp. 6-20.[10]G. E. Hinton, S. Osindero, and Y. W. Teh, "A fast learning algorithm for deep belief nets," Neural computation, vol. 18, pp. 1527-1554, 2006.[11]G. E. Hinton, "Training products of experts by minimizing contrastive divergence," Neural computation, vol. 14, pp. 1771-1800, 2002.[12]G. Hinton, "A practical guide to training restricted Boltzmann machines," Momentum, vol. 9, p. 1, 2010.[13]Y. Le Cun, "Modèles connexionnistes de l'apprentissage," 1987.[14]H. Bourlard and Y. Kamp, "Auto-association by multilayer perceptrons and singular value decomposition," Biological cybernetics, vol. 59, pp. 291-294, 1988.[15]G. E. Hinton and R. S. Zemel, "Autoencoders, minimum description length, and Helmholtz free energy," Advances in neural information processing systems, pp. 3-3, 1994.[16]X. Zhu, "Semi-supervised learning literature survey," 2005.[17]Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, pp. 2278-2324, 1998.[18]D. F. Wulsin, J. R. Gupta, R. Mani, J. A. Blanco, and B. Litt, "Modeling electroencephalography waveforms with semi-supervised deep belief nets: fast classification and anomaly measurement," Journal of Neural Engineering, vol. 8, p. 036015, 2011.[19]D. Erhan, Y. Bengio, A. Courville, and P. Vincent, "Visualizing higher-layer features of a deep network," Technical report, University of Montreal2009.[20]H. Lee, C. Ekanadham, and A. Ng, "Sparse deep belief net model for visual area V2," Advances in neural information processing systems, vol. 20, pp. 873-880, 2008.[21]H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng, "Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations," in Proceedings of the 26th Annual International Conference on Machine Learning, 2009, pp. 609-616.[22]A. Ng. (2013). UFLDL Tutorial. Available: [23]G. E. Hinton, "To recognize shapes, first learn to generate images," Progress in brain research, vol. 165, pp. 535-547, 2007.[24]P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, "Extracting and composing robust features with denoising autoencoders," in Proceedings of the 25th international conference on Machine learning, 2008, pp. 1096-1103.[25]P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol, "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion," The Journal of Machine Learning Research, vol. 9999, pp. 3371-3408, 2010.[26]F. Seide, G. Li, and D. Yu, "Conversational Speech Transcription Using Context-Dependent Deep Neural Networks," in INTERSPEECH, 2011, pp. 437-440.[27]G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," Signal Processing Magazine, IEEE, vol. 29, pp. 82-97, 2012.[28]Y. Bengio, "Neural net language models," Scholarpedia, vol. 3, p. 3881, 2008.[29]R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, "Natural language processing (almost) from scratch," The Journal of Machine Learning Research, vol. 12, pp. 2493-2537, 2011.[30]A. Krizhevsky, I. Sutskever, and G. Hinton, "Imagenet classification with deep convolutional neural networks," in Advances in Neural Information Processing Systems 25, 2012, pp. 1106-1114.[31]G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, "Improving neural networks by preventing co-adaptation of feature detectors," arXiv preprint arXiv:1207.0580, 2012.[32]I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, "Maxout networks," arXiv preprint arXiv:1302.4389, 2013.[33]A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, et al., "Physiobank, physiotoolkit, and physionet components of a new research resource for complex physiologic signals," Circulation, vol. 101, pp. e215-e220, 2000.[34]M. L?ngkvist, L. Karlsson, and A. Loutfi, "Sleep Stage Classification Using Unsupervised Feature Learning," Advances in Artificial Neural Systems, vol. 2012, 2012.[35]A. R. Osborne and A. Provenzale, "Finite correlation dimension for stochastic systems with power-law spectra," Physica D: Nonlinear Phenomena, vol. 35, pp. 357-381, 1989.[36]L. Zoubek, S. Charbonnier, S. Lesecq, A. Buguet, and F. Chapotot, "Feature selection for sleep/wake stages classification using data driven methods," Biomedical Signal Processing and Control, vol. 2, pp. 171-179, 2007.[37]E. Pereda, A. Gamundi, R. Rial, and J. Gonzalez, "Non-linear behaviour of human EEG: fractal exponent versus correlation dimension in awake and sleep stages," Neuroscience letters, vol. 250, pp. 91-94, 1998.[38]T. Gasser, P. B?cher, and J. M?cks, "Transformations towards the normal distribution of broad band spectral parameters of the EEG," Electroencephalography and clinical neurophysiology, vol. 53, pp. 119-124, 1982.[39]S. Zhou, Q. Chen, and X. Wang, "Active deep networks for semi-supervised sentiment classification," in Proceedings of the 23rd International Conference on Computational Linguistics: Posters, 2010, pp. 1515-1523.[40]D. D. Lewis and W. A. Gale, "A sequential algorithm for training text classifiers," in Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, 1994, pp. 3-12.[41]S. Tong, "Active learning: theory and applications," Citeseer, 2001.[42]T. Scheffer, C. Decomain, and S. Wrobel, "Active hidden markov models for information extraction," in Advances in Intelligent Data Analysis, ed: Springer, 2001, pp. 309-318.[43]A. McCallum and K. Nigam, "Employing EM and Pool-Based Active Learning for Text Classification," in ICML, 1998, pp. 350-358.[44]S. Tong and D. Koller, "Support vector machine active learning with applications to text classification," The Journal of Machine Learning Research, vol. 2, pp. 45-66, 2002.[45]S. C. Hoi, R. Jin, and M. R. Lyu, "Large-scale text categorization by batch mode active learning," in Proceedings of the 15th international conference on World Wide Web, 2006, pp. 633-642.[46]C. A. Thompson, M. E. Califf, and R. J. Mooney, "Active learning for natural language parsing and information extraction," in ICML, 1999, pp. 406-414.[47]B. Settles and M. Craven, "An analysis of active learning strategies for sequence labeling tasks," in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2008, pp. 1070-1079.[48]S. Tong and E. Chang, "Support vector machine active learning for image retrieval," in Proceedings of the ninth ACM international conference on Multimedia, 2001, pp. 107-118.[49]C. Zhang and T. Chen, "An active learning framework for content-based information retrieval," Multimedia, IEEE Transactions on, vol. 4, pp. 260-268, 2002.[50]R. Yan, J. Yang, and A. Hauptmann, "Automatically labeling video data using multi-class active learning," in Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on, 2003, pp. 516-523.[51]A. G. Hauptmann, W.-H. Lin, R. Yan, J. Yang, and M.-Y. Chen, "Extreme video retrieval: joint maximization of human and computer performance," in Proceedings of the 14th annual ACM international conference on Multimedia, 2006, pp. 385-394.[52]G. Tur, D. Hakkani-Tür, and R. E. Schapire, "Combining active and semi-supervised learning for spoken language understanding," Speech Communication, vol. 45, pp. 171-186, 2005.[53]N. Roy and A. McCallum, "Toward optimal active learning through monte carlo estimation of error reduction," ICML, Williamstown, 2001.[54]C. E. Shannon and W. Weaver, "A mathematical theory of communication," ed: American Telephone and Telegraph Company, 1948.[55]R. R. Cornelius, The science of emotion: Research and tradition in the psychology of emotions vol. 133001539: Prentice Hall Upper Saddle River, NJ, 1996.[56]D. Sander, D. Grandjean, and K. R. Scherer, "2005 Special Issue: A systems approach to appraisal mechanisms in emotion," Neural networks, vol. 18, pp. 317-352, 2005.[57]D. Novak, M. Mihelj, and M. Munih, "A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing," Interacting with Computers, vol. 24, pp. 154-172, 2012.[58]D. McDuff, A. Karlson, A. Kapoor, A. Roseway, and M. Czerwinski, "AffectAura: an intelligent system for emotional memory," in The ACM SIGCHI Conference on Human Factors in Computing Systems, Austin, TX, 2012, pp. 849-858.[59]E. W. Boyer, R. Fletcher, R. J. Fay, D. Smelson, D. Ziedonis, and R. W. Picard, "Preliminary Efforts Directed Toward the Detection of Craving of Illicit Substances: The iHeal Project," Journal of Medical Toxicology, pp. 1-5, 2012.[60]K. Plarre, A. Raij, S. M. Hossain, A. A. Ali, M. Nakajima, M. Al'absi, et al., "Continuous inference of psychological stress from sensory measurements collected in the natural environment," in Information Processing in Sensor Networks (IPSN), 2011 10th International Conference on, Chicago, IL, USA, 2011, pp. 97-108.[61]S. D. Kreibig, "Autonomic nervous system activity in emotion: A review," Biological Psychology, vol. 84, pp. 394-421, 2010.[62]J. Wagner, E. Andre, and F. Jung, "Smart sensor integration: A framework for multimodal emotion recognition in real-time," in Affective Computing and Intelligent Interaction and Workshops, 2009. ACII 2009. 3rd International Conference on, 2009, pp. 1-8.[63]M. E. Miiller, "Why Some Emotional States Are Easier to be Recognized Than Others: A thorough data analysis and a very accurate rough set classifier," in Systems, Man and Cybernetics, 2006. SMC '06. IEEE International Conference on, 2006, pp. 1624-1629.[64]C. Peter, E. Ebert, and H. Beikirch, "A wearable multi-sensor system for mobile acquisition of emotion-related physiological data," Affective Computing and Intelligent Interaction, pp. 691-698, 2005.[65]R. W. Picard, "Affective computing: challenges," International Journal of Human-Computer Studies, vol. 59, pp. 55-64, 2003.[66]M. M. Bradley and P. J. Lang, "Measuring emotion: Behavior, feeling, and physiology," Cognitive neuroscience of emotion, vol. 25, pp. 49-59, 2000.[67]R. A. Calvo and S. D'Mello, "Affect detection: An interdisciplinary review of models, methods, and their applications," Affective Computing, IEEE Transactions on, vol. 1, pp. 18-37, 2010.[68]G. Chanel, J. J. Kierkels, M. Soleymani, and T. Pun, "Short-term emotion assessment in a recall paradigm," International Journal of Human-Computer Studies, vol. 67, pp. 607-627, 2009.[69]J. Kim and E. André, "Emotion recognition based on physiological changes in music listening," Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 30, pp. 2067-2083, 2008.[70]P. Rainville, A. Bechara, N. Naqvi, and A. R. Damasio, "Basic emotions are associated with distinct patterns of cardiorespiratory activity," International journal of psychophysiology, vol. 61, pp. 5-18, 2006.[71]S. Thrun, "Is learning the n-th thing any easier than learning the first?," Advances in neural information processing systems, pp. 640-646, 1996.[72]S. Thrun, "Learning to learn: Introduction," in In Learning To Learn, 1996.[73]A. Krizhevsky and G. Hinton, "Learning multiple layers of features from tiny images," Master's thesis, Department of Computer Science, University of Toronto, 2009.[74]P. Vincent, H. Larochelle, Y. Bengio, and P. A. Manzagol, "Extracting and composing robust features with denoising autoencoders," in Proceedings of the 25th international conference on Machine learning, 2008, pp. 1096-1103.[75]G. W. Taylor, G. E. Hinton, and S. T. Roweis, "Modeling human motion using binary latent variables," Advances in neural information processing systems, vol. 19, p. 1345, 2007.[76]C. A. Frantzidis, C. Bratsas, M. A. Klados, E. Konstantinidis, C. D. Lithari, A. B. Vivas, et al., "On the classification of emotional biosignals evoked while viewing affective pictures: an integrated data-mining-based approach for healthcare applications," Information Technology in Biomedicine, IEEE Transactions on, vol. 14, pp. 309-318, 2010.[77]C. Setz, B. Arnrich, J. Schumm, R. La Marca, G. Troster, and U. Ehlert, "Discriminating stress from cognitive load using a wearable EDA device," Information Technology in Biomedicine, IEEE Transactions on, vol. 14, pp. 410-417, 2010.[78]D. J. Hand and K. Yu, "Idiot's Bayes—not so stupid after all?," International Statistical Review, vol. 69, pp. 385-398, 2001.[79]G. E. Sakr, I. H. Elhajj, and H.-S. Huijer, "Support vector machines to define and detect agitation transition," Affective Computing, IEEE Transactions on, vol. 1, pp. 98-108, 2010.[80]D. Wu, C. G. Courtney, B. J. Lance, S. S. Narayanan, M. E. Dawson, K. S. Oie, et al., "Optimal arousal identification and classification for affective computing using physiological signals: virtual reality Stroop task," Affective Computing, IEEE Transactions on, vol. 1, pp. 109-118, 2010.[81]J. N. Bailenson, E. D. Pontikakis, I. B. Mauss, J. J. Gross, M. E. Jabon, C. A. Hutcherson, et al., "Real-time classification of evoked emotions using facial feature tracking and physiological responses," International journal of human-computer studies, vol. 66, pp. 303-317, 2008.[82]V. Kolodyazhniy, S. D. Kreibig, J. J. Gross, W. T. Roth, and F. H. Wilhelm, "An affective computing approach to physiological emotion specificity: Toward subject‐independent and stimulus‐independent classification of film‐induced emotions," Psychophysiology, vol. 48, pp. 908-922, 2011.[83]F. Nasoz, C. L. Lisetti, and A. V. Vasilakos, "Affectively intelligent and adaptive car interfaces," Information Sciences, vol. 180, pp. 3817-3836, 2010.[84]E. L. van den Broek, V. Lis?, J. H. Janssen, J. H. Westerink, M. H. Schut, and K. Tuinenbreijer, "Affective man-machine interface: unveiling human emotions through biosignals," in Biomedical Engineering Systems and Technologies, ed: Springer, 2010, pp. 21-47.Publications[1] Dan Wang and Yi Shang, “Modeling Physiological with Deep Belief Networks,” in International Journal of Information and Education Technology, vol. 3, no. 5, pp. 505-511, 2013.[2] Dan Wang and Yi Shang, “Active Labeling in Deep Learning”, to be submitted.[3] Dan Wang, Peng Zhuang, and Yi Shang, "A new framework for multi-source geo-social based mobile classifieds searches," in Artificial Neural Networks In Engineering 2010, St. Louis, MO, USA, 2010, pp. 129-135.[4] Yi Shang, Wenjun Zeng, Dominic K. Ho, Dan Wang, Qia Wang, Yue Wang, Tiancheng Zhuang, Aleksandre Lobzhanidze, Liyang Rui, “Nest: NEtworked Smartphones for Target localization”, in 2012 IEEE Consumer Communications and Networking Conference (CCNC), Las Vegas, NV, USA, 2012, pp. 732-736.[5] Peng Zhuang, Dan Wang, and Yi Shang, "SMART: Simultaneous indoor localization and map construction using smartphones," in 2010 International Joint Conference on Neural Networks (IJCNN2010), Barcelona, Spain, 2010?IEEE. doi: 10.1109/IJCNN.2010.5596552.[6] Peng Zhuang, Dan Wang, and Yi Shang, "Distributed faulty sensor detection," in 2009 IEEE Global Telecommunications Conference, Honolulu, Hawaii, USA, 2009?IEEE. doi: 10.1109/GLOCOM.2009.5425702.VITADan Wang was born in Chengdu, China. He is currently a PhD student in Computer Science Department at the University of Missouri, Columbia, MO 65211, USA. He got MS in Communication and Information System and BS in Communications Engineering from Southwest Jiaotong University, Chengdu, China, in 2007 and 2003, respectively. He has published 5 papers. His research interests include mobile computing, machine learning, and wireless sensor networks. He was a summer intern with Amazon Web Services in Seattle, WA, in 2011, 2012, and 2013. He will be joining Google as a software engineer. ................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related searches
- university of missouri high school online
- university of missouri high school program
- university of missouri homeschool program
- university of missouri hospital jobs
- university of missouri jobs columbia
- university of missouri high school k 12
- university of missouri human resources
- university of missouri hr
- university of missouri transcripts
- university of missouri columbia careers
- university of missouri health care jobs
- university of missouri health jobs