Multiclass Logistic Regression

Multiclass Logistic Regression

Sargur N. Srihari

University at Buffalo, State University of New York USA

Machine Learning

Srihari

Topics in Linear Classification using Probabilistic Discriminative Models

? Generative vs Discriminative 1. Fixed basis functions in linear classification 2. Logistic Regression (two-class) 3. Iterative Reweighted Least Squares (IRLS) 4. Multiclass Logistic Regression 5. Probit Regression 6. Canonical Link Functions

2

Machine Learning

Srihari

Topics in Multiclass Logistic Regression

? Multiclass Classification Problem ? Softmax Regression ? Softmax Regression Implementation ? Softmax and Training ? One-hot vector representation ? Objective function and gradient ? Summary of concepts in Logistic Regression ? Example of 3-class Logistic Regression

3

Machine Learning

Srihari

Multi-class Classification problem

Categories K=10

Examples N=100

4

Machine Learning

Srihari

Softmax Regression

? In the two-class case p(C1|) =y() = (wT+b)

where =[1,.., M]T, w =[w1,.., wM]T and a=wT+b is the "activation"

? For K classes, we work with soft-max function instead of logistic sigmoid (Softmax regression)

p(Ck | ) = yk() =

exp(ak ) j exp(aj )

where ak=wkT +bk, k =1,..,K wk =[wk1,.., wkM]T and a={a1,..aK}

? We learn a set of K weight vectors {w1,.., wK}and biases b

? Arranging weight vectors as a matrix W

a =WT+b

W

=

w1

. . wK

=

w11

. . wK 1

w1M wKM

y = softmax(a)

yi =

exp(ai )

3

exp(aj )

j =1

5

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download