Linear Models for Regression - College of Engineering

Linear Models for Regression

Key concepts:

? Sum of squared error for regression loss ? Gradient descent for loss minimization ? Closed form solution for minimizing SSE ? Maximum likelihood estimation and SSE's probabilistic interpretation ? Overfitting and regularization

CS534

Example Regression Problems

? Predict housing price based on

? House size, lot size, Location, # of rooms ...

? Predict stock price based on

? Price history of the past month ...

? Predict the abundance of a species based on

? Environmental conditions, shrubs? trees? ...

A Basic Set Up

Given: a training set, containing a total of training examples x, , for = 1, ... , Goal: learn a function to minimize some loss function on the training data

To begin, we consider a linear hypothesis space (thus the name linear regression):

Let x = 1, 1, 2, ... , x = 0 + 11 + + = wx

where = 0, 1, ... ,

A Popular Loss Function: Sum-Squared Error

? Given a set of training examples (x1, 1), x2, 2 ..., (x, )

? Hypothesis space (representation): x = wx

? Loss function (objective):

1 = 2

x -

2

=1

Optimization of the loss function

? Many ways to optimize this objective function ? A simple approach is gradient descent

? Start with some random guess of the parameter ? Iteratively improve the parameter by following the steepest

descent direction

? Gradient: multivariate generalization of derivative, points in the direction of greatest rate of increase of the function

From Wikipedia Values of the function in black and white, black representing higher values

Gradient represented by blue arrows.

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download