STAT 515 -- Chapter 11: Regression



STAT 509 – Sections 6.1-6.2: Linear Regression

• Mostly we have studied the behavior of a single random variable.

• Often, however, we gather data on two random variables.

Response Variable (Y): Measures the major outcome of interest in the study (also called the dependent variable).

Independent Variable (X): Another variable whose value explains, predicts, or is associated with the value of the response variable (also called the predictor or the regressor).

• We wish to determine: Is there a relationship between the two r.v.’s?

• Can we use the values of one r.v. to predict the other r.v.?

Observational Studies vs. Designed Experiments

• In observational studies, we simply measure or observe both variables on a set of sampled individuals.

• In a designed experiment, we manipulate the predictors (factors), setting them at specific values of interest. We then observe what values of the response correspond to the fixed predictor values.

Example 1 (Table 6.1): We observe the Rockwell Hardness (X) and Young’s modulus (Y) for seven high-density metals. The resulting data were:

X: 41 41 44 40 43 15 40

Y: 310 340 380 317 413 62 119

Example 2 (Table 6.3): A chemical engineering class studied the effect of the reflux ratio (X) on the ethanol concentration (Y) of an ethanol-water distillation. For a variety of settings of the reflux ratio, the ethanol concentration was measured:

X: 20 30 40 50 60

Y: 0.446 0.601 0.786 0.928 0.950

We assume there is random error in the observed response values, implying a probabilistic relationship between the 2 variables.

• Often we assume a straight-line relationship between two variables.

• This is known as simple linear regression.

Yi = β0 + β1xi + εi

Yi = ith response value β0 = Intercept of regression line

xi = ith predictor value β1 = slope of regression line

εi = ith random error component

• We assume the random errors εi have mean 0 (and variance σ2), so that E(Y) = β0 + β1x.

• Typically, in practice, β0 and β1 are unknown parameters. We estimate them using the sample data.

Fitting the Model (Least Squares Method)

• If we gather data (Xi, Yi) for several individuals, we can use these data to estimate β0 and β1 and thus estimate the linear relationship between Y and X.

• First step: Decide if a straight-line relationship between Y and X makes sense.

Plot the bivariate data using a scatter plot.

R code:

> x y plot(x,y,pch=19)

• Once we settle on the “best-fitting” regression line, its equation gives a predicted Y-value for any new X-value.

• How do we decide, given a data set, which line is the best-fitting line?

Note that usually, no line will go through all the points in the data set.

For each point, the residual =

(Some positive residuals, some negative residuals)

We want the line that makes these errors as small as possible (so that the line is “close” to the points).

Least-squares method: We choose the line that minimizes the sum of all the squared residuals (SSres).

SSres =

Least squares prediction equation:

[pic]

where [pic]and [pic]are the estimates of β0 and β1 that produce the best-fitting line in the least squares sense.

Formulas for[pic]and [pic]:

Estimated slope and intercept:

[pic] and [pic]

where [pic] and [pic]

and n = the number of observations.

Example (see Table 6.4):

[pic] = [pic] =

[pic] = [pic] =

SSxy =

SSxx =

R code:

> x y lm(y ~ x)

Derivation of Formulas for[pic]and [pic]:

Recall that SSres =

To minimize the SSres with respect to [pic]and [pic]:

Interpretations:

Slope:

Intercept:

Example:

Avoid extrapolation: predicting/interpreting the regression line for X-values outside the range of X in the data set.

Model Assumptions

• Recall model equation: Yi = β0 + β1xi + εi

• To perform inference about our regression line, we need to make certain assumptions about the random error component, εi. We assume:

1) The mean of εi is 0. (In the long run, the values of the random errors average zero.)

2) The variance of the probability distribution of εi is constant for all values of X. We denote the variance of εi by σ2.

3) The probability distribution of εi is normal.

4) The values of εi for any two observed Y-values are independent – the value of εi has no effect on the value of εj for the ith and jth Y-values.

Picture:

We will discuss later how to check these assumptions for a particular data set.

Estimating σ2

Typically the error variance σ2 is unknown.

An unbiased estimate of σ2 is the mean squared residual (MSres).

MSres = SSres / (n–2)

where SSres = SSyy - [pic]SSxy

and [pic]

Note that an estimate of σ is

[pic]

Testing the Usefulness of the Model

For the SLR model, E(Y) = β0 + β1x.

Note: X is completely useless in helping to predict or explain Y if and only if β1 = 0.

So to test the usefulness of the model for predicting Y, we test:

If we reject H0 and conclude Ha is true, then we conclude that X does provide information for the prediction of Y.

Picture:

Recall that the estimate [pic]is a statistic that depends on the sample data.

This [pic] has a sampling distribution.

If our four SLR assumptions hold, the sampling distribution of [pic] is normal with mean β1 and standard deviation which we estimate by

Under H0: β1 = 0, the statistic [pic]

has a t-distribution with n – 2 d.f.

Test about the Slope

One-Tailed Tests Two-Tailed Test

H0: β1 = 0 H0: β1 = 0 H0: β1 = 0

Ha: β1 < 0 Ha: β1 > 0 Ha: β1 ≠ 0

Test statistic: t = [pic]

Rejection region:

t < -tα, n-2 t > tα, n-2 t > tα/2 or t < -tα/2

P-value:

left tail area right tail area 2*(tail area outside t)

outside t outside t

Example: In the ethanol example, recall [pic]=

Is the real β1 significantly greater than 0?

(Use α = .05.)

A 100(1 – α)% Confidence Interval for the true slope β1 is given by:

where tα/2 is based on n – 2 d.f.

In our example, a 95% CI for β1 is:

R code:

> x y summary(lm(y ~ x))

> plot(x, y, pch=19); abline(lm(y ~ x))

Correlation

The scatterplot gives us a general idea about whether there is a linear relationship between two variables.

More precise: The coefficient of correlation (denoted r) is a numerical measure of the strength and direction of the linear relationship between two variables.

Formula for r (the correlation coefficient between two variables X and Y):

[pic]

Most computer packages will also calculate the correlation coefficient.

Interpreting the correlation coefficient:

• Positive r => The two variables are positively associated (large values of one variable correspond to large values of the other variable)

• Negative r => The two variables are negatively associated (large values of one variable correspond to small values of the other variable)

• r = 0 => No linear association between the two variables.

Note: -1 ≤ r ≤ 1 always.

How far r is from 0 measures the strength of the linear relationship:

• r nearly 1 => Strong positive relationship between the two variables

• r nearly -1 => Strong negative relationship between the two variables

• r near 0 => Weak relationship between the two variables

Pictures:

Example (Rockwell hardness / Young’s modulus data):

> rock young cor(rock, young)

[1] 0.7759845

Interpretation?

Notes: (1) Correlation makes no distinction between predictor and response variables.

(2) Variables must be numerical to calculate r.

(3) Correlation only measures the linear association between two variables, not any nonlinear relationship.

The square of the correlation coefficient is called the coefficient of determination, R2.

Interpretation: R2 represents the proportion of sample variability in Y that is explained by its linear relationship with X.

[pic] (R2 always between 0 and 1)

For the Rockwell hardness / Young’s modulus data example, R2 =

Interpretation:

For the reflux ratio / ethanol concentration data example, R2 =

Interpretation:

Estimation and Prediction with the Regression Model

Major goals in using the regression model:

(1) Determining the linear relationship between Y and X (accomplished through inferences about β1)

(2) Estimating the mean value of Y, denoted E(Y), for a particular value of X.

Example: Among all columns with reflux ratio 35 units, what is the estimated mean ethanol concentration?

(3) Predicting the value of Y for a particular value of X.

Example: For a “new” column having reflux ratio 35 units, what is the predicted ethanol concentration?

• The point estimate for these last two quantities is the same; it is:

Example:

• However, the variability associated with these point estimates is very different.

• Which quantity has more variability, a single Y-value or the mean of many Y-values?

This is seen in the following formulas:

100(1 – α)% Confidence Interval for the mean value of Y at X = x0:

where tα/2 based on n – 2 d.f.

100(1 – α)% Prediction Interval for the an individual new value of Y at X = x0:

where tα/2 based on n – 2 d.f.

The extra “1” inside the square root shows the prediction interval is wider than the CI, although they have the same center.

Note: A “Prediction Interval” attempts to contain a random quantity, while a confidence interval attempts to contain a (fixed) parameter value.

The variability in our estimate of E(Y) reflects the fact that we are merely estimating the unknown β0 and β1.

The variability in our prediction of the new Y includes that variability, plus the natural variation in the Y-values.

Example (ethanol concentration data):

95% CI for E(Y) with X = 35:

> x y predict(lm(y ~ x), data.frame(x = c(35)), interval="confidence", level=0.95)

95% PI for a new Y having X = 35:

> predict(lm(y ~ x), data.frame(x = c(35)), interval="prediction", level=0.95)

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download