BASIC STATISTICS Random Sample. - Iowa State University

[Pages:20]BASIC STATISTICS

1. SAMPLES, RANDOM SAMPLING AND SAMPLE STATISTICS

1.1. Random Sample. The random variables X1, X2, ..., Xn are called a random sample of size n from the population f(x) if X1, X2, ..., Xn are mutually independent random variables and the marginal probability density function of each Xi is the same function of f(x). Alternatively, X1, X2, ..., Xn are called independent and identically distributed random variables with pdf f(x). We abbreviate independent and identically distributed as iid.

Most experiments involve n >1 repeated observations on a particular variable, the first observation is X1, the second is X2, and so on. Each Xi is an observation on the same variable and each Xi has a marginal distribution given by f(x). Given that the observations are collected in such a way that the value of one observation has no effect or relationship with any of the other observations, the X1, X2, ..., Xn are mutually independent. Therefore we can write the joint probability density for the sample X1, X2, ..., Xn as

n

f (x1, x2, ..., xn) = f (x1)f (x2) ? ? ? f (xn) = f (xi)

(1)

i=1

If the underlying probability model is parameterized by , then we can also write

n

f (x1, x2, ..., xn|) = f (xi|)

(2)

i=1

Note that the same is used in each term of the product, or in each marginal density. A different value of would lead to a different properties for the random sample.

1.2. Statistics. Let X1, X2, ..., Xn be a random sample of size n from a population and let T (x1, x2, ..., xn) be a real valued or vector valued function whose domain includes the sample space of (X1, X2, ..., Xn). Then the random variable or random vector Y = (X1, X2, ..., Xn) is called a statistic. A statistic is a map from the sample space of (X1, X2, ..., Xn) call it X, to some space of values, usually R1 or Rn. T is what we compute when we observe the random variable X take on some specific values in a sample. The probability distribution of a statistic Y = T(X) is called the sampling distribution of Y. Notice that T(?) is a function of sample values only, it does not depend on any underlying parameters, .

1.3. Some Commonly Used Statistics.

1.3.1. Sample mean. The sample mean is the arithmetic average of the values in a random sample. It is usually denoted

X? (X1, X2, ? ? ?, Xn) =

X1 + X2 + ... + Xn n

=

1 n

n

Xi

(3)

i=1

The observed value of X? in any sample is demoted by the lower case letter, i.e., x?.

Date: September 27, 2004. 1

2

BASIC STATISTICS

1.3.2. Sample variance. The sample variance is the statistic defined by

S2(X1, X2, ? ? ?, Xn)

=

1 n-

1

n

(Xi - X? )2

(4)

i=1

The observed value of S2 in any sample is demoted by the lower case letter, i.e., s2.

1.3.3. Sample standard deviation. The sample standard deviation is the statistic defined by

S = S2

(5)

1.3.4. Sample midrange. The sample mid-range is the statistic defined by

max(X1, X2, ? ? ?, Xn) - min(X1, X2, ? ? ?, Xn)

(6)

2

1.3.5. Empirical distribution function. The empirical distribution function is defined by

F^(X1, X2, ? ?

?, Xn)(x)

=

1 n

n

I(Xi < x)

(7)

i=1

where F^(X1, X2, ???, Xn)(x) means we are evaluating the statistic F^(X1, X2, ???, Xn) at the particular value x. The random sample X1, X2, ..., Xn is assumed to come from a probability defined on R1

and I(A) is the indicator of the event A. This statistic takes values in the set of all distribution functions on R1. It estimates the function valued parameter F defined by its evaluation at x R1

F (P )(x) = P [X < x]

(8)

2. DISTRIBUTION OF SAMPLE STATISTICS

2.1. Theorem 1 on squared deviations and sample variances.

Theorem 1.

Let x1, x2, ? ? ?xn be any numbers and let x?

=

x1

+x2

+...+xn n

.

Then the following two items

hold.

a: b:

mina

n i=1

(xi

(n - 1)s2 =

- a)2 = ni=1(xi -

ni=1(xi x?)2 =

- x?)2

n i=1

x2i

- nx?2

Part a says that the sample mean is the value about which the sum of squared deviations is minimized. Part b is a simple identity that will prove immensely useful in dealing with statistical data.

Proof. First consider part a of theorem 1. Add and subtract x? from the expression on the lefthand side in part a and then expand as follows

n

n

n

n

(xi - x? + x? - a)2 = (xi - x?)2 + 2 (xi - x?)(x? - a) + (x? - a)2

(9)

i=1

i=1

i=1

i=1

Now write out the middle term in 9 and simplify

n

n

n

n

n

(xi - x?)(x? - a) = x? xi - a xi - x? x? + x? a

(10)

i=1

i=1

i=1

i=1

i=1

= nx?2 - anx? - nx?2 + nx?a

=0

BASIC STATISTICS

3

We can then write 9 as

n

n

n

(xi - a)2 = (xi - x?)2 + (x? - a)2

(11)

i=1

i=1

i=1

Equation 11 is clearly minimized when a = x?. Now consider part b of theorem 1. Expand the

second expression in part b and simplify

n

n

n

n

(xi - x?)2 = x2i - 2x? xi + x?2

i=1

i=1

i=1

i=1

n

= x2i - 2nx?2 + nx?2

i=1

n

= x2i - nx?2

i=1

(12)

2.2. Theorem 2 on expected values and variances of sums.

Theorem 2. Let X1, X2, ? ? ? Xn be a random sample from a population and let g(x) be a function such that

Eg(X1) and V arg(X1)exist. Then following two items hold.

a: b:

EV (ar(ni=1ni=g1(Xg(iX))i=))

n(E g(X1 )) = n(V arg(X1))

Proof. First consider part a of theorem 2. Write the expected value of the sum as the sum of the

expected values and then note that Eg(X1) = Eg(X2) = ...Eg(Xi) = ...Eg(Xn) because the Xi are all from the same distribution.

n

n

E

g(Xi) = E(g(Xi)) = n(Eg(X1))

(13)

i=1

i=1

First consider part b of theorem 2. Write the definition of the variance for a variable z as E(z-E(z))2

and then combine terms in the summation sign.

n

n

n

2

V ar

g(Xi) = E g(Xi) - E

g(Xi)

(14)

i=1

i=1

i=1

Now write out the bottom expression in equation 14 as follows

n

V ar

g(Xi) = E [g(X1) - E(g(X1))]2 + E [g(X1) - E(g(X1))] E [g(X2) - E(g(X2))]

i=1

+ E [g(X1) - E(g(X1))] E [g(X3) - E(g(X3))] + ? ? ?

+ E [g(X2) - E(g(X2))] E [g(X1) - E(g(X1))] + E [g(X2) - E(g(X2))]2

(15)

+ E [g(X2) - E(g(X2))] E [g(X3) - E(g(X3))] + ? ? ? +? ? ?

+ E [g(Xn) - E(g(Xn))] E [g(X1) - E(g(X1))] + ? ? ? + E [g(Xn) - E(g(Xn))]2

Each of the squared terms in the summation is a variance, i.e., the variance of Xi = var(X1). Specif-

ically

E [g(Xi) - E(g(Xi))]2 = V arg(Xi) = V arg(X1)

(16)

4

BASIC STATISTICS

The other terms in the summation in 15 are covariances of the form

E [g(Xi) - E(g(Xi))] E [g(Xj ) - E(g(Xj))] = Cov [g(Xi), g(Xj)]

(17)

Now we can use the fact that the X1 and Xj in the sample X1, X2, ? ? ?, Xn are independent to assert that each of the covariances inn the sum in 15 is zero. We can then rewrite 15 as

n

V ar

g(Xi) = E [g(X1) - E(g(X1))]2 + E [g(X2) - E(g(X2))]2 + ? ? ? + E [g(Xn) - E(g(Xn))]2

i=1

= V ar(g(X1)) + V ar(g(X2)) + V ar(g(X3)) + ? ? ?

(18)

n

= V arg(Xi)

i=1 n

= V arg(X1)

i=1

= n V arg(X1)

2.3. Theorem 3 on expected values of sample statistics.

Theorem 3. Let X1, X2, ? ? ?Xn be a random sample from a population with mean ? and variance 2 < . Then

a: EX? = ?

b:

V arX?

=

2 n

c: ES2 = 2

Proof

of

part

a.

In theorem

2

let g(X)

=

g(Xi )

=

Xi n

.

This implies that

E g(Xi )

=

? n

Then

we

can write

EX? = E

1n n Xi

1 =E

n

n

1

Xi = n (nEX1) = ?

(19)

i=1

i=1

Proof of part b.

In theorem

2 let g(X)

=

g(Xi )

=

Xi n

.

This implies that V arg(Xi)

=

2 n

Then we

can

write

V arX? = V ar

1n n Xi

1 = V ar

n2

n

1

2

Xi = n2 (nV arX1) = n

(20)

i=1

i=1

Proof of part c. As in part b of theorem 1, write S2 as a function of the sum of square of Xi minus n times the mean of

Xisquared and then simplify

ES2 = E

1 n-1

n

Xi2 - nX? 2

i=1

1 =

n-1

nEX12 - nEX? 2

(21)

1 =

n(2 + ?2) - n 2 + ?2

n-1

n

= 2

The last line follows from the definition of a random variable, i.e.,

BASIC STATISTICS

5

V ar X = X2 = EX2 - (EX)2

= EX2 - ?2X

(22)

E X2 = X2 - ?2X

2.4. Unbiased Statistics. We say that a statistic T(X)is an unbiased statistic for the parameter of the underlying probability distribution if E T(X) = . Given this definition, X? is an unbiased statistic

for ?,and S2 is an unbiased statistic for 2 in a random sample.

3. METHODS OF ESTIMATION Let Y1, Y2, ? ? ? Yn denote a random sample from a parent population characterized by the parameters 1, 2, ? ? ? k. It is assumed that the random variable Y has an associated density function f ( ? ; 1, 2, ? ? ? k).

3.1. Method of Moments.

3.1.1. Definition of Moments. If Y is a random variable, the rth moment of Y, usually denoted by ?r, is defined as

?r = E(Y r )

=

yr f (y; 1, 2, ? ? ?k) dy

(23)

-

if the expectation exists. Note that ?1 = E(Y ) = ?Y , the mean of Y. Moments are sometimes written as functions of .

E(Y r) = ?r = gr (1, 2, ? ? ? k)

(24)

3.1.2. Definition of Central Moments. If Y is a random variable, the rth central moment of Y about a is defined as E[(Y - a)r]. If a = ?r, we have the rth central moment of Y about ?Y , denoted by ?r,

which is

?r = E[(Y - ?Y )r]

=

(y - ?y)r f (y; 1, 2, ? ? ? k) dy

(25)

-

Note that?1 = E[(Y - ?Y )] = 0 and ?2 = E[(Y - ?Y )2] = V ar[Y ]. Also note that all odd numbered moments of Y around its mean are zero for symmetrical dsitributions, provided such moments exist.

3.1.3. Sample Moments about the Origin. The rth sample moment about the origin is defined as

?^r

=

x?rn

=

1 n

n

yir

(26)

i=1

6

BASIC STATISTICS

3.1.4. Estimation Using the Method of Moments. In general ?r will be a known function of the parameters 1, 2, ? ? ? k of the distribution of Y, that is ?r = gr (1, 2, ? ? ? k). Now let y1, y2, ? ? ? , yn be a random sample from the density f(?; 1, 2, ? ? ? k). Form the K equations

1n ?1 =g1(1, 2, ? ? ? k) = ?^1 = n yi

i=1

1 ?2 =g2(1, 2, ? ? ? k) = ?^2 = n

n

yi2

(27)

i=1

...

1 ?K =gK (1, 2, ? ? ? k) = ?^K = n

n

yiK

i=1

The estimators of 1, 2, ? ? ? k, based on the method of moments, are obtained by solving the system of equations for the K parameter estimates ^1, ^2, ? ? ?^K .

This principle of estimation is based upon the convention of picking the estimators of i in such a manner that the corresponding population (theoretical) moments are equal to the sample moments. These estimators are consistent under fairly general regularity conditions, but are not generally efficient. Method of moments estimators may also not be unique.

3.1.5. Example using density function f(y) = (p + 1) yp. Consider a density function given by

f(y) = (p + 1) yp 0 y 1

(28)

= 0 otherwise

Let Y1, Y2, ? ? ? Yn denote a random sample from the given population. Express the first moment of Y as a function of the parameters.

E(Y ) = y f(y) dy

-

1

= y (p + 1) yp dy

0

1

= yp+1 (p + 1) dy

(29)

0

1

yp+2 (p + 1) =

(p + 2)

0

p+1 =

p+2

Then set this expression of the parameters equal to the first sample moment and solve for p.

BASIC STATISTICS

7

p+1 ?1 = E(Y ) = p + 2

p+1 1 n

=

p+2 n

yi = y?

(30)

i=1

p + 1 = (p + 2) y? = py? + 2y?

p - py? = 2 y? - 1

p(1 - y?) = 2 y? - 1

2 y? - 1 p^ =

1 - y?

3.1.6. Example using the Normal Distribution. Let Y1, Y2, ? ? ? Yn denote a random sample from a nor-

mal distribution with mean ? and variance 2. Let (1, 2) = (?, 2). Remember that ? = ?1 and 2 = E[Y 2] - E2[Y ] = ?2 - (?1)2.

?1 = E(Y ) = ?

(31)

?2 = E(Y 2) = 2 + E2[Y ] = 2 + ?2

Now set the first population moment equal to its sample analogue to obtain

1n

?= n

yi = y?

(32)

i=1

?^ = y?

Now set the second population moment equal to its sample analogue

2 + ?2 = 1 n

n

yi2

i=1

2 = 1 n

n

yi2 - ?2

(33)

i=1

=

1 n

n

yi2 - ?2

i=1

Now replace ? in equation 33 with its estimator from equation 32 to obtain

^ =

1 n

n

yi2 - y?2

(34)

i=1

^ =

n (yi - y?)2 n

i=1

This is, of course, from the sample standard deviation defined in equations 4 and 5.

8

BASIC STATISTICS

3.1.7. Example using the Gamma Distribution. Let X1, X2, ? ? ? Xn denote a random sample from a gamma distribution with parameters and . The density function is given by

f(x; , )

=

1 ()

x-1

e

-x

0x<

(35)

=0

otherwise

Find the first moment of the gamma distribution by integrating as follows

E(X) =

0

1 x

()

x-1

e

-x

dx

(36)

1 = ()

x(1+)-1

e

-x

dx

0

If we multiply equation 36 by 1+ (1 + ) we obtain

1+ (1 + ) E(X) = ()

0

1+

1

x(1+)-1

e

-x

(1 + )

dx

(37)

The integrand of equation 37 is a gamma density with parameters and 1 - This integrand will integrate to one so that we obtain the expression in front of the integral sign as the E(X).

1+ (1 + )

E(X) = ()

(38)

(1 + ) =

()

The gamma function has the property that (t) = (t - 1)(t - 1) or (v + 1) = v(v). Replacing (1 + ) with () in equation 38, we obtain

(1 + )

E(X) = ()

(39)

() =

()

=

We can find the second moment by finding E(X2). To do this we multiply the gamma density in equation 36 by x2 instead of x. Carrying out the computation we obtain

E(X2) =

0

x2 1 ()

x-1

e

-x

dx

1 = ()

x(2+)-1

e

-x

dx

0

(40)

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download