Applied Statistics for Political Scientists



POLS 7012

Spring 2008

Hypothesis Testing

Topic: Confidence intervals and hypothesis testing for means and proportions. Concepts of sampling and random data generation.

STATA commands and features: ci, sample, ttest, prtest

Data set: wvs.dta, taken from the World Values Survey 1991.

More information:

Readings: Alan Agresti and Barbara Finlay (1997). Statistical Methods for the Social Sciences, 3rd ed. Upper Saddle River, NJ: Prentice Hall. [CHAPTERS 4 -6]

1. Hypothesis Testing for Means

This week we look the issues surrounding hypothesis testing and sampling. Much of the theory is covered in the ‘Statistical Methods for the Social Sciences’ lectures, and the textbooks, and these notes concentrate on the STATA commands.

When we generate a sample mean in STATA it is an unbiased estimate of μ, the population mean. But it will certainly not be a perfect estimate since we are using a sample and not the entire population. Therefore if we took another sample we would expect the mean in this sample to be a little different; by chance, the mean we estimate is bound to be at least either a little bit higher or a little bit lower than the population mean.

For the estimate of the mean to be of value, we must have some idea of how precise it is. That is, how close to the population mean is the sample mean estimate likely to be? This is most commonly done by generating a confidence interval around the sample mean. Confidence intervals are calculated according to a particular method so that, under repeated sampling, the population parameter of interest (e.g. the mean) is contained in the confidence interval with a given probability. So, for instance, the population mean lies within the ‘95% confidence interval’ in 95% of random samples. This is not the same as saying that a 95% confidence interval contains the population mean with probability .95, although this is a common misinterpretation. Most often people use either the 95% or 99% confidence interval.

The process of generating confidence intervals for means in STATA is simple.

We use the .ci (confidence interval) command:

. ci [varlist] [weight] [if exp] [in range] [, level(#) binomial poisson exposure(varname) total ]

We will use this command to find a confidence interval for the mean of trust (the variable which we created last week from peoptrust). First, recreate this variable:

. recode peoptrust (1=1 "Trusting") (2=0 "Not Trusting"), gen(trust)

. summarize trust

Using our new trust variable, results are as follows:

. ci trust, binomial

-- Binomial Exact --

Variable | Obs Mean Std. Err. [95% Conf. Interval]

-------------+---------------------------------------------------------------

trust | 3432 .4769814 .0085258 .460151 .4938509

So given the confidence interval that is estimated in STATA, we see the range of possible values the mean might take in the population, due to the sample size the confidence interval is quite small. A rough rule of thumb is that the confidence interval is the mean plus or minus twice the standard error. The larger our sample the smaller our standard error and therefore the narrower the confidence interval and more precise estimate of the population mean. Note that we have included the binomial option on the .ci command. By doing so, the estimation takes into account the fact that variable cannot be less than 0 or greater than 1.

Exercise 1

Examine the confidence intervals for the means of the other variables we were interested in last week: pride, lifesat, and decision (recode the national pride variable as we did last week where 1 = proud and 0 = not proud, and remember to deal appropriately with missing data). Use the binomial option when necessary. Often opinion polls quote sampling error of +/- 2 percentage points. Why are our estimates from the WVS somewhat more precise?

2. A Note on Sampling

Having learned the .ci command, we should take a moment to look at the relationship between sample size and confidence intervals. You can use the .sample command to draw a random sample from the data in memory. We are currently using about 3500 cases. Use the .sample command to select a smaller random sample, anywhere between about 500 and 1000.

. preserve

. sample #, count

Then check the mean and confidence intervals for the trust variable again.

. ci trust, binomial

Note that as the sample size decreases, the confidence intervals increase. The reason this is true has to do with sampling theory, which is covered in the lectures. To return to using the full dataset, type:

. restore

Exercise 2

Experiment with the data to find what sort of sample size gives a confidence interval of around two percentage points either side.

3. Comparison of Means

If we find that two groups have different means for a given variable, how do we know whether this is due to chance in the particular sample that we have drawn? What we want to know is the probability that the means of the two groups are different in the population. Typically, when there is sufficient evidence in our sample to say that there is less than a five per cent chance that the underlying population difference is equal to zero, we say there is a statistically significant difference in means. Notice that ‘statistical significance’ is not the same as substantive significance. Even very small differences are statistically significant if we are confident that they reflect a real difference in the population.

The .ttest command in STATA tests the difference in means. We will use it to test the difference in life satisfaction in Britain and France, thus we have to exclude the USA and Sweden from the analysis using an if command:

. ttest lifesat if nation < 3, by(nation)

Two-sample t test with equal variances

------------------------------------------------------------------------------

Group | Obs Mean Std. Err. Std. Dev. [95% Conf. Interval]

---------+--------------------------------------------------------------------

France | 549 7.063752 .1854138 4.344383 6.699544 7.427961

Britain | 1052 7.613118 .1044634 3.388223 7.408137 7.818098

---------+--------------------------------------------------------------------

combined | 1601 7.424735 .0937565 3.751431 7.240836 7.608633

---------+--------------------------------------------------------------------

diff | -.5493656 .1970979 -.9359629 -.1627682

------------------------------------------------------------------------------

diff = mean(France) - mean(Britain) t = -2.7873

Ho: diff = 0 degrees of freedom = 1599

Ha: diff < 0 Ha: diff != 0 Ha: diff > 0

Pr(T < t) = 0.0027 Pr(|T| > |t|) = 0.0054 Pr(T > t) = 0.9973

Make sure you know which p-value to read from the chart, here we just want to know whether the two means are different; there is no directional component to the hypothesis, that is we don’t think one should be higher than the other. For example, a directional hypothesis would be that we expect that life satisfaction in France to be higher than in Britain. Our example here shows that life satisfaction is higher by a statistically significant margin in Britain.

Exercise 3

Compare means for Britain and France for decision. Where do significant differences exist? Interpret the results.

Now compare means for Britain to Sweden and the USA in turn. Again, where do significant differences exist? Hint: either recode into two new variables, where Britain equals 1 and the other country you are comparing equals 0 or use if statements to test this in each country.

If you are using a binary variable, like our trust variable, then testing the difference in means might not be the best way to proceed. This is because the ttest command relies on some distributional assumptions that may not be true for a 0/1 (binary or dummy) variable. Rather, you might consider testing the difference in proportions, using prtest. We will use it to examine the trust variable (which you will need to generate again if it was lost after the sampling exercise in section 2).

. recode peoptrust (1=1 "Trusting") (2=0 "Not Trusting"), gen(trust)

. prtest trust if nation|z| [95% Conf. Interval]

-------------+----------------------------------------------------------------

France | .2276867 .017897 .1926093 .2627641

Britain | .4448669 .0153217 .414837 .4748968

-------------+----------------------------------------------------------------

diff | -.2171802 .0235596 -.2633562 -.1710043

| under Ho: .0254254 -8.54 0.000

------------------------------------------------------------------------------

diff = prop(France) - prop(Britain) z = -8.5419

Ho: diff = 0

Ha: diff < 0 Ha: diff != 0 Ha: diff > 0

Pr(Z < z) = 0.0000 Pr(|Z| < |z|) = 0.0000 Pr(Z > z) = 1.0000

The final line of the output gives the relevant p-values for each test defined by the alternative hypothesis Ha. A p-value is the probability of observing the sample we have or one more extreme in a particular direction, assuming that the null hypothesis (Ho) is true. The null hypothesis is the same for all three tests: there is no difference between France and Britain in the level of trust. The middle test has an alternative hypothesis that the difference is not equal to zero. The interpretation of the p-value for this test is the probability of observing a sample difference between France and Britain that is as big or bigger than the one we have observed in our sample. Since this probability is practically zero (less than 0.0000) we can say that there is a statistically significant difference in the level of trust between France and Britain.

Exercise 4

Compare means for Britain and the other three countries for war (willingness to fight in a war if country was invaded). Note this may need recoding. Where do significant differences exist? Interpret the results.

Are there significant differences between men and woman in their willingness to fight? Are these differences significant in all countries (hint: use if statements to test this in each country). Interpret the results.

Stata Exercise:

1. According to authors like Putnam, education is positively associated with trust in other people. To what extent is this true for our WVS sample? Examine summary statistics for each education group in the dataset. Present one graph that summarises the relationship between education and trust. Is education also associated with life satisfaction? Present one graph that summarizes the relationship between education and life satisfaction. Describe your results.

2. Create a new two-group education variable, justifying your selection of the groups, and test the hypothesis that the means/proportions of the trust and life satisfaction variables are different between the two educational groups. Present the results in one or two tables and provide a brief description.

3. There is a possibility that the relationship between (1) education and (2) social trust or life satisfaction, varies from country to country. Using significance testing compare the means or proportions for social trust and life satisfaction by educational group (our recoded binary variable from above) in each country. Present your results concisely and describe your findings in no more than 500 words.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download