What a p Value Can and Can Not Tell You



What a p Value Can and Can Not Tell You

Richard G. Lambert, UNC Charlotte, RSCH 6110

Unfortunately, practitioners and students of statistics often place more value on p values than statisticians do. There seems to be an artificial reverence for p values that may not always be justified. The social sciences seem at times to have there own doxology:

“Praise God from whom all p values flow,

praise alpha, beta, and .05,

praise statistics all ye researchers here below,

praise alpha, beta, and .05.”

While this handout is not intended to diminish the importance and usefulness of the proper application of statistical methods, it may be helpful to reflect about exactly what information a p value is providing.

First, let’s review some basic definitions:

p Value – A p value indicates the probability that effects, of the size observed or greater, are due to sampling error under the condition that the null hypothesis is true. A p value answers the question: If this treatment really does not work, how likely is it that effects of at least the size observed could have been due to sampling error?

Statistical Significance - Are the results beyond what would be expected due to sampling error alone? Are the results large enough that it is very unlikely that they are due to sampling error? A finding is said to be statistically significant when the p value associated with the test statistic is smaller than the predetermined alpha level. Typically, alpha is set at .05. Therefore, when a p value less than .05 is observed the null hypothesis is rejected.

Practical Significance - Are the effects found in a particular study large enough to be of any practical value? Do the results translate into any change in actual practice? Do they translate into any change in policy?

What a p Value Can Tell You

1. A p value can tell you if your results are typical or rare in the context of the sampling distribution of the test statistic under the condition that the null hypothesis is true.

2. A p value tells you the probability that sampling error alone is responsible for your findings.

3. A p value helps you to make a decision at the end of the study about which hypothesis, the null or the alternative, is more reasonable to accept. A p value simply tells you whether the findings you obtained are likely to be members of the sampling distribution for the condition when the null hypothesis is true. If this probability is very low, less than alpha, it is more reasonable to conclude that the observed results are members of some sampling distribution that represents a case when the alternative hypothesis is true.

What a p Value Can Not Tell You

1. A small p value does not warrant use of the word “prove”. A single study, small p value or large, merely demonstrates evidence for the truefulness or falsehood of a hypothesis. It does not prove anything. Furthermore, a single study presents evidence that is specific to the conditions under which the study is conducted. Theoretical concerns, replication, validation, and judgements about the generalizability of findings are just as important to the scientific process when a p value is small as they are when a p value is large.

2. A p value can not tell you that you have an important finding. A small p value does not exempt the researcher from using judgement to arrive at the correct interpretation of a research finding. All the issues regarding theoretical, practical, or clinical significance of a research finding are always present.

3. Correlation is still not necessarily causality, even when a p value is small.

4. A small p value does not indicate an effect of a particular size. Studies with large sample sizes may yield small p values associated with very small effects, while studies with small sample sizes may yield large effects that are not statistically significant.

5. A p value cannot tell you that a statistical significance test was necessary at all. Statistical significance testing applies when researchers have used random sampling from a defined population, or when a researcher has obtained a sample that is believed to be reasonably similar to what would have happened if a scientifically credible sampling strategy had been employed. Statistical significance testing applies to situation in which researchers are testing hypotheses about population conditions by using information obtained from random samples drawn from those population conditions, or from samples that reasonably approximate random and representative samples. They do not apply to population data, to some single subject research situations, or to batches of data obtained by convenience.

6. A small p value does not indicate that the correct statistical procedure was used, nor does it indicate that the assumptions of the procedure were met.

7. A small p value does not inform the researcher about the cost benefit or cost effectiveness issues related to the treatment nor does it make any statement about the side effects of the treatment.

8. A small p value does not make any statement about the presence or absence of confounding variables in a study. Statistically significant results must still be qualified with respect to measurement error, potential flaws in the research design, or context specific factors that may influence the results.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download