Power Analysis in Education Research

Statistical Power Analysis in Education Research

APRIL 2010

Larry V. Hedges Christopher Rhoads Northwestern University

Abstract

This paper provides a guide to calculating statistical power for the complex multilevel designs that are used in most field studies in education research. For multilevel evaluation studies in the field of education, it is important to account for the impact of clustering on the standard errors of estimates of treatment effects. Using ideas from survey research, the paper explains how sample design induces random variation in the quantities observed in a randomized experiment, and how this random variation relates to statistical power. The manner in which statistical power depends upon the values of intraclass correlations, sample sizes at the various levels, the standardized average treatment effect (effect size), the multiple correlation between covariates and the outcome at different levels, and the heterogeneity of treatment effects across sampling units is illustrated. Both hierarchical and randomized block designs are considered. The paper demonstrates that statistical power in complex designs involving clustered sampling can be computed simply from standard power tables using the idea of operational effect sizes: effect sizes multiplied by a design effect that depends on features of the complex experimental design. These concepts are applied to provide methods for computing power for each of the research designs most frequently used in education research.

NCSER 2010-3006 U.S. DEPARTMENT OF EDUCATION

This report was prepared for the National Center for Special Education Research, Institute of Education Sciences under Contract ED-04-CO-0112/0006.

Disclaimer

The Institute of Education Sciences (IES) at the U.S. Department of Education contracted with Optimal Solutions Group, LLC to develop a guide for calculating statistical power for complex multilevel designs that are used in most field studies in education. The views expressed in this report are those of the author and they do not necessarily represent the opinions and positions of the Institute of Education Sciences or the U.S. Department of Education.

U.S. Department of Education Arne Duncan Secretary

Institute of Education Sciences John Q. Easton Director

National Center for Special Education Research Lynn Okagaki Acting Commissioner

April 2010

This report is in the public domain. While permission to reprint this publication is not necessary, the citation should be: Hedges, Larry and Rhoads, Christopher (2009). Statistical Power Analysis in Education Research (NCSER 2010-3006). Washington, DC: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education. This report is available on the IES website at .

Alternate Formats

Upon request, this report is available in alternate formats such as Braille, large print, audiotape, or computer diskette. For more information, please contact the Department's Alternate Format Center at 202-260-9895 or 202-205-8113.

Disclosure of Potential Conflicts of Interest

There are two authors for this report with whom IES contracted to develop the discussion of the issues presented. Dr. Larry Hedges and Dr. Christopher Rhoads are both employees of Northwestern University. The authors do not have financial interests that could be affected by the content in this report.

Disclosure of Potential Conflicts of Interest

iii

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download