Test Reliability—Basic Concepts
Research Memorandum
ETS RM?18-01
Test Reliability--Basic Concepts
Samuel A. Livingston
January 2018
ETS Research Memorandum Series
EIGNOR EXECUTIVE EDITOR
James Carlson Principal Psychometrician
ASSOCIATE EDITORS
Beata Beigman Klebanov Senior Research Scientist
Heather Buzick Senior Research Scientist
Brent Bridgeman Distinguished Presidential Appointee
Keelan Evanini Research Director
Marna Golub-Smith Principal Psychometrician
Shelby Haberman Distinguished Research Scientist, Edusoft
Anastassia Loukina Research Scientist
John Mazzeo Distinguished Presidential Appointee
Donald Powers Principal Research Scientist
Gautam Puhan Principal Psychometrician
John Sabatini Managing Principal Research Scientist
Elizabeth Stone Research Scientist
Rebecca Zwick Distinguished Presidential Appointee
Kim Fryer Manager, Editing Services
PRODUCTION EDITORS
Ayleen Gontz Senior Editor
Since its 1947 founding, ETS has conducted and disseminated scientific research to support its products and services, and to advance the measurement and education fields. In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public. Published accounts of ETS research, including papers in the ETS Research Memorandum series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes. Peer review notwithstanding, the positions expressed in the ETS Research Memorandum series and other published accounts of ETS research are those of the authors and not necessarily those of the Officers and Trustees of Educational Testing Service.
The Daniel Eignor Editorship is named in honor of Dr. Daniel R. Eignor, who from 2001 until 2011 served the Research and Development division as Editor for the ETS Research Report series. The Eignor Editorship has been created to recognize the pivotal leadership role that Dr. Eignor played in the research publication process at ETS.
Test Reliability--Basic Concepts Samuel A. Livingston
Educational Testing Service, Princeton, New Jersey January 2018
Corresponding author: S. A. Livingston, E-mail: slivingston@ Suggested citation: Livingston, S. A. (2018). Test reliability--Basic concepts (Research Memorandum No. RM-18-01). Princeton, NJ: Educational Testing Service.
Find other ETS-published reports by searching the ETS ReSEARCHER database at
To obtain a copy of an ETS research report, please visit
Action Editor: Gautam Puhan Reviewers: Shelby Haberman and Marna Golub-Smith Copyright ? 2018 by Educational Testing Service. All rights reserved. ETS, the ETS logo, GRE, MEASURING THE POWER OF LEARNING, and TOEFL are registered trademarks of Educational Testing Service (ETS). All other trademarks are the property of their respective owners.
Abstract The reliability of test scores is the extent to which they are consistent across different occasions of testing, different editions of the test, or different raters scoring the test taker's responses. This guide explains the meaning of several terms associated with the concept of test reliability: "true score," "error of measurement," "alternate-forms reliability," "interrater reliability," "internal consistency," "reliability coefficient," "standard error of measurement," "classification consistency," and "classification accuracy." It also explains the relationship between the number of questions, problems, or tasks in the test and the reliability of the scores. Key words: reliability, true score, error of measurement, alternate-forms reliability, interrater reliability, internal consistency, reliability coefficient, standard error of measurement, classification consistency, classification accuracy
RM-18-01 i
Preface This guide grew out of a class that I teach for staff at Educational Testing Service (ETS). The class is a nonmathematical introduction to the topic, emphasizing conceptual understanding and practical applications. The class consists of illustrated lectures, interspersed with written exercises for the participants. I have included the exercises in this guide, at roughly the same points as they occur in the class. The answers are in the appendix at the end of the guide.
In preparing this guide, I have tried to capture as much as possible of the conversational style of the class. I have used the word "we" to refer to myself and most of my colleagues in the testing profession. (We tend to agree on most of the topics discussed in this guide, and I think it will be clear where we do not.)
RM-18-01 ii
Table of Contents Instructional Objectives .................................................................................................................. 1 Prerequisite Knowledge .................................................................................................................. 2 What Factors Influence a Test Score? ............................................................................................ 2
The Luck of the Draw.............................................................................................................. 3 Reducing the Influence of Chance Factors .............................................................................. 4 Exercise: Test Scores and Chance ........................................................................................... 5 What Is Reliability? ........................................................................................................................ 6 Reliability Is Consistency ........................................................................................................ 6 Reliability and Validity............................................................................................................ 7 Exercise: Reliability and Validity............................................................................................ 8 Consistency of What Information? ................................................................................................. 8 "True Score" and "Error of Measurement" .................................................................................... 9 Reliability and Measurement Error ....................................................................................... 11 Exercise: Measurement Error ................................................................................................ 11 Reliability and Sampling............................................................................................................... 12 Alternate-Forms Reliability and Internal Consistency ................................................................. 13 Interrater Reliability...................................................................................................................... 15 Test Length and Reliability........................................................................................................... 16 Exercise: Interrater Reliability and Alternate-Forms Reliability........................................... 17 Reliability and Precision ............................................................................................................... 18 Reliability Statistics ...................................................................................................................... 19 The Reliability Coefficient .................................................................................................... 19 The Standard Error of Measurement ..................................................................................... 20 How Are the Reliability Coefficient and the Standard Error of Measurement Related? ...... 22 Test Length and Alternate-Forms Reliability ........................................................................ 22 Number of Raters and Interrater Reliability .......................................................................... 24 Reliability of Differences Between Scores............................................................................ 25 Demystifying the Standard Error of Measurement................................................................ 26 Exercise: The Reliability Coefficient and the Standard Error of Measurement .................... 26 Reliability of Essay Tests.............................................................................................................. 27
RM-18-01 iii
Reliability of Classifications and Decisions ................................................................................. 28 Summary ....................................................................................................................................... 30 Acknowledgments......................................................................................................................... 32 Appendix. Answers to Exercises .................................................................................................. 32
Exercise: Test Scores and Chance ......................................................................................... 32 Exercise: Reliability and Validity.......................................................................................... 33 Exercise: Measurement Error ................................................................................................ 33 Exercise: Interrater Reliability and Alternate-Forms Reliability........................................... 35 Exercise: The Reliability Coefficient and the Standard Error of Measurement .................... 35 Notes ............................................................................................................................................. 37
RM-18-01 iv
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related download
- validity and reliability in social science research
- reliability and validity
- why is scientific validity important wellsource
- test reliability and validity
- validity and reliability of qualitative data analysis
- validity and reliability in quantitative studies
- the importance of psychological measurement
- multimodal literacy scale a study of validity and reliability
- understanding validity for teachers activity what does a
- test reliability—basic concepts
Related searches
- basic grammar test with answers
- basic grammar test pdf
- basic math practice test questions
- basic math test for employment
- basic math test printable free
- basic math test for kids
- basic math test for adults
- basic math practice test free
- test of adult basic education
- basic math test for employment printable free
- basic math placement test printable
- basic skills assessment test practice