Running head: TEACHER MADE ASSESSMENT ACTIVITY: EDF …



Running head: TEACHER MADE ASSESSMENT ACTIVITY: EDF 6432

Teacher Made Assessment Activity: Accelerated Reader Teacher Made Tests

Margaret England

Florida Gulf Coast University

Abstract

This paper investigates the Validity and Reliability of one teacher made Accelerated Reader quiz. During Item Analysis it was determined that the book Two Crows Counting was not a valid selection for a 1.0 Accelerated Reader quiz.

The book contained sentence fragments, lacked verbs, and included incorrect punctuation. The book did not support the reading skills being taught

in the classroom.

Teacher Made Assessment Activity: Accelerated Reader Teacher Made Tests

Statement of Purpose:

This paper investigates decisions about managing classroom instruction by evaluating a teacher made Accelerated Reader quiz in an elementary school’s Accelerated Reader (AR) technology-based literacy program database. AR Criterion Reference Tests (CRT) are low-stakes formative assessments “intended to determine whether or not a student has read a book” (Renaissance Learning1999 p.2) within their tested Zone of Proximal Development (ZPD) reading level. “These quizzes are encouraging rather than intimidating, chiefly because a student who has read a book should be able to pass the quiz” (Renaissance Learning 1999, April). The AR program provides students with immediate feedback after they take a 5-20 question quiz. The quiz score, points, student’s average, number of quizzes, and goal are included on the TOPS (Three Opportunities to Praise Students) report. A computer-generated comment is included with space at the bottom for comments from the monitor and teacher.

The school’s database includes over 8,000 quizzes purchased from Renaissance Learning (AR). Up to 500 teacher made quizzes may be added to the database. The teacher made quizzes have not been evaluated for validity or reliability during the eight years AR has been implemented in the school. The purpose of this paper is to improve the validity and reliability of one teacher made quiz in level 0.2-1.9 through item analysis.

Table of Specifications

The Table of Specifications (Appendix A) for Two Crows Counting could be used with any Accelerated Reader Quiz or group of quizzes with adjustments being made for 10 or 20 item quizzes. The percentages of the evaluation are 50% Literal Comprehension, 15% Skill in Book Selection, 15% skill in Paired Reading, 15% Reading Independently, and 5% Recording in Reading Log. The main purpose of an AR quiz is to check the student’s literal comprehension of text as a way to motivate students and determine their understanding (knowledge) of the content of a book (Renaissance 1999 p.2). The table could be modified for use as a 9-week unit with percentage from quizzes scores used for the content score and the totals percentages from the book selection, reading log, dualog (paired) reading, and independent reading as the skill scores. When the student reads book independently or with a partner the text itself can provide instruction “that effectively promotes both comprehension development and the exercise of critical thinking” (Institute p.6). The teacher provides instruction through frequent monitoring of students during independent and paired reading. During Status of the Class (monitoring) the teacher has the opportunity to “assess ongoing work, monitor individual progress, provide informative feedback to students, adapt instruction as needed, and ultimate improve overall student performance” (Renaissance, 2002 p.1). The importance of the teacher guiding reading and intervening when needed are key elements to a successful AR program. (Institute 1999 p.6-7). Another skill in this table is the use of the Reading Log, “student-involved record keeping” (Willis), which is used to motivate students and to help them record and reflect on their improvement. Students are also assessed for selecting a book within their Zone of Proximal development (ZPD).

Choices of Assessment and Item Format

In order to choose a teacher made AR quiz to evaluate a Quiz Takers Reading Practice Report was run on all teacher made quizzes in the 0.2 - 1.9 ATOS (AR readability formula) range taken by students during the current school year (2003). The percentage scores were reviewed for these quizzes. The quiz, Two Crows Counting, taken by 21 different students, had the lowest percentage correct with 63.8%. The goal of AR is for students to earn an average score of 85 –92% on AR quizzes within their ZPD range (Institute 1997). Students in the school’s database who had taken the quiz during the previous 2002 school year scored an average of 65% and 74% in the 2001 school year. The AR database includes records for students currently enrolled in the school. The averages of students who have withdrawn are not included in the yearly averages.

In order to develop the teacher made quiz the teacher writes 5 - 20 multiple-choice questions, an answer and 3 distracters taken from a book. Five questions are usually written for shorter books and 10 or 20 questions for longer books. The directions for AR quizzes are the same for all quizzes and quiz takers. After reading a book, the student logs into the AR database using their name and password, they locate the quiz for the book they read, and then they take the selected-response AR Quiz. The questions are presented in the order that matches the chronology of the book and reinforces the story grammar of the book (Renaissance 1999, April). The program randomly changes the order in which the 4 responses to questions are presented. Only one answer may be selected to each question. Once a student selects an answer and goes to the next question the program does not allow them to go back to change an answer. The actual quiz taken by a student can’t be printed. Appendix I. shows the performance of students after the 2nd revision of the quiz.

In order to earn points on a quiz students “need to score at least 60 percent on a 5- and 10-quesiotn quiz and 70 percent on 20-question quizzes” (Renaissance1999, April). The number of questions is determined by the difficulty and content of the book. Lower level picture books usually have 5 questions. “Questions typically focus on significant events, character’s and literacy features of a book” (Renaissance 1999, April). If a student scores 60% or more the program will show them the correct answers to questions they answer incorrectly. Information concerning the questions is not retained in the database. Individual interviews or students writing the number of missed the questions are two ways to record the questions answered incorrectly.

Instrument and Performance

In order to determine the ATOS readability of Two Crows Counting, a text file (Appendix C) was sent twice to the Renaissance Learning ATOS (Readability For Books) Analyzer. The ATOS Analyzer results (Appendix D) included a word count of 113 words, book level of 3.7, and 0.5-point value if a student answers all questions correctly. When the teacher made quiz was developed the word count entered into the database the database required that a minimum of 1,000 words or more be entered in the word count field. Three years ago AR went from Fry Readability to ATOS Reading Levels. The book level using the Fry Readability Index (Advantage Learning, 1996) was 1.0.

Analysis of Instrument: Validity, Reliability, and Item Analysis

As stated earlier the purpose of taking an AR quiz is to determine if a student has read a book and if they have read a book then they should be able to pass the quiz. Is it valid to interpret the scores from Two Crows Counting as measuring reading comprehension (Nitko p.37)? Upon examination of Two Crows Counting it was noted that the book is a counting book written in rhyming sentence fragments (Appendix C). The book contains no verbs. Periods and commas were used in some of the sentence fragments. The text in Two Crows Counting was then compared with text in 1.0 and 3.7 level books in the school library collection. The 1.0 and 3.7 books contained complete sentences and punctuation. The vocabulary and word count in the 1.0 book more closely matched the Two Crows Counting vocabulary. Complete sentences, verbs, and punctuation are skills that are taught to beginning readers. In order for validity to be attained in assessments, including AR assessments, they should emphasize what you taught and contain content worth learning (Nitko, p.39). A quiz on Two Crows Counting is not a valid assessment of what is being taught in the classroom.

The scores for students taking the Two Crows Counting quiz were evaluated using Livingston’s Formulae for Reliability of a Criterion Referenced test (RCR). In 2001 ten students took the quiz with a Livingston’s RCR of .84 (Appendix E). In 2002 twelve students had a RCR of .83 (Appendix F) and in 2003 twenty-one students had a RCR of .85 (Appendix G). These scores support the stability of the performance of students’ with a 1.0 ZPD reading level that took the quiz during the past three years. The Formula was used to determine the RCR of .80 for 16 students (Appendix H) who to the first revision of the quiz and 11 students who took the 2nd revision of the quiz with the RCR of .81. The two revisions were administered one day only to a limited number of students. Appendix J. includes the student’s scores after the 2nd revision. Appendix B includes the original questions and the two revisions. .

Analysis of Instrument

Interviews were conducted after students took the original and first revision quiz. The changes in the instrument, observations, and comments follow.

Question 1:

One _____ rising

Answer: sun Distracters: crow bird balloon

Students who missed this question were looking at the picture clue of the crows flying towards the sun (p.6) and chose crow or bird for an answer. The question will be rewritten with a blank at the end of the stem and distracters that are not on the same page as the text for the stem.

Question 1 Revision 1:

One sun _______

Answer: rising Distracters: smiling shining hiding

Students missing this question were using prior knowledge and chose shining as the answer.

Question 1 Revision 2:

What was rising?

Answer: sun Distracters: boats cats people

After collaboration with a first grade teacher and reading the suggestions for improving the quality of stems of multiple-choice items (Nitko, p. 150) the question and stem were rewritten.

Question 2

Three people ______ and four trees blowing

Answer: rowing Distracters: running sitting eating

Students incorrectly chose running. When rewritten the blank was put at the end of the stem.

Question 2 Revison 1

Three people rowing and four trees ___________

Answer: blowing Distracters: growing crowing glowing

Using prior knowledge students incorrectly chose growing. After collaboration the stem was shortened.

Question: 2 Revison 2

Four trees __________

Answer: blowing Distracters: growing crowing glowing

Question 3

Seven ____wading and eight guess parading.

Answer: herons Distracters: crows birds ducks

Students missing this question chose ducks. The stem was shortened with the blank added at the end of the stem.

Question 3 Revision 1

Seven herons ___________

Answer: wading Distracters: fading parading swimming

The stem was shortened with the blank added at the end of the stem. Students missing this question chose swimming. The herons in the picture were walking in the water.

Question 3 Revison 2

Seven herons ________

Answer: wading Distracters: fading eating flying

The distracters were rewritten.

Question 4

Four trees standing and _______ boats landing

Answer: three Distracters: two five six

Students are not allowed to open their AR book during testing and did not remember how many boats were landing.

Question 4 Revison 1

Four trees standing and three boats ________-

Answer: landing Distracters: sailing rowing sinking

The question was rewritten with the blank at the end of the stem

Students chose rowing as an incorrect answer. After collaboration with another teacher the stem was shortened and another question selected.

Question 4 Revision 2

The socks are ________

Answer: flapping Distracters: sailing rowing resting

Question 5

Big crow and small crow are __________

Answer: fast asleep Distracters: counting eating flying

This question was rewritten and different distracters were selected.

Question 5 Revision 1

At the end of the story big crow and little crow are ________

Answer: fast asleep Distracters; counting sheep eating corn flying alone

One student chose flying alone.

Question 5 Revision 2

At the end of the story big crow and little crow are ________

Answer: fast asleep Distracters; counting sheep eating corn flying alone

No changes were made to the stem or distracter.

After two revisions of the quiz, review by a colleague, and review of the ATOS Analyzer there was evidence that the Two Crows Counting was not the appropriate book for a 1.0 teacher made AR quiz. The content did not assess the reading comprehension skills being taught to beginning readers. The incomplete sentences, lack of verbs, and incorrect punctuation were not an appropriated text for assessment. According to Popham reading vocabulary and/or sentence structure that is too difficult is a factor affecting the validity of an assessment.

Performance of Group and Individual Students

With each revision of the quiz there was an increase in the total percentage answers correct. The average for this school year was 63.0 %, after the 1st revision it was 66.2%, and 72.7 % after the 2nd revision. The target percentage was 80%. Only six students took the quiz for both revision 1 and 2 after they reread the book. Each student’s score showed an increased.

Student: Revision 1 # Correct Revision 2 # Correct

AA2 2 4

AA3 3 4

AA4 3 5

AA6 2 5

AA7 3 4

AA11 3 5

Five students took the quiz for the 2nd revision, but not revision 1.

AA1 5

AA5 1

AA8 1

AA8 3

AA10 3

The reading records for AA5 and AA8 were reviewed in the AR database. These students consistently score 60% or more on AR tests in the 1.0 - 1.2 ZPD range.

Conclusion

AR quizzes are intended to determine whether or not a student has read a book within their ZPD. Students who have read a book in their range should be able to pass a quiz. (Renaissance Learning, 1999, April). Teachers need to be skilled in developing high quality assessments (Nitko p.81). After careful analysis it was determined that a quiz from the book Two Crows Counting is not a valid assessment of what is being taught in the 1.0 reading level curriculum. The quiz and quiz records will be removed from the AR database. In order to improve the validity of quizzes in the database the teacher made quizzes will be evaluated using the Validity, Reliability, and Item Analysis approaches in order to improve provides high quality assessments. It is our professional responsibility to provide students with valid assessments (Nitko p.455-462).

References

Advantage Learning Systems. (1996). Accelerated Reader Installation User’s Manuel, 82-84

Hutchcraft, G. Table of Specification for a Weather Unit in Middle School Science. Message posted to

Hutchcraft, G. Livingston's Formulae for Reliability of a Criterion referenced test Relval.xls. Message posted to

Hutchcraft, G. Steps for Building a Teacher-made Classroom Assessment. Message posted to

Hutchcraft, G. Livingston's Reliability. Message posted to

Institute for Academic Excellence, Inc. (1997, November). Critical Thinking and Literature-Based Reading. Report from the Institute for Academic Excellence, 1-8.

Kehoe, F. (1995, October). Basic Item Analysis for Multiple-choice Tests. Eric/ AE Digest, ED398 237.

Nitko, A. J. (2001). Educational Assessment of Students. Upper Saddle River, New Jersey: Merrill Prentice Hall. (Original work published 1996)

Orgel, D. (1996). Two Crows Counting. Milwaukee, WI: Garth Stevens Publishing.

Paul, T. J., M.B.A. (1996, October 4). Learning Information Systems: Theoretical Foundations. Paper presented at National Research Center Conference.

Popham, W. J. What Do Classroom Teachers Really Need to Know about Validity?. Retrieved from Dr. Gilbert Hutchcraft:

Popham, W. J. What Do Classroom Teachers Really Need to Know about Reliability? Retrieved from Dr. Gilbert Hutchcraft:

Renaissance Learning. ATOS Analyzer,

Renaissance Learning. (1999, April). The Design of Accelerated Reader Assessments. Report from Renaissance Learning, 1-8.

Renaissance Learning. (2002). National Reading Studies Validate Accelerated Reader, Reading Renaissance. Scientifically Based Research Abstract Accelerated Reader and Reading Renaissance, L1542, 1-4.

Renaissance Learning. (2002, July). How Scientific Research Support the School Renaissance School Improvement Process. Report from Renaissance Learning.

Spencer, S. J. (1995, November). Modifying tests for diverse learners. Educational tests & measurements, 31(2), p.84. Retrieved April, 2003, from Academic Search Elite:

Stiggins, R. (2001, Fall). Educational Measurement: Issues and Practices. Retrieved March, 2003, from Dr. Gilbert Hutchcraft:

Willis, S. (1997, December). Using Assessment to Motivate Students. Educational Update, 39(8). Retrieved March, 2003, from Dr. Gilbert Hutchcraft:

Appendix A

Specifications:

Table of Specification for book: Two Crows Counting

Objectives

| |Knows |Skill In |Skill In |Skill In |Skill In |Total # |

| | | | | | |items |

|Content |Literal Comprehension |Selecting |Use status of |Use status of the |Use Reading Log | |

| | | |the class or |class or rating scale |To | |

| | | |ratting scale |to |Evaluate | |

| | | |to access. | |Recording of | |

|Vocabulary |5 |Book |Dualog |Access students |Title, Level |5 |

|Practice and | |In ZPD range |reading, |reading independently |And Score | |

|Reinforcement | | |students | |On Quiz by | |

| | | |reading with a| |students | |

| | | |partner | | | |

|% Of Evaluation |50% |15% |15% |15% |5% |100% |

The above table of specifications is the book Two Crows Counting by Doris Orgel. Knowledge, literal comprehension of the vocabulary in the book would be assessed through the Accelerated Reader Computer Program. Performances would be accessed through frequent monitoring and status of the class taken by the classroom teacher.

The numbers in the cells represent the number of items measuring the content area and the type of target objective. There are 5 items on the whole test so the percentages show what percent of the test measures each type of objective (you could show the same thing for each content area. The percentage of emphasis on the test should roughly match the percentage of emphasis given in instruction.

Appendix B

[pic]

Appendix C

Big crow, small crow, on the wing, counting, counting everything...

1 ONE sun rising

1 Two shadows gliding

3 Three people rowing

4 Four trees blowing

5 Five cats lapping

6 Six socks flapping

7 Seven herons wading

8 Eight geese parading

9 Nine farmers haying

10 Ten children playing

Small crow, big crow, home the go, counting things they see below...

10 Ten children snoozing

9 Nine farmers snoring

8 Eight geese resting

7 Seven herons nesting

6 Six socks hardly stirring

5 Five cats purring

4 Four trees standing

3 Three boats landing

2 Two shadows slowing

1 One sun glowing

Big crow, small crow, dreaming deep, counting nothing fast asleep.

Appendix D

Renaissance Learning, Inc.

Wednesday, April 09, 2003 4:39:45 PM

Message

From: ATOS

Subject: ATOS for book results (Confirmation # 8891)

To: Margaret L. England

Attachments:

Thank you for your inquiry. The file you submitted was analyzed, and these are the results:

Confirmation Number:    8891

Title of entry:         Two Crows Counting

Author First Name:      Doris

Author Last Name:       Orgel

ISBN:                   083681617x

ATOS book level:        3.7

AR Points:              0.5

Total Word Count:       113

Renaissance Learning, Inc.

Appendix E

[pic]

Appendix F

[pic]

Appendix G

[pic]

Appendix H

[pic]

Appendix I

[pic]

Appendix J

[pic]

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download