Assessment Form



A Procedure for Analyzing Science Assessment Items

Assessment Item Analysis Procedure

Enter the code for the targeted learning goal(s):

[pic]

I. Exploration

Before evaluating an assessment item, reviewers should first become familiar with the learning goal to which the item is targeted. They should carefully read the content standard or benchmark and any clarification materials that are provided. They should also become familiar with the assessment task and identify the specific knowledge and skill that students need to have in order to respond successfully. These initial observations will be used later in the analysis of the item.

In addition to the learning goal and the assessment task, materials provided may include:

• Clarification statements that describe in more detail what students are and are not expected to know.

• Assessment maps that show the relationship between the targeted learning goal and other learning goals, especially earlier and later learning goals.

• Summaries of research on student learning that describe difficulties students may have as well as misconceptions and other ideas they may hold.

IA. Exploring the Learning Goal and Related Materials. Hide Details

It is important to be as precise as possible in the interpretation of the learning goals.  For example, if the learning goal states that students should know that “when liquid water disappears, it turns into a gas and can reappear as a liquid when cooled” but it does not provide a molecular explanation for that phenomenon, then a molecular explanation should not be assumed to be part of the learning goal.  If a clarification statement is not available that makes such distinctions explicit, reviewers should see if there is a similar learning goal at an earlier or later grade level and make a comparison to how it is stated there.

Steps for reviewers:

a. Read any clarification statements that are available for this learning goal and review the accompanying assessment maps and summaries of research on student learning.

b. Complete the box below.

1. Is there anything you disagree with or do not understand in the learning goal, the clarification statement, the map, or the description of misconceptions from the research on student learning? Suggest changes here.

[pic]

IB. Exploring the Assessment Task. Hide Details

Before making judgments about alignment, the reviewers should become familiar with the assessment task.  They should think about the ways a student might approach the task or might misinterpret what is being asked.

Steps for reviewers:

a. Attempt to complete the task.

b. After noting the correct answer, examine student responses to the task if they are available from field tests or interview data.

c. Answer the questions in the boxes below.

1. State the knowledge needed to select or eliminate each answer choice. Hide Details

As much as possible, reviewers should state the knowledge that is needed to evaluate each answer choice in terms of a general principle, not simply as a verbatim restatement of the answer choice. Consider the following item: “If a container of water is sealed and kept at a constant temperature, what can you say about the motion of the water molecules?” If one of the answer choices is: “The average speed of the water molecules will increase a little bit,” the reviewer should not say that students need to know that the average speed of the water molecules will not increase a little bit. Instead, the reviewer should say that students need to know that the average speed of the molecules of a substance will not change if the temperature stays constant.

[pic]

2. Describe the different strategies that could be used to complete the task. Hide Details

Some questions can be answered using multiple strategies. In the context of this analysis, answering a question by using the information in the stem vs. the information in a diagram or using ideas from another (non-targeted) learning goal instead of the targeted ideas would qualify as using different strategies, but “eliminating distractors” (vs. identifying the correct choice) or “intuition” would not.

[pic]

3. What common misconceptions might affect how a student answers?  For each answer choice, indicate the misconceptions that might affect how a student answers.

[pic]

4. What general cognitive abilities beyond recall of scientific knowledge are needed to successfully complete this task? Hide Details

Examples include predicting, explaining, analyzing, comparing, evaluating, inferring, or interpreting a representation.

[pic]

5. What mathematical knowledge and skills are needed to successfully complete this task? Hide Details

Examples include understanding concepts such as probability, interpreting graphs and tables, or using various computational skills.

[pic]

II. Content Alignment

In this part of the procedure, reviewers evaluate the alignment of the task to the targeted learning goal. Two criteria, necessity and sufficiency, are used.  The necessity criterion addresses whether the learning goal is needed to complete the task, while the sufficiency criterion addresses whether the learning goal is adequate by itself to complete the task.

Note: The analysis of the assessment tasks should use the learning goal as originally stated even if reviewers made suggestions to revise it.  Suggested revisions to the learning goal and other materials made during the exploration should not be used in the task analysis. Those revisions should be considered during future item development.

II. A. Necessity Hide Details

This section examines whether having the knowledge described in the learning goal and the clarification statement, and/or knowing that certain commonly held misconceptions are not true, is needed for students to answer correctly.

Guide for reviewers:

• If the knowledge needed to complete the task is described in the targeted learning goal, then the targeted learning goal is necessary. This is true even if the part of the learning goal that is needed is a small part.

• Reviewers should list the part of the learning goal that is needed to complete the task (or if the majority of the learning goal is needed, reviewers can simply list the part not needed).

• If an item is specified to be aligned to more than one learning goal, reviewers are to imagine a single, combined learning goal and assess the item’s alignment to that combined learning goal. However, if one of the specified learning goals is actually not needed, then reviewers can lower the item’s score for necessity by one point (see Scoring Guidelines below).

• Reviewers should determine which answer choices require that students have the knowledge described in the learning goal and its clarification statement and/or know that any idea referred to in the list of misconceptions is incorrect.  In judging whether the necessity criterion is met, reviewers should focus on whether the targeted content knowledge (as opposed to some other content knowledge) is what is needed to answer correctly.  Reviewers should NOT consider whether test-wiseness or general cognitive abilities can be used to determine the correct answer. The focus here is strictly on content knowledge. 

Using your responses in the Exploration section as a guide, check all applicable boxes and provide explanations as appropriate.

1. [pic]Evaluation of every answer choice requires knowledge of the ideas specified in the learning goal and clarification statement, and/or knowledge that a commonly held misconception is incorrect. State the part of the learning goal that is needed for each answer choice or state the listed relevant misconception.

[pic]

2. [pic]Evaluation of some of the answer choices requires knowledge of the ideas specified in the learning goal and its clarification statement, and/or knowledge that a commonly held misconception is incorrect. State the part of the learning goal that is needed for each answer choice or state the listed relevant misconception. [pic]

3. [pic]Evaluation of none of the answer choices requires knowledge of the ideas specified in the learning goal and clarification statement, and/or knowledge that a listed commonly held misconception is incorrect.

[pic]

4. [pic]Evaluation of the answer choices may come from having learned a specific instance of the general principle (SIGP) being tested. Explain. Hide Details

When testing a student’s understanding of a general principle, it is sometimes difficult to distinguish the student’s knowledge of the general principle from knowledge of the particular instance used in the assessment task. For example, if the learning goal states the general principle, “every object exerts gravitational force on every other object,” and the assessment task asks about the attraction between the earth and the moon, we cannot be sure that the student understands the general principle. It is not unlikely that he/she may have learned the specific fact that gravity is the force of attraction between the earth and the moon without knowing that every object exerts gravitational force on every other object. When it is likely that a student would know the specific instance in the test item without knowing the generalization being tested, check the corresponding box and explain why you think that is likely to be true. SIGP (Necessity) scoring instructions are given below.

[pic]

score [pic] Guidelines

Scoring Necessity for Multiple-Choice (4-Choice) Items: Based on your comments in boxes 1-3 (and not yet taking into account SIGP), assign 1 point for each distractor and 2 points for the correct answer if any part of the learning goal is needed to determine the truthfulness of that answer choice and its relevance to the question being asked. (See below for scoring of SIGP)

|If the learning goal is needed to evaluate: |Score |

|All four answer choices |5 |

|Two distractors and the correct answer |4 |

|Three distractors |3 |

|One distractor AND the correct answer |3 |

|Two distractors |2 |

|The correct answer |2 |

|One distractor |1 |

|Zero distractors |1 |

SIGP Penalty for Necessity: After scoring the item as described in the table above, subtract an additional 2, 1, or 0 points, depending on the extent to which knowledge of the SIGP enables students to complete the item without knowing the targeted learning goal.

Scoring notes:

[pic]

II. B. Sufficiency Hide Details

This section examines whether having the knowledge described in the learning goal and its clarification statement, and/or the knowledge that certain misconceptions are not true, is enough by itself for students to evaluate the task and each of the answer choices.

Guide for reviewers:

• Determine if any more knowledge than is specified in the learning goal and its clarification statement, or the knowledge that certain listed student misconceptions are not true, is required to complete the task.

• Do not remark on knowledge that can be considered within the scope of the general knowledge or abilities that students of this age would be expected to have.  If questions arise about whether something can be considered “general knowledge,” explain any reservations you may have.

• When two or more answer choices can be evaluated as being true statements using the knowledge in the learning goal, but it is not possible to determine which of those is the correct answer using either the knowledge in the learning goal or what would be considered general knowledge, the sufficiency score should be lowered.

Using your responses in the Exploration section as a guide, check all applicable boxes and provide explanations as appropriate.

1. [pic]Evaluation of each of the answer choices does NOT require any more knowledge or skill than is described in the learning goal or the clarification statement, knowledge that a listed commonly held misconception is not true, or that can be assumed as general knowledge and ability for students of this age.

[pic]

2. Successful evaluation of some or all of the answer choices requires:

a. [pic]more knowledge or skill than is described in the learning goal or the clarification statement, knowledge that a listed commonly held misconception is not true, or knowledge that can be assumed as general knowledge and ability for students of this age. Specify the additional knowledge or skill that is needed to evaluate each answer choice. 

The assessment task may require knowledge of a more advanced idea than is stated in the learning goal.  For example, the task might require a molecular explanation of a phenomenon (e.g., evaporation, chemical reactions) whereas the learning goal itself specifies a substance explanation rather than a molecular explanation.  Students may very well know the exact ideas or skills in the learning goal (the substance explanation) but not the more advanced knowledge or skills required by the assessment task.  If more advanced ideas and skills are required, a brief explanation should be given in the box below.

Sometimes successfully completing a task requires the application of mathematical knowledge and skill even though it is not stated in the learning goal.  This may involve computation, graphing, solving equations, or reading tables and charts that contain numerical information.  Because it is often difficult for students to apply skills that they have learned in a mathematics class to problems in science, it should not be assumed that just because a mathematics idea has been taught in an earlier grade, students will have learned it well enough to use it in new contexts.  If there is any question about whether students should generally be expected to have the mathematics skills needed to successfully complete the task, it should be noted below.

[pic]

b. [pic]familiarity with science or mathematics terminology that may be unfamiliar to students.  State the specific terms.

In some cases, successful completion of a task requires understanding the meaning of technical science or mathematics vocabulary. If the terminology used in the assessment task goes beyond that which is used in the learning goal itself, students may very well understand the ideas or skills specified in the targeted learning goal but respond unsatisfactorily because they do not understand the technical vocabulary used in the task.

[pic]

c. [pic]knowledge of specific instances of the general principle (SIGP) that all students are not expected to have.  Explain.

If a learning goal specifies knowledge of a scientific generalization or principle, but the assessment task expects students to be familiar with a particular instance of that generalization that not all students would be expected to know, this box should be checked. For example, if the learning goal says that students are expected to know that some animals eat only plants and other animals eat animals that eat plants, a test question should not expect students to know which animals eat plants and which animals eat animals that eat plants. If the item assumes knowledge about a specific instance that is outside the expectation of the learning goal, check this box. (There are no additional scoring instructions for the case of SIGP in the context of evaluating sufficiency; please follow the table below.)

[pic]

d. [pic]familiarity with specific representations or models of ideas in the learning goal that all students are not expected to have.  Explain.

Does the task require the student to be familiar with specific representations or models of ideas in the learning goal that all students are not expected to have? For example, the strength of a force on an object can be represented by the length or the width of the arrow. Is the representation that is used in the assessment task one that students should be familiar with?

[pic]

e. [pic]Other.  Provide a brief explanation.

[pic]

score [pic] Guidelines

Scoring Sufficiency for Multiple-Choice (4-Choice) Items: Assign 1 point for each distractor that can be evaluated using no more knowledge than what is described in the targeted learning goal, and 2 points if the correct answer can be evaluated using no more knowledge than what is described in the targeted learning goal.

|If the learning goal is sufficient to evaluate: |Score |

| | |

|All four answer choices |5 |

|Two distractors and the correct answer |4 |

|Three distractors |3 |

|One distractor AND the correct answer |3 |

|Two distractors |2 |

|The correct answer |2 |

|One distractor |1 |

|Zero distractors |1 |

Scoring notes:

[pic]

III. Confounding Factors

In this part of the procedure, reviewers focus on features of the assessment task that could reduce the likelihood that students’ responses will provide information about what they know and do not know.  Students should not get a question wrong if they understand the targeted idea and they should not get it right if they do not.  Three criteria are provided: comprehensibility, appropriateness of task context, and resistance to test-wiseness.  There is also a section on cost effectiveness, in which reviewers reflect on the amount of time needed to complete the task.

III. A. Comprehensibility Hide Details

This criterion addresses whether students are likely to understand the task statement, diagrams, and symbols.  Reviewers should look for unfamiliar general vocabulary (technical, scientific, or mathematical terminology is addressed under Sufficiency) and language that may not be familiar to poor readers or students whose first language is not English.  Reviewers should also note whether photographs, diagrams, graphs, and symbols are accurate, well labeled, and intelligible.

Check all applicable boxes and provide explanations as appropriate.

1. [pic]It is not clear what question is being asked.  Explain.

[pic]

2. [pic]The task uses unfamiliar vocabulary that is not clearly defined, or words or phrases that have unclear, confusing, or ambiguous meanings.  Explain.

Some words, although correct, may lead students astray.  For example, in the phrase “scientific finding,” the word “finding” has a special meaning in science that may be unfamiliar to students.  Note all places where words, both general and scientific, do not have clear and straightforward meanings.

[pic]

3. [pic]The task uses unnecessarily complex sentence structure when plain language could have been used, or ambiguous punctuation that make the task difficult to comprehend.  Explain.

[pic]

4. [pic]There is inaccurate information in any part of the task (including diagrams and data tables) that may be confusing to students who have a correct understanding of the science. Explain.

[pic]

5. [pic]There are comprehensibility issues with the diagrams, graphs, and data tables.  Explain.

[pic]

6. [pic]There is more than one correct answer. Explain

[pic]

7. [pic]Other.  Provide a brief explanation.

[pic]

Score [pic]Guidelines

Scoring Comprehensibility for Multiple-Choice (4-Choice) Items: Problems in the stem should be weighted more heavily than problems in the answer choices because the stem applies to the entire task. Reviewers are expected to use their professional judgment regarding the overall effect of the various factors affecting comprehensibility.  The following scoring scheme should be used when some aspect of the item is thought to be incomprehensible to students.

|There are comprehensibility issues with: |Score |

|None of the answer choices |5 |

|One answer choice |4 |

|Two answer choices |3 |

|Three answer choices |2 |

|The stem |1 |

|All four answer choices |1 |

Note: If box 6 is checked (i.e., there is more than one correct answer), subtract one additional point for each answer choice that can be considered correct.

Scoring notes:

[pic]

III. B. Appropriateness of Task Context Hide Details

A task context sets up particular situations or stories involving people, objects, or events, including real-world occurrences or idealized conditions.  The context should be understandable, interesting, and sensible to students.

Although judgments about the suitability of task contexts are difficult in the absence of knowledge of students’ particular life experiences, reviewers should note contexts that they feel may be inappropriate.  In addition, all situations that are described should include plausible quantities and dimensions and should be accurate and credible.  Students should not have to spend time wondering if there is a mistake in the question.  Task contexts should be scientifically and mathematically accurate. Idealizations (such as a frictionless world) should be clearly noted.

Steps for reviewers:

a. If the task is not based on a particular situation, story, event, or object, check the first button below and move on to the next section.  Reviewers should consider any object or event that is more specific than the general statement in the learning goal to be a task context.

b. If the task does make use of a particular situation, story, event, or object (e.g. a cow, a speeding car, etc.), check the second button and consider the statements that follow. Reviewers should check these statements if they believe that the problem is significant enough to affect the student’s response.

[pic]This task has no context.  Check here and skip to the next section (III.C).

[pic]The task makes use of a context. Check all applicable boxes and provide explanations as appropriate.

1. [pic]The context may be unfamiliar to most students. Explain.

[pic]

2. [pic]The context may advantage or disadvantage students because of their interest or familiarity with the context. Explain.

[pic]

3. [pic]The context is complicated so that students might have to spend a lot of time trying to figure out what the context means. Explain.

[pic]

4. [pic]The information and quantities that are used are not reasonable or believable. Explain.

[pic]

5. [pic]The context does not accurately represent scientific or mathematical realities or make clear when idealizations are involved. Explain.

[pic]

6. [pic]Other. Provide a brief explanation.

[pic]

score [pic]Guidelines

Scoring the Appropriateness of Task Context for Multiple-Choice Items: If you checked none of the boxes above, enter a score of 5 points. Deduct 0-1 point (from a total of 5 points) for each statement box that you checked above. Deduct points only if you think students’ answers could be significantly affected because of this task context.

Scoring notes:

[pic]

III. C. Resistance to Test-Wiseness Hide Details

This criterion addresses characteristics of the task that might (1) allow students to make a satisfactory response using only general test-taking skills without understanding the idea being tested or (2) mislead students into choosing an incorrect answer.  For example, in multiple choice questions, distractors should be plausible to students and one answer (especially the correct answer) should not be distinctly different from the other answers.  One answer should not be significantly longer or more elaborate.  It should not be more qualified than the others, using terms such as “always,” “never,” or “everyone.”  It is also important to pay attention to the use of logical opposites that may make it easy to eliminate answer choices.  Lists of common student misconceptions (and interview data, if available) are provided to help identify plausible distractors.  These test-wiseness issues should be noted and explained in the boxes below.

Check all applicable boxes and provide explanations as appropriate.

1. [pic]Some of the distractors are not plausible. Explain.

[pic]

2. [pic]One of the answer choices differs significantly in length or contains a different amount of detail from the other answer choices. Explain.

[pic]

3. [pic]One of the answer choices is qualified differently from the other answer choices, and/or uses giveaway words such as “usually” or “never.” Explain.

[pic]

4. [pic]The use of logical opposites leads the student to eliminate the other answer choices.  Explain.

[pic]

5. [pic]One of the answer choices contains vocabulary at a different level of difficulty from the other answer choices. Explain.

[pic]

6. [pic]The language in one of the answer choices mirrors or is obviously closely related to the language in the stem. Explain.

[pic]

7. [pic]Other.  Provide a brief explanation.

[pic]

score [pic]Guidelines

Scoring Resistance to Test-Wiseness for Multiple-Choice (4-Choice) Items: If you checked none of the boxes above, enter a score of 5 points. Deduct 0-1 points (from a total of 5 points) for each distractor, and 1-4 points for the correct answer choice, if they can be evaluated (successfully or unsuccessfully) through test-wiseness. The table below gives scoring ranges for some possible situations. The ranges are intended to allow reviewers to use their professional judgment regarding the overall effect of test-wiseness. In particular, the large range associated with the correct answer choice allows reviewers to assess a severe penalty if the student can select the correct answer without any content knowledge. Note: Successfully evaluating the correct answer choice means identifying it as the correct answer, and successfully evaluating a distractor means identifying it as a wrong answer. Unsuccessfully evaluating the correct answer choice means thinking that it is a wrong answer, and unsuccessfully evaluating a distractor means thinking that it is the correct answer.

|  |Score |

|No test-wiseness issues |5 |

|One distractor can be eliminated using test-wiseness methods |4-5 |

|Two distractors can be eliminated using test-wiseness methods |3-5 |

|Three distractors can be eliminated using test-wiseness methods |1-4 |

|The correct answer can be determined using test-wiseness methods |1-4 |

Scoring notes:

[pic]

III. D. Cost-Effectiveness Hide Details

This section is provided to determine whether the task seems to require a great deal of time and effort for students to complete, relative to other items of this type. The focus here is primarily on the amount of time and effort required, and the goal is simply to flag the most costly items.  Do not consider comments made under necessity, sufficiency, comprehensibility, and test-wiseness in the determination of cost-effectiveness.  This is not a measure of how much we are able to learn from the item about what students know.

Check the applicable box and provide explanations as appropriate.

• [pic]The item does not require an inordinate amount of effort or time to complete. If you have any comments, please enter them here.

[pic]

• [pic]The item requires an inordinate amount of effort or time to complete.  Provide a brief explanation of why the task is not cost-effective and how the same information might be elicited more efficiently. 

[pic]

Compared to other items of this type, does it take a long time for students to:

[pic]Read the question

[pic]Decipher the answer choices

[pic]Interpret the graph, read the data table

[pic]Make calculations

[pic]Other

score [pic] Guidelines

Scoring Cost-Effectiveness: If you checked none of the boxes above, enter a score of 5 points. If you checked any of the boxes above, use your professional judgment to deduct points accordingly.

Scoring notes:

[pic]

IV. Revisions

Suggestions for improving the task or reasons for eliminating the task from further consideration should be listed below. If the item is satisfactory as written, the corresponding box should be checked.

Check the appropriate box and provide suggestions and explanations as appropriate.

• [pic]The task is effective and does not need to be revised.

[pic]

• [pic]The task should be revised.  Please make specific suggestions for revision.

[pic]

• [pic]The task should be eliminated from further consideration. Please explain.

[pic]

[pic]

Bottom of Form

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download