Testing Listening - Research, Spring 2013.docx



ITBE Workshop, April 20, 2013Assessing Listening Comprehension: Test Format DecisionsBrian Hampson, Purdue University Calumet - brian.hampson@purduecal.eduHeather Torrie, Purdue University Calumet – torrieh@purduecal.edu Test Development Stages (adapted from Hughes, 2003)Stating the problemWhat is the purpose?What abilities are to be tested?ContentWhat tasks should students perform?What texts should be used? What is the overall format (number of passages, number of listenings, etc)?What items should be included?Pilot items – using both native and non-native speakersValidity: Does the test measure what it is supposed to measure? (Hughes, 2003) Sometimes format can affect validity Overview of Various Item FormatsBuck (1991) “…successful listening comprehension involves an interaction between linguistic skills, knowledge of the context, background knowledge and inferencing skills. Thus, listening test items, even those written to test one particular skill, turn out on examination to be testing a number of different skills.”FormatResearch NotesProsConsOther skills being testedMultiple-choiceTrue/FalseMatchingSelection In’nami & Koizumi, 2009 - Meta-analysis shows MC is easier than SAYi’an 1998 - people often choose the right answer for the wrong reasons-Reliability -Less stressful for testees-Takes less time for testees-Encourages bottom-up processing -Cognitive Load: difficult for listeners to hold four options in their mind while listening-Guessing-Choose the right answer for the wrong reason-Difficult to write good distracters-Cheating -Reading ability-Only word recognition (rather than true comprehension)Short AnswerIn’nami & Koizumi, 2009 - Meta-analysis shows SA more difficult than MCBuck 1991 – Supports validity, but addresses concerns with reliability-Less guessing -Easier to write items-More authentic-More top-down, especially for main idea questions-Reliability issues (scoring difficult, especially for inference items)-Takes time for them to answer -Writing ability-Reading (understanding the question and determining which information to write down)FormatResearch NotesProsConsOther skills being testedTable/Outline/Chart CompletionSong 2011 - Filling out a table is easier than blank notesBrindley & Slatyer, 2002 – found that a chart/table structure was easier than SA/blank notes, and easier than cloze- Constrains the contents of test takers’ notes to a given framework-Authentic-emphasizes top-down processing -Scoring difficult; reliability issues -Writing abilityRe-callUsed in research (eg, Jung 2003; Sherman 1997); not so much in classroom -Authentic -Good measure of intake -More difficult to score (compile a list of key information units)-Writing and speaking ability-Memory-Note-taking abilityCloze / Dictation / Partial Dictation-Reliability -Emphasizes bottom-up listening -Less authentic -Writing ability Test Delivery Format Options:# of ListeningsRationaleQuestion Preview*RationaleOne time-Authentic Before-Helps students focus on particular informationTwo times through-Affective value for students -Reflects the way listening is taught in the classroom Sandwiched -Promotes top-down processing during the first listening; and bottom-up processing during the second listeningMore than twice / student-controlled-Also can be authentic (listening to recorded lectures, online material, conversation)Afterwards -Further promotes top-down processing*Sherman (1997) and Buck (1991) suggest that question preview seemed to have more of an affective benefit than actual performance benefit. Examinees thought it helped them more than it actually did. ReferencesBrindley, G & H. Slatyer. (2002). Exploring task difficulty in ESL listening assessment. Language Testing, 19(4). Uses charts and sentence completion. One listening only. Buck, G. (1991). The testing of listening comprehension: an introspective study. Language Testing, 8(1).Using verbal report, this study looked at the test-taking process of answering short-answer questions based on listening to segments of a narrative. While suggesting strength in validity with using short-answer items, the study reveals concerns with reliability over the various answers. Cross, J. (2009). Effects of listening strategy instruction on news videotext comprehension. Language Testing Research, 13(2). 151-176. One of the ways listening comprehension was measured in this study was using written recalls, where they had to write down everything they remembered from the videotext.Hughes, A. (2003). Testing for Language Teaching. Cambridge University Press. This book is a great overview on test development for all language skills.In’nami & Koizumi. (2009). A meta-analysis of test format effects on reading and listening test performance: Focus on multiple-choice and open-ended formats. Language Testing, 26(2).After reviewing 56 or so studies, found that multiple-choice items were indeed easier than open-ended formats.Jung, E.H. (2003). The role of discourse signaling cues in second language listening comprehension. The Modern Language Journal, 87(4). 562-577. Learners performed a written recall task, after listening to a lecture. Assessment was based on how many key “information units” learners included in the recall. Sherman, J. 1997. The effect of question preview in listening comprehension tests. Language Testing, 14. 185-213.Very interesting study design! Lots of great citations too. Seems that question preview often makes students feel better, but doesn’t necessarily help them. Could interfere with processing. Song, M. (2011). Note-taking quality and performance on an L2 academic listening test. Language Testing, 29(1).Studies how effective note-taking using a partially-filled outline. “it would seem that notes taken in the outline format in particular, because it constrains the contents of test takers’ notes to a given framework, might have more potential as a listening measure than notes in the blank format.”Yi’an, W. (1998). What do tests of listening comprehension test? - A retrospection study of EFL test-takers performing a multiple-choice task. Language Learning.Did a qualitative study on 6 learners and why they chose the answers they did (multiple-choice). Showed a lot of people chose the correct answer, but for the wrong reason. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download