Item Writers’ Course: Līgatne, 14 - 17 February, 2002
Writing tests at Universities
ESF Testing project, Vilnius, January 29-30, 2007
Vita Kalnberzina, Dr.Phil., Assist.Prof., Latvia University
Contents
Writing tests at Universities 1
Contents 1
List of Tables 1
Course Programme 2
'Writing' as a concept 3
What do you understand by the term 'writing'? 3
What does the term 'writing' mean to the student/test-taker? 5
How is written language different from spoken? 5
Is writing informational, interactional or both? 6
Teaching Writing at University Level 7
Product approach 7
Conventions For Term Paper Writing 7
Process approach 8
Help for students at Dartmouth University 8
Instructions to Writing assistants: 8
Moving from Writer-Based to Reader-Based Prose 9
Writing test development 9
Indirect approach 9
Direct approach 10
What do you need to know to develop a task? 10
Writing Tests at University 13
Qualspell 13
What task types can be used to test writing proficiency at the University? 13
Which text/task-types are appropriate for your students? 14
How do you know that you have produced a good task? 15
What does a test-taker need to know to do a task: develop the test specification 15
Marking writing tests 15
Formative Assessment 16
Summative Assessment 16
Holistic marking scales 16
Analytic marking scales 18
Assurance of test quality 20
Validity 20
Reliability of the examination 21
Inter and Intra-marker reliability 21
Bibliography 23
Online sources 23
List of Tables
Table 1 You as a writer 3
Table 2 What is writing? 4
Table 3 Writing from the point of view of the test-taker 5
Table 4 Comparison between written and spoken language 5
Table 5 Interactional versus informational writing 6
Table 6 Comparison between direct and indirect writing tests 10
Table 7 Topics for writing tasks, Compare with Language Domains in CEF 11
Table 8 Writing as a process (Callaghan, Knapp, and Noble 1993, p.193) 14
Table 9 Impressionistic marking scale 16
Table 10 CEF levels of writing production 16
Table 11 Holistic scale (British Council IELTS written production evaluation) 17
Table 12 BC analytical writing evaluation scale 18
Table 13 Validity of the writing tasks 21
Table 14 Reliability of the Year 12 examination in 1999 22
Table 15 Intercorrelations between the different tests in 2001 22
Course Programme
|January 29 |
|9:00 – 10:30 |session |Introduction, Writing as a concept |
|10:30 |coffee break | |
|11:00 – 12:30 |session |Approaches to testing writing |
|13:30 |Lunch | |
|15:00 – 16:30 |session |Writing tests and tasks |
|16:30 |tea break | |
|17:00 – 18:30 |session |Marking scale types and marking reliability |
| |
|30 January |
|9:00 – 10:30 |session |Development of new writing tasks |
|10:30 |coffee break | |
|11:00 – 12:30 |session |Development of marking scales |
|13:30 |Lunch | |
|15:00 – 16:30 |session |Pretesting of the new writing tasks and marking scales |
|16:30 |tea break | |
|17:00 – 18:30 |session |Feedback on the new writing tasks and marking scales and|
| | |evaluation of the course |
'Writing' as a concept
Objective: Discuss the concept of writing and hence validity of writing tests
What do you understand by the term 'writing'?
Take a couple of minutes to answer the following questions:
Table 1 You as a writer
|Question |Answer |
|What do you write? | |
| | |
|How do you write? | |
| | |
|What are three main things you want your students to learn about | |
|writing? | |
| | |
|How would you define 'writing' as a skill? | |
| | |
Read the 5 texts and fill in the table 3 below:
1. Writing is both tool and art, but it is not a body of knowledge. Those who have taught writing and observed writers at work are convinced that writing is a complex, multifaceted, dynamic skill, affected by cognitive demands, affect (whether the writer feels good on that day, whether he or she likes the topic, or the reader of the writing, or the reasons of writing) and variations in test conditions. Above all writing is a cognitive activity that calls on many components of personal experience, attitude, knowledge of ideas, issues and factual information, but that requires a person to take and shape some parts of this complex web of material into a rhetorical, syntactic and mechanical whole (Hamp-Lyons 1993, p.11).
2. The writing test tests the test-takers’ ability to write correct and appropriate sentences, use conventions peculiar to written language, think creatively and develop thought excluding irrelevant information, manipulate sentences and paragraphs to use language effectively, write in an appropriate manner with a particular audience in mind, organise and order the written material (Year 12 test specification).
3. Textual competence includes the knowledge of the conventions for joining utterances together to form a text consisting of two or more sentences which are structured according to the rules of cohesion (marked relationships among sentences) and rhetorical organisation (development of the text) (Bachman 1990)
4. A successful writer needs to have content knowledge (knowledge of the concepts involved in the subject area), context knowledge (social context in which the text will be read, the reader's expectations, knowledge of the cotexts alongside which this text will be read), language system knowledge (lexis, syntax) and writing process knowledge (the most appropriate way of preparing for a writing task) (Tribble 1995).
Table 2 What is writing?
| |Year 12 test |Hamp-Lyons (1993) |Tribble/Bachman (1990) |
| |specification (1999) | | |
|What is writing? | | | |
| | | | |
| | | | |
|What subskills does | | | |
|writing consist of? | | | |
| | | | |
| | | | |
| | | | |
| | | | |
|What is it affected by? | | | |
| | | | |
| | | | |
| | | | |
What does the term 'writing' mean to the student/test-taker?
Table 3 Writing from the point of view of the test-taker
|Question |Answer |
|What do your students write? | |
|In what settings do they like/do not like to write? | |
|What factors inhibit/encourage them to write? | |
|What contexts stimulate them to write well? | |
How is written language different from spoken?
Read the extract below and say what language aspects are lost when we transform spoken language to written language? How can we compensate for the loss? Is this enough?
Contemporary views of the differences between written and spoken language support the idea that
1. written and spoken language do possess distinctive features and
2. that texts can be distributed along the continuum from the most typically spoken to the most typically written.
Apart from the loss of the contextual information that is available to the people who participate in a conversation (for example, knowledge of the background and the histories of the individuals concerned) two of the most important aspects of speaking that are normally lost in any transcription are prosodic (rhythm, phrasing, pauses) and paralinguistic features (the way someone is speaking, facial expressions and gestures). To compensate for this loss we use punctuation and other features (Tribble 1995, p.17, 18).
Fill in the table choosing from the box below:
Table 4 Comparison between written and spoken language
| |spoken |written |
|vocabulary | | |
| | | |
|Grammar | | |
| | | |
| | | |
|Text | | |
Incomplete utterances, full word forms, rare words, 'incorrect' grammar, vocabulary of Latin origin, complete sentences, hesitations, reduced word forms, informal vocabulary, many phrasal verbs,
common words, conventional abbreviations, polysyllabic words, high density of content words, sophisticated and intricate grammatical resources, heavily nominalized style, clauses instead of sentences as a minimum unit, carefully planned paragraphs
Is writing informational, interactional or both?
Weir (1995) considers that there are two types of writing operations: interactional and informational.
Fill in the table choosing the operations from the box below
Table 5 Interactional versus informational writing
|Type of operation |Operations/action |Product |
|Interactional in social| | |
|and service texts: | | |
| | | |
|Informational in | | |
|academic contexts | | |
Summary checklist of writing operations (Weir 1995)
• Describing process which might involve: purpose, describing means, results, process, change of state, sequential description, instructions, summary
• Describing phenomena and ideas: definition, classification, identification, comparison and contrast, exemplification,
• Arguing: stating a proposition, stating assumptions, induction, deduction, substantiation, concession, summary, generalisation, speculation, comment, evaluation
• Expressing thanks, requirements, opinions, attitude, confirmation, apology, wants, needs, lacks, ideas, information, complaints, reasons, justifications
• Eliciting information, directions, service, clarification, help, permission
• Directing: ordering, instructing, persuading, advising, warning
Teaching Writing at University Level
Product approach
Conventions For Term Paper Writing
Students having registered for the academic BA programme (English Philology) will be eligible to write 3 term papers before writing their Bachelor Paper (BA paper).
Students can choose themes for term papers in the following areas:
1. Aspects of the English Language (Phonetics, Grammar, Lexis);
2. British (American, Canadian) studies;
3. English Language Teaching (ELT) Methodologies;
4. Linguistics;
5. Translation and Interpreting;
6. Literature.
Length Of The Term Paper
The term paper will be between 5000 words and 7500 words.
You are not required to produce a term paper embodying totally original work and ideas that will be a contribution to knowledge, although if you do, you will receive appropriate credit for it.
Organisation And Content Of The Term Paper
Conventionally, there will be:
Title page
Declaration of academic integrity
Abstract in English and Abstract in Latvian
Table of contents
Introduction;
Chapters;
Conclusions;
Theses;
Bibliography;
Appendix (or appendices).
Plagiarism
You plagiarize if you take someone else’s work, ideas, words and use them as if they were your own. You can avoid plagiarism by using correct methods for quoting, paraphrasing and referencing. Quoting means using the exact words of the writer/speaker, whereas paraphrasing means restating the words and ideas from a book, an article, or a lecture in your own words.
Failure to produce references amounts to plagiarism, and your work will be deemed invalid.
Layout And Presentation
Use A4 size white paper on one side.
Word process the text using Times New Roman.
Use twelve point for the main text; fourteen point for headings.
Use margins: top/bottom/right -2.5 cm; left-3.5 cm (for binding);
Use 1.5 line spacing;
Indent quotes longer than three lines by five characters; use single space between the lines;
Process approach
Principles of process pedagogy at Dartmouth University:
1. writing can be understood as the culmination of several steps in a complicated process, which include prewriting, writing, rewriting, and all their attendant strategies
2. use of dialogue as a teaching technique.
3. challenge the traditional, authoritative models of teaching, in which professors (who know everything) talk "at" students (who know very little).
4. empower students by getting them to talk about their writing at every step of the writing process.
Help for students at Dartmouth University
• If you are a first-year student, we will help you understand the expectations of college writing.
• We will help you generate, research, and organize ideas.
• We will talk to you about your structure, grammar, and style.
• If you are a student for whom English is a second language, we will help you work on English grammar and style. We will also help you understand the conventions for writing an essay in English.
• If you are a senior, we will provide strategies for writing a thesis or culminating experience paper.
• We will even help you with multi-media compositions, such as Web sites, videos, or PowerPoint presentations.
Instructions to Writing assistants:
• Re-read the assignment, if the professor has written one. Make careful notes regarding what, precisely, the professor is asking the students to do.
• Read through all (or several) of the papers once, without marking. This strategy helps you to get a sense of the range of essays you'll be reading.
• Read through each paper twice: once to get the gist, the second time to respond. This strategy helps you to see where the writer is going before you jump in with your advice.
• Respond the first time through, but in pencil. That way, you can erase your remarks if something the writer says on page four changes the way you feel about page two.
• Take notes, on the side. Sometimes writing assistants want to keep a running commentary of their reading responses, but don't wish to share these responses with the writer. Keep a pad and a pen nearby.
Moving from Writer-Based to Reader-Based Prose
At some point in the writing process, writers must turn their attention from the writing process to the written product. They must transform writer-based prose into reader-based prose. To do this, they must be ready to revise, paying close attention to their prose and to the effect it may be having on their audience.
Often it's precisely at this point in the process that a writer will come to you for help. He has a draft completed, his argument seems logical and even persuasive, but he wants a second opinion. You read the essay and have some trouble following his line of reasoning. You point to a paragraph that is particularly confusing, and you ask what the writer is trying to say. He responds, "But it's all right there!" and goes on to summarize a point that he clearly hasn't made. What's happened? The writer's point is so firmly entrenched in his mind that he really believes that it's "in" the essay. You show him that it's not. You've just given him a lesson in the difference between writer-based and reader-based prose.
Writing test development
How do you develop writing tests? How do you test writing?
How often do you test? How do you mark writing?
What problems do you have?
Indirect approach
Hamp-Lyons (1993, p.5):
Indirect tests (recognize correct word order or structure, choose the correct phrase for the gap, underline the phrase that contains error etc.) do not require the test-taker to write continuous prose, there is little room for personal interpretation, a key is provided for the marker, the role of human judgement is less obvious, as scores are not tied to essay samples or rating scales, but to the judgement of the test developers who have developed questions and decided on the correct answers. In this case training of markers is not necessary and scoring can be done clerically.
In indirect discrete element writing tests (TOEFF, MELAB) knowledge of grammar rules and vocabulary items tend to dominate. While these skills may be related to proficient writing, most of us do not accept that they can represent what proficient writers do. Therefore indirect writing assessment is invalid. It has extremely negative educational consequences on the learning/teaching process as it creates the impression that writing is the same as recognizing errors in sentences, knowing grammar rules, and of not writing about ideas and emotions.
Direct approach
A direct test of writing has the following characteristics:
1. each individual taking the assessment must actually physically write at least one piece of continuous text of 100 words and may write several considerably longer pieces
2. the writer is provided with a set of instructions and a text, picture or other prompt material, he or she is given considerable room within which to create a response to the prompt
3. every text written by a candidate is read by at least one , usually two or more, human reader-judges who have been through some form of preparation process
4. the judgements of the readers are tied in some way to some common yardstick, such as a set of sample essays, description of the expected performance or one or several marking scales
5. scores on the text are recorded and can be retrieved for review
The ability of a writing test to give reliable results is affected by the amount of variation in things like:
1. topic he or she is asked to write about
2. the type of written task (narrative, argumentative, etc.)
3. the amount of time available
4. whether it is done on word processors
5. how many samples we get from each writer
Fill in the table:
Table 6 Comparison between direct and indirect writing tests
| |Sample task |Positive |Negative |
|Direct approach to testing writing| | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
|Indirect approach to testing | | | |
|writing | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
What do you need to know to develop a task?
Questions:
1. What is a test task? How is it different from a learning task?
2. What does a test task consist of?
3. What are the criteria in a test task evaluation?
Hughes (1989) offers a following writing task framework:
1. Operations (describe, explain, compare, contrast, argue)
2. Text type: letter, form, essay, postcard
3. Addressee: native speaker, non-native speaker or as specified under topics
4. Topic (see Table 7)
Table 7 Topics for writing tasks, Compare with Language Domains in CEF
|Topic: |Rank order of appropriacy |
| |for your student |
|social interaction with native and non-native speakers of English | |
|shopping and using services | |
|travelling and arranging for travel | |
|dealing with official and semi-official bodies | |
|visiting places of interest and entertainment | |
|using media for information and entertainment | |
|studying for academic, occupational and social purposes. | |
|medical attention and health | |
| | |
Use Hugh's (1989) framework to analyse Year 12 examination tasks in 2001
What operations, addressee, text type and what topic?
| |Task 1 |Task 2 |Task 3 |
|Operations | | | |
|Text type: | | | |
|Addressee: | | | |
|Topic | | | |
Year 12 Writing test
Task 1
Your friend wants to rent his seaside house to Mr and Mrs Crawford for the summer holidays. They asked him to describe it in detail, but his English is not very good. Help him to write the letter (100 words). Here are his notes:
A wooden cottage near the Lielupe
A kitchen and living room downstairs
Two bedrooms and a bathroom upstairs
Garden, a garage
Task 2
Describe the diagram in 100 words and compare the popularity of the different study programmes (Diena June 2000).
Task 3
The role of sports in the modern world
Justice Earl Warren says that the sports page records people's accomplishments, but the front page nothing but their failures.
Write an essay on the role of sports in the modern world (200-250 words) and discuss:
What sports means to you and your friends
Whether you follow the results of Latvian athletes in international competitions
Whether it is important for Latvian athletes to win international competitions or not (why)
Whether the government of Latvia should support sports clubs, teams (if yes then how and why) or spend money for more urgent needs of the Latvian people
Writing Tests at University
Qualspell
ESP:C, an English Course for Chemists With the support of [pic]
Task 1
Give written instructions to a new laboratory assistant on how to prepare a solution of salt in water. Say where the necessary chemicals can be found, in which vessel(s) they must put in order to dissolve the salt, how the solution is treated after use and how it is stored.
(It is important that your instructions are logically structured, easy to understand and very clearly expressed. Give your instructions in 8 – 10 short sentences.)
Task 2
Genetics is most important in modern science.
Do you agree with this statement or would you rather disagree with it?
Present your arguments in 120 - 150 words. Structure your ideas logically. Do not forget to make clearly visible paragraphs.
Task 3
Write an e-mail to an important customer in which you inform the company about a serious delay and in which you apologize for this delay.
The reasons for the delay: You bought a new analytical instrument but you have problems with this machine as the machine cannot be operated. You hope the maintenance works will not take too long.
You have had very good relations with this customer for many years. So you hope that despite this delay you will be able to continue this good the co-operation.
Your reply should be very polite and explain the situation as given.
What task types can be used to test writing proficiency at the University?
Year 12 test specification says: The test-takers demonstrate their ability to do three of the following: to write a letter, a postcard, a set of instructions, a report or a guided essay or fill in forms. In all cases test-takers are asked to respond to leaflets, notices, announcements, personal notes and messages, directions, tables and graphs.
Which text/task-types are appropriate for your students?
Look through the Table 11and tick the text types that would be appropriate for your students, choose one type and develop the task as you would present to your students
Include the text type, length, addressee and time limit, criteria, optional.
Table 8 Writing as a process (Callaghan, Knapp, and Noble 1993, p.193)
|Action |Process |Product |Yes/No |
|Describe |Through the process of sequencing phenomena |Personal descriptions | |
| |in temporal and causal relationships | | |
| | |Technical descriptions | |
| | |Information reports | |
| | |Scientific reports | |
| | |Definitions | |
|Explain |Through the process of sequencing phenomena |Explanations of how | |
| |in temporal or causal relationships | | |
| | |Explanations of why | |
| | |Elaborations | |
| | |Illustrations | |
| | |Accounts | |
| | |Explanation essay | |
|Instruct |Through the process of logically ordering a |Procedures | |
| |sequence of actions or behaviours | | |
| | |Instructions | |
| | |Manuals | |
| | |Science experiments | |
| | |Recipes | |
| | |Directions | |
|Narrate |Through the process of sequencing people and |Personal recounts | |
| |events in time and space | | |
| | |Stories | |
| | |Fairytales | |
| | |Myths and fables | |
| | |Narratives | |
|Argue |Through the process of persuading readers to |Essays | |
| |accept a logical ordering of propositions | | |
| | |Expositions | |
| | |Discussions | |
| | |Debates | |
| | |Reviews | |
| | |Evaluations | |
How do you know that you have produced a good task?
How difficult the task will be?
| |Concrete |Abstract |
|Personal |1 |2 |
|Public |3 |4 |
Weir 1995 p.22-25: Specific guidelines for moderation of a task: format
1. Is the level of difficulty appropriate?
2. Will the task discriminate?
3. Is it appropriate?
4. Is it clear?
5. Is the timing appropriate?
6. Is the layout appropriate?
7. Is there a bias against some group of students?
What does a test-taker need to know to do a task: develop the test specification
What are the student's rights? What do they know before they start writing?
Hamp-Lyons (1993, p.61) considers that a student has the right to full knowledge about the test she or he is taking and should be informed about:
|test purpose |
|format |
|test-length |
|number of questions and score weighting if any |
|kinds of writing to be evaluated |
|criteria |
|scoring method |
|score reliability |
|qualification of judges |
Tick the information in the table above that was available for the year 12 exam test takers in year 2001 (see below)
Marking writing tests
Formative Assessment
Formative assessment is often done at the beginning or during a program, thus providing the opportunity for immediate evidence for student learning in a particular course or at a particular point in a program. Classroom assessment is one of the most common formative assessment techniques. The purpose of this technique is to improve quality of student learning and should not be evaluative or involve grading students.
Summative Assessment
Summative assessment is comprehensive in nature, provides accountability and is used to check the level of learning at the end of the program. For example, if upon completion of a program students will have the knowledge to pass an accreditation test, taking the test would be summative in nature since it is based on the cumulative learning experience.
Holistic marking scales
Holistic scoring involves the assignment of a single score to a piece of writing on the basis of an overall impression of it (hence the name 'impressionistic scoring'). It is rapid and therefore each paper can be scored several times. Sample scale used at an English medium university
Table 9 Impressionistic marking scale
|NS |Native speaker standard |
|NS- |Close to a native speaker standard |
|MA |Clearly more that adequate |
|MA- |Possibly more than adequate |
|A |Adequate for study at this university |
|D |Doubtful |
|NA |Clearly not adequate |
|FBA |Far below adequacy |
Holistic scales are based on overall level descriptions, which can also be expressed in 'can statements'.
CEF describes its levels of overall written production in a following scale starting with A1 (the lowest level) and finishing with C2 (the top level). Fill in the names of the levels in Table 13
Table 10 CEF levels of writing production
|Description |Level |
|Can write straightforward connected texts on a range of familiar subjects within his field of interest by | |
|linking a series of shorter discrete elements into linear sequence | |
|Can write simple isolated phrases and sentences | |
|Can write clear, well structured texts of complex subjects, underlining the relevant salient issues, expanding | |
|and supporting points of view at some length with subsidiary points, reasons and relevant examples, and | |
|rounding off with an appropriate conclusion | |
|Can write clear, detailed texts on a variety of subjects related to his/her field of interest, synthesising and| |
|evaluating information and arguments from a number of sources | |
|Can write a series of simple phrases and sentences linked with simple connectors like 'and', 'but', 'because' | |
|Can write clear, smoothly flowing, complex texts in an appropriate and effective style and a logical structure | |
|which helps the reader to find significant points | |
What are the criteria used to differentiate between the levels in CEF writing production level description?
Table 11 Holistic scale (British Council IELTS written production evaluation)
|9 |The writing displays an ability to communicate in a way which gives the reader full satisfaction. It displays a |
| |complete logical organisational structure which enables the message to be followed effortlessly. Relevant arguments are|
| |presented in an interesting way, with the main ideas prominently and clearly stated, with completely effective |
| |supporting material, arguments are effectively related to the writer's experience or views. There are no errors of |
| |vocabulary, spelling punctuation or grammar and the writing shows an ability to manipulate the linguistic systems with |
| |complete appropriacy |
|8 |The writing displays an ability to communicate without causing the reader any difficulties. It displays a logical |
| |organisational structure, which enables the message to be followed easily. Relevant arguments are presented in an |
| |interesting way, with main ideas highlighted, effective supporting material and they are well related to the authors |
| |own experience or views. There are no significant errors of vocabulary, spelling, punctuation or grammar and the |
| |writing reveals an ability to manipulate the linguistic systems appropriately. |
|7 |Writing displays an ability to communicate with few difficulties for the reader. It displays good organisational |
| |structure, which enables the message to follow without much effort. Arguments are well presented with relevant |
| |supporting material and an attempt to relate them to writer's experience or views. The reader is aware of but not |
| |troubled by occasional minor errors of vocabulary, spelling, punctuation and grammar or some limitations to the |
| |writer's ability to manipulate the linguistic systems appropriately. |
|6 |The writing displays an ability to communicate although there is occasional strain for the reader. It is organized well|
| |enough for the message to be followed throughout. Arguments are presented but it may be difficult for the reader to |
| |distinguish main ideas from supporting material; main ideas may not be supported; their relevance may be dubious; |
| |arguments may not be related to the writer's experience or views. The reader is aware of errors of vocabulary, |
| |spelling, punctuation or grammar, and/or limited ability to manipulate the linguistic systems appropriately, but these |
| |intrude only occasionally. |
|5 |The writing displays an ability to communicate although there is often strain for the reader. It is organized well |
| |enough for the message to be followed most of the time. Arguments are presented but may back relevance, clarity, |
| |consistency or support; they may not be related to the writer's experience or views. The reader is aware of errors of |
| |vocabulary, spelling, punctuation or grammar, which intrude frequently, and of limited ability to manipulate the |
| |linguistic systems appropriately. |
|4 |The writing displays a limited ability to communicate, which puts strain on the reader throughout. It lacks a clear |
| |organizational structure and the message is difficult to follow. Arguments are inadequately presented and supported; |
| |they may be irrelevant; if the writers experience or views are presented, their relevance may be difficult to see. The |
| |control of vocabulary, spelling, punctuation and grammar is inadequate, and the writer displays inability to manipulate|
| |the linguistic systems appropriately. |
|3 |The writing does not display an ability to communicate although meaning comes through spasmodically. The reader cannot |
| |find any organizational structure and cannot follow a message. Some with an argument is mainly irrelevant. The reader |
| |is primarily aware of gross inadequacies of vocabulary, spelling, linguistic appropriacy, although there is no evidence|
| |of sentence structure. |
|2 |The writing displays no ability to communicate. No organisational structure or message is recognizable. A meaning comes|
| |through occasionally but it is not relevant. There is no evidence of control of vocabulary, spelling, punctuation or |
| |grammar and no sense of linguistic appropriacy |
|1 |A true non-writer who has not produced any assessable strings of English writing. An answer which is wholly or almost |
| |wholly copied from the input text or task is in this category |
|0 |Should only be used where a candidate did not attend or attempt this part of the test in any way |
Analytic marking scales
Methods of scoring that require a separate score for each of a number of aspects of tasks are said to be analytic.
Transform the holistic marking scale in Table 11 into an analytical marking scale in Table 12
Table 12 BC analytical writing evaluation scale
| | | | | | |
|9 | | | | | |
|8 | | | | | |
|7 | | | | | |
|6 | | | | | |
|5 | | | | | |
|4 | | | | | |
|3 | | | | | |
|2 | | | | | |
|1 | | | | | |
QUALSPELL Marking scale
| |Grammar |Mechanical Accuracy |Vocabulary |Structure |Content and Style |
|5 |Wide range and correct|Almost no errors of |Wide range of |Completely logical, |Arguments effectively related|
| |use of complex |spelling, |vocabulary, appropriate |varied use of cohesive|to specific area within |
| |structures, very few |punctuation, |register, consistent |structures, complex |topic, interesting |
| |minor errors |capitalization |style |sentence structures |presentation of topic, |
| | | | | |appropriate tone and register|
|4 |Good general command |Occasional errors |Good range of vocabulary|Generally logical, |Arguments mostly related to |
| |of grammar with no |that do not obscure |(not necessarily |sufficient use of |the specific area within the |
| |systematic errors |the meaning |outstanding, some |various cohesive |topic, mostly appropriate |
| | | |circumlocutions, some |devices, complex |tone and register |
| | | |errors of word forms, |sentence structure | |
| | | |choice, usage, but |with occasional errors| |
| | | |meaning not confused | | |
|3 |Accuracy high, some |Frequent or |Limited vocabulary, but |Inconsistent |Arguments related to the |
| |gross or systematic |systematic errors, |largely correct, |structure, but easily |general topic, occasional |
| |errors and more minor |but meaning mostly |sometimes “adventurous” |to follow most of the |inappropriate tone and |
| |errors which do not |not confused or |choice of words, meaning|time, occasionally |register |
| |distort communication |obscured |not confused |repetitive, some | |
| | | | |variety of sentence | |
| | | | |structure, some use of| |
| | | | |cohesive structure | |
|2 |Has a basic grasp of |Frequent or |Range inadequate to |Loosely organised, but|Written mostly to the point |
| |the grammar system |systematic errors, |express ideas |main ideas expressed, |with occasional lapses, |
| |which is not applied |meaning confused or |effectively, frequent |can be followed, but |inconsistent in tone and |
| |consistently, frequent|obscured, poor |errors of words, idioms,|requires some |register |
| |occurrence of errors |handwriting |form, usage, meaning |re-reading, sentences | |
| | | |sometimes confused |short and stereotyped,| |
| | | | |attempts at using | |
| | | | |cohesion | |
|1 |Very frequent |Dominated by errors,|Limited range, |Lack of basic |Limited treatment of topic, |
| |occurrence of gross |meaning confused or |Misuse of words makes |organisation, only |major points left out, |
| |errors, grammatical |obscured, serious |communication difficult,|simple sentences or |insufficient length, mostly |
| |patterns based on |misspellings of |essentially translation |serious, frequent |inappropriate style |
| |direct translation |elementary words |from L1 |errors in sentence | |
| | | | |structure, not | |
| | | | |cohesive | |
|0 |Completely |Misspellings of |Misuse of words breaks |Vague, confused and | |
| |ungrammatical |basic words, |down communication |disconnected ideas | |
| | |illegible | | | |
| | |handwriting | | | |
Assurance of test quality
Validity
Validity is the degree to which a test or assignment actually measures what it is intended to measure. There are five important aspects of validity (Hamp-Lyons 1991; Jacobs et al. 1981):
1. Face validity Does the test appear to measure what it purports to measure?
2. Content validity Does the test require writers to perform tasks similar to what they are normally required to do in the classroom? Does it sample these tasks representatively?
3. Concurrent validity Does the test require the same skill or sub-skills that other similar tests require?
4. Construct validity Do the test results provide significant information about a learner’s ability to communicate effectively in English?
5. Predictive validity Does the test predict learners’ performance at some future time?
Hughes (1989): Specific guidelines for moderation of a task: contents
1. Is the task properly representative of tasks that we should expect the test-takers to be able to perform
2. Will the tasks elicit samples of writing which truly represent the students' ability?
3. Will the tasks help markers score reliably?
4. Does the task test writing ability and nothing else? In language testing we are not normally interested in knowing whether students are creative, imaginative, or even intelligent, have wide general knowledge or have good reasons for the opinions they hold. For that reason we should not set tasks which measure these abilities.
What do you think the following tasks test?
Table 13 Validity of the writing tasks
|Task |What does the task test? |
|Write the conversation you have with a | |
|friend about the holiday you plan to have | |
|together | |
| | |
|You spend a year abroad. While you are | |
|there, you are asked to talk to a group of | |
|young people about life in your country. | |
|Write down what you would say. | |
| | |
|The advantages and disadvantages of being | |
|born into a wealthy family. | |
| | |
|'Envy is the sin which most harms the | |
|sinner' Discuss | |
| | |
Reliability of the examination
Reliability is the degree to which the scores assigned to students’ work accurately and consistently indicate their levels of performance or proficiency.
Inter and Intra-marker reliability
Correlation coefficients of .80 and above between readers’ scores (inter-rater reliability) as well as between the scores assigned by the same reader (intra-rater reliability) to the same task are considered acceptable for decision making (Bachman 1990).
There is research that indicates that the gender, background, and training of the reader can affect the reliability of scores (Brown 1991; Cushing-Weigle 1994).
Thus, to maintain reliability many programs put heavy emphasis on the training of raters and as a result have obtained high positive correlations (Jacobs et al. 1981; Hamp-Lyons 1991).
Hughes (1989) suggests the following steps to make the scoring of writing tasks more reliable:
1. set as many tasks as possible : the more scores that scorers provide for each candidate, the more reliable the total score
2. restrict candidates: the greater the restrictions imposed on the candidates (writing tasks should be well defined and candidates should know just what is required of them) the more reliable should be the total score
3. give no choice of tasks: making the candidates perform all tasks also makes comparisons between candidates easier
4. ensure long enough samples: the samples should be long enough for judgements to be made reliably, especially if you are evaluating organisation
5. Once we have written a task and pretested it, we can start developing a marking scale (holistic or analytical)
Table 14 Reliability of the Year 12 examination in 1999
|Task |points |SEM |Alpha |Correlation with the |
| | | | |total*** |
|Reading |37 |2.4 |0.895 |0.885 |
|Matching |11 |1.3 |0.711 |0.640 |
|Gap-filling |20 |1.7 |0.866 |0.858 |
|Multiple-choice |6 |1.0 |0.622 |0.643 |
|Listening |40 |2.3 |0.776 |0.812 |
|Gap-filling |17 |1.2 |0.708 |0.683 |
|Multiple-choice |12 |1.4 |0.555 |0.683 |
|true/false |11 |1.3 |0.421 |0.558 |
|Language Use |51 |2.9 |0.932 |0.700 |
|Multiple choice |19 |1.7 |0.815 |0.642 |
|Editing |16 |1.7 |0.799 |0.608 |
|Gap-filling |16 |1.5 |0.875 |0.671 |
|Writing |60 |1.3 |No data* |0.761 |
|Postcard |15 |0.2 |No data |0.542 |
|Letter |20 |0.5 |No data |0.670 |
|Report |25 |0.7 |No data |0.725 |
|Speaking |25 |0.6 |.572** |0.823 |
|Total |100% |0.2 |Not appropriate |Not appropriate |
* In 1999 CEC data package did not contain the results of the first and second marking
**Correlation between the first and the second marking
***See Intercorrelations between the different tests in Appendix 12
Table 15 Intercorrelations between the different tests in 2001
|Total |Listening |Reading |Lang.use |Speaking |Writing | |Total |1.000 |0.915 |0.884 |0.872 |0.849 |0.869 | |Listening |0.915 |1.000 |0.814 |0.766 |0.688 |0.709 | |Reading |0.884 |0.814 |1.000 |0.742 |0.651 |0.682 | |Language use |0.872 |0.766 |0.742 |1.000 |0.638 |0.685 | |Speaking |0.849 |0.688 |0.651 |0.638 |1.000 |0.763 | |Writing |0.869 |0.709 |0.682 |0.685 |0.763 |1.000 | |
Bibliography
Bachman, L.F. 1990: Fundamental considerations in language testing. Oxford: Oxford University Press.
Common European Framework of language learning, teaching and assessment, Strasbourg 1996
Cushing-Weigle S., Assessing Writing Cambridge University Press, 2002
Hamp-Lyons, L. & W. Condon (1993) ‘Questioning the assumptions about portfolio-based assessment’ College Composition and Communication, 44/2: 176-190.
Hughes, A. Testing for Language Teachers. Cambridge: Cambridge University Press, 1989.
Tribble C. Writing. Oxford University Press,1995
Weir, C. Communicative Language Testing. Prentice Hall, 1995
Online sources
lithuania-education-exams-ielts.htm
Testing Writing Seminar Evaluation Sheet
Please evaluate the usefulness of the seminar from 1 (not useful)- 5 (very useful), please give your comment and suggestions for the further seminars
The topic of the seminar 1…2…3…4…5……………………………………………………
The format of the seminar 1…2…3…4…5……………………………………………………
Session 1: Writing as a concept 1…2…3…4…5…………………………………………………
Session 2: Approaches to testing writing …2…3…4…5…………………………………………
Session 3: Writing tests and tasks 1…2…3…4…5…………………………………………………
Session 4: Marking scale types and marking reliability 1…2…3…4…5…………………………
Session 5: New task development 1…2…3…4…5…………………………………………………
Session 6: Marking scale development 1…2…3…4…5……………………………………………
Session 7: Pretesting test tasks 1…2…3…4…5……………………………………………………
Session 8: Feedback on the tasks developed 1…2…3…4…5………………………………………
Suggestions for the further seminars…………………………………………………………………
Thank you for your help!
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.