Sarah Johnson - Purdue University



Sarah Johnson

English 680W

Shirley Rose

December 17, 2003

Third Wave Writing Assessment for Writing Program Administrators: 1990 – 2003

In her 1999 essay entitled, “Looking Back as We Look Forward: Historicizing Writing Assessment,” Kathleen Blake Yancey identifies three overlapping waves of writing assessment beginning in 1950. She characterizes each wave of writing assessment by analyzing them through five lenses: method employed, emphasis on validity and/or reliability, location of expertise, relationship to the classroom, and relationship to the self. Whereas the first wave of writing assessment used objective tests designed by testing (psychometric) experts and emphasized reliability, the second wave used the holistically scored essay test, which was argued by composition specialists to be more valid. The third wave focuses on portfolio and programmatic assessment designed and rated by composition specialists concerned with both validity and a new conception of reliability. Because third wave assessment is designed by composition specialists who are primarily concerned with validity, it is based in and reflective of classroom practices, and therefore can be used to improve upon those practices. Moreover, whereas first wave writing assessment allowed the tested self who was passive and had no agency, and the second wave permitted the self as producer with limited agency, the third wave allows for multiple and reflective selves with greater agency.

This essay attempts to orient new WPAs and graduate students intending to pursue careers as WPAs to salient issues in third wave writing assessment. Yancey’s characterization of third wave writing assessment highlights many of these issues. Validity has indeed become increasingly valued in discussions of assessment, which is perhaps why portfolio assessments have become the preferred method of writing assessment. Because of this preferred status, compositionists have explored a variety of contexts and purposes for using portfolios. In all of these contexts, issues of power, politics, and ideology have become increasingly significant.

Issues of Validity

Yancey claims that “[w]riting assessment is commonly understood as an exercise in balancing the twin concepts of validity and reliability. Validity means that you are measuring what you intend to measure, reliability that you can measure it consistently” (487). One way she characterizes the history of writing assessment is by its movement from an emphasis on reliability to an emphasis on validity. In first wave assessment, the emphasis was on reliability, and so objective tests were the preferred method. But as composition became a field of study, and compositionists became more involved in assessment, concern for validity began to eclipse concern for reliability and the holistically scored essay exam came into being. Yancey characterizes this as a middle step in which greater validity is sought, but it is argued for within the discourse of psychometrics: it was not until compositionists could argue that they could achieve sufficient interrater reliability that the holistically scored essay was actually implemented. Another important move toward validity in the second wave involved questioning the importance of reliability. Yancey explains how Rexford Brown undermined the concept of correlation by showing that just as objective tests correlate with writing ability, so too do the number of bathrooms and cars in one’s family or one’s parents’ level of education, but we do not use those factors to assess writing ability. Reliability, then, is claimed to not be sufficient without validity. In the third wave, portfolios are seen as still more valid than the holistically scored essay because rather than one text there are multiple texts written on different occasions and in different genres. Further, Yancey argues that in addition to greater reliability, portfolio assessment relies on a new kind of reliability – one that is based on reading, interpretation, and negotiation.

O’Neill, Schendel, and Huot also stress the importance of validity in writing assessment. They identify the preoccupation with testing technologies – especially reliability – as an obstacle to a nuanced understanding of validity, which may include but is not limited to reliability issues. They argue that by viewing assessment as research, WPAs can ask questions that move beyond reliability toward validity, which in turn can reposition assessment from being something done to WPAs to something done by them to answer legitimate questions about how to improve teaching and learning. A valid assessment accurately reflects what happens in the program and may involve rethinking and redefining what is meant by consistency. The authors cite William Smith’s research as an example of redefining consistency as a “community’s consensus of what counts as successful writing” (15).

In their 1997 book chapter entitled, “Assumptions About Assessing WAC Programs: Some Axioms, Some Observations, Some Context,” Yancey and Huot do not mention reliability at all, but do stress the importance of validity in two contexts. First, they argue that because part of validity is “its ability to measure what it purports to measure” (8), program assessment relies on sampling because it is focused on the program, not individual students. Second, they assert that “[t]he validity of a specific assessment depends, in part, on the impact of the decisions made because of the assessment” (11). In this sense validity is not only concerned with measuring the thing it claims to measure, but also with the effects of that measurement.

The issue of systemic validity is addressed in both the CCCC’s position statement on writing assessment and Royer and Gilles’s book chapter addressing placement issues. The CCCC’s position statement addresses systemic validity in general, simply stating that because assessment tends to drive pedagogy, it must demonstrate systemic validity, or in other words, “encourage classroom practices that harmonize with what practice and research have demonstrated to be effective ways of teaching writing and of becoming a writer” (432). Royer and Gilles, however, discuss not only systemic validity, but also content and predictive validity within the context of placement methods. They define content validity as “the relationship between the placement method and the curriculum into which students are being placed”, predictive validity as “the relationship between placement results and student success in the curriculum”, and systemic validity as “the relationship between the placement method, the curriculum, and the rest of the students’ academic lives” (267). In terms of using portfolios for placement assessment, Royer and Gilles assert that there is greater content and systemic validity than in either objective measures or holistically scored essays, but that there is no evidence that portfolios have greater predictive validity. They suggest that portfolios may indicate how successful a writer is, but not necessarily how successful that writer will be in the curriculum into which s/he will be placed.

Despite the greater content and systemic validity of portfolios, they are not always used because they are time intensive and expensive. For example, Anson and Brown describe a situation in their article “Large-scale Portfolio Assessment: Ideological Sensitivity and Institutional Change,” in which they had actually convinced other faculty members and their institution that portfolio assessment would be the best method of assessment, but they were never able to implement it because of a lack of funding. On the other hand, in “Shooting Niagara,” Haswell, Johnson-Shull, and Wyche-Smith actually do implement a portfolio proficiency exam, but have to make significant compromises concerning validity. Although they acknowledge that theoretically, the more samples of student writing the more valid a portfolio is, they only require three samples because the large number of students at their institution mean that in terms of time and cost, the fewer writing samples, the better.

Huot and Williamson discuss validity and reliability within a very different context: the standardization of writing assessment. Citing Pamela Moss, they point out that validity problems are inherent in large-scale, standardized assessment because instead of adequately representing the ability to be tested and how the results of the test will effect teaching and learning, standardized assessment “tends to reduce the curriculum to what can be measured” (46). Moreover, standardized assessment procedures “sacrifice validity for the objectivity of reliability, often resulting in the trivialization of the goals of assessment itself” (47). Huot and Williamson also indicate that new theories are being formed that “conceive of reliability as a ‘critical standard’ or ‘confirmation’” because “rigid and simple conceptions of reliability” simply do not work for making “sophisticated judgments about complex activity like that exhibited in a portfolio of student writing” (50).

Multiple Purposes and Contexts

The combination of an increased concern for validity and the high degree of content and systemic validity in portfolio assessment have contributed to the experimentation with using portfolios for a variety of purposes and in a variety of contexts. Portfolios have been used to emphasize the process nature of writing, to encourage reflection and critical thinking, to allow students to focus on improving their writing, to allow teachers to focus on improving their teaching, to train new teachers, to assess writing programs, to place incoming students, and to measure the proficiency of outgoing students.

Two possible ways of using portfolios are for placement and proficiency. When Royer and Gilles address the use of portfolios for placement purposes, they acknowledge the content and systemic validity as a strength, but point out the problem that portfolios do not necessarily have great predictive validity. Predictive validity is key in placement because ultimately the purpose of the placement is to predict how well a student will do in the class s/he is placed in. When Anson and Brown discuss the portfolio based assessment plan they designed but were never able to implement, they describe a placement portfolio that would contain many genres of writing. They thought that if they required a high school portfolio for placement, it would encourage high schools to build writing across the curriculum programs. What neither Royer and Gilles nor Anson and Brown address are the cumbersome logistical challenges that would accompany such a placement portfolio. Ideally, the writing samples in such a portfolio would represent work submitted during high school and then revised. However, the kinds of writing done in high schools vary greatly. The WPA requiring the placement portfolio would either have to dictate the types of work they would like students to include, knowing that some of those pieces would be first drafts completed specifically for the placement portfolio, or be flexible with the types of writing that could be included, knowing that portfolio readers would then be in the position of comparing widely different kinds of texts (apples and oranges, so to speak).

Using portfolios to assess proficiency, especially during the junior year and especially in writing across the curriculum programs is a much more common practice. One reason for this may be that proficiency assessment is not concerned with predictive validity. Another reason has to do with the development of portfolio assessment. Yancey points out that whereas first and second wave assessment were developed in the context of the placement exam, which is extra-curricular, “in early iterations of programmatic portfolio assessment, the initial reference point is curriculum based, occurring, […] at the end of a course” (493). Because portfolio assessment is based in classroom practices, it is ideal for measuring what is learned in classes, and in the case of WAC programs, over the course of many classes. Smit, Kolonosky, and Seltzer implemented a portfolio system to reflect their program’s process-approach to writing and to enable their proficiency assessment to be consistent across courses. Holt and Baker put a twist on portfolios for proficiency by using portfolios as a follow-up option in their proficiency test, which consists of two, timed essay tests. If students do not pass the essay tests, they may either continue taking the essay tests until they pass or compile a portfolio. Haswell, Johnson-Schull, and Wyche-Smith also modify typical proficiency portfolios; in their case they do so in order to be able to use a portfolio system on a large-scale. Anson and Brown, on the other hand, simultaneously design a portfolio assessment for exit from the junior year that requires writing from students’ chosen disciplines and attempt to implement a writing in the disciplines program with designated “w” courses.

Portfolios are most commonly used to support instructional goals, to improve the teaching and learning of writing within the classroom. Thus, like Smit, Kolonosky, and Seltzer, Irwin Weiser explains that the programmatic use of portfolio assessment within his developmental writing program supports the process-oriented goals of the program. Moreover, he claims that portfolio assessment of student work is especially suited to the needs of basic writers, who are likely to produce poor papers early in the course and are also likely to become discouraged for the rest of the course by a receiving a low grade that confirms their suspicions that they are not capable of writing well. Mills-Courts and Amiran also cite the improvement of teaching and learning as the reason for their use of a portfolio system. In their case, however, they are attempting to improve students metacognitive skills of reflective and critical thinking.

Another way WPAs have used both student and teacher portfolios is to enhance teacher training and development. Wendy Bishop describes her first attempt at requiring student portfolios as a teacher-training strategy in her 1991 article, “Going up the Creek Without the Canoe: Using Portfolios to Train New Teachers of College Writing.” Although this attempt was wrought with difficulties, Bishop claims that new teachers learned many things about evaluation than they otherwise would have, that their use of the portfolio system resulted in meaningful dialogue, and that the teachers enjoyed the camaraderie of the evaluation sessions. Irwin Weiser also articulates the benefits of requiring new teachers to use portfolios to assess their students’ writing. One reason that portfolio assessment is particularly well-suited to new teachers is that it temporarily relieves them of the pressure of grading. This allows them to gain confidence and focus on learning how to teach before they have to learn how to evaluate their students’ work fairly. It also allows them to concentrate on learning how to respond to student writing before having to evaluate it, which can help new teachers avoid the temptation to use their comments to justify a grade. Moreover, Weiser asserts that using portfolio assessment helps new teachers learn not only about process-based pedagogy, but also about the writing process itself. Many new teachers are not as familiar with the nature of the writing process as WPAs assume they are. Using portfolios in their teaching, they learn how to teach and encourage revision in an environment where revision is focused on the improvement of writing rather than the improvement of the grade. They also learn about the decision making processes that accompany revision in a portfolio model, including collection, reflection, selection, and evaluation. Weiser also points out that using a portfolio model may help new teachers negotiate their desire for student centered classrooms with their anxiety about authority by positioning them primarily as coaches and instructors, and only secondarily as evaluators. Finally, just as portfolios help students become reflective writers, Weiser argues that they help teachers become reflective practitioners.

Teacher development can also be promoted through the use of teacher portfolios. In “Building Community Through Reflection: Constructing and Reading Teaching Portfolios as a Method of Program Assessment,” Schendel and Newton begin by acknowledging the potential for teaching portfolios to encourage reflective practice and to create a meaningful discourse about teaching. They quickly complicate this view by pointing out how conflating teacher development and teacher assessment and professionalization in one teaching portfolio compromises the developmental potential of the portfolio. Schendel and Newton then assert that these purposes should be separated. Further, they argue that these portfolios are constructed by many discourses in their teaching lives, especially those of their particular writing program, those of the field of composition, and those shaped by their own experiences. Because the writing program helps shape the teaching portfolios, Schendel and Newton argue that teaching portfolios should also help shape the writing program by becoming part of the writing program’s assessment. The model of teacher-portfolio based writing program assessment that they advocate not only contributes to the development of the writing program, but feeds back into the development of the teachers because teachers read each others’ portfolios.

Carrie Shively Leverenz further complicates the use of teacher portfolios by articulating the ethical dilemmas of requiring teacher portfolios. Leverenz also begins by enumerating the benefits of using teaching portfolios both for professional development and for evaluation. Among these benefits she lists the ability of teachers to evaluate their teaching behaviors, be self-reflective, set goals, and support each other in constructing their portfolios and meeting their goals. Then Leverenz begins listing the ethical challenges of required teaching portfolios, many of which stem from the dual purpose of the portfolio for teacher development and evaluation. Leverenz makes no mention of intentionally separating these functions, as do Schendel and Newton, but she does assert that she is much more interested in formative evaluation than summative. Despite her primary interest in formative evaluation, she claims that in reality, the two can never be fully separated, that “there really is no such thing as an evaluation-free zone” (114). The primary challenge for instructors is that they struggle both to represent their teaching in a positive light and to identify areas for improvement. Related challenges for instructors include how to represent teaching philosophies which conflict with the stated goals of the writing program and how to represent their teaching when they are giving different assignments than they are supposed to be using. WPAs struggle not only to respond to teaching portfolios both critically and supportively, but also to use the knowledge gained from the portfolios to create a coherent writing program that allows for difference.

Program assessment is another context in which portfolios are used. For example, as we have already seen, Schendel and Newton argued for the benefits of using teaching portfolios in program assessment. However, many third wave program assessments do not specifically require the use of portfolios. Yancey and Huot offer five guiding principles (which they call assumptions) for WAC program assessment that work for most other kinds of writing program assessment as well. They say that WAC assessment (1) is focused on whole classes or programs, not individual students; (2) is inquiry driven; (3) is based on an understanding of the nature of writing; (4) relies on diverse methods; and (5) is concerned with both learning and teaching, both faculty and student development. Although Yancey and Huot do not specifically mention the use of student or teacher portfolios, it is clear that these portfolios could serve as one of the diverse methods based on an understanding of writing that is concerned with both learning and teaching. James Slevin is particularly concerned about the role of faculty in writing program assessment. He argues that the value of a writing program lies in how the program selects, retains, supports, and empowers faculty to create courses based on their intellectual interests and those of their students, and that therefore faculty should play a greater role in program assessment.

Susan McLeod echoes Yancey and Huot’s claim that program assessment is inquiry driven in her 1992 article, “Evaluating Writing Programs: Paradigms, Problems, Possibilities.” McLeod focuses on the methods that WPAs use to evaluate their programs, dividing them into quantitative and qualitative methods. While she asserts that qualitative methods are more well suited to writing program assessment, she acknowledges those who are interested in the accountability of education (legislators, administrators, and others) are most interested in quantitative and positivist evaluation. What she advocates is using questions about one’s audience and purposes to motivate the contextual design of any particular program assessment. Using a combination of methods assumes that program assessment can be both formative and summative. Yancey and Huot, however, maintain that program assessment is always formative, though they do concede that different stakeholders will be interested in different aspects of the assessment.

Issues of Power, Politics, and Ideology

One of the defining characteristics of the current discourse about portfolio and program assessment, whatever the specific context, is its concern with issues of power, politics, and ideology. One of the central issues of power in assessment revolves around who controls assessment. As Huot and Williamson point out in “Rethinking Evaluation: Issues of Assessment and Power,” “[t]o control testing is to control education, to control what will be valued and taught within the schools” (44). Huot and Williamson’s article is an extended argument against the standardization and centralization of assessment, especially portfolio assessment, because such standardization takes power and control away from the student, the teacher, and the local school. “Decisions about assessment,” they argue, “ultimately involve decisions about where to locate power in educational and political institutions” (48). They support this claim by pointing out that “the aspects of the curriculum for which we are held accountable will determine what we emphasize in our teaching” (49) and that assessment often functions as a form of surveillance by administrators or “other powerful stakeholders” (49). Huot and Williamson then cite Kentucky and Vermont as states that are mandating portfolio assessment and thereby “confounding the political with the educational” (51). In these examples, Huot and Williamson point out that the learning going on in classrooms is secondary to the scores that are being generated to support state reform movements. Decision making is thus “predicated on political rather than educational rational” (51). Huot and Williamson argue that “to preserve the integrity of portfolios and to harness their ability to truly alter the power relationships in assessment, it is necessary to maintain their localized character and to resist any attempts to centrally evaluate them” (54). They also advocate replacing the language of accountability, which carries with it the meaning of reporting to someone with more authority, with responsibility.

On a smaller scale, Irwin Weiser explores the issue of who should control assessment when he discusses whether or not the instructor should evaluate student portfolios. In his developmental writing program, the classroom teacher is responsible for evaluation of the portfolio because s/he is considered the best judge of the student’s progress in the course relative to the course goals. Weiser does mention, however, that there are times when the teacher is not the evaluator, in placement and proficiency assessments, for instance. Marcia Dickson’s article, “The WPA, The Portfolio System, and Academic Freedom,” is also concerned with issues of who controls assessment, again on a local, departmental level. Her narrative of an angry faculty member blowing up at a new WPA about the un-American breech of academic freedom imposed by a portfolio assessment demonstrates the intense emotions that often surround these issues. In this case, the irate professor does not want to negotiate a grade for his students’ portfolios with other professors.

James Slevin’s article, “Engaging Intellectual Work: The Faculty’s Role in Assessment,” is very concerned with faculty’s control of assessment. In fact, he writes that “the evaluation of students […] is the sole province of the faculty member who teaches them” (304). His argument, that writing program assessment should be primarily concerned with faculty development and that faculty should be centrally involved in conducting the assessment, is framed by his concern that if faculty do not take the responsibility to become involved in evaluation, the increasing corporatization of education and commercialization of knowledge will go unchecked.

Whereas Huot and Williamson are concerned with who should control writing assessment, specifically portfolio assessment, Susan McLeod is concerned with what methods, or combination of methods, WPAs should use when assessing their programs. Her concern about methodology dovetails with Huot and Williamson’s concern about who controls assessment because she concludes that although the evaluation method best suited to writing programs is qualitative, those who are interested in the accountability of education (legislators, administrators, and others) are most interested in quantitative evaluation. Therefore, she recommends that WPAs use methods most appropriate for their audience. McLeod highlights the political nature of this compromise through her analysis of the ontological, epistemological, axiomatic, and causal paradigms underlying both qualitative and quantitative methodologies. Huot and Williamson might disagree with her compromise on principle, but practically speaking McLeod offers a reasonable solution to WPAs designing program assessment.

O’Neill, Schendel, and Huot also position writing assessment as political in nature when they argue that WPAs should approach assessment as research. Writing assessment, they point out, has far-reaching affects: “assessments define good and bad writing; they promote certain pedagogies and discourage others; they have real-world consequences for students and teachers; they function to endorse certain positions or sites with authority; and they define our values – whether accurately or not – to others” (13). Moreover, writing assessment is often tied to public policy. By doing the intellectual work to ensure that writing assessments are valid, the authors argue that WPAs “can promote [their] values and theories while limiting the costs of assessment to ourselves, our students, and our programs” (14). O’Neill, Schendel, and Huot argue that not only do WPAs need to do the intellectual work required by assessment, but that they also need to make their knowledge widely accessible. They claim that “if WPAs and teachers of writing want our work to connect to the culture outside of academia, then we must engage publicly the conversations about literacy and education that circulate through the media, local school board meetings, civic events, and casual discussion” (21). Barbara Gleason’s narrative about how her research findings went unheeded due to the heated political controversy over remedial writing is testimony to the need for those researching assessment to “go public” with their findings – both to contribute their knowledge to the discourse and to begin to change the tenor of it.

Works Cited

Anson, Chris M. and Robert L. Brown, Jr. “Large-Scale Portfolio Assessment: Ideological Sensitivity and Institutional Change.” Portfolios: Process and Product. Eds. Pat Belanoff and Marcia Dickson. Portsmouth: Boynton/Cook, 1991. 248-269.

Bishop, Wendy. “Going up the Creek Without a Canoe: Using Portfolios to Train New Teachers of Writing.” Portfolios: Process and Product. Eds. Pat Belanoff and Marcia Dickson. Portsmouth: Boynton/Cook, 1991. 215-227.

CCCC Committee on Assessment. “Writing Assessment: A Position Statement.” CCC 46 (1995): 430-437.

Dickson, Marcia. “The WPA, The Portfolio System, and Academic Freedom: A Cautionary Tale with an Optimistic Ending.” Portfolios: Process and Product. Eds. Pat Belanoff and Marcia Dickson. Portsmouth: Boynton/Cook, 1991. 270-278.

Gleason, Barbara. “Evaluating Writing Programs in Real Time: The Politics of Remediation.” CCC 51 (2000): 560-588.

Haswell, Richard, Lisa Johnson-Shull, and Susan Wyche-Smith. “Shooting Niagara: Making Portfolio Assessment Serve Instruction at a State University.” WPA. 18.1-2 (1994): 44-53.

Holt, Dennis and Nancy Westrich Baker. “Portfolios as a Follow-up Option in a Proficiency-Testing Program.” Portfolios: Process and Product. Eds. Pat Belanoff and Marcia Dickson. Portsmouth: Boynton/Cook, 1991. 270-278.

Huot, Brian and Michael M. Williamson. “Rethinking Portfolios for Evaluating Writing: Issues of Assessment and Power.” Situating Portfolios: Four Perspectives. Eds. Kathleen Blake Yancey and Irwin Weiser. Logan: Utah State UP, 1997. 43-56.

Leverenz, Carrie Shively. “The Ethics of Required Teaching Portfolios.” Composition, Pedagogy, and the Scholarship of Teaching. Eds. Deborah Minter and Amy M. Goodburn. Portsmouth: Boynton/Cook, 2002. 110-120.

McLeod, Susan H. “Evaluating Writing Programs: Paradigms, Problems, Possibilities.” Journal of Advanced Composition. 12 (1992): 373-382.

Mills-Courts, Karen and Minda Rae Amiran. “Metacognition and the Use of Portfolios.” Portfolios: Process and Product. Eds. Pat Belanoff and Marcia Dickson. Portsmouth: Boynton/Cook, 1991. 101-112.

O’Neill, Peggy, Ellen Schendel, and Brian Huot. “Defining Assessment as Research: Moving from Obligations to Opportunities.” WPA 26.1-2 (2002): 10-26.

Royer, Daniel J. and Roger Gilles. “Placement Issues.” The Writing Program Administrator’s Resource: A Guide to Reflective Institutional Practice. Eds. Stuart C. Brown and Theresa Enos. Mahwah: LEA, 2002. 263-274.

Schendel, Ellen and Camille Newton. “Building Community Through Reflection: Constructing and Reading Teaching Portfolios as a Method of Program Assessment.” Composition, Pedagogy, and the Scholarship of Teaching. Eds. Deborah Minter and Amy M. Goodburn. Portsmouth: Boynton/Cook, 2002. 121-131.

Slevin, James F. “Engaging Intellectual Work: The Faculty’s Role in Assessment.” College English 63 (2001): 288-305.

Smit, David, Patricia Kolonosky, and Kathryn Seltzer. “Implementing a Portfolio System.” Portfolios: Process and Product. Eds. Pat Belanoff and Marcia Dickson. Portsmouth: Boynton/Cook, 1991. 46-56.

Weiser, Irwin. “Portfolio Practice and Assessment for Collegiate Basic Writers.” Portfolios in the Writing Classroom. Ed. Kathleen Blake Yancey. Urbana: NCTE, 1992. 89-101.

Weiser, Irwin. “Revising Our Practices: How Portfolios Help Teachers Learn.” Situating Portfolios: Four Perspectives. Eds. Kathleen Blake Yancey and Irwin Weiser. Logan: Utah State UP, 1997. 293-301.

Yancey, Kathleen Blake. “Looking Back as We Look Forward: Historicizing Writing Assessment.” CCC 50 (1999): 483-503.

Yancey, Kathleen Blake and Brian Huot. “Assumptions about Assessing WAC Programs: Some Axioms, Some Observations, Some Context.” Assessing Writing Across the Curriculum: Diverse Approaches and Practices. Eds. Kathleen Blake Yancey and Brian Huot. Greenwich: Ablex, 1997. 1-6.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download