Effective Evaluation of Alternative Education Programs in ...



A Concept Paper on

Effective Evaluation of Alternative Education Programs in Oregon

D R A F T FOR DISCUSSION ONLY

Changhua Wang, Ed.D.

Northwest Regional Educational Laboratory

wangc@

Chet Edwards, MPA

Education Options Associates

cedwards5@

Presented at 2009 Superintendent’s Summer Institute

Eugene, Oregon

August 3–5

INTRODUCTION

In Oregon law, an alternative education program is defined as a school or separate class group designed to best serve students’ educational needs and interests and assist students in achieving the academic standards of the school district and the state (ORS 336.315). Historically, this broad definition encompassed all types of educational programs deemed to be innovative, school-reform oriented, or different from “traditional schools.” However, Oregon school systems’ alternative education programs have usually been either in-school dropout prevention programs for at-risk students or dropout retrieval programs designed to return to a school setting those students who have already dropped out. These programs can be operated by school districts, community colleges, or by community-based organizations that have formed private alternative programs. Subgroups of students these programs serve are defined in compulsory attendance law as having needs related to school discipline, attendance, or academic standards concerns. The legal status of school districts is that “school boards have control of the district schools and are responsible for educating children residing in the district” (ORS 332.072), which underpins the provision of alternative education programs by school districts.

State law therefore requires school districts to “adopt policies and procedures for approval and annual evaluation of alternative education programs…that receive public funds” (OAR 581-022-1350(2). It is further required that “Each school district operating, participating in or contracting for a public or private alternative education program shall evaluate the program at least annually. The district shall provide the public or private alternative education program with a copy of the written evaluation” (ORS 336.655).

Faced with these legal requirements, school districts in Oregon may choose to do one of four things: 1) use the “Alternative Education Program Toolkit for Program Evaluation” (offered by the Oregon Department of Education); 2) carry out annual “program reviews” using a checklist derived from state laws and designed for legal compliance; 3) call their dropout prevention and retrieval programs something other than “alternative education programs”; or 4) opt not to have alternative education programs at all.

It is clear in Oregon that the quality of alternative education programs rests in the hands of the school districts and eventually those who are running the programs on a daily basis. The current quality control of these programs is mostly in compliance with Oregon statutes and administrative rules relating to alternative education. To assist individual programs in complying with these statutes and rules, the Oregon Department of Education has developed an “Alternative Education Program Toolkit for Program Evaluation.” While this toolkit is useful for individual programs to use on an annual basis to ensure compliance with various statutes and rules, it is inadequate to provide the guidance for a program evaluation. Except for some larger school districts such as Portland Public Schools, few alternative education programs are evaluated independently and systemically. In most cases, alternative educations programs use the toolkit, usually at the last minute, to ensure that their legal compliance justifies their existence. The annual evaluation reports provided by most of these programs contain little evaluative information regarding the quality of these programs. This kind of evaluation, though conducted annually, is neither useful for program improvement nor to give decision-makers a good sense how these programs are serving their students. We discuss in this concept paper the importance of effective evaluation of alternative education programs and propose ideas on how to achieve it.

IMPORTANCE OF ALTERNATIVE

PROGRAM EVALUATION

Most students in alternative education programs are disengaged from traditional schools. For these students, alternative education programs are their last resort in order to graduate from high school. For example, 23 Portland Community-based Alternative Education Programs served close to 3,000 students. Over 40 percent of these students do not live with either parent, 4 percent of them are homeless, and close to 90 percent of them have had serious attendance or academic issues before enrolling in these alternative education programs. Most students in Portland Community-based Alternative Education Programs are at risk of dropping out or have previously dropped out of school. Even though specific statistics regarding characteristics of alternative education students statewide are not available, it is safe to conclude that alternative education programs statewide have the highest percent of at-risk students when compared with those traditional public schools in Oregon. Statewide, only 68 percent of students graduated from high schools within four years (Oregonian, June 30, 2009).

Data reported by school districts to the Oregon Department of Education show that in April 2008, alternative education programs were serving 15,018 students, up from 12,448 students served in 2006–07, a 20.6 percent increase. Facing these many at-risk students, alternative education programs remain in the frontline in preventing students from dropping out or failing to graduate from high school with a diploma. Failure of alternative education programs directly contributes to the increased number of dropouts or students who fail to graduate from high schools. These failing students, instead of contributing to our state’s economic well-being and competitiveness, will be a long-term drain on our state’s economy. A sizable percentage of those dropout students will wind up unemployed or underemployed throughout much of their lives. A recent study by a group of economists concluded that if the number of dropouts in the present cohort of 20-year-olds was cut in half, the nation would reap $45 billion through extra tax revenues and reduced costs of public health, crime and justice, and welfare payments (Levin et al., 2007).

Despite many dedicated educators in our alternative educations programs and many stories of students turning around in these programs, it is not uncommon for us to hear negative comments such as the following regarding the controversial nature of alternative education programs.

Alternative schools are alternatives to education.

Alternative schools are a dumping ground for bad kids.

Kids are “warehoused” at alternative schools.

Kids at alternative schools can get away with murder.

There are no standards at alternative schools. They give credits away there.

Alternative programs cost too much money.

Alternative programs reward bad behavior.

Alternative programs are havens for outcasts, misfits, and losers.

We don’t have statewide data to support or reject these negative comments, but educators in alternative education programs have to work hard every day with their students to tell their side of the story. Instead of associating alternative education programs with educational innovations, the general public tends to view alternative education programs as providing a second rate education that isolates difficult students from the mainstream. Given the importance of alternative education programs serving our most vulnerable student population and the unknown quality of the majority of these programs, a thoughtful and effective evaluation of these programs will be important for practitioners of these programs as well as for policymakers at the district and state levels.

HOW TO EVALUATE ALTERNTIVE EDUCATION

PROGRAMS EFFECTIVELY

The following section of the paper outlines what an effective evaluation of alternative education programs looks like based on our management and evaluation experiences with such programs. Our outline also incorporates key elements of effective alternative education schools/programs backed by research over the past 25 years. In our view, an effective evaluation of alternative education programs needs to examine the following components: 1) the context of the school or program, 2) the program’s mission, 3) the school governance and operation, 4) the school curriculum, 5) community involvement, 6) feedback from students, parents, and staff, and 7) student outcomes.

Context

Alternative education programs vary in terms of students served and approaches used in serving these students. Even though we have a general definition for what alternative education programs (ORS 336.615 to 336.665) are in a legal sense, the term can mean different things to different people. The term has been attached to magnet schools for subjects such as science, mathematics, and performing arts; to programs for pregnant teens and teen mothers; to school within schools; to schools without walls; to schools located on college campuses, inside shopping malls, churches, museums, zoos, and amusement parks; to schools for chronically disruptive, chronically disaffected, homeless youth, and adjudicated youth; to schools for the intellectually gifted, or emotionally disturbed, or for students who require special education; to schools that hold classes during the usual daytime hours, that hold classes in the late afternoon or evening, or that don’t hold classes at all. Adding to the confusion is that most alternative education programs do not attach the “alternative” label to their school names, for fear of the bad reputation associated with the term.

Examining the context an alternative education program is the first step toward a systemic, rather than a fragmented, evaluation of a complete program. Non-systemic evaluation tends to focus on isolated outcomes such as testing scores without looking into the intricacies of relationships within the system. The results of non-systemic evaluation often do not reflect what is really happening and are of little use for program improvement. To enter into a systemic evaluation of an alternative education program, we need to look at the context of the program by addressing the following questions:

• Under what circumstance is this alternative education program established?

• What are the characteristics of the student population the program is intended to serve?

• How do students get recruited for this program and how are teachers assigned to teach in it?

• How does this program relate to the school district and/or the community in which it is located?

• Who are the key players in running this program?

Program Mission

This part of the evaluation examines to what extent the program mission, philosophy, or purpose fits the context of the alternative education program, particularly the characteristics of the student population the program intends to serve. An effective alternative education program needs to have a clear mission, a sense of community and commitment, and shared values. Unlike traditional schools that have to cater to everyone, an alternative education program could have a distinctive mission for serving a targeted student population with special needs that are difficult to meet in a traditional school setting. In examining the program mission, we can ask the following questions:

• Is the program mission relevant to students in the program?

• Does the mission distinguish this program from other programs?

• Is the mission achievable given the context and the community of the program?

• Can staff members, students, parents, and other stakeholders know and verbalize the program mission and related goals?

• Does the program have a list of objectives relating to the school mission and goals? Are the objectives measurable?

School Governance and Operation

An effective alternative education program needs to have some degree of autonomy. It needs to have a separate administrative unit that can make decisions independently and quickly to meet students’ needs. When students become reengaged in an alternative education program, this program is usually characterized by a culture of concern and caring. In this culture, students feel they are treated with respect and given a lot of freedom in their learning while being held accountable for their progress toward their leaning goals. There is a strong sense of family among all participants: students, teachers, counselors, support staff, and administrators. The questions that need to be addressed in this area are:

• To what extent is the program independent in making decisions relating to students needs?

• What does the program do to make students feel they belong to this program?

• To what extent do students and teachers have a real voice in the day-to-day operation of the program?

• To what extent are community members involved in the program?

• Is class size small enough to allow students individualized learning and personal attention?

• To what extent are technologies integrated into learning, allowing students access to more learning resources and to learn at their own pace?

• Are disciplinary rules fair and equitably enforced?

• Is the attendance policy flexible and designed to meet student needs?

School Curriculum

Successful alternative education programs are not that different from traditional schools in trying to achieve many of the academic goals. As indicated in the Toolkit provided by Oregon Department of Education: “The program ensures that students receive adequate instruction in the educational standards adopted by the State Board of Education for the grade level(s) the program serve[s] for students to meet state and local benchmark standards.” What makes alternative education programs different from traditional schools are innovative approaches used to attaining these goals. An effective alternative education curriculum tends to be student centered and relevant to students’ academic and personal concerns, with plenty of contextualized learning opportunities or hands-on experiences. Some evaluation questions relating to an alternative education curriculum include:

• Does the alternative education program have a curriculum or a curriculum framework that is aligned with state and district academic standards?

• What is innovative about the program curriculum that allows it to meet the needs of individual students (e.g., integration of academic and career education, differentiated instruction, contextual learning/hands-on experiences)?

• To what extent can students earn their credits based upon proficiency instead of “seat time”?

• How are students accessed in making progress toward their academic goals?

• How is individual assistance provided for students when they need it?

Access to Social Services

Alternative education programs often serve disruptive and disaffected students. Some of these students could be court ordered to attend the school. While a big part of an alternative education program is to help students modify their behaviors in a positive way, some student behavioral issues, such as those caused by drug use, domestic violence, and homelessness, are beyond what can be handled by alternative education programs. Therefore, providing students with access to various social services on an as-needed basis is crucial to helping them deal with personal issues that such agencies are in a better position to handle. This is also important to run a safe alternative program for all students. Evaluation questions in this area include:

• Is each student assigned a personal counselor?

• Does the program provide on-going instruction on conflict resolution?

• For programs serving teen parents, is effective parenting skills instruction provided?

• How does the program assist students to assess social and health services?

• Is childcare provided for students (as appropriate)?

Feedback from Students, Parents, and Staff

No evaluation of an alternative education program is complete without feedback from its students, parents, and staff members. The feedback is typically collected via survey and/or focus groups. It is important to design survey or focus group questions with key elements of a successful alternative education program (such as individualized instruction, contextual learning, and a supporting, caring learning environment) as a framework. Some of these key elements could also be goals and objectives of the alternative education program. The same set of questions need to be used with students, parents, and staff members so that data collected can be triangulated to identify communality as well as gaps in their perceptions of various aspects of the program. The following are sample questions that can be used to obtain feedback from students, parents, and staff members (these questions can be broken into more specific questions when used in the survey):

• What is unique about the alternative education program in comparison with traditional schools?

• How does individualized instruction take place in the program?

• How do students get connected to this program?

• How does this program make learning relevant and meaningful for students?

• What strategies does this program use to deal with students’ behavioral issues?

• How are parents, community members, and social service agencies involved in the program?

Student Outcomes

Every school should be held accountable for its student outcomes; alternative schools or programs are no exception. However, in our current education environment, with alternative education programs included, student outcomes are often disproportionately focused on basic, narrowly defined academic skills. Success is often measured by how many students meet or exceeded the state benchmarks in reading and mathematics.

By this measure, most alternative programs appear to fail their students despite significant progress these students may have made in these programs. This single measure takes the context out of alternative education programs. As mentioned above, students in most alternative education programs were dropouts or at risk of dropping out. They usually entered these alternative education programs a few grade levels behind and have had to overcome a variety of other personal issues just to come into the school. Many of these students move closer to meeting academic benchmarks each year, but the simple percentage of students meeting or not meeting academic benchmarks will still mark them as school failures without acknowledging their academic progress. Using this indicator alone will not allow us to capture the whole picture of an alternative education program. We propose to use student average scores on the statewide assessment and measure the distance between student average scores and the cutoff scores (benchmarks). Doing so annually and longitudinally will allow us to see the students’ progress (or the lack thereof) in an alternative education program in meeting statewide academic benchmarks.

In measuring student outcomes, we also propose to use multiple indicators that are more aligned with the designs of individual alternative education programs. For example, for some job-related alternative education programs, students’ self-efficacy for employment will be an appropriate indicator to measure the extent to which students are prepared for work. The number of credits earned by students each year will be a legitimate indicator of student productivity when students’ credits are well defined.

SUMMARY

Even though alternative education programs are in the frontline of school dropout prevention and provide a second chance for those who have already dropped out, the status of quality control of most of these programs is currently that of following a checklist to ensure compliance with Oregon statues and administrative rules relating to alternative education. Results of such a checklist provide little information on the program quality. This paper therefore calls for a systemic evaluation of these programs by looking at their various components: context, mission, governance and operation, curriculum, access to social services, feedback from students, parents, and staff, and student outcomes.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download