ASES Policy Brief - Resources (CA Dept of Education)



ASES and 21st CCLC Policy Brief:

What Have We Learned from the California Statewide Evaluation

CDE4/CN077738/2012/Policy Brief - February 2012

Denise Huang & Deborah La Torre Matrundola

CRESST/University of California, Los Angeles

National Center for Research on Evaluation,

Standards, and Student Testing (CRESST)

Center for the Study of Evaluation (CSE)

Graduate School of Education & Information Studies

University of California, Los Angeles

300 Charles E. Young Drive North

GSE&IS Bldg., Box 951522

Los Angeles, CA 90095-1522

(310) 206-1532

Copyright © 2012 The Regents of the University of California.

The work reported herein was supported by grant number CN077738 from California Department of Education with funding to the National Center for Research on Evaluation, Standards, and Student Testing (CRESST).

The findings and opinions expressed in this report are those of the authors and do not necessarily reflect the positions or policies of California Department of Education.

ASES and 21st CCLC Policy Brief:

What Have We Learned from the California Statewide Evaluation

Denise Huang & Deborah La Torre Matrundola

Research on after school programs tends to show mixed effects concerning the impact of after school participation on students. Some studies have found positive outcomes such as greater self-confidence, increased civic engagement, better school attendance, improved academic achievement, and decreased delinquency (Fredericks & Eccles, 2006; Goerge, Cusick, Wasserman, & Gladden, 2007). Other studies have reported no differences, mixed, or even negative findings for students in afterschool programs related to academic performance, school retention, feelings of safety, and behavior (Cooper, Charlton, Valentine & Muhlenbruck, 2000; Dynarski, et al., 2003; Vanderhaar & Muñoz, 2006).

Similarly, the recent statewide evaluation of ASES and 21st CCLC programs in California showed neutral outcomes when examined as a whole (Huang & Wang, 2012). For example, when comparing after school participants to non-participants, some minor negative effects were found for English-language arts assessment scores, but positive effects were seen concerning physical fitness and school attendance. Similarly, satisfaction levels among the different stakeholders was overwhelmingly high. In particular, after school staff and parents were pleased with the quality of the relationships students developed with staff and peers and felt that students’ academic and emotional needs were being met.

|Snapshot: |

|ASES and 21st CCLC programs in California |

|93% of programs had an academic improvement |

|goal set. |

|78% of programs had increased program |

|attendance goals set. |

|56% of programs set skill development goals.|

|98% of site coordinators reported that they |

|offered homework assistance. |

|56% of site coordinators reported that they |

|offered youth development activities. |

|69% of elementary school programs and 83% |

|of middle school programs had student |

|participation barriers. |

|37% of parents reported that they picked |

|their children up early an average of 2 days|

|per week. |

|Why support after school programs for ASES |

|and 21st CCLC students? |

|1.1 million elementary school children and |

|3.7 million middle school youth do not have|

|adult supervision during the after school |

|hours. |

|The hours between 3pm and 6pm are the peak |

|hours for juvenile crime and |

|experimentation with drugs and alcohol |

|Students attitudes about school tend to |

|become less positive as they transition to |

|middle school. |

|Sixth graders who fail in their core |

|content areas, attend school infrequently, |

|or have poor behavior in school have less |

|chance of graduating on time. |

|Involvement in extracurricular activities |

|is an important factor in the promotion of |

|positive youth development. |

What Circumstances Lead to Inconsistent Findings?

The California Statewide Evaluation (Huang & Wang, 2012), across all outcome measures analyzed, revealed substantial variation across schools in the effect of after school programs. The variations indicates that some after school programs likely have a strong positive effect while other after school programs are less successful. Interaction analyses further revealed that neighborhood contextual variables and program qualities influence program outcomes.

Study findings indicated that contextual variables such as neighborhood facilities; general family, health, and educational services provided in the neighborhoods; crime rates; availability of extended family members; etc. may be affecting the dynamics and value of after school programs in specific neighborhoods. When the community resources are high, this opens up more enrichment opportunities for the non-participants even if they don’t attend the ASES and 21st CCLC programs; but in communities where resources are scarce, these programs may serve even more important roles in broadening the accessibility for those in need.

Furthermore, when quality features were examined at a subsample of 40 elementary and middle schools, the California Statewide Evaluation found that students participants in higher quality programs perceived higher academic competence. Specifically, the students in the programs that were rated as higher quality by the evaluation team reported greater academic benefits than did the students in programs of lower quality.

These students relative to those in lower quality programs, were more positive about program impact on their grades, motivation for school, and comfort in taking tests. Survey results from the middle school students further suggested that their ASES and/or 21st CCLC programs provided them with the skills and knowledge to be successful leaders, to get to college or vocational school, to resist drugs and alcohol, and that their programs helped them to believe that they could go to college and get a good job after school.

In their analysis of findings, the California Statewide Evaluation produced the following conclusions:

Important Contextual Variables Cannot be Examined Because they are Not Available

One of the stipulations of the California Statewide Evaluation was that all participants in all after school programs would be examined compared to non-participants, regardless of the quality at the programs attended, differences in neighborhood context, and potential differences between participants and non-participants. While available data and advanced statistical methodologies, such as propensity matching and HLM analysis, were used to try to account this variation, unmeasured differences between after school participants and non-participants likely remain that are influencing study results. A range of contextual and environmental data, for example, may influence who chooses to enroll in afterschool program, and important variables thought to contribute to self-selection (e.g., inherent student motivation, family structure, parent expectation, and services and supports in the immediate community) were not available for propensity matching.

Not all ASES and 21st CCLC Programs are Equal

As previously mentioned, evaluation results indicated there was some or great variation in the effects across program sites. Qualitative data from the study further revealed that not all programs were functioning at optimal levels. Because program quality indicators were not available beyond the subset of 40 sites, the study could not adequately investigate the relationship between program quality and student outcomes and/or the effect of programs of at least modest quality. Thus, it is important to acknowledge the possibility that the efforts and successes of the higher quality programs in this study were likely undermined by the programs that were functioning at sub-par levels and did not produce positive outcomes.

Program Targeting Practices

The ASES and 21st CCLC programs are encouraged to target academically and emotionally at risk youths. While over half of the nearly 2500 site coordinators across the state who completed a profiling questionnaire reported that they enrolled students using a first-come, first-serve basis, student needs were often taken into account. For example, over one-quarter of these site coordinators reported that they enrolled students who had academic needs prior to enrolling other students. Furthermore, over three-quarters of these sites actively targeted these academically at-risk students during their yearly recruitment process. Nearly half of these site coordinators also reported that they actively tried to recruit English learners and/or students with emotional/behavioral issues. When interpreting the negative findings such as suspension and achievement, programs that targeted emotionally distressed students or academically at-risk students would likely have more students that would be suspended due to behavioral issues or academic issues. Thus program targeting practice should be considered when examining student outcomes in achievement or suspension so that programs that have specific missions would not be penalized.

Study findings also produced the following recommendations:

How Can ASES and 21st CCLC Programs Improve Student Outcomes?

First, a few implementation strategies are suggested

Establish a common quality indicator. After school programs may provide important opportunities to students. However, as was found in the statewide evaluation as well as other recent studies, the positive impacts of participation in after school programs occur when the students participate regularly, are exposed to a variety of activities, and are cognitively engaged in the programming (Durlak et al., 2010). Thus, it is important for the California Department of Education (CDE) to collect quality indicators from their grantees so that program quality can be taken into consideration when measuring program outcomes.

Making sure students received the dosage intended. One program barrier that staff at the 40 ASES and 21st CCLC programs brought up very frequently was that students were constantly being picked up at various time intervals. Even with effective programming and quality staff, for after school programs to function as intended, students must receive the entire “treatment.” The fact that students were picked up at various different times not only reduces the dosage for the students that left early, but also disrupts the flow of the programming and concentration for the rest of the participants. This common and on-going practice may have negative impacts on student outcomes. Specific regulations or policies concerning early dismissals should be enforced to address this issue.

Importance of the linkage to the day school. Under Title 22, the Federal emphasis is placed in the context of standards-based accountability, and after school programs have been increasingly seen as a way to promote better student academic outcomes. To fulfill this expectation, partnership with the day schools is critical. Within the statewide evaluation, stronger partnerships appeared to have positive impacts on program implementation, such as obtaining more physical resources (such as space and equipment) and providing students with a “seamless” transition between the school day and after school by having teachers share student records on homework completion, achievement, and behavior with staff. Alignment between the day school and after school curriculums further enhances students’ academic and behavioral development, especially when teacher and staff work together in assisting students in their weakest performance and behavioral areas. Thus, schools and principals should be actively recruited into partnerships with the after school programs.

Secondly, the other important observation from the statewide evaluation is the benefit of separating the implementation structure, delivery strategies, and staff qualifications between the elementary and middle school programs.

Catering to Ages and Stages

Early adolescence is a very developmentally specific and critical period for youth development. In this study, some frequently mentioned obstacles for middle school ASES and 21st CCLC programs were student disinterest and staff perceptions of the troubling attitudes and behaviors of their students.

The literature has shown that students are especially vulnerable to initiating or escalating their behavioral problems during transitional periods such as from elementary to middle schools and from middle to high schools (Ayers, et al., 1999). Transitions break the social bonds students have formed with their teachers and peers, and these students must then negotiate new social relations and adapt to the practices and routines of the new school. These uncertainties and challenges may result in behavioral problems, weaker attendances, and poor grades. Thus, more time needs to be spent preparing youth during these crucial years of transition (Weiss & Bearman, 2007). Unfortunately, most ASES and 21st CCLC agencies still place the main focus on serving elementary age children and employ program delivery strategies that are designed to fit this age group for all programs.

Thus, it is not surprising that after school attendance tends to decline as students age. According to a recent fact sheet, only 19% of the 21st CCLC programs funded nationally exclusively target middle school students (Afterschool Alliance, 2008). This was also evidenced in the California Statewide Evaluation (Huang & Wang, 2012) with over half of elementary school participants attending 108 or more days during the final year of the study and about 40% of middle school participants attending only 72 or more days. Similarly, in a study by Grossman and colleagues (2002) it was found that only about half of middle school students attended all possible days, while approximately two-thirds of the elementary school students in the study did the same.

The literature on after school programs suggest that a variety of reasons may exist for this lack of interest in participation. First, middle school is a difficult time for children. As youth go through great developmental changes, they have a growing sense of autonomy and changing interests (Herrera & Arbreton, 2003). They also have more options and responsibilities than young children do during the after school hours (Deschenes et al., 2010). For example, research has found that many adolescents have access to multiple after school activities (Cross et al., 2009; Deschenes et al., 2010; Huang & Wang, 2012), and they may be asked to take on family responsibilities such as watching younger siblings or contributing financially through part-time jobs (Lauver, Little, & Weiss, 2004).

Thus different recruitment and retention strategies need to be catered specifically for the middle school students.

Recruiting and Retaining Middle School Students

Stakeholders who participated in the surveys, interviews, and focus groups for the California Statewide Evaluation provided valuable insight into the reasons why middle school students participated in the ASES and 21st CCLC programs. While it was not surprising to find that parents most likely enrolled their their children to get help with homework, key academic subjects, and in school in general; research on adolescents indicates that emphasizing principles of youth development such as leadership, student autonomy, and social competence is an equal or more important factor in appealing to middle school students (Larson, 2000). In this study qualitative findings also revealed that programs that emphasize engaging students in intellectual, productive, and constructive ways are more welcomed by students.

Creating Programs that Appeal to Middle School Students

While all ASES and 21st CCLC programs need to provide a physically and psychologically safe environment for all students, the emphasis for each age group differs (Arbreton, Bradshaw, Metz, Sheldon, with Pepper, 2008). In reviewing the literature on retention strategies for middle school students, the following themes emerged:

Provide nurturing and safe environments

Staff should respect student opinions, recognize student contributions, and treat students as responsible individuals (Kauh, 2010).

Students should be allowed to interact with their peers and friends they don’t get to see during the school day (Arbreton, Bradshaw, Metz, Sheldon, with Pepper, 2008; de Kanter, 2001).

Providing developmentally appropriate activities

As the curriculum in middle school begins to specialize under disciplines, programs should hire staff who have content specializations (Deschenes et al., 2010)

Most students want to learn, not just hang out at their program (de Kanter, 2001). Therefore, students should be allowed to explore their emerging interests by providing a selection of academic and non-academic activities (Arbreton, Bradshaw, Metz, Sheldon, with Pepper, 2008; Arbreton with Bradshaw, Sheldon, & Pepper, 2009; Deschenes et al., 2010; Kauh, 2010).

Activities should be age-appropriate and challenging to the adolescents (de Kanter, 2001; Gootman, 2000).

Focus on teaching methods that emphasize engagement

Students should be given opportunities to practice new skills through hands-on experiences, cooperative learning, and structured reflection (Gootman, 2000).

Students should be given project or field-based opportunities to learn ( Deschenes et al., 2010).

Activities should be student-centered with staff facilitating and guiding lessons rather than dominating (Gootman, 2000).

Provide students with leadership opportunities

Offering leadership opportunities is significantly related to retention and attendance levels of adolescents (Arbreton, Bradshaw, Metz, Sheldon, with Pepper, 2008; Deschenes et al., 2010).

Students should be provided with opportunities to be involved in program decision-making, design, and the leading of activities (Kauh, 2010).

Develop attendance policies that work with rather than against youth

In contrary to the elementary ASES and 21st CCLC programs, where students need to attend the entire duration of the program to reap the benefits, middle school program policies should emphasize engagement and quality of instruction, not just specific hours attended. As stakeholders in a study by Deschenes and colleagues (2010) pointed out, “Quality often matters more than quantity” (p. 18).

As middle school students take on personal, school, and family responsibilities, programs should consider having flexible attendance policies, such as providing activities on a “drop-in” basis, in the evenings, or on weekends (Roth, Malone, & Brooks-Gunn, 2010).

Students could be provided with meaningful rewards – such as point-based prizes, access to special events or activities, and priority enrollment for the next year – for consistent and/or high participation so as to motivate students to attend by choice rather than obligation (Kauh, 2010).

The following policy recommendation can facilitate the success of ASES and 21st CCLC programs in California.

Policy Recommendations

Set up an on-line database system to collect neighborhood and other environment contextual data for all grantees.

Establish a common quality indicator system across all ASES programs. It would be especially efficient if elementary and middle school quality indicators are specified to their unique age groups.

Set up policy and guideline to encourage and expand collaborations between the day schools and the after school programs.

Create separate legislatures and regulations that are designed specifically for middle school and elementary students.

Emphasize the knowledge of childhood development in hiring elementary staff and emphasize the knowledge of adolescent development in hiring after school staff for middle school students.

Provide professional development that are specific to working with the intended age group.

Consider setting separate attendance and programmatic policies that specifically target the unique assets and barriers of elementary and middle school students.

References

Afterschool Alliance. (2008). 21st Century Community Learning Centers Providing Afterschool Supports to Communities Nationwide. Retrieved from

documents/factsResearch/21stCCLC_Factsheet.pdf

Afterschool Alliance. (2009, October 6). 18 million children need—but don’t have—afterschool programs, according to new “American after 3pm” study [News release]. Retrieved from

Afterschool Alliance (2011, January). Afterschool innovations in brief: Focusing on middle school age youth. Retrieved from

Compendium.pdf

Arbreton, A. Bradshaw, M., Sheldon, J., & Pepper, S. (2009). Making every day count: Boys & Girls Clubs’ role in promoting positive outcomes for teens. Philadelphia: Public/Private Ventures.

Arbreton, A., Bradshaw, M., Metz, R., Sheldon, J., & Pepper, S. (2008). More time for teens: Understanding teen participation—frequency, intensity and duration—in Boys & Girls Clubs. Philadelphia: Public/Private Ventures.

Ayers, C. D., Williams, J. H., Hawkins, D. J., Peterson, P., Catalano, R., & Abbott, R. D. (1999). Assessing correlates of onset, escalation, de-escalation, and desistance of delinquent behavior. Journal of Quantitative Criminology, 15(3), 277-306.

Cooper, H., Charlton, K., Valentine, J. C., & Muhlenbruck, L. (2000). Making the most of summer school: A meta-analytic and narrative review [Serial No. 260]. Monographs of the Society for Research in Child Development, 65(1).

Cross, A. B., Gottfredson, D. C., Wilson, D. M., Rorie, M., & Connell, N. (2009). The impact of after-school programs on the routine activities of middle-school students: Results from a randomized, controlled trial. Crimonology & Public Policy, 8(2), 391-412.

De Kanter, A. (2001). After-school programs for adolescents. NASSP Bulletin, 85, 12–21.

Deschenes, S. N., Arbreton, A., Little, P. M., Herrera, C., Grossman, J. B., Weiss, H. B. (with Lee, D.). (2010). Engaging older youth: Program and city-level strategies to support sustained participation in out-of-school time. Cambridge, MA: Harvard Family Research Project, Philadelphia: Public/Private Ventures.

Durlak, J. A., Weissberg, R. P., & Pachan, M. (2010). A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents. American Journal of Community Psychology, 45, 294-309.

Dynarski, M., Moore, M., Mullens, J., Gleason, P., James, Burdumy, S., Rosenberg, L., Pistorino, C., Silva, T., Deke, J., Mansfield, W., Heaviside, S., & Levy, D. (2003). When schools stay open late: The national evaluation of the 21st Century Community Learning Centers Program, first-year findings. Princeton, NJ: Mathematica Policy Research.

Fredericks, J. A., & Eccles, J. S. (2006). Is extracurricular participation associated with beneficial outcomes? Concurrent and longitudinal relations. Developmental Psychology, 42(4), 698-713.

Goerge, R. M., Cusick, G. R., Wasserman, M., & Gladden, R. M. (2007). After-school programs and academic impact: A study of Chicago’s After School Matters. Chicago: Chapin Hall Center for Children.

Gootman, J. A. (Ed.). (2000). After-school programs to promote child and adolescent development: Summary of a workshop. Washington, DC: National Academy.

Grossman, J. B., Price, M. L., Fellerath, V., Jucovy, L. Z., Kotloff, L. J., Raley, R., Walker, K. E. (2002). Multiple choices after school: Findings from the extended-services schools initiative. Philadelphia: Public Private Ventures.

Herrera, C., & Arbreton, A. J. A. (2003). Increasing opportunities for older youth in after-school programs: A report on the experiences of Boys & Girls Clubs in Boston and New York City. Philadelphia: Public/Private Ventures.

Huang, D., & Wang, J. (2012). Independent Statewide Evaluation of High School After School Programs: May 1, 2008-December 31, 2011. Los Angeles: University of California, National Center for Research on Evaluations, Standards, and Student Testing (CRESST).

Kauh, T. J. (2010). Recruiting and retaining older African American and Hispanic boys in after-school programs: What we know and what we still need to learn. Philadelphia: Public/Private Ventures.

Larson, R. W. (2000, January). Toward a psychology of positive youth development. American Psychologist, 55(1), 170-183.

Lauver, S., Little, P. M. D., & Weiss, H. (2004). Moving beyond the barriers: Attracting and sustaining youth participation in out-of-school time programs. Issues and opportunities in out-of-school time evaluation #6. Cambridge, MA: Harvard Family Research Project

Roth, J. L., Malone, L. M., & Brooks-Gunn, J. (2010). Does the amount of participation in afterschool programs relate to developmental outcomes? A review of the literature. American Journal of Community Psychology, 45(3-4), 310-324.

U.S. Department of Education Mentoring Resource Center. (2008, September). Making the transition to middle school: How mentoring can help. Mentoring Resource Center Fact Sheet. Folsom, CA: Author. Retrieved from

Vanderhaar, J., & Muñoz, M. A. (2006). Educating at-risk African American males: Formative and summative evaluation of the Street Academy Program. (ERIC Document Reproduction Service No. ED 495 958).

Weiss, C. C., & Bearman, P. S. (2007). Fresh starts: Reinvestigating the effects of the transition to high school on student outcomes. American Journal of Education, 113(3), 395-422.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download