The Oklahoma Center for Educational Policy



The Oklahoma Center for Educational Policy

The Community School Effect

Evidence from an Evaluation of the Tulsa Area Community School Initiative

November 2010

Curt M. Adams

University of Oklahoma

The Oklahoma Center for Educational Policy

This report was prepared for the Tulsa Area Community Schools Initiative (TACSI). Do not cite without permission from the TACSI Director. Questions about the research design and findings may be directed to Curt Adams at Curt.Adams-1@ou.edu. The research was supported by the Oklahoma Center for Educational Policy. The investigator does not have a financial interest in TACSI or community schools that could bias the findings of the report.

ACKNOWLEDGEMENTS

This report would not have been possible without the support of colleagues at the Oklahoma Center for Educational Policy. Their assistance in conceptualizing the design and providing critical feedback was invaluable. In particular, the qualitative data collected and analyzed by Dr. Gaetane Jean-Marie was instrumental for understanding how principals in the mentoring and sustaining schools diffused core components of the community school model. Additionally, her ideas, reflections, and critical questions challenged the research to extend beyond merely providing evidence on achievement effects. Dr. Lisa Bass and doctoral candidate Kathy Curry also made helpful contributions to the evaluation design, data collection, and analysis. Finally, I want to thank the management team of TACSI and the principals of TACSI schools for their openness and willingness to share their story.

ABOUT THE AUTHOR

Curt M. Adams is an assistant professor in the Department of Educational Leadership and Policy Studies at the University of Oklahoma. He is also a research scientist with the Oklahoma Center for Educational Policy. His research addresses school improvement through the lens of social conditions in school organizations. Recent publications include: Collective Trust: Why Schools Can’t Improve Without It (with Patrick Forsyth and Wayne Hoy, Teachers College Press). The Formation of Parent-School Trust: A Multi-level Analysis (Educational Administrative Quarterly), The Nature and Function of Trust in Schools (Journal of School Leadership), and Social Determinants of Student Trust in High Poverty Elementary Schools (a chapter in Analyzing School Contexts: Influences of Principals and Teachers in Service of Students).

ABOUT THE CENTER

The Oklahoma Center for Education Policy (OCEP) was established as a resource for the State of Oklahoma and its education policy makers at the State, district, community and school levels. OU scholars from both the Norman and Tulsa campuses and from various disciplines are tapped as needed to address research needs as they are identified. Some projects are identified by the core scholars of OCEP and result in policy white papers, evaluation reports, or scholarly publications. Other research projects are contracted with civil and governmental agencies.

.

EXECUTIVE SUMMARY

This study used data collected from 18 community schools associated with the Tulsa Area Community Schools Initiative (TACSI) and 18 comparable non community schools to test the achievement effect attributed to the community school model. The primary research questions were: Is there an achievement difference between students in TACSI schools and students in comparable non TACSI schools? Does diffusion of the community school model make a difference in student achievement? If an achievement effect exists, what social conditions contribute to differences in student achievement?

Is there an achievement difference between students in TACSI schools and students in comparable non TACSI schools?

Significant math and reading achievement differences were not found when comparing students in TACSI schools and comparable non TACSI schools. Graph one shows that the average TACSI student performed slightly below the average non TACSI student in math. These differences were eliminated when individual poverty was entered in the model, suggesting that differences between the two groups were largely a function of differences in individual student poverty rather than systematic differences between schools.

Does diffusion of the community school model in TACSI schools make a difference in student achievement?

Significant achievement differences were found when accounting for the development of the community school model. The evidence suggests that bringing the community school model to scale in TACSI schools has the potential to enhance student achievement and to narrow the achievement gap attributed to poverty. Controlling for level of diffusion provided a different achievement picture than a simple comparison between TACSI and non TACSI schools. Students in TACSI schools that had reached the mentoring and sustaining levels of diffusion significantly outperformed other students in the original sample. Nearly all of the mentoring and sustaining students were high poverty students. When isolating the poverty effect, results indicate that students in mentoring and sustaining schools significantly outperformed comparable students in other schools. In short, the efficacy of the community school model as operationalized in TACSI is greatest when core components have been fully diffused to the school level.

Post Hoc Analyses: Closing the Poverty Gap

Two additional samples were drawn in order to test the durability of the achievement effect when comparing student performance in schools with a more affluent student composition. Comparison schools in the first sample had an average school poverty rate of 48 percent whereas the average poverty rate for the second sample was 20 percent. Results from the first post hoc showed there were no significant differences in the math achievement between groups of students. There was, however, a significant difference in the poverty gap, with students in mentoring and sustaining schools significantly outperforming poverty students from schools with a more affluent composition. A similar relationship was found with the second sample. Students in mentoring and sustaining TACSI schools significantly outperformed free/reduced lunch students in the comparison schools where the average poverty rate was 20 percent.

If an achievement effect exists, what social conditions contribute to differences in student achievement?

Reforms, much like policies or planned change, do not directly influence achievement. The effect is more indirect, operating through social conditions in schools to shape student and school performance. Conditions for learning targeted by TACSI are the mechanisms to promote effective teaching and to satisfy the learning needs of students. Two school level conditions in particular were found to significantly predict student achievement: student trust in teachers and faculty trust in students and parents. Student performance is likely to improve when collective trust defines the behaviors and social interactions of teachers, parents, and students. In short, collective trust mediated the relationship between student poverty and achievement.

I. INTRODUCTION AND PURPOSE

The theoretical roots of community schools extend back to Dewey and his idea that schools should function as social hubs of communities (Benson, Harkavy, & Puckett, 2007). Of more recent vintage, is the organic emergence of community schools as a type of whole school reform. The growing popularity of community schools is evidenced by Secretary of Education Arne Duncan’s advocacy of the model. In a speech to the US Chamber of Commerce, Duncan stated, “I'm a big believer in community schools—keeping school buildings open for 12 hours a day and opening up the computer lab, the library and the gym on weekends for our children and their families” (Duncan, 2009, p. 5).

There are reasons why policy makers like Arne Duncan are proponents of community schools. Schools like the Harlem Children’s Zone and the community schools in New York City operated by the Children’s Aid Society have attracted considerable acclaim by the national media for their educational achievements. By myriad indicators (e.g. test scores, satisfaction with school, and student motivation), these schools are effectively educating children and transforming communities. But, what general understanding about the effectiveness of community schools as a whole school reform can be drawn from their success? As compelling of a case successful community schools make for the effectiveness of the model, it is imprudent to draw general conclusions about the efficacy of a reform without warrants derived from scientific research.

Many internal evaluations of community schools exist (Coalition for Community Schools, 2009), but most of the evidence consists of descriptive data on achievement outcomes. Few studies use research designs that meet scientific research standards. Without credible evidence, it is hard to know if the community school model is a viable reform or just another intervention that falls short of its performance claims. While not without limitations, this study used data collected from 18 community schools associated with the Tulsa Area Community Schools Initiative (TACSI) and 18 comparable non community schools to test the achievement effect attributed to the community school model. Previous research on TACSI schools examined the association between the community school model and conditions for learning (Adams, 2009). The purpose of this study was to test the assumed relationship between development of the community school model and student academic performance. The primary research questions were: Is there an achievement difference between students in TACSI schools and students in comparable non TACSI? Does diffusion of the community school model in TACSI schools make a difference in student achievement? If an achievement effect exists, what social conditions contribute to differences in student achievement?

II. COMMUNITY SCHOOLS AS WHOLE SCHOOL REFORM

Unlike many whole school reforms that are based on standardized designs and practices (Rowan, Correnti, Miller, & Camburn, 2009), program components used to carry out the vision of community schools reflect the unique characteristics of different schools and networks of schools. To illustrate, some community schools embed health clinics and social services in the schools, others address the out of school time needs of students and families, and some like TACSI are redesigning traditional structures to align with each core component of the community school framework (Blank & Harkavy, 2002). In short, community schools are grown and nurtured locally. Strong external control would be antithetical to the core propositions of local control, partnerships, community empowerment, and social democracy upon which community schools are built (Blank, Melaville, & Shah, 2003). Even though community school designs reflect each school’s or network’s unique context, a common theoretical framework at the national level provides a cohesive vision for community schools. This framework is reflected in the theory of action for TACSI.

Figure one displays the core components of TACSI’s community school model. Cross-boundary leadership; holistic programs, services, and opportunities; family and community engagement; and community based learning work collectively to shape six conditions for learning (i.e. early childhood development, a core instructional program, motivated and engaged students, holistic needs, family-school partnership, and safe school environment) that combine to form the necessary human and social capacity for effective performance. Conditions for learning mediate outcomes associated with school and student success; healthy and socially competent children; adult preparation; and safe and supportive, families, schools, and neighborhoods.

Figure 1: Theory of Action, Tulsa Area Community Schools Initiative

Core Components Conditions for Learning Outcomes

Community schools are different than many comprehensive school reforms in that the reform is not based solely on adopting and implementing new structural components and practices. Rather, schools become community schools when the normative environment establishes a dense and cohesive relational network, a shared culture, and is responsive to the needs of all school and community members. Whereas some school reforms are defined by specific practices, community schools are defined largely by the functional capacity of the core components to cultivate and sustain supportive conditions for learning. Harkavy and Blank (2002) note, “A community school is not just another program being imposed on a school. It is a way of thinking and acting that recognizes the historic central role of schools in our communities—and the power of working together for a common good” (p.50).

Fidelity takes on a different meaning with community schools. Many conceptions of fidelity define it as the alignment between intended components of an intervention and actual practice (O’Donnell, 2008). But, as Copland (2003) found, compliance with a reform model does not always lead to changed cultures or improved outcomes. Reforms are seldom implemented with complete fidelity. In fact, evidence suggests that adaptation (Borman & McLaughlin, 1977; McLaughlin, 1990) and local sense making (Spillane & Thompson, 2007; Spillane, Reiser & Reimer, 2002) are necessary for reforms to transform practices and cultures. It is not fidelity to prescribed practices that transform a school into a community school. Instead, fidelity to the vision of creating responsive school communities where healthy conditions for learning support the social and psychological needs of all school members is the linchpin of the community school model.

Core Components

The core components of TACSI schools are based on structures, processes, and practices advanced by the National Coalition for Community Schools (Blank, Melaville, & Shah, 2003). Research supporting the effectiveness of the core components has a long history. Effective schools research by Edmonds and other scholars (see Block, 1983; Downer, 1991; Edmonds, 1979; Zigarelli, 1996) identified strong leadership, an instructional focus, high expectations, and quality instruction as characteristics of high performing schools. Today, variations of these elements are found in concepts like collective leadership (Leithwood, 2000) parent-school partnerships (Epstein, 2001), and quality teaching (Goe, 2007). From a general perspective, there is not much difference between characteristics of community schools and the above concepts associated with quality school performance. What is different are the nuanced distinctions in the structures and norms used by community schools to bring the characteristics of effective schools into existence. These structures and norms are illustrated in the figures that precede the discussion of each core component.

Figure 2: Cross-boundary Leadership

Cross-boundary leadership is based on the idea that educational and social problems are more effectively addressed when school and community members share responsibility for the performance and wellbeing of children. Cross-boundary leadership brings together community leaders, leaders on the ground, and leaders in the middle to work collaboratively within the educational process. These leaders represent the civic and business community, the local neighborhood, and different school role groups (i.e. teachers, support staff, parents, students, administrators) (Blank, Berg, & Melaville, 2006). An active and diverse community site team and a full-time community school coordinator are structural features of cross-boundary leadership; whereas, a culture of shared influence and responsibility is a normative condition that facilitates effective interactions across role boundaries.

Unlike conventional site-based management models, cross-boundary leadership does not obfuscate the authority of principals by transferring power to a representative council (Malen & Cochran, 2008). A transfer of principal power to a governing board would seem to contradict the extensive evidence supporting the necessary role of principals in leading school improvement (Bryk, et al., 2010; McLaughlin, 1987). Rather than take away leadership authority of principals, cross-boundary leadership is a mechanism to empower all school role groups to share responsibility for student and school performance and to work collectivity toward common outcomes. Cross-boundary leadership supports the commitment and motivation of school agents to improve performance by providing a social architecture that makes open communication around instructional performance possible.

Figure 3: Holistic Programs, Services, and Opportunities

Holistic programs, services, and opportunities emphasize the development of the whole child, not just his/her cognitive learning. Services, programs, and opportunities that address the emotional, physical, cognitive, and social needs of students and families, as identified by the school, are integrated with traditional school services and programs to ensure the effective development of the whole child (Blank & Berg, 2006). Family support, out of school time, and professional capacity make up three core categories of programs, services, and opportunities found within community schools. It is not sufficient to simply provide supplemental services and opportunities to students and families; these must meet the needs of students and families and they must be integrated with the theory of change. A lack of coherence among programs, services, and the instructional plan diminishes predictability and can have an adverse effect on performance (Newmann, Smith, Allensworth, & Bryk, 2001). For supplemental services to be effective, processes and coordinating structures need to create operational symmetry among the interdependent elements of schools.

Wilson’s (1987) study of the truly disadvantaged in Chicago provides compelling evidence as to why holistic programs, services, and opportunities are needed in high poverty communities. Social and economic structures have left such communities with limited capacity to address the psychological and social needs that families and communities traditionally satisfy. A lack of psychological and social support coming from families and neighborhoods leaves many children incapable of leveraging educational and life opportunities that schools can provide (Wilson, 1987). Increasing and embedding more services in schools is only one variable in the equation; other variables relate to the social ties and relational attachments that help children develop the efficacy and motivation to take advantage of increased services, programs, and opportunities (Coleman, 1987). Cognitive, behavioral, and affective states are as dependent on the third component of the community school model – family and community engagement – as they are on supplemental services.

Figure 4: Family and Community Engagement

Family and community engagement is based on the belief that relationships defined by mutual trust and reciprocity can be a resource for individuals and groups (Coleman, 1990). Coleman (1988) found a strong association between social capital and human capital in his study of high school students. Relationships and social context are strong determinants of student and school performance. Bryk, Sebring, Allensworth, Luppescu, and Easton (2010) found that the level of social capital in Chicago communities, as partly measured by relational ties, contributed to achievement improvements of schools in truly disadvantaged communities. The lack of social capital in these communities led to achievement stagnation. Quite simply, relationships among school members based on trust and reciprocity matter for school performance (Bryk & Schneider, 2002; Goddard, 2003).

Turning relationships into a resource capable of facilitating goal attainment requires strong social bonds within schools if relational bridges that connect school members with community residents, community leaders, and business partners are to be effective. Community and family engagement is TACSI’s means to develop a strong social network where shared responsibility for student learning is a norm (Blank & Berg, 2006; Blank, Melaville, & Shah, 2003). Strong connections between families and schools maximize resources provided by the community.

Figure 5: Community Based Learning

Community-based learning is an instructional model that emerged from multiple theoretical frameworks and empirical evidence on how students learn best. The combined evidence suggests that young people are more likely to engage in learning when the content has personal meaning, builds on what students already know, and is situated within their social environment (Brown, Collins, & Duguid, 1989). Moreover, students are more likely to retain and transfer knowledge when given opportunities to apply their learning to real world issues and problems. Meaningful content, voice and choice, personal and public purpose, and assessment and feedback make up the interdependent properties of community-based learning (Melaville, Berg, & Blank, 2006). These practices are effective instructional strategies because they promote a classroom environment that supports autonomous learning in students (Reeve, Ryan, Deci, & Jang, 2008). Rather than regulate student behavior with purely external motivators, the elements of community based learning draw on commitment and motivation to foster high quality student performance.

Conditions for Learning

Community schools are defined as much by their conditions for learning as by their core components. Schools do not become social centers of communities without dense social relationships and strong normative bonds that foster a sense of belonging among school members and community partners. Structural mechanisms without convergent normative conditions are incapable of bringing the community school philosophy to scale in schools. To use an analogy, just as plants wither without water and sunlight, community schools cannot grow, nor can they achieve their intended outcomes, without nurturing environments. Normative conditions targeted by community schools function as nutrients that give life to the school’s theory of action.

TACSI conceptualized conditions for learning as a core instructional program, motivated and engaged students, holistic needs of students and families, cooperative family-school partnership, and a safe school environment. Measurable factors that make up the conditions for learning include instructional leadership, collective responsibility, collective trust, and collective efficacy. Instructional leadership and collective responsibility capture behavioral patterns that are associated with academic performance (Lee & Smith, 1996). Collective efficacy accounts for past experiences and performance of the faculty that shape their shared beliefs about future performance. A culture of collective efficacy is associated with higher student achievement (Goddard, Hoy, & Woolfolk Hoy, 2000). Collective trust is an affective condition that lubricates cooperative interactions within and between school groups (i.e. teachers, administrators, community members, students, and families), and enhances school effectiveness (Forsyth, Adams & Barnes, 2006; Forsyth, Adams, & Hoy, 2011; Goddard, Hoy, & Tschannen-Moran, 2001; Bryk & Schneider, 2002).

In short, conditions for learning represent cognitive, behavioral, and affective norms that are found within highly effective organizations and schools. The focus on actual behaviors, conditions, and interactions in schools is more helpful for school improvement than are indicators of achievement (Forsyth, Adams, & Hoy, 2011). Achievement indicators are incapable of detecting sources of performance problems and explaining reasons for achievement outcomes (Adams & Forsyth, 2010). In effect, a focus on conditions for learning directs attention to the determinants of quality performance that are controllable by school leaders.

III. EVALUATION DESIGN

This evaluation was part of a larger, ongoing investigation on the implementation and effectiveness of the community school model as developed across TACSI schools. The overall design is longitudinal and uses both qualitative and quantitative methods of observation and analysis. For this specific evaluation, data were cross-sectional and ex post facto, representing teachers’ perception of the community school model, teacher and students perceptions of social interactions and relationships within schools, and student achievement on state math and reading exams.

Data Source

Data for this study were collected from 2,130 students and 1,095 faculty members from 36 schools during the 2008-2009 school year. Fifth grade students from 18 TACSI schools and 18 comparable non-TACSI schools were sampled. All faculty members from the 36 schools were surveyed. School demographics presented in table one suggest that TACSI and non TACSI schools are comparable across indicators of poverty, average teacher experience, average teacher educational attainment, school size, and student ethnicity. Comparable school demographics provide modest support for the homogeneity of contextual factors empirically linked to conditions for learning and student performance. Similar demographic representations among comparison schools and TACSI schools reduce the probability that any achievement difference would be the result of confounding factors or selection bias. In fact, the demographic data would suggest that TACSI schools may be at a slight disadvantage for creating healthy social conditions for their higher average poverty levels and larger average size.

Table One: School Demographics

| |Non TACSI |TACSI |TACSI Mentoring |TACSI Non Mentoring |

|% Federal Lunch |61 |78 |92 |74 |

|% Minority |69 |77 |80 |74 |

|Avg. Teaching Exp. |11.5 |9.2 |10.4 |8.1 |

|% Advanced Degrees |21.9 |21.6 |19.3 |23 |

|School Size |439 |515 |552 |479 |

Evaluators administered student surveys to 5th grade students during a regular class period in the school day. Participating students returned surveys directly to evaluators. Faculty surveys were distributed and collected at two time points. Teacher surveys administered in the fall measured the development of the community school model and teacher surveys in the spring measured properties of a core instructional program. For both time periods, evaluators met with the faculty during a regular scheduled faculty meeting and informed them of the surveys. Surveys were delivered electronically using survey monkey to faculty members the day following the information meeting. Student achievement data were gathered from the participating school districts and school level demographic data were collected from the state department of education. The resultant data set was hierarchical with individual teachers and students nested within elementary schools.

Optimal Design version 2.0 was used to test the power of the sample for detecting significant achievement differences. With an expected small effect size and an estimated average of 50 students per school, results yielded a power estimate of .66 for a sample of 36 schools. Power improves to .95 with an expected medium effect size. Based on these calculations, the sample’s probability of finding a significant relationship if one exists in the population was adequate.

Measures

The Community School Development Scale (Adams, 2009) was used to measure development of the community school model in TACSI schools. The scale operationalizes the observable structural and normative properties of TACSI’s community school model. Items were written to reflect development criteria established by the management team of the network. Structural items captured the within-school spread of practices associated with the core components. Items were based on behaviors and conditions that facilitate the diffusion of reform: establishing a shared understanding; implementing, testing, and evaluating new practices; developing collective expertise; and establishing a strong social network. Sample items are presented for each core component and diffusion level.

Cross boundary Leadership. “School administrators, faculty, and staff are engaged in on-going conversations about the leadership role of the community site team.” “The community site team has appropriate representation from faculty, staff, parents, and community members.” “The school site team has united the school community around a shared vision and core beliefs for the school.” “The community school coordinator is building relationships with community members and community organizations to meet the needs of students, families, and the school.”

Holistic Programs, Services, and Opportunities. “The school community agrees on the importance of providing health care services to students and families.” “ The faculty is engaged in on-going conversations about ways to meet the holistic needs of students during out of school time.” The school community has developed a behavioral support model to promote positive behavior of all students.” “The school has formed partnerships to sustain out of school time programs and activities.”

Community-Based Learning. “Teachers in this school collectively inquire into the type of instructional strategies that have personal meaning for students.” “Teachers in this school regularly connect teaching with students’ out of school time activities.” “Teachers use both formative and summative achievement data to inform their instructional activities.” “Teachers have opportunities to lead professional development activities on community-based learning.”

Family and Community Engagement. “The school community is inquiring into ways to better engage parents in the educational process.” “The needs of the school are regularly communicated to community organizations/partners.” Home visits are conducted by faculty members.” “There is social cohesion among parents/guardians in the school community.”

Normative measures capture the functional capacity of the core components. Cross-boundary leadership is not effective unless it fosters a culture of collective responsibility and shared influence. To measure these conditions, Tschannen-Moran’s (2001) collaboration scale and Logerfo and Goddard’s (2008) collective responsibility scales were used. Holistic Programs, Services, and Opportunities need to be integrated into the overall theory of action for the school. Newmann, Smith, Allensworth, and Bryk’s (2001) instructional program coherence scale was used to measure the alignment between supplemental services and instructional designs. Family and community engagement is designed to foster parent responsibility. Green, Walker, Hoover-Dempsey, and Sandler’s (2007) role construction and parent efficacy scales were used to measure parent responsibility. Finally, community-based learning is supported by effective professional development and open interactions around instructional issues. Items from the Consortium on Chicago School Research measured the effectiveness of professional development and the prevalence of faculty interactions.

The Community School Development Scale was scored to discriminate among four levels of diffusion: inquiring, emerging, mentoring, and sustaining. Individual teacher responses were aggregated to the school level to reflect the shared perception of the faculty regarding processes and practices of the site team, the community school coordinator, and normative conditions. Development stages were determined by cumulative school means across both structural and normative factors. Development stages ranged from: 121 – 135 sustaining; 108 – 120 mentoring; 94 – 108 emerging, and less than 94 inquiring. Six of the 18 TACSI schools had reached mentoring and sustaining levels at the time of the survey. This translated into about one fourth of the students in the overall sample.

Analytical Technique

The purpose of this evaluation was to assess the school effects on individual achievement. In particular, emphasis was placed on detecting a TACSI and/or diffusion achievement effect. Because data were multilevel – students nested in schools – multilevel modeling with HLM 6.04 was used to evaluate the achievement effect attributed to the TACSI model. Focus was placed on random intercepts; the variation in individual student achievement that exists across schools. The primary school level factor was diffusion of the community school model in TACSI schools. Two types of dummy coding were used: 1) TACSI schools were coded as 1 and non TACSI schools as 0; 2) TACSI schools at the mentoring and sustaining stages were coded as 1 and all other schools (including TACSI schools at the inquiring and emerging stages) in the sample as 0. The latter coding technique provided evidence on the diffusion effect; the degree to which diffusion of cross-boundary leadership explained differences in student achievement.

Three types of models were fitted: 1) Random effects ANOVA that reported the Intraclass Correlation Coefficient (ICC), or the amount of school and student level variance in student achievement, 2) Random Intercepts Means-as-Outcomes Model that tested the school effects on individual achievement, and 3) Random Intercepts and Slopes as Outcomes Model that allowed the relationship between qualification for the federal lunch subsidy and student achievement to vary randomly across schools. Using math achievement as an example, level I and level II equations are presented for each model:

Random Effects ANOVA

Level I: Math Achievement = β0 + r

Level II: β0j = γ00 + u

Random Intercepts with School Level Predictors

Level I: Math Achievement = β0 + r

Level II: β0j = γ00 + γ01 (TACSI) + γ02 (Diffusion) + u0

Random Intercepts and Slopes as Outcomes Model

Level I: Math Achievement = β0 + β1j (F/R Lunch) + r

Level II: β0j = γ00 + u0

β1j = γ10 + γ11 (TACSI) + γ12 (Diffusion) + γ13 (SES) + γ14 (SIZE) + u1

β0j = is the school mean for math achievement

γ00 = grand mean for math achievement

γ01 = TACSI Effect: Difference in math achievement between TACSI schools and non TACSI schools.

γ02 = Diffusion Effect: Difference in math achievement between TACSI schools at the mentoring and sustaining level and all other schools in the sample.

γ11 = TACSI Effect on the social distribution of achievement: Difference in the social distribution of achievement attributed to TACSI schools.

γ12 = Diffusion Effect on the social distribution of achievement: Difference in the social distribution of achievement attributed to mentoring and sustaining TACSI schools.

Post Hoc Analysis. Based on findings from the multilevel models, two post hoc analyses were conducted to better understand the relationship between the community school model as developed in TACSI schools and student achievement. The first analysis fitted multilevel models with a different sample of students. Schools in the top quartile in the distribution of school level economic status within the district were sampled as the comparison group and students in the six mentoring and sustaining TACSI schools comprised the treatment group. The purpose of this sampling approach was to test school level differences in 5th grade math achievement between students in the mentoring and sustaining schools and students in schools with less poverty. Table two shows that there was a 44 percent differential in the average school level poverty rate between the TACSI schools at the mentoring and sustaining levels and the comparison schools.

The second post hoc analysis was an ANCOVA that examined achievement differences between students in the mentoring and sustaining TACSI schools and students from the seven most affluent schools in the district. The poverty differential was approximately 72 percent between these schools (table two). Emphasis was placed on the main effects between the two groups and the interaction of qualifying for the lunch subsidy and being in a mentoring and sustaining school.

Table Two: School Level SES Comparisons for Post Hoc Analyses

| |First Post Hoc |Second Post Hoc |

|Mentoring/Sustaining Schools Average Federal Lunch |92 % |92 % |

|qualification | | |

|Comparison Schools Average Federal Lunch Qualification |48 % |20 % |

IV. RESULTS

Schools, reform policies, and external interventions are judged by their achievement effects. As recent evidence on the limitations of value-added teacher evaluation policies indicate (Baker, Barton, Darling-Hammond, Haertel, Ladd, Linn, Ravitch, Rothstein, Shavelson, & Shepard, 2010), there are no assurances that assumed outcomes of specific reforms will occur. Thus, any claim about an achievement effect should be based on valid evidence and made judiciously. It is with this cautionary assertion in mind that the value of the community school model in TACSI schools is examined. Findings from this initial evaluation are intended to test the efficacy of TACSI schools to enhance student achievement. Results are organized by the three evaluation questions.

Is there an achievement difference between students in TACSI schools and students in comparable non TACSI Schools?

In addressing the above question, cross-sectional data were used to evaluate differences in student achievement that can be attributed to differences in school membership; that is, whether students were in a TACSI school or a comparison school. There was an average school SES difference of approximately 20 percent, with TACSI schools having a higher average school poverty level. Based on findings from the multilevel models, evidence does not support significant achievement differences between students in TACSI schools and students in comparison schools. As indicated in table three, there were no significant differences in math (γ01 = -12) or reading achievement (γ01 = -9) between the two groups of schools (see table 3 and graphs 1 and 2). While the mean difference in math and reading achievement was lower for students in TACSI schools, the initial comparison is a bit misleading because controls for student poverty were not entered into the model.

Graph One: Differences in average math achievement between TACSI students and non TACSI students without controlling for free lunch status

[pic]

Graph Two: Differences in average reading achievement between TACSI students and non TACSI students without controlling for free lunch status

[pic]

Individual student poverty had a significant effect on both math and reading achievement. Students who qualified for the lunch program scored significantly lower on the math exam (β01 = -28.74, p ≤ .01) and reading exam (β02 = -24.89, p ≤ .01) than non free/reduced lunch students. When controlling for individual free lunch status, as well as school level SES, the estimated average difference in math and reading decreased approximately nine points respectively. Graphs three and four show the changes in average math and reading achievement when accounting for individual and school poverty. Individual poverty status explained most of the achievement variance between TACSI students and non TACSI students. In short, mean achievement differences between TACSI and non TACSI schools were not significant; average performance was comparable.

Table Three: Random Intercepts Results on TACSI Effect

|Fixed Effect |Math Model 1 |Math Model 2 |Reading Model 1 |Reading Model 2 |

| | | | | |

|TACSI |-9.1 (11.43) |-2.0 (11.6) |-9.4 (8.24) |-0.09 (10.9) |

| | | | | |

|School SES | |-0.2 (10.75) | |-0.20 (.23) |

| | | | | |

|F/R Lunch Slope | |-28.43 (5.39)** | |-24.87 (8.02)** |

**p ≤ .01

Graph Three: Differences in average math achievement between TACSI students and non TACSI students after controlling for individual free/reduced lunch status

[pic]

Graph Four: Differences in average reading achievement between TACSI students and non TACSI students after controlling for individual free/reduced lunch status

[pic]

Does diffusion of the community school model in TACSI schools make a difference in student achievement?

The second research question was based on extensive evidence that points to the importance of implementation in the effectiveness of comprehensive school reform models (Honig, 2009; Rowan, Barnes, & Camburn, 2000). The level of reform diffusion was isolated to determine if development of the community school model in TACSI schools, not simply adopting the community school philosophy, had a significant effect on student achievement. When diffusion level was entered as a school level predictor, achievement differences were significant. Students in schools at the highest diffusion levels (i.e. mentoring and sustaining) significantly outperformed all other students in the sample on math (γ02 = 32, p ≤ .01) and reading (γ02 = 19, p≤ .05) achievement tests (see table four). These results suggest an approximate 32 and 19 scale point achievement difference between students in the six TACSI schools that reached the mentoring and sustaining diffusion levels and all other students in the sample. Additionally, diffusion of the community school model was a stronger school level predictor of math and reading performance than the average poverty level (γ03 = -0.45; γ03 = -0.48) of schools. The significant achievement effect attributed to bringing the community school model to scale in schools is illustrated in graphs five and six.

Table four: Random intercepts results on diffusion effect

|Fixed Effect |Math |Reading |

| | | |

|Diffusion |-32.03 (10.43)** |-19.2 (10.6)* |

| | | |

|School SES |-0.45 (0.3) |-0.47 (0.22) |

| | | |

|F/R Lunch Slope |-28.43 (5.39)** |-24.87 (8.02)** |

**p ≤ .01

*p ≤ .05

Graph Five: Differences in average math achievement between students in mentoring and sustaining TACSI schools and all other students in the sample

[pic]

Graph Six: Differences in average reading achievement between students in mentoring and sustaining TACSI schools and all other students in the sample

[pic]

Not only were math and reading achievement significantly higher for students in mentoring and sustaining schools, results suggest that a fully diffused community school model has the potential to moderate the achievement gap attributed to individual student poverty. Qualification for the federal lunch subsidy was a significant individual level predictor of math and reading performance. Students in the overall sample who qualified for the lunch subsidy scored on average 28 points lower on the state math test (β01 = -28, p ≤ .01) and 24 points lower on the state reading test (β01 = -24, p ≤ .01) than more affluent students (see table five). Narrowing the poverty gap and equalizing the social distribution of achievement is the target of most federal, state, and district policies. Evidence from this evaluation suggests that mentoring and sustaining TACSI schools are having an effect on reducing the poverty gap. Poverty students in the mentoring and sustaining schools performed on par with non free/reduced lunch students and significantly higher than free/reduced lunch students in the comparison schools (see graphs seven and eight).

Table Five: Intercepts and Slopes-as-Outcomes Results on Diffusion Effect

|Fixed Effect |Math |Reading |

| | | |

|Poverty Gap |-28.43 (6.00)** |-24.87 (5.51)** |

| | | |

|School SES |-0.24 (0.26) |-0.20 (.24) |

| | | |

|Diffusion |31.1 (12.79)** |18.93 (10.75)* |

**p ≤ .01

*p ≤ .05

Graph Seven: Moderating effect of mentoring/sustaining TACSI schools on the poverty gap

[pic]

Graph Eight: Moderating effect of mentoring/sustaining TACSI schools on the poverty gap

[pic]

Post Hoc: Closing the Poverty Gap

Post hoc analyses purposefully compared math achievement of students in mentoring and sustaining schools against the math achievement of students in schools with a more affluent student composition. The purpose of comparing student performance in two different school contexts was to test the strength of the achievement effect found in mentoring and sustaining schools. Recall from table three that comparison schools in the first post hoc had an average school SES of 48 percent compared to an average school SES of 92 percent for the mentoring and sustaining schools. Given the significant effect of concentrated poverty on student achievement one would reasonably expect student achievement to be lower in the mentoring and sustaining schools. This, however, was not the case. There was not a significant difference in the average math achievement between the two groups of students. Additionally, students in mentoring and sustaining schools outperformed free/reduced lunch students in the comparison schools by nearly 36 scale points (β01 = 36, p ≤ .05) (see table six and graph nine).

Table Six: Intercepts and Slopes-as-Outcomes Results on Diffusion Effect

|Fixed Effect |Math |

| | |

|Poverty Gap |-33.83 (6.00)** |

| | |

|School SES |-0.63 (0.35) |

| | |

|Mentoring/Sustaining |36.1 (17.79)** |

**p ≤ .01

*p ≤ .05

Graph Nine: Moderating effect of mentoring/sustaining TACSI schools on the poverty gap

[pic]

The second post hoc was intended to drill farther down into the mentoring and sustaining effect on the poverty gap. Even though 19 schools with the highest composition of affluent students were selected for the first post hoc, the average school poverty level was still around 50 percent in these schools. What about achievement differences between students in schools with the highest representation of affluent students? To address this question, seven schools with the most affluent student compositions from both districts in which TACSI schools are located were selected. There was nearly a 72 percent difference in school level poverty between these two groups of schools. Without controlling for free/reduced lunch qualification, results suggest a difference in marginal math means of approximately 18 scale points. This difference was only marginally significant (F = 3.324, p = .07) with less than one percent of the overall variability being explained by school membership (partial eta squared = .005) (see table seven).

Controlling for free/reduced lunch qualification revealed a different picture, with low income students in schools with the highest percentage of affluent students performing approximately 35 scale points lower than students in the mentoring and sustaining schools (see graph 10). Additionally, the within-school poverty gap in schools with a large affluent student population averaged approximately 63 scale points. Post hoc findings suggest that the achievement effect attributed to mentoring and sustaining schools was durable when comparing student performance in contrasting school demographics.

Table Seven: Test of between-subjects effects

|Fixed Effect |Sum of Squares |Df |F |Sig |Partial Eta Squared|

| | | | | | |

|Intercept |153731676 |1 |20925 |.00 |.97 |

| | | | | | |

|Mentoring/Sustaining |24418 |1 |3.33 |.08 |.005 |

| | | | | | |

|F/R/ Lunch |41003 |1 |5.58 |.01** |.009 |

| | | | | | |

|Mentoring/Sustaining/ F/R Lunch |108408 |1 |14.76 |.00** |.031 |

** p ≤ .01

Graph Ten: Moderating effect of mentoring/sustaining TACSI schools on the poverty gap

[pic]

In summary, evidence from three different samples provides support for an achievement effect attributed to fully diffused TACSI schools. Fifth grade students in these schools significantly outperformed their peers on the state math and reading exams in comparable Title I elementary schools. They also performed on par with students not qualifying for the free/reduced lunch subsidy in the sample of 19 schools with the lowest school level poverty, and significantly higher than free/reduced lunch students in these schools. The average math achievement of students in mentoring and sustaining schools was also significantly higher than comparable low income students in seven schools with the most affluent compositions. TACSI schools that reached a mentoring and sustaining diffusion level demonstrated the potential of the community school model to equalize the social distribution of achievement.

If an achievement effect exists, what social conditions contribute to differences in student achievement?

Reforms, much like policies or planned change, do not directly influence achievement. The effect is more indirect, operating through social conditions in schools to shape student and school performance (Forsyth, Adams, & Hoy, 2011). Conditions for learning targeted by TACSI are the mechanisms to promote effective teaching and to satisfy the learning needs of students. With evidence on an achievement effect attributed to mentoring and sustaining schools, it is important to flush out specific conditions in schools that contributed to better student and school performance.

Recent research on school reform has established trust as an essential lubricant for school improvement (Bryk & Schneider, 2002; Bryk et al. 2010; Forsyth, Adams, & Hoy, 2011). Faculty trust in teachers and students, student trust in teachers, parent trust in schools, and faculty trust in colleagues, are different forms of trust that have both direct and indirect links to student and school achievement (Goddard, Tschannen-Moran, & Hoy, 2001; Goddard, Salloum, & Berebitsky, 2009; Forsyth, Barnes, & Adams, 2006; Tschannen-Moran, 2004). Given this existing evidence, different forms of collective trust were treated as social determinants of student achievement. The purpose was to explore how normative conditions contributed to student performance.

Results from a random intercepts means-as-outcomes model (see table eight)found that student trust in teachers (γ11 = 11, p ≤ .01; γ11 = 9, p ≤ .01) and faculty trust in students and parents (γ12 = 18, p ≤ .01; γ12 = 19, p ≤ .01 ) were stronger school level predictors of math and reading achievement than school SES (γ13 = .05). Findings also show that these forms of collective trust explain differences in the relationship between individual qualification for the lunch subsidy and individual student achievement. Student achievement of low income students was significantly higher in schools with entrenched cultures of collective trust (see graphs eleven and twelve).

Table Eight: Social conditions and average student achievement

| |Math Model 1 |Math |Reading Model 1 |Reading Model 2 |

| | |Model 2 | | |

| | | | | |

|School SES |-4.03 (7.5) |9.71 (14.0) |-7.5 (6.2) |10.0 (7.9) |

| | | | | |

|School Size |-1.59 (4.7) |-4.89 (4.3) |2.1 (3.5) |-.39 (2.5) |

| | | | | |

|Student Trust in Teachers | |10.6 (2.6)** | |8.8 (3.3)** |

| | | | | |

|Faculty Trust in Students and Parents | |17.8 (4.1)** | |18.8 (6.5)** |

| | | | | |

*p ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download