NEW YORK CITY’S CHARTER SCHOOLS

NEW YORK CITY'S CHARTER SCHOOLS

What the Research Shows

Marcus A. Winters

Senior Fellow

REPORT | February 2018

New York City's Charter Schools

About the Author

Marcus A. Winters is a senior fellow at the Manhattan Institute and an associate professor at Boston University. His research focuses on education policy, including school choice, accountability, and teacher quality. Winters's papers have been published in the Journal of Public Economics, Journal of Policy Analysis and Management, Educational Researcher, Educational Evaluation and Policy Analysis, Education Finance and Policy, Educational Finance, Economics of Education Review, and Teachers College Record. His op-eds have appeared in the Wall Street Journal, Washington Post, and USA Today, and he is often quoted in the media on education issues. Winters holds a B.A. in political science from Ohio University and a Ph.D. in economics from the University of Arkansas.

Acknowledgment

The author thanks the Walton Family Foundation for its generous support.

2

Contents

Executive Summary...................................................................4 Introduction...............................................................................5 What Is the Effect of Attending a Charter School on

Student Achievement?...............................................................6 Are All Charter Schools Equally Effective?..................................9 What School-Based Factors Explain a Charter School's

Effectiveness Relative to Other Charter Schools?.......................9 Do Differences in Resources Explain Charter School Effects

Relative to Traditional Public Schools and Other Charters?.......10 Do Charter Schools Experience Disproportionate Student

Attrition? Are Low-Performing Students Systematically Pushed Out of Charters?..........................................................11 How Do the Characteristics of Students Attending Charter Schools Differ from Students Attending Traditional Public Schools?.......................................................................12 What Explains the Difference in the Characteristics of Charter and Traditional Public School Students?.......................13 Has Charter School Expansion Harmed the Performance of Students Attending Traditional Public Schools?....................14 What Are Areas for Future Research?......................................15 Conclusion..............................................................................16 Appendix: Research on New York City Charter Schools............................16 Endnotes................................................................................. 18

3

New York City's Charter Schools

Executive Summary

This report evaluates the current state of research on New York City charter schools. Overall, their effect on student performance is unambiguously positive. But the research is more dated and limited in scope than proponents and critics of charters appreciate.

While the research on charter schools across the U.S. is growing, this paper focuses exclusively on studies that evaluate one or more aspects of New York City charter schools. This focus is important because the characteristics and effects of charter schools vary from city to city and, indeed, from charter to charter.

Key Findings

Students who attend a New York City charter instead of a traditional public school do much better on math tests and somewhat better on English language arts (ELA) tests. These positive effects appear to have remained similar over time, even as the number of charters and students has expanded. However, the studies that employ the strongest potential research design are dated or targeted only at a few highly effective schools that do not represent the full charter sector today.

New York City charter schools are not equally effective. About half appear to be more effective than the traditional public school that students would have otherwise attended. A small percentage of charters appear to have negative effects on student test scores.

New York City charters operated by a charter management organization appear to be more effective than other charters, on average. School attributes--such as frequent teacher feedback, data-guided instruction, tutoring, high standards for students, and additional instruction time--are related to larger charter school effects. Variations in class size and the number of teachers with advanced degrees are not related to the effectiveness of charter schools.

Differences in resources do not explain differences in effectiveness between charter schools and traditional public schools or between charters in New York City. To date, charter schools have received fewer public resources than have traditional public schools, but recent changes in how charters are funded might narrow the gap. There is large variation in the resources that charter schools receive from nonpublic sources. However, differences in overall per-pupil funding do not appear to be associated with school quality.

There is no evidence that New York City charter schools systematically push out low-performing students. In fact, low-performing students are less likely to exit charter schools than they are to exit traditional public schools, especially after accounting for differences in their demographic characteristics.

Competition from New York City charter schools has either no effect, or a positive effect, on the performance of students in the nearby traditional public schools.

4

NEW YORK CITY'S CHARTER SCHOOLS

What the Research Shows

Introduction

If you ask five people what the academic research says about New York City charter schools, you might get seven different answers. Indeed, the empirical evidence is now so broad and long-standing that it is worth stepping back to reflect on where it stands.

This report reviews existing research, which shows that New York City charter schools have an unambiguously positive effect on student performance. But the research is more dated and limited in scope than is commonly appreciated.

I focus exclusively on the results from studies that specifically evaluate one or more aspects of New York City's charter sector because the characteristics and impacts of charter schools vary dramatically by locality.1 In New York and elsewhere, conversations about the impact of charter schools are often contaminated by discussions of the national charter school literature. That literature provides a broad understanding of the charter school movement and offers insights into why charters appear to be more effective in some localities than in others. But in cases like New York City, where broadly informative local research is available, the policy conversation should focus explicitly on the local body of research. (In cases where research on non?New York charter school sectors might inform this report, I include a brief description in the endnotes.)

I have done my best to collect all of the relevant academic literature on New York City charter schools in this review. (See the Appendix for further readings.) I do not impose a firm cutoff date to include a paper, but I do exclude early studies that are clearly no longer relevant.2 In addition to employing my own knowledge of the literature, I searched on Google Scholar, browsed the websites of organizations--supporters as well as opponents--that frequently write about the city's charter sector in the popular press,3 and collected further relevant papers that were cited in the initial papers that I identified.

I do not restrict the review to published academic work. Peer review is an imperfect signal of a study's quality. In some cases, I reference published critiques of a particular study, but I do not treat such reviews as evaluations. I organize this review as a series of questions and answers--a Q&A format seemed best for addressing the most pressing claims and controversies among the public, as well as among the scholarly community.

5

New York City's Charter Schools

What Is the Effect of Attending a Charter School on Student Achievement?

Large and positive. Attending a charter school instead of a traditional public school has a large, positive effect on student achievement in math and a smaller positive effect in ELA. The effect of attending a charter school appears to have remained similar over time, even as the sector has expanded dramatically. Unfortunately, studies using the strongest potential research design are dated or targeted at only a few highly effective schools that do not represent the full charter sector today.

New York City charter schools post substantially higher test scores, on average, than do the city's traditional public schools. Alas, the aggregate test scores of charter and traditional public schools are not directly comparable. Differences in the performance of charter and traditional public school students might reflect differences in the quality of schooling, student differences, or both. Measuring the effect of attending a charter school on student performance requires rigorous statistical analysis.

Seven studies estimate the causal effect of attending a New York City charter school on student performance--i.e., they attempt to directly answer the question of whether the same charter school student would have performed differently had he or she attended a traditional public school.4 These studies use one of two research designs, both of which have benefits and limitations that should be considered when interpreting their results.

Five use a randomized field trial (RFT) design. This design takes advantage of the fact that oversubscribed charter schools are required to use a random lottery to offer students the opportunity to enroll. The researchers compare the later outcomes of students who were randomly offered a charter school seat with those of students who also applied but were randomly denied the opportunity to enroll.

Because winning the charter lottery is not related to any other student characteristic, any differences in the later outcomes of lottery winners and losers are either random or are directly due to the opportunity to enroll in a charter school. Researchers then statistically adjust the estimated effect of being offered a charter seat (the "reduced form" or "intention-to-treat" estimate) to account for the fact that not all students who are offered a charter school seat actually attend the school

(the treatment-on-the-treated estimate). Because the RFT design accounts for all pretreatment differences between the treatment and control groups, it is widely considered to be the gold standard for research designs in the social sciences.

From a policy perspective, RFT studies produce limited information because their results strictly apply only to those represented in the lottery pools. Thus, if students who apply to charter schools are systematically different from those who do not apply to them, then the estimates from an RFT study likely would not apply to non-applicants.5 Many RFT studies of New York City charter schools also focus exclusively on one or several charter operators; and the schools included are typically among the highest-performing. Such studies reveal a great deal about the effect of attending a particular charter school, but they do not offer a complete picture of the effectiveness of New York's charter school sector.

The second research design, "matching," uses a computer algorithm to compare the performance of each student attending a charter school with a similar student or students attending a traditional public school. The Center for Research on Educational Outcomes (CREDO) at Stanford University has used this design to estimate the effects of charter schools on student performance in New York City and elsewhere (two of its matching studies are discussed below). Unlike the RFT design, matching allows the researcher to include all charter schools, regardless of whether they are oversubscribed or participate in the evaluation.

Matching requires stronger assumptions than RFT to produce estimates of the causal impact of attending a charter school. RFT studies control for differences between applicants who are offered and not offered a charter seat, whether or not the difference is observed in the researcher's data set. Matching studies match students only according to characteristics that are observed in the data set, even though students attending charters may differ from those attending traditional public schools in unobserved ways, such as the level of parental involvement.

Figures 1?2 plot the main estimates for the impact of attending a New York City charter school on student math and ELA scores, as reported in the seven studies.6 The five RFT studies (blue dots) list the school included in the analysis, as well as the studies' authors. The two matching studies (yellow dots) are CREDO (2013) and CREDO (2017); their estimates hold for all charter schools operating in a given year in New York City.

Figures 1?2 show that the estimated effect of attending

6

FIGURE 1

Estimating the Effect of Attending a New York City Charter (ELA)

0.30

RFT Matching

0.25

Corcoran & Cordes (2015)-- Democracy Prep

0.20

0.20*

Per-Year Effect

0.15

0.10*

0.10

0.06 *

0.05

Hoxby & Murarka (2009)--All

0.00

0.09*

0.02*

Dobbie & Fryer (2011)-- Promise Academy

0.11

Dobbie & Fryer (2013)-- 29 Schools 0.06*

0.02*

0.02*

Unterman (2017)-- Success Academy

0.09

0.09*

0.03* 0.00

0.04*

-0.05

2006

2007

2008

2009

2010

2011

2012

2013

2014

2015

2016

Final Observed Year

*Indicates whether the result is statistically significant at the 95% confidence level.7 The x-axis represents the final observed year in the study's sample. The y-axis is the magnitude of the study's main estimate (i.e., the effect of a year of charter schooling, measured in standard deviation units).8 For example, to obtain the study's estimate for the effect of four years of charter schooling, one would multiply the study's estimate by 4.9 For comparison, the most cited study, Krueger (1999), evaluating the impact of small class sizes, found that students in small classes outperformed students in regular-size classes by about 0.22 standard deviations after four years, which--when translated into a constantly accumulating single-year effect--would be an average of 0.055 standard deviations.10

Source: Author's compilation of seven studies cited in endnote 4

FIGURE 2

Estimating the Effect of Attending a New York City Charter (Math)

0.30

Per-Year Effect

0.25

0.20

0.15

0.12*

0.10 0.09*

Hoxby & Murarka (2009)--All

0.05

0.14*

0.13*

Dobbie & Fryer (2011)-- Promise Academy

0.19 0.17*

0.11*

0.11*

Dobbie & Fryer (2013)-- 29 Schools

0.11*

RFT Matching

0.29* Corcoran & Cordes (2015)-- Democracy Prep

Unterman (2017)-- Success Academy

0.14

0.12*

0.13*

0.07*

0.00

-0.05

2006

2007

2008

2009

2010

2011

2012

2013

2014

2015

2016

Final Observed Year

*Indicates whether the result is statistically significant at the 95% confidence level.11 The x-axis represents the final observed year in the study's sample. The y-axis is the magnitude of the study's main estimate (i.e., the effect of a year of charter schooling, measured in standard deviation units).12 For example, to obtain the study's estimate for the effect of four years of charter schooling, one would multiply the study's estimate by 4.13 For comparison, the most cited study, Krueger (1999), evaluating the impact of small class sizes, found that students in small classes outperformed students in regular-size classes by about 0.22 standard deviations after four years, which--when translated into a constantly accumulating single-year effect--would be an average of 0.055 standard deviations.14

Source: Author's compilation of seven studies cited in endnote 4

7

New York City's Charter Schools

FIGURE 3

Number of Charter Schools in New York City

250

197

205

216

227

200

183

159

150

125

135

99

100

78

47

58

60

50

0

2006

2007

2008

2009

Source: New York City Charter School Center and CREDO

2010

2011

2012

2013

2014

2015

2016

2017

2018

a New York City charter school is positive in all but one case; 23 of 28 estimates are significantly positive. Attending a New York City charter appears to have a larger effect in math than in ELA. The magnitude of the positive effect in math is substantial, especially given that the estimate covers only a year of charter school attendance. The smallest estimate suggests that four years of attending a charter school would lead to a 0.28 standard deviation increase in student math scores (0.07 x 4 years). The estimates in math are also quite stable over time and across studies.

Meanwhile, the estimated effects on student performance in ELA are more modest and less consistent. However, the evidence suggests a significant positive effect in ELA that would be meaningful after accumulating over time.

As noted, RFT and matching studies have strengths and weaknesses. Three of the RFT studies evaluate only a single, high-performing charter network (Promise Academy, Democracy Prep, or Success Academy). A fourth uses information from a more expansive group of 29 schools. Hoxby and Murarka (2009), the fifth RFT study and the only one to include all of the city's then-operating charter schools, found a significant positive impact in math and ELA.

Though Hoxby and Murarka remain highly influential in the policy discussion, their results are quite dated. New York City's charter sector has grown dramatically (Figure 3) since 2006, the final year of study in Hoxby and Murarka. In 2006, only 47 charter schools operated in New York, compared with 227 today. Because of this big expansion, the effect of attending a charter school may have changed over time.

If, say, the city successfully closed ineffective charters and encouraged the opening of only highly effective ones, the impact of attending a charter school would tend to increase over time. On the other hand, the quality of charter schools might diminish as the sector expands and digs deeper into the labor pool for administrators and teachers, or if the additional students who enroll are, for some reason, less responsive to charter schooling than the average student who enrolled previously.

Compare matching and RFT still further. The fact that the RFT studies often include only a small minority of charter schools likely explains why they tend to produce substantially larger effect size. Meanwhile, each matching estimate produces a statistically significant result; but four of 10 RFT estimates are statistically insignificant. The reason: the RFT estimates are measured imprecisely, partly because they utilize far fewer schools and students than do the matching estimates.

Are the matching estimates reasonable? As of 2006, the estimated impact of attending a charter school using RFT (0.06 standard deviations in ELA and 0.09 standard deviations in math) was similar to the estimated impact of attending a charter school in 2007 using matching (0.10 standard deviations in ELA and 0.12 standard deviations in math). The fact that these approaches--applied at about the same time, using nearly the same sample of schools--produced relatively similar results suggests that matching is capable of producing causal estimates for the effect of attending a New York City charter school. Indeed, the matching estimates have been relatively consistent over time, especially in math. This suggests that the average effectiveness of the city's charter sector has not diminished, even as it has expanded.15

8

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download