Education Technology: An Evidence-Based Review

[Pages:6]NBER WORKING PAPER SERIES

EDUCATION TECHNOLOGY: AN EVIDENCE-BASED REVIEW

Maya Escueta Vincent Quan Andre Joshua Nickow Philip Oreopoulos

Working Paper 23744

NATIONAL BUREAU OF ECONOMIC RESEARCH 1050 Massachusetts Avenue Cambridge, MA 02138 August 2017

We are extremely grateful to Caitlin Anzelone, Rekha Balu, Peter Bergman, Brad Bernatek, Ben Castleman, Luke Crowley, Angela Duckworth, Jonathan Guryan, Alex Haslam, Andrew Ho, Ben Jones, Matthew Kraft, Kory Kroft, David Laibson, Susanna Loeb, Andrew Magliozzi, Ignacio Martinez, Susan Mayer, Steve Mintz, Piotr Mitros, Lindsay Page, Amanda Pallais, John Pane, Justin Reich, Jonah Rockoff, Sylvi Rzepka, Kirby Smith, and Oscar Sweeten-Lopez for providing helpful and detailed comments as we put together this review. We also thank Rachel Glennerster for detailed support throughout the project, Jessica Mardo and Sophie Shank for edits, and to the Spencer Foundation for financial support. Any errors or omissions are our own. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.

NBER working papers are circulated for discussion and comment purposes. They have not been peer-reviewed or been subject to the review by the NBER Board of Directors that accompanies official NBER publications.

? 2017 by Maya Escueta, Vincent Quan, Andre Joshua Nickow, and Philip Oreopoulos. All rights reserved. Short sections of text, not to exceed two paragraphs, may be quoted without explicit permission provided that full credit, including ? notice, is given to the source.

Education Technology: An Evidence-Based Review Maya Escueta, Vincent Quan, Andre Joshua Nickow, and Philip Oreopoulos NBER Working Paper No. 23744 August 2017 JEL No. I20,I29,J24

ABSTRACT

In recent years, there has been widespread excitement around the potential for technology to transform learning. As investments in education technology continue to grow, students, parents, and teachers face a seemingly endless array of education technologies from which to choose-- from digital personalized learning platforms to educational games to online courses. Amidst the excitement, it is important to step back and understand how technology can help--or in some cases hinder--how students learn. This review paper synthesizes and discusses experimental evidence on the effectiveness of technology-based approaches in education and outlines areas for future inquiry. In particular, we examine RCTs across the following categories of education technology: (1) access to technology, (2) computer-assisted learning, (3) technology-enabled behavioral interventions in education, and (4) online learning. While this review focuses on literature from developed countries, it also draws upon extensive research from developing countries. We hope this literature review will advance the knowledge base of how technology can be used to support education, outline key areas for new experimental research, and help drive improvements to the policies, programs, and structures that contribute to successful teaching and learning.

Maya Escueta Teachers College Columbia University 525 W 120th St New York, NY 10027 mme17@tc.columbia.edu

Vincent Quan Abdul Latif Jameel Poverty Action Lab, North America (J-PAL North America) 400 Main Street, E19-201 Cambridge, MA 02142 quanv@mit.edu

Andre Joshua Nickow Northwestern University Department of Sociology 1810 Chicago Ave. Evanston, IL 60208 a-nickow@northwestern.edu

Philip Oreopoulos Department of Economics University of Toronto 150 St. George Street Toronto, ON M5S 3G7 CANADA and NBER philip.oreopoulos@utoronto.ca

1. Introduction

Technological innovation over the past two decades has indelibly altered today's education landscape. Revolutionary advances in information and communications technology (ICT)--particularly disciplines associated with computers, mobile phones, and the Internet-- have precipitated a renaissance in education technology (ed-tech), a term we use here to refer to any ICT application that aims to improve education. In the United States, the market for PreK-12 software alone had exceeded $8 billion1, and a recent industry report projects an estimated value of $252 billion for the global ed-tech industry by 2020.2 Governments, schools, and families increasingly value technology as a central part of the education process, and invest accordingly.3 In the coming years, emerging fields like machine learning, big data, and artificial intelligence will likely compound the influence of these technologies even further, expanding the already dizzying range of available education products, and speeding up cycles of learning and adjustment.

Collectively, these technologies offer the potential to open doors and build bridges by expanding access to quality education, facilitating communication between educators, students, and families, and alleviating frictions across a wide variety of educational contexts from early childhood through adulthood. For example, educational software developers work to enable educators to deliver the latest learning science advances to schools in inner cities and remote rural areas alike. The proliferation of cell phones and growing ease in connecting them to

1 SIIA, 2015. Digital-Content. 2 Morrison, 2017.

king/#32966ae927a6.

3 Bulman and Fairlie, 2016.

2

Internet-based information systems has enabled the scaling of automated text messaging systems that aim to inform, simplify, and encourage students and their parents as they traverse difficult sticking points in education, like the transition to college. And online educational institutions may bring opportunities to earn degrees to students who would otherwise be constrained by work, families, disabilities, or other barriers to traditional higher education.

But the rapid proliferation of new technologies within education has proved to be a double-edged sword. The speed at which new technologies and intervention models are reaching the market has far outpaced the ability of policy researchers to keep up with evaluating them. The situation is well-summarized by a recent headline: "Ed-Tech Surges Internationally--and Choices for Schools Become More Confusing."4 While most agree that ed-tech can be helpful under some circumstances, researchers and educators are far from a consensus on what types of ed-tech are most worth investing in and in which contexts.

Furthermore, the transformations associated with ed-tech are occurring in a context of deep and persistent inequality. Despite expanding access to some technologies, the digital divide remains very real and very big. While 98 percent of children in United States households with incomes exceeding $100,000 per year have a computer at home, only 67 percent of children in households with incomes lower than $25,000 have them.5 Even when disadvantaged students can physically access technology, they may lack the guidance needed for productive utilization--a "digital-use divide."6 Depending on design and implementation, education technologies could alleviate or aggravate existing inequalities. Equity considerations thus add another layer to the need for caution when implementing technology-based education programs.

4 Molnar, 2017.

confusing/.

5 Bulman and Fairlie, 2016.

6 Brotman, 2016. . 3

Of course, not every intervention model can be evaluated, and the extent of success inevitably varies across educational approaches and contexts even within well-established fields. But the speed and scale with which many ed-tech interventions are being adopted, along with the enormous impact they could have over the next generation, demand a closer look at what we know. To confront this issue, the present review takes stock of rigorous quantitative studies on technology-based education interventions that have been conducted so far, with the goal of identifying policy-relevant insights and highlighting key areas for future inquiry. In particular, for reasons explained in the following section, we assembled what we believe to be a comprehensive list of all publicly available studies on technology-based education interventions that report findings from studies following either of two research designs, randomized control trials or regression discontinuity designs, and based our analyses primarily on these studies.

In the next section, we discuss our literature review methodology in greater depth. Sections 3-6 constitute the core of the review--these sections respectively synthesize the evidence on the four topic areas that encapsulate the overwhelming majority of studies that we included: 1) access to technology, 2) computer-assisted learning, 3) online courses, and 4) behavioral interventions. Section 7 offers concluding observations and considers several of the priority areas for future research that we consider vital to ongoing efforts at more effectively and equitably leveraging technology for learning.

4

2. Literature Review Methodology

Several recent reviews have synthesized empirical evidence relevant to aspects of ed-tech policy.7 The present paper aims to contribute to these efforts in two main ways. First, while existing reviews have covered subsets of ed-tech, no recent review has attempted to cover the full range of ed-tech interventions. In particular, no previous review to our knowledge brings together computer- and internet-based learning on one hand and technology-based behavioral interventions on the other. Of course, expanding our scope must come with some sacrifice--it would not be feasible to meaningfully integrate all studies relating to all areas of ed-tech into a single paper. Instead, we focus on studies presenting evidence from randomized control trials (RCT) and regression discontinuity designs (RDDs). Our core focus on RCT- and RDD-based studies constitutes a second unique contribution of this review--we argue that, in addition to helping us define sufficiently clear and narrow inclusion conditions, a focus on RCTs and RDDs adds a productive voice to broader and more methodologically-diverse policy research dialogues in an environment characterized by complex tangles of cause and effect.

Why focus on RCTs and RDDs? In the fields of program evaluation and applied microeconomics, RCTs--when properly implemented--are generally considered the strongest research design framework for quantitatively estimating average causal effects.8 RCTs are randomized experiments, studies in which the researcher randomly allocates some participants into one or more treatment group(s) subjected to an intervention, program, or policy of interest, and other participants into a control group representing the counterfactual--what would have

7 Bulman and Fairlie, 2016; Lavecchia, Liu, and Philip Oreopoulos, 2014; Means et al., 2010. 8 Angrist and Pischke, 2008.

5

happened without the program.9 Randomization assures that neither observable nor unobservable characteristics of participants predict assignment, "and hence that any difference between treatment and control...reflects the impact of the treatment."10 In other words, when done correctly, randomization ensures that we are comparing apples to apples and allows us to be confident that the impacts we observe are due to the treatment rather than some other factor. Yet as a result of cost, ethics, and a variety of other barriers, RCTs are not always possible to conduct.

Over the past several decades, methodologists have developed a toolkit of research designs, known broadly as quasi-experiments, that aim to approximate experimental research to the greatest extent possible using observational data. Commonly used examples include instrumental variable, difference-in-difference, and propensity-score matching designs. Regression discontinuity designs (RDDs) are quasi-experiments that identify a well-defined cutoff threshold which defines a change in eligibility or program status for those above it--for instance, the minimum test score required for a student to be eligible for financial aid. While very high-scoring and very low-scoring students likely differ from one another in ways other than their eligibility for financial aid, "it may be plausible to think that treatment status is `as good as randomly assigned' among the subsample of observations that fall just above and just below the threshold."11 So, when some basic assumptions are met, the jump in an outcome between those just above and those just below the threshold can be interpreted as the causal effect of the intervention in question for those near the threshold.12

9 Duflo, Glennerster, and Kremer 2008; Glennerster and Takavarasha, 2013. 10 Banerjee and Duflo, 2017. 11 Lee and Card, 2008. 12 Imbens and Lemieux, 2008; Thistlewaite and Campbell, 1960.

6

RDDs can only be used in situations with a well-defined threshold that determines whether a study participant receives the intervention. We chose to include them but not other quasi-experimental designs because they can be as convincing as RCTs in their identification of average causal effects. With minimal sensitivity to underlying theoretical assumptions, RDDs with large samples and a well-defined cut-off produce estimated program effects identical to conducting RCTs for participants at the cut-off.13 Although RDDs are quasi-experiments, in the remainder of this review we refer to the RCTs and RDDs included in this review as experimental research for simplicity. We chose to focus on RCTs and RDDs not because we believe they are inherently more valuable than studies following other research designs, but because we felt that the policy literature on ed-tech is flooded with observational research and could benefit from a synthesis of evidence from the designs most likely to produce unbiased estimates of causal effects. Furthermore, we introduce, frame, and interpret the experimental results in the context of broader observational literatures.

RCTs and RDDs estimate the impact of a program or policy on outcomes of interest. But the estimates they come up with are sometimes difficult to compare with one another given that studies test for impact on different outcomes using different measurement tools, in populations that differ in their internal diversity. While these differences can never be completely eliminated and effect sizes must always be considered in the contexts within which they were identified, standard deviations offer a roughly comparable unit that can give us a broad sense of the general magnitude of impact across program contexts. Standard deviations essentially represent the effect size relative to variation in the outcome measurement. Economists studying education generally follow the rule of thumb that less than 10 percent of a standard deviation is small, 10

13 Berk et al., 2010; Cook and Wong, 2008; Shadish et al., 2011.

7

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download