Skip to main content
San Diego Education Research Alliance San Diego Education Research Alliance
  1. HOME
  2. Research Projects
  3. Completed Projects
  4. School Choice

A Lottery-Based Evaluation of the Impact of Public School Choice Programs on Short- and Medium-Term Academic and Behavioral Outcomes

A research project funded by the Institute of Education Sciences (IES) - U.S. Department of Education (Award Number: R305A19012)   

 

Project Overview                                                                                                                                                  

In 2019, SanDERA received fuding from IES to conduct a retrospective evaluation of the impact of three types of public school choice programs: magnet schools, open enrollment, and a traditional busing system designed to integrate a district's schools. The research team studied school choice lottery data from a large, urban school district to examine the impact of applying to and enrolling in public schools of choice. The analysis studied the impact of winning a lottery to any of the three programs, as well as winning a lottery to the individual programs.  An advantage of combining different types of school choice is that the impact of treatment on the treated can adjust for participation in other programs even if a student loses a given lottery to a given program.  Researchers tested demographic characteristics, including race/ethnicity, English Learner status, and special education needs, as moderating variables. The research team also conducted an exploratory analysis of potential school-based mediating influences, such as school means of peer group characteristics, school test scores and value-added in reading and mathematics, and teacher experience and other qualifications. 

Key Findings

  • Winning a school lottery did not increase math or English Language Arts (ELA) scores, but did increase the probability of graduating on time and normal grade progression. The positive impact on graduation time derived mainly from high schools. 
  • Winning a school choice lottery was not significantly linked to non-cognitive outcomes, including absences, elementary school report card information on perseverance and respect, secondary school citizenship GPA or suspensions.  
  • Winning a school choice lottery was not significantly related to postsecondary outcomes. 
    The positive impacts on time to graduation and normal grade progression were partly mediated by school test scores and value added on reading achievement tests, and by student demographics.  Teacher demographics and especially teacher qualifications were less important as mediators of these effects.  

  • Among subgroup analyses by student race/ethnicity, English Learner status and special education status, the most common pattern was that winning a lottery had positive impacts for black students more often than for any other student subgroup, both in the models that combined all three types of school choice, and in models of the individual types of school choice.  Another pattern was that by winning a seat at a magnet school, special education students, unlike other students, benefited significantly in terms of reading scores, normal grade progression, and the percentage of days absent. 

 ----------------------------

Structured Abstract

Setting: The project took place in the San Diego Unified School District, a large urban district in California. 

Sample: The sample included K-12 students who applied to any of the three public school choice systems centrally administered by the district, over the years fall 2001 to fall 2008 and fall 2010 to fall 2013. Over this period, about 40% of students attended public schools of choice. 

Intervention: The intervention was admission to a public choice school. During the period of study, students in the district who wished to attend a public school other than their neighborhood school had four main options available: magnet schools, the Voluntary Enrollment Exchange Program (VEEP), the School Choice program (the state-mandated open-enrollment program), and charter schools. This study focuses on the magnet program, the traditional busing program (VEEP) and the open enrollment program (Choice). 

Research Design and Methods: The team leveraged the fact that access to public choice schools in this district is lottery-based. The use of admission lotteries approximated a randomized controlled trial, which produces causal estimates. The unit of randomization is the individual student’s school application. The researchers treated each school lottery across the three types of programs as a separate “experiment” and also combined the lotteries and examined the overall impact of winning a lottery and enrolling a school of choice. The researchers ensured baseline equivalence and conducted a variety of robustness checks. 

Control Condition: The control condition was students in admission lotteries who did not win the lottery to attend a choice school. In addition to applying to the same school in the same year, comparison group students must also have been in the same grade level and priority group. 

Key Measures: Academic outcomes included test score changes in math and English Language Arts (ELA), progression to the next grade, and graduation on time.  Additional measures included absences, elementary school report card ratings for perseverance and respect, secondary school “citizenship GPA” and suspensions. Postsecondary measures included enrollment in the first year after high school, the number of years enrollment in the first four years after high school graduation, obtaining a Bachelor's degree within five years of high school graduation, and obtaining an Associate's degree or certificate from a two-year college within five years of high school graduation.   

Data Analytic Strategy: Using regression techniques, the research team estimated both the impact of winning a school choice lottery (intent to treat, or ITT) and of winning a lottery and attending (treatment on the treated, or TOT). The team estimated the impact of TOT using winning a lottery as an instrumental variable (IV) for enrolling in a school receiving students as part of the given choice program. The team conducted moderation analyses by testing for interactions of winning a lottery with student background variables related to race/ethnicity, English Learner and special education status.  The analyses were also repeated separately for elementary, middle and high schools. For results with robust non-zero impacts, a mediator analysis was performed.  Mediating school-level variables included school demographics, school mean test scores and value-added in reading achievement tests, and teacher qualifications and demographics. 

 ----------------------------

Project Timeline

July 1, 2019 through June 20, 2022

Availability of Data 

The Data Analysis and Reporting Department at the San Diego Unified School District (SDUSD) has received the final dataset used for the analysis along with a data codebook. Other researchers wishing to replicate the results can do so by applying for access to the data to the RPRP (Research Proposal Review Panel) in the aforementioned office. The current contact for the RPRP is Mr. Ronald Rode who can be reached at rrode@sandi.net. The district website is updated from time to time; if a researcher finds that the above link has been removed or changed, please contact Professor Julian Betts for help, at jbetts@ucsd.edu.

Contact Information

For additional information, please contact Julian Betts, Department of Economics, UC San Diego (jbetts@ucsd.edu)