Experimental Design for the Study of Treatment Effect Heterogeneity in Education Research

Peko-Spicer, Sarah

doi:https://doi.org/10.21985/n2-tfpz-g143

Work

Experimental Design for the Study of Treatment Effect Heterogeneity in Education Research

Public

Download PDF

Download All Files (.zip)

In recent years, the social sciences have been ensnared in a crisis in which many research findings cannot be replicated (Ioannidis, 2005; Open Science Collaboration, 2015; Camerer et al., 2016; Makel & Plucker, 2014). This crisis has been attributed to a variety of problems including lack of transparency about research (Wagenmakers et al., 2012) and publication bias (Francis, 2012) and has led to numerous improvements to the rigor of research practice, most notably in reporting practices (Anderson et al., 2019). Although improvements to practice have been made, concerns about the lack of replicable findings persist. One reason that the social sciences might still be grappling with this crisis is because many of the fields that fall under its umbrella presume that effects are constant. As Andrew Gelman notes, this presumption “corresponds to a simplified view of the world that can impede research discussion” (Gelman, 2015). This dissertation picks up on this observation and further asks how the design and analysis of research programs might differ if we assume effects to be just as complex as the world to which they correspond. What if we presume that effects vary over time, populations, and contexts? As Gelman notes, this problem is “more complex than simply changing methods. It requires changing mindsets” (Gelman, 2015). So, Chapter 1 of this dissertation begins with a discussion of the philosophical foundations of current practice in education research and how it might change if we work to align our understanding of the world with our understanding of effects. In doing so, it weaves together questions about the study of heterogeneity with principles of an emerging literature in education research called QuantCrit. It asks education researchers to dream about where a new foundation might take us, rather than to build defenses (or critiques) for what already exists. The primary statistical contribution of this chapter is the introduction of an effect surface as an alternative estimand to the global average treatment effect. However, in my opinion, the primary contribution of this chapter is the demonstration that methods do not exist in a vacuum and are truly part of a constellation of practices, theories, and beliefs that comprise scientific research. Effect surface estimation is a rather unwieldy problem that has been suggested before as a path forward in the social sciences (Rubin, 1992), but has yet to be seriously considered. To get the field started on this path, this dissertation distills this problem to its simplest form: estimating an effect curve that depends on a single covariate. It groups covariates into two categories: malleable covariates that can be manipulated by an investigator or research team and observed covariates that cannot be manipulated and can only be, well, observed. Chapter 2 explores how we might design a research program to estimate an effect curve when the covariate of interest is malleable, like implementation dosage. Chapter 3 explores how we might do so when the covariate of interest is observed, like a measure of school climate. In both cases, we draw comparisons to current practice – or at least what we envision alternative options to be.

Creator