Methods for Synthesizing and Translating Statistical Evidence in Education


This dissertation is a collection of three papers on synthesizing and translating statistical evidence in education research. Chapter 1 serves as an introduction and executive summary, and Chapters 2 - 4 contain the three substantive papers respectively. Chapter 2 presents methods for pooling sample variances across studies to improve properties of standardize mean difference treatment effects. It leverages the fact that there is often external data from other studies that have used the same outcome measure, made available via clearinghouse databases. I derive pooled variance estimators using both an ANOVA and a meta-analytic framework and find that using the pooled estimators as the denominator in the standardized mean difference can improve the precision of treatment effects and protect against the bias that can occur when study samples are homogeneous and cannot capture the total variation in the population. Chapter 3 presents a statistical cognition experiment conducted to evaluate the effectiveness of four data visualizations in communicating meta-analytic evidence to education practitioners. Drawing on evidence from the data visualization and statistical cognition literatures, I propose a new visualization - the Meta-Analytic Rain Cloud (MARC) plot - as an alternative to forest plots currently used to visualize meta-analytic data. The experiment finds that the MARC plot is more effective than current versions of forest plots in helping people understand meta-analytic evidence. Chapter 4 argues the need for a translation science in education research, which I define as a science for communicating scientific findings and concepts to lay audiences for the purposes of decision-making by policy-makers and practitioners. Because much of the evidence we communicate is statistical in nature, I point to existing literature regarding people’s statistical reasoning and data visualization cognition that is pertinent to effective communication of education research. Drawing on evidence from cognitive science, data visualization, and statistical cognition, I develop a conceptual framework for the type of translation science research that is needed to move towards more evidence-based communication practices as a field.

Alternate Identifier
Date created
Resource type
Rights statement