13 Experiments and Quasi-Experiments
This chapter discusses statistical tools that are commonly applied in program evaluation, where interest lies in measuring the causal effects of programs, policies or other interventions. An optimal research design for this purpose is what statisticians call an ideal randomized controlled experiment. The basic idea is to randomly assign subjects to two different groups, one that receives the treatment (the treatment group) and one that does not (the control group) and to compare outcomes for both groups in order to get an estimate of the average treatment effect.
Such experimental data is fundamentally different from observational data. For example, one might use a randomized controlled experiment to measure how much the performance of students in a standardized test differs between two classes where one has a “regular”" student-teacher ratio and the other one has fewer students. The data produced by such an experiment would be different from, e.g., the observed cross-section data on the students’ performance used throughout Chapters 4 to 8 where class sizes are not randomly assigned to students but instead are the results of an economic decision where educational objectives and budgetary aspects were balanced.
For economists, randomized controlled experiments are often difficult or even indefeasible to implement. For example, due to ethic, moral and legal reasons it is practically impossible for a business owner to estimate the causal effect on the productivity of workers of setting them under psychological stress using an experiment where workers are randomly assigned either to the treatment group that is under time pressure or to the control group where work is under regular conditions, at best without knowledge of being in an experiment (see the box The Hawthorne Effect on p. 528 of the book).
However, sometimes external circumstances produce what is called a quasi-experiment or natural experiment. This “as if” randomness allows for estimation of causal effects that are of interest for economists using tools which are very similar to those valid for ideal randomized controlled experiments. These tools draw heavily on the theory of multiple regression and also on IV regression (see Chapter 12). We will review the core aspects of these methods and demonstrate how to apply them in R using the STAR data set (see the description of the data set).
The following packages and their dependencies are needed for reproduction of the code chunks presented throughout this chapter:
- AER (Kleiber & Zeileis, 2017)
- dplyr (Wickham et al., 2018)
- MASS (Ripley, 2018)
- mvtnorm (Genz et al., 2018)
- rddtools (Stigler & Quast, 2015)
- scales (Wickham, 2018)
- stargazer(Hlavac, 2018)
- tidyr (Wickham & Henry, 2018)
Make sure the following code chunk runs without any errors.
library(AER) library(dplyr) library(MASS) library(mvtnorm) library(rddtools) library(scales) library(stargazer) library(tidyr)
Kleiber, C., & Zeileis, A. (2017). AER: Applied Econometrics with R (Version 1.2-5). Retrieved from https://CRAN.R-project.org/package=AER
Wickham, H., François, R., Henry, L., & Müller, K. (2018). dplyr: A Grammar of Data Manipulation (Version 0.7.8). Retrieved from https://CRAN.R-project.org/package=dplyr
Ripley, B. (2018). MASS: Support Functions and Datasets for Venables and Ripley’s MASS (Version 7.3-51.1). Retrieved from https://CRAN.R-project.org/package=MASS
Genz, A., Bretz, F., Miwa, T., Mi, X., & Hothorn, T. (2018). mvtnorm: Multivariate Normal and t Distributions (Version 1.0-8). Retrieved from https://CRAN.R-project.org/package=mvtnorm
Stigler, M., & Quast, B. (2015). rddtools: Toolbox for Regression Discontinuity Design (’RDD’) (Version 0.4.0). Retrieved from https://CRAN.R-project.org/package=rddtools
Wickham, H. (2018). scales: Scale Functions for Visualization (Version 1.0.0). Retrieved from https://CRAN.R-project.org/package=scales
Hlavac, M. (2018). stargazer: Well-Formatted Regression and Summary Statistics Tables (Version 5.2.2). Retrieved from https://CRAN.R-project.org/package=stargazer
Wickham, H., & Henry, L. (2018). tidyr: Easily Tidy Data with ’spread()’ and ’gather()’ Functions (Version 0.8.2). Retrieved from https://CRAN.R-project.org/package=tidyr