**Open Review**. We want your feedback to make the book better for you and other students. You may annotate some text by selecting it with the cursor and then click "Annotate" in the pop-up menu. You can also see the annotations of others: click the arrow in the upper right hand corner of the page

## 13.2 Threats to Validity of Experiments

The concepts of internal and external validity discussed in Key Concept 9.1 are also applicable for studies based on experimental and quasi-experimental data. Chapter 13.2 of the book provides a thorough explanation of the particular threats to internal and external validity of experiments including examples. We limit ourselves to a short repetition of the threats listed there. Consult the book for a more detailed explanation.

#### Threats to Internal Validity

**Failure to Randomize**If the subjects are not randomly assigned to the treatment group, then the outcomes will be contaminated with the effect of the subjects’ individual characteristics or preferences and it is not possible to obtain an unbiased estimate of the treatment effect. One can test for nonrandom assignment using a significance test (\(F\)-Test) on the coefficients in the regression model \[X_i = \beta_0 + \beta_1 W_{1i} + \dots +\beta_2 W_{ri} + u_i \ \ , \ \ i=1,\dots,n.\]

**Failure to Follow the Treatment Protocol**If subjects do not follow the treatment protocol, i.e., some subjects in the treatment group manage to avoid receiving the treatment and/or some subjects in the control group manage to receive the treatment (

*partial compliance*), there is correlation between \(X_i\) und \(u_i\) such that the OLS estimator of the average treatment effect will be biased. If there are data on*both*treatment actually received (\(X_i\)) and initial random assignment (\(Z_i\)), IV regression of the models (13.1) and (13.2) is a remedy.**Attrition**Attrition may result in a nonrandomly selected sample. If subjects systematically drop out of the study after being assigned to the control or the treatment group (systematic means that the reason of the dropout is related to the treatment) there will be correlation between \(X_i\) and \(u_i\) and hence bias in the OLS estimator of the treatment effect.

**Experimental Effects**If human subjects in treatment group and/or control group know that they are in an experiment, they might adapt their behaviour in a way that prevents unbiased estimation of the treatment effect.

**Small Sample Sizes**As we know from the theory of linear regression, small sample sizes lead to imprecise estimation of the coefficients and thus imply imprecise estimation of the causal effect. Furthermore, confidence intervals and hypothesis test may produce wrong inference when the sample size is small.

#### Threats to External Validity

**Nonrepresentative Sample**If the population studied and the population of interest are not sufficiently similar, there is no justification in generalizing the results.

**Nonrepresentative Program or Policy**If the program or policy for the population studied differs considerably from the program (to be) applied to population(s) of interest, the results cannot be generalized. For example, a small-scale program with low funding might have different effects than a widely available scaled-up program that is actually implemented. There are other factors like duration and the extent of monitoring that should be considered here.

**General Equilibrium Effects**If market and/or environmental conditions cannot be kept constant when an internally valid program is implemented broadly, external validity may be doubtful.