Quasi-Experimental Design: Types, Analysis, Reporting

A quasi-experimental design tests the effect of an intervention, like an experiment does, but without random assignment to conditions. Researchers use it when randomization is impossible, unethical, or impractical: a policy that applies to one region, a teaching method adopted by some schools, a hospital protocol introduced on a set date. Because groups are not randomized, the central task is ruling out other explanations for any difference you find.

This guide covers the main designs, the threats to validity each one handles, how to analyze them, and how to report them.

Quasi-experiment vs true experiment vs observational study

The line between a quasi-experiment and an observational study is not sharp. What defines a good quasi-experiment is a deliberate design (a comparison group, pretests, multiple time points, a known assignment rule) that makes specific alternative explanations less plausible.

Main types of quasi-experimental design

1. Nonequivalent groups design

Two existing groups (for example, two classes, two clinics) are compared; one receives the intervention. Without a pretest (posttest-only), any difference could simply reflect who was in each group, so this version is weak. Adding a pretest makes it much stronger: the nonequivalent groups pretest-posttest design. You can then check baseline similarity and analyze change over time.

2. One-group pretest-posttest design

One group is measured, receives the intervention, and is measured again. This is common in pilot studies and practice-improvement projects, but it cannot separate the intervention from anything else that changed over the same period (history, maturation, practice effects, regression to the mean). Treat its results as preliminary.

3. Interrupted time series (ITS)

The outcome is measured at many time points before and after the intervention, for example monthly infection rates for 24 months before and 24 months after a new protocol. Analysis asks whether the level (an immediate jump) or the slope (trend) changed at the interruption, beyond what the pre-intervention trend predicted. A common model is segmented regression:

y_t = b0 + b1 × time + b2 × post + b3 × time_after + e_t

where post is 0 before and 1 after, and time_after counts time since the intervention. b2 is the level change and b3 is the slope change. Account for autocorrelation and seasonality. Adding a comparison series that did not receive the intervention strengthens the design considerably.

4. Regression discontinuity (RD)

Assignment is based on a cutoff on a continuous score: students below a test score get tutoring, patients above a risk score get a drug. Units just above and just below the cutoff are very similar, so a jump in the outcome at the cutoff estimates the effect for units near it. RD is often considered the strongest quasi-experimental design when the assignment rule is strictly followed. Key checks: no manipulation of the score around the cutoff, and results that hold across reasonable bandwidths and functional forms. The estimate applies to units near the cutoff, not to everyone.

5. Difference-in-differences (DiD)

Measure the outcome before and after the intervention in a treated group and in a comparison group. The effect estimate is the change in the treated group minus the change in the comparison group. This removes stable differences between groups and common changes over time. The key assumption is parallel trends: without the intervention, both groups would have changed by the same amount. Checking pre-intervention trends with several earlier time points makes this assumption more credible.

DiD is closely related to panel fixed effects models; see fixed effects vs random effects for the within-unit logic.

Threats to internal validity

The threats below follow the classic framework from Campbell and Stanley and later Shadish, Cook, and Campbell. Plan which ones your design addresses.

External validity matters too: an effect found in two volunteer schools may not generalize to all schools.

A worked example: pretest-posttest with a comparison group

A school district introduces a new reading program in one school (treatment, n = 40 students). A similar school continues as usual (comparison, n = 40). Reading scores are measured before and after the semester.

Difference-in-differences estimate: 8.0 − 3.0 = 5.0 points. The comparison school's 3-point gain represents what would have happened anyway (maturation, ordinary teaching, practice effects); the extra 5 points are attributed to the program, assuming parallel trends.

Testing the difference. Each student's change score is computed. The treatment group's gains have SD = 6.0; the comparison group's have SD = 6.5. An independent-samples t-test on the change scores gives:

• Pooled SD = 6.25, standard error of the difference = 1.40
• t(78) = 3.57, p < .001
• 95% CI for the difference in gains: [2.22, 7.78]
• Cohen's d based on the pooled SD of change scores = 0.80

You can reproduce this with the t-test calculator using the summary statistics above.

The same answer from a mixed ANOVA. A 2 (group: treatment, comparison) × 2 (time: pre, post) mixed ANOVA gives a group × time interaction of F(1, 78) = 12.78, which is exactly t² (3.575² ≈ 12.78, using the unrounded t). In a 2 × 2 design, the interaction is the DiD. For background on interactions, see two-way ANOVA; for comparing more than two groups, the ANOVA calculator.

Note the baseline gap: the treatment school started 2 points lower. That is a reason to check whether lower-scoring students simply have more room to grow (a selection-maturation threat). If you have earlier test scores for both schools, plotting them is the best evidence for or against parallel trends.

Choosing an analysis

ANCOVA vs change scores. In a randomized experiment, ANCOVA adjusting for the pretest is usually preferred because it is more precise. In nonequivalent groups, the two can give different answers (Lord's paradox), because they make different assumptions: ANCOVA assumes the groups would have the same posttest given the same pretest, while the change-score approach assumes the groups would have changed by the same amount. Choose based on which assumption fits your setting, state it, and consider reporting both as a sensitivity check.

Propensity scores balance measured covariates only. They do not fix bias from unmeasured differences, which is the main limitation of every non-randomized design.

How to report a quasi-experimental study (APA 7)

State the design name, explain why randomization was not possible, describe how groups were formed, and report baseline equivalence. For nonrandomized evaluations of interventions, the TREND reporting guideline is a useful checklist in health and behavioral research.

Methods section:

Results section:

Discussion: name the threats your design did not rule out, such as history or selection-maturation, and avoid causal language stronger than the design supports.

Common mistakes

• Calling a one-group pretest-posttest study evidence of effectiveness without discussing history, maturation, and regression to the mean.
• Comparing posttests only when a pretest exists.
• Running separate paired t-tests in each group and concluding there is an effect because one is significant and the other is not. Test the difference in change directly.
• Ignoring clustering. If the intervention was assigned to schools or wards, the students or patients are not independent; with only one cluster per arm, the school effect and the program effect cannot be separated, which should be stated as a limitation.
• Not checking parallel trends in a DiD design when earlier data are available.
• Using causal language without qualification. "Was associated with a greater improvement" is often more honest than "caused."

FAQ

Is a quasi-experimental design quantitative or qualitative?

It is a quantitative design. Qualitative data can complement it, for example to understand how an intervention was implemented.

What is the strongest quasi-experimental design?

Regression discontinuity and interrupted time series with a comparison series are generally considered among the strongest, because their assumptions can be partly checked with data. A well-designed difference-in-differences with several pre-intervention periods is also strong.

Can a quasi-experiment show causation?

It can support a causal interpretation when the design rules out the main alternative explanations, but it rests on assumptions that randomization would make unnecessary. Report those assumptions and the evidence for them.

What sample size do I need?

Do a power analysis for the planned test, and account for clustering if the intervention is assigned to groups. See the power analysis guide.

Is a natural experiment the same as a quasi-experiment?

A natural experiment is a type of quasi-experiment in which an external event or rule, rather than the researcher, creates the treatment and comparison groups in a way that is plausibly unrelated to the outcome.

Writing it up

If you have a complete dataset from your study and want a full paper draft with the analysis, tables, figures, and results text, Data2Paper Generate Paper produces one from a CSV or XLSX upload. The first half is free to preview, and the full draft is a one-time $19.90. It is an AI-generated draft, so check the design description, analysis choices, and numbers before you use it.