Experiments
The principles of experimental design
Causation has a tricky relationship with data. We can’t observe the precise counterfactual that would allow us to identify an individual cause. Instead, statisticians use a host of methods to find a good approximation of the counterfactual. In the last set of notes, the methods focused on matching each unit with a near-twin; another unit that is similar in every respect except for the variable of interest. In these notes, we consider what can be gained if we zoom out and focus on causation at the group-level.
The distinction between individual and group level causation is demonstrated in following two statements.
- Evelyn got a good job because she graduated from Cal.
- Graduating from Cal helps people get a good job.
The first is a strong statement about a single individual, Evelyn. The second is a much more general statement that compares people who have have graduated from Cal with a counterfactual group who has not.
Group-level causation is the focus of many sciences, which aim to make general claims about the causal mechanisms of the world. The goal is to estimate the average treatment effect.
- Average Treatment Effect
- A population parameter that captures the effect of being administered a particular treatment, averaged over each group. Most often estimated by a difference in sample means or a difference in sample proportions.
In statement two above, a natural estimate for the average treatment effect would be the difference between the proportion of Cal graduates who got a good job and the proportion of non-Cal graduates who got a good job. That can be visualized in a simple example of three students who graduated from Cal and three who did not. The difference in the proportion with a good job is \(2/3 - 1/3 = 1/3\).
| Name | Cal Grad | Good Job |
|---|---|---|
| Evelyn | TRUE | TRUE |
| Grace | TRUE | FALSE |
| Juan | TRUE | TRUE |
| Alex | FALSE | FALSE |
| Monica | FALSE | FALSE |
| Sriya | FALSE | TRUE |
| Cal Grad | P(Good Job) |
|---|---|
| TRUE | 0.67 |
| FALSE | 0.33 |
By broadening our focus to the average treatment effect across groups, we’re able to bring to bear one most convincing forms of evidence of a causal link: data from experiments.
Principles of Experimental Design
An experiment is generally characterized as being a setting where the researcher actively assigns subjects or units to one particular condition or another. The most potent design of an experiment to determine whether one variable, the treatment, affects the outcome at the group level is the aptly named Randomized Controlled Trial (RCT).
As a running example, consider an immediately relevant question: will reading these course notes as a pdf that you annotate result in a deeper understanding and a correspondingly better score on the final exam? Let’s run through each term one-by-one to think through how to design an RCT and effectively answers this question.
Control
- Control
- (noun) A second condition to which a subject or unit could be assigned that draws a contrast with the treatment condition.
-
(verb) Actions taken to eliminate other possible causal factors and ensure that the only difference between the treatment and control group is the factor of interest.
When designing an RCT, an essential decision is the nature of your control group. Our research question is: will reading these notes as a pdf result in a deeper understanding? Deeper . . . than what? Deeper than if you didn’t read the notes at all? Deeper than if you read them aloud?
If we’re most interested in any difference between reading a pdf and reading a webpage, we would declare those who read the pdf part of the treatment group and those who read the webpage as part of the control group. Small changes to the control group can make an important difference in the precise question that you’ll be answering.
We also speak of “controlling for” other variables that might provide alternate explanations for any effect that is found. Imagine that the pdfs were consistently posted 3 hours later than the website (😬). To rule timing out as a potential cause, we would want to control for time by ensuring that the two versions were released at the same time.
Random Assignment
- Random Assignment
- The process of assigning subjects to either the treatment or control group in a random fashion, where they’re equally likely to be assigned to each group.
Because we are conducting an experiment, we are intervening in the process and directly assigning subjects to either the treatment (pdf readers) and control (website readers). There are many ways we could do this. The morning sections could all be assigned the pdf and the afternoon sections the website. Or the instructors could choose to exclusively assign either the pdf or the website version of the notes to their individual sections.
The problem with both of these approaches is that our two groups might differ on more characteristics than just their reading format. The morning sections perhaps have students who are early risers and more conscientious students. Or perhaps the instructors who choose the website are more tech-savvy and more effective at teaching computing. In both cases, we invite the skeptical observer to question whether it was truly the medium of the notes that led to a difference in course grades or if it was something else.
The mechanism of random assignment is brilliant because it can snuff out every such skeptical inquiry in one fell swoop. If instead we assigned each student to read the pdf or the website totally at random, every other possible characteristic between these groups should, on average, be balanced. If students are assigned at random, we’d expect an equal number of early-risers to be in both the pdf and the website group. We’d also expect the students with the more effective instructors to be evenly represented in both groups. And so on and so forth, for every possible other characteristic that might muddy the waters of our causal claim.
Replication
- Replication
- The practice of assigning multiple subjects to both the treatment and control group. Also, the practice of repeating an experiment a second time to see if the result is consistent.
The careful reader will have noted a weakness in the brilliance of the random assignment mechanism for balancing the characteristics between the groups. What if, purely due to bad luck, we happen to randomly assign all of the early-risers to the pdf group and all of the late-risers to the website group? That would indeed bring us back to the problem of there being many ways in which our treatment group is different from our control group.
The random assignment mechanism will balance out all possible confounding factors on average, but for a given experiment that is not guaranteed. However, it becomes much more likely if we have a large sample size. If you just have four students total in the class, two of whom are early risers, it’s quite easy for both of them to end up in the pdf group if they were assigned at random. If instead you have 800 students, 400 of whom are early risers, it’s very very unlikely that all 400 will have made their way into the pdf group.
Other considerations
Randomized controlled trials have long been considered the gold standard for establishing a group-level causal claim, but care must be taken to ensure that your result means what you think it means. Here we reconsider a study where a new drug was used to treat heart attack patients. In particular, researchers wanted to know if the drug reduced deaths in patients.
These researchers designed a randomized control trial because they wanted to draw causal conclusions about the drug’s effect. Study volunteers were randomly assigned to one of two study groups. One group, the treatment group, received the drug. The control group did not receive any drug treatment.
Put yourself in the place of a person in the study. If you are in the treatment group, you are given a fancy new drug that you anticipate will help you. If you are in the control group, you are not treated at all but instead sit by idly, knowing you are missing out on potentially life-saving treatment. These perspectives suggest there are actually two effects in this study: the one of interest is the effectiveness of the drug, and the second is the emotional effect of (not) taking the drug, which is difficult to quantify.
In order to control for the emotional effect of taking a drug, the researchers decide to hide from patients which group they are in. When researchers keep the patients uninformed about their treatment, the study is said to be blind. But there is one problem: if a patient does not receive a treatment, they will know they’re in the control group. A solution to this problem is to give a fake treatment to patients in the control group. This is called a placebo, and an effective placebo is the key to making a study truly blind. A classic example of a placebo is a sugar pill that is made to look like the actual treatment pill. However, offering such a fake treatment may not be ethical in certain experiments. For example, in medical experiments, typically the control group must get the current standard of care. Oftentimes, a placebo results in a slight but real improvement in patients. This effect has been dubbed the placebo effect.
The patients are not the only ones who should be blinded: doctors and researchers can also unintentionally affect the outcome. When a doctor knows a patient has been given the real treatment, they might inadvertently give that patient more attention or care than a patient that they know is on the placebo. To guard against this, which again has been found to have a measurable effect in some instances, most modern studies employ a double-blind setup where doctors or researchers who interact with patients are, just like the patients, unaware of who is or is not receiving the treatment.
Summary
You may heard the phrase “Correlation does not imply causation … unless it’s seen in data from a randomized control trial.” While this statement is misleading - there is a rich set of methods that, when used correctly, can make compelling causal claims from correlations found in data - it does highlight the particular strength of RCTs. RCTs are able to isolate the effect of interest by creating a carefully selected control group and then assigning subjects to groups at random. When the number of replicates is large enough, we can be confident that our control group on average serves as a close counterfactual to our treatment group.