When is randomization not possible

The underlying problems are that randomization cannot guarantee equivalence between groups when the sample is small and that this small sample may result in an underpowered test , reducing the ability to detect a true effect.

For example, the U. However, this belief has given rise to the impression that RCTs are the only acceptable and scientifically valid evaluation method for nudge interventions. As the previous example clearly shows, this is not the case. We are not saying that RCTs are without their merits—far from it. As Ronald Fisher, the prominent statistician, argued the number of factors that may differ between two groups is endless, and randomization is the only method that can—under the right conditions—guarantee the intervention is the only systematic difference between the groups.

RCTs are indeed the safest method for establishing cause—effect relationships, and they have often been used to rigorously evaluate nudge interventions, contributing considerably to our knowledge about the effectiveness of different nudges in various contexts. RCTs should be used to evaluate nudge interventions whenever appropriate. However, they are not always appropriate. In some cases they are a not feasible or practical, b considered unethical, and c not free of limitations.

In schools, for example, randomization at an individual level is not usually possible. This is because school interventions often happen in the context of the classroom, where all students are exposed to it.

Other times, schools refuse to apply educational programs unequally to their students. Under such constraints, other research designs are better suited to provide relevant insights. In an RCT, one group receives the intervention while the other does not. This raises ethical issues. For example, exposing only some students to an intervention that helps them create a plan to enroll in college might be perceived as unfair by schools, parents, and students.

While researchers understand that having a control group is the best way to see if the intervention really works, school directors and teachers may not be willing to deny half of their students a potential benefit. Even if the control group will be, in a subsequent phase, the target of the intervention, it still can be considered unfair or unethical. RCTs Have Limitations. As previously mentioned, one of the most important limitations of RCTs is that they are a poor evaluation method when the sample size is small.

Administering a similar nudge in the control condition—one designed not to have an effect, like a message about a different topic—may still have an effect, leading to an underestimation of the size of the effect common in health care settings. Not only do RCT have limitations but nonrandomized designs may be less problematic than they may seem.

The theoretical limitations of nonrandomized designs are not always observed in practice. For example, a common criticism of pretest—posttest design is history —that is, that the results may be explained by an external event that co-occurred with the intervention, and not by the intervention itself. However, this is greatly mitigated in shorter interventions. Other strategies can also be used to overcome the limitations of nonrandomized designs.

For example, showing a difference between a treatment group and several different control groups in a nonrandomized design will increase the confidence that the effect of the treatment is real.

Finally, the authors present Propensity Score Matching PSM , which matches control and treatment groups based on covariates that reflect the potential selection process. The purpose of this paper is to give an introduction of each of the three quasi-experimental designs. For an in-depth discussion on each design, please refer to the included references.

In addition, the authors discuss novel techniques to improve upon these designs. These techniques address the limitations often inherent in quasi-experimental designs. As well, illustrative examples are provided in each section. Regression Point Displacement is a research design applicable in quasi-experimental situations such as pilot studies or exploratory causal inferences. The method of analysis for this design is a special case of linear regression where the post-test of an outcome measure is regressed on to its own pre-test to determine the degree of predictability.

Treatment effectiveness is estimated by comparing a vertical displacement of the treatment unit s on the posttest against the regression trend of the control group Linden et al. If the treatment did have an effect, the treatment group would be significantly displaced from the control group regression line.

In this case, the treatment condition would be evaluated for whether it is statistically different from the control. A regression equation in the form of Linden et al. This effect can be visually observed by plotting a regression line and inspecting whether or not the treatment condition is out of the confidence interval of the trend for the control groups.

First, it requires a minimum of only one treatment unit Trochim, Because of this minimum requirement, however, the data may be highly variable, so it is a good idea to use aggregated units e. Second, this design is applicable in contexts where randomization is not possible, such as pilot studies Linden et al. The effect of the covariates can be interpreted visually by using residual differences between pre and posttests.

By regressing the pretest and the posttest on the covariate, a plot with more than one predictor using the resulting residuals can be created. The residuals of the regression on the covariate should be saved for both pre-test and post-test and used in the regression equation just as before. In this way, the residuals are representative of the pretest and the posttest with the influence of the covariate taken out.

As an example, the regression point displacement design was used to estimate the effect of a behavioraltreatment on twenty-four schools. One of the schools was selected to receive the treatment. The pre and posttest outcomes were operationalized by the number of disciplinary events for their respective years. Figure 1 demonstrates that the treatment school was displaced by disciplinary class removals from the trend - this residual value provides a tangible effect size estimate that has real and direct interpretation.

In other words, this large number can be interpreted as a real difference in removals between the trend of the control schools and the treatment school. The p value indicates that the displacement of the treatment unit was significant.

Table 1. Figure 1: Displacement of the Treatment School x from the control group regression line. Table 1: Regression Model Statistics. Regression point displacement designs also have inherent limitations.

If the treatment unit is not randomly selected, the design will have the same selection bias problems as other non-RCT designs Linden et al. Due to this limitation, it is possible that the treatment unit may not generalize to the population of interest.

On the other hand, the treatment unit can be thoroughly scrutinized prior to treatment. As a result, prior knowledge and prudent selection of the context of the treatment, mitigates these issues particularly in sight of the benefits. The RPD design studies are inexpensive and perfectly suited for exploratory and pilot study frameworks Linden et al.

That is, a single program can be evaluated by selecting a number of control programs and using the RPD design to evaluate the selected unit.

The Regression Discontinuity RD design is a quasi-experimental technique that determines the effectiveness of a treatment based on the linear discontinuity between two groups. The cut point should be a specific value on the assignment variable decided a priori.

Figure 2 illustrates a hypothetical example of an RD design that is depicting the effect of a program intended to increase math test scores. In the RD design, the y- axis represents the outcome variable, in this case math test scores, and the x-axis represents the screening measure. In Figure 2 , the trend for the control group, called the counterfactual regression line shows what the regression line would be if the treatment had no effect.

Figure 2: Hypothetical results of a treatment designed to increase math test scores. The discontinuity in the solid line indicates a treatment effect. The counterfactual line is usually smooth across the cut point, as seen in Figure 2. RD designs have three main limitations. First, RD designs are dependent on statistical modeling assumptions. Participants must be grouped solely by the cut point criterion Trochim, ; Second, it may not be appropriate to extrapolate the results to all the participants as only the scores immediately before and after the cut point are used to calculate the treatment effect.

To remedy these limitations, Wing and Cook propose the addition of a pretest comparison group. The reasoning for using pretest scores is to provide information about the relationship between the cut point and outcome prior to treatment. The first advantage of this approach is that the differences between pre and post measures will give an indication of bias in assignment, thereby attenuating the limitation of controlled assignment.

Second, the treatment effect can be generalized beyond the cut point to include all individuals in the treatment group. This extended generalizability is so because adding a pretest allows for extrapolation beyond the cut point in the posttest period.

Third, the inclusion of the pretest strengthens the predictive power of RD, making it comparable in power to an RCT. The addition of a comparison function gives the RD design all the benefits of an RCT design but is coupled with the dissonance reduction that serving the neediest provides. The pretest RD design equation from Wing and Cook is defined by the following:.

The variable Y 1 it represents the outcome for the treatment group at time t. Conversely, if 0 was in place of 1, it would be the outcome of the untreated group.

Pre it is a dummy variable identifying observations during a pretest period where the treatment has yet to be implemented. An unknown smoothing function is represented by the g A i , and it is assumed to be constant across the pre- and posttest for further discussion of smoothing parameters see Peng, In the original study, disabled Medicaid beneficiaries were randomly assigned to obtain two types of healthcare services to examine the differences on a variety of health, social, and economic outcomes.

In the subsequent analysis, Wing and Cook used baseline age as the assignment variable to reexamine the outcomes in an RD framework. The researchers identified three age cut points i. Additionally, the pretest was used to estimate the average treatment effect for everyone older than the cut point in the pretest RD design.

They found that the prepost RD design leads to unbiased estimates of the treatment effects both at the cut point and beyond the cut point. Also, adding the pretest helped to obtain more precise parameter estimates than traditional posttest-only RD designs.

Therefore, the results from the within-study comparisons showed that the pretest helped to improve the standard RD design method by approximating the same causal estimates of an RCT design. This example demonstrates that the pre-post Regression Discontinuity design is a useful alternative to and can rival the performance of RCT designs.

Propensity score matching attempts to rectify selection bias that can occur when random assignment is not possible by creating two groups that are statistically equivalent based on a set of important characteristics e.

Here, each participant gets a score on their likelihood propensity to be assigned to the treatment group based on the characteristics that drive selection termed, covariates. A treatment participant is matched to a corresponding control participant based on the similarity of their respective propensity score. That is, the control participants included in the analysis are those who match treatment participants on the potential confounding selection variables; in this way, selection bias is controlled.

Before propensity scores can be estimated, the likely selection covariates must be identified. In practice, propensity scores are typically estimated using logistic e. The preferred strategy is to enroll the entire treatment group within a narrow time frame. An alternative option is to have periodic enrollment periods with their respective treatment and control cohorts. The concept proposed in this article is intended to offer a robust alternative to the inadequate strategies currently being used in many health care settings where study findings may not be trusted, and thus decision makers remain uninformed as to whether an initiative is worth continuing or cancelled.