When the Abdul Latif Jameel Poverty Action Lab (J-PAL) was established in 2003, very few randomized control trials (RCTs) had previously been conducted to answer social policy questions. RCTs had primarily been used as a means to conduct clinical trials.
This past weekend, J-PAL celebrated their 10th anniversary. Over the course of 10 years, they have conducted 445 ongoing or completed RCTs in the areas of health, education, agriculture, environment and energy, political economy and governance, finance and labor markets alongside several partners. The organization has created strong partnerships with governments and NGOs, and 91 affiliated professors, and has improved policy for 165 million people.
The debate on the utility of RCTs has been prominent and widespread. I find that most people who are strongly opposed to RCTs have the expectation that these trials should be the magic bullet to development. "...a common misperception directed at advocates of RCTs is that they can and should be conducted on every program..." said Dean Karlan, founder of Innovations Poverty Action (IPA), J-PAL partner.
I'm a proponent of RCTs used in this manner particularly because they give us an evidence base of both information gathered from the end user and observed information. However, I think there are several limitations to RCTs that J-PAL and other entities have strongly articulated. I would like to separate actual limitations from perceived limitations. Below, I will briefly explore three perceived limitations of RCTs while highlighting some interesting points made during the J-PAL@TEN conference.
RCTs are resource intensive: This is an often cited limitation by opponents of RCTs. I would ask relative to what? One RCT can cost anywhere from $50,000 to $500,000 according to GiveWell's 2011 evaluation of IPA. An RCT can take anywhere from a few months to several years to complete depending on the nature of the RCT. On the other hand, the U.S. government spends approximately $49 billion on foreign aid in one fiscal year. Most foreign aid spending is often invested in areas that have no proven impact. I would argue that spending millions of dollars over the course of each year (IPA spent $27.85 million on research in 2012) to develop an evidence base is well worth the investment if foreign aid dollars will subsequently be redirected to proven interventions. As Esther Duflo, co-founder of J-PAL, said, conducting these RCTs are like chipping away until you find the lever. It takes a lot of resources, but the payoffs are tremendous as well.
RCTs aren't externally valid: External validity is the effectiveness of an intervention in contexts outside of that in which the RCT was conducted. In an RCT, external validity is dependent on the environment that the RCT was conducted including the beneficiaries, the service providers, and the feasibility of providing the intervention.
Although individual RCTs are context specific, J-PAL and partners have tested some theories using multiple RCTs in varying contexts to build a more comprehensive picture. As Annie Duflo, Executive Director of IPA highlighted at the conference, RCTs were conducted in Kenya, Ghana and India yielding empirical evidence that identifying and focusing instruction on actual learning levels of students is a cost-effective approach in remedial education. This approach is contrary to the traditional approach of providing more inputs - teachers, textbooks, etc. One method of doing this is to split the class by achievement levels.
When asked about his vision for J-PAL in the next 10 years, Abhijit Banerjee, co-founder of J-PAL, specifically addressed the issue of external validity. The organization started out conducting RCTs with extremely small sample sizes, and have been able to learn and grow. He hopes to see affiliates conduct RCTs with even larger sample sizes which will allow for greater external validity.
RCTs do not yield action: Gone are the days when RCTs are conducted in a vacuum. J-PAL's strong partnerships such as that with Pratham in India, allow implementing organizations to be closely involved in the design of the study. As Rukmini Banerji, Director-Programs at Pratham stated, the partnership with J-PAL has not only helped Pratham to understand the results and act upon them, but it has also created a culture of curiosity within the organization. Pratham, like many other J-PAL partners, is an equal partner in the study design process.