While Randomized Control Trials (RCTs) have become the dark horse of contemporary development microeconomics, they have engendered controversy. This article sifts through the controversy and concludes on how RCTs can continue to advance understanding of real-world problems.
New Gold Standard: Abhijit Banerjee and Esther Duflo in their book titled ‘Poor Economics’ (2011) use RCTs and data sets from 18 countries to measure the effectiveness of specific program interventions designed for the poor. Their ‘new way of doing economics‟ explains how poor live, how specific institutions function and what policies work or do not work, for e.g, deworming pills bring short-run increases in school attendance and positive effects on earnings over time in Kenya. They conclude that relatively straightforward but intelligently designed interventions do significantly help the poor. Secondly, in their assessment of research and field experience to understand the behavior of the poor and their fundamental problems; they attribute failures to three Is – ideology, ignorance, inertia. They advocate helping the poor via transfers, behavioral nudges and subventions, rather than poverty reduction through economic growth.
On the basis of their research, they prescribe five lessons to reshape institutions, reorder politics, change lives: i) Reliable public information can nudge poor people towards education etc. ii) Easy availability of goods e.g. chlorine dispensers can reduce the poor man’s burden of taking decisions. This is consistent with the success of India’s Jan Dhan Yojana in bringing about financial inclusion. iii) Missing or unfavourable markets can be developed by technological innovations and government support. Their prescription finds resonance in India where ‘Aadhaar’ a unique ID serves as a primary key for receiving entitlements. iv) It is possible to improve governance and policy with existing social and political structures. v) Changing expectations about success can begin a virtuous cycle of improvements.
Randomized control trials have come to be regarded as the new ‘gold standard’ in development economics. A few illustrations of randomized control experiments carried out to test the impact of policy interventions are summarized below:
- A randomized experiment in Dominican Republic found that providing families with information on the actual returns to education could change their schooling decisions. Boys were provided information on the actual returns to education in ‘treatment’ schools. As a result, boys in the treatment group attended 0.2 additional years of schooling over the next four years relative to their counterparts in the control group. This relatively inexpensive one-time intervention proved to be a very cost-effective way of increasing students' time in school (Jensen 2010).
- A randomized evaluation suggested that providing official textbooks to randomly selected rural Kenyan primary school students did not increase test scores of the average student. Textbooks increased scores for students with initial high academic achievement and increased the probability that the students who had made it to the selective final year of primary school would go on to secondary school. Students with weaker academic backgrounds did not benefit from the textbooks at all. The study confirmed the view that the Kenyan education system and curricular materials were oriented to academically strongest students rather than to typical students (Glewwe, 2009).
- Kyle Emerick et al (2013) used a randomized field experiment in 128 villages of Orissa to show that Swarna-Sub1, a submergence-tolerant rice variety, had significant positive impacts on rice yield when fields were submerged for 7 to 14 days with no yield penalty without flooding. It was estimated that Swarna-Sub1 offered an approximate 45% increase in yields over the current popular variety when fields were submerged for 10 days. Additionally low-lying areas prone to flooding tended to be more heavily occupied by people belonging to lower caste social groups. Thus, a policy relevant implication of their findings was that flood-tolerant rice can deliver both efficiency gains, through reduced yield variability and higher expected yield, and equity gains in disproportionately benefiting the most marginal group of farmers.
- Using a dataset collected in 265 Panchayats in West Bengal and Rajasthan, it was found that women elected in reserved Panchayats invested in public goods more closely linked to women’s concerns such as drinking water and roads in West Bengal and drinking water in Rajasthan. They invested less in public goods that are more closely linked to men’s concerns viz. education in West Bengal and roads in Rajasthan. This study by Chattopadhyay and Duflo (2004) suggested that direct manipulation of the identity of the policymaker can have important effects on policy.
- Mckenzie (2016) in an experiment on New Zealand visas assigned by lottery found that randomly selected workers moving from a poorer country to a richer country appear to be immediately more productive and that the observed wage gains are stable over time. The experiment supported the view that cross-country wage differences are due to better institutions, higher quality capital and other factors in rich countries that serve to raise the productivity of all workers, rather than attributes that are embedded in native workers and take time for migrants to accumulate.
- In an experiment with lottery used for Hajj participants, it was found that successful pilgrims showed increased belief in equality and harmony among ethnic groups and Islamic sects. They also developed more favourable attitudes towards women, including greater acceptance of female education and employment. Increased unity within the Islamic world was not accompanied by antipathy towards non-Muslims. These positive changes occurred due to increased exposure and interaction with Hajjis from around the world (Chingingsmith, et al, August 2009).
- Lai et al (2008) estimated that children’s school outcomes were negatively affected by the poor choices made by parents during the school selection process. Children of parents who made judgment errors in school selection were admitted to lower quality schools and achieved lower test scores in Beijing’s High School Entrance Examination. Parents who had less education, whose children performed at lower levels in primary school, and who were less attentive to teachers’ opinions about schools were more prone to make these errors. They concluded that providing assistance to parents, especially those less prepared to make informed choices about school selection, was important for supporting more efficient and equitable open enrolment programs.
- In Colombia, Angrist (2002) used an experiment to study the impact of one of the largest school voucher programs, the Programa de Ampliación de Cobertura de la Educación Secundaria (PACES). Vouchers partially covering the cost of private secondary school were randomly distributed. Three years after the lotteries, winners had completed 0.1 more years of schooling, were about 10 % more likely to have finished 8th grade, primarily because they were less likely to repeat grades, and scored 0.2 standard deviations higher on achievement tests. In the longer-run, the PACES program increased secondary school completion rates by 15-20%.
These few illustrative studies support the view that RCTs offer simple ways of testing the success and limits of new policy initiatives and further sharpening policy interventions.
Criticism: In his critique of RCTs, Martin Ravallion (2012) argues that RCTs tall claim of identification of causal effects with fewer assumptions than econometric methods breaks down. He points out that the exclusion restriction ceases to hold. Further, RCTs only indicate the mean impact on the treated group and say nothing about distribution of the impact. He adds that factors such as external validity, compliance, interference, spillovers, and ethical considerations reduce its utility. He is concerned that questions such as why intervention did not have an impact or whether impact would be replicated in a different setting or how certain countries have succeeded in reducing poverty, remain unanswered. He doubts efficacy of RCTs in explaining poverty traps which need big development push or where market failures are a concern. Further RCTs apply selectively to policies and settings and their extrapolation beyond experimental environments is questionable. In fact he argues that what may work in a pilot may encounter governance issues when scaled up. He cautions against drawing broad lessons from casual observations from field work which fail to meet “standards of rigorous qualitative and mixed method work in social sciences‟.
Mark Rosenzweig (2012) criticizes the approach for overselling the suggested remedies as the detected absolute gains are small and the benchmark is “not level of living of non-poor but typical level of relevant outcome variable in poor population”. Small absolute gains on small baseline levels produce large percentage gains, but cannot bridge the poor-non poor gap over time. He highlights that reporting increases in percentages may be suitable for income, but using them in schooling given the log-linear relationship between wages and schooling, potentially “obscures what is really at stake”. He argues that the studies do not impute family labor in family labor enterprises, use proxies for earnings, ignore externalities and suffer from selectivity bias. The reported 8% return on schooling in INPRES reduces to a mere 3.2% with inclusion of self-employed in the sample. He cautions against extrapolating program effects based on experiments for a small segment of the population He states that the authors disregard that Governments in developing countries may lack the resources to subsidize all the activities as cost of implementation from ‘thinking small‟ can be large. Tweaks at macro level may be cheaper. He thinks that “thinking big and focusing on marginally improving the lives are ultimately complementary. Other criticisms of RCTs are: that they expensive; are hampered by program selection; lack theoretical deduction; give misleading data to econometric studies ; suffer from Hawthorne effect; cannot achieve perfect randomization; have limitations with conceptions of causation, challenges of implementation, internal validity etc.
Way forward: RCTs serve as a handy tool to test theories, recognize patterns, and pre-test new institutions or mechanisms. They can tailor treatment to questions and measure program effectiveness on a small scale prior to large-scale implementation at affordable costs. This is helpful to flesh out the strengths and limitation s of strategies for poverty alleviation. To increase their efficacy, it becomes important to control implementer effects, have deduced hypothesis guiding treatment and examine distributional differences with non-parametric tests. Ravallion (2016) emphasizes testing that treatment and control groups are balanced in terms of relevant covariates. More thoughtful use of non-experimental findings to structure the behavioral models could underpin good RCT design. RCTs can be complemented by qualitative appraisal and baseline surveys. Relevance to policy making can be enhanced by keeping the experimental pilot close to the scaled up program, spanning all the relevant dimensions of variation in impact by scale and context, adopting structural parameters with best of methods and using them to simulate alternative designs. External validity can be assured by replications with different settings, estimating behavioral parameters, exploiting heterogeneity in impact and experimentation across several contexts. Scholars advocate principal agent or mechanism design approach to improve external validity. RCTs can inform better design and implementation of critical policy interventions in conjunction to achieve political sustainability and consistency and coherence with the overall effort. Validity of different quantitative methods depends on program implementation and type of data. Development challenges require healthy methodological pluralism for success. RCTs offer one scientific approach to give context with limited control.
1. Banerjee, Abhijit V and Esther Duflo (2011), ‘Poor Economics – A radical rethinking of the way to fight global poverty’, Public Affairs, April 26, 2011.
2. Ravallion Martin (2012), ‘Fighting Poverty One Experiment at a Time: Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty: Review Essay’, Journal of Economic Literature Vol. 50, No. 1, March 2012 (pp. 103-14).
*These views are personal and do not represent the views of NITI Aayog.