View: The much talked-about randomised controlled trials are more sinned against than sinning

RCTs, by creating treatment (those who get the development programme) and control groups (those who don’t), create further ethical problems by creating short-term haves and have-nots that could potentially skew observed responses.

BCCL
Mr. Banerjee, Ms. Duflo and Mr. Kremer applied RCT to the field of economics beginning in the 1990s.
By Sujoy Chakravarty

The Nobel Prize in economics this year has received way more attention than most Nobels given in other disciplines. The recipients, Abhijit V Banerjee, his colleague at Massachusetts Institute of Technology (MIT) and partner Esther Duflo, and the purported pioneer of randomised evaluations Michael Kremer from Harvard University, have become the champions of ‘alternative’ development strategy — or agents of the illuminati, depending on which camp you’re in.

But what exactly is this ‘new experiment-based approach [that] has transformed development economics’, that the Nobel committee is referring to? At the heart of the literature on randomised controlled trials (RCTs) is a randomised experiment — selecting two or more groups in a randomised manner from the appropriate population and statistically testing if a certain attribute, intervention or programme alters behavioural or strategic outcomes for the ‘treatment’ group, which is characterised by the attribute or intervention of interest vis-à-vis the ‘control’ group.


The control group is statistically identical to the treatment group, but doesn’t possess this attribute or intervention.

Arbit, Not Fluke
Though field and laboratory experiments to test proclivities or preferences at the individual level such as risk tolerance, time preference and cooperation had been previously done by several experimental social scientists such as Daniel Kahneman, Richard Thaler, Elinor Ostrom and Hans Binswanger, applying the principles of experimentation to specific developmental contexts (health, education, sanitation, governance) and studying the impact of interventions (free mosquito nets, after-school boosts in reading and numeracy, designating minority leadership in rural governance) on outcomes (adoption of best practices in health, ability scores and creation of local public goods) characterise this new alternative approach to determining policy efficacy that was pioneered by several researchers, including this year’s Nobel recipients.

Of course, while typical economic experiments have much more limited goals (to find if people adhere to predictions of economic models of individual behaviour) with much more homogenous populations (like college students), randomised programme evaluation adds significant complexity in sample selection, controls for heterogeneity of individuals and the monitoring of data collection, which is typically an exercise involving thousands of households.
ADVERTISEMENT

This method of evaluation is common in pharmaceutical research and many large pharma companies continue with large randomised interventions to study the effect of newly created drugs. A January 2019 study in the scientific journal Plos One puts the average cost of RCTs approved by ethics committees in Switzerland in 2012 and 2016 at a whopping $72,000 (about Rs 51 lakh).

Designing these uber-interventions, therefore, bumps up the cost of such observational exercises to well beyond research budgets for average development researchers, particularly in developing countries. The proponents aver that this detailed, rigorous measurement is necessary, as without this mega-randomised impact assessment exercise, we would never know the true overall impact of many policy interventions where data on outcomes are not immediately (or ever) observable.

This is particularly true for goals that go beyond measuring typical shortterm economic outcomes (consumption, labour supply) to more behavioural ones (adopting norms and attitudes towards educating girl children, adopting more sustainable agricultural practices, health investments) that often generate significant social welfare in the long term.

However, along with the accolades for RCT have come criticisms that can be grouped largely into three heads: (1) ideological and ethical; (2) methodological issues; and (3) exorbitant costs leading to a bias in disseminated results.
ADVERTISEMENT

In the first group, the main argument is that existing RCTs focus on small micro-level issues that seemingly miss the wood for the trees, largely ignoring the main issues plaguing humanity, such as conspiracies and structural issues with respect to elite capture, primitive accumulation and unequal distribution of wealth. The contention is that the positive economics framework adopted by the proponents stops short of actually assigning responsibility for conditions of deprivation they explore, choosing instead to document and recommend conservative, pragmatic solutions that gloss over these overarching systemic problems.

Casually Causal
ADVERTISEMENT
Furthermore, RCTs, by creating treatment (those who get the development programme) and control groups (those who don’t), create further ethical problems by creating short-term haves and have-nots that could potentially skew observed responses.

The first of these contentions is largely a straw-man argument, as many subfields within economics and social sciences can be pulled up for such lapses, especially if they study aspects of micro-behaviour, local norms and decision-making that don’t directly explore systemic failure at a macro level.

Second, researchers of any repute using the RCT framework do agree that they work in certain contexts and fail spectacularly in others. The scholarly literature on RCTs is full of disclaimers and robustness checks, and rarely makes blanket pronouncements.

If today RCTs are being mindlessly performed by international agencies to study behaviours invasively in inappropriate contexts, it is a travesty to say the least. But it does not negate the randomised experiment as a useful tool if applied with insight.

The second group of critics, that includes eminent economists such as Angus Deaton and Pranab Bardhan, alludes to methodological concerns primarily to do with representativeness of the so-called randomised cohorts, and the external validity of results when the intervention is scaled up in a policy.

Even assuming that the randomisation is mostly effective, its size (1,000 villages or 10,000) will affect the statistical significance of impact metrics. So, for example, deleterious effects of certain variables may not be revealed with 1,000 villages, but can spell disaster for the policy programme if scaled 100 times.

The second aspect is that RCTs can explore causality, but are mostly clueless about the specific process driving the causality. This sometimes creates uninteresting associations that are sufficient but not necessary. Here’s an example offered by Deaton: though the causal statement — ‘The house burned down because the TV set was left on’ — may be correct, but it is by no means interesting or relevant in another context.

Serious as these shortcomings may be, they are present in every data set generated, and in the analysis of secondary empirical data that continues to be the mainstay for empirical development economists the world over.

The third group of criticism really sticks, and largely informs the problems highlighted in the first two groups. Simply put, these interventions just cost too much. Leaving aside ethical issues, many vocal critics would be more comfortable with the RCTs’ shortcomings if the costs involved were not so exorbitant, which could lead indirectly or directly to freeing up public money or research resources to fund a larger set of more diverse research questions.

If RCTs become the gold standard for research in development policy, a potential public bad this could trigger is a small set of results generated only by a handful of individuals who have the resources to perform such evaluations.

If randomisation is, indeed, felt to be an important tool in the study of development, then it is vital that cheaper data collection metrics and procedures are developed in an ethical manner. This would go a long way in democratising randomised experiments and their interpretation.

(The writer is an experimental and behavioural economist, who teaches at Jawaharlal Nehru University, New Delhi)
(Disclaimer: The opinions expressed in this column are that of the writer. The facts and opinions expressed here do not reflect the views of www.economictimes.com.)
Download
The Economic Times Business News App
for the Latest News in Business, Sensex, Stock Market Updates & More.
Download
The Economic Times News App
for Quarterly Results, Latest News in ITR, Business, Share Market, Live Sensex News & More.
READ MORE
ADVERTISEMENT

READ MORE:

LOGIN & CLAIM

50 TIMESPOINTS

More from our Partners

Loading next story
Text Size:AAA
Success
This article has been saved

*

+