Dutch scientists have a long and respected history of carrying out research into global issues. Much of the academic work that underlies this research.. » read more
Poor people as guinea pigs?
Erwin Bulte has adopted a revolutionary approach for assessing the impact of development interventions through social experiments. These assessments provide governments and NGOs with robust data that they can use to focus their efforts on what really works.
The international community is spending hundreds of millions of dollars on development aid, but the effects of their interventions are seldom measured systematically. Measuring impact is extremely difficult, not least because you generally do not know what would have happened without the particular intervention. Even if you measure a certain change, it is usually impossible to prove a causal relationship with the intervention, as there are so many other variables that could have contributed to the change.
In recent years, development aid organizations have increasingly been expected to demonstrate impact, but have lacked the proper tools and methods to do so in a systematic way. They must have looked with envy at medical researchers. In the medical sciences, it is common to measure the effect of interventions. The introduction of new medicines, for example, involves sophisticated experimental approaches and yields robust results. Either the medicine works, or it doesn’t. The core of the experimental approach is the random selection of a treatment group (those who receive the medical treatment) and a control group (those who do not receive the treatment). Unfortunately it is not possible to apply such an approach to development projects. Or is it?
Measuring impact
Erwin Bulte thinks it is possible. He is involved in a large number of research projects in collaboration with NGOs, governments and multilateral organizations, measuring the impacts of their development interventions on the lives of ordinary people using a truly experimental approach.
Early in his career, Bulte was especially interested in the relationship between natural resources and development (see ‘Resource Curse’ box), but over the years more and more of his findings indicated that local institutions are more important than biophysical circumstances in determining development opportunities. Consequently, he gradually focused his attention on the relation between development processes and the characteristics of local institutions such as the accountability of local governments, the level of tenure security, local norms of cooperation and solidarity, and the quality of regulation and judicial systems. ‘The problem with this type of research, however, is that most of these local institutions are interconnected. It’s like a plate of spaghetti,’ says Bulte. ‘But to understand development, we have to know more precisely which institutions are the crucial ones.’ And that is how Bulte came to the idea of experimental approaches to assess development interventions.
The reasoning is simple. If you are able to measure the impact of development projects that are directed at different local institutions, you will learn a lot about development processes in general. Inspired by the scientific possibilities this approach would offer, Bulte became involved in numerous projects with development organizations. These organizations received him with open arms, as they were looking for better ways to measure the impact of their interventions. And Bulte had ambitious plans. He believed that impact should best be measured in a systematic and controlled manner, and at a sufficiently large scale. Indeed, much like a medical trial.
Bulte adopted an innovative approach that is based on treatment and control groups. He works closely with the intervening organizations and helps them to design their projects as social experiments, i.e. they randomly select a large number of people or villages to receive the intervention and an equally large number as the control group. The intervention could, for example, be the creation of a local peasant organization, the launch of a microcredit project or the introduction of a new seed variety. Bulte and his colleagues then measure the impact of the intervention in both the treatment and control communities using a large range of indicators, including income, health and education levels. They are then able to find out what works in a particular context, for whom, and under what circumstances. Eventually this information enables practitioners to tailor interventions in the future.
Innovation platforms
As an example Bulte mentions the ‘innovation platforms’ being developed by the Forum for Agricultural Research in Africa (FARA). These innovation platforms are established at the village level and are meant to provide a place for all relevant stakeholders in the village – think of farmers, traders, entrepreneurs, village leaders and women’s groups – to come together and discuss their problems, needs and solutions. This discussion is then used as the basis for designing agricultural support and building new local institutions. FARA asked Bulte and his team to help monitor the effects by comparing the developments in villages with and without the innovation platforms. Bulte jumped at the challenge, and is now involved in an experiment to measure the impact of 36 innovation platforms, covering a total of 540 villages and 5400 households in seven African countries.
The first results recently came in and they were quite surprising, says Bulte. ‘I really had not expected to see any significant changes between the control and treatment villages – not this soon at least. But the data clearly showed that poverty levels had gone down in the treatment villages, in a statistically significant manner.’ Thinking about the possible reasons for this success, Bulte realized that the result simply confirmed the pivotal importance of local institutions in determining development processes. ‘After all, these platforms enable people to address exactly those local institutions that they consider to be the main bottlenecks in their particular village, which can be very different from the situation in a neighbouring village.’
Double blind
People usually know when they are in the treatment group in these experiments. But even double-blind experiments are possible. Recently Bulte started a project in the village of Morogoro in Tanzania, where they are experimenting with the double-blind method for the first time. In Morogoro, local farmers grow cowpeas for their own consumption and for sale on the local market. Hoping to increase the farmers’ yields, agricultural specialists developed a new variety of cowpea, which performed significantly better than the traditional variety in the laboratory. But this does not automatically mean that the ‘improved variety’ will perform better under real-life circumstances, and so field tests are needed.
Such agricultural trials are not new, and are fairly straightforward. But the problem is that they do not take a potential placebo effect into account: farmers who know they are part of an agricultural trial may adjust their behaviour. They may, for example, spend more (or less) time weeding and apply more (or less) fertilizer than they would normally have done, or they may plant the improved peas on higher (or lower) quality plots. As a result, analysts will confuse the effect of the improved cowpea variety with the behavioural response of the farmers.
In Morogoro, Bulte and his team therefore designed a doubleblind experiment. In addition to the standard treatment and control groups, a random sample of farmers received common peas and another random sample received the new seed variety. But neither the farmers nor the extension workers who distributed the beans knew which variety they were handling. According to Bulte, double-blind trials are truly innovative when applied in social experiments and allow researchers to identify placebo effects. He expects that their experiment in Tanzania will be the first step towards a major methodological improvement in impact assessments.
Theory and practice
Is it ethical to randomly assign people to control or treatment status? Bulte is prepared for this question, and his answer is clear: ‘Yes. It is the only way to learn. Any intervention includes and excludes people. The only thing we ask is that the people who receive treatment and those who don’t are randomly selected. This actually makes it fairer. And in the end our work will improve the intervention in the future. No costs, just benefits.’
Bulte has been working in developing countries for more than 20 years. ‘I have seen many examples of worthwhile interventions, but I have also seen many projects that were poorly designed and not successful at all. This made me interested in the question of impact. Part of our new research agenda now enables us to explore what factors are conducive to successful interventions.’ For Bulte the main challenge is to form a bridge between hands-on impact assessment work and the more lofty academic theories. ‘I am continuously switching between working on grand academic theories and working with organizations in the field on questions of impact. And the best thing is, they reinforce each other.’
‘Without a good theory about how development works, we will never know how to promote it,’ says Bulte. ‘So, assessing the impact of development aid is crucial not only for improving future interventions, but also for building more robust development theories. And, in the end, nothing is as practical as a good theory.
Download
RESEARCHERS
Erwin BulteSHARE THIS PROJECT
RESEARCH INSTITUTES
- The World Bank
- Forum for Agricultural Research in Africa (FARA)
- ZOA Refugee Care
- Royal Society for the Protection of Birds (RSPB)
RELEVANT ARTICLES
Relevant articles on impact assessment
- Banerjee, A. and E. Duflo, 2009. The Experimental Approach to Development Economics. Annual Review of Economics, Vol. 1: 151-178.
- Duflo, E., R. Glennerster, and M. Kremer, 2007. Using Randomization in Development Economics Research: A Toolkit. In: T. Paul Schultz and John Strauss (eds.) Handbook of Development Economics. Elsevier Science Ltd.: North Holland, Vol. 4, pp. 3895-62.
Some articles by Erwin Bulte
- Brunnschweiler, C.N. and E.H. Bulte, 2008. Are Resource-Rich Countries Cursed? Linking Natural Resources to Slow Growth and More Conflict. Science 320 (May 2, 2008): 616-617.
- Voors, M.J., E.E.M. Nillesen, E.H. Bulte, B.W. Lensink, P. Verwimp and D.P. van Soest, 2011. Violent Conflict and Behavior: a Field Experiment in Burundi. American Economic Review, In Press.
ERWIN BULTE
Erwin Bulte is professor of development economics at Wageningen University, and professor of environmental and natural resource economics at Tilburg University. He is also a research fellow at the University of Oxford Centre for the Analysis of Resource Rich Economies (OxCarre) and at the University of Cambridge Department of Land Economy. Bulte is a member of the board of WOTRO.
DEBUNKING THE RESOURCE CURSE THEORY
Erwin Bulte is known in the academic world for his analysis of the relationship between economic growth and natural resources, with which he overthrew a persistent scientific theory.
According to this theory, known as the natural resource curse, resource abundance leads to conflict and poor economic performance. This idea was widely accepted among academics in the 1990s and the common explanation was that an abundance of natural resources triggers corruption and invites grabbing, or enables undemocratic autocrats to retain their grip on power. This certainly was an appealing idea, with countries like Nigeria and the Democratic Republic of Congo as obvious examples.
Based on a multi-country econometric data analysis, however, Bulte and his colleagues found that natural resource wealth does not lead to conflict or poor development at all. Digging a little deeper, they realized that the statistical analysis underlying the resource curse theory was based on measures of resource dependence (e.g. the extent to which a country is dependent on exports of resources) rather than abundance.
Eventually Bulte and his colleagues proved that the resource curse theory was fundamentally flawed. There certainly is a correlation between dependence on natural resources and poor growth, but the causality is the other way around: countries with bad institutions and conflict attract few investments, and as a result they grow more slowly, and therefore remain dependent on exports of primary commodities.
Text by Koen Kusters
Video by Koen Kusters and Allard Detiger
Comments on this article
Congratulations to NWO/WOTRO on the new website, which brings the findings of research on development attractively in the public domain, thus encouraging the debate. Let me take up the gauntlet and react to the article which I also received in hard copy as a showcase of the new initiative.
‘Poor people as guinea pigs? Assessing the impact of development projects’ provokes me to some critical comments, as the methodology for impact assessment is less simplistic than presented: 1) the experimental method is not as revolutionary as the article claims, and 2) the method is not the egg of Columbus for measuring results of development interventions.
Not revolutionary
For several decades, evaluations of development interventions – of bilateral donors, international agencies and non-governmental organisations alike – have frequently applied a qualitative analysis. A wide range of methods has emerged to deal with the complexity of development processes, including theory-based and participatory techniques. Recently, the qualitative approach to evaluating development aid has come under attack. “When will we ever learn?” was the out-cry from a distinguished group of economists, who criticized the existing portfolio of evaluation studies, commenting that these studies lacked rigour to show evidence of the successes or failures of aid. (Centre for Global Development, 2006, When will we ever learn? Improving lives through impact evaluation, Report of the Evaluation GAP Working group, Washington DC). As a real phoenix, the experimental approaches rose out of the ashes were they had been since the 1970s. Hard and quantifiable results have become the central orientation of aid management and aid evaluations, and under the term ‘impact assessments’ experimental methods are being promoted.
There’s nothing really new. Experimental designs in evaluation are a revival of positivist approaches in social sciences with which evaluation research started in the 1960s in the USA to assess the social programs of the ‘great society’. The founding father of the ‘experimenting society’ Donald Campbell, laid down his basic formula of a treatment and control group with pre-test and post-test in 1963. The logic of experimentation, taken from the natural sciences, was applied to social programs, the main intention being to detect the attribution of a program to the desired outcome. Since these approaches were mainly able to find out whether an intervention was working on the average participant, but failed to provide answers to crucial questions for any social intervention notably why this was or was not the case and under which conditions results were forthcoming, their popularity faded. In the following decades, other approaches took over which were more qualitative and more participatory in nature. And now the pendulum has swung back to experiments, because “the international community is spending hundreds of million of dollars on development aid, but the effects of their interventions are seldom measured systematically”. This sentence in the WOTRO article parrots the CGD report of 2006 and is highly exaggerated. Over the years, a lot of good evaluation studies have been informing about the spending of aid, often in a very systematic and rigorous way. Of course, there have been shortcomings in the evaluation practice as well, but the distinguishing factor between good and bad is not per se the quantitative or qualitative research technique applied. More problematic is it when studies, of whatever kind, leave assumptions untested or jump to unproven conclusions. In this regard, Bulte’s example of the innovation platforms requires more explanation.
Not the egg of Columbus
I welcome the return of experimental methods to the menu of approaches for evaluating development interventions. But let’s treat them for what they are: one among many approaches to be applied when possible and useful. And just as it is short-sighted to go for participatory and process evaluations only, it is equally naive to propagate the quantitative, experimental approaches and more specifically the randomized control trials as THE methodology to measure development results, as the WOTRO article seems to do. Randomized experiments, advocated by Bulte, have their advantages and their limitations, as any other research method. They are not the egg of Columbus for measuring all results of development interventions. There are more reasons for this than the above mentioned inability of experimental assessments to explain why interventions are working or not working.
A major point is that Randomized experiments, or Randomized Control Trials (RCTs), can only be applied in a small number of evaluation cases. Bamberger and White (2007) have calculated that probably not more than some five percent of the total value of development finance is amenable to the RCT approach. Rather than unwillingness or inability to implement such evaluations, as CGD had suggested, the limited applicability is the reason for the small number of experimental evaluations. To make it a bit technical: the use of a random control group is applicable when the intervention is
i) assigned to certain specified units receiving the program while others are not (households, categories of people, villages, etc.),
ii) homogenous and applied in a uniform way (cash transfers to a defined group of poor people are the classical example of experimental evaluations), and
iii) applied in a stable program environment.
However, the practice of development interventions is increasingly complex, with multi-dimensional programs and no standard single input, addressing a broad group of beneficiaries rather than distinguished units. It is often difficult to control the conditions under which the programs operate throughout the program period, especially in the unstable environments of many development programs. For example, institutions that coordinate a program may not function well, the inputs may not be available in time for all kinds of reasons, there may be political pressure to change the group of beneficiaries, or some parts of the control group may become absorbed in the treatment group. As Ravallion (2009) has put it: “Randomization is clearly only feasible for a non-random subset of policies and settings”.
The limitations of the use of randomized experiments has led to the preference for quasi- experimental evaluations where a control group is constructed on the basis of characteristics of the treatment group and data are collected on both the project and a representative control group before and after the intervention. Even more promising are the possibilities of interplay between quantitative and qualitative methods, e.g. by combining them as was done in a recent series of IOB impact evaluations, or by using qualitative analysis to prepare for quantitative research (e.g. the evaluation of social funds described by Carvalho and White 2004).
One of the main messages I aim to get across to the MA students in my evaluation course at the International Institute of Social Studies (ISS) is that methods are not the right starting point of an evaluation, whether they be qualitative, participatory, quantitative or experimental. Evaluation methods have to serve the evaluation purpose. They are to be supportive to generate the knowledge we need, and they should be suitable for the sort and the setting of the program under evaluation. That, in my experience, is more useful than blindly following one particular method. There are more ways to measure impact of development interventions than the impact assessments claimed by the experimentalists.
Ria Brouwers
Senior Lecturer International Development Policy
International Institute of Social Studies
Erasmus University
Reading suggestions:
• Pawson, R. and N. Tilley (1997) ‘Out with the Old – Weaknesses in Experimental Evaluation’, in Realistic Evaluation, pp. 30-54. London: Sage Publications.
• Carvalho, Sonyia, and Howard White (2004), ‘Theory-based evaluation, the Case of Social Funds’, in: American Journal of Evaluation, Vol. 25, No. 2 2004, pp 141-160.
• Bamberger, Michael & Howard White (2007) ‘Using strong evaluation designs in developing countries: Experiences and challenges’, in: Journal of Multi Disciplinary Evaluation, Volume 4, Number 8, October 2007, pp 58-73.
• White, Howard (2008), Of Probits and Participation: The use of Mixed Methods in Quantitative Impact Evaluation, NONIE Working Paper No.7, January 2008.
• Chambers, Robert, Karlan, Dean, Ravallion, Martin Rogers, Patricia (2009) Designing Impact Evaluations: Different perspectives 3ie Working paper 4. http://www.3ieimpact.org/3ie_working_papers
• Ravallion, Martin (2009), Evaluation in the Practice of Development, International Bank for Reconstruction and Development / The World Bank, Published by Oxford University Press.
• Deaton, Angus S. (2009), Instruments of Development, Randomization in the tropics, and the search for the elusive keys to economic development, NBER Working Paper Series nr 14690, National Bureau of Economic Research, Cambridge MA.
I would like to provide a brief response to the comments and questions by Jur Schuurman and Ria Brouwers. First, Jur Schuurman wonders about a potential asymmetry between the number of treated and control villages. I would like to apologise for the confusion. The solution to the riddle is that each innovation platform serves multiple villages. So, the number of platforms equals 36, the number of treated villages equals 540, and the number of control villages is equally large (also 540).
Ria Brouwers argues that the experimental approach to impact assessment is neither revolutionary nor without problems. I fully agree. We have encountered many challenges during our attempts to measure impact. There are ethical questions, statistical issues (e.g., spill-over effects, non-compliance), concerns about external validity of the findings, etc. etc. However, in spite of the fact that it is difficult to properly organise meaningful randomised control trials for certain interventions (and impossible for others), I do believe this approach holds great promise for the future. It remains difficult to create a “credible counterfactual” otherwise, so my motto would be to “randomise treatment if you can (and if it is cost-effective).” But even if randomisation is feasible, experiments do not provide one-stop shopping for results. I support the plea for combining quantitative and qualitative approaches to measuring impact. One reason is that we are not only interested in whether there is impact (an average treatment effect). Ideally, we also want to know why and how certain interventions work (“understanding the mechanism”), and for whom they work. Another reason is that ultimately we are much more interested in welfare effects than in treatment effects. This requires a need to collect additional information, and a need for mixing methods.
Posted: 29 augustus 2011 om 20:30 by Jur Schuurman says:
Very interesting stuff! There is just one thing that is not clear, at least not to me. After explaining that the innovation platforms (the ‘treatment group’) are established at the village level, the author goes on to say that the experiment studies the impact of 36 such platforms, and covers a total of 540 villages. Given the above, this seems to mean that the treatment group consists of 36 villages, and the control group of the rest (504 villages). Is that not a bit asymmetrical?
I look forward to your reply!