In Praise of Results Claire Melamed

January 31, 2011 | More on Economics and development | 10 comments

There’s much anxiety in development-land these days. New, frightening beasts like ‘results’ and ‘value for money’ are stalking the defenceless and helpless herds of ‘empowerment’ and ‘rights’. With their new slashing, tearing, efficiency-driven ways, the fear is these interlopers will exterminate the shyer folk and irretrievably warp the development process.

Actually, scrap the analogy. Firstly, it’s possibly a bit overwrought, and secondly, it’s just not true. It is absolutely wrong and quite dangerous to equate a concern for results with the view that development is just about handing out food parcels.

If you think results don’t matter, then consider the alternative. Without measuring results, we would never know if aid money made any difference. We would have no way of reporting back to communities about what governments were doing. We would have no way of judging between different claims on scarce development budgets, other than the whims of aid administrators and the vagaries of development fashion. And we would never, ever, know if things were getting better or worse.

It seems extraordinary to me that anyone could object to having more information about whether a policy change or a development project has an impact. More information can only help the people who really should be driving development – poor people themselves – to make choices and take control of what is done in their name.

This is not to say that what is counted doesn’t matter. A concern for results shouldn’t mean that we just reach for the nearest indicator and call it development. Quite rightly, there are many voices out there pointing out that development is a complex social and political process, involving power, rights and justice – all concepts that are difficult to capture on a spreadsheet. And, yes, there is a danger that pursuit of ‘results’, narrowly defined, could steer development projects and policies towards what is best counted, not what is best.

Now that is the argument worth having. What should we count? Those who are concerned with empowerment, rights and other such slippery but absolutely essential ideas should be rushing to the statisticians, and calling out for better ways of measuring if what they are doing in the name of empowerment and rights is actually working. Yes, as the saying goes, ‘not everything that can be counted, counts’, but ‘not everything that counts can be counted’ just isn’t good enough. We just ask people to take it on trust that things are working because the professionals say so? I think not.

Don’t get mad, get counting. The results agenda is completely compatible with a view that people are the agents of development and that it is their experiences and relationships that define progress. There are ways of measuring how people feel about things – in the UK, for example, we have things like the British Social Attitudes survey which asks people what they think and then reports it, and measures changes in things like attitudes to inequality, feelings about old age or views about the effectiveness of education policy over time. And we know that there are many ways of involving poor people in defining indicators and collecting numbers.

It could be so exciting. Let’s ask poor people what they call results, and what they consider the best value for money? Surely, surely you want to know?

10 comments »

Dave Algoso on January 31st, 2011 at 1:57 pm:

I love the push for measuring results. You're right that those dealing with the more "slippery" issues should get on board with this wave rather than try to stand in its way. But I disagree that measuring results always means counting something. You imply that blind faith is the only alternative to quantitative measures for describing change and causality. I don't think that's true. Quantitative measures have to be paired with qualitative descriptions in order to truly understand what took place. This is the same way historians use data combined with descriptions of events to create understandable narratives of human society. A reliance on one or the other tells an incomplete story.

To your last point: What if a participatory evaluation process yielded primarily qualitative measures, such as stories of significant changes in the community? If the intended beneficiaries of a program felt this more accurately portrayed a program's impact than numerical survey results, would we tell them they're wrong?
c ellis on February 3rd, 2011 at 9:50 am:

what i would like to know is people's understanding of why it is that they are poor. of course, some may not be able to articulate why it is that they are poor. after all, it's potentially quite emotionally painful to examine such things. some may not wish to examine such facts because it is too upsetting to their equilibrium.

i would suggest some have simply accepted their lot: growing up as a child requires a lot of trust, indeed, isn't a trustful child a sign of a 'good' upbringing? the natural progression of this emotional mindset is to extend the behaviours you learn in your family to the wider community, ie, those of trust. imagine then, such people's bewilderment when they find that people in the wider community do not necessarily care about their wellbeing, indeed, on the contrary, they actively exploit such innocence for their own financial gain.
c ellis on February 3rd, 2011 at 9:51 am:

continued/…

to my mind, it is the dissonance between the domestic and the wider national culture in a country which may result in poverty: ie: in the family, members are socialised to be caring, kind, non-competitive, honest, obedient, loyal and co-operative and then, when it is time for them to enter into the bigger community, completely different values are called for to survive, such as competitiveness, disingenuity, sleuth, dominating and entitlement. the incongruence is too great, confusing, bewildering and contrasting with what you have learned at home. what do you do? you stick with your original values and you resign yourself to whatever may happen, because you couldn't live with yourself if you changed those values to the ones predominant in the greater society in which you live, as it would make you feel alienated from yourself and your past and betray those who have cared for you up until now.

so, as well as concentrating on the traditional "power, rights and justice", i think it might be good to psychologically assess poor people's feelings of entitlement to a share of basic resources.
c ellis on February 3rd, 2011 at 9:52 am:

continued/…

but then, what do you do with this information? do you instigate programmes to make people feel more entitled? would such programmes work, given that their content contradicts the early 'acculturisation' this hypothetical person may have received? and, of course, such an enterprise could only be one prong in an holistic intervention to alleviate poverty.

but, given that resources for such endeavours are also limited, perhaps this information could be presented to governments in a persuasive way, to encourage them to support their poor in the best ways possible, now that they had a good idea of the general psychological characteristics of the poor.
c ellis on February 3rd, 2011 at 9:55 am:

nb: the middle bit of this comment has been lost
Simon Maxwell on February 9th, 2011 at 7:30 am:

Aid effectiveness is the theme of the new Independent Commission on Aid Impact, whose membership was announced just the other day, and which will be accountable to parliament through the IDC. As I discussed in a previous contribution on my own blog (www.simonmaxwell.eu), the political imperative behind a focus on results is obvious, and right, but the technicalities are complicated. Aid has complex and often indirect effects, for example in the field of governance. The macro-economics are also more difficult than simply drawing a straight line between a gift of cash in the UK and the building, say, of a primary school in Tanzania: sometimes, for example, aid is used, legitimately, to bolster foreign exchange reserves or pay down internal debt. Fungibility is also an issue: was the Government of Tanzania planning to build a primary school anyway, and does the aid therefore allow some other expenditure to take place?
There is continual pressure in the aid business to deliver measurable short-term results, with much current enthusiasm for randomised controlled trials. Claire has made a spirited contribution to this debate, favouring a results-based approach even for social and political interventions. The new Independent Commission on Aid Impact will succeed if it can be equally determined to understand the impact of aid, but subtle enough to recognise that different kinds of aid work in different ways, and that all have multi-faceted impacts on the politics and institutions which drive development.
Tim on February 15th, 2011 at 11:43 am:

Yes, at a fundamental level it is nonsensical to be in favour of development but opposed to a focus on results. But in practice, the results focus tends to collapse rapidly into a push for numbers. I like numbers; I like what they can tell us; I think there is lots of scope for the donor community to get better at measuring. But anyone with a little statistical or management literacy also knows how *not* to use numbers. Some things are genuinely hard to measure: if we devise a proxy or composite and then treat this as a 'hard' number rather than a best-possible, will-do-for-now-as-a-rough-guide, internal estimate, then we risk pervert ing a results focus and misleading the public.
Tim on February 15th, 2011 at 11:44 am:

By way of illustration: I was talking with someone who works for an Indian NGO. In a meeting with a donor about a proposed project for reducing domestic violence, she was told they had to come up with a cost-benefit / value for money (VfM) estimate of how much it would cost to stop one woman being battered; and if they said this unit cost was (say) £1.17 and a different country office said they could do it for £1.03, then funds would go to the other country. This is nonsense, and it makes the donor look foolish. Such a figure could only ever be a very rough guess; different offices will arrive at it in different ways, with different available data; and the incentives are clearly skewed to revise the estimate down.
Tim on February 15th, 2011 at 11:48 am:

One other thought: if donors are serious about measuring, then we need to get much more serious about investing in data generation, including not only donors’ own monitoring and evaluation but also basic, fundamental national statistics. This is a public good which would have a significant effect on the effectiveness of development activities. It is necessary if we want governments and citizens in the countries in which we work to have any hope for evidence-based, results-focused policy making; but it is also, ultimately, necessary if we want to benchmark our impact against background change and other interventions, or look at how different problems and interventions interact. So yes, highly controlled randomised control trials of a nicely bounded, discrete intervention are great, but extremely expensive and have their limitations: please can we also have more money for vital registration of births and deaths; building the capacity of national statistical organisations; better, more consistent and more frequent household surveys; introducing ICT innovations to get us closer to real-time data; improving line ministry management information systems; and supporting independent think tanks in our partner countries.
Tim on February 15th, 2011 at 11:49 am:

So while I’d agree with the thrust of Claire’s argument – the push for results is essential – I do think that there are real dangers that the principle becomes warped in practice. Unrealistic expectations of donors’ ability to measure and attribute, magnified through extreme simplification for the purposes of public communication, entails some serious risks. To paraphrase Reinhold Niebuhr, what we need now is the courage to measure what we can, the serenity to accept there are some thing we cannot measure (or can measure only imperfectly), and – critically – the wisdom to know the difference. Right now, I’m not sure this wisdom is guaranteed.

10 comments »

About Global Dashboard

Follow us

Authors