New Republic: Stop Trying to Save the World: Big ideas are destroying international development, Michael Hobbes
One of the first classes I took in the economic development program at Eastern University was a class where we spent the entire semester studying the wide variety of economic development models that had been tried. Few worked. The best attempts led to very modest improvements. The worst had perverse unintended consequences. The overall message? Economic development is hard to do well!
I missed this article by Michael Hobbes from New Republic last November. It is a 6,000+ word essay, but it is one of the best reads I have seen on the need for careful ongoing assessment, and it is a warning of the inefficient - even perverse - consequences when we do not empirically test our assumptions. Here are some key excerpts.
Maybe the problem isn’t that international development doesn’t work. It’s that it can’t.
He points to these examples:
In the late ’90s, Michael Kremer, then an economics professor at MIT, was in Kenya working on an NGO project that distributed textbooks to schools in poor rural districts. Around that time, the ratio of children to textbooks in Kenya was 17 to 1. The intervention seemed obvious: Poor villages need textbooks, rich donors have the money to buy them. All we have to do is link them up.
But in the early stages of the project, Kremer convinced the researchers to do it differently. He wanted to know whether giving kids textbooks actually made them better students. So instead of handing out books and making a simple before-and-after comparison, he designed the project like a pharmaceutical trial. He split the schools into groups, gave some of them the “treatment” (i.e., textbooks) and the others nothing. Then he tested everyone, not just the kids who got the books but also the kids who didn’t, to see if his intervention had any effect.
It didn’t. The trial took four years, but it was conclusive: Some of the kids improved academically over that time and some got worse, but the treatment group wasn’t any better off than the control.
Then Kremer tried something else. Maybe the kids weren’t struggling in school because of what was going on in the classroom, but because of what was going on outside of it. So again, Kremer split the schools into groups and spent three years testing and measuring them. This time, the treatment was an actual treatment—medication to eradicate stomach worms. Worm infections affect up to 600 million children around the world, sapping their nutrition and causing, among other things, anemia, stomachaches, and stunting.
Once more, the results were conclusive: The deworming pills made the kids noticeably better off. Absence rates fell by 25 percent, the kids got taller, even their friends and families got healthier. By interrupting the chain of infection, the treatments had reduced worm infections in entire villages. Even more striking, when they tested the same kids nearly a decade later, they had more education and earned higher salaries. The female participants were less likely to be employed in domestic services.
And compared with Kremer’s first trial, deworming was a bargain. Textbooks cost $2 to $3 each. Deworming pills were as little as 49 cents. When Kremer calculated the kids’ bump in lifetime wages compared with the cost of treatment, it was a 60-to-1 ratio.
This is perfect TED Talk stuff: Conventional wisdom called into question, rigorous science triumphing over dogma. As word of Kremer’s study spread, he became part of a growing movement within international development to subject its assumptions to randomized controlled trials.
Based on his analysis, Kremer went on to ramp up a deworming NGO, but Hobbes notes the NGO stopped testing after their initial research. Additional testing by others revealed more nuance.
It’s an interesting question—when do you have enough evidence to stop testing each new application of a development idea?—and I get that you can’t run a four-year trial every time you roll out, say, the measles vaccine to a new country. But like many other aid projects under pressure to scale up too fast and too far, deworming kids to improve their education outcomes isn’t the slam-dunk its supporters make it out to be.
In 2000, the British Medical Journal (BMJ) published a literature review of 30 randomized control trials of deworming projects in 17 countries. While some of them showed modest gains in weight and height, none of them showed any effect on school attendance or cognitive performance. After criticism of the review by the World Bank and others, the BMJ ran it again in 2009 with stricter inclusion criteria. But the results didn’t change. Another review, in 2012, found the same thing: “We do not know if these programmes have an effect on weight, height, school attendance, or school performance.”
Kremer and Evidence Action dispute the way these reviews were carried out, and sent me an upcoming study from Uganda that found links between deworming and improved test scores. But the evidence they cite on their own website undermines this data. Kremer’s 2004 study reporting the results of the original deworming trial notes—in the abstract!—that “we do not find evidence that deworming improves academic test scores,” only attendance. Another literature review cited on Deworm the World’s website says, “When infected children are given deworming treatment, immediate educational and cognitive benefits are not always apparent.”
Then there’s the comparison to textbooks. Kenya, it turns out, is a uniquely terrible place to hand out textbooks to kids and expect better academic performance. When Kremer reported that textbooks had no overall effect, he also noted that they did actually improve test scores for the kids who were already at the top of the class. The main problem, it seems, was that the textbooks were in English, the second or third language for most of the kids. Of the third-graders given textbooks, only 15 percent could even read them.
In the 1980s and early ’90s, a series of meta-analyses found that textbooks were actually effective at improving school performance in places where the language issues weren’t as complex. In his own paper reporting the Kenya results, Kremer noted that, in Nicaragua and the Philippines, giving kids textbooks did improve their test scores.
Here is the crux of it:
But the point of all this is not to talk shit on Kremer—who has bettered the world more with his career than I ever have with mine—or to dismantle his deworming charity, or to advocate that we should all go back to giving out free textbooks. What I want to talk shit on is the paradigm of the Big Idea—that once we identify the correct one, we can simply unfurl it on the entire developing world like a picnic blanket.
There are villages where deworming will be the most meaningful education project possible. There are others where free textbooks will. In other places, it will be new school buildings, more teachers, lower fees, better transport, tutors, uniforms. There’s probably a village out there where a PlayPump would beat all these approaches combined. The point is, we don’t know what works, where, or why. The only way to find out is to test these models—not just before their initial success but afterward, and constantly.
I can see why it’s appealing to think that, once you find a successful formula for development, you can just scale it up like a Model T. Host governments want programs that get more effective as they get bigger. Individual donors, you and me, we want to feel like we’re backing a plucky little start-up that is going to save the world. No international institution wants to say in their annual report: “There’s this great NGO that increased attendance in a Kenyan school district. We’re giving them a modest sum to do the same thing in one other district in one other country.”
The repeated “success, scale, fail” experience of the last 20 years of development practice suggests something super boring: Development projects thrive or tank according to the specific dynamics of the place in which they’re applied. It’s not that you test something in one place, then scale it up to 50. It’s that you test it in one place, then test it in another, then another. No one will ever be invited to explain that in a TED talk.
Hobbes goes on to explain that testing means more money spent on overhead. That overhead would lead to greater effectiveness and, in the long run, lead to a bigger bang for the overall buck, but everything we do now is oriented toward keeping overhead as lean as possible. We ramp up projects that end up being incredibly wasteful. Sometimes they can be downright destructive.
This is the paradox: When you improve something, you change it in ways you couldn’t have expected. You can find examples of this in every corner of development practice. A project in Kenya that gave kids free uniforms, textbooks, and classroom materials increased enrollment by 50 percent, swamping the teachers and reducing the quality of education for everyone. Communities in India cut off their own water supply so they could be classified as “slums” and be eligible for slum-upgrading funding. I’ve worked in places where as soon as a company sets up a health clinic or an education program, the local government disappears—why should they spend money on primary schools when a rich company is ready to take on the responsibility?
There’s nothing avaricious about this. If anything, it demonstrates the entrepreneurial spirit we’re constantly telling the poor they need to demonstrate.
My favorite example of unintended consequences comes, weirdly enough, from the United States. In a speech to a criminology conference, Nancy G. Guerra, the director of the Institute for Global Studies at the University of Delaware, described a project where she held workshops with inner-city Latina teenagers, trying to prevent them from joining gangs. The program worked in that none of the girls committed any violence within six months of the workshops. But by the end of that time, they were all, each and every one, pregnant.
“That behavior was serving a need for them,” she says in her speech. “It made them feel powerful, it made them feel important, it gave them a sense of identity. ... When that ended, [they] needed another kind of meaning in their lives.”
The fancy academic term for this is “complex adaptive systems.” ...
So do we give up?
First, let’s de-room this elephant: Development has happened. The last 50 years have seen about the biggest explosion of prosperity in human history. ...
Development, no matter how it happens, is a slow process. ...
The ability of international development projects to speed up this process is limited. ...
And this is where I landed after a year of absorbing dozens of books and articles and speeches about international development: The arguments against it are myriad, and mostly logistical and technical. The argument for it is singular, moral, and, to me anyway, utterly convincing: We have so much, they have so little. ...
To this, I would add one note about faith-based economic development. There is a tendency to turn a tactic into a sacrament. Christians and congregations are frequently using two metrics for mission. First, there is a desire to help those in need. Second, there is a desire for congregants to be engaged in helping others in ways that are meaningful to the congregant. If the latter becomes particularly strong, it is very difficult to alter tactics, no matter how much data you show that demonstrates ineffectiveness and even harm. In my book, the first consideration is an absolute must. To do development that does not achieve the first criteria, no matter how meaningful it is to the congregation, is to dehumanize those in need as instruments for stroking our spiritual self-esteem. And that is why addressing economic issues from a Christian perspective requires both warm hearts AND cool heads.
Comments