Randomised trials and public policy

[Gups]

I really liked this article by Ben Goldacre (a few months old now) and thought I'd share just to give Timmy a break (that's a hint Tim).

http://www.badscience.net/2011/05/we-should-so-blatantly-do-more-randomised-trials-on-policy/

Politicians are ignorant about trials, and they're weird about evidence. It doesn't need to be this way. In international development work, resources are tight, and people know that good intentions aren't enough: in fact, good intentions can sometimes do harm. We need to know what works.

In two new books published this month – "More Than Good Intentions" and "Poor Economics" – four academics describe amazing work testing interventions around the world with proper randomised trials. This is something we've bizarrely failed to do at home.
Is business training useful? There's a randomised trial on it in Peru. What about business mentors? In Mexico, they ran a randomised trial. Now think about all the different initiatives in the UK to support small businesses, or to help people find work. Do they work? No idea: you can have no clear idea.

Randomised trials are our best way to find out if something works: by randomly assigning participants to one intervention or another, and measuring the outcome we're interested in, we exclude all alternative explanations for any difference between the two groups. If you don't know which of two reasonable interventions is best, and you want to find out, a trial will tell you.

Microfinance schemes help small producers buy in bulk to make larger profits, and they change lives. But are group-liability loans better, because people default less, so the project is more sustainable? Or do anxieties about shared reponsibility restrict recruitment? Some academics ran a trial.

Do free uniforms improve school attendance, especially in pupils who don't own one at all? Someone ran a trial. Contingent payments improve attendance: but what's the best time to pay, and how? There's another trial. What about streaming in Kenyan schools, with high and low ability classes? Do all kids do better? Someone ran a trial. Maybe different strategies to encourage saving work best in different places? Innovations for Policy Action ran a series of trials to find out, in the Philippines, in Bolivia, in Peru.

I won't tell you the results, for any of those projects: because this isn't about good news on what works, or bad news about what doesn't. What matters is that someone ran a randomised trial and found the answer.

This week the papers and parliament were filled with uninformed wittering on sex education. If the goal is to delay sexual activity, or reduce sexually transmitted infections, and you don't know what age to start, or what to teach, then stop wittering: define your outcome, randomise schools to different programmes, and you'll have the answer by the end of next parliament.

Do long prison sentences work? At the moment sentences are hugely variable anyway: so randomise properly and run a trial. Different teaching approaches? Run a trial. Harder exams? Run a trial. Job-seeking support? Run a trial. This isn't rocket science: the first trial was in the bible. And I'm certainly not saying these are the best UK policy trials you could run. In fact, the most important part of evidence based policy is identifying where there is uncertainty.

So here is my fantasy. We sack the Behavioural Insights Team – all they'll do is overextrapolate from behavioural economics research – and open a Number Ten Policy Trials Unit instead.

They sit down to write a giant list of unanswered questions, for situations where we don't know if an intervention works: this will be most of them. Then we filter down to questions where a randomised trial can feasibly be run. Then we do them.

This won't cost money: it will save money, in unprecedented amounts, by permitting disinvestment in failed interventions, and it will transform the country. It's efficient, it's sensible, and it will never happen, because politicians are too ignorant of these simple ideas, too arrogant to have their ideologies questioned, and too scared – let's be generous – of hard data on their good intentions.

[LaCroix]

i prefer a timmay thread to this. he's been slacking off lately, though

[Brazen]

Randomised trials suggest that articles pasted in all-italics are near-impossible to read.

Public policy is perhaps harder to apply randomised trials to because whoever implements it doesn't want to be seen be behind the failed policy. Would you want to be behind a sex education scheme that increased teen pregnancy, for example?

[Malthus]

I can foresee some ethical problems with (for example) randomized trials to test the effectiveness of longer prison sentences. "Yeah, we are giving you a longer sentence than your buddy who committed the exact same crime - part of this trial we are running, don't you know ... ".

In most cases, participants have to agree to participate in testing.

[Gups]

Near instant results!

It's a fair point but that would suggest we never have any sex education schemes for fear of failure.

[Brazen]

It's a fair point but that would suggest we never have any sex education schemes for fear of failure.

But there wouldn't be a different one designed for direct statistical comparison in the next school where they're not all pushing prams home.

[Gups]

I can foresee some ethical problems with (for example) randomized trials to test the effectiveness of longer prison sentences. "Yeah, we are giving you a longer sentence than your buddy who committed the exact same crime - part of this trial we are running, don't you know ... ".

But as Goldacre points out, sentences are hardly uniform in any event. Although I agree that randomly selecting sentences is never going to be politically acceptable.

In most cases, participants have to agree to participate in testing.

I think that's only medical trials.

[Gups]

It's a fair point but that would suggest we never have any sex education schemes for fear of failure.

But there wouldn't be a different one designed for direct statistical comparison in the next school where they're not all pushing prams home.

So we never know which is better.

[Scipio]

The question to ask is not "do long prison sentences work?" but "What do long prison sentences work at?" and "what should the purpose of prison be?"

[Malthus]

But as Goldacre points out, sentences are hardly uniform in any event. Although I agree that randomly selecting sentences is never going to be politically acceptable.

No, but there is a world of difference between sentence inequality caused by individual variation between judges, and sentence inequality caused by deliberate "policy testing".

I think that's only medical trials.

I dunno about that. What about tests like the infamous test used to determine if the average person could be persuaded to torture people? (which found that it was disturbingly easy). The commentary to that test is that today it would be considered "unethical". It wasn't any sort of medical test.

Really, I dunno why medical trials would be any different from other sorts of trials on people, except that the risk of harm is (usually) greater in a medical trial. But a prision sentence trial imposes direct harm on some of the participants (a longer prision sentence than they would otherwise get) for no benefit to themselves (the alleged benefit, if any, is to society as a whole). I can't see how this would possibly be ethical, without consent, and I can't imagine who would consent to being imprisioned longer without some sort of payment or other reward.   

[The Brain]

Really, I dunno why medical trials would be any different from other sorts of trials on people, except that the risk of harm is (usually) greater in a medical trial. But a prision sentence trial imposes direct harm on some of the participants (a longer prision sentence than they would otherwise get) for no benefit to themselves (the alleged benefit, if any, is to society as a whole). I can't see how this would possibly be ethical, without consent, and I can't imagine who would consent to being imprisioned longer without some sort of payment or other reward.

You could give one group shorter prison sentences.

[Gups]

OK I agree about prison sentences (as I thought was already clear), the best one can do is seek to learn about effectiveness (having defined your goals first of course) from existing variations on sentences.

But most other policies can be tested without consent. We do it all the time already. Kids aren't asked to consent to the teaching method their teacher wants to use oor patients as to the method by which they will be allocated a doctor.

I think you may have the wrong end of the stick. This is about testing policies not individuals. Obviously you would need a sane adult's consent before undertaking a psychological test on them.

[grumbler]

Gups, I think you are missing the objection:  with randomized trials, the negative outcomes would be seen as a deliberate policy choice by the trial-approvers.  The "losers" in the trial would (rightly) see that the reason the winners were winners was because the trail organizers deliberately set out to inflict the losing policy on the losers.  While the argument would be made that people as a whole benefited from the knowledge that the trial provided, the fact would remain that the "victims" of the trial would still have suffered because of deliberate policy choices (choices made in the knowledge that the losers would lose).  The positive outcome for the whole would still leave the losers as losers.

Now, you can argue that the whole point if the trial is that, at the outset, no one knew who the losers would be, but that's not how politics works.  Randomly chosing losers is still seen as "unjust" by the losers, and many people would sympathize with them.

This was the reason why serious suggestions to have the Federal government pay for random school systems (or non-random ones - the DC School system was specifically mentioned as a target) to convert to the voucher model in the 1990s to see if it provided the perceived benefits were dropped - if vouchers worked, the politicians and bureaucrats stood to be accused of unfairly advantaging the targeted students, and if it didn't they stood to be accused of "unfairly" disadvantaging them.  therefor, the funds were thinly distributed and told us nothing.

I agree with you in theory, but think putting the ida into practice is a political non-starter.  Politicians hate to be unambiguously responsible for anything.