Â鶹´«Ã½

Helen Pearson,
Princeton University Press 

Often, when I read a non-fiction book, I think “this could have been an essay”. An argument that could have been made in 10,000 words gets dragged out to 100,000 with filler anecdotes, repetition and, worst-case, bad extrapolations into topics the writer isn’t qualified to discuss. I won’t name names – we’ve all read the books.

Beyond Belief: How evidence shows what really works is the rare example of the opposite: I genuinely wanted it to be longer. It is a book about the seemingly dry topic of evidence-based policy, exploring how experiments and trials can be used in fields like international development, policing and management. It talks a lot about systematic reviews. Yet because it is so readable and punchy, I burned through the whole thing in a weekend.

Author is a journalist and a senior editor at the journal Nature. I should say, I know her slightly: she has edited a few of my pieces over the years and we have sometimes been in the same London pubs.

Pearson’s topic is the “evidence revolution”: the global movement to ensure that decisions are based on evidence from research, rather than the authority of supposed experts or just conventional wisdom. She begins with medicine, where it is common to test a new treatment using a randomised controlled trial: some people get the treatment while others don’t, and their outcomes are compared.

In 1747, naval surgeon James Lind conducted a key early trial aboard the warship HMS Salisbury. At that time, sailors often had scurvy: a horrific condition of swollen limbs, decaying gums and ultimately haemorrhaging. Lind recruited 12 men and split them into six pairs, each receiving a different dietary supplement. The two men given oranges and lemons recovered rapidly. Nobody knew why – the key chemical, vitamin C, wouldn’t be identified until the 20th century – but it worked and that was enough. Within decades, citrus juice became standard on ships and scurvy became rare.

This story is part of the mythology of modern medicine. So I was completely taken aback when Pearson wrote that “the term evidence-based medicine is barely 35 years old”. For most of the 20th century, even though many controlled trials were run, most medical decisions were still being made on the say-so of senior doctors. Pearson tells the story of Iain Chalmers, who, as a junior doctor in the 1970s, was puzzled to see that “when two different doctors treated women with the same condition, they would often give wildly different advice”.

Chalmers, working with colleagues like Archie Cochrane, set out to change that. Their chosen methods were systematic reviews and meta-analyses: they would pull together the published evidence about a given question, scouring huge numbers of scientific journals, then go through it piece by piece to work out which bits were reliable and which weren’t, and what the combined evidence said. This ultimately led to institutions like the , which conducts systematic reviews on a huge range of topics.

Few reasonable people would argue against any of this. While controlled trials and systematic reviews aren’t the last word in good healthcare, they are crucial tools and have been underused. The same is true of other areas. Pearson has many horror stories about utterly ineffective projects (sometimes unresearched) that wasted money, such as the efforts to build “bat bridges” over roads in the UK, which the bats never used.

But the story gets gnarlier as the evidence revolution moves into other areas. Pearson tackles the use of randomised-controlled trials in social policy, such as welfare payments, international development, policing, parenting, conservation and education.

These topics are harder to study because they are such complex systems. For one thing, they tend to include humans, with our pesky free will and pig-headedness. Pearson is fully aware of these dimensions, that trials of social policies are less reliable than trials of medical interventions, and less likely to generalise well.

She describes how a poverty-reduction measure worked in one community, based on a controlled trial, but didn’t necessarily work elsewhere because communities differ so much. This is a recurring problem, and for me it means social policy trials are just less useful than medical trials, so we mustn’t over-emphasise them.

I am not arguing against trials and systematic reviews for such policies – on the contrary. But I do feel Pearson and her interviewees are too convinced of the benefits.

There are plenty of examples of policies with good evidence bases that fail because their proponents have neglected core political tasks, such as getting informed consent from the affected communities. Efforts like reintroducing wolves, vaccinating children and taxing polluting vehicles often stumble, not because they are unsupported by science, but because the people who have to live with them don’t trust the authorities to act in their interests. These are problems of community cohesion and democratic deficit, of trust and equity, of power. They are solved only by working with those communities to develop policies.

Again, Pearson raises this. She describes how some practitioners of evidence-based conservation now work more closely with groups like Indigenous peoples, often shut out of decision-making despite their vast trove of knowledge. But in the end, she treats these sociopolitical barriers just as wrinkles, while I think they are the heart of the matter. For example, if you want to understand why much teaching is bad, as the husband of a teacher, I’d say it’s not lack of evidence; it’s that teachers are overworked and underpaid. Most don’t have time to follow the latest research, let alone use it.

Beyond Belief is a fascinating account of people who have tried to apply one of science’s favourite methods to knotty and subjective areas of human life – with all the triumphs and failures that implies. I just wish it was a bit longer, so Pearson could explore the practical and sociopolitical barriers to evidence-based decision-making. On the plus side, that could be her next book.

Michael Marshall is a science writer based in Devon, UK

Three more great books on following the evidence

by Ben Goldacre

Scourge of quacks Ben Goldacre, a doctor, academic, writer and broadcaster, exposes how often the media promotes nonsense as fact, when even simple checks would reveal the reality.

The Golem at Large: What you should know about technology
by Harry Collins and Trevor Pinch

A pair of sociologists, Harry Collins and Trevor Pinch, explore how scientific methods struggle when faced with the complexities and uncertainties of the real world, the place where everything happens.

by Anthony King and Ivor Crewe

Variously hilarious and infuriating, this account of stupid government mistakes reveals why British politics often goes so very wrong. Lack of evidence is just one issue among many, and far from the biggest.