麻豆传媒

WHEN Stuart Ritchie was a graduate student in Edinburgh, UK, in 2011, he was involved in an incident that shook his faith in science. With two colleagues, he tried and failed to replicate a famous experiment on precognition, the ability to see the future. They sent their results to the journal that published the original research and received an immediate rejection on the grounds that the journal didn鈥檛 accept studies that repeated previous experiments.

Ritchie remained a scientist 鈥� he is a psychologist at King鈥檚 College London with a focus on studying human intelligence 鈥� but ever since that rejection, he has been on a crusade to air science鈥檚 dirty laundry. His latest book is , in which he shows how, all too often, we can鈥檛 rely on the facts that science provides.

Graham Lawton: The grand and scary claim of your book is that something is rotten in the kingdom of science.

Stuart Ritchie: Absolutely. We think of science as being this objective thing that tells us facts about the world and produces all these scientific papers, which are almost sacred things. But a lot of people don鈥檛 see how the sausage is made. I think if they had more of an idea of how the process happens, they would question the truth status of those papers much more. In a lot of cases, the science is useless, not worth the paper it is written on.

You identify four main causes of rot.

First there鈥檚 fraud, when people deliberately alter or make up results to try to get a paper published. That鈥檚 rare, but not as rare as we would like to think. If you ask scientists whether they鈥檝e committed fraud, only a very small number say yes. But something like 14 per cent will tell you that they think they know of an instance where a colleague has committed fraud.

That blurs into the second thing, which is bias. Everyone wants to change the world, so people are biased towards finding significant results. And that can make them see things that aren鈥檛 there. They might make arbitrary changes to their statistical analysis 鈥� called p-hacking 鈥� to find the result they want. Drop a few participants here, change a number there and convince yourself that it鈥檚 the right thing to do.

The third cause is unforced errors, like a typo in a spreadsheet that knocks the whole result off track. That is negligence.

The last category is hype, where scientists are pushed towards writing up their results as if they are much more exciting than they are. The way papers are written now often makes it sound as if the findings are going to revolutionise the way we think about things. But we鈥檙e not making more groundbreaking discoveries than we used to. It鈥檚 just that there鈥檚 pressure on how we鈥檙e supposed to write up science, and lots of media excitement.

If you put all those four together, it looks as if something is very rotten.

How did we let the rot set in?

It crept in due to a system of perverse incentives. There鈥檚 huge pressure to publish papers and huge pressure to bring in grants 鈥� which is an incentive to publish papers and apply for grants, but not an incentive to discover the truth. We focus far too much on rewarding people who have brought in big grants or published papers in prestigious journals, which isn鈥檛 necessarily getting us what we want.

But there are checks and balances, like peer review, where independent experts vet papers before they are published鈥�

It works in some cases, but it鈥檚 nowhere near the filter it needs to be. Some of the worst papers ever went through the peer-review system of the world鈥檚 best journals.

For instance, the system isn鈥檛 set up in a way that peer reviewers can easily get raw data. It鈥檚 absurd. The people who are supposed to be checking whether the analysis is correct rarely see the data that the claims are based on.

Reviewers are also often rushed. They themselves are busy scientists. Journals want there to be only a short time between articles being submitting and accepted, so peer reviewers are pushed towards accepting things really quickly. And journals only want to accept the flashiest and most exciting claims.

That鈥檚 before you get into any personal vendettas that some peer reviewers have for one another. Even if the paper is anonymous, you can usually tell who wrote it. If it鈥檚 your rival lab, maybe you鈥檒l give it a very harsh review.

Another check in the system is replication. But that seems to have gone wrong.

I remember being told when I was an undergrad that a paper should be written in such a way that anyone could replicate the experiment. But when people try and do that, often they can鈥檛 even set the experiment up because the paper doesn鈥檛 contain enough information. When they can replicate the experiment, they often find results that are different or not statistically significant.

Scientists are not being fully open or transparent. They can often be quite reluctant to give more details when someone comes along and says, 鈥淚 want to replicate your work鈥�, because there鈥檚 this culture of, 鈥淥h, I know best how to do this experiment.鈥�

How much published science isn鈥檛 replicable?

In my own subject, psychology, replication was recognised as a major problem from about 2011. About 50 per cent of studies in the literature don鈥檛 seem to be replicable.

That replication crisis spread into pretty much every subject that has been looked at. In most subjects, if you ask scientists, 鈥淒o you think the research is completely robust?鈥�, a huge proportion say no. But it鈥檚 hard to put a number on it because almost nobody else has made a systematic attempt to replicate results. My guess would be that the picture is less bad in other subjects, but we don鈥檛 know.

鈥淪cientific papers are seen as this almost sacred thing. But a lot of people don鈥檛 know how the sausage is made鈥�

I think that is itself a demonstration of the problem. We are not clear what proportion of papers from biomedicine or chemistry or physics and so on would be replicated because replication is not incentivised. Going back and doing someone鈥檚 experiment again, is, in most cases, not what the journal system and the university system want.

Is the replication crisis getting worse?

I think there鈥檚 evidence that things are worse now than they were 30 years ago in terms of this obsession with publication. At least in some cases, people who finish their PhDs now are expected to have some astonishingly high number of peer-reviewed publications, something like 19. A few years ago, you鈥檇 be expected to have five or six. The quality of the work inevitably has to suffer.

This a long-standing problem.

Why write a book about it in 2020?

We鈥檝e got to the point where we have a handle on the issues. We鈥檝e also got a great deal of so.called meta science, the science of science. People have spent their entire careers running large replication attempts. So I thought it was a good time to summarise where we are.

A lot of it is still a bit ivory tower, discussed only within science. When we鈥檙e communicating to the public, we just go back to 鈥渉ere鈥檚 a new exciting paper that tells us this is true鈥� rather than revealing how the process really works.

Why should people who aren鈥檛 scientists care about this?

So much of what we do relies on science. So if the whole system is off, then that has major consequences. It couldn鈥檛 be more clearly illustrated than with covid-19. All of the things I鈥檝e talked about have come up in the pandemic: accusations of fraud, retractions from top journals, screwed-up statistical analyses, lack of data sharing, negligence. And it has put rocket boosters under hype.

Another lesson we鈥檝e learned is the dangers of pushing science out too quickly. We鈥檝e seen a huge upsurge in research about covid-19, but a lot of it is probably completely useless. It鈥檚 kind of unbelievable. There couldn鈥檛 be a better illustration of the need for change.

Also, a huge amount of taxpayers鈥� money is wasted on useless science. It actually makes it harder to discover the truth in future because it muddies the picture.

What is at stake? If science just shrugs and carries on as normal, what will happen?

At the moment, people have a high degree of trust in science and I think that鈥檚 justified to some degree, certainly compared with politicians. But if people keep publishing these hyped results which end up not actually delivering on the promise, trust in science will decline.

We see it already in nutrition research. I think most people don鈥檛 take that seriously any more. I would hate to get into a situation where other sciences are in that same boat and research becomes a laughing stock.

Do you worry about giving ammunition to anti-science voices?

I do. I worry particularly about already politicised areas like climate change, vaccines and evolution. The last thing we want is to give climate deniers and creationists yet more reasons to be critical of science.

I鈥檓 critical of the way science is done, but I鈥檓 not critical of the idea that science is the best way we have of discovering how the world works, of making new things that improve the world for everyone. That is the whole reason that we must be sceptical and critical of it. I would be disappointed if people took this book as saying, 鈥淲ell, we can just chuck away science.鈥� That鈥檚 absolutely the opposite of what I鈥檓 trying to advocate.

How can we fix it?

I think the broad answer is scientists being more open and transparent about what they actually did in their research. But I don鈥檛 think that鈥檚 the only answer. Universities need to stop hiring people just on the basis of how many papers and citations they have and start rewarding a different kind of science: people who contribute data to the world, people who are part of large replication projects, people who are creating new tools to help other scientists do research more efficiently. Being a good scientific citizen, essentially.

Journals need to change the way they publish articles. They can incentivise people to preregister their work, so they鈥檝e published their data analysis plan before they actually get to the analysis and can鈥檛 do stuff like p-hacking.

Ultimately, I feel like this is an optimistic book, even though it is essentially a litany of human failure and weakness.

The amazing thing is that even with all these problems, we can still know stuff and make progress. But imagine how much more we could know if we managed to build a better system than the one we have. We can do much better. We can change the way the system works.

But I also think we need to change what we think science is, and have different expectations of it. We should expect scientists to be much more open, but also more boring.

The perverse incentive to make research look exciting is at the root of a lot of the problems. Science is not an endless march of exciting, flashy findings. There are transformative discoveries, and we should try and encourage those. But in general, science is incremental and small scale and requires a new kind of intellectual humility. I think we will be much better off when we realise that.

Ritchie鈥檚 view on the need to have peer-reviewed papers to complete a PhD has been clarified