Nearly 10 years ago you caused a stir when you showed that most published scientific findings are wrong. Have things improved since then?
We have seen lots of improvement. Some fields have made more progress than others, but many scientists have been sensitised to the issues that compromise good research and are trying to come up with solutions.
So scientists didn’t take it personally?
A few may have been annoyed. But I’m not interested in shaming individuals. I’m interested in the big picture – in showing problems that may affect many thousands of papers. If you say a single paper is wrong, that can easily get you into trouble. It’s different when you say that all of the papers are wrong.
In what way have things improved?
Many disciplines have moved to a team approach, so scientists across the field work together, share data and repeat the original study to see if they arrive at the same results. Such changes have transformed human genome epidemiology from an unreliable discipline with a 99 per cent non-replication rate to one with a 99 per cent replication rate, for example.
Advertisement
It would be interesting to further assess the value of replication to understand when, how and by whom it should be done. Right now I’m collecting data to see if solutions that work for some fields could also apply to others.
Why is replication still rare?
It may be entrenched tradition. The dominant paradigm has been that, to be a good scientist, you shouldn’t do what has been done before. I’m not saying this is completely wrong. We do need different approaches to the same question. But if we have no replication then everything we do is unclear. In fact, the psychological sciences just because, when people finally did try to replicate some key findings, they found that they couldn’t.
Are some disciplines more susceptible to flawed research?
All scientific fields face challenges, but they have adopted safeguards to different extents. Many of the physical sciences use large-scale collaboration, for instance. They also have a culture where the credibility of a finding has to be very high before they publish it. In biomedical disciplines, if you find something even nominally significant, you publish it.
That said, I don’t think standards should be exactly the same across fields. The question is whether we can convey that a finding is of low credibility, that “this is interesting, but it’s just 1 per cent likely to be true – it needs to be replicated and we will have to wait”.
Your proposed solution is to introduce a credibility estimate. How might that work?
Each scientific paper and major result would have a credibility estimate based on the quality of the research. So for the finding that smoking causes lung cancer, the credibility estimate would be 99.99999 per cent. Conversely, for findings where we have only seen a few cases that drug X may cause severe bleeding, for instance, the credibility estimate might be 1 per cent. It may still be worth publishing because it is important to make sure the drug doesn’t cause severe bleeding.
At the moment we have no guidance when we see yet another paper that claims to have “significant results”. Nearly all papers in the biomedical sciences claim significant results – but even so we know that less than half of findings get replicated.
Could credibility estimates stifle research?
Weird, extravagant, bold ideas should be out there, presented with whatever evidence there is. We cannot wait for perfect certainty; there is no such thing in science anyway. I am happy to see low credibility papers published, otherwise we do risk stifling scientific advances. But they should come with the caveat that it’s very early in the game.
A lot of research never sees the light of day, particularly if the results are negative. Does this skew our understanding?
We know that for even the most visible type of research – clinical trials – about half aren’t published. Even when they are, and the outcomes are also published, many aren’t analysed in the way that was originally intended. A big problem is that investigators don’t want to write up negative results, so they often write them up in a way that makes them seem to be positive. For other types of research, it may be even worse.
How do you spin a negative result as a positive one? Isn’t that borderline fraudulent?
Oh, there are many ways. You can change the analysis so even though your intention was to compare A with B, you compare A with C. Or you follow up after six months instead of nine. Or you use a different scale, or some subscale of that scale. It’s endless.
This isn’t rare. About 75 per cent of scientists use these questionable research practices. They are distinct from fraud, though. Fraud is when you present a result that clearly isn’t there, or say data exists that doesn’t. In these cases the result is there but you have really explored and data-dredged to get to it.
This has been described as “torturing the data until it confesses”. What drives this?
Under the current reward system you get credit for publishing, for getting grants, for being the first – no matter how credible the work. Real innovation is important, but it is also extremely rare. Science is a communal enterprise. People don’t operate in a vacuum and just wake up one morning with an idea that no one else had.
We need to realign our incentives to reward good quality, replicable research and sharing practices instead of how many papers you publish with how many extravagant claims.
Is that kind of systemic change possible?
It’s not impossible. And I’m optimistic because I have already seen changes in several fields.
Scientists have good intentions. We are trying to get to the truth. That’s what we want. At the same time we are under pressure from funding agencies and competition to come up with extreme results. So the main challenge is how to align the important stakeholders – not just scientists but funding agencies, journals, universities and research institutions. We don’t need to apologise for science, it is the most wonderful endeavour, but we shouldn’t try to claim more than we can.
“We don’t need to apologise for science, but we should not claim more than we can”
What role do the media play in all of this?
You amplify signals. That means you can play a pivotal role if you disseminate credible information, show how difficult science is and help to teach the public what it means to have strong evidence, when to be cautious and that there are limitations in scientific work.
How would you feel if you were wrong – if someone found problems in your own work?
Totally wonderful. I love finding that there’s something that could be improved. This is what keeps me alive. I think that science is an evolutionary process where people find out that something can be done in a better way. Working on the science of science has exactly the same rules as any other field. So I’m very open to criticism – it can be very productive.
Profile
John P. A. Ioannidis is co-director of the University, California, and also professor of disease prevention, health research and policy, and statistics. He studies ways to improve scientific credibility, most recently in the journal PLoS Medicine ().
This article appeared in print under the headline “How not to be wrong”