It gives dangerous power to governments. Most of us fail to grasp the subtle nuances that threatens our liberty and livelihoods.
A sloppy attitude towards statistics has led to exaggerated and unjustified claims becoming commonplace in science, according to one of Britain’s most eminent statisticians.
Speaking ahead of his president’s address to the Royal Statistical Society, Prof Sir David Spiegelhalter, said that questionable practices such as cherry-picking data and “hacking statistics” to make findings appear more dramatic threatens to undermine public trust in science.
“We’re not concerned with either lies, utter falseness or fabrications. It’s to do with exaggerations or misleading claims,” he said. “It’s more difficult to deal with.”
The lecture, entitled Trust in Numbers, draws parallels between concerns around the reliability of published scientific research and the rise of fake news and alternative facts in politics.
“Both are associated with claims of a decrease in trust in expertise, and both concern the use of numbers and scientific evidence,” he said.
By the time the general public comes to read about a scientific result, often after it has been filtered through a university press office and a media outlet, they would be right to treat it with suspicion, Spiegelhalter argues.
“In my darkest moods I follow what could be called the ‘Groucho principle’: because stories have gone through so many filters that encourage distortion and selection, the very fact that I am hearing a claim based on statistics is reason to disbelieve it,” he will say in the address on Wednesday evening.
He said that an overwhelming pressure to publish even marginal and mediocre work was partly to blame for dragging down the reliability of published scientific findings.
“The sheer quantity of scientific research published must mean the quality has gone down,” he said. “There should just be less published.”
The lecture highlights multiple ways in which the poor use of statistics leads scientists into a grey area, somewhere between rigorous science and outright misconduct.
A common problem, he said was the misuse of “p-values”, a statistical tool designed to show whether a blip in the data is real or caused by random chance. In one survey, around a quarter of scientists admitted to “rounding down” p-values to get their results below the significance threshold (0.05) required for publication, or including or excluding data in order to boost the apparent statistical strength of results.
The complexity of statistical tools available has also led to less transparency around what calculations actually show, he said. “The whole practice has changed towards greater sophistication and leads to greater opportunities for misuse, often with no malevolence intended at all,” he said.
Spiegelhalter’s intervention will fuel concerns over an apparent replication crisis, in which many high profile results have been shown to be unreliable. One recent study suggested that 64% of psychology findings in top journals cannot be reproduced and similar fears have been raised over the reliability of biomedical research.
“There’s a dispute about the extent of it, but I don’t think anyone would disagree that things could be improved a lot,” said Spiegelhalter.
Spiegelhalter, who is professor for the public understanding of risk at the University of Cambridge, criticises scientific journals for not providing a platform for people to “call out” research that is flawed – but not sufficiently problematic to warrant a retraction.
“It’s preposterous … in the modern world of critique and commentary and open discussion that that’s not possible,” he said. “At the moment a bad paper, if it’s not actually completely wrong and has to be retracted, just sits there sending out its message to the world.”
Hetan Shah, executive director of the Royal Statistical Society, said that the society is actively working to raise statistical standards across science. “The assumption is that scientists have got this covered, it’s just social science where the problem is,” he said, adding that behind closed doors many fields acknowledged there were deficits.
“Absolutely this is a concern and a problem,” he said.
