According to Charles Seife, “A well-wrapped statistic is better than Hitler’s ‘big lie’; it misleads, yet it cannot be pinned on you.” Twenty-some years ago I bought and read Darrell Huff’s little gem of a book: How to Lie with Statistics. And it seems I wasn’t the only one, particularly when I read about some of the problems with medical science and psychology research. Huff said that while his book might appear to be a primer in the ways to use statistics to deceive others, honest people must also learn them in self-defense.
Thesaurus.com gave 48 synonyms for the verb form of “lie,” including: deceive, mislead, misrepresent, exaggerate, fabricate, misstate, fudge and BS. One or more of these synonyms will be found regularly in the discussion (and linked articles) that follow. But make no mistake—the discussion is still about how the public can be lied to in what they read health science news. Along with a previous article, “The Reproducibility Problem,” this is meant in inform. So let’s look at some of the ways that we are lied to about psychology and medical science research news.
Gary Schwitzer wrote about the problem of exaggeration in health science news releases. He commended an editorial by Ben Goldacre and a research paper by Sumner et al. published in the BMJ, a peer reviewed medical journal, on exaggerations in academic press releases and the news reporting they generate. Sumner et al. found that most of the exaggerations identified in their study did not occur ‘de novo’ in the media reports, but were “already present in the text of the press releases produced by academics and their establishments.” And when press releases contained misleading statements, it was likely that the news would be as well. “Exaggeration in news is strongly related with exaggeration in press releases.”
The study’s three main outcome measures of exaggeration were: whether causal statements were made about correlational research; if there was advice to readers about behavior changes; and were there inferences made to humans from animal studies that went beyond those already accepted in the literature. The authors concluded:
Our findings may seem like bad news but we prefer to view them positively: if the majority of exaggeration occurs within academic establishments, then the academic community has the opportunity to make an important difference to the quality of biomedical and health related news.
Goldacre noted that while some fixes to the problem were in place, they were routinely ignored. He further suggested that press releases should be treated as part of the scientific publication and then subjected to the same accountability, feedback and transparency of the published research. “Collectively this would produce an information trail and accountability among peers and the public.” Schwitzer noted to how the academic community had the opportunity to “make an important difference to the quality of biomedical and health related news.”
A review of 2,047 biomedical and life-science research articles by Fang, Steen and Casadevall indicated that only 21.3% of retractions could be attributed to error. A whopping 67.4% were attributable to misconduct, including fraud or suspected fraud (43.4%). The percentage of scientific articles retracted since 1975 for fraud has risen 10-fold. “We further note that not all articles suspected of fraud have been retracted.”
But these weren’t the only problems with the current academic research and publication process. A November 2014 article in Nature described a peer-review scam, where journals were forced to retract 110 papers involved in at least 6 instances of peer-review rigging. “What all these cases had in common was that researchers exploited vulnerabilities in the publishers’ computerized systems to dupe editors into accepting manuscripts, often by doing their own reviews.” Recommendations in the article were also made for changing the way editors assign reviewers, particularly the use of reviewers suggested by a manuscript’s author. Cases of authors suggesting friends and even themselves—using a maiden name—were noted.
A further concern is with p-hacking, also known as: data-dredging, fishing, and others. Uri Simonsohn and Joe Simmons, who jointly coined the term, said p-hacking “is trying multiple things until you get the desired result.” They said p-hacking was particularly likely in “today’s environment of studies that chase small effects hidden noisy data.” Their simulations have shown that changing a few data-analysis decisions can increase the rate of false-positives to 60%. Confirming how widespread this is would be difficult. But they found evidence that “many published psychology papers report P values that cluster suspiciously around 0.05, just as would be expected in researchers fished for significant P values until they found one.”
According to Charles Seife, a journalist and author with degrees in mathematics from Princeton and Yale, a “p-value” is a rough measure of how likely your observation could be a statistical fluke. The lower the p-value, the more confident you are that your observation isn’t a fluke. Generally, statistical significance is a p<0.05. The YouTube video of Seife’s talk is about 45 minutes, with another twenty some minutes of question and answer. It gives an understandable presentation of how statistics, including p-hacking, can be misused.
Simonsohn and Simmons devised a method consisting of three simple pieces of information that scientists should include in an academic paper to indicate their data was not p-hacked. Whimsically, they suggested these three rules could be remembered as a song (they need to work on their musical composition skills). First, they preach to the choir. You’ll recognize the “melody” they used when you read the lyrics:
Choir: There is no need to wait for everyone to catch-up with your desire for a more transparent science. If you did not p-hack a finding, say it, and your results will be evaluated with the greater confidence they deserve.
If you aren’t p-hacking and you know it, clap your hands.
If you determined sample size in advance,say it.
If you did not drop any variables,say it.
If you did not drop any conditions,say it.
A mere 21 words, included in the Methods section of a paper would declare the above: “We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study.”
Scientific theories are in principle subject to revision. And sometimes people’s desires drive them to find explanations that harmonize with their desires and with a worldview that reinforces those desires. (Vern Poythress, Redeeming Science)