A bogus study of chocolate and diets -- and the media that swallowed it whole

John Bohannon is a journalist who made something of a name for himself with a 2013 exposé of the credulousness of science journals. This week he unveiled a new sting, in which he duped a host of media outlets in Europe and the U.S. with a patently bogus study purporting to find that eating chocolate made diets more effective.

Bohannon promptly attracted kudos for exposing the laziness of the mass media when it comes to checking out superficial scientific claims. But he has also drawn brickbats for his own investigative technique and questions about his own ethics. All these reactions are justified.

It shouldn’t be overlooked that among the victims of Bohannon’s sting were the readers of the magazine and websites that fell for it. Indeed, Bohannon seems rather proud about scamming these innocents. The headline on his report of the sting, on the website io9, is: “I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here’s How.�

Let’s examine the sting.

Working with a pair of German documentary filmmakers looking into the “junk-science diet industry,� Bohannon refashioned himself as Johannes Bohannon, Ph.D., and designed an “authentic� but hopelessly flawed study to show that the daily consumption of 1.5 ounces of chocolate enhanced the weight loss effect of a low-carbohydrate diet.

They divided 15 real-life subjects into three groups--one to go on a low-carb diet for three weeks, the second to augment the diet with chocolate, and the third to keep to their normal dietary habits as a control group. As it turned out, both of the low-carb groups lost weight, but the chocolate eaters lost a bit more on average.

They wrote up the study and shipped it out to a list of non-peer reviewed journals known to be less than exacting about the papers they accept. They quickly found one willing to publish the paper for a fee of merely 600 euros ($660).

The next step was to gin up a catchy press release: “Can you indulge your sweet tooth and lose weight at the same time? If it’s chocolate you crave, then the answer seems to be: yes. That is the surprising conclusion of a study by German researchers...� Etc., etc.

Plenty of publications and media sites bit, including the U.K.’s Daily Star, foreign sites of the Huffington Post, and Shape magazine. Bohannon reports that he got very few calls from reporters or editors seeking to verify his bona-fides or the claims in the press release. The gulled publications seemed more interested in taking the story at face value and using it as an excuse to run what Bohannon calls “vaguely pornographic images of women eating chocolate.�

Bohannon justifies his project as an effort to expose “journalists’ incredible laziness.... If you lay out the information just right, you can shape the story that emerges in the media almost like you were writing those stories yourself.� He’s right, up to a point. None of the publications that used his study did more than cursory checking, if even that.

But he concludes too much from this. For one thing, not all the publications that swallowed the story are really journalistic institutions; they’re popular mass media, and their predilection for bogus diet prescriptions are only part of what makes them discreditable.

Indeed, one aspect that made them vulnerable to Bohannon’s pitch is that it fed into what they, and their readers, want to hear; many of them have been pushing the purported health benefits of chocolate for years. Bohannon’s study seemed tailor-made to feed his targets’ prejudices, as an example of “confirmation bias.� He and his partners were well aware that diet nostrums are especially likely to be devoid of facts; one need only look at the history of the Dr. Oz show to know that.

Did Bohannon prove that “journalists� are credulous about scientific claims? Not really. No major U.S. outlet appears to have published his bogus study. Maybe what he proved is that journalists aren’t so credulous.

Since Bohannon published his exposé, it’s become the target of more pushback. Several commentators have questioned the very ethics of Bohannon’s project, asking whether the subjects were made aware of its flaws and its purpose and whether the study was subjected to third-party Institutional Review Board scrutiny expected of studies involving human subjects. There’s no indication in Bohannon’s report that it was.

“How does this sting’s risk-benefit balance measure up?� asks Michelle N. Meyer of Harvard Law School on the Faculty Lounge blog. “Not very well, I think.� She observes that “this sting tells us almost nothing that we didn’t already know,� and further points out that Bohannon performed a disservice to the readers of Shape and other outlets by allowing the publication to go ahead without warning the publications in time that they had perpetrated a sting.

“Bohannon et al. caused shoddy science to be published in numerous outlets with a collective circulation that pales in comparison to the readership of io9, where Bohannon announced his sting,� she writes. “Just as when a sensational story is printed on page 1 and its correction is printed on page 14, it seems likely that far more people will read and be affected by the shoddy science than by the announcement that the latter was all a ruse.�

Keep up to date with the Economy Hub. Follow @hiltzikm on Twitter, see our Facebook page, or email [email protected].