The International Archives of Medicine published a study in 2015 that showed that adding high-cocoa chocolate to a low-carb diet accelerates weight loss, as opposed to a low-carb diet with no chocolate. Johannes Bohannon and others of the Institute of Diet and Health conducted the study.
Bohannon selected a random sample of people aged 19 to 67. He separated the subjects into three groups: a control group, a low-carb group, and a low-carb-plus-chocolate group. The control group ate as usual, the low-carb group ate a low-carb diet, and the low-carb-plus- chocolate group ate a low-carb diet supplemented with 42 grams of high-cocoa chocolate each day.
The low-carb-plus-chocolate group lost weight 10 percent faster than the low-carb group. The low-carb-plus-chocolate group also showed positive changes in cholesterol levels. The weight loss shifts and the cholesterol improvement were each found to be statistically significant. That means the likelihood of the differences being explained by chance alone were statistically tiny.
The idea of losing weight by eating chocolate resonated with people. The results of the study were published in more than 20 countries and translated into six different languages, usually accompanied by an enticing photo of a sumptuous piece of chocolate. It was a definitive study published in a legitimate journal.
The study was a real one, but it was conducted not by a scientist, but by a journalist. The journalist was frustrated by the lack of rigor in scientific studies and wanted to demonstrate how easy it was, in his words, “to turn bad science into big headlines.” He designed the study to be scientifically believable, yet intentionally deceptive.
The study was technically a clinical trial. The subjects were indeed randomly selected, and the results were, in fact, statistically significant. The study was a real one, yet what Bohannon chose to disclose—and, importantly, what he chose not to disclose—made the study sound more definitive than it really was.
One thing Bohannon chose not to reveal was the size of the group that participated in the study. Just fifteen people volunteered to participate. That meant each group had just five people. In statistics, that means he had a very small sample size.
Bohannon also chose not to disclose the number of variables he tested. In addition to weight and cholesterol, the clinical trial tested sodium levels, blood protein, and a variety of other things. He tested 18 different variables in total. Weight loss and cholesterol just happened to be the two variables that showed a statistically significant difference.
Statisticians call this “p-hacking.” The p-value is the probability that a relationship occurs only by chance—the lower the p-value, the less likely the relationship occurs just through some quirk in the data. Studies that show low p-values are said to be statistically significant. P-hacking is when a scientist tests for lots of different variables and chooses the variables that show low p-values to feature in the study. Bohannon “p-hacked” the data. A low p-value allowed him to use the “statistically significant” label.
Bohannon chose to omit a few other little goodies about the study that, if disclosed, would have shed a little doubt on his findings. He specifically chose high-cocoa chocolate (aka, “dark” chocolate), for instance, because he felt it would sound more believable than other forms of chocolate (like milk chocolate.)
He sent the study to 20 different journals in order to get it published. All 20 were hand-selected for their lack of peer reviews. The International Archives of Medicine actually charged him to publish the study. The Institute of Diet and Health, the “institution” Bohannon associated with, was nothing more than a website.
The study was scientifically worthless. Bohannon had an agenda, and he designed the study and all that went with it in a way that promoted his agenda.
Beware of so-called definitive studies. At KP7, we don’t believe the results of any definitive study unless we have conducted it ourselves. In the investment business, we get bombarded with data each and every day. We heed the results of someone else’s study only when we know who performed it and only when we are absolutely certain they have no hidden agenda or bias. We ignore the results of all other studies, definitive or not.
