You’re probably already seen the story about the chocolate bar hoax. “A recent study” brought joy to chocolate lovers everywhere by concluding that chocolate helps you lose weight. And oh yes, reduces your cholesterol and gives you an improved sense of well-being as well. Alas, like pixie dust and Republican tax policy, “a recent study” turned out to be too good to be true (well, except for that improved sense of well-being). A journalist set up a chocolate bar hoax study to show how much junk science was breathlessly reported by unquestioning newsfolks.
What does this have to do with Parkinson’s? Nearly every week, in my various Parkinson’s news feeds, I get headlines like “Scientists achieve breakthrough…” or “[Blank] Reported to Lead to Parkinson’s”. Wow, they’ve found the cause! Wow, they’ve found a cure! Not so fast…you can see it’s important to be a critical reader of science journalism.
So let’s go back to the chocolate bar study. What was wrong with the study? This hoax was put together by a journalist named John Bohannon (or as he called himself in the study report, “Johannes Bohannon, Ph.D., research director of the Institute of Diet and Health”.) He and his cohorts put out ads to recruit research subjects, got 16 persons, and randomly assigned them to three different diets: 1) Lo-carb, 2) Lo-carb with a bar of chocolate every day, and 3) whatever the person was eating now. They weighed themselves each morning for 21 days, and completed beginning and ending blood tests and questionnaires.
Results: Both of the treatment groups lost about 5 pounds over the course of the study, while the control group’s average body weight fluctuated up and down around zero. But the people on the low-carb diet plus chocolate? They lost weight 10 percent faster. Not only was that difference statistically significant, but the chocolate group had better cholesterol readings and higher scores on the well-being survey.
So what’s wrong with the study? Several things:
—Study is too small: To quote Bohanon: “If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people. (One subject was dropped.) That study design is a recipe for false positives.” Too small of a study is a perennial problem for Parkinson’s research, because of the difficulty of obtaining Parki research subjects. That’s why the Michael J. Fox Foundation did such a service for PD research in setting up Fox Trial Finder to link research studies with research volunteers (over 47,000 signed up so far!.)
—Study is too large: There was deliberately no focus or hypothesis in this junk study. Even with a larger number of research subjects, when you measure so many factors, one of them’s bound to show a pattern. The magic word in study design is “Reproducibility”. If you repeated the study, would you get the same result? Parkinson drug studies (or any FDA drug studies) require at least these three phases before a drug can be released to market, meaning favorable results will need to be reproduced:
Phase I: Safety — What is safe dosage range?
Phase II: Efficacy — Does the drug work?
Phase III: Efficiency — Does the drug work better than existing drugs for same purpose?
—Correlation does not equal causation: Hey, the people with the chocolate bars lost weight! Thus, chocolate must help you lose weight, right? Particularly with this small of a sample size (5 people), the chocolate may have had nothing to do with the weight loss. The classic statistics class example is: “All criminals drank milk as children, thus milk leads to criminality.”
Studies of what causes Parkinson’s and biomarkers to diagnose PD seem to be particularly prone to causation/correlation booby traps. I just received in my emailbox this headline: “People with depression may be more likely to develop Parkinson’s disease”. Hmmm…
–Does this mean depression causes Parkinson’s?
–Does this mean people who have Parkinson’s will develop depression?
–Does this mean depression might be a biomarker for PD?
–Does this mean it’s a coincidence that some people have depression and PD?
You know the answer…all of the above.
As it happens, this is a quite a large and well-designed study. The study description (below) indicates that the sample size was very large, there was an abundance of control participants, and researchers had the luxury of tracking subjects over time (25 years!). The last factor is particularly critical to a progressive disease like Parkinson’s.
For the study, researchers started with all Swedish citizens age 50 and older at the end of 2005. From that, they took the 140,688 people who were diagnosed with depression from 1987 to 2012. These people were then matched with three control participants of the same sex and year of birth who had not been diagnosed with depression, for a total of 421,718 control participants.
The participants were then followed for up to 26 years. During this time, 1,485 people with depression developed Parkinson’s disease, or 1.1 percent, while 1,775 people, or 0.4 percent of those who did not have depression, developed Parkinson’s disease.
Parkinson’s disease was diagnosed an average of 4.5 years after the start of the study. The likelihood of developing Parkinson’s disease decreased over time. People with depression were 3.2 times more likely to develop Parkinson’s disease within a year after the study started than people who did not have depression. By 15 to 25 years after the study started, people with depression were about 50 percent more likely to develop Parkinson’s disease.
–If it’s on the Internet, it must be true — Remember the “lead author”of the chocolate bar study? Johannes Bohannon, Ph.D., research director of the Institute of Diet and Health. Turns out the Institute of Diet and Health was an official sounding but totally phony website. The Internet is a double-edged sword, however. A little more Googling by the journalists breathlessly reporting this press release would have found out that the Institute was not quoted anywhere else in cyberspace – and Dr. Johannes Bohannon doesn’t exist. I Googled “chocolate helps you lose weight” and depressingly got nine entries before getting to Bohannon’s expose article. And oh yes, one of the listings was another study…with 1000 people!