Organic Foods Suck! ... says sucky study

Last fall, one of those deliciously contrarian studies came out that grabs headlines but doesn't deserve it. It turned out the flashy study from Stanford University was as much a case of false advertising as the organic foods it portrayed in that light. Annoyed by the study's flaws, I took it to task for The Aquarian. Here's a tweaked version of what I wrote followed by a brief update. 

From The Aquarian, Winter 2012

Organic Foods Suck: Study

Actually, the study does

By SYD BAUMEL

Like a tsunami of cold water on the good name of organic food, the story was all over the media early in September. You pretty much had to be living off the grid to miss it. The headlines were withering enough to dry out a rack of organic grapes:

"Stanford scientists cast doubt on advantages of organic meat and produce" (New York Times)

"Organic foods may not be much healthier" (National Post/Canada.com)

"Organic food no healthier than non-organic: study" (Reuters)

"New study finds scant evidence of health benefits from eating organic foods" (Globe and Mail)

Stanford University's PR department had spun the study for maximum media impact. What editor or columnist doesn’t love an emperor-has-no-clothes story? And when the emperor is organic food, for some it's a feeding frenzy:

"The organic fable" (New York Times columnist Roger Cohen)

"Stanford study shows organic food no safer or healthier than conventional food" (National Post columnist Marni Soupcoff)

"Study sticks fork in organic claim" (Washington Times)

Below the schadenfreude headlines, the details of the study – as spoon-fed to the media by Stanford and the study's media-friendly authors – unfolded. An ambitious meta-analysis (a study that pools the results of previous studies) of over 200 studies comparing organic and conventionally grown food mostly came up “meh.”

The study (published September 4 in Annals of Internal Medicine) reported scant statistically significant evidence that organic foods are nutritionally superior.

It claimed that organic produce was less likely to harbour pesticide residues, but only by 30%.

And it found that “the risk for isolating bacteria resistant to 3 or more antibiotics was 33% higher among conventional chicken and pork than organic alternatives.”

In other words, modest benefits for foods with less than modest price tags.

The take-home message was as predictable as the headlines. Senior author Dena Bravata, MD, summed up the sentiment in a much-reprinted quote from the press release: “There isn't much difference between organic and conventional foods, if you're an adult and making decisions based solely on your health.”

It wasn't until days and weeks after the tsunami had come and gone that well-informed critiques of the study began to surface. But by then the story was cold. The critiques seldom travelled further than minor blogs, low profile news releases or alternative media. To this day, most people have heard the bad news but not the rebuttals.

Those critics included many credible voices: New York Times food columnist Mark Bittman, veteran sustainability guru Francis Moore Lappé, Scientific American “Greengrok” blogger and scientist, Bill Chameides. One of the best critiques was written by Mother Jones food and agriculture columnist Tom Philpott. His column was my gateway drug to the study's own emperor-has-no-clothes issues. It led me to an even more penetrating take-down by Charles Benbrook, Ph.D., an agricultural economist and professor at Washington State University, and eventually to the study itself, ensconced behind an expensive paywall.

Misleading numbers

So what was wrong with the Stanford study? To my mind, grossly misleading numbers trumpeted to a scientifically naive public (and media) by a major university were the root of the study's evil.

Take those 30 percentish differences between organic and conventional food samples.

What if I told you that in multiple studies, pesticide residues have been found in 7% of organic food samples and 38% of conventional food samples. Which of the following would you consider to be the most meaningful way of communicating that difference to you?

• There's a 31% difference between organic and conventional foods (38 minus 7).
• Organics carry 18.5% of the pesticide risk of conventional produce (7 over 38) or, conversely, they're 82% less likely to contain pesticide residues (38 minus 7 = 31 over 38 = 81.5%).
• Conventional produce is 5.4 times more likely to contain pesticide residues than organics (38 over 7).

I hope you'll agree that the latter two examples, which unlike the first one calculate the relative difference between food samples, give you a much better idea of the practical, real world difference between organic and conventional foods.

Relative differences are the default language for reporting findings in scientific research. When you read that patients who took Drug A enjoyed 33% fewer heart attacks than patients who took a placebo, that's a relative difference. The raw numbers in the study might have been 30 people out of 1000 (3%) taking the drug had a heart attack vs 45 out of 1000 (4.5%) on the placebo. If the researchers had reported the absolute difference in percentages – 4.5% minus 3% – the headline would have been:

Heart Drug Flops

Ceminex cuts heart attack risk less than 2%

But that's how the Stanford scientists chose to report their results, making big differences seem meagre. The numbers I quoted earlier – 7% vs 38% – are the actual numbers the Stanford team reported in their study. So far so good. But then they subtracted the 7 from the 38 to arrive at the absolute difference of just 30% (not 31% because the numbers had probably been rounded). As Benbrook – a former Executive Director of the National Academy of Sciences's Board on Agriculture – critically observed: “Their seemingly unimpressive finding of '30% lower risk' corresponds to an overall 81% lower risk or incidence of one or more pesticide residues in the organic samples compared to the conventional samples.” Statistically, the odds of this organic advantage being insignificant (just a chance variation) were less than 1 in 1000, as the researchers themselves reported.

It doesn't stop there. The methodology chosen by the Stanford team ignored any reported differences in the pesticide concentration in samples (organic and conventional) that tested positive. Not having checked the studies they reviewed, I'm ignorant too. But knowing what we know about how liberally pesticides are used in conventional agriculture and shunned in organic farming, it isn't a stretch to speculate that in the 7% of organic samples that contained pesticide residues, the pesticide concentrations were much lower than those in the 38% of conventional samples that did. If the average contaminated orange from a conventional orchard contained, say, a total of 10 parts per million of two or three different pesticides, the average contaminated organic orange might have contained just one or two pesticide residues totalling 2 or 3 parts per million. Such differences weren't quantified in the Stanford study. But Benbrook, who knows the research like the back of his hand, says that when these differences are factored in, “the potential health risk of pesticide residues in organic foods compared to conventional foods typically averages 10 to 20-times smaller than that in conventional foods.”

That's a long way from 30% smaller.

Sweeping superbugs under the rug

And so it went for most of the study's other findings.

That 33% difference in antibiotic-resistant bacterial contamination? Absolute difference. Crunching the numbers reported in the study – 57 out of 358 (15.9%) organic samples contained antibiotic-resistant bacteria; 166 out of 343 (48.4%) conventional samples did – shows that, relatively speaking, the non-organic meats were three times more likely to be contaminated with antibiotic-resistant bacteria, again with odds less than 1 in a 1000 that the difference was due to chance. But armed with their absolute difference of just 33%, the Stanford team didn't even bother to comment on the health implications. And the news release all but blew the finding off, noting that “the clinical significance of this is ... unclear.”

The same release had branded the study as a doctor's quest to better serve her patients. The study, it stated, “stemmed from Bravata’s patients asking her again and again about the benefits of organic products. She didn’t know how to advise them.”

Well, how about advising them they have a threefold greater risk of catching a superbug from a conventionally produced pork chop or chicken wing? (She might also want to advise them that plant-based meat substitutes – beans, hummus, tofu, veggie burgers, soy hot dogs etc. – carry close to a 0% risk of being contaminated by antibiotic-resistant bacteria.)

And what about the health implications of those “30%” higher pesticide residues? Because most of the analyzed studies hadn't reported if or by how much the residues exceeded regulatory safety limits, the Stanford team essentially dismissed the clinical implications as “unclear.” But even if pesticide residues never exceeded officially safe limits for any single food, if every non-organic carrot, apple or bowl of soup Johnny eats has a 38% risk of containing some pesticide residues, it all adds up – perhaps to a level that clearly is unsafe. Ken Cook, President of the Environmental Working Group, had some pointed words on this subject for the Stanford scientists:

“Studies that have come out in the last two years have linked exposures to organophosphate pesticides with increased risks of ADHD and lower IQ in children and to low birth weight and early gestation among newborns,” Cook said in an EWG news release. “The authors of this study, for whatever reason, decided not to focus on this new and troubling research showing that a diet of food high in certain pesticides could pose such serious and lasting health impacts in children. That’s a glaring omission, in my opinion.”

The other big finding in the study was that only one nutrient of the 14 studied was significantly and unequivocally higher in organic foods: phosphorous. The researchers were probably right to write this difference off, because phosphorous is abundant in any diet and I've never heard of a little bit more equating to better health.

The Stanford team also reported weaker evidence that organic produce is better endowed with useful phytochemicals called phenols (resveratrol, the possibly life-lengthening compound in grapes and red wine, is a phenol), with omega-3 fatty acids in milk and chicken and with vaccenic acid (a fatty acid of uncertain healthfulness) in chicken.

Based on how the nutrition data was crunched and presented, the Stanford team reasonably concluded: “Despite the widespread perception that organically produced foods are more nutritious than conventional alternatives, we did not find robust evidence to support this perception.”

But neither did they find robust evidence that organic foods aren't more nutritious. Fourteen nutrients is a small fraction of the beneficial compounds found in foods. Even most vitamins and essential minerals – including ones like selenium and zinc whose availability to plants is dependent on the quality of the soil, an organic farming selling point – weren't included in the study due to insufficient data. Just a year earlier, researchers from the University of Newcastle analyzed pretty much the same literature and reported that organic produce is approximately 12% more nutrient-dense than non-organic – a small, but real (statistically highly significant) difference. Looking over the raw numbers reported in the Stanford study, one finds an echo – undiscussed by the Stanford team – of the Newcastle results. To begin with, although the differences in the Stanford study weren't statistically significant in most cases, organic foods were better endowed with 10 of the 14 nutrients. Non-organics beat on just 3 (there was a tie for potassium). Similarly, while 133 nutrient comparisons between organic and conventionally produced foods favoured the conventional samples, 199 favoured organic. Advantage organic: 50%.

So, in terms of nutrient density, the jury is still out on whether organic food is trivially or significantly superior to non-organic. Meanwhile, the pesticide advantage is confirmed. As for the credibility of the Stanford study, the jury is in.

Syd Baumel is editor of The Aquarian. He often pays extra for organic food, but mostly for the social and environmental benefits.

_______________________________

UPDATE, May 2013. In its February 19 issue, Annals of Internal Medicine published reader feedback to the Stanford study. All five letters were by scientists and one clinician, including Charles Benbrook and Kirsten Brandt (lead author of the Newcastle study described above), and all were critical. Among the new wrinkles:

Sari Lisa Davison, MD, wrote:

As an internist who relies on the Annals to publish articles that are free from bias and for which authors’ potential conflicts of interest are clearly stated, I was dismayed that Smith-Spangler and colleagues’ article on organic food did not indicate that some of the authors are affiliated with Stanford’s Freeman Spogli Institute for International Studies, which receives funding from agribusiness and agricultural chemical companies, such as Cargill and Monsanto.

In their response, the study's authors denied any funding from or financial relationship to organizations "that could be perceived to influence our published work."

Preston K. Andrews, Ph.D, of Washington State University wrote:

The authors also neglected to include a 2010 study (3) that compared organically and conventionally grown strawberries in California in which cultivar and environmental factors were meticulously controlled. This study found increased concentrations of vitamin C and total phenolic compounds, as well as higher antioxidant capacity, in organic strawberries. (For the sake of full disclosure, I am a coauthor of this study.)

The authors attributed the omission to a coding error on their part. But they questioned that the vitamin C difference in the other study would have been big enough to change the results of their meta-analysis, which found no significant difference in vitamin C concentration between organic and conventional produce.