UNTHINKING: The Meaning of Research

The Off-Topic forum for anything non-LDS related, such as sports or politics. Rated PG through PG-13.
Post Reply
_moksha
_Emeritus
Posts: 22508
Joined: Fri Oct 27, 2006 8:42 pm

UNTHINKING: The Meaning of Research

Post by _moksha »

This is from Bill Killpatrick, one of the philosophers of the Mormon Issues forum on Beliefnet.

UNTHINKING AS A UNIVERSAL TRAIT

This "unthinking" is a problem both in and out of the Church, one which LDS people face all the time, particularly when dealing with people who already know everything and can't be bothered to examine anything for themselves.

Currently, there's a wholescale dismissal of the Book of Mormon, on the basis of genetics, and every time a new report comes out - lauding the dietary effects of coffee or tea - Mormons end up taking it on the chin.

But not all that glitters is gold. Consider the headline, "Diet Soda May Carry Heart Risks." For many, the headline alone is enough to generate lots of water-cooler buzz. Until the next study comes out, we'll see lots of talk about how a can of diet soda can drop you like Mike Tyson.

But if we bother to read past the headline, we get this:

"(CBS/AP) People who drank more than one diet soda each day developed the same risks for heart disease as those who downed sugary regular soda, suggests a large but inconclusive study."

Whoops! While the headline says that diet soda "marry carry heart risks," the article, itself, just says that people who drink diet soda developed "the same risks" as regular soda drinkers. What a difference! And then, to top it all off, the last line sources this to "a large but INCONCLUSIVE study."

It gets better.

"The results surprised the researchers who expected to see a difference between regular and diet soda drinkers. It could be, they suggest, that even no-calorie sweet drinks increase the craving for more sweets, and that people who indulge in sodas probably have less healthy diets overall."

Did you catch that? Researchers "expected to see a difference" between regular and diet soda drinkers. They set up their study to prove that. It didn't. And yet, these same researchers - operating under a grant - are now marketing their results as if they'd actually proven something else!

Let's go back to 7th Grade Science. Remember that little thing called The Scientific Method? Researchers look at a phenomenon, construct a hypothesis, then design a test. Designing the experiment is the hardest part. Just administering it is no big deal, but making sure it actually proves something is a completely different matter. Ideally, the test should eliminate all other factors and/or explanations. The researchers' explanation should be the only reasonable one to accounts for the data.

If the test data prove the hypothesis, the next step is to have the experiment validated through peer review. Legitimate scientists publish their data within the scientific community - such as through a refereed journal - inviting other scientists to test their data. If you can't get validation through repeated results, you don't have science; you have Cold Fusion.

That's why this study is such a joke. The researchers looked at a phenomenon (obesity), hypothesized that soda was the/a culprit, devised a study to show just that (regular soda versus diet soda), ran their study and were "surprised" that the data didn't support their hypothesis. Having shot the wad on a study that didn't pan out, there's no way to peer review a failed study. So, instead of releasing their data to the scientific community, these jerks have held a press conference - a surefire indication that they're selling a carnival ride, not science.

But there's more:

"The study's senior author, Dr. Vasan Ramachandran, emphasized the findings don't show diet sodas are a cause of increased heart disease risks. But he said they show a surprising link that must be studied."

Isn't it funny - even after the study's "senior author" went to the trouble of ephasizing that "the findings don't show diet sodaas are a cause of increased heart disease risks" - that CBS, in its quest for journalistic integrity, would headline the article, "Study: Diet Sodas May Carry Heart Risks"?

But even if CBS ignored the content of the report to drum up something sensational and inaccurate, what self-respecting scientist would brag about a failed study and then say his results "show a surprising link that must be studied"?

Let's get it straight. I'm no scientist, but I at least passed 7th Grade Science and I at least know that there's a huge difference between proving X and arguing, because you failed to prove X, that the answer is "not X." Until this study, these guys were so sure that sodas were a major factor in obesity and heart disease. Now, because their test results flatlined, they think they've shown the opposite, "a surprising link that must be studied."

But wait, there's more:

"It's intriguing, and it begs an explanation by people who are qualified to do studies to understand this better," said Vasan, of Boston University School of Medicine.

I love that part where doctor dougnuts is saying this matter should be taken up "by people who are qualified to do studies." Damn skippy! Anybody but these guys!

Maybe that's why, further down the story - and nowhere even hinted at in the lead - we get this:

"However, a nutrition expert dismissed the study's findings on diet soda drinkers. 'There's too much contradictory evidence that shows that diet beverages are healthier for you in terms of losing weight that I would not put any credence to the result on the diet (drinks study),' said Barry Popkin, of the University of North Carolina in Chapel Hill, who has called for cigarette-style surgeon general warnings about the negative health effects of soda."

And this:

"Susan Feely, president of the American Beverage Association, said the notion that diet drinks are associated with bulging waistlines defies common sense. 'How can something with zero calories that's 99 percent water with a little flavoring in it ... cause weight gain?' she said."

It's only at the bottom of the story that we actually get any real details. This is the stuff the average consumer of news is not ready to digest:

"The research comes from a massive, multi-generational heart study following residents of Framingham, Mass., a town about 25 miles west of Boston. The new study of 9,000 observations of middle-aged men and women was published Monday online in the journal Circulation."

"The researchers found those who drank more than [one]soda per day — diet or regular — had an increased risk of metabolic syndrome, compared with those who drank less than one soda. Metabolic syndrome is a cluster of symptoms that increase the risk for heart disease, including large waistlines and higher levels of blood pressure, blood sugar, cholesterol and blood fats called triglycerides."

In other words, BU got a grant to study the effects of soda on metabolic syndrome. It set up a study in Framingham, 25 miles away, made "9,000 observations" of men and women and found a correlation between those who drink more than one soda per day and "metabolic syndrome" - which is basically "large waistlines, high blood pressure, high blood sugar, high cholesterol and blood fats."

The problem, of course, is one of cause and effect. It's one thing to say that big, fat, heavy people are bit fans of soda. It's quite another to say that soda makes you a big, fat, heavy person.

"At the start of the study, those who reported drinking more than one soft drink a day had a 48 percent increased prevalence of metabolic syndrome compared to those who drank less soda. Of participants who initially showed no signs of metabolic syndrome, those who drank more than one soda per day were at 44 percent higher risk of developing it four years later, they reported."

This is the real grit of the study. At the beginning of the study, soda drinkers are 48% more likely to end up with metabolic syndrome; of those who didn't have it going in, the soda drinkers were 44% more likely to have it four years later. But what does that mean?

"Researchers expected the results to differ when regular soda and diet soda drinkers were compared, and were surprised when they did not, Vasan said."

Here's the kicker. BU set up this four-year experiment, conducting "9,000 observations of men and women," hoping to show a link between sugary soda and metabolic syndrome. Diet soda was supposed to help prove the impact of sugary sodas because if the diet-soda group had a lesser prevalence of metabolic syndrome, it would show what an impact sugar - particularly corn syrup - plays in creating metabolic syndrome.

But the study failed. Instead of showing a difference, it showed no major difference. Does this mean that diet sodas - which have no sugar - contribute to metabolic syndrome, or does it simply fail to show that soda is the culprit they're after? Even further buried in the article is the following paragraph:

"But Popkin said that result isn't that surprising. He said much of the market for diet sodas are people who have unhealthy lifestyles and know they need to lose weight — with the other portion being thin people who want to stay that way. That means many people drinking diet sodas have unhealthy habits that could lead to increased heart disease risks, whether they drink diet soda or not."

Did you hear a BOOM as the whole study fell apart? In designing this study, its "authors" never accounted for the possibility that diet soda drinkers would skew not just the fitness-conscious but people who were already at great risk for metabolic syndrome.

Buried even deeper in the article is the fact that this study is at variance with others:

"In studies in which some users were randomly given diet sodas and others were given regular soda, diet soda drinkers lost weight and regular soda drinkers gained weight, Popkin said."

That's why this study was conducted - to prove (with better data) what other studies have suggested (that sugary sodas are a factor in weight gain). But having failed to prove that (since diet-soda drinkers didn't fare better), it's a little late to now suggest that diet sodas play a link in heart disease (as previous studies show benefit, not harm).

"In a statement, the American Heart Association said it supports dietary patterns that include low-calorie beverages. 'Diet soda can be a good option to replace caloric beverages that do not contain important vitamins and minerals,' the association said, adding further study is needed before any association between diet soda and heart risk factors would lead to public recommendations."

Clearly, the authors of this study are out on a limb, as nobody is really standing by their newly-tweaked results. Not surprisingly, the best they can offer is more idle speculation:

"Vasan also said poor overall health habits may be one reason diet soda drinkers did not show lower heart disease risks in the Framingham study, but there hasn't been enough research to say for sure. Another possible reason is a controversial theory called 'dietary compensation,' which holds that if someone drinks a large amount of liquids at a meal, they aren't satisfied and will tend to eat more at the next meal .... Other theories, Vasan said, are that people who drink a large amount of sweetened drinks are prone to develop a taste for sweeter foods, or that the substance that gives soda its caramel color promotes resistance to insulin, which is needed to process calories."

Okay, so imagine this. You conduct a four-year study, intent on proving X. When that doesn't happen, you suggest it means "not X." In the meantime, as a substitute for proving one thing, you offer three competing possibilities. That's science?

"Without a more definitive explanation, Vasan offers only this advice to diet soda drinkers: 'Consume in moderation and stayed tuned for more research.'"

Think back that to the article's title, "Study: Diet Soda May Carry Heart Risks." Does that really seem like an apt title for an article where the study intended to prove one thing (sugary drinks cause metabolic syndrome), failed to do so (diet drinks fared no better), has been spun to suggest a new possibility (diet drinks have a link to metabolic syndrome), which is contradicted by nutritionists, the American Heart Association and previous studies and for which the study's senior author can only say, "Stay tuned?" It's a mess. If there's any real science in there, I challenge anyone to find it with a flashlight and a map.

But this didn't prevent CBS from trying to bolster its headline with a completely irrelevant claim:

"Meredith Weise told CBS News medical correspondent Dr. Jon LaPook she was hooked on soda, drinking eight glasses a day. 'My blood pressure went high and I felt very dehydrated,' Weise said. But she has cut her habit and is now down to three or four sodas a day, dropping 53 pounds in the process."

Does this really support the headline? I thought this was supposed to be an article claiming "DIET Sodas May Carry Heart Risks." This is a testimonial about what happened when somebody drank a lot of sugary soda, and then stopped. If we're not talking about DIET SODA, what difference does it make? Dr. Jon LaPook - CBS's "medical journalist" is no better than Sanjay Gupta. He's just a guy with a medical license and a camera, putting his licensed face onto AP news items. Apparently, "unthinking" is second-nature to him, as he used a testimonial about sugary sodas to end a report about diet sodas.

What morons!

WHAT DOES THIS ALL MEAN FOR Mormons?

The CBS hatchet job on that little bit of pseudo-science is not meant as a dismissal of good science, or good journalism. Sometimes, when discussing issues with people of goodwill, you have to concede that the facts are against you, though you hope - as faith would have you do so - that the facts will someday be more supportive of your position.

But it's a great example of how stupid people are, including educated people using billion-dollar megaphones - whether we're talking about BU or CBS. There's a tendency to package and simplify to meet someone's interests or agenda. It's a cold, hard, reality of the information age.

Americans want to eat junk food and drink gallons of sugary soda. They also want to sit on their hinds and bulge like balloons.

Corporate America, knowing the profit margin of salty snacks and sugar water, wants to sell this junk to our children, who are turning obese at remarkably younger ages.

Health advocates want to get rid of the junk food, or at least scale it back. They want the FDA to get involved. They want more labels. They want more restrictions. They want the snack machines out of our schools.

Legislatures and school boards want campaign contributions, and there's simply a lot more to be gotten from the snack and beverage industries than from tree-hugging granolas.

That means the health activists need data, hard data, scientific data, to show a public health crisis - and clearly identifiable culprits.

In the meantime, universities need money, and they know that one way to get it is to advertise themselves as research centers. Soliciting grants from public and private sources, they are in the grant-writing and research-conducting business. And, like gun runners, they play both sides of the fence, taking money wherever they can get it.

Not surprisingly, both sides of the junk food wars are working the research-industrial complex, funding studies each hopes will give their side a leg up. If you want to be a partner at a major law firm, you got to be a rainmaker. If you want to be a fellow of a major research institution, you've got to bring in the grant money. It's the same principle at work.

This is not to say that all studies are to viewed through urine-colored glasses but that it's an act of innocent naïvété to think all these studies are simply being done because America loves science. Prince Henry of Portugal was known as "the Navigator" because of his navigational college - but his motives had less to do with pure science than with replacing the Italy as the trading center of Europe.

With science, as with lots of speculative projects, the rich get richer and the poor get poorer. You're only as good as your last grant award. Grant money flows to those institutions which produce the most useful results. It isn't "Science" so much as "$cience." If you take in millions of dollars of grant money, you'd better come up with useful results or you're a dry well - and you'll have a hard time getting more money to run with.

Not surprisingly, most grants "produce" something - even if it's just a carefully orchestrated headline. Even after trashing its own reputation, the University of Utah still made money off of "cold fusion."

So, if you're BU, and you've just spent four years studying the effects of soda on "metabolic syndrome," you'd better have something to report. And if you can't show that sugary sodas are worse for you than their dietary counterparts, maybe it's time to suggest "Diet Soda May Carry Heart Risks."

It's a risky maneuver, since it will open you up to widespread ridicule. What is it you people do over there, predict the future by reading chicken entrails?

But if that's all you've got, it's your move to make.

In the meantime, CBS has to find something to get the public to "stay tuned," even if it means doing a hatchet job on a study that would dubious to begin with. Are these "journalists" so stupid they can't see the holes in their own story? Possibly. (I was a communications major for 9 weeks, just long enough to realize I was surrounded by morons.)

More likely, these chicken little stories are exactly what they're supposed to be - the closest thing network news gets to stories about Bat Boy.

Fair enough. Let the viewer beware. And let him laugh as well as we get Pinnocchio's rendition of the evening news:

Pinnochio: "Soda may carry heart risks?"

Viewer: "Really?"

Pinnochio: "Well, according to a BU study that's inconclusive."

Viewer: "They actually found that diet soda hurts your heart?"

Pinnocchio: "Not quite. They were actually trying to show that sugary sodas are a factor in metabolic syndrome, which includes a number of maladies - high blood sugar, high blood pressure, high blood cholesterol, et cetera."

Viewer: "But that was for sugary sodas."

Pinnocchio: "Right. The study distinguished between those who drink sugary sodas and those who drink diet sodas. The diet-soda drinkers were supposed to be so much better off. But when the study failed to show that, the researchers decided this means that diet soda is just as bad for you as sugary soda."

Viewer: "But they didn't actually prove that sugary soda is bad for you."

Pinnocchio: "No, they were trying to but diet-soda drinkers didn't do better, so they concluded that there could be a link between diet soda and heart disease."

Viewer: "How could diet soda have anything to do with heart disease? How does something that has no sugar and is 99% water pose a heart risk?"

Pinnocchio: "They didn't say it did. They said it raised an interesting question that should be studied by those who are qualified to look into it."

Viewer: "But aren't they the researchers? Weren't they qualified to look into it? Wasn't that what they were trying to do in the first place?"

Pinnocchio: "No, they were trying to show that sugary drinks are harmful, and they were using diet drinks to do that."

Viewer: "But they didn't."

Pinnocchio: "Which is why it's so intriguing. Clearly, there is a link between diet sodas and heart risk that is just begging to be studied."

Viewer: "Where's this link? How do you show anything from this study?"

Pinnocchio: "You can't. It's inconclusive. Which is why we need more money for another study. In the meantime, drink in moderation and stay tuned."
Cry Heaven and let loose the Penguins of Peace
_moksha
_Emeritus
Posts: 22508
Joined: Fri Oct 27, 2006 8:42 pm

Post by _moksha »

This really is a worthwhile read.
Cry Heaven and let loose the Penguins of Peace
Post Reply