Yesterday, Childhood Obesity News went into more detail about Dr. Pretlow’s reflections on the two (so far) groups of young people who tried out the W8Loss2Go smartphone app. They started by filling out applications and motivation questionnaires, but Dr. Pretlow’s team “found little correlation with the results of this questionnaire and the success with the app program.”
The obese youth who signed up were asked to rate some questions on a 1 to 10 scale. Two of the questions were “How much does being overweight bother you?” and “How committed are you to losing weight?” and most of the applicants answered both with 10 — the highest. And that is probably how they felt at the time.
But later on, once the study was underway, it appears that many of the kids “would not admit what they were actually eating, adamantly claiming they were eating only small amounts, in spite of the fact that they continued to gain weight.”
The perils of self-reporting
All studies are different in their particulars, but the ones that involve what are called self-report measures are susceptible to problems that impact the validity of whatever conclusions are drawn. The first problem with self-reporting is that it’s subjective, a condition that is the very antithesis of scientific objectivity.
Basically, what self-reporting means is that the researchers are taking the subjects’ word for it, and this can go wrong in a surprising number of ways, all of which were catalogued by Rob Hoskin, a PhD student in Sheffield University’s neuroscience department. We mention a few of them here, with the strong recommendation to refer to the source article for some eye-opening concepts.
A range of choices designated 1 to 10, for instance, provides for a more nuanced response than a simple yes-or-no vote. But unless a study is carefully designed, the rating scale tool can be as much a liability as an asset, because people tend to interpret the meanings of the increments differently.
People often experience a conflict between honesty and image management, which can come into play strongly when sensitive issues are involved. Sexual behavior and drug use are two examples that Hoskin gives, and no doubt eating habits are a similarly loaded topic. He warns that “it is unwise to assume, even when you are measuring something relatively benign, that participants will always be truthful.”
And of course, people sometimes lie even in an anonymous survey! So who knows what they might do when their name is attached.
Then there is a thing called “response bias,” where the participants very much wish to be honest, but all their training and experience has taught them what the answer “should” be. For example, how many parents would admit that there have been times when they wanted to put their kid in a duffel bag, tie the top securely, and drop the bag into the nearest body of deep water? How many parents would admit to having that impulse even once? Even in a totally anonymous questionnaire?
Adding another layer of complexity, subjects who wish to be totally honest may not have enough introspective ability to make sense of their own responses or behavior. Truer words were never spoken than, “Undoubtedly we are all to some extent unable to introspectively assess ourselves completely accurately.”
Your responses and feedback are welcome!
Source: “The dangers of self-report,” ScienceBrainWaves.com, 03/03/12
Source: “What Is an Anonymous Survey?” Wisegeek.com, 01/23/14
Image by Sergey Vladimirov