Half a truth is often a great lie. Benjamin Franklin
A few weeks ago I conducted a survey (see questions here) and subsequently published a blog that featured some commentary on the responses. At the time, I had not fully analyzed the results and my assessment was preliminary. Some of the questions were not addressed at all, nor were the flip sides of some of the percentages. None of the readers got the full story. For that matter, neither did the author.
Seriously. Did I know better? Heck, yes. I’ve looked at hundreds of surveys, many in a professional capacity, and I understand how they roll. I would like to say I did it to you on purpose, as a social experiment. As mentioned, I had a point I wanted to make, but the fact is that I learned a lot while looking at the data more closely. Just as I warned of life safety concerns when there’s a lack of awareness, there are also issues when you just swallow what you’re handed. Without looking at it. Without chewing.
To illustrate, we’ll go back to the impetus for my survey, the correlations between some individuals’ choices and their skillsets. In this instance, take a look at how different my husband and I are in approach and preferences. And talents.
Andy
Gayle
- Prefers Geometry
- Enjoys looking at paper (or electronic) maps
- Never gets lost (ok, rarely)
- Loves sweets
- Prefers robust bread
- Loves sub shop sandwiches, cold ones
- Doesn’t drink tea or coffee, but loves juice. It’s sweet, you see.
- Loves watching sports
- Never plays board or card games
- Eschews Geometry; loves Algebra
- Prefers GPS (with a British voice)
- Was often lost in the 20th century, when forced to use maps
- Loves salty snacks
- Wonder bread is perfect
- Avoids cold sandwiches when possible
- Known to drink coffee and lots of tea. Lots. Juice only at breakfast.
- Loves to sleep to the sound of sports
- Loves card games; tolerates others
There HAS to be a pattern that I haven’t found a way to prove. That is, are brains wired to go in one direction or the other with skillsets as basic as different types of math and visual skills? Since I have admitted that I lean to drawing correlations, I was eager to see if I could arrive at conclusions that are more widespread than my four little walls. That skews the results – just that, all by itself. Not consciously, but it had to be a factor. Also, while it is a given that my survey would have achieved more accurate results with better designed questions, there are other issues.
As explained by the National Library of Medicine, survey summaries are fraught with error, whether intentional or otherwise. No matter who you are, there will be a number of recipients who do not respond at all. The conclusions drawn from those who participated cannot be assigned to those who did not complete the survey, even if a 75% response rate exists. This fact is even more critical with the survey I sent out, since it leaves me wondering whether the lack of response is yet one more value to consider. That is, what if those who completed the survey are not fully representative of any subset of the population?
That makes two issues. The first was a predisposition to find what I was looking for and the second was a lack of credibility. I didn’t have enough responses to generate a useful result. But let’s say 500 people responded. What then? What if all of them went to the same high school? What if the Algebra teacher was a star and the Geometry teacher longed to teach at a university and was unmotivated in front of 30 sleep-deprived sophomores? Could the quality of the experience impact the results years later?
Then there are the questions. The answers you see are a function of the questions you ask. A variation on the classic is “How long have you been enjoying your co-worker’s lunch?” Obviously, there’s no good way to answer that one. A realistic example this time around was when I asked about geometry. Not everyone took that course and there was no way to get around the question.
But that’s not all. It’s what you don’t say. In my survey, 10 of the responders who preferred Heinz catsup also preferred Crime dramas. Perhaps they had issues with their precious little psyches? No. Well, that isn’t proof, anyway. Of the people who watch TV (not everyone does, actually, and a few prefer HGTV), 7 preferred Heinz. That’s 7 out of 10 and a majority, and I could word that as a large majority when reporting results. I wouldn’t be wrong. I could go a step further and say that all Hunt’s catsup lovers prefer sweets, using their GPS over a map, and high energy music, but it’s 2. Just 2 people.
Omitted data can be as crucial to the report as the detail included. When reviewing survey results, it is wise to take them with a grain of salt. And cynicism. Even when no malice is intended and fraud is not a factor, the end product can easily be flawed. Imperfectly designed questions or lack of diversity in the distribution, or in those who actually respond, can easily distort the bottom line.
You’ve heard the old joke about the guy who sees a man standing next to a dog and asks him if ‘his’ dog bites. The man says he doesn’t. The newcomer is subsequently bitten because he didn’t confirm that the dog lying there was, in fact, owned by the man he asked.
Am I saying you should ignore every, “Survey says”? No. But when I’m being sold a bill of goods, I try to make sure I understand what I’m buying and whether there’s a curtain I need to pull back. When I walk in the woods I am more concerned about the snakes I don’t see than the ones I do.
You?
I kept thinking “polls”! Maybe because on the rare occasion the TV is on, a poll is being reported. There are so many variables when questioning and so many ways to interpret results. One thing I remember from a stats class was, how to lie with statistics
I know. I am not sure there’s any way to run a survey/poll without impacting the results with the way the question is worded or the way the answers are ordered or even the way the survey is delivered. And that’s when people don’t actually intend to mislead the respondents. I got nothin’.