Tuesday, June 10, 2014

Poll/Survey Results: Advocacy or Knowledge for Associations?

One of my high school math teachers told us “Figures lie and liars figure.”

Over the last 20 or so years, the U.S. has become a nation fascinated with polls and surveys. Several are released daily (I’m excluding political polls which are too many to count.) Many become stories in the news media.

Conducting and releasing polls or surveys is a great tool to advance your association’s profession/industry and/or the issues it is advocating.

I know. From Christmas trees and soybeans to flood relief and health care issues, I’ve used polling data for more than 25 years to generate publicity along the lines of issues we were advocating. 

The reality: all survey research or polls are biased!

  • Who conducted the survey? What were their motives?
  • What questions did the survey ask?
  • How were the questions (and response options) worded?
  • How was the survey conducted? Telephone (did the sample include cell phones)? Email (did the survey include those without emails)? An online panel? 
  • What sample was polled?  Was it random?  If an online survey, what about non-responders?
  • Which results were included in the announcement? All? Or, only those which favorably position the issue your organization is advocating?
A market research professional once told me: “Steve, you can’t ask the question that way ... it will bias your results.” I trusted her with future market research!

Over the last week, multiple news stories – based on “new research” – raised a lot of issues for me. Let’s look at some of them:

Tech: Where the women and minorities aren't

  • Google released its diversity numbers Wednesday after it (and most other tech firms) have spent years without disclosing such figures.
  • Just 1% of its tech staff are black. Two percent are Hispanic. The one well-represented minority group is Asians, who make up 34% of the company's tech workers. Eighty-three percent of Google's tech workers internationally are male. For non-tech jobs, the number is 52%.
  • The numbers are especially astounding for California, where 38% of the population is Hispanic, according to the U.S. Census Bureau. Asians make up 14% of the state.
Implied in the USA Today full story was that tech company staff in Silicon Valley did not reflect the population in surrounding communities. Well, dah?! That’s like saying a predominantly African-American NBA basketball team doesn’t reflect its fans.

Bird deaths from car crashes number in millions
  • A new study shows that crashes with cars and trucks kill as many as 340 million birds on U.S. roads every year — a much higher toll than bird deaths from many other human activities.
  • Sounds terrible, doesn’t it. Read deeper into the story:
  • To compile a nationwide estimate of avian road kill, the study's authors extrapolated from 13 small-scale surveys of birds that died after being hit by vehicles. The results show that 89 million to 340 million birds suffer fatal injuries from vehicle encounters annually. 
  • So, perhaps the death toll is only 89 million ... one-fourth the amount in the headline. The new study is "the most comprehensive analysis … to date," statistician Wally Erickson of Western Ecosystems Technology, an environmental consulting group, says. 
  • According to one study that Loss and his colleagues relied on, some 500 birds a year died on a two-mile stretch of highway through a Florida wildlife refuge. Many would've been tiny sparrows or warblers that drivers may not have even noticed.
Think of this “scientific” methodology: take 13 local studies and “extrapolate them into a national crisis. Really? We should base national policy on extrapolations of 13 small studies that results in bird deaths with a 251 million variance?

Termites, climate change investigated
  • “What termites and fungus eat locally may play a bigger role in global warming than scientists think, according to a new study. How quickly dead trees rot strongly influences how much carbon stays in forests to offset the carbon released to the atmosphere from fossil fuels and other sources. That makes wood-rot rates crucial in detecting potential climate changes. But those rates are a big uncertainty in climate models because of scant data.”
  • Dig deeper and you’ll discover the “research” methodology: “During the 13-month experiment, the researchers distributed 160 blocks of pine tree wood across five sub-regions of temperate forest in the eastern U.S.”
Two other polls came via Twitter today.
I have not bothered to review them to see whether they are extrapolations or biased polls but the headlines make me wonder!
  1. "New poll reveals almost 50% of Americans under the age of 35 have been harassed online  depressing but important info."
  2. "Of those who use Twitter, 94% consider it most valuable blog.”

Two take-aways for association professionals:

  1. Consider how you can use survey research and public opinion to help advocate issues for your profession or industry. Note: you may want to review my post on this topic: Should associations use faux research to advocate a cause? 
  2. Carefully review methodology before accepting the various polls and surveys you see or read or hear.

No comments:

Post a Comment