March 3, 2008

The Polling Follies

By James H. Miller

The day before the Wisconsin primary a number of polls suggested that Hillary Clinton was closing ground fast. One poll showed Hillary Clinton had assumed a 49% to 43% lead. The next day Barack Obama absolutely hammered Hillary Clinton. Obama won by 17 points, which meant that the poll that had showed him losing by 6 points was off by an astonishing 23 points. In this political year the media and the pundits continuously blame pollsters for having bad numbers. The criticisms started after New Hampshire and continued into the results on Super Tuesday. The pollsters answered with the difficulty of polling the primaries, the problem of cell phones, and the fact that turnout figures were hard to predict in a primary election.

All these alibis are true. I have been polling in Wisconsin for 21 years and for 15 years before that. I know a little bit about this business. One point no one wants to discuss is that most media and political pundits do not have the slightest idea of how polls are put together or how they work. One of the explanations for the dreadful polling results this year is that a lot of the companies that do polls are totally incompetent. When you miss an election by 23 points within two days of the election, there are no excuses. You should not be in the business.

Over the last several years one growing trend in survey research is that anybody can run a poll and have the media cover it. In all the years I have been in Wisconsin, no one has ever asked me specifically how our poll was put together, how it actually worked, where the phone calls were made, etc.? Nobody cares. They just want a headline, and if the headline isn’t true they can just explain it away with either “it was a snapshot,” “margin of error,” or whatever.

The reality is that polling is an art and a science that is, in fact, changing, but the expertise of those in the business is almost comical. Few pollsters are academically trained. They drifted into the business because it has become more and more profitable, especially in regards to corporate surveys. Almost all the companies you see running political polls do it as a sidelight to simply get their name in the paper. They don’t care if the data is wrong, and neither does the media. My favorite now is that a number of pundits have discovered if you average all the polls together it gives you a clearer picture of what is going on. Really? How does one average a poll that is 23% wrong with any other poll?  What the media may have to actually do, if they want to get serious about reporting polls, is to examine who is doing the poll, how much experience they have, and candidly, how often are they right.

In our December 2007 poll we released data that showed an enormous movement in the population of Wisconsin toward declaring themselves politically Independent, They strongly disagreed with the direction of the country. They also thought that the economy was becoming the major issue in Wisconsin. Several months later these results foreshadowed the primary results in Wisconsin. Polling is a sophisticated art and a science that will face some very serious problems over the next decade. But it is still a business where we should demand accountability from the people who are doing it. The media should be examining the qualifications of those doing the polls to see if their results merit the attention of the general population.