Credibility Watch

One of the good/bad things about blogs is that it's all in writing, so you can often go back and see what was written in the past by various people. Occasionally I like to take advantage of this and cherry pick some recent event, where I was right in some prediction and others were wrong, in order to highlight my superior predictive abilities.

For instance, here’s a recent post by Andrew Coyne regarding the polls leading up to the B.C. election – the post was written before the B.C. election took place, so now that the election has happened, we can assess whose interpretation of the polls was more accurate:

Andrew wrote:

I dunno. This makes no sense to me:

"Halfway through the B.C. election campaign the race for the leadership is tighter than ever. The BC Liberals and New Democrat parties are separated by just three percentage points, according to a new Angus Reid Strategies/CTV poll.
The online survey of 822 voters found the BC Liberals are holding top spot with 42 per cent, with the opposing NDP not far behind at 39 per cent. The Green Party remains in third place with 13 per cent."

The same poll shows that, by a wide margin, the public thinks the economy is the number one issue. And that, by a margin of three to one – 48% to 16 — they pick the Liberals’ Gordon Campbell as the best person to handle it. Nearly two to one — 40% to 23% — also prefer Campbell as “best premier.” Yet the Liberals are only three points ahead?

Going into the campaign, the Liberals enjoyed a 17-point lead. Nothing much has happened since. The NDP have run a scattershot, unfocused campaign. And suddenly they’re only three points behind?
Colour me skeptical.

And then in the comments:

dan in van says:

Smacks of a push-pull that was manipulated, whether accidentally or purposely, to ensure that Liberal forces are motivated to vote. It is the only poll in a few years that have shown this kind of narrow margin. But while my theory may sound a little conspiracy-like, in any event I take this with a long tall one.

Critical Reasoning says:

I never trust online polls like this one. Here’s the justification by CTV/Angus Reid:
Our polls are conducted online, and not over the telephone or through face-to-face interviews. The rationale behind this approach is simple. Canada is now one of the most connected countries in the world, and Internet access has spread to every key demographic group in the country, including seniors and people with below average household incomes.

Conversely, telephone-based research has been greatly affected by declining response rates and the absence of land-lines in the homes of many younger Canadians.
Codswallop! Online polls are only used because they’re cheap. Almost any online poll is more likely to magnify the young, left-leaning “can’t be bothered to vote” demographic at the expense of the older, more conservative demographic who always vote, but only venture online to check email.

Derek Pearce says:

Heh. Where can I online-vote that Unicorns are the prettiest of all the fabled creatures? Actually I bet that you’d get a better than 19 out of 20 result for that, so it’s a bad example. What was the methodology of survey even if online? If people had the option of clicking one party or another from a public site and meandering off, then the NDP will be over-represented by zealots who did their quick-click duty and then went web-surfing elsewhere, while those who cared stayed to answer more in depth questions will more accurately reflect the public opinion of Campbell.

bigcitylib says:

A response to the speeding scandal, which arrived plop in the middle of the polling period?

As for AR’s online polls, I’ve complained alot about them but during the last Federal they did as well as (maybe better than) anyone else’s.

catherine says:

These results are only a small shift from the last Angus Reid poll (done in March which put the Liberals at 43 and NDP at 37) and the small shifts could be fluctuations in small numbers (MOE) or due to the speeding scandals.

Mike G says:

Well, no one ever accused British Columbians of being sensible.
Here’s the thing, though: I would contribute to those statistics. I think Gordon Campbell is probably a better administrator than Carole James. I am not entirely happy with either platform; the economy is certainly the biggest challenge ahead, but I don’t actually think that the provincial government can do much about it. I’m very unhappy with a lot of the things that the Liberal government here has done over its current tenure, I think there’s been a lot of hamhandedness and ignorance on their part, and I think a bit of shakeup could be good for them. On top of that, my local MLA, Shane SImpson (NDP, Vancouver-Hastings) seems to be just fine and I see no need to boot him.

Dale says:

The 17 point lead poll is a Mustel poll. This company was started by a current MLA for the Liberals. If any poll has a tendency to bias it is that one.
The A. Reid poll is very close to Robbins Reasearch poll wich uses telephone polling. You can complain about one but it is backed up by the other. The BC election will be won or lost in each riding. Liberals are strong is some but the NDP are equally strong in others. Who ever gets the vote out will win.

Declan says:

Libs won by 4% last election. Seems pretty plausible to me that things haven’t changed much since then. I guess we’ll see soon enough.

---


Election result:
Liberals get 46.02% of the vote, NDP gets 42.06%, Liberals have a margin of victory of almost exactly 4%.

Credibility Increase:
Angus Reid, Me, Catherine, bigcitylib, Dale and Mike G.

Credibility Decrease:
Mustel, Andrew Coyne, dan in Van, critical reasoning and Derek Pearce

That is all. Just trying to enforce a little accountability around here...

Potpourri

While researching my post on polls I came across this discouraging poll which estimated that 44% of Canadians believe the Federal government is running a deficit - up from 37% two years ago! Yikes.

Of course, when was the last time you saw the results of any poll / survey / test / whatever and your reaction was, 'wow, people really seem to be informed about what's going on'? There seems to generallly be a wide gap between what people know and what we (I?) think they should know.

---------------------------------

It's been a while since I linked to 'No More Shall I Roam', but the last two posts there are both worth a look. This one on why we need science to be able to truly understand the world around us is a gem.

Also, not only does Jonathan read Ezra Levant's column so you don't have to, he even writes about it afterwards. It seems like a lot of effort to devote to someone so lacking in interesting ideas, but it makes for a fun read. The latest episode is worth reading not so much for the Ezra related content but rather for how Jonathan picks apart the claim that Alberta's provincial spending is up 90% since 1996. Why use 1996 as a benchmark you ask? So does Jonathan.

The graphs and discussion which follow in his post are a great demonstration of why your pundit sneak-o-scope should immediately go off any time you see a claim that Item X has changed by Y% since year Z - especially where Z seems like an arbitrary choice.

For example, 'airlines are a great investment because air travel is up 50% since fall 2001.', or 'The number of tourists visiting Montreal has been flat with ony a 5 increase since 1976', or 'The tech era is over with tech stocks down 30% in value since mid-2000.' I could go on.

-----------------

Bringing together the themes of ignorance and people getting more attention than is warranted, I have to say I was left typeless by Time magazine's embarrassing decision to feature right wing hatemonger Ann Coulter on their cover. Luckily Eric Alterman wasn't and neither was Digby:.

"Ann Coulter is not, as Howie Kurtz asserts today, the equivalent of Michael Moore. Michael Moore is is not advocating the murder of conservatives. He just isn't. For instance, he doesn't say that Eric Rudolph should be killed so that other conservatives will learn that they can be killed too. He doesn't say that he wishes that Tim McVeigh had blown up the Washington Times Bldg. He doesn't say that conservatives routinely commit the capital offense of treason. He certainly doesn't put up pictures of the fucking snoopy dance because one of his political opponents was killed. He doesn't, in other words, issue calls for violence and repression against his political enemies. That is what Ann Coulter does, in the most coarse, vulgar, reprehensible way possible.

Moore says conservatives are liars and they are corrupt and they are wrong. But he is not saying that they should die. There is a distinction. And it's a distinction that Time magazine and Howard Kurtz apparently cannot see."

This Post Worth Reading 19 Times Out of 20

With B.C. heading to the polls in one month (vote yes!), and Federal politics in a state of uncertainty, potentially leading to an election sometime this year, I thought this would be a good time to add another dropdown box, this time with Canadian Polling firms in it (if I missed any important ones, drop me a line).

Like with many topics, it seems, I have mixed feeling about polls. On the one hand, I'm a numbers person with a bit of a stats background, so I enjoy digging into poll results to see what can be made of them. On the other hand, I find that polls feed into the 'horserace' mentality in which all people care about is whose numbers are good and whose are bad, rather than looking at actual issues. Furthermore, because the general population often seems to be a little lacking in its ability to interpret numbers, statistics tends to be an area where unscrupulous people will try to manipulate the numbers to suit their point. And in no area of statistics is this more true than in regards to polls.

The most common way to manipulate a poll is through the structuring of the questions. The pathetic B.C. referendum on native issues was a classic example of how to write a question to get the answer you want to hear. If this fails, the second line of defense is to interpret the results in a way which isn't consistent with the answers received. My recent post on my STV blog, is a good example where the 'lead' headline from a poll gave the opposite result from the poll results themselves.

Another problem with polls is that people try to interpret small movements which could well just be random fluctuations. This is especially true when the numbers are broken down into regions or different demographic groups which typically have much higher levels of uncertainty than the national totals.

There is legislation which outlines disclosure requirements for polls but as this article by Claire Durand, Professor at Université de Montréal shows, compliance with these rules is inconsistent at best. Of course I'm not sure how much good these rules do since I'm not sure people really pay attention to the poll fine print anyway.

Even if the pollsters are doing their job honourably and competently (as many are) and the media is interpreting the results clearly and objectively (this happens every now and then) there are still issues with polls. One of the biggest is the continuing rise in refusal rates. Veteran pollster Angus Reid refers to high refusal rates as polling's 'dirty secret' in this worthwhile article on polling in the Tyee. Besides increasing costs and putting downward pressure on sample sizes, this potentially biases the results if the refusal rates are different for different parties. Reid figures the solution is for the media to spend more on polling (shocking!) to make sure they get things right and notes that in the era before polls became prevalent, people were still devoting a lot of effort into 'horserace' politics and attempts to predict the outcome.

So far, there's no word on the possible rise of a new generation of pop-culture-referencing-smart-asses who are happy to talk to pollsters and just make up false answers as they go along (count me in with this group) and how that could skew the results.

It's been a rough year for the pollsters. They completely failed to see the Liberal minority government coming and down in the U.S. the massive discrepancy between the exit poll¹ results and the vote count still has not been resolved half a year later.²

Looking to the future, it seems like times will only get tougher for the pollsters. While on the one hand our growing demand for up-to-date information will drive more and more polling, our growing reluctance to talk (honestly) to pollsters will make it harder for them to get accurate results. People are already skeptical about relying too heavily on poll results or trusting media interpretations of poll results and this isn't likely to help. Darren's negative reaction to polling (which proved justified when the results came in in the Federal election) may become increasingly common. Polls make for good amusement in the punditocracy but their usefulness in the real world is pretty limited. If nothing else, I recommend taking all poll results with a grain (or two) of salt.


----------------
If you're interested in polling and recent Canadian polls, here are some links:

HillWatch's polling station which has a good historical log of political polls.

Maple Leaf Web has a really good FAQ (list of Frequently Asked Questions) on the topic.

CRIC has a great archive of polls organized by polling company

Michigan State University's Canadian Studies Centre has a dated (but still useful for those companies which are still around) list of polling companies with descriptions of each one.

----------------------

Polling companies tend to match up with media outlets:

Typical pattern is:

Environics = CBC
Ipsos-Reid = Globe and Mail / CTV
SES Research = Toronto Star / Sun Media
COMPAS = National Post / CanWest

although there are certainly exceptions and this can change as contracts/relationships end.

----------------------



¹ Exit polls are polls conducted as people are leaving the voting station. Because they record what people actually voted (of they are honest) as opposed to just who they are going to vote for, exit polls are generally much more accurate than pre-election polls.

² The firm responsible for the exit polling released a report on what went wrong which said that, "Our investigation of the differences between the exit poll estimates and the actual vote count point to one primary reason: in a number of precincts a higher than average Within Precinct Error most likely due to Kerry voters participating in the exit polls at a higher rate than Bush voters.".

But they didn't really do any analysis to back that up and as this report shows, their explanation doesn't really hold water: "Edison/Mitofsky did not come close to justifying this position, however, even though they have access to the raw, unadjusted, precinct-specific data set. The data that Edison/Mitofsky did offer in their report show how implausible this theory is."

Why I Wouldn't Want to be a Politician

The last 2 poll questions on the Globe and Mail website:

Should a university education be free? 50% say yes.
Given the highly misleading TD report from a couple of weeks back, should tax relief be the first priority in the upcoming federal budget? 67% say yes.

In a previous poll 40% were in favour of a child care program, despite the question being worded negatively.

And I'm guessing polls on more military spending would get a pretty big yes vote as well.

And we haven't even talked about health care yet.

I have this dream that someday people will realize that you can't have big tax cuts and big spending increases at the same time without going (further) into debt.

Which is all my way of leading up to the fact that the federal budget is coming up on February 23 and I'm not too optimistic about the prospects for my personal number one priority: maintaining a healthy budget surplus.

More to come on this as the budget approaches...