Who got it right last time?

My friend Grant has a great way of explaining data issues. I was talking to him about the variation between the different polls at the last election, and the problem of averaging the variation when some differences are positive and some are negative, which inevitably leads to a reduction in the apparent variation.

He sagely wrote ‘standing with one foot in ice cold water and the other in boiling water does not warm make’. His solution was to use absolute values, where the focus is the scale of difference rather than if it’s negative or positive.

The table below shows the eve of election poll results published before the 2011 election, and the actual election results. The green cells show which were closest for each party.  Interestingly, two polls back then (Fairfax and Roy Morgan) didn’t include the Conservatives in their polling, who ended up with 2.7% of the vote.

The next table shows the variation of each poll from the election result. The blue cells show over-estimation from the final result, and the red cells under-estimation. The yellow cells show the average variation, using absolute values, for each party and for each poll overall.

A number of interesting things emerge:

  • All the polls over-estimated National’s performance, by 3.7 percentage points on average. If repeated this week, that could be the difference between government and opposition, depending on how other parties perform.
  • Most under-estimated Labour and over-estimated the Greens.
  • All underestimated NZ First, the Conservatives and United Future.
  • The poll which was closest overall to the election result was TVNZ’s Colmar Brunton, while the Fairfax poll, at the time conducted by Research International, was least accurate overall (note that Fairfax now use Ipsos as their polling provider).

A note though on the Roy Morgan poll. Although it appears second worst, its combined Labour and Green vote was very close to the final outcome. As shown in the table below, when these are taken into account, its average variation drops to only 0.9, making it the most accurate poll overall. This does suggest that the Labour-Green vote is very soft and that people will shift allegiance between the two in the lead up to election day.

So based on the last eve of election polls, this week, I’ll be paying particularly close attention to the TVNZ and Roy Morgan polls.

And if the last pre-election polls are anything to go by, it may also be worth mentally deducting a few percentage points from National in the final polls, and see where that may leave post-election coalition arrangements.