As Britain moves into pre-election spring, we can expect the opinion polls to flower in profusion. But what do they really tell us?

Something; but generally a lot less than the pollsters and their media partners pretend. For example take this, from The Times from December 2008:

“The boost [in Gordon Brown’s ratings as most trusted to deal with recession] has improved Labour’s position with the Conservative poll lead narrowing to four points…..”

 The first flaw with this is the post hoc ergo propter hoc fallacy. It might be of course that any narrowing of the Tory lead was due to Brown’s increased standing. It could also be because the previous poll had been conducted on a wintery day and this one on a sunny day, or because voters preferred Sarah’s hat to Samantha’s, or any other cause.

However, there is a yet more fundamental flaw: that, statistically, there is little or no reason to believe that there had been a narrowing at all. The poll showed the Tories on 39 per cent, down 2 percentage points, and Labour on 35 per cent, unchanged.

However, all poll ratings, even the pollsters admit, are subject to a margin of error due to the fact that only a sample of voters are being interviewed, rather than all voters. That statistical margin of error depends on such variables as sample size and whether the result is say 50:50, 90:10 or whatever. Strictly it tells us within what range of the central figure quoted 95 per cent of observations will fall (the other 5 per cent being the dreaded rogue polls).

In this case, the statistical margin of error is rather more than 2 per cent +/- on each figure. So the poll actually tells us that the Tories are on between 37 per cent and 41 per cent, Labour on between 33 per cent and 37 per cent and therefore the Tory lead is between 0 per cent and 8 per cent. You can see why they don’t put it like that in the stories! So in the case of the Times story there is little evidence that there has been any change at all in the parties’ share of the vote, certainly not that there is one that is statistically significant.

That is bad enough, but it is only part of the story. For the term “margin of error” is itself a sleight of hand. It seems to be a concession of weakness but actually is a claim of strength for it is designed to suggest to the unwary that the polls are accurate to within it.
This is not the case. This statistical margin of error measures one thing only: the error that results from the statistical technique of using samples. However it makes a very strong assumption: that the statistical technique is itself perfect, yielding samples that are representative of the population as a whole.

This is extremely difficult to achieve. There are broadly two possible methods: random sampling and quota sampling.

Random sampling involves, for example, using a random telephone dialling system and questioning whoever answers. The first problem here is not the old one – that many households don’t have phones – but a new one, that many people have only mobile phones and we cannot tell if their views are the same as those who have fixed lines.

Worse is the problem of refusers. Random digit dialling yields usable interviews in only 20-25 per cent of cases, according to Nick Sparrow, who runs ICM, a pollster. Who knows if those 20-25 per cent are representative of the population as a whole?

In this case, what is done is to weight the sample. So, for example, if it contains fewer young people than are known to exist in the population as a whole, the weight given to their response is boosted to reflect the national position. This is similarly the basis of quota sampling, where interviews are conducted either by phone or face-to-face and the date weighted to be demographically representative of the population as a whole.

Tremendous methodological controversy exists between pollsters as to what you should weight by. Many use past voting. Unfortunately, pollsters do not know how people voted last time, and have to rely on voters to tell them. And equally unfortunately, in huge numbers they give the wrong answers: in one case, for example, 51 per cent of voters said they voted Labour in the 2001 general election when in fact only 41 per cent did.

Weighting to get accurate political results has got much more difficult. Classically, voting behaviour in Britain was largely a matter of class. Broadly, the working class voted Labour, the middle class Tory. So as long as the percentage of each class in your sample matched the known profile of the population, you did not go wrong. Today, however, the link between vote and class has been eroded. No other simple demographic fact is closely coordinated with vote though some (eg public against private sector employment) may have a relationship. In consequence, it is much harder to weight samples to produce accurate voting intention figures.

None of this means that polls are a waste of time. They still give an indication which is more accurate than the pricking of politicians’ thumbs. They will more usually get close to the right result of an election (as they did in 2005) than the wrong result (as they famously did in 1992). If a lot of polls using a lot of different methodologies are showing a consistent and clear lead for one party, that party is probably in front. In other words, by all means partake – but do not inhale.