Eight out of ten cats

A polling problem

As I alluded to in my last post, there’s a polling puzzle that has been bothering me for some time. On the face of it, it isn’t important and it certainly looks tangential. More and more, however, I am coming to the conclusion that it is important and that it potentially is central to the question of where we are heading.

The puzzle that is Reform UK’s polling

In the time that Rishi Sunak has been Prime Minister, Britain has had nine by-elections, a local election round and a gazillion opinion polls.

Reform UK have held their deposit in exactly one of the nine by-elections in the last year: Tamworth. Of the eight seats they contested (they didn’t stand in Uxbridge), five are estimated to have voted for Leave, so you would have thought there was plenty of fertile ground for the party formerly known as Brexit. Tamworth, where Reform just held their deposit, voted Leave by just under two to one.

In the local elections, Reform UK fought hundreds of seats and only won six, averaging just 6% of the vote in the wards they stood in, which had been selected for their favourable demographics. This was a dismal performance. At the same round of local elections, the Greens took a council and more than doubled their seat tally to nearly 500.

Yet when we look at the opinion polls, Reform UK have been consistently polling above 5% throughout that period, dipping below that mark in only three out of 54 polls published in September and October. Why is there such a mismatch between the polls and Reform UKs actual electoral performance?

Here are some possible explanations. They aren’t mutually exclusive, of course.

You don’t know Reform UK supporters, they live in a different constituency

Perhaps Reform UK have underperformed because their strength lies in other areas. This seems implausible. The by-election constituencies in the last year have offered a fairly good cross-section and if anything have been skewed against the metropolitan elite. And Reform UK chose their own seats to fight in the local elections. This one can, I think, be discounted.

Reform UK are rubbish at the ground war but will be fine once the air war of a general election kicks in

I can well believe that Reform UK don’t have the capacity to fight in depth the way that Labour, the Conservatives or even the Lib Dems do. Still, I don’t think that lets them off the hook. In the local elections they husbanded their resources, fighting only a few hundred seats. They still flopped.

We saw something very different from their predecessors revolting on the right, in by-elections at least. UKIP consistently outperformed their polling at by-elections in suitable constituencies despite also being rubbish at the ground war. They tallied 22% in the Newark by-election in 2014 at a time when they were averaging about 15% in the national polls. They got 21% at the Rotherham by-election in 2012 at a time when they were averaging about 11% in the national polls. Even at the Feltham & Heston by-election in 2011, when UKIP was averaging about 4% in the national polls, they kept their deposit.

Indeed, it doesn’t make any sense at all that Reform UK would always underperform its national polling at by-elections, even if their ground game isn’t up to much. Turnout is always down at by-elections but those who are motivated to send a message are going to go out and vote. Reform UK supporters are presumably angry enough with the Conservatives to have defected, so you’d expect them to be marching with their pitchforks to the polling booth to make their point. Reform UK are not really competing with Labour or the Lib Dems for such voters, so if there are all these fervent Reform UK supporters out there, they should be making an outsize impression at the ballot box in by-elections in suitable constituencies. The opposite is happening.

Pollsters are picking up the low information voters who will turn out at general elections but who don’t bother at by-elections or local elections

I’m completely unconvinced by this theory. It’s theoretically possible but it makes no sense in the real world. The type of person who joins an opinion pollster’s panel or who agrees to be interviewed is unlikely to be someone whose top priority is whether they’re having chips for tea.

The opposite is likely to be true. Members of such panels and telephone interviewees are disproportionately likely to be engaged with politics and have views: they’re the people who are willing or even enthused to give ten minutes of their time to tell you what they think about political matters.

(As a practical matter, I’d like to know how much name recognition Reform UK has. Unlike its previous name of the Brexit Party, its name gives no clue what its policies and it’s not getting that much media coverage outside GB News and the hard right media ecosystem. Do low information voters even know who it is?)

Which leads me to a further possible explanation.

The polls are wrong, at least so far as they report Reform UK vote shares

This looks to me to be much the simplest explanation. It’s easy to see how. If pollsters are interviewing the disproportionately politically engaged, their sample will be skewed by their priorities. Those priorities may or may not match those of the wider public.

You don’t have to go far online to find a particular type, red-pilled by the Mail and GB News and with union jacks in their profiles, who remain fervently pro-Brexit and regard Rishi Sunak as a closet socialist. They are notably keen to express their opinions whether or not those opinions are solicited and often when it would be apparent to even the most bullish opinion-offeror that those opinions were not going to be warmly received. Such individuals are not going to balk at the opportunity to participate in opinion polls.

It’s notable that Ipsos, which (unusually) polls by telephone, has not recorded Reform UK above 4% in any poll in the last year. The 1–4% band Ipsos have had Reform UK in is much more consistent with the actual election results Reform UK has achieved than the 6–9% band recorded by YouGov for Reform UK in the calendar year 2023.

Note, I’m not saying that Reform UK’s support could not genuinely increase. It’s easy to see mechanisms by which it does, particularly if the Conservatives stuff things up further. What I’m saying is that present polling is very possibly flattering to deceive.

So what?

I can hear you saying: OK Meeks, you’ve made your point, but where are you going with this? Well, the first thing is to note this isn’t just a problem with Reform UK. The Greens too have consistently outpolled their actual results in by-elections. Their average vote share at by-elections in the last year is under 3%. A crude average of their polling in October puts them above 5%.

Now this I do think is a bit more explicable. Unlike Reform UK, the Greens are vulnerable to being squeezed by Labour and the Lib Dems. That definitely seems to happen in some by-elections (though, note, the Greens held their deposit in both Selby & Ainsty and Somerton & Frome — there seems to be Green support in some places that is quite hardcore).

Also, respondents may be naming the Greens as an expression of enthusiasm rather than a final voting decision. It’s one thing telling a pollster you’re a Green supporter. It’s another thing entirely voting in a Conservative/Labour marginal for the Greens when Labour want your vote.

The Greens have a proven track record in local elections showing that the public will come out and vote for them when the circumstances are right, as this year’s results showed.

Nevertheless, the combination of these considerations means that the Greens too are probably being overstated in the polls relative to what they are currently on course to achieve at the next election. At the last election, the Greens took just under 3% of the national vote share. The Greens at election after election have generally been overstated in the opinion polls at some distance out until the election comes. I expect the same is true this time too (though perhaps the Greens will hold more of their enthusiastic voters this time, given how firmly Sir Keir Starmer’s Labour are clinging to the centre).

There’s an immediate point and a behind-point to think about here.

The immediate point is that if both Reform UK and the Greens are being substantially overstated, there’s a big chunk of the polling that is being misallocated. If Reform UK and the Greens should really be on, say, 3%, then YouGov’s latest poll which allocates 9% to each of them has 12% of the vote misallocated. Depending on where that 12% belongs, we could be looking at a hung Parliament or Conservative obliteration. That seems rather important.

The behind-point is still more important. If these two parties are being misstated by artefacts of polling, how much confidence can we place in the vote shares being attributed to the bigger parties? What artefacts of polling might be leading us astray about them?

A warning

It’s tempting to try to adjust polls to compensate. You might say that if most of these Reform UK voters don’t really exist and most of the Green voters are really Labour voters, we should transfer, say, 5% from the Greens to Labour, reduce Reform UK’s total by 5% and then scale all parties up by 100/95. Simple, eh?

Tempting, but quite wrong. If the polls are wrong, they’re wrong. You have to decide how much weight you’re going to put on their findings. You can’t start doing your own adjustments.

Right now, the weight I’m putting on them is “big Labour lead, low Conservative headline count, likelihood of extensive tactical voting against the Conservatives”. I’m not comfortable going beyond that without corroborating evidence.

Some concluding nice words about pollsters

This really is not an attack on pollsters. They have a desperately difficult job to do and given how difficult it is, they do it well. The wonder is that their polls have any degree of accuracy at all.

We don’t have many sources of evidence about what the public are going to do and opinion polls are one of those sources. It’s important not to place more weight on them than they can bear. They should be taken seriously but not literally.

These are not new problems and pollsters are very aware of the theoretical challenges of obtaining a representative sample, taking great pains to address the point. Still, it seems at the moment as though their countermeasures may not be wholly effective. We need to be particularly careful about how we use polls just now.