Polls can't measure hypotheticals

From Biden's withdrawal to Trump's legal woes, the polls this year have been really bad at predicting how voters will react to big events

A few days after the June 27th debate, Biden-Harris deputy campaign manager Rob Flaherty released an infamous memo that implicitly dissed Kamala Harris (and anyone on the Democratic bench who might replace Biden). He was writing off a post-debate Data for Progress poll showing Biden losing to Trump—but every other Democrat doing the same or worse.

This passage is quite a read in light of recent events:

The bedwetting brigade is calling for Joe Biden to "drop out." That is the best possible way for Donald Trump to win and us to lose.

First of all: Joe Biden is going to be the Democratic nominee, period. End of story. Voters voted. He won overwhelmingly. And if he were to drop out, it would lead to weeks of chaos, internal foodfighting, and a bunch of candidates who limp into a brutal floor fight at the convention, all while Donald Trump has time to speak to American voters uncontested. All of that would be in service of a nominee who would go into a general election in the weakest possible position with zero dollars in their bank account. You want a highway to losing? It's that.

And at the end of the day, we'd switch to candidates who would, according to polls, be less likely to win than Joe Biden -- the only person ever to defeat Donald Trump.

At least in the short term, this doomsday scenario was wrong. There was no internal food fight. Democrats do not have zero dollars in the bank. By all measures, they are doing much better than they were when Joe Biden dropped out.

Dropping Biden was unquestionably the right move for Democrats—even if the polls didn’t say so at the time.

Similarly, a number of polls earlier in the cycle tested what might happen to the presidential horserace if Donald Trump were convicted of a crime in the middle of the campaign. Last fall, the gold-standard New York Times/Siena poll showed a Biden landslide in the making if that happened, with a 10 point victory across the battlegrounds. The initial horserace number for this poll was Trump +4.

This too did not happen. After (maybe) a 2 point shift in Biden’s direction following the conviction in the New York hush money case, the race reverted to its previous baseline, with Trump ahead in the polls going into the debate.

Republican voters were no better at predicting their response to the initial Alvin Bragg indictment. When we asked them to consider a hypothetical indictment by the Manhattan DA before it happened, the gap between Trump and his closest competitor at the time, Ron DeSantis, narrowed by 13 points.

Again, nope. The actual indictment turned out to be rocket fuel for Trump’s primary campaign, effectively ending any hope his challengers ever had.

The point here is not that the polls are fake or that respondents are lying. Polls are a reflection of the political moment when the poll is taken. And merely asking a hypothetical can’t begin to approximate the dynamics of a change once it actually takes place.

Especially as July wore on, some polls were directionally right about Harris polling better than Biden. But these polls got the psychology of the electorate wrong, suggesting a replacement would fare almost as badly as Biden, with all of the downsides Flaherty outlined in his memo.

What polls didn’t capture was the enormous sense of relief that would wash over the Democratic Party once the Biden drama was resolved, leading at least for now to a consolidation of the party’s base behind Harris. Three weeks ago, the party was anything but unified, so the polls taken at the time, with their dejected and despondent Democratic samples, weren’t that helpful in making the case for a switch.

What was needed to finally push Biden out was not data, but vibes. Nate Silver has been pilloried by the left this cycle for becoming something of a “vibes” guy, but his common sense read on the situation was absolutely correct. Simply going from a moribund candidate to one who could actually wage a vigorous campaign transformed the psychology of the party almost overnight—even if Harris was not the ideal candidate.

Similarly, prior to the first indictment of Donald Trump, there had not yet been a rallying-of-the-flag behind Trump, with a groundswell of grassroots pressure leading even his opponents to swear off this apparent gift from the political gods. In the poll, respondents were clinically updating their previous views of Trump with a negative piece of new information about him. But there is no way for a poll to simulate the group psychology that takes hold once an event actually takes place. The eventual reaction to the Trump indictment could be predicted by vibes and common sense, not by polls.

The critique I’m making is one that’s circumscribed to the types of polling questions that ask respondents to engage in hypotheticals. And it’s well known in the field that respondents are actually bad at predicting their future behavior. So even when these questions do get asked, it’s incumbent on pollsters to warn audiences to take them with several grains of salt.

Given the looming potential for any number of game-changing developments in this election, we’ve seen our share of these sorts of polling questions this cycle. And now that these developments have happened, we can evaluate the quality of their predictions.

The results are not good.