The US polling industry has been suffering a crisis of insight over the past decade or so; its methods have become increasingly bad at telling which way America is leaning. ... The classic pollster’s technique known as random digit dialing, in which firms robo-dial phone after phone, is failing, because an ever-dwindling number of people have landlines. ... whereas a survey in the 1970s or 1980s might have achieved a 70 percent response rate, by 2012 that number had fallen to 5.5 percent, and in 2016 it’s headed toward an infinitesimal 0.9 percent. And finally, the demographics of participants are narrowing: An elderly white woman is 21 times more likely to answer a phone poll than a young Hispanic male. So polling samples are often inherently misrepresentative. ... Today’s polling landscape appears so fraught that Gallup, long the industry leader, opted out of presidential horse-race polls this year; the reputational risk of being wrong was simply too high. Civis, on the other hand, promises a paradigm that could rescue American politics from confusion. ... Today, campaigns realize they have to look elsewhere for their intelligence, which has caused a major change in how the political industry functions. In the past, an entire campaign’s data and infrastructure would go poof after Election Day. Now Civis and similar firms are building institutional memory with permanent information storehouses that track America’s 220 million–odd voters across their adult lives, noting everything from magazine subscriptions and student loans to voting history, marital status, Facebook ID, and Twitter handle. Power and clients flow to the firms that can build and maintain the best databases of people’s behavior over time.
The opinions of experts concerning the future are accorded great weight ... but they’re still just opinions. Experts may be right more often than the rest of us, but they’re unlikely to be right all the time, or anything close to it. ... A lot of people's lives would be more tranquil and more productive if they accepted that what the media says about an upcoming event - and whether you watch of not - won't have any impact on the outcome. ... Today many analysts seem preoccupied with central bank behavior, government actions, trends in interest rates and currencies, and the movement of markets, as opposed to the fortunes of individual companies. … Most people don’t want to tempt fate by saying things will go well forever, and in fact they know they won’t. It’s just that they can’t decide what it is that will go wrong. The truth is that while I can enumerate them, the obvious candidates (changes in oil prices, interest rates, exchange rates, etc.) are likely to already be anticipated and largely priced in. It’s the surprises no one can anticipate that would more markets most if they were to happen. But (a) most people can’t imagine them and (b) most of the time they don’t happen. That’s why they’re called surprises. ... People began to ask me what inning we’re in during the financial crisis of 2008, and they’ve continued ever since.