Matt Ginsberg’s training is in astrophysics. He got his Ph.D. from Oxford when he was 24 years old. His doctoral advisor there was the famed mathematical physicist Roger Penrose, and he recalls rubbing elbows with the academic rock stars Stephen Hawking and the late Richard Feynman. He created an artificial intelligence crossword puzzle solver called Dr. Fill and a computer bridge world champion called GIB. ... Unsurprisingly, there’s pretty heavy math involved to make this real-time sports predictor work. For one element of the system’s calculations, Ginsberg sent me a pdf with eight dense pages of physics diagrams and systems of equations and notes on derivations. It uses something called the Levenberg-Marquardt algorithm. It requires Jacobians and the taking of partial derivatives and the solving of quartics, and code efficient enough to calculate it all up to the split second. If predicting the future were easy, I suppose everybody would do it. ... One thing this project can’t predict, however, is its own future. Its uses are, so far, largely speculative, and cashing in on a minor superpower might not be easy. Even gamblers who bet during play would struggle to make much money from a half-second heads-up that a shot is going in. But Ginsberg’s system would find a natural place in the long line of sports technologies that have been used for a singular end — TV.
Famous German soccer coach Sepp Herberger once said, “After the match is before the match.” The same can be said for financial markets: After the crisis is before the crisis. The complication, of course, is that while soccer players usually know exactly when the next match will kick off, the timing of the next crisis is always uncertain for financial players. All we know is that, eventually, there will be another one. ... What’s perhaps less obvious is that the global savings glut also helps to explain the occurrence of financial bubbles and subsequent crises of the past few decades. ... the global savings glut plays an important role in explaining this evidence. How? Excess savings not only pushed down r* and actual interest rates but also drove up asset prices and caused serial asset bubbles in equities, emerging markets, housing, credit, eurozone peripheral bonds and commodities. Whenever a bubble burst, it sparked financial distress and crisis. ... there is a feedback loop between financial crises and the savings glut. This is because a financial crisis and the related destruction of wealth leads to even higher desired saving (or deleveraging), and because the depressing impact on growth reduces investment and thus the demand for savings. ... now that exhaustion has set in almost everywhere for many unconventional policy tools, such as quantitative easing, there is a significant risk that central banks may not be able to deal effectively with the next crisis. ... It’s likely the only viable way out would be a joint effort by the major countries to raise public spending on infrastructure, education, and more in order to absorb excess savings and raise r*.
Guinness brewer William S. Gosset’s work is responsible for inspiring the concept of statistical significance, industrial quality control, efficient design of experiments and, not least of all, consistently great tasting beer. ... Because he used a pseudonym, his name isn’t even familiar to most people who frequently use his most famous discovery. Gosset is the “student” of the Student’s T-Test, a method for interpreting what can be extrapolated from a small sample of data. ... Born in 1876 in Canterbury, England, Gosset entered a world of enormous privilege. His father was a Colonel in the Royal Engineers, and though he intended to follow in his footsteps, he was unable to due to bad eyesight. Instead, Gosset attended the prestigious Winchester College, and then Oxford, where he studied mathematics and natural sciences. Soon after graduating from Oxford, in 1899, Gosset joined the Guinness brewery in Dublin, Ireland, as an experimental brewer.
We are exposed to possible events all the time: some of them probable, but many of them highly improbable. Each rare event—by itself—is unlikely. But by the mere act of living, we constantly draw cards out of decks. Because something must happen when a card is drawn, so to speak, the highly improbable does appear from time to time. ... It is the repetitiveness of the experiment that makes the improbable take place. The catch is that you can’t tell beforehand which of a very large set of improbable events will transpire. The fact that one out of many possible rare outcomes does happen should not surprise us because of the number of possibilities for extraordinary events to occur. The probabilities of these singly unlikely happenings compound statistically, so that the chance of at least one of many highly improbable events occurring becomes quite high. ... Persi Diaconis, professor of statistics at Stanford University, describes extremely unlikely coincidences as embodying the “blade of grass paradox.” If you were to stand in a meadow and reach down to touch a blade of grass, there are millions of grass blades that you might touch. But you will, in fact, touch one of them. The a priori fact that the blade you touch will be any particular one has an extremely tiny probability, but such an occurrence must take place if you are going to touch a blade of grass. ... The devil is in the details of how we interpret what we see in life. And here, psychology—more so than mathematics or logic—plays a key role. We tend to remember coincidences such as the one I experienced with my editor Scott and conveniently forget the thousands of times we may have met someone and had a conversation finding absolutely nothing in common.
The shikiri (pre-match ritual) takes several minutes. The wrestlers clap to attract the attention of the gods, lift their hands to show they are unarmed, stomp the ground to scare away demons and throw salt in the ring to purify it. They repeatedly crouch as if about to start the match and then stand up after a few moments of glaring at each other. When they are finally ready, they creep toward their starting stance. ... There is no bell. The match starts with a tachi-ai (initial charge), which generally happens the instant the opponents are set. ... Harumafuji lunged from his crouch, low, exploding toward Hakuho in an effort to take control of the bout early. Instead, he caught a quick palm to the face — and then air. His momentum carried him clear out of the other side of the ring, like he’d tried to bull-rush a ghost. ... Commentators didn’t quite know what to say; one of the English announcers let out a long “hmmmmm.” The crowd booed its champion. ... This is not normally how a match of this scale plays out. Side-stepping an opponent’s charge is legal but considered beneath the dignity of top sumotori. The move is known derisively as a henka (変化), which translates to “change” or “changing,” while connoting the root “strange” (変). That it would be used by an all-time great in one of the most consequential matches of his career was strange indeed. ... The first known professional tournament was held in 1684, and the first sumo organizations began issuing written rankings in the mid-1700s — just in time to document the rise of sumo’s most legendary figure.
Statistics were designed to give an understanding of a population in its entirety, rather than simply to pinpoint strategically valuable sources of power and wealth. In the early days, this didn’t always involve producing numbers. In Germany, for example (from where we get the term Statistik) the challenge was to map disparate customs, institutions and laws across an empire of hundreds of micro-states. What characterised this knowledge as statistical was its holistic nature: it aimed to produce a picture of the nation as a whole. Statistics would do for populations what cartography did for territory. ... the aspiration to depict a society in its entirety, and to do so in an objective fashion, has meant that various progressive ideals have been attached to statistics. The image of statistics as a dispassionate science of society is only one part of the story. The other part is about how powerful political ideals became invested in these techniques: ideals of “evidence-based policy”, rationality, progress and nationhood grounded in facts, rather than in romanticised stories.
Statcheck had read some 50,000 published psychology papers and checked the maths behind every statistical result it encountered. In the space of 24 hours, virtually every academic active in the field in the past two decades had received an email from the program, informing them that their work had been reviewed. Nothing like this had ever been seen before: a massive, open, retroactive evaluation of scientific literature, conducted entirely by computer. ... Statcheck’s method was relatively simple, more like the mathematical equivalent of a spellchecker than a thoughtful review, but some scientists saw it as a new form of scrutiny and suspicion, portending a future in which the objective authority of peer review would be undermined by unaccountable and uncredentialed critics. ... When it comes to fraud – or in the more neutral terms he prefers, “scientific misconduct” ... Despite its professed commitment to self-correction, science is a discipline that relies mainly on a culture of mutual trust and good faith to stay clean. Talking about its faults can feel like a kind of heresy. ... Even in the more mundane business of day-to-day research, scientists are constantly building on past work, relying on its solidity to underpin their own theories. If misconduct really is as widespread as Hartgerink and Van Assen think, then false results are strewn across scientific literature, like unexploded mines that threaten any new structure built over them.
For decades, poultry had been volatile in a frustratingly predictable way: When times started getting good, companies flooded the market with chicken, causing prices to crash. ... At first the transformation puzzled industry watchers. Some speculated that a merger spree during the 1980s and 1990s was responsible—with fewer decision-makers in charge and fewer competitors, the remaining companies could more easily survey and predict the landscape. But Sanderson’s conference call suggested another source for the shift: Agri Stats, a private service that gathers data from poultry processors, produces confidential weekly reports, and disseminates them back to companies that pay for subscriptions. ... Many industries, such as health care and retail, make use of information-sharing services, but Agri Stats provides chicken producers with a rare level of detail, in uncommonly timely fashion. ... Agri Stats has for years maintained that its reports don’t violate antitrust laws, in part because the information provided is historical. A typical report doesn’t say how much a company plans to charge for a cut of meat, only what it charged last month or last week. ... In 2013, according to SEC filings, Eli Lilly purchased Agri Stats for an undisclosed sum and folded it into its farm animal drug division. ... Illegal collusion occurs when companies plan with one another to cut production ahead of time with the specific intent of raising prices.
Over the past 12 years, Lt. Megge has increased the speed limit on nearly 400 of Michigan’s roadways. Each time, he or one of his officers hears from community groups who complain that people already drive too fast. But as Megge and his colleagues explain, their intent is not to reduce congestion, bow to the reality that everyone drives too fast, or even strike a balance between safety concerns and drivers’ desire to arrive at their destinations faster. Quite the opposite, Lt. Megge advocates for raising speed limits because he believes it makes roads safer. ... This “nationally recognized method” of setting the speed limit as the 85th percentile speed is essentially traffic engineering 101. It’s also a bit perplexing to those unfamiliar with the concept. Shouldn’t everyone drive at or below the speed limit? And if a driver’s speed is dictated by the speed limit, how can you decide whether or not to change that limit based on the speed of traffic? ... The answer lies in realizing that the speed limit really is just a number on a sign, and it has very little influence on how fast people drive.
Here we have a basketball mystery: a player is widely regarded inside the N.B.A. as, at best, a replaceable cog in a machine driven by superstars. And yet every team he has ever played on has acquired some magical ability to win. ... The virus that infected professional baseball in the 1990s, the use of statistics to find new and better ways to value players and strategies, has found its way into every major sport. Not just basketball and football, but also soccer and cricket and rugby and, for all I know, snooker and darts — each one now supports a subculture of smart people who view it not just as a game to be played but as a problem to be solved. Outcomes that seem, after the fact, all but inevitable — of course LeBron James hit that buzzer beater, of course the Pittsburgh Steelers won the Super Bowl — are instead treated as a set of probabilities, even after the fact. The games are games of odds. Like professional card counters, the modern thinkers want to play the odds as efficiently as they can; but of course to play the odds efficiently they must first know the odds. Hence the new statistics, and the quest to acquire new data, and the intense interest in measuring the impact of every little thing a player does on his team’s chances of winning. In its spirit of inquiry, this subculture inside professional basketball is no different from the subculture inside baseball or football or darts. The difference in basketball is that it happens to be the sport that is most like life. ... the player who seems one step ahead of the analysts, helping the team in all sorts of subtle, hard-to-measure ways that appear to violate his own personal interests.