What I’m about to say…David Brooks wrote a piece today, and I actually wholeheartedly agree with it. In fact, it led to the discovery of one of the more interesting articles I have read in a very long time. Brooks discusses a highly interesting project (funded by the Intelligence Advanced Research Projects Agency) where a group of researchers are trying to discover optimal forecasting methods for near-term (primarily political) events.
In the fall of 2011, the agency asked a series of short-term questions about foreign affairs, such as whether certain countries will leave the euro, whether North Korea will re-enter arms talks, or whether Vladimir Putin and Dmitri Medvedev would switch jobs. They hired a consulting firm to run an experimental control group against which the competitors could be benchmarked.
Tetlock and his wife, the decision scientist Barbara Mellers, helped form a Penn/Berkeley team, which bested the competition and surpassed the benchmarks by 60 percent in Year 1. In the second year of the tournament, Tetlock and collaborators skimmed off the top 2 percent of forecasters across experimental conditions…These “super forecasters” also delivered a far-above-average performance in Year 2. Apparently, forecasting skill cannot only be taught, it can be replicated.
If I were President Obama or John Kerry, I’d want the Penn/Berkeley predictions on my desk. The intelligence communities may hate it. High-status old vets have nothing to gain and much to lose by having their analysis measured against a bunch of outsiders. But this sort of work could probably help policy makers better anticipate what’s around the corner. It might induce them to think more probabilistically.
I find this kind of research fascinating–if you read the piece I linked to earlier (the conversation with Tetlock), it turns out that forecasting improves significantly when people work in teams (or with the use of prediction markets), and Tetlock and his team have identified an algorithm that corrects for systematic human biases in estimating probabilities (it turns out that forecasts tend towards 50%; that is, on average, groups of people systematically overestimate the likelihood of unlikely events and underestimate the likelihood of probable events). I find this line of research fascinating, and I am excited to see how far these researchers can push the limits of accurate forecasting.
Of course, I can’t let Mr. Brooks go without a few quibbles. First, I actually suspect that this sort of decision-making is already fairly common-place in the intelligence community–I know the intelligence community made a big push toward probabilistic analysis when Dr. Tom Fingar was in charge of the National Intelligence Council, and while the methodology might vary b/w the CIA and these outside groups, I know that there is some degree of probabilistic decision-making occurring throughout the broader intelligence community. President Obama even makes reference to the use of probabilistic thinking in the Oval Office (though Felix Salmon makes it clear that what is deemed “probabilistic analysis” by Obama might be a far stretch from what the quants at Penn, MIT, and Berkeley are doing), so I actually anticipate far less resistance to this idea than Brooks does.
Second, the country recently experienced an extraordinary example of highly skillful probabilistic analysis at work in the run-up to the 2012 election…I wonder how David Brooks felt about the use of complicated computer models to forecast the probabilities of future events before the 2012 election?? David Brooks quoted by Dylan Byers at Politico:
“The pollsters tell us what`s happening now. When they start projecting, they`re getting into silly land.” Brooks doubled down on this charge in a column last week: “If there’s one thing we know, it’s that even experts with fancy computer models are terrible at predicting human behavior…”
So, it appears that 2013 Brooks has a completely different feeling about probabilistic forecasting than 2012 Brooks. Or, perhaps probabilistic analysis is only good when it is not projecting your favored Presidential candidate to lose the electoral vote by a significant margin. But, we can’t blame David for making a bad prediction about the accuracy of prediction models; as he says in his article: “pundits…are terrible at making predictions.”
PS: Brad Delong with the takedown of pundits attacking Nate Silver’s probabilistic forecasting in advance of the 2012 election. How did that one turn out? Nate Silver called all 51 electoral results (50 states and DC) and was in the upper echelon of 2012 forecasters (his model, which was ripped for its inherent liberal bias, failed to beat out a couple other poll aggregation models because Silver underestimated the probability of an Obama victory.)