Tuesday, December 13, 2005

Expert prediction quality

As an IT architect, I make a lot of predictions about where technology is headed. This fascinating review (via Kottke) of Expert Political Judgment : How Good is It? How Can We Know? discusses the efficacy of experts in making political predictions. In short, get out your dartboard and you can outdo them. While the reviewer doesn't make this connection, it doesn't sound very different from success at picking stocks.

Indeed, while the realm is political prediction, what Tetlock describes seems to apply very generally to predictions. His recommendation also seems apply beyond policy debates:
Tetlock also has an unscientific point to make, which is that “we as a society would be better off if participants in policy debates stated their beliefs in testable forms”—that is, as probabilities—“monitored their forecasting performance, and honored their reputational bets.”
Probablities without accountability are worthless. Gartner has irritated me for years by saying assigning probabilities to predictions and then saying, "As we predicted ..." and never saying "which took us by surprise" or "where we whiffed badly" or even "happened much more rapidly than we thought possible" much less providing a statistical analysis of their analysts' accuracy.

We don't assign probabilities, but we clearly have accountability.


Post a Comment

<< Home