Forecasting, Pt 2

Never, ever, over-trust the forecast output

Models are unintelligent. They have no means of deciding to leave out unnecessary information or retrieving information that they actually need. Unfortunately, there is a tendency to trust the outcome of previously successful models. The thought process goes: “It was right before, it will be right now.” Until suddenly it’s not; a market condition has changed leading to massive failure.

Flawed models are at the heart of the recent economic downturn. Every financial and rating institution was using the same flawed models. For more than five years the models worked with enormous success. Analysts would put in their data points, the model would return the likely outcome and the analyst would execute against that decision. A highly effective system…until it broke.

Once a model is discovered to be flawed, analysts often dive for cover under the excuse of “the model told me to.” This takes us back to the concept of trusting a model’s output. Don’t. Do not ever put too much faith in the outcome of a forecast (moving average, Monte Carlo, or anywhere in between).

Forecasting models are only tools in the digital toolbox of decision makers. The moment that they become elevated to a higher status is the moment that they should lose all credibility. Decision makers should be held responsible for their actions no matter their process for reaching the decision. Anything less leads to abuse of the system.