Forecasting, Pt 4

Forecasting’s ultimate goal is to accurately understand how your business will run

The chart below shows the input that a forecast has on any given business decision. As you can see it is only one of six possible inputs (although past market conditions may also be captured in the forecast). Tribal knowledge, gut feel and market conditions are also significant inputs into the decision.

For the purposes of this discussion it is important to define Tribal Knowledge and Gut Feel with regards to forecasting. Tribal Knowledge is the information known by workers of a company that is not captured in any database, qualitative or numerical format. Tribal Knowledge is best classified as the information critical to the function of any business that is simply known and un-captured.

Gut feel is the concept of “knowing” an answer. Any decision maker will hesitate on a decision that they are uncomfortable with. Humans are capable of capturing and processing information subconsciously which leads to the sense of making decisions without solid input. Most effective decision makers will “trust their gut” often enough that it will impact many critical decisions. This also happens to be one of the best error finding methods in model creation.



You will also notice the three inputs that go into the Revised Forecast Model. All three have the ability to be both good and bad. As we’ve discussed previously, past model performance is no indicator of future performance. There are many variables that could unbalance the tenuous nature of the interactions. Past performance does provide the main driver of model improvement as refinements are made to fine tune the variables actually in the model.

Past bad decisions is an important input to the model structure. Any good analyst will design their model to avoid past mistakes. The situations that led to those decisions will have specific avoidance mechanisms built-in. This is both good and bad. It’s always good to avoid past mistakes, just not at the expense of making new ones.

When modeling in past scenarios many analysts fall into the trap of over simplifying the causes of the past error which causes the model to detect the scenario in more situations than would actually be called for. To use a simple example, a modeler in the Northern US state may put in a mechanism that says rain in January will lead to frozen roads which leads to significantly reduced revenue because the model would predict fewer people driving. However, it is entirely possible to have a very mild month where rain either has no impact or positive impact (depending on the business and specific location).

On the other extreme is the analyst who overcomplicates the situation. To use the same example as above, that same modeler instead programs a situation that rain in January but only on Tuesday’s leads to frozen roads. This leads to six days being excluded from the situation. Neither situation leads to effective decision making.

The third input to the model setup is Forecaster Bias which is very similar to the Past Bad Decision input. Any modeler creating a new forecast will have their concept of what should be included or excluded. This will be modified slightly based on the input of others, but ultimately decided by the model-builder. They are in the position they are because they probably have enough experience on the topic to make good predictions of what will be necessary but they may also have blinders to other inputs.

Ultimately, a forecast model is no better than the people who create it, the time it has been used and revised over, the data input and the situations that are predicted to occur. This still leaves quite a large opportunity for any model to be wrong in many situations. For this reason it is important to reiterate that no model should have the final say in any given business decision.

Forecasting, Pt 3

Iterate regularly to test for variable sensitivity

Every model has the ability to be stress tested. Test them regularly to see how the result changes based on varying conditions. Test for sales dropping 10%. What if prices drop 5% and sales rise 10%? What if your entire logistics department is out sick for the month of October?

To go back to the role of forecasts in the economic downturn, most of the models used in financial houses had no capability to test for the possibility of an across the board housing price drop. Their inputs would not allow them to enter a negative number. It’s possible that those institutions wouldn’t be in the situation that they find themselves in if they could test the sensitivity of their outputs based on negative home price increases.

If your forecast model doesn’t allow you to test for a given condition change that could occur, it is time for a revision. Never let yourself be held hostage by the constraints of your overgrown spreadsheet. It is up to you to make decisions based on the criteria you feel you need to make them against.

A common question is “how do you test for sensitivity?” It surprises everyone who asks how simple it actually is: draw up five scenarios that could happen over the next forecast period (typically this will be high variable growth, medium variable growth, no change, medium variable decline and high variable decline). Use your imagination because it changes from business to business and period to period.

Brainstorm how those five scenarios would make the numbers look. Some data points will go up, others down. Use your best estimation abilities; accuracy is not vital since the scenario didn’t actually happen. Put the revised data into the model and see what the outcome is. Compare all five scenarios against what actually happened. You now have a possible range of outcomes that could have occurred.

This gives you a greater ability to understand the impact of shifting market conditions. If you believe that there is a 5% chance of high growth 15% chance of medium growth, 40% chance of no change and 40% chance of medium decline you can apply those probabilities to the outcomes to get a blended decision. You are essentially hedging against the possibilities that are likely to occur. You will be a better decision maker for this process.

Forecasting, Pt 2

Never, ever, over-trust the forecast output

Models are unintelligent. They have no means of deciding to leave out unnecessary information or retrieving information that they actually need. Unfortunately, there is a tendency to trust the outcome of previously successful models. The thought process goes: “It was right before, it will be right now.” Until suddenly it’s not; a market condition has changed leading to massive failure.

Flawed models are at the heart of the recent economic downturn. Every financial and rating institution was using the same flawed models. For more than five years the models worked with enormous success. Analysts would put in their data points, the model would return the likely outcome and the analyst would execute against that decision. A highly effective system…until it broke.

Once a model is discovered to be flawed, analysts often dive for cover under the excuse of “the model told me to.” This takes us back to the concept of trusting a model’s output. Don’t. Do not ever put too much faith in the outcome of a forecast (moving average, Monte Carlo, or anywhere in between).

Forecasting models are only tools in the digital toolbox of decision makers. The moment that they become elevated to a higher status is the moment that they should lose all credibility. Decision makers should be held responsible for their actions no matter their process for reaching the decision. Anything less leads to abuse of the system.

Forecasting, Pt 1

Forecasts are no more than a cog in a Prediction Process

Any process that ends in a decision involves forecasting the future. No decisions can be made that do not account for possible future actions. At their most basic level, forecasts are simply predictions of what will occur in the future.

Every C-level executive has a job that almost exclusively involves making numerous business decisions. Sometimes they use crude tools such as gut feel or personal preference and sometimes they use highly sophisticated, probabilistic models involving multitudes of disparate variables. The sophistication of a model does not necessarily make it better than its lower level brethren.

Moving Average forecasts are still around for a reason – they work. In environments not prone to sudden, dramatic changes a moving average forecast holds a lot of appeal: it adjusts fairly quickly to new conditions and it is easy to understand/fix. It will never predict future market condition changes but that’s understood when it is implemented.

Sophisticated Monte Carlo simulations that account for internal and external pressures are more likely to predict future condition changes. The issue with using this style model is that it is dependent on the variables it makes decisions on. Forecaster bias is a significant risk with complex forecasts. Forecaster bias is the condition where the forecaster includes only the variable he/she believes will impact the outcome. There is no way to include every possible variable and someone has to make the selection.