Sushi Rice and Tofu Bowl

Don’t assume no meat mean no taste!

Monte Carlo analysis provides a way to handle answering questions with significant uncertainty in the inputs that influence the outcome of the work so you can have the difficult “when, what, and how much” type conversations with sponsors, stakeholders, and marketing people. That definition is an explicit admission that almost ALL of the hard questions asked about projects cannot be answered using simple a + b = c formulas or arguments. This leads us to use tools like Monte Carlo analysis. There are four common assumptions often overlooked or misunderstood when using Monte Carlo methods.

  1. The trap of assuming that estimates, used to answer what, when and how much questions are deterministic (that knowing the input parameters and initial conditions mean that a specific outcome can be known). Monte Carlo analyses are stochastic models that assume inherent randomness in both the inputs and outputs. Randomness means that the output is a range of probabilities. If you want a single number — Monte Carlo is not for you (perhaps an Ouija board?).
  2. Garbage in equals garbage out. Monte Carlo analysis uses historical data and assumptions of risks (a number of observations of yesterday’s weather to paraphrase the common agile adage) to determine the future. Simply put, before using tools like Monte Carlo and assigning a value to the output of the analysis, perform due diligence on the historical and input data. Bad data generates bad answers. When garbage in generates gospel out, train wrecks occur.
  3. Most Monte Carlo software uses a normal distribution (normal distribution assumes that outcomes are equally distributed around the mean) as a standard assumption. The distribution that drives your equations needs to be a good proxy of the development processes. Plot out the data you are using for the analysis and then make sure that you use the correct distribution. A simple rule of thumb to alert you to scenarios that don’t follow normal distribution is to calculate the median and average. If they are significantly different, review the distribution very carefully. Different distributions will yield VERY different answers. Note that while the distribution of human attributes might be normal, distributions impacted by human behavior are often not.
  4. The past can, and does, influence the future in software development. Most of probability statistics casually used in business have a hidden assumption that each observation is independent (when flipping a coin the last flip does not influence the next flip). Humans are learning machines! Anyone that has measured or observed a knowledge worker delivering a product will recognize that the person doing the work is learning and increasing their capability over time. Many kinds of work are serially correlated therefore assuming acceleration is often rational (note: crappy management can really mess this assumption up easily).

Monte Carlo analysis is harder to generate and understand than creating a chart and plotting a line through the data is using EXCEL.  However, if you keep the assumptions in mind and find a decent tool, Monte Carlo analysis is an important tool every decision maker with a high stakes decision in their sights must have at their fingertips (or at their keyboard).