Sunday, November 27, 2016

Absolute vs. Relative Accuracy in Cost Modeling

Cost modeling, like all other modeling activities, is fraught with weaknesses.  A well-known quote from George Box, “Essentially, all models are wrong, but some are useful,” [1] is appropriate for describing cost modeling.  First, cost modeling is a “garbage in, garbage out” activity – if the input data is inaccurate, the values predicted by the model will be inaccurate. That said, cost modeling is generally combined with various uncertainty analysis techniques that allow inputs to be expressed as ranges and distributions rather than point values.  Obtaining absolute accuracy from cost models depends on having some sort of real-world data to use for calibration.  To this end, the essence of cost modeling is summed up by the following observation from Norm Augustine [2]:

“Much cost estimation seems to use an approach descended from the technique widely used to weigh hogs in Texas.  It is alleged that in this process, after catching the hog and tying it to one end of a teeter-totter arrangement, everyone searches for a stone which, when placed on the other end of the apparatus, exactly balances the weight of the hog.  When such a stone is eventually found, everyone gathers around and tries to guess the weight of the stone.  Such is the science of cost estimating.”

Nonetheless, when absolute accuracy is impossible, relatively accurate costs models can often be very useful.  Relatively accurate cost models produce cost predictions that have limited (or unknown) absolute accuracy, but the differences between model predictions can be extremely accurate if the cost of the effects omitted from the model are a “wash” between the cases considered--that is, when errors are systematic and identical in magnitude between the cases considered.  While an absolute prediction of cost is necessary to support the quoting or bidding process, an accurate relative cost can be successfully used to support making a business case for selecting one alternative over another.

As an example for a relatively accurate cost model (with little or no absolute accuracy), consider a model for selecting the best way to maintain a system.  In this case the question to be answered is not whether or not to maintain a system, nor how much to budget to maintain a system, but rather which of several alternative approaches one should use.  Presumably once an approach is chosen, more accurate modeling can be performed for budgeting purposes.  The alternatives might be preventative, predictive, and corrective maintenance, and combinations thereof.

[1] Box, G. E. P. and Draper, N. R. (1987). Empirical Model-Building and Response Surfaces (Wiley, Hoboken, NJ).
[2] Augustine, N. R. (1997). Augustine’s Laws, 6th Edition (AIAA, Reston, VA).