Cost estimating for development programs has apparently been too widely credited in the Defense Department, in industry, in the Congress, and by the public with a potential for accurate prediction which is belied by the inherent technical uncertainties in developments… The precise problems which may be encountered… cannot be foreseen with accuracy. It should be axiomatic that one cannot place a price on an unknown, particularly when the development may take 10 years.
In the development of a complicated new weapon system, there are two types on unknowns. Research engineers, based on experience, can plan for some types of unknowns, but there always seem to be surprises in the form of unknown unknowns, familiarly known as unk-unks.
That was Gilbert Fitzhugh, Chairman of the President’s Blue Ribbon Defense Panel. “Weapon Systems Acquisition Process” Hearings Before the Committee on Armed Services United States Senate, Ninety-Second Congress, First Session, December 3-9, 1971.
If it’s axiomatic that you can’t put a price on an unknown, then why are lifecycle cost estimates performed ahead of development in DoD? When faced with uncertainty, the availability of options to hedge against alternative contingencies is paramount. That’s one of the things difficult to comprehend about markets: the apparent duplication brought on by unrestrained competition where the success of some business plans depend on the failure of others, all this waste is actually a source of innovation and resilience.
The Decker-Wagner study in the Army concluded that it isn’t that hard to estimate development costs for the kinds of acquisition that ought to make up ~80% of Army acquisition. That would include commercial buys, minor modifications of existing systems, and development of new systems using only mature technologies. The remaining 20% — the genuinely new technologies with major new capabilities — are harder, but that risk can be mitigated by judicious use of prototyping.
Unfortunately, that’s not the mix we buy, and we don’t do sensible prototyping. The incentives in the requirements development and budgeting systems favor claims of major leap-ahead and jumping into development before you know what you’re developing — with predictable results.