When should cost-effectiveness studies start?

… you can’t apply cost effectiveness to really basic research. Otherwise you never would discover new genes in biology for example if you had to prove ahead of time you were going to find something in order to get money and make the study.

That was Gilbert Fitzhugh, Chairman, President’s Blue Ribbon Defense Panel, in a December 1971 testimony to Congress, “Weapon Systems Acquisition Process” Hearings Before the Committee on Armed Services United States Senate, Ninety-Second Congress, First Session, December 3-9, 1971.

I think that’s a really important concept. If you have to know ahead of time what will be found — and have a precise plan to get there — in order to justify costs and have resources allocated, then you can only seek minor advances in the state of the art.

There is another tension, that new developments should also be bigger, better, and faster than their predecessors. So this usually means taking existing principles and scaling them up, sort of like the C-5 cargo aircraft was expected to be just a “linear extrapolation” from previous, smaller aircraft. This method provides quick access to end states, detailed plans, and cost estimates, but it neglects the problems encountered by nonlinearities or complexity of integration. And so it often leads to large cost growth, whereas a completely “sideways” approach may have revealed a much more cost-effective solution.

Here’s more from Fitzhugh:

Cost estimating for development programs has apparently been too widely credited in the Defense Department, in industry, in the Congress, and by the public with a potential for accurate prediction which is belied by the inherent technical uncertainties in developments… The precise problems which may be encountered… cannot be foreseen with accuracy. It should be axiomatic that one cannot place a price on an unknown, particularly when the development may take 10 years.

 

In the development of a complicated new weapon system, there are two types on unknowns. Research engineers, based on experience, can plan for some types of unknowns, but there always seem to be surprises in the form of unknown unknowns, familiarly known as unk-unks.