Is requirements stability the problem?

Question: We all know that what drives our cost estimates is not better CERs, or better data bases, or better mathematics, or better communications. In my view it is this constant, incessant, undocumented, creeping problem of changes in requirements. So I would throw a challenge to each of you individually to influence and to do what you can to at least put a pause to this constant change in requirements so that those of us who have to make cost estimates can do that better job that you ask us to do.

 

Response (Wayne Allen): We are talking about a systemic problem. It is a problem of how do we obtain increased stability in the acquisition process, or, said another way, how do we minimize disruption? That, in the Army’s case, gets back to something as fundamental as our internal requirements-setting process, how many players we have in that process, and how many times those players are second-guessed as requirements issues pass up and down the hierarchy. The number of voices that must be accommodated contributes to the instability in the requirements over time.

 

All of which says, I don’t have a good answer to your question. I have noted that, within the Army, we are talking about requirements stability and this is at the three- and four-star level. However, institutionally and systematically, we tend to shy away from the word “freeze” as we want “flexibility.” It has to do obviously with such things as the annual funding or annual budgeting process and how that process causes instabilities, and a lot of that is beyond our control.

 

But we can’t carry that thought too far or it can become a cop-out for not trying to bring stability. There are some mechanisms that we are working with in the Department of the Army to try to bring about increased stability. One such mechanism is a better statement of materiel system requirements from the outset to reduce downstream turbulence.

That’s an excellent answer from former Army cost chief Wayne Allen. Basically, the thinking had been that inaccurate cost estimates caused decision-makers to change their requirements, such as cutting capabilities or ordering fewer quantities. But cost estimators responded that they couldn’t put a number to a requirement that changes, as though knowledge in a technology growth environment stands still. We still hear often how getting the requirements “right” and locking them down is the most important step in setting up a successful program or contract. 

Wayne Allen, however, realizes that requirements flexibility is needed so that requirements and technology are updated in tandem as more information is learned along the way. He pointed out that what people think they want in the DOD acquisition system is stability. They want to minimize disruption by setting “good” plans and executing to those plans as tightly as possible. Any deviation from the plan is seen as bad, usually because deviations are on the high side for cost and low side for capabilities.

But Allen also points out that we don’t want to “freeze” requirements. He points to the obvious mechanism of creating a “better statement of materiel system requirements,” but also notes that the requirements are often corrupted by the huge number of players that get involved on issues that pass up and down the chain of command many times. This generates a lot of space for uncertainty absorption and other ways in which errors are introduced into the system.

I think requirements are destined to be a forum to insert all sorts on unvalidated design choices based on bias or ignorance. “Creating the better requirement” outside of creating a better culture — a better atmosphere of adaptation — is doomed to failure in my mind. That is because the requirement is basically viewed as a binding obligation, an obligation that is necessarily formulated ahead of the relevant information that is needed to decide upon its efficacy.

I think Ed Catmull’s maxim holds from Creativity, Inc: 

If you give a good idea to a mediocre team, they will screw it up. If you give a mediocre idea to a brilliant team, they will either fix it or throw it away and come up with something better.

Here’s another (perhaps solid requirements — the appearance of knowledge — is what stifles innovation):