Podcast: Risk, Cost, and Project Management with Christian Smart

Christian Smart joined me on the Acquisition Talk podcast to discuss his new book, Solving for Project Risk Management. He is the chief scientist at Galorath Federal, and before that he was the cost chief at the Missile Defense Agency. We touch on a number of important issues, including:

• Whether Augustine’s Laws still have relevance
• The track record of NASA’s better, faster, cheaper program
• How to do cost estimates on data projects like AI/ML
• Whether the DoD is trying to jam too many programs into the budget
• The effect of MDA’s matrixed organizational design

In his book, Christian talked about a prevailing belief in the 2000s that the Department of Defense could benefit from a “free lunch” when it funded to portfolios of projects. Similar to how diversification between uncorrelated assets gives investors the chance to get the same return with lower risk (or a higher return for the same risk), funding a group of projects at the 60th percentile cost estimate could achieve an 80 percent confidence level for the portfolio overall.

Christian argues that development projects have asymmetric probability distributions for cost and schedule. You are more likely to see black swans in the cost growth direction than you are in cost savings. When more projects are put under a portfolio, the likelihood that one of them will have relatively extreme cost growth increases. We discuss the implications of this result, and what procedures managers can take to mitigate or even remedy the effects.

Cost growth rules

One of the things researchers have worked on a lot is cost growth factors. For example, Augustine’s Las Vegas factor of planning says that a development project will grow by 52 percent. Experience from a number of sources seem to roughly bear that number out.

But there seemed to me a problem: let’s say a cost estimator used best practice cost-schedule risk analyses — shouldn’t this estimate be exempt from application of a cost growth factor? Wouldn’t setting aside more funds simply lead to Parkinson’s Law: that work expands to fill the time [or budget] allotted? Here’s Christian:

Is the estimate an independent estimate or is it, or are you working for the project? Because you’ve worked on the project and then your inputs are going to be biased by the optimism of the project manager.

I think that’s a good point. The internal machinations of program advocacy often leads to pressure on cost estimators to come up with a number that fits into their expectation of what the budget will allow. This is called a “must cost” estimate, rather than a “will cost” estimate.

MDA matrix organization

In the military services, the cost estimator in the program office reports to a single boss, the program manager. The services also have an independent cost estimating office (DASA-CE in the Army, AFCAA in the Air Force, and NCCA in the Navy) which reviews program cost estimates. The Office of the Secretary of Defense in turn has its own independent cost office, CAPE.

The Missile Defense Agency has a different organizational design, as Christian explains:

[MDA has] a hybrid approach where you have a functional chain of command and a matrix chain of command. So it’s a criticism of something of attention. So there was always some tension between the functional chain of command and the project chain of command.

 

But you do get the insight across organizations when you do it that way. [It] took an agreement between the functional chain of command and the program chain of command before they can go forward. And that does slow things down a little bit, but if you do get concurrence and you do get the insight across the functional organizations that need to have insight into specific areas.

Does agile reduce risk?

Christian says that development projects using agile have perhaps seen some decreases in costs (though it’s hard to tell), but wonders whether it reduces risk:

That’s the part I don’t really see, where is it that agile is reducing risks? They are breaking the work into smaller chunks. There is always a trade off between cost schedule performance. So to some extent, that sound sounds like they’re almost in the short-term sacrificing performance to achieve a lower cost and a shorter schedule.

I think that’s one of the points, perhaps. You want to put risk on the performance side. That way, you don’t experience extreme cost/schedule growth to achieve a program requirement. The flip side is that you open yourself up to new opportunities that could revolutionize performance.

One of the key questions is: Is there such a thing as a “must-have” requirement? For example, was NASA’s James Webb Satellite Telescope important enough keep throwing money at it at the expense of a great number of other astronomy programs? Or are requirements pretty flexible?

One thing about agile that I think is underappreciated is that it should actually benefit cost estimation. Consider a world of once-in-a-generation, exquisite, fully-integrated programs. You simply won’t have enough degrees of freedom to do any realistic cost model. With agile development across an ecosystem of standard components/subsystems, there will be a far greater number of data points to draw inferences from.

Thanks Christian Smart!

I’d like to thank Christian Smart for joining me on the Acquisition Talk podcast. Be sure to check out his blog, Smart Remarks, where he writes about these subjects and more including coronavirus statistics, as well as his podcast Smart Remarks, Howarth States. Here’s me blogging about Christian’s blog. His book is now out in stores, Solving for Project Risk Management, so pick up a copy! Christian is a serial best paper winner at the International Cost Estimating and Analysis Association, so much so that they dedicated a page to his collected works. Transcripts for this podcast are available here.