Edward Keating joined me on the Acquisition Talk podcast to talk about a wide range of issues. He is the Deputy Assistant Director for National Security at the Congressional Budget Office (CBO), and before that, he was a longtime RAND analyst researching a number of areas including sustainment, shipbuilding, and compensation.
Edward provides us insights into the Navy’s 30 year shipbuilding plan, how preventative maintenance can create lasting impacts, whether cost escalation can persistently outpace economy-wide inflation, if defense planning should focus on inputs or outputs, how modern ships are floating computers, why acquisition history is important, and much more.
In the episode, we discuss Edward’s excellent article in the Acquisition Review Journal, “Approaches to F-35 Depot-Level Maintenance: Insights from Other Systems.” It showed readiness and cost-per-flying-hour data for a variety of fighter aircraft. We discuss the data, and how there is a great deal of heterogeneity depending on the aircraft model or even the tail number.
Edward sets me straight about my characterization of the F-35. Neither is cost-per-flying hour the sole determinate of sustainment costs, nor is the $1 trillion lifecycle figure cited useful for thinking about the opportunity cost. For example, the $1 trillion lifecycle estimate of the sustainment cost includes anticipated inflation, which over many decades amounts to a sizeable proportion of the figure.
Podcast annotations.
Let’s pick up with Edward discussing trends in the availability of fighter aircraft, or the proportion of aircraft capable of achieving missions.
The rate of degredation of availability varied fairly considerably across the airframes. I mentioned earlier the A-10 – the A-10 is an interesting system. It had a large number of availability problems in the 1990s but has stabilized in the last 15 years.
The other side of that is the F-18, the Navy’s legacy Hornets in particular, which have had very sharply diminished availability since 2000 roughly. So if we had been talking in the 1990s, we probably would have been looking at the legacy Hornets as a laudatory system that has done very well availability-wise and the A-10 would be the converse. Today, it’s the opposite.
Edward notes that the A-10 received a great deal of depot-level maintenance in the 1990s, which perhaps put it on a good footing for the future. It bodes well for advocates of preventative maintenance. Listen to the podcast to hear how some of those lessons are translating to the F-35.
What we didn’t address was the fact that the A-10 executed a high sortie rate during the Gulf War, and many got shot up quite severely. Indeed, the high sustainment cost of those aircraft is a testament to the fact that the A-10 was so survivable that it could take incredible fire and return its pilot home safely. See more on that here, and a great interview with Pierre Sprey here.
So to me, it is not clear that the cost-effectiveness of preventative maintenance is the only lesson to be learned. It is also plausible to believe that the competitively prototyped A-10 (it’s paper design was not expected to win over the OA-9) that was also thoroughly tested (went through some 3 years of testing before the production decision) resulted in a more reliable aircraft than the F-18, which looked more like a traditional weapons program particularly after McDonnell took the lead from Northrop’s YF-17 design in 1975. It’s difficult to say, however, because they are very different systems used for different missions.
Here’s a good part from Edward on the importance of testing:
To some extent if you don’t have some failures in test & evaluation then you’re not doing test & evaluation correctly. I think the issue shouldn’t be whether you “passed” test & evaluation, but what is it that you learned. It’s certainly pernicious if we expect weapon systems to always get 100 percent favorable results…
He goes on to say that a lack of failures and error correction in operational testing means that they are more likely to be found out in the field. You hear the same story, particularly from the software acquisition folks, that the acquisition system is pushing risk onto the military operators, who are handed over systems that may not have been adequately tested, or they might be made operationally effective but not suitable.
Often, costly engineering fixes might be expedient from an acquisition perspective but have adverse consequences in sustainment or even production. As a result, systems have a lower operational tempo and require more maintenance, particularly from contractor specialists. The CBO, for example, has found that aircraft sustainment costs have experienced real growth at 3 and 7 percent annually. That growth is unsustainable over the long run.
I asked whether these rapidly growing sustainment costs, coupled with steady or growing operational requirements, would “crowd out” new system investments as has been sometimes claimed. Edward has an excellent response:
Eric: Do you think that this trend [rising sustainment costs] will actually crowd out RDT&E investment?… How can we keep maintenance costs from accelerating and creating a structural problem?
Edward: Let me provide a somewhat different perspective as someone who has spent my career more worried about O&M [operations & maintenance] issues than acquisition issues, even though they are complementary.
I think one could turn the statement around and argue that rising acquisition costs has crowded out necessary O&M. In other words, I don’t necessarily accept the logical premise that its because O&M costs have grown we can’t get all the acquisition we want.
Listen to the podcast to hear the whole thing. Here are a couple other good parts:
If you’re talking about the long run development of a weapon system, particularly technology that is nascent or isn’t yet developed, it would be impossible to do an output based contract because we simply don’t know what the product is going to look like, and how we know whether we succeeded or not from a contracting perspective.
So at that point I think you’re forced into input-based cost-plus type contracting by the nature of the uncertainty of the product. So what you’re able to measure and how you measure it certainly drives how you set up the contractual structure.
That I think is an important insight, and I’m not sure how often it is followed in the Department of Defense, or how deep into the RDT&E process it applies. Certain elements of the DOD have some sort of input-based financing, like the Joint Improved Threat Defeat Organization. Many still justify project line-items through the budget and other acquisition processes. That implies targeting some output at a cost, or else it is not following the output-oriented PPBE acquisition process.
When there is uncertainty, and when you in many circumstances want to affect a wide range of options and take advantage of unintended outcomes, the definition of “success” from a contractual standpoint is vague, as Edward pointed out. Meeting the specifications of the contract is considered success, and constraining changes rather than pouncing on them is considered a success.
Of course, there are no easy answers to defense contracting, as Edward reminds us. Here’s a final excerpt:
It turns out for any military ship, the steel itself is really a very small percentage of the entire ship. It’s less than 5 percent. The way to think about a military ship frankly is that it’s a very large computer system that happens to be on water and has steel that surrounds it. But the steel input itself is really a very small percentage of the total cost of any modern ship.
I’ll follow up on this important point and more in another blog post. For now, I’d like to thanks Edward for joining me on Acquisition Talk. Be sure to check out his article in the ARJ, Approaches to F-35 Depot-Level Maintenance, and browse through his treasure trove of RAND reports. Find more from the CBO’s national security division here, including their analysis of the Navy’s FY 2019 shipbuilding plan. I also highly recommend reading some of the work from Mark Lorell, who is Edward’s former RAND colleague and has done a good job finding lessons in acquisition history.
Endnotes and corrections:
- We discussed Arthur Herman’s Freedoms Forge about military production in WWII.
- The steel types Admiral Rickover argued cost and quality over for nuclear submarines was HY-80, largely as a result of the Thresher incident.
- VAMOSC is the generic term used for the DOD’s operating & support cost collection effort. The Navy’s specific system is also called VAMOSC. ATFOC is the Air Force’s system, and OSMIS is the Army’s.
Leave a Reply