Eric Lofgren: [00:00:00] I’m speaking with Christian smart today, who is the chief scientist at Galorath federal, and his new book is called solving for project risk management. Christian also runs a blog called smart remarks in a podcast with Doug Howarth named Smart Remarks, Howarth States. And before that Christian was the cost chief that the missile defense agency. So Christian, thanks for joining me on acquisition talk.
Christian Smart: thanks for having me on Eric.
Eric Lofgren: You’re welcome. So the book is terrific. The book is out. And we’re going to be dancing around a lot of those topics here today. But the first thing I want to start off with is — give you a question about software cost estimating.
So if work units like ESLOC and function points are used to estimate software, then what do you use to estimate data projects such as for artificial intelligence and machine learning?
Christian Smart: That’s a great question. I don’t know that I have a fulsome answer, but I would like to talk about it because that’s a fantastic idea. It’s a big thought in a way, because I don’t think that when we do data projects, we tend to look at the scope in the same way we tend to look at, Oh, we got to do this work.
Here’s some money we throw at it. And then we try to get as much done as we can. The biggest part. It depends on, if you have data or not, that’s one of the key things. If you have to go out and collect data and then normalize the data, that’s the biggest part of the project that’s, as from screenings, probably that’s probably 89% of the work that’s involved.
That’s the problem that we have a trouble getting our hands around on in terms of scope. Because some extent there is a matter of — you collect what you can get and then you had some way to stop and then you have to get the data cleaned up and get ready for an analysis part, depending on, using modern tools like R and Python, that can be, you could look at, what are the lines of code or that kind of thing.
I guess the question is how do you count that? Like you use R using a lot of built-in library. So is that, is that counted as auto-generated code and your ESLOC or, how is that counted kind of thing. which, accounts for something, but usually when you use our, you don’t really make any changes to that.
So there’s really no debit to your lines of code count or, using a built-in library. that’s good. It’s a great question. I don’t [00:02:00] know that we have a really good answer when I, and I’ve done these kinds of projects in the past — when I was at the cost chief at missile defense agency, we had a research group and we adopted an agile approach to doing all kinds of research projects there.
the kind of the, the sizing metric for that, aren’t you been story points. So that’s what we’re going to use there.
Eric Lofgren: Yeah. It’s interesting that you were bringing back the data piece into ESLOC, the source lines of code, or, try to use like story points. So use the software metrics that we’re using today, but it seems like a lot of people are also Dissatisfied with those metrics as a proxy for what work is actually involved in how to estimate those costs. You said something actually pretty interesting in the book. And I want to get your view on this, because it was just like a short sentence and I want you to expand on it.
So you said the agile approach quote has potential to reduce costs, but the jury is still out on reducing risk. can you explain what you meant by that?
Christian Smart: Yeah, agile is highly touted right now, and a lot of people are adopting it and using that framework. And, there’s been some analysis that’s done on agile projects and how it compares to costs.
And, it’s kinda some mixed bag of the analysis that in terms of, maybe it decreases cost a little. It hasn’t been, the results I’ve seen have not been statistically significant, but I think there’s some potential there. really about taking an agile approach, are you really reducing risk?
And that’s the part I don’t really see, where is it that agile is reducing risks? They are breaking the work into smaller chunks. some extent there is always a trade off between cost schedule performance. So to some extent, that sound sounds like they’re almost. Some extent in the short-term sacrificing performance to achieve a lower cost and a shorter schedule.
So they’re working within that framework to try to achieve lower costs and shorter schedules, but are they sacrificing performance? Because some extent it’s we’re giving ourselves this amount of time. We’re going to try to achieve this small goal and maybe that’s at a lower cost, but is that really offsetting the risk of the overall broader picture, the broader project?
And that’s just where I’m just struggling to see [00:04:00] where agile really reduces risk. maybe it’s reducing risk a little by taking a slightly lower performance to achieve shorter schedule and lower costs. But I’m not really seeing. how it’s getting rid of the risks are out there.
Cause there’s a lot of, a lot of the risks that are outside of the project. whatever approach you adopt, you can’t really offset those risks.
Eric Lofgren: Yeah. This is an interesting discussion. So I want to actually get into this and I want to maybe push back on you a little bit, but first, we’re talking about, the risk of these projects and one of the beliefs, and I want you to expand on this and then I want to get into a discussion.
So one of the beliefs in the early 2000s, and you were highlighting this in your book, is that there was like a free lunch when you fund a portfolio of projects rather than the individual project. If I fund the portfolio project, I can get away with funding it to a lower confidence level because of something like a diversification effect, and that was like the belief that people had and you devoted a good portion of your book to refuting that.
So can you describe what the story was there and then what you were like getting at with your solution or your proposal?
Christian Smart: Sure. So the portfolio effect, is there idea that, like you said, it’s an idea that you could, achieve, for example, an overall 80% confidence level at the portfolio, funding each individual project to a lower confidence level, such as a 60% confidence level, there was a report that came out I think it was the national defense council, in the early 2000s that talked about the recurring problem of costs growth for venture projects and advocated funding, each individual project to an 80% confidence level, doing a risk analysis, and then funding that to the 80% confidence level. There’s two kind of key aspects there, even though mainly defense branches do risk analysis and missile defense agency does risk analysis on all this projects.
the CAPE still does not, do risk analysis. it doesn’t do cost risk analysis. so there’s still, there’s this advocacy to do risk analysis and then also do the risk analysis. How does that inform your budget? And so they need to do the risk analysis and fund to an 80% confidence level.
So [00:06:00] there was a paper written, that year or the next year that gave a simple example that showed if you were to fund 10 projects to a 62% confidence level, just a notional example, you would achieve an overall portfolio level confidence of 80%. So the idea was. if you fund each individual project or an 80% confidence level, you’re going to be an extremely high confidence level for the entire portfolio.
So that’s probably a bad idea. so they use this idea without actually adopting any sort of portfolio level analysis. Just use the idea. Okay. We can fund a lower levels. In many organizations, they said, okay, we’ll fund it at 50% confidence level. that’s problematic for multiple reasons.
One of which is you’re below the mean. So when you’re below the mean, now you’re setting up a negative portfolio level. So that’s one issue. The other is the analysis that was done. This notional example was too simplistic. It was, it didn’t account for correlation between projects. It didn’t account for a realistic amount of risk, a couple of issues with that.
So once you accounted for that, Pretty much that portfolio effect more or less went away.
Eric Lofgren: so here’s my kind of like interpretation of the thing. So, when you’re doing your cost risk analysis, and you put them into a portfolio. So now I have multiple projects. I think the presumption is that cost growth for each project will follow some kind of distribution that’s like log normal. So it’s more like if there’s an asymmetry, so you’re more likely to have cost growth than you are to have cost decay or cost savings. So that’s one of the issues with the distribution. And then when I put them into a portfolio, It’s now more likely that I’ll have one of these extremistisan, right from Nassim Taleb, we’ll have a black Swan, and then that will be higher.
So like in order to fund that portfolio, I need a higher confidence level relative to the individual projects, because now I have a bigger sample and that bigger sample is more likely to include an extremististan cost growth. Is that kind of right? The right way of thinking about it?
Christian Smart: yeah. So even if everything is log normal, you will see some, you won’t [00:08:00] necessarily see a negative portfolio effect if you fund the mean, but you won’t really see much savings, but if there is, one of those, you have a, do have a chance of having something in there where you have uncontrolled cost growth, which is, in those extremistan events, then you can have a negative portfolio effect.
yeah. even if you find a high confidence level, so you could find everything to an 80% confidence level and still be at an overall portfolio level confidence of less than 80% in such a case.
Eric Lofgren: I think this is where for me, it comes back to agile, Because I think, and you grappled with this in your book as well, but I think one of the points here is that when we have a portfolio of 10 projects, it’s not like we’re going to let all 10 projects run their entire waterfall course. And then we will experience those big cost growths. So I think some of it, and I think Nassim Taleb, and then also Benoit Mandelbrot in the anti-fragile book, they were like, what’s the best thing to do is diversify across all of your — if you have 10 projects, then your funding should be like one over 10 to each project or something like that, spread them. But I think some of their point was also like, you want to be able to have a big filter. So before I get to the extremistan cost growth, I should be, I should have cut that.
And then, so now I’m left with an overall portfolio that like I’ve cut out the biggest losers, the ones that are like the black swans to me. And then I keep the other ones and that’s how I move forward. And there was like this idea that you grappled with in your book that’s like, — what if you just had a 25% cost growth rule?
anything that grows over 25%, I just cut it out. that relates to me for, Agile because it’s like, it’s a risk management procedure where you incrementalize these projects. And then over time you can see where they’re going and they’re competing or whatever, and you can make those trade-offs and make sure you don’t get into an extremistan.
So that’s like my version of risk management is just like making sure you, you collect the 10 X or a hundred X gains and make sure you don’t collect the 10 X, a hundred X cost growths.
Christian Smart: Yeah, that’s a good [00:10:00] point. so if it done right, you would say that agile basically would allow you to recognize the things that are not working and just cut those out or things that are not going well. That’s a good point.
Eric Lofgren: I guess one of my things is we have let’s just say we have a log normal distribution on cost and schedule growth. But then the flip side of that equation is. What is the benefit, right? And so the benefits, usually we think of in the waterfall thing where it’s just the benefit is defined and then we have a cost schedule distribution around getting to that singular benefit or specification.
But if we also have like this, if we open it up a little bit, and have this range of options in terms of performance outcomes, that might be right. So if you have agile is more uncertain and more risky, but then the question is. Is there an institutional process for getting rid of the bad guys and collecting the good guys, Such that you’re really transforming yourself. So what if you have a pareto distribution or a wider distribution on your benefits relative to your costs? So you don’t really worry as much about cost growth, right? Because like overall your portfolio would be collecting the big guys, the big game changers.
Christian Smart: Yeah. You wouldn’t worry as much about cost growth or scheduled lays. You would worry about performance right. In that case. Yeah. That would you would, if performance were, pareto, then you would definitely want to take. Performance as your driver, unfortunately pipeline, project managers treat performance as pareto and everything else is secondary because they tend to focus exclusively on performance.
Eric Lofgren: Yeah, I think that’s right. So it’s interesting. Because I had, Dan ward on the podcast like a, about a year ago now and he was actually looking at these kind of cost growth rules. And it was just like, if you grow over 15 or 25%, just make that a rule. you didn’t seem to come down on that side as much.
And so do you see like where you and him might be like having a difference of opinions on whether you follow the cost growth rule or whether you like fund to the portfolio estimate that you would have a [00:12:00] suggested,
Christian Smart: yeah, so this is a little bit hard because the, the things that wound up. No having the biggest cost growth are often high priority projects. NASA, that would probably be one of those would be James Webb space telescope. do you want to just get rid of James Webb space telescope and cancel it? now to some extent they’ve already spent $10 billion on it, You shouldn’t make your, decisions based on some costs. there’s some cost fallacy, but, you do want this, next generation of space telescope to come out as the successor to Hubble. And if NASA makes that their priority and that’s what they decided to do, then, I support that.
It’s just a matter of, you need to have the reserves in place so that, you don’t wind up having these many successions of delays and then further cost growth caused by hitting funding constraints.
Eric Lofgren: I believe I heard this a while ago, but I’m not really sure about it there. I heard that NASA actually does have a portfolio — they’ll like fund to the cost adjusted risk estimate.
And then they’ll take that slice between 70 and 50 for each project. And put that into kind of like a central kind of like cost growth management fund. Does that exist, one, for NASA, and two, is that something that you see as being useful to projects?
Christian Smart: Yes, NASA does. So NASA has basically there’s two levels. they would, for large projects, they would, they would look at a joint 70% cost and schedule confidence level. So a little more stringent than just cost alone. and I think that’s a good thing. So they would hold that internally. And then, they would give the project manager a 50% a number, and then the contractor would come in and bid well below that.
So they might send a contract for level below that. So that would be a lower number. So there’d be two levels of reserves. One is the contract, would have a contract for say, yeah, X and then the project manager would, would have some levels of its own reserves up to a 50% confidence level. Now, if the contractor blew through his contract and then, cost typically cost plus contract, although, [00:14:00] NASA was trying to make a human landing system, I fixed price contract.
So it is interesting. I’ll have to get your thoughts on that, but for large multi-billion dollar development contract, but, So once you blow through that and you get to a 50% confidence level of the project manager, you blew through that. And then the project manager had to go back to NASA and say, Hey, I need some more money.
And they would have some reserves in place to offset additional growth. So that’s not a bad way to go. It’s Yeah, I think that’s a better, than some organizations do. It’s still, you’re still ignoring some of the really extreme risks around the tail looking at that. So you’re not looking at those extreme risks, but I think that is a better practice than in other organizations I think you’ll see. Especially like DOD.
Eric Lofgren: Yeah. So you were at the missile defense agency, which actually. It’s like separate and they have a little bit more control over what they’re doing there. They’re their own component. how could you implement this reserve budget, kind of what NASA is doing, in the department of defense where it seems like, we fund directly to programs and having these kinds of — a lot of times Congress looks at them as slush funds for department of defense. I’m not really sure why, or how NASA was able to get that authority.
But how do you think about implementing that in the department of defense or, is that something that the DOD should really be looking at?
Christian Smart: I think so, yeah, that is the, that is the issue is that, money that’s just sitting there, there’s, there are people that are gonna want to use that and, Congress, people, they have companies that work in their district. And they’re, if see, there’s money sitting over here, that’s not being used. They’re they’re gonna want to try to get it to their constituencies. so there is that temptation. I think that is something that they should do.
If you look at the missile defense, missile defense would get an annual budget. the past has been around $10 billion a year and, and then they would, have some authority on how to fund individual projects. And then there was actually some amount of rectors reserved. Now there’s usually a very small amount, but there was some amount of records reserved that he could apply to, to overruns.
But, there was some, a small amount there. but yeah, I think that would, I think that would be good if the, the services could get some authority to hold some reserves in place, and [00:16:00] have their own reserves. it is very tempting. It’s very hard to implement because like you said, Congress and other folks, see that idle funds and they want to be able to, they want to use them.
And, that’s one of the weird incentives in DOD is to try and spend your money as fast as you can — get reserves in place.
Eric Lofgren: I could imagine like the DOD, one, it depends on where these pots are. Is it like for each component, has their own pot, or is it like adjudicated at a higher level?
And then you could imagine a lot of these guys like, Oh, we want to do all these new starts with this funding. And then how does Congress feel about that and what would those processes be? I thought it was interesting that you brought up the fixed price, NASA program, artimus. So why does that interest you? what you’re thinking it’s a develop, it’s a big development program. Billions of dollars. It should like fixed price seems weird for it. So why do you. have a question there?
Christian Smart: It’s just is NASA really — or is a contractor really willing to sign up for whatever the number ends up being. Let’s just throw out 10 billion, just for example, say, is NASA really willing to sign up for the $10 billion contract — is a contractor will they be willing to sign up for a $10 billion contract? If they. If they think, really is probably going to cost say 15 billion. I’m making these numbers up.
I’m having a little bit of insight into that because I’ve done some estimates for it. I’m just making some numbers up. so you know, the real thing is 15 billion. are they really going to — it is a competitive bid, so there’s, Dynetics, blue origin and space X they’re all bidding on this work.
Now is the contractor willing to take that risk of eating potentially massive cost overruns by trying to establish a competitive bid. Is NASA, willing to, really does it necessarily think that they’re willing to fight that fight? On the other hand, if it, if they want to, try to, have a $10 billion contract, are they willing for the contractor to have a 25% profit, for example, because that’s the other side of fixed price contracts is what if they think it’s going to be 10 billion? they signed a contract for 10 billion, but the contractor can do it for 8 [00:18:00] billion. Then they make $2 billion in profit and it’s 25% of profit. is that a good thing to do? So there’s, it’s kinda those two sides of it.
Eric Lofgren: my reaction would be like, if they made someone made 2 billion off of it, shouldn’t that really encourage the competition to enter? but
Christian Smart: Oh yeah. No, and yeah, so it’s a good question. And I just don’t, given that Dynetics, space X and blue origin, space X is probably going to come in with a very low number.
So is it. can space X execute, maybe. space X can do things cheaper than NASA and traditionally, I think do things faster, but can, can they also meet the 2024 goal too, but people on the moon?
Eric Lofgren: Yeah, I just heard. Elon Musk just recently he was claiming 2026 is the date or the year for getting humans onto Mars.
Christian Smart: that’s, pretty ambitious.
Eric Lofgren: Yeah, that was my reaction. I think one of the things that we’ve seen is that, and NASA actually admitted this, That Space X is doing things better and for cheaper than they could have done it without them.
So there’s some, and I think NASA is also I guess my view on the fixed price is that, first this program doesn’t really seem to have the same production and sustainment payoffs, right? Oh, I need to buy into this program because there’s going to be these huge backend sole source things going on for me.
But potentially, Elon Musk might really be thinking like that [buying in] too, Like I just need to be the first to build this huge moat in terms of, space exploration, but. And that should be good from the taxpayer’s point of view that he’s going to be taking that risk. But I think the way that NASA actually approaches this, like when they first started doing business with space X, they actually used an other transaction. and then they broke it down into milestone increments. So I’m not really sure what the structure is. I’d like to actually look into that, but, if they can modularize the fixed price contract, it seems there are steps along the way where you could probably, put that into a fixed price term, and then you move along so you can know okay, these guys are way behind and we can either like, cancel this or restructure it at that [00:20:00] point.
So I think like when you break them, break down the contracts, they almost start looking a little bit more like a cost plus to some degree. but with like real test and evaluation pieces there.
Christian Smart: Yeah, my understanding is right now, they’re going to carry all three contractors through PDR and then get the fixed price bid at PDR. And then that’s the rest of the contract down select from PDR.
Eric Lofgren: So one big contract after PDR to take them through literally the landing on the moon?
Christian Smart: Yeah.
That’s going to be, that’s going to be a big experiment though. So I guess DOD should be watching to see what happens.
Christian Smart: Yeah, no matter which way, when I was at missile defense and notice this, that, whatever happened. The contractor always wound up doing okay. It’s what also is case of heads? They win. Tails, we, the government would lose
Eric Lofgren: a while ago I took down all, I dumped all of the financials from a bunch of sectors and then I just categorize them by sector. And I was just like, okay, when you look at defense, yes, they make about 10%, profit on sales, but it’s a constant, it’s like very little volatility around that over time.
it doesn’t seem like they’re going through the swings of a lot of the commercial markets where like retail and healthcare have these like very wide distributions in terms of returns.
Christian Smart: in our own industry, you don’t see the, seeing the layoffs and that kind of thing. It’s just fairly steady, in terms of defense and aerospace.
Eric Lofgren: Which is also interesting because the very big winner take all type programs, at least within a company, it creates those mountains of ramp on and then ramp off onto these projects. And it’s it becomes very scary when you’re looking at the edge of a cliff.
And you don’t have another big program to backfill that. And so I feel like that’s also one of the reasons, right? You see a lot of these, you’re like first you have the funding to fill that up, but then the companies need that as well. So you get into those bidding wars.
Christian Smart: you do see that. companies do spend, a lot of their own money and generating these bids on the other hand they make it back up because then if they spend the [00:22:00] money, so Lockheed built a building in Huntsville bidding on the contract years ago, spent their own money on that. They lost the contract, but then that costs wound up going into their overhead, which they then charged on other government contracts.
So they wound up getting paid back for it know, in certain senses, all legal, DCMA, DCAA had no issues with it. It’s just every this there, they’re not doing anything they’re just, they’re just working the system.
Eric Lofgren: And you would expect that. when you, I think a lot of the companies, the big primes, they have that Advantage because they have a big portfolio of programs that they can smear these things across and they can like Boeing, for example, has a big portfolio.
They have a lot more commercial business, so they’re more willing to take some of those hits in the near term and even if they do, like any kind of For a Lockheed Martin, 100% of its business, pretty much it’s government. So those costs have to be just like reimbursed somewhere or else, like it’s just hard. Yeah.
so I wanted to stick with NASA for a second because you were pretty critical of NASA is a better, cheaper, faster program that they had in the nineties and maybe into the early two thousands. So can you just explain what was NASA doing and then what was your critique?
Christian Smart: NASA to their credit– and then they were boxed into a certain problem because they a cost for increasing, systems getting more expensive. The budget was getting flat. And so something had to give right there, you can’t it’s unsustainable trend. and so they had to do something.
And so in, in around 92 or 93, when, Bill Clinton took office and Dan Golden took over as the NASA administrator, he decided he was going to do things cheaper, and with a goal of, trying to get more done for less. And so the problem with that is. That they took on a lot of technical risk.
I talk about the trade off between cost schedule performance and NASA for most of its life has tended to focus on [00:24:00] performance exclusive to cost and schedule. At that time, they went the other way. They’d focused on cost and schedule disclosure performance, and that led to several, quite a few failures, aerospace, Dr. [George] Mueller showed that, that resulted in more failures. I developed a model — can I show that trade-off showed that you had more failures based on that? The problem was they didn’t really look at the analysis up front to see what the interrelationship was between cost schedule and performance, and trying to trade those off.
I think NASA and other organizations should try to trade those off. there’s a, there was a initiative called CAIV cost is independent variable years ago. That kind of has fallen by the wayside that kind of looked at that, trade-off cost schedule performance, but you got to do the analysis up front.
You can’t just simply say, we’re going to do this for less. And then when you’re taking more technical as a result. NASA, wound up, bouncing one spacecraft off the atmosphere of Mars, and off into deep space. And then they’re supposed to be an orbiter for Mars. And then embedded another one in the Marsh and a soil that was supposed to be on a Lander is going to be on the polar Lander.
there’s a couple of high-profile failures where they just try to do things too fast and too cheaply.
Eric Lofgren: yeah, I’m wondering, is, I don’t know enough about the whole subject, but I guess some of it is like, We’ve been hearing from leadership that you want to fail more often so that you can succeed.
And so it’s what’s, I just don’t know what the balance of that is. Because I feel like there could be a benefit to that if you’re like — you implement these rules, no big cost growth. Let’s do things cheaper, faster, and then, Oh, a lot of things fail, but some things like really succeed. And then you get into this, do-loop that you’re like growing the winners and making sure the losers stay away. But then there’s also that trade off where it’s we’re, where’s that line because it’s not necessarily always true that the winners will be more important than the losers. So like when the losers start stacking up, then it’s a real problem. So I guess that’s like the trade off there. It’s just where was NASA? So you think NASA was closer to the side where the winners weren’t really compensating for the losers overall to the status [00:26:00] quo of, making sure you define your risk up front.
Christian Smart: Yeah. there were two kinds of, the, some of the bigger projects that they were just, they were forced into a box of taking on more technical risks. So there was more failure. And I don’t think that overall it ended up paying for itself. the other side of that is that some of the, the centers like Goddard space flight center that was supporting at the time, they, shifted a little bit by doing more, very small single instrument satellites so that they could stay within the box.
So cost and performance, but then you lose out on economies of scale by having a single platform that have multiple instruments. So you wind up making. it’s not a very cost-effective way to achieve science objectives because you have a big platform that has multiple instruments.
There’s gotta be some sort of economies of scale. That’s another thing that, the NASA didn’t really look at was, where’s the right balance, in terms of — but what is the right balance between the cost and the schedule and the performance and how much general career should we take on?
it’s just something that’s not done upfront. I developed a tool that kind of look at kind of a logistic regression model that would say. Hey if you have this balance of cost schedule performance upfront, you have about 90 per 90% chance or 80% chance of succeeding. and you can look at that trade off and so trying to stay within the box.
So Goddard would use that model when they would evaluate a new mission to see is our cost schedule performance in the box in order to achieve their objectives.
Eric Lofgren: So You don’t want to have these extreme kind of problems where you’re pushing — you’re trying to do too much technical capability, but you can train the cost and then that creates these trade-off to these problems. you also said in your book that if cost growth was more extreme than log normal, a point estimate would be useless. And so what makes a point estimate useless?
And then like, when I’m thinking about, actually making programmatic decisions, I can’t operate with a range, I have to choose a number. Can you explain that breakdown a little bit? What did you mean by if it’s more than log normal a point estimate is useless and then like, how do you relate the [00:28:00] cost range of outcomes to a point estimate in actual decisions.
Christian Smart: Okay. A great question. So if you look at — you’re more extreme than log normal. So you look at something that is extremely heavy tail as a pareto type districts that’s and talent talks about these things — denizens of extremist stand, these projects really extreme outcome then, then there really is no, there really is no variance if you get really beyond that.
And there’s really no, there’s really no mean. There’s no defined mean. Sometimes they call it infinite and we live in a finite world. if you look at these distributions, they suggest that there is infinite mean variance. So examples of that are the, number of deaths due to pandemics applicable to what’s going on right now.
that’s if you model, you look at the history of that and he modeled that, you’ll see that they have an infinite, suggest an infinite mean infinite variance. it, you look at the, The fluctuations in the stock market, things are going up right now. And we look at this fluctuations, financial prices, it suggests infinite, mean different variants type of model sometimes.
definitely infinite variance, depends on the product. It could be finite. So if you get into the realm of infinite mean infinite variance, there’s no sample mean. So any sample that you look at, it’s not indicative of an underlying population mean because there is no population mean.
Because when we had to kick like statistics and we look at, for example, cost growth, if things were really extremistan, whatever cost growth number we came up with as an average would not really be indicative of what the overall population mean was, because there’s not one. so there’s, so point estimates are relatively meaningless in the sense, if you’re trying to estimate a mean, you can’t really do it.
So if you look at, you were looking at traditional linear regression, linear regressions aligned through the meat and that, so that doesn’t really. That doesn’t have a, something meaningful in that case, because whatever you’re doing is not indicative of what’s going on in the entire population.
So there are ways of getting around that. You can look at extreme value theory is one way to [00:30:00] handle extreme statistics. And so then you would wind up looking at something like a percentile the distribution instead, and you funded that, or you could look at some sort of risk measure around that.
So you come up with a distribution and you can still come up with a budget. Maybe based on a risk measure and you can do that for the project, or you could do a portfolio analysis and then do it for the entire portfolio and then allocate the reserves back to individual projects.
Eric Lofgren: in the pandemic where you could have these extreme outcomes where everybody dies and it’s like that’s the most extreme outcome.
And you’re just trying to say okay, what’s the impact of this next, of this next virus that comes out or disease is I don’t really know. It could be way out there. And that might be very unlikely, but that’s just going to skew everything that’s in my statistics.
And so it becomes uncertain and I guess the. project management point of view of that would almost be like, I have a project it’s very high priority and it’s a big project. And then it starts growing, and I let it grow. And then now it takes up the entire budget I have and now I’m going into debt and now it’s like consuming everything I have and it kills me.
Christian Smart: and to some extent, James Webb, you can argue James Webb space. Telescope is like that because it was called the, the telescope that ate astronomy because it ate up the entire budget for astronomy projects and NASA for a while.
Eric Lofgren: when I think of Nassim Taleb how he talks about, pandemics, I guess like in that situation, what he was — I think one of his things was like, okay, you need to have a robust rule in terms of protocols that you will enact to make sure you never get to that super extreme thing.
The one thing, your risk management strategy is never to have that extreme thing. And so does that not translate to project management? And is that not something like a cost growth rule? no matter what, we can’t make this thing, get so big that it consumes us and then we’re left with this one capability, even if it’s a sweet capability, it just ruined us.
Christian Smart: Yeah. So the precautionary principle and the Taleb was right saying, we need to not just react, but it would look like overreaction. and yeah, that would be basically, [00:32:00] and, I don’t know if you’re familiar with Bent Flyvbjerg , , the Oxford professor that, he calls it, cutting the tail.
You can cut the tail off. so you would basically, you would cut the tail and that would basically, you grow more than a certain amount. You would wind up, cutting, canceling the project, stopping it. So that’s one way to get that fat tail and get rid of that fat tail is to cut it.
the other thing is, upfront, pretty much everyone, cost estimators knew that, James Webb space telescope would be expensive, but NASA management assistant, it would be a billion dollars. No more risk management practices were in place upfront. It would have an if NASA administrators have been willing to listen to, the cost estimators, they would realize that this is a multi-billion dollar project, not a single $1 billion project.
So you know, that upfront planning, and having those practices in place and. having some discipline and establishing budgets would help with that as well. But yeah, there’s cutting the tail. That’s one thing. So once you started the project, you realized this is way more expensive than what we originally thought.
actually just canceling the project and starting over or doing something else is one way to handle that. And that would be the correlate that is the way to do a cut the tail.
Eric Lofgren: one of that is. And from Bent, I always get his name, kind of pronounce it wrong. I always say Bent Flyvbjerg how do you say it?
Christian Smart: I think it’s actually Flyvbjerg. I actually looked it up to see how he pronounced it.
Eric Lofgren: But one of the things that he also talks about is You want to break down a project as much as you can, to smaller projects where possible. And some things like maybe a massive suspension bridge is very hard to do that, but in most cases, You really want to do that and modularized.
So like when I think of James Webb, could they have de-risked that by decomposing it and doing some parts of it to experiment upfront? maybe not, but think overall I don’t see the value necessarily of always, these economies of scale, because you were also talking about it.
And that’s what I was wanting to talk about before on the better, cheaper, faster, you said on the satellite, there’s gotta be economies of scale. But what it looks like SDA is moving towards — the space development [00:34:00] agency — they’re going for a proliferated architecture. And that’s what the commercial industry is moving towards too.
And so that’s what worries me sometimes, about. The interaction between cost estimating and project technical decision-making because it’s like, if I need the cost estimate, then usually those are the things that I’ve done in the past. And so your cost estimate and your project will look like something that came in the past and that might not be optimal for one reason or another. If you could do it in a different way, or if you tried it out differently, you might see a different economies. Because just thinking about like a lot of, some of the cost estimating models, For space or just the bigger it is, the more expensive it’s going to be.
But ultimately I want to get as much sensor capability on something as small as possible. So shouldn’t the smaller, it is like the more expensive it would be. but there’s just like these kind of weird, like logical loops. I feel like when I do like a cost estimating relationship, for example.
Christian Smart: if you look at, those kinds of satellites. So if you look at, the legacy, this historical standard is these very large, expensive satellites up in, MEO or even geo.
So those are very expensive, billion dollar plus satellites. they’re actually able to, with recent advances they’re able to, put up very small, inexpensive satellites, like in the five million dollar range and you put up, 500 of those, it still winds up being quite a bit of, still quite a bit of money, but they, the one thing is, you don’t have to worry about, an entire satellite program, shooting down your one big satellite in LEO.
So are you going to have, a harder to kill the entire constellation when you have more of them. So there is a, there’s a trade-off in terms of the defense strategy as well. and, back by going to the really small, low level, though, you do it’s likely they do get the, do get pretty cheap and, you can launch a whole lot of them at one time.
So it is potentially a way to, to save some money, but it’s not only just in saving money, it’s also in terms of trying to achieve a reliable satellite network as well.
Eric Lofgren: Yeah, definitely. So I guess sticking to the similar [00:36:00] topic, you brought up, Norman Augustine, and I guess he also reviewed your book, which is pretty cool. one of Augustine’s famous laws was basically, that. like he was projecting from 1984, that if you just follow the trajectory of fighter aircraft, procurement costs, per unit — if you just extrapolate that line into the future by, 2034, 2054, it would be the size.
Like one aircraft would cost the size of the defense budget and then it would like surpass gross domestic product overall, a hundred years later. but there was actually an interesting article this year from ICEAA — international cost estimating and analysis association, from Brent Johnstone.
And he basically was like, Hey look, Augustine’s law broke down. Augustine would have thought that using that statistic, in 2020, today, a fighter aircraft would cost about a billion dollars. And it’s about a 10th of that, right? Like an F 35 about let’s just call it a hundred million.
What happened to Augustine’s law. And what do you think that says about, using historical cost data to project future costs?
Christian Smart: Yeah, that’s a great, that’s a great question. So yeah, so I guess that’s, he, Augustine’s, book, Augustine’s laws consist of 52, laws.
And, so this is when you’re talking about law 16, which is probably his arguably his most famous law. That’s often quoted. yeah. And so in the year 2054, he said that it would consume the entire budget. Then he also, interestingly, he said that the aircraft would be shared so that the air force and the Navy would have the aircraft three and a half days a week, a piece.
And then on leap here, the Marines would have it on leap day. So it was pre-staging I guess, joint strike fighter in a way. I don’t know. But, that didn’t turn out to be the case. Like you said, it’s only about a hundred million dollars and, in terms of a unit cost, and Johnstone looked at, what Augustine did.
Augustine has some really interesting analysis, back at that time, that was the kind of analytics he did as soon as his book was really ahead of his time. you think about the late seventies, early eighties when he was writing that no one else was really doing that. but, it’s, it wasn’t clear, in terms [00:38:00] of what Augustine was looking at it wasn’t clear what kind of unit costs he was looking at, is it a flyway cost? is it a, is it APUC, just procurement costs. is it, is it ruling out the development costs, so it wasn’t clear. He didn’t really, normalize for the units, get you to think he’s just like a total sort of total costs.
Now he got some kind of unit costs, but it sounds unclear how he got it. He was also using then your dollars. it was no accounting for inflation, and the, One of the things that happened in the sixties and seventies was high inflation rates and that since ameliorated quite a bit, so that’s a factor.
it’s, it was really extrapolating from an unreasonable trend. He was like looking at the early eighties and then extrapolating out to 2054 so way outside of, historical experience. So he was a really way out there. So it’s –you ought to be careful about how far you extrapolate.
It’s almost like it. More of a more, that little chart was more futurism than forecast.
Eric Lofgren: So you’re saying that, Using historical data, I would be able to better estimate something that’s coming up like right now. But if you’re trying to be like, all right, I’m going to use an F 4, an F 14 and an F 15 data to like project what the next generation air dominance is going to be, many years later and then I just have this huge index in between. that’s just not gonna cut it.
Christian Smart: So you can’t extrapolate that far. I’m just stumbling way too, out way too far on the future. So what, Brent Johnstone did find though, was that it, it did, it doesn’t increase on a yearly basis, tuned around 4.4% per year, that unit cost of an aircraft over time.
so there was a trend, it was much lower than what Augustine was projecting. some of his stuff has a little bit tongue in cheek. so he, No, but he does have a lot of wisdom. And a lot of the writings that I do, I did find in my research that I wound up, rediscovering a lot of things that he had said like average cost growth to development programs 50%.
So multiple projects tend to indicate that does indicate that there is a sort of a reliable mean potentially for or costs. We can do some point estimates, but it, is that to be a little careful, but, So there, so he does have a lot of interesting things like that about, the more you produce, [00:40:00] the less you get.
that’s the problem of too many projects at the same time, he’d loved things like that. another thing that, that some, something, someone had an underappreciated was, is that, the production rate of these systems is so low that we’re getting reaching in on an uneconomical point.
And defense where, basically not everyone’s gainfully employed, producing these products all the time. They’re standing around having to do some sort of training or do something else because the production rate is low.
Eric Lofgren: Yeah. I think I was looking back at like the F 16, the first, I think it’s first year of production, something like 1976, 77. they had 180 units and they were like consistently above a hundred, every year. And it’s just JSF, it took, I, it took them five, six, seven, eight years to get to something even close to that. Yeah. actually, when you look at just like the annual, like what’s the.
The cost per unit in each year in for the joint strike fighter, it looks like it’s coming down a lot, but that’s because a lot of the learning curve Is being spread out over that time. Whereas most of them learning was actually done in like the first year or two of F 16, because they were able to just pound out so much.
It was just started from a lower level and it didn’t look like it was going down as much, but the performance overall. Like the actual learning curve that they were experiencing were actually relatively similar. So there’s just like an optical effect between the difference in production rates and what that looks — makes the price look like over time.
Christian Smart: Yeah. So Augustine didn’t account for anything like that. He didn’t, but Brent Johnstone did correctly was, he normalized everything to a T 100. the hundredth unit. there’s, that’s not fair what, what Augustine was doing with, for that, but yeah, yeah, so it was because of Johnstone’s paper.
I thought, maybe I could get Augustine to review my book. Because I saw him several times. And so I asked him because he actually in the paper, he says, this graph reproduced the permission of the author. So I said, maybe he must have Augustine’s email address. So I reached out to him and he said, yeah, he still has a Lockheed Martin because.
Brett is a Lockheed employee. He said, yeah. Augustine still has a Lockheed email address and he answers it. [00:42:00] So I emailed him and sent him the book. and he, he endorsed the books. it’s an endorsement of sin on the back cover of the book. was happening. He’s 80, 85 years old.
He is, he’s still going strong. The other thing, was the Augustine said you alluded to was, that this, the schedules for these, developing these aircraft was pretty consistent, around an average around three years, but it took a lot longer to develop JSF.
Eric Lofgren: Yeah. That’s and, I just saw a paper out recently that was claiming that the time between milestone B an IOC has actually been pretty constant over time since the sixties. And I just don’t believe it. it’s I think there’s just too much going on in terms of what do we, what does a milestone B versus an IOC actually mean in those different times?
I would just want to stick with the Augustine law for one more second. Sure. and this is some of the difference I think, of accounting over time. and the differences and why it’s hard to use costs because it’s like, it’s easy to measure the costs. But then a lot of these times, like when we were looking at those. unit costs, JSF costs a hundred million.
But that’s like excluding the spreading of the RDT and E costs, the research and development mod costs, and what’s happening in the operations and support area. And then when you’re actually getting capability, and the , was it the capability you thought? So I guess like when I’m looking at Augustine’s Law and just like looking at this.
There’s growth in terms of cost per aircraft. It’s what’s going on underneath that? I think that we’re often we do a decent job tracking costs and schedule, but then it’s hard to relate that to performance.
And when we do, it’s just what’s the correlation between cost and performance along this trajectory. But. yeah, I don’t know if you had anything to say there in terms of just like the completeness of these analyses.
Christian Smart: Yeah. Yeah. Sometimes it’s hard to get your hands around it like a missile defense, cost per kill was, it was a pretty good way to relate cost and performance.
what is, which take into account both the cost of the missile and then the probability of the missile, shooting down another missile. you can, there are some ways to, to quantify it. people get in kind of murky area when you start using, trying to mix qualitative and quantitative and a lot of [00:44:00] performances qualitative.
So you have to be careful about that, but there are some hard numbers that you can get your hands around. Missiles, for example, cost per kill is one way to look at that.
Eric Lofgren: So the last thing I want to touch on with, Augustine, you highlighted the, the Las Vegas factor development program planning, which is like 50 — Development programs are likely to grow at 52%. And you advocated in your book for something like growth factors. And I think Bent Flyvbjerg, he also says the same thing, right? It’s like I have this big historical database of costs and on average projects that look like yours grow, let’s just say 30%.
So whatever the cost estimator comes out with, let’s put 30% on it. And now we call it like the risk adjusted and. that’s what we should go with the funding for, even if that’s not what we’re trying to target the cost for, but we should recognize that we probably need to put that amount of money aside, but then I’m also wondering what if I’m just like, I read everything that Christian smart does.
And I understand these pitfalls. And I do a correct integrated cost schedule analysis in my output is like the 70, if they’re whatever percentile cost estimate from that. And here’s my cost estimate. Why — there’s almost like some kind of, Willpower verse like, you know, what’s can I actually get away from that 52% cost growth?
is every estimator biased in the same way? Or like what happens when Christian smart produces a cost estimate? I should not use that cost. that 52% factor.
Christian Smart: Well you got to be careful. I mean, yeah, so that’s a good, that’s a good, a wrinkle. the key is. If you’re doing, is the estimate an independent estimate or is it, or are you working for the project? Because you’ve worked on the project and then your inputs are going to be biased by the optimism of the project manager.
So even you have the best model in the world, if your inputs are low and your cost is going to be low, so you have to be, be careful about that. So you gotta look and see, how much optimism is it and, take a look, we did, I worked on the constellation program years ago and, the cost risk analysis for areas one which you norman augustine and help cancel that program.
and [00:46:00] around 2009, 2010, and the, our initial estimates were very low. when you looked at an independent analysis that was done and, they were more realistic around the same time. So I, and their S-curve actually matched up to the final budget, whereas ours, not but we were influenced by the project management optimism.
They made a lot of assumptions that they weren’t realistic once we started getting into the details.
Eric Lofgren: Yeah. It’s hard for me sometimes to like discern between that. Because sometimes I feel like I want my project managers to be like, we’re going to do something new. We’re going to be optimistic, like at a startup, you can’t be recruiting people and having an entrepreneur, like not being an evangelist, thinking like I’m going to go change the world.
So they have to have that mindset. And then, I guess there’s also you’re in a government bureaucracy, You’re not necessarily taking the risks on your own self. And so there has to be like this oversight and in terms of, realism and making sure, the whole system actually works out.
So do you have any views about that? Like we almost want the program offices to be like that,
Christian Smart: you want to be success oriented, right? yeah, I was talking to, Doug Howarth, and I had a podcast with him. [Name?] Who worked at the skunkworks for a long time, and he’s one of the Guggenheim, award, he designed the propulsion system for the JSF and, one of his, one of the things he said was that, and Scott works is known for doing things quickly and relatively cheaply.
And he said their, motto was one miracle per program. if you look at a lot of the programs that a lot of cost growth require multiple miracles, JWST required multiple technological advances. So you want to challenge people, but you also, you want to set some realistic limits on what you’re going to try to do with the project.
if you want to be able to challenge yourself and succeed. project just try to take on too much, so there’s optimism, but it needs to be tempered a little bit with some men reality. So I’m not agree. you want optimism, you want stretch goals. You want people to work hard to do things quicker and cheaper, but, in many cases, like you said, these project managers don’t have skin in the game, they’re, they’re not the ones that are suffering as a result of cost growth and schedule the delays, this other [00:48:00] projects that want up paying that price.
sometimes, like you look at, military leaders, there’s a lot of these project managers are, military leaders, colonels, generals, they’re only around for two to three years before they, they, that’s the way the system works is they rotate in and out. So they’re not around to see the end of the program.
Eric Lofgren: Actually, when I think about like the skunkworks, for example, the F 17, that prototype, like all they did was like the airframe, They took like the landing system from a C130 [actually, the environmental controls from C-130] the fly by wire from, the F 16. And they just high TRL stuff for everything. That’s not what they’re making the miracle on, which is just the airframe. yeah, I feel like a lot more that kind of thinking needs to be done and I think that’s where some of the agile process I think, comes in and breaking down these things. So I’m not estimating out a 10 year long development program where I have all these miracles, right? Like I’m really just focused. I have high TRL stuff. I know the cost of those things because there’s been other projects and hopefully other cost estimates on those.
So there’s data on that. And then I’m really like constraining where my uncertainty is and hopefully that speeds everything up and actually get to higher technical end states faster than if I tried to do it all in one big bang.
Christian Smart: I guess skunkworks was agile before it was agile.
Eric Lofgren: I think the department of defense just had those things built into it. They just didn’t know how effective their management really was. And they were like throwing out the good for the hope of the perfect, Oh, we can perfect this portfolio in these programs. And maybe that just didn’t work out, but I wanted to, there was something actually interesting that you brought up in your book that I actually think gets to this distinction between how things were done in the past and how things are done now.
And that was the idea that cost growth equals the square root of the number of lead organizations involved in the project. So that’s if I have. two lead organizations that are cooperating on, on a single, like major [00:50:00] project, then I’ll get whatever the square root of two, that’s my factor growth in as I add more organizations, I can expect more and more cost growth out of it.
And I think that kind of aligns with how the department of defense shifted itself, right? Because back in the forties and the fifties, if I wanted to accomplish a major program, I had these different organizations, they all had their own budgets and they all had to like, Coordinate themselves. for example, an aircraft carrier, you had to do Bureau of ships, we build the ship Bureau of ordinance, we’ll build the ordinance parts, and then you have the Bureau of aeronautics and other organizations, potentially all involved in this one project. And that would, we would expect it to. Lead to cost growth. And then they moved over to a lead systems, like one program office, one lead systems integrator, every aspect of a project should be under the hands of one guy and fund him to go do that program.
And that’s where we are now. And it seems like this has actually led to. now the problem is stovepipes and inter-operability I’ve, now that I’ve created all these systems that are under one leader, maybe that did reduce cost growth. And I was able to get some efficiencies out of that, but then it seemed like the networking between them was kind of the expense.
So that was like the higher enterprise level that now I’m going to have to see this cost growth and you’re starting to see. the services tackle that. So how do you reconcile this need for kind of like project leadership to make sure you’re not getting too much cost growth and there’s like a single vision versus enabling the enterprise services and tools and the joint force networking.
Christian Smart: Yeah, that’s a great question. so one of the issue, one of the things that they have a military, you go into battle, you have to have a single chain of command and you want to call a unified chain of command. Otherwise people are going to get killed. So you’ve got to have one, one boss tells you what to do and you get to do what he says.
and so the services, some extent of that kind of adopted that approach over time, too. the, you want a unified chain of command or a single chain of command. You don’t want multiple people telling you what to do. So that’s the beauty of that is it does cut down on some of that —
some of that complexity of having multiple project managers, you look at, historically [00:52:00] one of the worst examples of cost growth was the Sydney opera house. there was no single project manager. It was a kind of a sort of prolonged time as collaboration between the architect and the construction team.
having a single chain of command does help, with that and, does simplify things, but like you said, it does, stovepipe things. One of the things that, the way that missile defense agency was organized, I think that Mike [?] Helped with this is, it was a hybrid structure.
So you think about there’s, in terms of management, there’s a matrix structure where, so for example, as the cost director or matrix structure, I would matrix, my employees out to programs. But then I would have no say over what they did. So they worked for the THAAD program office. Then the THAAD program director were told what to do, and I would have no influence.
The other option would be, they could be totally functional and they could make it work for me. They threw somewhere for the third project manager. He, but he told them what to try and tell them something to do. They didn’t have to do it, that it has to do what I told them to do or her one of my employees told them to do.
But, MDA was the balance of the two kind of a hybrid approach where you have a functional chain of command and a matrix chain of command. So it’s a criticism of something of attention. So there was always some tension between the functional chain of command and the project chain of command.
But you do get the insight across organizations when you do it that way. they would, so the, someone worked for the THAAD program office. and that then they would, whatever they did, they’d also had to report to their functional chain of command and to do it, the functional, they took an agreement between the functional chain of command and the, program chain of command.
what they did before they can go forward. And that does slow things down a little bit, but if you do get concurrence and you do get the insight across the, the functional organizations that need to have insight into specific areas. So that makes sense,
Eric Lofgren: complete sense. like like the foundations after world war two of defense organization was that.
See, like single chain of command hierarchy, where everyone has a single boss and that quickly Fell apart. Because they were claiming that the functionals didn’t have enough [00:54:00] say, and there wasn’t enough coordination. And you had these assistant secretaries of defense that actually started popping up in the fifties for each functional.
And they would actually, since her assistant secretaries of defense, they didn’t have any line power or over the functionals, but they had to go like up through the, secretary of defense or at the service level up through like the headquarters or the secretary for the service in order to get anything done.
And so eventually it just DDR and E you’re going to have line control over the research and development programs. And then you started to see some of the growth of that. But I think what you’re saying in MDA is there’s a true within that component kind of functional organization.
So you have leaders for like costs, potentially financial management, life cycle sustainment, each one of those functionals. They would like control their workforce, but they would matrix them out. So once they went to a project, they didn’t have control, but they would still have to report.
So they literally did have two bosses, The project manager. and then the functional lead. Interesting.
Christian Smart: And that the functionals review the work, and then also any major, like anything that would come up with in terms of costs so that they had program office said, we’re going to spend a hundred million dollars on this test.
And they had to come to me for concurrence. I had to sign off on it before they could budget it.
Eric Lofgren: So I guess some of that, that’s let’s just say like each one of these functionals has like a responsibility to the enterprise, right?
So the cost person has a responsibility to make sure that there’s enough budget available. So such that each of these programs can succeed. like potentially the systems engineering guy has a responsibility to make sure that each of the systems networks with the ground architecture. And Eventually I’m a program manager, I’m trying to execute this program, but each of my functional areas as a different boss and a different objective, and I’m trying to do something else, but it has to force fit in there. So it’s. going back against the idea of a unity of [00:56:00] command, but then it also is getting towards the idea of you have to negotiate between each of those stakeholder interests.
So you’re bringing a lot more stakeholder interest into it, in order to get that networking and interoperability, but then you’re sacrificing something on your program manager side, and that’s just the reality of it. You can’t have it both ways, right?
Christian Smart: That’s right. Yep. That’s right. So they can’t say, Oh, we’re going to spend $50 million through the test. If I say, no it’s really going to cost a hundred million dollars.
Eric Lofgren: yeah. So with the services, let’s just look at the cost functional because that’s kinda where we’re at right now. And maybe they’re pretty related with the financial management.
So business FM and business CE, but. you said in your book that, like an optimal profile for expenditures for each program. And the problem is maybe that goes up pretty high and I need to start bringing a lot of people. Optimally to get this program done.
But then when I have a portfolio of programs and now they have budget constraints and potentially I have to like chop off the top, you have to go for longer and you don’t ramp up as much. And that creates like a problem in terms of. optimizing the portfolio. So each project is actually taking longer and costing more than it should if I just knew that and I budgeted for it correctly.
So are the services, are, is that kind of just like a general proposition that you said, or do you think that like the services and the department of defense are actually trying to pack too many programs into the budget and they’re actually getting into this kind of funding constraint where they’re not able to execute the programs to the level that efficiency would dictate.
Christian Smart: Yes. I think that’s the case. I saw this a missile defense agency. I’ve got some scattering of, of, evidence of that anecdotal evidence and other services. That’s not captain quite a bit of missile defense agency, norman Augustine talks about it a little bit in his book.
I think it’s a fairly, really common phenomenon, but it, so for example, At a time when this defense agency, the budget was actually declining. and then projects were a little bit constrained. [00:58:00] the organization decided to try to embark on a new project. So it’s why aren’t you doing that?
Because we already, budgets being cut. no projects were going away, nothing was getting canceled. And then they’re trying to start a new major new initiative just to develop a new missile program. actually, that required an independent estimate from the CAPE. And then the CAPE recommended the program get canceled and they wound up canceling the program as a result.
So it happened a couple of times there were a couple of projects that got canceled like that, and that was wisely — they were wisely canceled because it really, it was adding too much to the budget trying to do too much with a limited amount of resources. And so it’s just, it’s not, it’s kinda Just because the organization was not looking at the whole portfolio and how much are things costing and looking at the whole sand chart of what is, where are all the projects fit within the colors of sand and where is the budget?
Are we had this new project to end in five years from now we’re going to be running out of money. Yeah. they don’t, organiz organizations often don’t look at that now. by the time I left MDA that, our group was looking at that for, for the entire organization and looking at the future years, planning out through around 10, 15 years to see how potential new programs would fit within the total portfolio.
In terms of, are we going to wind up, we started a project this year. We’re already near our budget ceiling. two or three years from now, we’re going to be having to delay other projects because the ramp up will require us to hit a funding constraint and then required to delay other things.
And the other thing is you also see you wound up in efficiency. You mentioned production. You want it with inefficiencies in production because organizations will say, I want us to spend more money on developing this new capability. So I’ll cut production. in out years, you can’t do that, in a year of execution because of the different colors of money.
But in future years planning, you can trade some of that in and out and you can cut the production quantities to pay for, research and development. so organizations would do that too. And then you cut the production, then you’re achieving less efficiencies in production and the cost of each individual item that you’re producing.
Ends [01:00:00] up costing more as a result. you’re also losing from that perspective as well.
Eric Lofgren: Yeah. I think that’s an Chuck Spinney. He had a great report on that. About just like looking at the future years defense program and being like, every program wants to do something like this. And then they’re all just getting extended out and reducing quantities.
And as he just said, like, when you do that, you get less efficiencies from, rate effect and from learning curves and all that and spreading overhead. And so that really jacks up — and I think that’s a lot of the problem with. With those trade-offs because once you have a program and it’s wanting to go that way and you say, there’s other, these other priorities.
We want to take money from you. And then they come back and say, you take out 10% of my budget. I’m going to have to get like 20 or 30% less product out of that because of those effects. And we had been planning to this thing and everyone was that was our shelling point. So now we got to bite the apple and that seems to lead to a lot of, like inter program, squabbling.
Christian Smart: Yeah. And, and you saw this too with, there were a couple of initiatives to try to do multi-year procurements on a couple of projects when I was there. And, They ended up getting squashed to some extent, because in one case, the contractor was very interested in that, but, that would be basically tie the hands of the government in terms of, okay, if we had to buy 30 or 40 or 50 of these missiles a year, we’re gonna have to commit to this production budget, for the next five years.
And that they won’t have, they wouldn’t have the, the wiggle room to take money from that to spending on something else. but that would have saved them a lot of money. that the multi-year, is essential to save quite a bit of money, but
Eric Lofgren: yeah, I always go back and forth on those prospects because.
you lose optionality when you do that. And then you’re essentially locking yourself into a suboptimal outcome in terms of what the program is. and what, and now that contributes to the force structure. If you just like lock in this plan and say, five years from now, we’re still going to be by the same baseline specification of this thing.
And potentially we rushed it through, testing. So we might have to go [01:02:00] back in, do some odds or fixes.
Christian Smart: Yeah, that’s true. And then in this particular missile, it was light to mature over. They had a lot of technical issues. it was a relatively late to mature. so that is a good point because when, once you do that, you’re basically locking in that design for the next five years.
You can’y make any changes
Eric Lofgren: I want to get, you should get your view on something real quick. Because I always think of It would be nice to have this separation of R and D from procurement. So like those decisions seem very different. One is very, uncertain, lots of risk. And the other one, I can more use these analytical techniques to say okay, I know what the costs are going to be and what the outcomes are and how they fit in.
So I can go into that. And I think, the more you break that apart, and you’re not in these kinds of like concurrent situations, then you can say, the MYP. Makes sense here because I know what I’m getting. but then there’s also this idea. I think that — like Elon Musk, for example, with Starlink he’s like actually going through concurrency, he says, you have to do concurrency so that you learn in production. you learn what designs are most producible, because production is much harder than development and you have to get get that right together.
And then there’s also the kind of point from DevSecOps, right? for software you’re never done, you’re always simultaneously in development and production how do you walk that line? How do you think about that from a cost estimating perspective or just PR project management perspective?
do we want to separate those things out? should they be like a bright line between R and D and production or operations? Or should they basically be like merged together?
Christian Smart: the problem is, when you see this over and over again, is it projects start trying to produce before they even have a design or design complete, it does want to pay increasing risks, GAO, consistently points out, issues with concurrency as a major source of risk.
And so it’s a problem that projects set upon themselves by trying to do things too quickly. it’s, I guess there’s a fine line between those two things. but I think for like big projects, big weapon systems, Things that are complex. You want to have the design straightened out before you start building it and you’re being in a vicious cycle.
I’ve seen this, what happened and missile defense agency for one project. I won’t name it. [01:04:00] it, the missile, having, technical issues, if they had just stopped taking the time to iron out the design upfront and stop the production, I think it would have ended up saving money in the long run.
it just, it depends somewhat on the context, but I think, For complex systems getting the design, before you start producing it saves money.
Eric Lofgren: Yeah. I usually have to agree, especially hardware intensive type systems, I think, one of the issues is that procurement money is just staring you in the face and you have to have that lined up from years ahead.
you know that you’re going to have gone through that testing phase. And like when procurement money staring you in the face, The last thing you want to do is be like, I got to push that off. So I guess some of it is just almost treating these programs by themselves and not like lining up, follow on stuff until you’re ready.
So you need to. For some reason, I guess there has to be like, I think the cost estimator is job becomes much easier when the funding becomes more flexible because I don’t have a lock-in of the next 20 years. I can actually say Oh, I’ve learned something in this stage of the prototype or in this dis-aggregated subsystem.
And now they’re ready to go here and I can cost that out relatively quickly, but will the funds be there? And like all those gaps that might occur might be the problem. So I guess the question is, do you believe funding flexibility and better cost estimating actually go hand in hand?
Christian Smart: I think so. I think the, more funding flexibility does help with estimating the costs. I think that’s, I think that’s the case. that with NASA, they have, they only have one color, more or less one color of money at NASA. and I think they would, that would probably help with DOD, not me not having those separate things broken out, or at least more cleanly broken out, having the procurement funding coming in and much later, but having more flexibility I think helps with cost estimating.
Eric Lofgren: as we’re wrapping up here, is there anything else that you’d like to leave our audience with?
Christian Smart: no, I’ll just, mention that, a book is out now, the bell art cover and ebook version available through Amazon Barnes and [01:06:00] noble. Amazon now has it for $39 is the one part of online book shopping. so it’s out there, you should check it out. you can also, you can find the, And the book online. So I think that was the main thing that I would like to be the leaders we’ve had a good discussion. I appreciate having you having me on
Eric Lofgren: And also don’t forget that Christian has a podcast and a blog too, so you can find out he writes and talks a lot about risk management of course, cost estimating actually a lot about the Corona virus as well in statistics from that. So I’ve been kept up to date with what’s going on with Corona virus from you and on LinkedIn.
So Christian smart. Thanks for joining me. And everybody makes sure you pick up solving for project risk management.
Christian Smart: Thanks Eric.
This concludes another episode of acquisition talk. If you have comments, interview recommendations, or just want to chat. Please contact us at acquisitiontalk.com. Thanks again. And until next time.
Leave a Reply