I was pleased to have Colin Carroll join me on the Acquisition Talk podcast to discuss the acquisition of machine learning in the Department of Defense. He is the Director of Government Relations at Applied Intuition, a company that enables autonomous vehicles through simulation development and validation. Before that, Colin had a number of positions including Chief Operating Officer at the JAIC, mission integration lead for Project Maven, and a decade of active service in the Marine Corps.
- 2:30 -Project Maven started with Bob Work and 10 slides
- 10:30 – There’s not yet an urgency like in 2009 with MRAP
- 15:00 – DoD autonomy programs often have zero data
- 19:00 – The Joint Common Foundations is no more
- 24:40 – Most DoD’s data is owned by industry
- 27:00 – DoD is buying brittle AI/ML models
- 31:00 – Separating HW acquisition from SW
- 37:00 – DoD’s $2B AI/ML spending estimate likely high
- 42:00 – We don’t win by reforming SBIR
- 59:20 – The buzzword of JADC2
- 1:05:16 – The idea behind Title 10 failed
- 1:09:50 – Force Design 2030 and the future fight
- 1:20:10 – How to build a federal team at a tech company
Download the full-text transcripts
How Project Maven differed from the JAIC
Project Maven is the DoD’s effort to collect all the ISR data coming in from drones and satellites, label it, and create AI/ML models to help identify objects and potential targets. Colin described how Colonel Cukor developed this idea, drafted a 10 slide powerpoint, and got approval from Bob Work to move forward. The project quickly became the largest such effort in DoD at more than $200 million a year. More than one hundred million images are labeled for the algorithm.
Colin describes how that experience differed from the objectives given to the JAIC, which recently got wrapped up into the Chief AI and Data Office (CDAO).
Maven — the algorithmic warfare cross-functional team — it was literally designed to accelerate purely AI capability development for geospatial intelligence to war fighters. It didn’t inherit a lot of the, I’ll call it baggage, but a lot of the other tasks that the JAIC inherited, which was how to train and educate a workforce, AI policy to include international policy, acquisitions policy. Maven was able to stay very lean. It was mostly reservists with some details and then some contractors in the office.
… The JAIC It took its money and hired 300 people. I think when I left the JAIC, there were 300 people. I couldn’t tell you what 200 of those people did. I didn’t know them. They all worked under me somewhere, but I didn’t know what they did. Now that’s, some of that’s probably the nature of onboarding 150 people during Covid. A lot of it’s nature of what Congress and the department tasked the JAIC to do.
This may have contributed to an alignment problem at the JAIC, where service representatives detailed to the JAIC brought all their hobby projects. The JAIC had 35 projects, and Colin said he worked hard to whittle that number down. “I would’ve loved to hack it down to five,” he added, “but people got upset with me getting it down to 13. When you take someone’s money away, they tend not to be happy about it.”
Strategy vs. the Funnel
Colin doesn’t agree with the hundred flowers bloom methodology. “I’m a very top-down strategy person,” he said. While he believes in building and empowering teams to achieve the strategy, he thinks the grassroots method of sourcing problems through the classic “funnel” has flaws:
Basically the assumption of the funnel was the JAIC doesn’t know what to do because we don’t understand how the joint force fights and when it needs to win. So we’ll just hang a shingle out that says “the JAIC is open for business” and then the services will provide us a thousand requirements and then we’ll go through some weird process and down select the 10 that we need to do. The funnel.
Or we can say, Hey, we’re the joint force. We do a little bit of operational design about what we’re supposed to do, where the gaps are, what the services aren’t filling, and we invest our money to fill those gaps when it comes to either enterprise capability or development. I’m much more of the latter.
Whereas a young officer is trained to think about what they’re going to do when they command a division, the same kind of incentive structure does not exist around acquisition programs. They are viewed as rotational tours. And it doesn’t help your career to be aggressive in acquisition. “Where there have been PMs that are super aggressive, guys like Colonel Cukor, they wind up not getting a star. And so that people look at that and go okay, I don’t really wanna be like that.” That must change.
Data is Critical in Defense Programs
Colin stresses the importance of investing and managing data as a fundamental prerequisite to autonomous capabilities:
Program offices have a requirement to deliver some kind of unmanned capability — autonomous, semi-autonomous — this decade to a war fighter where they have zero data. They’ve not collected any data from any of the current, enduring, or legacy systems that are out driving, flying, or sailing. No one spent money to do that.
Colin said he recommends sending defense officials to Tesla to observe how commercial industry sets up an autonomy development platform. Data is critical. “Most of these programs spent years strapping sensors on Ford fusions and driving around collecting data before they even started development, or at least in parallel with the development.” He relates that strategy to how Project Maven pursued its objectives:
Maven is $257 million a year in RDT&E. A lot of that went to industry. Probably a hundred million of that was on the DevSecOps platform side. Data acquisition, data curation, data labeling. Maven had a team of 400 data labelers annually, eight hour shifts, just labeling data from all different platforms, all different sensors, unclassified and classified.
There’s no other program in the Department that’s even remotely looking like that. It’s because they either don’t know, or they’ve got a limited budget and they’re on the hook to deliver 12 prototype tanks or 12 prototype airplanes. They’re not gonna get what they want by just pumping money out the industry. They need to actually design their program properly.
Colin tells a story of a service that awarded a contract to Applied Intuition and wants them to deploy to a network still under development and could take a year, whereas there are existing solutions in DoD that could enable the company get to underway within the month. Service politics can sometimes impede the adoption of enterprise solutions.
This ultimately hampers DoD’s progress with autonomy. Colin points to the Air Force Skyborg effort:
Most of the DOD’s data is out in industry and it’s owned by, whoever’s the current kind of incumbents are on some of these contracts. I think they’ve narrowed it down to one or two — Skyborg is dead, but there’s a follow-on of Skyborg out of AFRL. On the whole, when those companies flew their prototype vehicle, one, they’re collecting very little limited data. That thing might fly once a month for a couple days. And then two, any data that was collected is at each disparate vendor.
Primes of the Future
The Army has separated its autonomy efforts into a hardware pathway and a software pathway, whereas in the past these are usually lumped into one and put directly under a single lead systems integrator. Colin is interested to see how this works out:
Now you have to merge those things together. Most of these hardware platform providers are not super accustomed to, or probably willing to integrate, someone else’s software into their vehicle. And the government then becomes the systems integrator. But the people that have designed these programs will all have moved on two years from now.
Colin discusses a spectrum of potential models. One one end, hand the autonomy program over to a traditional prime contractor. On the other end, a software company primes the next-generation system. “I don’t think either of those are the right answer,” he concludes. It will be somewhere in the middle. But he thinks it would be interesting for some of the new entrant SHARPE companies to prime major hardware.
Delivering for 2027
But for what is needed by 2027 to deter China from a Taiwan invasion will require the primes. “We are gonna fight with the things that we have today,” Colin says, “and that is a hard thing for people to understand.” He points to the SBIR reauthorization as a major side-show.
The other thing I think I’m gonna say that’s probably controversial coming from a small business who’s nontraditional is, at the end of the day, we win in conflict not by companies like mine and not by SBIR, and not by reforming SBIR, which is, I don’t know, half a percent of the duties budget. We win by the big primes. Being able to produce hardware with a software mindset and iterate on the software within that hardware… If you look at the POM and how the department budgets, builds capability, and then transitions to sustainment, SBIR is designed to fail every single time.
What is Needed in the Pacific
Colin thinks that the Marine Corps’ Force Design 2030, intended to realign the service around distributed operations, counter-UAS, long-range fires, electronic warfare, and so forth. While Colin personally thinks the intent is directionally correct, the program tradeoffs create a dip in capability “at the worst possible time this decade.” At the same time, many programs like the Amphibious Combat Vehicle look very traditional. How would logistics work in a denied environment? Here’s Colin:
Nobody said, Hey, I need to be able to have a Marine go to Advanced Auto Parts Philippines and get parts for that thing, right? So I don’t know, it just makes me, it makes me a little nervous about the words Marine Corps saying with the capability that they’re buying and how those things add up.
Here’s a slice from what I think is an important story for how the Marines might be fighting in a future conflict in the Pacific.
Your to weapon when you get there is not a M4. It’s not a Mark 19 grenade launcher on your MRAP. It’s a GSA credit card and you’re gonna go rent a truck and you’re gonna drive around that thing. That’s how it’s going to happen. Why? Because a DF-21 from China isn’t gonna be stopped by an MRAP anyway, and you’re not shooting your M4 at a bunch of bad guys shooting at AK 47 back at you. You’re guarding airfields and rapidly moving around the island trying to get logistics in trying to survive, manage your signature.
Building a Federal Team
Colin emphasizes that if a commercial company or startup wants to do business with government, and not be hanging around the edges, it will have to invest in a government team. Here’s what that consists of:
The government team consists of some BD [business development] people. It consists of cyber security, so for, DoD risk management framework and all the information insurance requirements. It consists of a legislative policy teams, or government relations. It might consist of a marketing team. And then all your customer facing engineers.
Thanks Colin Carroll!
I’d like to thank Colin Carroll for joining me on the Acquisition Talk podcast and John Mark Wilson for making it all happen. Colin is an industry commissioner on the Atlantic Council Commission of Defense Innovation Adoption, which I am supporting as well. You can watch him on a panel The state of defense & commercial autonomy.
Leave a Reply