Can you pass the “Ships Test?” Issues of defense requirements

Here’s an interesting slice of Lockheed Martin’s Inside Skunk Works podcast on something called the “ships” test:

The bottom line is don’t add rules that are not there. Let’s go through the ships test, and you can grade yourself. Name different kinds of ships.

 

If you thought of aircraft carrier, ocean liner, row boat, destroyer, and canoe, congratulations, you’re a really well educated adult. Give yourself an “F.” If you thought of a submarine, give yourself a “D.” I did not ask you to name things that float on water, I asked you to name different kinds of ships. If you thought of an airship, give yourself a “C,” it has nothing to do with water even. If you said “spaceship,” give yourself a “B,” it’s not even buoyant or floating. If you thought of friendship, ownership, or relationship, give yourself an “A.”

 

All too often you hear the question, “name a ship,” and it’s something that floats on water. You added rules that I did not impose. You added them artificially. By doing so, you constrained the design space. For breakthroughs or new ideas to happen, you cannot allow artificial rules to be there.

 

I failed the ships test. When I saw it on television I was the classic adult. I have to remind myself of that example and be very conscious of that, then I have to find ways to test my teams and challenge them in a way that allows them the ability to think freely. The biggest trick there is not to artificially add rules.

This gets to the idea of outcomes-focused requirements. Too often in DoD, a military requirement gets overly specified. Section 5 and 6 of the Capabilities Development Document, for example, outlines product features including key performance parameters and key system attributes with a list of threshold and objective thresholds.

Ostensibly, the CDD could be written to enable the best of outcome rather than the best platform, but often it presupposes the technical solution. Otherwise, it would be labeled “high risk” and probably could not get all the approval signatures. How else could you estimate the lifecycle cost to get budget if you didn’t have a good idea of what the hardware or software will look like?

The preconceived solutions then make their way into Sections L (instructions, including technical) and M (evaluation factors) of the contract solicitation (request for proposal). Contractors must then respond not to the capability need, but to what the government anticipated the solution would be.

I talk a little about how this can be improved by separating strategic from tactical requirements and creating an iterative process with stakeholders in the Acquisition Next playbook.