The problem of metric selection for operational testing

I would be perfectly willing operationally to test anything anybody can name without having measures of effectiveness or criteria for the test. I am also perfectly willing to use them, but they can easily produce the wrong answer if you over-concentrate on them.

 

Let’s look at a few examples. If we are talking about testing an eight-inch gun aboard ship, the kind of criteria that I will find, if I go to the specifications documents, are that it must fire 10 rounds a minute, or that it must fire 30,000 yards using full charges, or 22,000 yards using reduced charges. I don’t really care whether it meets these requirements or not. Eight rounds a minute or 12 rounds a minute isn’t going to make that big a difference. 30,000, 25,000, 35,000 – – these aren’t the important things to the operational tester. What he will use are the basic criteria by which gunnery has been measured over the years: Hits per gun per minute, early hits, who’s afloat after the engagement is over, that kind of thing.

 

In R&M (Reliability and Maintainability) the same thing applies. I simply can’t use technical measures. If you asked me to test an air-to-ground missile and tell me its MTBF (Mean Time Between Failures) should be 30 hours, I don’t even know where to start.

 

Are you talking about hours that the missile is under power on the bench, or hours that it is aloft, or hours that it is aloft under power, or hours that it’s on the aircraft under power on the flight line during pre-flight and post-flight testing, or what. How do these various kinds of times relate to each other? I don’t think anybody knows.

That was Rear Admiral Robert R. Monroe, US Navy, Commander Operational Test and Evaluation Force, at the National Security Industrial Association Conference in Washington, D.C. 23 September 1976.

When we select a narrow set of metrics in advance for operational testing, program participants may neglect the unmeasured but equally important characteristics. Rather than design a test plan, the test should allow users to try it out and use judgment to determine what tests should be run. If testers have flexibility, then designers will focus on making the best product rather than optimizing to a list of targets.