Traditional acquisition metrics can be a major problem for adaptable systems. The Earned Value Management System (EVMS) is a common tool for measuring progress in acquisition programs. It is designed around breaking down a program’s master schedule throughout its entire development into discrete work packages, which register as earned value when they are completed at, or below, expected costs. As traditionally implemented, however, EVMS requires an almost entirely static program baseline to function. When the content of work packages is subject to continuous change, the ability of EVMS to meaningfully track progress decays rapidly.
That was from the CSIS report, “Acquisition of SoftwareDefined Hardware-Based Adaptable Systems“. I couldn’t agree more. Note how they wrote “As traditionally implemented.” See here for an overview of EVMS.
If you look at the history, there are two conflicting rationales for EVMS (1) reduce uncertainty about large/complex developments, and give early indication of cost/schedule growth so that budgets — which have a long planning cycle — can be adjusted in advance, (2) it was created to show everyone they were top managers using fancy whiz-bang charts and keep the Navy/Congress off the backs of the Fleet Ballistic Missile program.
Here is some good discussion on rethinking EVMS from the excellent Bryon Kroger:
It occurs because we don’t measure what matters (customer outcomes). Measure milestones completed? You’ll get completed milestones. Measure money spent (EVM)? You will get money spent. But that’s not the worst part. Most of this is *manually self-reported* with no accountability. What we measure, then, is the ability to lie about what the Pentagon cares about…
Where I think EVM still has potential, though perhaps reformulated, is in measuring risk defined as “resources spent without validation”. In software for instance, if value = performance / cost / schedule and we care about delivery as a proxy for validation, value = (deployment frequency, MTTR, chg fail%, availability) / cost / lead time. Agile/DevOps, done correctly, actually decrease risk according to this formula.
See Bryon’s YouTube video for more on that, particularly starting at 28:30. I wrote a piece recently on why EVMS doesn’t work well with Agile. The Defense Innovation Board recommends removing EVMS for software projects. Can the project management technique, invented for an industrial era, be saved?
Leave a Reply