DevOps Metrics

Overview

There is an old adage that goes “you can’t manage what you can’t measure”. Despite this, it’s surprising how many organizations are unable to accurately measure the cost, reliability, and effectiveness of their development to operations lifecycle.

Often you can find isolated pockets of metrics in various development (code quality, test coverage, etc.) or systems administration (server performance, monitoring events, etc.) groups. But what’s missing is an end-to-end picture of how the entire development to operations lifecycle is performing.

Simple questions that are critical to management are unable to be answered with certainty. Such as:

  • Cost of moving a unit of change through your application lifecycle?
  • Percentage of time spent on unplanned change or service restoration activity?
  • Success/repeatability rates for each key change process?
  • Percentage of changes that we not documented?
  • Number of releases from Development that impacted Operations procedures or automation?
  • Average cycle time for moving change from Development to Production?
  • Impact of lifecycle bottlenecks on Development and QA costs?
  • Amount of Management time spent on lifecycle problems?
  • It’s important to determine what the key performance indicators (KPI) are for your specific organization. The goal is not to collect every piece of data but to focus on 5 – 10 core KPIs that are the strongest indicators of current performance. As you work through various improvement projects these are the indicators that will show the first signs of improvement. The return on investment for these types of projects is substantial and ongoing.

The Process

The process is a fairly straightforward one, but it requires discipline and an independent eye if it is going to be effective. After the initial upfront effort to build the KPI model, the effort required to continually measure and report on the KPIs is minimal.

  1. Inventory existing metrics being collected across your organization
  2. Develop organization-specific KPI model
  3. Baseline the KPIs
  4. Implement procedures for ongoing measurement/collection
  5. Create organization-specific KPI scorecard
  6. Cycle through improvement projects and measure KPIs at regular intervals (generally 2-week cycles)
  7. Adjust KPI model on a quarterly basis