Measuring Engineering in the AI Era
Replacing Activity Metrics with Evidence of Flow, Quality, Trust, and Business Outcomes
AI makes software activity easier to generate and easier to mismeasure. The enterprise must stop treating code volume, commits, story points, prompt counts, and agent utilization as evidence of productivity. Engineering performance should be measured as the organization’s ability to convert intent into valuable, trustworthy, and sustainable outcomes.
Executive Summary¶
Software engineering has always been difficult to measure. Output is intangible, work is highly interdependent, and many of the most valuable contributions prevent problems rather than create visible artifacts. Artificial intelligence intensifies the challenge because developers and agents can now generate code, tests, documentation, and pull requests at a rate that makes traditional activity measures even less meaningful.
Executives will be tempted by readily available telemetry: lines generated, prompts submitted, agent tasks completed, commits, pull requests, story points, or hours saved. These measures can describe activity. They do not establish business value, system quality, engineering health, or trustworthy delivery. Used as targets, they invite gaming and can accelerate the wrong work.
EB-005 argues that engineering measurement in the AI era must begin with business and user outcomes, preserve the DORA software-delivery foundation, use a multidimensional productivity model, and add targeted AI-era measures where they support consequential decisions.
The brief introduces a measurement architecture spanning business outcomes, flow and delivery, quality and engineering health, trust and governance, human capability, and economics. It also identifies new measures for AI-assisted and agentic engineering, including verification cost, delegated-work effectiveness, context quality, evidence completeness, review capacity, authority integrity, and cost per accepted outcome.
The central discipline is not collecting more metrics. It is connecting each measure to an owner, definition, source, interpretation, privacy boundary, and decision. Metrics should help leaders determine where to invest, what to stop, which authority to expand, where risk is accumulating, and whether AI is improving the engineering system or merely increasing visible activity.
The executive position is direct: measure what the organization can responsibly deliver, defend, operate, and improve. Do not confuse accelerated artifact production with engineering performance.
Why Read This Brief?¶
EB-005 gives senior leaders a practical measurement model for evaluating engineering performance in AI-assisted and agentic environments without rewarding noise, activity, or superficial adoption.
After reading it, you should be able to:
- explain why AI further weakens the link between activity and value;
- identify measures that should not be used as executive productivity targets;
- begin measurement with business and user outcomes;
- preserve and interpret the DORA delivery-performance baseline;
- apply a multidimensional productivity model;
- add AI-era measures for verification, delegation, context, evidence, authority, and economics;
- measure evidence quality rather than document volume;
- use architecture and repository health as leading indicators;
- connect operational trust and human capability to performance;
- build a coherent measurement architecture and 120-day executive agenda.
Key Topics¶
Intended Audience¶
What the Brief Examines¶
- Why AI breaks the remaining link between activity and value.
- Which activity measures should not be used as executive productivity measures.
- Why measurement must begin with business and user outcomes.
- The continuing importance of the DORA delivery-performance foundation.
- Multidimensional productivity through SPACE and related models.
- AI-era measures for verification, delegation, context, evidence, authority, and economics.
- Evidence quality, architecture health, and repository health.
- Operational trust and the human system.
- Measurement architecture rather than metric accumulation.
- A 120-day executive agenda.
Relationship to ETIS¶
Related Publications¶
- WP-001 — Engineering Trustworthy Software in the AI Era
- WP-002 — Repository-Centered Engineering
- WP-003 — Engineering Evidence
- WP-006 — Engineering Governance
- WP-007 — Engineering Review and Readiness
- WP-008 — Operational Readiness
- WP-009 — Context Engineering
- WP-011 — Engineering Trust
- WP-012 — The ETIS Manifesto
- EB-001 — Why AI Changes Software Governance
- EB-002 — The Future Software Engineering Organization
- EB-003 — Preparing Engineering Organizations for Agentic Development
- EB-004 — Building an AI Engineering Platform
Citation
IEEE
W. T. O’Connell, “Measuring Engineering in the AI Era: Replacing Activity Metrics with Evidence of Flow, Quality, Trust, and Business Outcomes,” ETIS Executive Brief Series, EB-005, ver. 1.0, July 2026.APA 7th Edition
O’Connell, W. T. (2026). Measuring engineering in the AI era: Replacing activity metrics with evidence of flow, quality, trust, and business outcomes (EB-005, Version 1.0). Engineering Trustworthy Intelligent Systems.Chicago
O’Connell, William T. “Measuring Engineering in the AI Era: Replacing Activity Metrics with Evidence of Flow, Quality, Trust, and Business Outcomes.” ETIS Executive Brief Series, EB-005, version 1.0. July 2026.BibTeX
@techreport{oconnell2026measuringengineering,
author = {William T. O'Connell},
title = {Measuring Engineering in the AI Era: Replacing Activity Metrics with Evidence of Flow, Quality, Trust, and Business Outcomes},
institution = {Engineering Trustworthy Intelligent Systems},
type = {ETIS Executive Brief},
number = {EB-005},
year = {2026},
month = {July},
note = {Version 1.0},
url = {https://etisframework.org/publications/executive-briefs/eb-005/}
}Version History
| Version | Date | Status | Notes |
|---|---|---|---|
| 1.0 | July 2026 | Current | Initial publication. |