What to measure instead of AI usage

One of the easiest ways to make an AI programme look healthy is to show rising usage: more prompts, more active users, more workflow runs, more model calls. Those numbers are visible, easy to collect, and often reassuring in steering meetings. They are also incomplete.

High activity can mean genuine adoption. It can also mean confusion, duplicated work, weak process design, or expensive use of AI where deterministic automation would have done the job better. If leadership treats activity as proof of value, teams quickly learn to optimize for movement rather than results.

Why usage is a weak proxy for value

The OECD's 2025 report on AI adoption in firms makes the business case for measuring outcomes clearly: wider AI adoption can contribute to higher labour productivity, lower defect rates, and reduced material inputs. None of those outcomes is the same thing as volume of model usage. They are business effects, not system exhaust.

The World Economic Forum's 2025 report AI in Action adds a second warning sign. It notes that 74% of companies report challenges in adopting AI at scale. That matters because scale problems are rarely solved by pushing more usage into the system. They are usually solved by better operating design, better controls, clearer ownership, and more disciplined measurement.

NIST's AI Risk Management Framework points in the same direction from a governance angle. It treats risk management as a continuous activity across the AI system lifecycle and emphasizes measurement, documentation, monitoring, and accountability. In other words, serious AI oversight is not just about whether people are using the tool. It is about whether the system is producing acceptable outcomes under real operating conditions.

What leaders commonly misunderstand

The mistake is not collecting usage data. The mistake is promoting it to a headline success metric. Usage can tell you whether something is being touched. It does not tell you whether work is better, safer, cheaper, faster, or more auditable.

In fact, some of the most useful AI programmes reduce visible activity. A better routing model may lower manual triage. A stronger retrieval layer may reduce repeated prompting. A cleaner approval workflow may cut unnecessary escalations. If leadership only rewards volume, teams can miss improvements that actually matter.

A more useful scorecard: six measures that travel well across functions

1. Outcome improvement

Start with the operational result the workflow exists to change. That may be time to resolution, first-pass quality, conversion quality, exception clearance time, policy turnaround, or case completion time. If AI is not improving a real business outcome, rising usage is not a meaningful victory.

2. Human rework rate

Measure how often a human has to fix, repeat, override, or complete AI-generated work. Rework reveals whether the system is genuinely removing effort or simply moving it downstream where it is harder to see.

3. Exception and escalation volume

Healthy AI operations do not eliminate exceptions; they make them visible and manageable. Track how many cases need fallback logic, manual review, or supervisor approval. A system that appears fast only because exceptions are hidden is usually creating future cost.

4. Control adherence

For sensitive workflows, ask whether required reviews, approvals, logging, and evidence capture actually occurred. This matters beyond internal discipline. Under the EU AI Act, for example, deployers of high-risk AI systems are required to assign human oversight, monitor operation, and keep logs under their control for at least six months. Even when a use case is not legally classified as high-risk, the operating principle is useful: value claims should be accompanied by proof that the system remained governable.

5. Unit economics

Track cost per resolved case, per approved document, per completed workflow, or per qualified lead, not just total model spend. This prevents a programme from looking successful simply because it is busy.

6. Eligible-work coverage

Measure the share of suitable work that the AI-enabled process can handle reliably. This is different from raw usage. It shows whether the system is dependable enough to be trusted in the portion of work it was designed to support.

Why this matters now

As AI budgets move from experimentation to operating expense, weak measurement becomes a capital-allocation problem. Leaders do not just need to know whether a team likes a tool. They need to know whether the operating model is improving and whether the risks remain acceptable as usage expands.

This is also where many programmes begin to drift. A pilot may look promising on a narrow task. Six months later, the organization has more users, more subscriptions, and more model calls, but no consistent evidence on error reduction, throughput, exception handling, or oversight burden. At that point, the organization is funding momentum rather than performance.

How to put the scorecard into practice

Keep the first version small. Choose one workflow, name one outcome metric, add two operating metrics, and add one control metric. For example: turnaround time, rework rate, escalation rate, and evidence completeness. Review them at the same cadence as cost.

If a workflow is important enough to scale, it is important enough to measure from three angles at once: business result, operational stability, and control integrity. That framing usually improves investment decisions faster than any dashboard of token counts.

Honest limits

Not every AI use case deserves a detailed scorecard on day one. Internal knowledge assistance, drafting aids, and low-risk productivity tools may begin with lighter oversight. But even there, organizations should decide in advance which outcomes would justify broader rollout and which signals would show the tool is adding noise instead of value.

It is also possible to over-measure. A useful scorecard should clarify judgment, not bury teams in reporting. The goal is not to create a compliance theatre around every experiment. The goal is to ensure that scaling decisions rest on evidence that operations leaders and risk owners can both accept.

Three questions for leadership teams

• Which AI metrics on our dashboard reflect business outcomes rather than activity alone?
• Where are we still mistaking human cleanup for successful automation?
• Can we show, with evidence, that the workflows we want to scale remain reviewable and controllable?

A practical first move

Pick the most important workflow currently described as an AI success and remove every vanity number from the review for one month. Replace them with one outcome metric, one exception metric, one rework metric, and one control metric. That exercise usually changes the quality of the conversation immediately.

Where NOVA's perspective fits

NOVA's view is that organizations make better AI decisions when they measure systems as operating components, not as novelty layers. That means asking whether a workflow is producing better outcomes, whether its failure modes are visible, and whether the organization can defend the process when scrutiny arrives.

If you want a broader governance foundation, see What is governed AI?. If you are moving from pilots into managed operations, From scattered experiments to disciplined operations provides a useful maturity lens. And if the workflow may face formal scrutiny, Audit readiness, step by step explains how evidence has to hold up once someone asks to see it.

The central point is simple: activity can be encouraging, but it is not the same as progress. In AI, the measures that deserve executive attention are the ones that still matter after the demo ends.