How to measure AI ROI: moving from technical metrics to business outcomes with business observability

Measure AI ROI through the business process: baseline, cohorts, the real cost of tokens and conversion into euros with business observability in practice.
Mesurer le ROI de l'IA : passer des métriques techniques aux résultats métier avec la business observability

Measuring AI ROI has become the awkward question in many boardrooms: budgets have been committed, pilots have multiplied, but few teams can show a quantified effect on the profit and loss account. The problem is often not the technology itself, but what is being measured. To measure AI ROI credibly, you have to connect technical signals to business indicators.

AI project dashboards show model accuracy, latency, token consumption and error rates. These indicators tell you whether the system works well, not whether the business is better off. Between the two, a link is missing: the business process into which the AI fits.

This article proposes a method for moving from technical metrics to business outcomes, based on business observability, that is, the ability to follow a business process transaction by transaction, with the same rigour applied to IT services.

Why technical metrics are not enough to measure AI ROI

Several recent studies agree: MIT’s report on the “GenAI Divide” estimates that only a small minority of integrated pilots produce measurable value, and McKinsey’s annual AI survey indicates that while a large majority of organisations use AI, only a minority see an effect on their operating profit. Gains exist at the use-case level; they are much harder to demonstrate at the scale of the company.

Take an example. A retail company deploys an agent that reads supplier invoices, extracts the amounts and proposes an accounting allocation. The model reaches 97% correct extractions in under two seconds. Good news? Impossible to say. If the invoices then wait three days in a controller’s approval queue, the payment time has not moved. If the agent gets the largest 3% of invoices wrong, the cost of corrections can exceed the gain.

The model dashboard is identical in both scenarios. Only the process view makes the difference.

The business process as the unit of measurement

Almost all AI projects are grafted onto an existing process: invoice processing, account opening, ticket resolution, order preparation. AI automates a step, assists a person or guides a decision, but the value shows up across the whole process. The right question is therefore not “is the model good?” but “has the process become faster, cheaper, more reliable or more pleasant for the customer?”

For the invoice agent, relevant indicators would be, for example:

  • the time between receipt of the invoice and payment;
  • the processing cost per invoice, including AI;
  • the rate of invoices requiring manual correction;
  • late payment penalties avoided and early payment discounts obtained.

Do not forget the real cost of AI

An honest ROI calculation also includes the denominator. The cost of an AI solution is not limited to the licence or the model subscription:

  • token consumption or GPU time, which varies with volume and document length;
  • the surrounding infrastructure: vector database, storage, network;
  • human time spent on supervision and correction;
  • evaluation and observability tooling.

Token consumption is easier to track when it is traced per request and linked to the relevant service, as explained in our guide to AI observability.

In practice: four steps to measure AI ROI continuously

Step 1 — Instrument the whole process, not just the AI

Emit a business event at each step of the process, with a common identifier that lets you follow each invoice end to end. In Dynatrace, this role is played by business events, which can then be assembled into a process view. An event might look like this:

{
  "event.type": "facture.etape",
  "facture.id": "F-2026-018342",
  "etape": "imputation_proposee",
  "traitement": "agent_ia",
  "montant_eur": 12480.00,
  "fournisseur.categorie": "transport",
  "duree_etape_s": 1.8
}

Step 2 — Establish a baseline before go-live

Measure the process for several weeks before the AI arrives: median time, cost per invoice, correction rate. Without this baseline, any claimed improvement remains an assertion. If the AI is already in place, keep a sample processed through the old route.

Step 3 — Attribute the effect to the AI

Seasonal volume, a new hire in the accounting team or a bug fix upstream can also move the figures. Compare cohorts: invoices processed by the agent versus invoices processed manually over the same period. A DQL query like this one allows a first comparison (field names to adapt):

fetch bizevents
| filter event.type == "facture.payee"
| summarize factures = count(),
            delai_median_h = median(delai_total_h),
            corrections = countIf(correction_manuelle == true),
            by: {traitement}

Step 4 — Convert the verified difference into euros

Data-entry hours saved multiplied by the hourly cost, penalties avoided, early payment discounts, minus the full cost of the AI. Present the result with its margin of uncertainty rather than a single figure.

These four steps are not a quarterly exercise. Once the process is instrumented, measurement runs continuously and also reveals silent regressions: if the correction rate rises again after a model change, you see it the same day, not at the next review.

Linking AI degradations to business indicators

The benefit of tracking the process in the same place as IT is being able to work back up the chain when a business indicator degrades: an infrastructure problem, model drift, a change in input data or simply a shift in activity. The journey experienced by the user also brings valuable insight, for example with Session Replay on Grail when the AI is exposed in an interface. On the operations side, SRE teams benefit from building these business indicators into their objectives, as described in our article on AI and reliability.

Best practices

  • Choose three to five business indicators per AI project, agreed with the process owner.
  • Measure before deploying: a baseline is worth more than a sophisticated dashboard.
  • Include human supervision time in the cost of AI.
  • Compare cohorts rather than periods, to isolate the effect of AI.
  • Make measurement continuous and shared between technical and business teams.

Platform teams can make this easier by offering business event emission in their project templates, a topic covered in the challenges of platform engineering in 2026.

Frequently asked questions

How do you calculate the ROI of an AI project?

By measuring the improvement in a business process against a baseline, isolating the share due to AI, converting it into financial value and subtracting the full cost of the solution.

What is the difference between business observability and classic BI?

BI analyses aggregates after the fact. Business observability follows transactions in real time, step by step, alongside technical signals, which makes it possible to link a business degradation to its cause immediately.

Can you measure the ROI of an internal productivity assistant?

It is harder, as there is no single process. You can track indirect indicators: ticket resolution time, code review time, internal user satisfaction.

Conclusion

AI ROI is not calculated once and for all in a spreadsheet: it is observed continuously, at the level of the business process, with the real cost set against it. By shifting the question from “does the model work?” to “has the process improved?”, teams can finally give management a verifiable answer.

Further reading: From AI experiments to business outcomes: How to measure what matters (Dynatrace blog).

span.observer is an independent community blog, not affiliated with Dynatrace LLC or the projects mentioned. Trademarks belong to their respective owners. Tools change quickly: check commands and options in their official documentation.

Total
0
Shares
1 comment
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
IA et fiabilité SRE : ce qui change au quotidien, de la supervision des modèles à l'autonomie par paliers

AI SRE: what changes for reliability day to day, from model monitoring to staged autonomy

Next Post
Platform engineering en 2026 : cinq défis et le rôle de l'observabilité pour les relever

Platform engineering 2026: five challenges and how observability helps meet them

Related Posts