After the pilot: how to track whether an AI initiative delivers the value it promised

In brief

Track AI value after the pilot as UK government tracks project benefits: a pre-launch baseline, an owner in operations, paired measures and variance reviews.

Steve Elliott

Steve Elliott · · 6 min read


On this page
  1. Why do AI initiatives struggle to show value after the pilot?
  2. What does tracking inherit from the business case?
  3. Which measures should you track?
  4. An illustrative benefit profile: a customer service assistant
  5. How often should you review, and who acts on a variance?
  6. How long should you keep tracking after go-live?
  7. When should you scale or stop an AI initiative?
  8. Sources

Track an AI initiative after the pilot the way UK government guidance tracks benefits from its projects and programmes. That means a baseline measured before launch, a named owner in the operation that has to change, a forecast trajectory to target, and reviews at set points that can stop the work. In our view, adoption figures tell you whether people use the system, not whether any money moved.

This post is about what happens after approval. It borrows from the UK government's benefits management guidance, refreshed in July 2026, and the FinOps Foundation's guidance on AI value. It sets out measures by use case, an illustrative benefit profile for a customer service assistant, how to read a variance and how long to keep tracking.

Why do AI initiatives struggle to show value after the pilot?

In our view, few organizations measure it well. McKinsey's State of AI 2026, an online survey of 1,719 participants in 97 nations fielded in May and June 2026, identifies about 6% of respondents as high performers: they attribute 5% or more of EBIT to AI and call the impact significant. Those high performers are twice as likely as others to report defined processes to measure the impact of AI initiatives.

Bain measures the gap with what it calls the realization rate: the share of expected value that becomes measurable business results. Its October 2026 brief, drawing on a survey of 951 companies, puts generative AI at 38% across functions, against 52% for traditional automation. Nearly 40% of companies that measured AI cost savings realized less than 10%, despite targets as high as 20%. Bain's verdict: "Program-level metrics (licenses deployed, workflows automated, hours saved) are no longer enough."

We think the cause is often structural, not technical. A pilot's success criteria measure the model. The value depends on the operation changing around it, and nobody in the pilot team controls that.

What does tracking inherit from the business case?

Three things. What goes into the case itself is covered in what a fundable AI business case contains.

  • A baseline. The Teal Book, the UK government's project delivery guidance, says: "The current performance level for each benefit should be established before work starts." Once the system is live, the old process has changed.
  • A benefit profile. The Teal Book's profile records each benefit's measures, baselines, targets and timescales, assumptions, dependencies and risks. Once approved, its forecasts and targets become the baseline for tracking.
  • A benefit owner. A named individual who confirms the benefit and is responsible for realising and reporting it. UK guidance notes that owners often sit in operations.

If any of these is missing at go-live, we think the first job is to reconstruct it, labeling a late baseline as an estimate. Without them, an AI business case can't be tracked against anything.

Which measures should you track?

Both kinds, paired. Leading measures, such as adoption, task completion and acceptance, move within weeks and show whether the lagging ones can move. Lagging measures, such as cost, revenue and cycle time, are what the case promised. In our view, AI adoption metrics reported alone are a progress report on the rollout, not on the benefit.

Add a quality guard to each pair, so a cost gain can't come from worse service. This is the same rule we argued for unit costs that drive decisions. The examples below are ours, not from any standard.

Use case typeLeading measuresLagging measuresHow value becomes money
Customer service assistant or agentShare of contacts resolved without handoff, agent acceptance of drafted repliesCost per resolved contact, repeat-contact rateFewer outsourced or contractor hours; avoided hires as volume grows
Document processing (invoices, claims)Share processed with no manual touch, reviewer acceptanceCost per task, cycle time, error rateCapacity released; earlier-payment discounts
Software engineering assistantActive use, share of suggestions accepted, review timeLead time for changes, change failure rate, roadmap deliveredMore delivered for the same budget; rarely cash
Sales and marketing contentAssets produced, time to launch a campaignConversion rate, pipeline per seller, agency spendRevenue; lower agency fees
Internal knowledge assistantWeekly active use, answers acceptedTickets deflected from HR and IT desks, onboarding timeFewer tickets; often non-cash

The FinOps Foundation's AI working group asks: "Is AI genuinely saving money, or is it simply automating existing inefficiencies?" Time saved becomes cash only when someone removes cost. The Teal Book puts cash-releasing benefits in the financial case as negative costs; in our view, a benefit that can't be written that way should be labeled non-cash.

An illustrative benefit profile: a customer service assistant

The numbers below are made up to show the mechanics. They don't come from any company.

A service operation resolves 50,000 contacts a month, partly through an outsourced partner billed by the hour. Measured over three months before launch, the fully burdened cost is $8.00 per resolved contact, or $400,000 a month, and 12% of customers contact again within 7 days. The case promises $6.00 per resolved contact within 12 months, AI run cost included: $2.00 × 50,000 = $100,000 a month, or $1.2 million a year.

The last automation wave delivered 60% of its forecast savings. Following Bain's advice not to fund the next wave as if it will deliver 100%, the funding decision assumes a $1.20 saving: $6.80 per contact, or $720,000 a year. The owner is the head of customer operations, who controls the partner contract.

MeasureTypeBaselineMonth 3Month 6Month 12
Contacts resolved by the assistant without handoffLeading0%15%25%35%
Agent acceptance of drafted repliesLeadingn/a70%80%85%
Cost per resolved contactLagging$8.00$7.60$6.90$6.00
Repeat contact within 7 daysQuality guard12%12% or less12% or less12% or less

At month 6, two readings are possible. If the assistant resolves 28% of contacts but cost is still $7.70, the model is working and the partner's hours haven't come down: the gap belongs to the owner. If cost has hit $6.90 but repeat contacts have risen to 15%, the assistant is closing contacts it hasn't solved, and part of the saving is illusory.

How often should you review, and who acts on a variance?

The Teal Book says to monitor each benefit against agreed metrics at an agreed frequency, set by how quickly it accrues and how easily it can be measured. The FinOps Foundation notes that many organizations meet monthly, that less predictable AI projects should be reviewed more often, and that a pilot seeking to scale should trigger a review. We suggest the benefit owner reviews leading measures and run cost monthly, and the portfolio review compares lagging measures with the profile quarterly.

In our view, the pattern of the variance tells you who acts:

PatternLikely causeWho acts
Leading on track, lagging behindFreed capacity not removedBenefit owner
Leading behindSolution quality or adoptionDelivery team
Lagging on track, quality guard slippingGain bought with worse serviceBoth; hold any scale-up
Both on trackBenefit realisingPortfolio review considers scaling

How long should you keep tracking after go-live?

Until the benefit is realised, not until the project ends. The Teal Book says a benefit is not closed simply because the associated work is completed, and records each closure in the benefits register with its basis, date and decision-maker. It also stresses clear ownership of benefits reporting in operations once the solution is in use.

After the project closes, the Teal Book allows further reviews through assurance, evaluation or a post-implementation review, agreed with the sponsoring body. UK gate reviews can include one 6 to 12 months after transition to operations. Where a comparison group exists, such as sites not yet rolled out, use it: UK guidance notes that evaluation often compares groups that received a change with similar groups that did not.

We suggest handing the profile to the operational owner before the delivery team disbands, and closing a benefit once its lagging measure has held at target for two consecutive quarters.

When should you scale or stop an AI initiative?

The Teal Book says work that is no longer likely to realise enough benefit to justify continued investment should be terminated, and that when assumptions stop holding, the impact goes through change control. The UK practitioner workbook names the failure mode: unwillingness to stop because of previous investment. Kill criteria belong in the case; tracking supplies the evidence. The rules we would apply at each review:

  1. Scale when the lagging measure is on profile at two consecutive quarterly reviews and the quality guard holds.
  2. Fix the operation when leading measures are on track but lagging measures aren't, with a dated plan from the owner.
  3. Fix or stop the solution when leading measures miss for two consecutive reviews.
  4. Re-baseline only through change control, with the reason recorded.

We think the hard part is keeping the baseline, owner, funding decision and measures connected for a year after the pilot team moves on. That is what an operating model is for: each AI dollar traceable to the benefit it was approved to deliver, which is what AI value realization means in practice.

Sources

Frequently asked questions

How long after go-live should you track AI benefits?

Until each benefit is realised, which usually outlasts the project. The UK government's Teal Book says a benefit is not closed simply because the work is completed, and UK gate reviews can include one 6 to 12 months after transition to operations. In our view, close a benefit once its lagging measure has held at target for two consecutive quarters, and record who decided.

Who should own the benefits of an AI initiative?

In our view, the operational leader whose budget or P&L would show the change, not the AI team. UK guidance says each benefit needs a named owner, often in operations, responsible for confirming, realising and reporting it. After launch, that leader acts on variances: releasing capacity, changing the process or renegotiating a contract, which is usually where AI benefits stall.

Are adoption metrics enough to prove AI value?

No. Adoption, acceptance and task completion are leading measures: they show whether the system is used and trusted, and whether lagging measures can move. Value shows up in cost per transaction, cycle time, revenue or error rates. Bain's October 2026 brief says program-level metrics such as licenses deployed and hours saved are no longer enough. Report both, paired with a quality guard.

What should happen when an AI initiative falls behind its benefit profile?

Read the pattern first. If leading measures are on track but cost hasn't moved, the benefit owner must remove the freed capacity. If leading measures are behind, the delivery team fixes the solution or its adoption. If misses persist and the benefit can no longer justify the investment, the Teal Book says the work should be terminated, and UK guidance warns against continuing because of previous investment.

Free playbook

Run the enterprise from any AI.

Read The Headless Enterprise
Steve Elliott

Steve Elliott

Published November 22, 2026

More on AI Economics

Tokens, inference cost, cost per task.