After the pilot: how to track whether an AI initiative delivers the value it promised
In brief
Track AI value after the pilot as UK government tracks project benefits: a pre-launch baseline, an owner in operations, paired measures and variance reviews.
Steve Elliott · · 6 min read
On this page
- Why do AI initiatives struggle to show value after the pilot?
- What does tracking inherit from the business case?
- Which measures should you track?
- An illustrative benefit profile: a customer service assistant
- How often should you review, and who acts on a variance?
- How long should you keep tracking after go-live?
- When should you scale or stop an AI initiative?
- Sources
Track an AI initiative after the pilot the way UK government guidance tracks benefits from its projects and programmes. That means a baseline measured before launch, a named owner in the operation that has to change, a forecast trajectory to target, and reviews at set points that can stop the work. In our view, adoption figures tell you whether people use the system, not whether any money moved.
This post is about what happens after approval. It borrows from the UK government's benefits management guidance, refreshed in July 2026, and the FinOps Foundation's guidance on AI value. It sets out measures by use case, an illustrative benefit profile for a customer service assistant, how to read a variance and how long to keep tracking.
Why do AI initiatives struggle to show value after the pilot?
In our view, few organizations measure it well. McKinsey's State of AI 2026, an online survey of 1,719 participants in 97 nations fielded in May and June 2026, identifies about 6% of respondents as high performers: they attribute 5% or more of EBIT to AI and call the impact significant. Those high performers are twice as likely as others to report defined processes to measure the impact of AI initiatives.
Bain measures the gap with what it calls the realization rate: the share of expected value that becomes measurable business results. Its October 2026 brief, drawing on a survey of 951 companies, puts generative AI at 38% across functions, against 52% for traditional automation. Nearly 40% of companies that measured AI cost savings realized less than 10%, despite targets as high as 20%. Bain's verdict: "Program-level metrics (licenses deployed, workflows automated, hours saved) are no longer enough."
We think the cause is often structural, not technical. A pilot's success criteria measure the model. The value depends on the operation changing around it, and nobody in the pilot team controls that.
What does tracking inherit from the business case?
Three things. What goes into the case itself is covered in what a fundable AI business case contains.
- A baseline. The Teal Book, the UK government's project delivery guidance, says: "The current performance level for each benefit should be established before work starts." Once the system is live, the old process has changed.
- A benefit profile. The Teal Book's profile records each benefit's measures, baselines, targets and timescales, assumptions, dependencies and risks. Once approved, its forecasts and targets become the baseline for tracking.
- A benefit owner. A named individual who confirms the benefit and is responsible for realising and reporting it. UK guidance notes that owners often sit in operations.
If any of these is missing at go-live, we think the first job is to reconstruct it, labeling a late baseline as an estimate. Without them, an AI business case can't be tracked against anything.
Which measures should you track?
Both kinds, paired. Leading measures, such as adoption, task completion and acceptance, move within weeks and show whether the lagging ones can move. Lagging measures, such as cost, revenue and cycle time, are what the case promised. In our view, AI adoption metrics reported alone are a progress report on the rollout, not on the benefit.
Add a quality guard to each pair, so a cost gain can't come from worse service. This is the same rule we argued for unit costs that drive decisions. The examples below are ours, not from any standard.
| Use case type | Leading measures | Lagging measures | How value becomes money |
|---|---|---|---|
| Customer service assistant or agent | Share of contacts resolved without handoff, agent acceptance of drafted replies | Cost per resolved contact, repeat-contact rate | Fewer outsourced or contractor hours; avoided hires as volume grows |
| Document processing (invoices, claims) | Share processed with no manual touch, reviewer acceptance | Cost per task, cycle time, error rate | Capacity released; earlier-payment discounts |
| Software engineering assistant | Active use, share of suggestions accepted, review time | Lead time for changes, change failure rate, roadmap delivered | More delivered for the same budget; rarely cash |
| Sales and marketing content | Assets produced, time to launch a campaign | Conversion rate, pipeline per seller, agency spend | Revenue; lower agency fees |
| Internal knowledge assistant | Weekly active use, answers accepted | Tickets deflected from HR and IT desks, onboarding time | Fewer tickets; often non-cash |
The FinOps Foundation's AI working group asks: "Is AI genuinely saving money, or is it simply automating existing inefficiencies?" Time saved becomes cash only when someone removes cost. The Teal Book puts cash-releasing benefits in the financial case as negative costs; in our view, a benefit that can't be written that way should be labeled non-cash.
An illustrative benefit profile: a customer service assistant
The numbers below are made up to show the mechanics. They don't come from any company.
A service operation resolves 50,000 contacts a month, partly through an outsourced partner billed by the hour. Measured over three months before launch, the fully burdened cost is $8.00 per resolved contact, or $400,000 a month, and 12% of customers contact again within 7 days. The case promises $6.00 per resolved contact within 12 months, AI run cost included: $2.00 × 50,000 = $100,000 a month, or $1.2 million a year.
The last automation wave delivered 60% of its forecast savings. Following Bain's advice not to fund the next wave as if it will deliver 100%, the funding decision assumes a $1.20 saving: $6.80 per contact, or $720,000 a year. The owner is the head of customer operations, who controls the partner contract.
| Measure | Type | Baseline | Month 3 | Month 6 | Month 12 |
|---|---|---|---|---|---|
| Contacts resolved by the assistant without handoff | Leading | 0% | 15% | 25% | 35% |
| Agent acceptance of drafted replies | Leading | n/a | 70% | 80% | 85% |
| Cost per resolved contact | Lagging | $8.00 | $7.60 | $6.90 | $6.00 |
| Repeat contact within 7 days | Quality guard | 12% | 12% or less | 12% or less | 12% or less |
At month 6, two readings are possible. If the assistant resolves 28% of contacts but cost is still $7.70, the model is working and the partner's hours haven't come down: the gap belongs to the owner. If cost has hit $6.90 but repeat contacts have risen to 15%, the assistant is closing contacts it hasn't solved, and part of the saving is illusory.
How often should you review, and who acts on a variance?
The Teal Book says to monitor each benefit against agreed metrics at an agreed frequency, set by how quickly it accrues and how easily it can be measured. The FinOps Foundation notes that many organizations meet monthly, that less predictable AI projects should be reviewed more often, and that a pilot seeking to scale should trigger a review. We suggest the benefit owner reviews leading measures and run cost monthly, and the portfolio review compares lagging measures with the profile quarterly.
In our view, the pattern of the variance tells you who acts:
| Pattern | Likely cause | Who acts |
|---|---|---|
| Leading on track, lagging behind | Freed capacity not removed | Benefit owner |
| Leading behind | Solution quality or adoption | Delivery team |
| Lagging on track, quality guard slipping | Gain bought with worse service | Both; hold any scale-up |
| Both on track | Benefit realising | Portfolio review considers scaling |
How long should you keep tracking after go-live?
Until the benefit is realised, not until the project ends. The Teal Book says a benefit is not closed simply because the associated work is completed, and records each closure in the benefits register with its basis, date and decision-maker. It also stresses clear ownership of benefits reporting in operations once the solution is in use.
After the project closes, the Teal Book allows further reviews through assurance, evaluation or a post-implementation review, agreed with the sponsoring body. UK gate reviews can include one 6 to 12 months after transition to operations. Where a comparison group exists, such as sites not yet rolled out, use it: UK guidance notes that evaluation often compares groups that received a change with similar groups that did not.
We suggest handing the profile to the operational owner before the delivery team disbands, and closing a benefit once its lagging measure has held at target for two consecutive quarters.
When should you scale or stop an AI initiative?
The Teal Book says work that is no longer likely to realise enough benefit to justify continued investment should be terminated, and that when assumptions stop holding, the impact goes through change control. The UK practitioner workbook names the failure mode: unwillingness to stop because of previous investment. Kill criteria belong in the case; tracking supplies the evidence. The rules we would apply at each review:
- Scale when the lagging measure is on profile at two consecutive quarterly reviews and the quality guard holds.
- Fix the operation when leading measures are on track but lagging measures aren't, with a dated plan from the owner.
- Fix or stop the solution when leading measures miss for two consecutive reviews.
- Re-baseline only through change control, with the reason recorded.
We think the hard part is keeping the baseline, owner, funding decision and measures connected for a year after the pilot team moves on. That is what an operating model is for: each AI dollar traceable to the benefit it was approved to deliver, which is what AI value realization means in practice.
Sources
- Government Project Delivery, The Teal Book, Chapter 19: Benefits management, first published July 29, 2024, last modified July 1, 2026
- Government Project Delivery, Quick guide to benefits management in government, July 29, 2026
- Government Project Delivery, Practitioner workbook for benefits management in government, Part A: An overview of benefits management in government, July 29, 2026
- Government Project Delivery, Benefits management in government, July 30, 2026
- FinOps Foundation, Managing AI Value Using FinOps Practice Operations, updated February 23, 2026
- FinOps Foundation, Unlocking AI Business Value with FinOps, updated May 13, 2025
- McKinsey & Company, The state of AI in 2026: On the road to ROI, August 25, 2026
- Bain & Company, Realization Rate: The Metric Reshaping Enterprise AI, October 8, 2026
Frequently asked questions
How long after go-live should you track AI benefits?
Until each benefit is realised, which usually outlasts the project. The UK government's Teal Book says a benefit is not closed simply because the work is completed, and UK gate reviews can include one 6 to 12 months after transition to operations. In our view, close a benefit once its lagging measure has held at target for two consecutive quarters, and record who decided.
Who should own the benefits of an AI initiative?
In our view, the operational leader whose budget or P&L would show the change, not the AI team. UK guidance says each benefit needs a named owner, often in operations, responsible for confirming, realising and reporting it. After launch, that leader acts on variances: releasing capacity, changing the process or renegotiating a contract, which is usually where AI benefits stall.
Are adoption metrics enough to prove AI value?
No. Adoption, acceptance and task completion are leading measures: they show whether the system is used and trusted, and whether lagging measures can move. Value shows up in cost per transaction, cycle time, revenue or error rates. Bain's October 2026 brief says program-level metrics such as licenses deployed and hours saved are no longer enough. Report both, paired with a quality guard.
What should happen when an AI initiative falls behind its benefit profile?
Read the pattern first. If leading measures are on track but cost hasn't moved, the benefit owner must remove the freed capacity. If leading measures are behind, the delivery team fixes the solution or its adoption. If misses persist and the benefit can no longer justify the investment, the Teal Book says the work should be terminated, and UK guidance warns against continuing because of previous investment.
Steve Elliott
Published November 22, 2026
More on AI Economics
Tokens, inference cost, cost per task.
Keep reading
Initiative vs Program: Understanding the Key Differences
An initiative is a single effort toward a goal. A program is a set of related projects and initiatives managed together for benefits none of them would deliver alone.
Initiative vs Epic vs Feature: Understanding the Key Differences
An initiative is a large goal-driven effort spanning months. An epic is a body of work inside it, too big for one sprint. A feature is a piece of functionality users can use.
TBM after an acquisition: how to merge two IT cost models
After a deal, two IT cost models rarely add up. Map both to one TBM taxonomy, conform capitalization, and baseline run cost per tower before synergy targets.
How to Track Customer Retention: A Comprehensive Guide
Track customer retention with retention and churn rates, cohort analysis and engagement signals. Learn which metrics to use and how to act on them.