AI Agent Pilot to Production: Why Projects Stall

11 August 2026 · Nicholas Oneill

A practical guide for professional-services leaders to fix outcomes, workflows, permissions, evaluation, and ownership before scaling an AI agent.

Pilot worked but production decision did not

Consider a hypothetical professional-services firm testing an agent on a research or document task. It works on selected material, saves time for the people involved, and appears ready for wider use.

Real work adds client data, conflicting sources, system-of-record actions, approvals, and exceptions. The pilot proved that the agent could perform a task, but not that the firm could run it inside the workflow.

That is why a promising pilot can stall. The next decision is whether to scale, redesign, narrow, pause, or stop.

Production is a different test

A pilot tests whether a model and its tools can complete a bounded task. Production tests whether the business can depend on that capability inside a workflow with a defined outcome, a named owner, safe boundaries, useful evidence, and a plan for change.

Pilot proves capability — bounded task, selected material, useful output — feeding a production decision; production must establish a defined outcome, redesigned workflow, safe boundary, evidence loop, and named owner

"A successful task proves capability. Production requires evidence that the firm can run it inside the workflow"

Deloitte's 2026 State of AI in the Enterprise survey found that 25% of respondents had moved 40% or more of their AI pilots into production. Only 30% were redesigning key processes around AI, and the survey fieldwork ran in August and September 2025.

For professional-services firms, the value case is also hard to defend when nobody defined the result. Thomson Reuters' 2026 report says 18% of respondents report that their organisations track AI return on investment, while 40% do not know whether it is measured.

Neither survey establishes a universal failure rate. Together, they show why a task-level pilot can leave the production decision unresolved.

Where the decision breaks

The task passed, but the outcome is unclear

An agent can complete a task and still miss the result the firm needs.

Before expanding scope, define what should change. Set a baseline, a target, and a named owner for measuring the difference.

That measure might be turnaround time, rework, client response time, margin, risk exposure, capacity, or a quality threshold that matters more than speed.

If an agent saves time but adds a second review, the workflow may be slower. Measure the work around the agent as well as its output.

The workflow was never redesigned

An agent can sit beside an existing process while people compensate for it. They copy information between systems, check the same work twice, handle exceptions in untracked messages, or keep an approval step that the pilot ignored.

Map the end-to-end workflow. Name the affected roles, important handoffs, exception path, and decision the agent is allowed to prepare or take.

If an exception has no owner, the pilot does not yet have a clear safe path to production.

A task-level pilot can therefore run into an organisation-level problem when the surrounding workflow stays unchanged.

The operating boundary was never designed

Production forces specific answers about data, context, tools, permissions, reversibility, approval, and escalation. An agent that drafts a suggestion has a different risk profile from one that changes a client record, sends a message, or commits money.

For a buyer, a useful way to assess autonomy is by the actions the system can take, not by an instruction telling the model to be careful. Google Cloud's production guidance points to state, delegated approval, identity and tool governance, monitoring, orchestration, and evaluation as concerns beyond a demo.

OpenAI's practical guide notes that a deterministic approach may be enough for some tasks and that guardrails sit alongside authentication, authorisation, and access controls.

Match the control to the consequence. Microsoft's risk guidance distinguishes assistive work from actions that execute changes in a system of record and recommends matching oversight to risk.

The release boundary should sit around the action that changes a system, client record, or external state, not around every internal reasoning step.

Read-only work may need a lighter review path. Actions affecting clients or systems of record need clear permission limits, meaningful approval, records of what happened, and a way to contain or reverse the action where possible.

Drift has no owner

A production agent changes as its source material, integrations, users, and instructions change. Microsoft's observability guidance notes that uptime and error rates are not good indicators of AI quality and reliability.

Those measures may not show that an agent used the wrong context, called the wrong tool, escalated too late, or produced a plausible answer that failed the workflow.

Anthropic's guidance on agent evaluations distinguishes capability evaluations from regression evaluations. One asks what the agent can do; the other asks whether it still handles work that used to pass.

Tool-call checks, outcome checks, transcript review, and human grading can all contribute evidence. Microsoft's agent lifecycle guidance also calls for a named owner, monitoring, evaluation, improvement, and a route to retirement after go-live.

Run the same evaluation set before and after changes. Review sampled runs, escalations, incidents, permissions, and source freshness through a recurring operating rhythm matched to the agent's risk.

The post-launch evidence loop: agent runs in production, evidence is recorded (sampled runs, escalations, incidents), operating conditions change (permissions, integrations, source freshness), named owner reviews, operating decision (continue, adjust, restrict, pause, retire)

"Production ownership means reviewing evidence, responding to change, and adjusting the agent or its authority when needed"

Sometimes the use case is beyond the required quality, latency, cost, or risk threshold. Narrow it, use a conventional automation or point tool, pause, or stop.

Six steps before funding the next build

A pilot should earn more scope only after the team completes these steps.

Each step should leave a record the next person can inspect.

Define the outcome. Set the baseline, target, and measure. Assign someone to own the result, rather than using adoption or token counts as the outcome.
Map the workflow. Trace the work from start to finish, including handoffs, exceptions, approvals, the roles affected, and what happens after the agent produces its output.
Set the boundary. Specify the data and tools the agent can access, what it can write or change, and which actions are reversible. Start with the narrowest permission surface that can test the use case.
Place the controls. Decide where human approval is required and what triggers escalation. Keep approval at the actions with real consequence.
Test the evidence loop. Use production-like scenarios, evaluations, telemetry, feedback, and incident records to check quality and drift. Keep the test set small enough to run after every change.
Name the owner. Assign responsibility for the agent, context, permissions, integrations, updates, incidents, and retirement. Give that owner authority to change, pause, or retire it.

Six steps before funding the next build: define outcome, map workflow, set boundary, place controls, test evidence loop, name owner — leading to a decision: scale narrowly, redesign, narrow, pause, or stop

"The gate should produce a decision, not automatic permission to scale"

Strong capability does not compensate for an unsafe permission, absent escalation path, or unnamed owner. Any of those can stop the rollout even when the demo looks good.

The gate should produce a clear decision. Scale narrowly when the outcome, owner, boundary, and evidence are credible. Redesign when the capability is promising but the workflow or evidence is incomplete; narrow, pause, or stop when the use case is outside the firm's tolerance.

Where Aikin's Agentic OS point of view fits

Aikin's Agentic OS point of view starts with the workflow, not a list of agents. Aikin defines Agentic OS as the operating layer that connects context, tools, permissions, approvals, outcome tracking, and ownership to the work.

That begins with mapping handoffs and exceptions, separating human judgement from delegable work, and setting escalation thresholds before granting access or authority.

This operating layer is not needed for every task. A narrow read-only task, stable rule, or simple handoff may be better served by conventional software or a point tool.

It becomes useful when work crosses tools, carries persistent context, contains exceptions, involves consequential decisions, or needs ownership after launch.

Start suggest-only, then expand authority only when evaluation and operating evidence support the next boundary.

Production is a decision, not a reward

A pilot is useful when it shows what the agent can do and what the business still needs to make the work dependable.

Before funding another build, record the outcome, workflow, boundary, controls, evidence, and owner. Then choose the next step: scale narrowly, redesign, narrow, pause, or stop.

~ From Nicholas O'Neill and the Aikin team.

Frequently asked questions

What is the difference between an AI pilot and production?

A pilot tests capability under selected conditions. Production makes that capability dependable inside real work, with ownership, controls, measurement, and a plan for change.

What should an AI agent pilot prove before scaling?

It should show a credible outcome, fit a real workflow, use safe data and tool boundaries, pass production-like evaluations, and have a named owner.

When should a team stop an AI pilot?

Stop when the outcome is not credible, the capability or risk threshold is not met, or no safe operating boundary and named owner can be established.