CVS Health teams moved a four-week feature from idea to production in roughly a day and a half, then completed a legacy rewrite in one month after estimating six to nine months. In their Arize Observe 2026 talk, Matt Turner and Lagan Khare explain how specifications, evaluation harnesses, AI observability, guardrails, and cost-per-outcome metrics can make that speed durable.
Enterprise AI projects often stall at the point where a convincing demo has to become a dependable production system. Evaluation, AI observability, governance, and cost controls suddenly carry as much weight as model performance, and teams quickly discover that these capabilities cannot be added casually after the build.
At CVS Health, lead director of architecture, Matt Turner, and executive director of engineering and enablement, Lagan Khare, have spent years developing the practices required to close that gap. Turner said their teams had taken a feature from idea to production in roughly a day and a half, and completed a legacy rewrite in one month after estimating six to nine months. Although the acceleration was dramatic, making it repeatable required a disciplined AI-native software development lifecycle supported by clear specifications, trusted tests, production monitoring, and measurable business outcomes.
Their experience points to a broader lesson for engineering teams: AI delivery speed depends on the systems surrounding the model. As developers generate more code, prototypes, and workflows at machine speed, evaluation-driven development and observability become core parts of the development process. Together, they provide the evidence teams need to grant agents greater autonomy without losing control of quality, cost, or risk.
Why enterprise AI pilots stall before production
Enterprise AI pilots stall when the surrounding operating system of software delivery cannot evaluate, govern, and absorb work at the pace a model produces it. The demo room tends to conceal that mismatch because a polished interaction can prove capability before anyone has settled the harder questions about release criteria, production monitoring, ownership, access, or return on investment.
Turner began the session by asking how many people had seen an AI-assisted demo that looked immediately useful, then followed with a question about systems that had reached production and produced measurable value. Together, those questions framed what he called “pilot purgatory,” a state in which a promising system waits for the organization around it to become ready.
Model improvements do little to resolve the missing discipline. Evaluation, governance, deployment controls, and measurement arrive through deliberate engineering work, while stronger base models can simply deliver more output into an unprepared process. When a pilot sits for more than a quarter, Turner argued, teams should inspect the surrounding system before assuming that another model release will rescue the project.
Khare approached the same problem through durability, which she described as the operating model that continues to hold after a demo has established capability and a deployment has shown that the team can ship. Durable value emerges when that model survives growing usage, shifting source data, accumulating costs, and newly visible failure modes.
That distinction changes the question developers should ask. Production becomes an engineering path that teams need to design before the first agent begins making consequential decisions.
What is an AI-native software development lifecycle?
An AI-native software development lifecycle uses agents across individual development, team coordination, and production validation while preserving human ownership of intent, evidence, and release decisions. Turner described three layers of discipline, each of which creates a new form of leverage and a new class of failure.
Layer 1: AI coding tools and specifications accelerate individual developers
Personal productivity is the usual on-ramp because coding assistants produce immediate, tactile gains. An engineer can explore a codebase, scaffold a service, generate a working spike, and draft tests before a conventional handoff would have begun. At CVS Health, general coding tools sit alongside shared Model Context Protocol (MCP) servers, source-control and ticketing integrations, and reusable skills that encode repeatable practices.
One internal skill, called Clarify, helps align product and engineering intent before implementation starts. Its purpose reflects a larger shift toward specifications that agents and humans can evaluate: the specification becomes a durable source of truth containing structured requirements, edge cases, acceptance criteria, and system context.
“The spec is as much for the engineer as it is for the agent building it,” Turner said.
That obligation matters because an agent can produce code that compiles while misunderstanding the feature. A specification forces the engineer to articulate the intended behavior before delegating the work, which gives reviewers a stable artifact against which they can judge the implementation. As a library of well-formed specifications grows, new features become faster to scaffold because previous context, contracts, and decisions remain available.
Turner has also seen teams prototype live while stakeholders remain in the room. Because the working system arrives while the conversation is still active, product, design, and engineering can test assumptions before they harden into a backlog item or an architectural commitment. The prototype becomes an instrument for collective reasoning, allowing the team to validate the idea before a conventional post-meeting handoff would return.
The same acceleration creates traps. Vibe coding encourages developers to approve code they do not understand, while agent-generated tests may confirm the assumptions already embedded in the implementation. Large tasks can overwhelm the available context until the agent loses coherence, so engineers need to decompose work into smaller, executable units with clear intent. Shared tooling also needs enough consistency for practices to compound across teams, since isolated tool choices prevent one group’s learning from becoming organizational infrastructure.
Layer 2: Shared context and dependency maps align engineering teams
Once individual output rises, coordination becomes the pacing system. Product intent, design decisions, service contracts, and dependencies must leave people’s heads and enter artifacts that humans and agents can both access, because undocumented knowledge becomes an immediate ceiling on agent performance.
Specifications gain a second function at this layer. They align product, design, and engineering around the same acceptance criteria, while revealing gaps before generated code turns them into rework. Turner cautioned against polishing every spec into a monument because the useful artifact only needs enough structure to support fast iteration, with a scope small enough for both the team and the agent to reason about clearly.
Dependency mapping becomes equally important because agentic changes can cross service boundaries faster than organizational memory can track them. “If an agent doesn’t know what its change is going to break, it will break it,” Turner says.
A dependency map gives the agent a view of the blast radius before it changes a contract, a data flow, or a shared component. Without that context, the time saved during generation can disappear into cleanup, incident response, and cross-team negotiation.
AI can also compress the distance between roles, allowing product managers, designers, and engineers to work in parallel around the same prototype and specification. Turner heard several teams describe this emerging identity as “builders,” a term that captures how the boundaries between defining, designing, and implementing a feature can become more porous.
The organizational constraint does not vanish with the handoff. Turner described features that developers completed in hours and then watched stall for weeks because user validation, product review, or dependent teams could not move at the same speed. AI amplifies the coordination system already in place, so healthy teams gain more parallel capacity while brittle silos collide earlier.
Layer 3: Evaluation and observability earn agent autonomy
Production autonomy depends on evidence that the system behaves correctly and can be stopped when its behavior changes. Turner framed this layer as the point where trusted tests, agent observability, staged rollouts, behavioral dashboards, and rollback mechanisms become prerequisites for speed.
Deterministic tests provide the foundation, although coverage percentages alone offer little comfort when the tests fail to represent meaningful behavior. AI can propose cases and draft test code, while humans remain responsible for verifying that those tests catch failures the business and the user would care about.
Agentic workflows add nondeterminism, which requires evaluators that can inspect whether outputs and end-to-end behavior satisfy the intended outcome. A pull request may look reasonable line by line while producing an unexpected behavior after deployment, so review shifts toward intent, system impact, and the evidence supplied by traces and evaluations.
User validation belongs in the same loop. Faster shipping creates value only when users can confirm that the system solves the intended problem, and a rapid implementation without rapid feedback can turn acceleration into efficiently produced waste.
When teams increase generation volume before building this layer, the resulting failures become harder to locate and more expensive to contain. In a regulated environment, an undetected error can also become a compliance event, which means explainability and audit trails influence the ceiling on autonomy as directly as model capability does.
What should an AI evaluation harness include?
An AI evaluation harness should include golden datasets, regression suites, and continuous production monitoring that together define acceptable behavior, catch regressions, and reveal drift. The evaluation harness gives teams a repeatable way to answer whether an agent is good enough to ship and whether it remains good enough after release.
“Build the harness before the agent,” Khare asserts.
Khare compared evals for generative AI systems to unit tests in conventional software engineering. The analogy becomes especially useful when teams place evaluation at the beginning of development, because the cases and thresholds force them to define quality before prompt tuning and model selection consume the conversation.
Golden datasets and regression suites define release quality
Golden datasets encode the cases that matter, including expected behavior, known edge cases, and failures that would make a release unacceptable. Regression suites then preserve those lessons by rerunning the cases whenever the agent or its surrounding system changes.
This practice turns evaluation into a design activity. When a team has to decide which examples deserve permanent protection, it exposes disagreements about intent while those disagreements remain inexpensive to resolve. The resulting suite also gives reviewers a body of evidence that can support a release decision without relying on a handful of manually selected demos.
Human judgment remains essential because a weak evaluator can reward the same shallow behavior that a weak test would permit. Teams need to review the cases, scoring criteria, and failure thresholds with the same seriousness they apply to production code, especially when an agent’s output has several acceptable forms.
Continuous evaluation detects model, data, and user drift
Continuous evaluation extends the harness into production, where models, source data, and user behavior can all drift. A launch score captures a moment; an operational evaluation program reveals whether that score changes over the next 30 days and which slices of traffic account for the movement.
That history allows developers to separate a broad regression from a localized failure, such as a tool-call issue, a newly ambiguous user request, or a data-quality problem in one source. Traces and production samples can then feed new cases back into the golden dataset, which lets the harness mature alongside the system.
This feedback loop gives AI agent evaluation its production value. In evaluation-driven development, the harness becomes the mechanism through which teams learn what the agent does under conditions the original test set did not anticipate, while each newly observed failure sharpens the next release decision.
How should teams measure AI agent cost and ROI?
Teams should measure AI agent economics through cost per outcome and connect that figure to a business KPI such as cycle time, error rate, cost per transaction, or time to resolution. Token usage remains useful for infrastructure analysis, although it cannot capture the retries, rework, escalations, and review burden created by an unreliable workflow.
Cost per outcome captures retries, rework, and review
Khare argued that the meaningful unit is cost per outcome because a cheaper model can produce a more expensive operating system when its outputs require additional cleanup. A model that saves pennies during inference may add dollars downstream when a human has to verify, correct, rerun, or escalate the work.
The unit should match the job the agent performs. A support workflow might track cost per resolved inquiry, a document workflow might track cost per processed document, and an internal automation might track cost per completed task. Those measures incorporate the full path to a usable result, which makes them more informative than a monthly token bill detached from task success.
Production planning should therefore include per-feature spending ceilings, fallback paths, and observability on spend. These controls allow engineering and finance to understand how costs behave under real volume, where retries and exceptions can make pilot economics misleading.
Business KPIs belong in the AI intake process
A production candidate should enter development with a business metric already attached. Khare named cycle time, throughput, documentation turnaround, error rate, cost per transaction, and time to resolution as examples that can connect an AI system to operational value.
When the metric appears during intake, engineering can design the evaluation suite and production monitoring around the result the organization expects. The same metric then supports prioritization, release decisions, and post-launch analysis, which prevents ROI from becoming a story assembled after the budget has already been spent.
Khare’s rule was simple: the metric comes before the model. Finance belongs in this conversation early because enterprise scale changes the economics. Although a pilot may look inexpensive while serving a limited audience and routing difficult cases around the system, production exposes the true cost of retries, human review, fallback models, data access, and ongoing monitoring. Cost per outcome gives finance and engineering a common unit for deciding whether the workflow deserves to expand.
Which parts of the agent harness survive better models?
The durable parts of an agent harness are the data, workflow design, evaluation suites, permissions, and trust fabric that remain valuable as model capabilities improve. Teams should scrutinize custom orchestration, retrieval, and routing layers because a future model or API may absorb functions that currently require extensive scaffolding.
“The model will eat your stack,” Khare said, describing how quickly capabilities can migrate from surrounding infrastructure into the model layer.
The observation redirects engineering investment toward the parts of the system that retain value as the model changes. A workflow that captures domain intent, constrains access, records traces, and measures outcomes can continue to create value after the underlying model changes. A brittle layer built around a temporary limitation may become maintenance work with a short half-life.
Data quality, workflow design, evals, and trust remain durable
Data quality moves onto the critical path because agents amplify ambiguity at scale. A messy column that an experienced analyst can interpret through context may lead an agent toward a confident error, and the resulting answer can propagate through a workflow before anyone notices the source of the mistake.
Clean source material, relevant context, and well-defined semantics therefore remain part of production engineering. The work may lack the spectacle of a new model demo, although it determines whether the system can use that model safely.
Workflow design also survives because a model does not inherit the organization’s operating context. Teams still have to decide which problem deserves automation, how responsibilities move between systems and people, where approval belongs, and what evidence should accompany an action. Evaluation suites preserve those decisions as executable expectations, while the trust fabric enforces the permissions and auditability required to operate them.
Why can AI workflow automation create downstream bottlenecks?
AI workflow automation creates downstream bottlenecks when one step accelerates without a corresponding increase in the capacity of the teams and systems that receive its output. Because AI often performs only a small part of a larger process, local throughput can rise while end-to-end completion time becomes worse.
Khare described an example in which Team A’s agent ran continuously and buried Team B under the resulting tasks. The first group could point to a faster local process, yet the organization had merely moved the queue to a place with less capacity.
The pattern is easy to miss during a low-volume pilot, which allows downstream teams to absorb the extra work, whereas production changes the shape of the system. A workflow that generates cases, documents, code changes, or review requests around the clock can overwhelm a step that still depends on a small group of specialists.
Map the full workflow before automating a step
Teams should map the complete workflow, including upstream inputs, downstream queues, approval points, and exception paths, before choosing the automation target. That map reveals where increased volume will land and whether another step will become the new constraint.
The exercise also changes prioritization because the easiest manual task may offer little system-level value when it feeds a scarce reviewer, while a less obvious intervention could remove a handoff, improve source data, or reduce the number of cases that require escalation. Developers need to evaluate the automation across an end-to-end workflow containing users, reviewers, compliance functions, dependent teams, and other software.
This systems view also protects the business metric. When teams measure cycle time or cost per completed task across the full workflow, a local speedup cannot disguise a slower overall process.
How do AI agent guardrails work in regulated environments?
AI agent guardrails in regulated environments combine mechanical controls that constrain what an agent can physically do with text-based controls that make its intended behavior reviewable. Khare argued that both layers belong in the architecture because failures carry asymmetric costs, and a defect that would cause a retry in a consumer application can trigger audit exposure, notification obligations, or a breach in a regulated workflow.
Mechanical guardrails constrain production access
Mechanical guardrails use deterministic controls to enforce boundaries around data, tools, and actions. They include hard checks on information that cannot leave the system, scoped service accounts that grant only the access an agent needs, audit trails for every action, and fallback to human review when confidence drops.
Least-privilege access becomes especially important because an agent cannot misuse a capability it never received. When a hallucination leads the system toward an inappropriate action, the permission layer should prevent execution before the action reaches production.
Mechanical controls can also combine deterministic checkpoints with nondeterministic evals. A policy rule may block protected data from leaving a boundary under any condition, while an evaluator examines whether a generated explanation, summary, or recommendation satisfies a domain-specific rubric. Observability-driven guardrails can then provide the trace needed to see which control fired and how the agent arrived at the attempted action.
Text-based guardrails make agent intent reviewable
Text-based guardrails document what the agent must do, may do, should do, and cannot do. Behavior-driven acceptance criteria let reviewers evaluate intent before they inspect implementation details, which becomes increasingly important when agents generate large code changes or execute multi-step workflows.
These constraints should remain traceable to a rule or a source. When an agent produces an output or takes an action, the system should preserve enough evidence to show which instruction, retrieved context, policy, or approval informed the behavior. That traceability supports debugging, regulatory review, and continuous improvement.
Layered review strengthens the same discipline. Reviewers can first confirm that the proposed behavior matches the intended outcome, then inspect whether the implementation enforces that behavior under realistic conditions. This ordering keeps a technically polished change from bypassing a flawed assumption.
Human review requires a designed intervention loop
“Human in the loop only counts if the loop is designed,” Khare said.
A useful intervention loop identifies the responsible reviewer, the precise step at which they enter, the evidence they receive, and the authority they hold. Without those decisions, a nominal human checkpoint can become a queue with unclear ownership or a ceremonial approval that lacks the context needed to catch a failure.
Confidence thresholds should route ambiguous or high-risk cases to the appropriate person, while audit trails should record the agent’s proposed action, the reviewer’s decision, and the evidence available at the time. Explainability then supports both practitioner trust and a defensible account of how the system behaved.
Governance scales this pattern across use cases. Shared standards for data boundaries, evaluation, identity, permissions, review, and traceability allow each deployment to build on previous learning, whereas one-time approvals force every team to rediscover the same controls under a different project name.
What does production readiness look like for enterprise AI agents?
Production readiness for enterprise AI agents requires a defined business outcome, an evaluation harness, observable production behavior, controlled access, cost-per-outcome measurement, and explicit rollback or human-review triggers. These elements give a team enough evidence to decide when an agent can act, when its autonomy should narrow, and when the system should stop.
Define the business metric before development begins
Khare advised teams to concentrate their effort on a small number of concrete workflows, which keeps a large portfolio of weak pilots from consuming attention without producing durable learning: “Three concrete bets beat 30 weak pilots every time.”
Each workflow should carry a success metric written in business terms before development begins. A metric such as cycle time, error rate, cost per resolved inquiry, or documentation turnaround gives the team a stable target, while a general claim that users liked the demo provides little basis for investment or release.
Turner’s three-layer model offers a useful way to choose those bets. A team can identify one workflow where individual productivity matters, one where cross-team coordination creates friction, and one where validation and observability constrain autonomy. That scope allows the organization to learn across the full lifecycle without attempting to transform every process at once.
Set evaluation, rollback, and human-review thresholds
Taken together, Turner and Khare’s framework produces a practical production gate:
- Define the business outcome and the workflow-level metric that represents success.
- Build a golden dataset and regression suite around the cases that matter.
- Establish release thresholds, production monitors, and a process for adding newly discovered failures to the harness.
- Instrument traces, costs, agent actions, and behavioral changes so the team can explain what happened.
- Constrain access through scoped permissions, deterministic checkpoints, and auditable actions.
- Document the confidence levels and conditions that trigger rollback, fallback, or human review.
- Track cost per outcome after launch and compare it with the business result the system was built to move.
The final test is the question Khare left with the audience: How will the team know whether the agent is working, and how will it stop the agent when the evidence says otherwise?
Models will continue to improve, which means the volume and ambition of agentic software will continue to rise. The teams that convert that progress into durable production systems will have already built the specifications, evaluation harnesses, observability, guardrails, and economic measures that make autonomy accountable.