How to pitch AI observability to your VP

You’ve seen the traces, so you’re convinced: the agent fails in ways nobody’s catching, the team debugs by archaeology, and every release ships on hope. Your VP hasn’t seen the traces, and won’t. The person who owns the budget line, whether that’s a VP of engineering or a VP of product, experiences the agent through a different interface: escalation counts in the ops review, a board question about AI quality, an engineer’s time that was supposed to go to the roadmap. Pitches that fail here fail the same way, by describing a tool to someone who’s weighing an outcome.

This playbook is built for the internal champion, so you can translate what you know from the traces into what your VP is actually deciding. For a deeper look at what the tooling category covers, see our guide to AI observability tools.

Understand what your VP is considering

Before writing a word of the pitch, map it to the three things any budget owner is balancing, since these are the axes every purchase gets judged on:

• Risk. Your VP answers upward when the agent embarrasses the company, and the question they fear is “how long was this happening before we knew?” A pitch that shortens that window speaks directly to their exposure.
• Velocity. The roadmap is their commitment, and every hour engineers spend paging through traces by hand is an hour taken from it. Framing observability as returned engineering capacity lands harder than framing it as a new capability.
• ROI. New spend competes with headcount and existing tools, so the comparison your VP will run is platform cost against what the team currently pays absorbing failures, and your job is making that second number visible, since today it’s spread across support tickets, delayed releases, and engineer time where nobody totals it.

A useful exercise before the meeting: write down the last question your VP asked about the agent. It was almost certainly a risk, velocity, or cost question in disguise, and it’s the opening line of your pitch.

Build the baseline first

The pitch is only as strong as the numbers under it, and four are enough. Pull them together for one production agent before any meeting gets scheduled:

1. Hours per week engineers spend investigating agent behavior by hand, gathered honestly from the people doing it.
2. Median time from a customer reporting a problem to a fix verified in production, reconstructed from the last few incidents.
3. Escalation volume attributable to the agent, from support’s own numbers, which also earns you an ally in support.
4. Release cadence for agent changes, and how many shipped with a real before-and-after versus a spot check.

These are the same four numbers a bounded pilot will re-measure, which is the point: the pitch and the proof use one yardstick. If the numbers are hard to assemble, that difficulty is itself evidence, and it belongs in the pitch as the current state of measurement.

Try Arize AX

Build better agents with Arize

Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.

Get started View AX docs

Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.

How to structure the pitch

The meeting needs three moves, in order, and the discipline is keeping the tool out of the first two:

Open with an incident instead of a category

Pick a real failure, ideally one your VP already heard about, and walk the timeline: when it started, when the team found out, who found it (usually a customer), and what the diagnosis took. The incident does the educating that a category explanation can’t, because it shows the detection gap in your own operation rather than asserting one exists in general.

Price the current state

Lay the four baseline numbers over the incident, and let them total quietly: the engineering hours at loaded cost, the escalations at support’s cost per ticket, the releases slowed or shipped blind. Resist projecting savings, since invented ROI invites skepticism, and the honest version, “here’s what we’re spending today to not know,” is stronger.

Make a small ask with a written finish line

The ask is a 30-day pilot on a high-traffic agent you’re already cleared to instrument. Pick one where you can get traces flowing in days — regulated or revenue-critical agents tend to get mired in PII review and change management, so steer toward something you can actually move quickly on.

Thirty days splits naturally into two phases. The first two weeks are about getting visibility established: instrumentation in, traces flowing, failures surfacing that nobody was tracking before. That’s worth showing your VP early, before the pilot’s over. The back half is where the workflow starts to pay off — investigation time comes down, recurring failures get caught before customers hit them, and the team’s debugging process changes in ways that are easy to demonstrate.

For success criteria, scope them to what a 30-day window can credibly move:

• Investigation hours returned, re-measured from the people doing the work
• Failures caught before customers reported them, with a count
• Report-to-fix time, projected from the incidents the pilot surfaces

Incident rates and support volume need more runway. Enterprise environments with lower incident frequency typically need 4 to 6 weeks before those numbers move in ways that hold up to scrutiny. Project them from what the pilot surfaces rather than trying to re-measure a median that won’t shift in a month.

Write the success criteria down before it starts—lightweight EvalOps gates for the pilot—and a pilot that produces its own data turns the second meeting from a pitch into a review.

What objections to expect

Four objections cover most VP responses, and each has an answer that respects the question:

• “Don’t we already have Datadog – or whichever APM we’re running?” Datadog confirms the agent responded fast with no errors and the incident you opened with did both while failing the customer. The gap is not visibility, it’s the improvement loop APM doesn’t own: evals tied to datasets and experiments to prove a change is better, discovery for the unknown failures no dashboard is watching, and a path from a production trace to a verified fix (agent evals from traces). That loop is what agent observability platforms are built to close. Datadog tells you the system stayed up, but it can’t tell you the agent did the job right, or help you close the loop when it didn’t.
• “Can’t the team build this?” The team built scripts, and scripts cover offline testing of known cases. The expensive failures are the unknown ones in live traffic, and the in-house version of continuous production evaluation and discovery is a platform team, which is a headcount conversation rather than a savings. Compare the tradeoffs in our build-versus-buy guide for AI observability.
• “Why now, can it wait a quarter?” Every month without measurement adds incidents that get found by customers and instrumentation debt that makes starting later more expensive. The pilot costs two weeks and nothing, so the real question is whether waiting a quarter buys anything, and it doesn’t.
• “What’s this going to cost?” Nothing until the pilot proves the case, and after that, a platform priced on data volume, modeled at production scale before any contract, with the pilot’s numbers beside it. Bring the buyer’s guide to choosing an AI observability platform for review questions and a comparison of the best AI observability tools for agents when the shortlist forms. Committing to bring the model rather than a quote keeps the meeting from becoming a negotiation before there’s data.

Notice each answer ends by returning to the pilot, which is deliberate, since the pilot is the answer that doesn’t require your VP to trust your judgment about tooling.

De-risk the ask completely

The champion’s advantage is that Phoenix, Arize’s open source project, carries zero license cost through the pilot. The two costs worth naming upfront are engineer time and model tokens if you’re running LLM-as-a-Judge evaluators, both of which your baseline already accounts for. Model-based judges are a real line item; see LLM evaluation costs for how to model them. Naming them before the VP asks is what keeps the “approve a pilot, not a purchase” framing credible.

Write the success criteria down before it starts. Because Phoenix is built on OpenTelemetry and OpenInference, the instrumentation work you do during the pilot carries forward to whatever platform wins procurement, which tends to pre-empt the “what if we pick wrong” conversation before anyone raises it. For the product split after the pilot, see Arize AX or Phoenix.

What to pitch, in one slide

For the champion who needs the literal artifact, one slide could read:

• The incident: one line, with the detection gap in days and who found it.
• What not knowing costs today: the four baseline numbers.
• The ask: a two-week pilot, zero license cost, on [agent name].
• The finish line: the same four numbers re-measured, plus failures caught before customers, reviewed together on [date].

Everything else, including the category education and the vendor landscape, waits for the second meeting, where the pilot data does the talking.

The infrastructure that backs the pitch

The pitch above only works because the free-pilot path exists, and it’s worth knowing the shape of what’s underneath. Phoenix, the open source project from Arize, runs the pilot itself: tracing on OpenTelemetry and OpenInference, offline and session level evaluation, and datasets and experiments for the before-and-after.

When the pilot lands and the conversation turns to production, Arize AX adds the managed layer. Your OpenTelemetry/OpenInference instrumentation carries forward, so the traces you captured in the pilot aren’t thrown away, though the production capabilities are AX’s own: continuous online evals and the Evaluator Hub, Signal for automated failure discovery and investigation, and project dashboards and monitors your VP can read without opening a trace. Pricing is on data volume with no per-seat charges, which keeps the year-two bill from becoming the next pitch you have to make. The Why Arize page carries the full differentiator story for the procurement stage.

Related reading

• AI observability tools: features and selection
• Agent observability: tracing and debugging
• How to choose an AI observability platform
• Best AI observability tools for agents (2026)
• Build vs buy AI observability
• EvalOps release gates
• Online LLM evaluations
• Build agent evals from traces
• LLM evaluation costs
• Arize AX or Phoenix

Pitching AI observability FAQs

What convinces a VP fastest?

An incident they already know about, walked as a timeline with the detection gap visible, followed by the four baseline numbers. Category arguments and feature tours convince slowly, while a specific failure with a price attached convinces in one meeting.

What if the answer is “let’s build it ourselves”?

Agree that the team should own its evaluators, since evaluation logic is genuinely system-specific, and separate that from the platform underneath—what our agent observability guide covers at implementation depth—where the in-house version of continuous production evaluation, discovery, and retention is an ongoing platform team. Offer to include a build-cost line in the pilot readout so the comparison happens with numbers rather than instincts.

What if budget is frozen this quarter?

The pilot needs no budget, which makes a freeze the ideal time to run it. Arriving at the next planning cycle with two weeks of your own data, a caught-failure list, and a modeled cost beats arriving with a vendor deck, and the instrumentation work done on open standards carries forward regardless of when procurement opens.

How long should the pilot run?

Two weeks on one high-traffic production agent is enough to return investigation hours and catch failures worth showing, and short enough that the ask stays easy. Don’t expect it to re-measure everything: median time from report to verified fix and escalation volume need more incidents than two weeks reliably produces, so re-measure the metrics that move fast and project the slower ones from the failures the pilot surfaces. Choose the agent for traffic and instrumentation access, not stakes. The highest-stakes agent is often the one mired in change management or a PII review that eats the window. Extend only if traffic is too low to surface anything, and resist letting the pilot sprawl across agents, since a sharp result on one beats a shallow result on five.