How PagerDuty turns AI agent quality signals into engineering action

PagerDuty is building GenAI directly into the workflows engineers use to understand systems, manage on-call schedules, and respond to incidents. Today, the company has over four agents live in production for customers, applying AI to workflows where reliability and fast response matter.

These agents are rolled out in PagerDuty Advance. First, the Insights Agent helps users ask questions about operational performance without digging through dashboards. Second, the Shift Agent helps engineers manage on-call schedules and find coverage. Thirdly, AI Site Reliability Engineer (SRE) Agent brings together incident context, runbooks, and other information to recommend what responders should do next.

As those systems became more agentic, PagerDuty encountered a new challenge: understanding what was actually happening between a user’s question and the agent’s final answer.

With a simple LLM chain, the team could follow a relatively straightforward sequence of questions and responses. However, agents were different. They could call multiple tools, make intermediate decisions, take different paths through a workflow, or even get stuck in loops.

“The biggest benefit we’ve seen is that Arize AX has opened up the black box of what an agent is,” says Ralph Bird, who has been leading work on PagerDuty Advance. “You need to have another way to see the inner workings of the agent, what it’s doing, where it’s getting things right, where it’s getting things wrong, and where it’s getting stuck in loops.”

That need has shaped how PagerDuty approaches observability, AI agent evaluation, and the development of its AI agents.

Evals are how you know what your agent is doing

For Ralph, evaluating an agent starts with accepting something fundamental about LLMs: getting an answer doesn’t mean getting a good answer.

“Evals are the way that you know what your agent is doing,” Ralph says. “Agents are a black box.”

LLMs make that particularly challenging because they tend to produce confident responses regardless of whether those responses are actually useful.

“The great thing about LLMs, and the bad thing about LLMs, is they will always give you an answer and they will give you that answer in a way that makes you want to believe that answer,” Ralph says. “So you need to be able to check that the answer it is giving you is a good, useful answer.”

PagerDuty uses different evaluation strategies depending on what an agent is being asked to do.

For some tasks, the team can create a known correct answer. If a user asks the Insights Agent how many incidents occurred yesterday, for example, PagerDuty can query its own data to determine the answer and evaluate the agent against it.

But even that isn’t static. Yesterday changes every day, so PagerDuty generates updated ground truth as part of the evaluation process.

Ralph describes it as “a living set of evals,” rather than a fixed golden dataset.

Evaluating the journey, not just the answer

Not every agent task has one correct answer.

Some problems can be solved in multiple ways. Others depend on live systems, calendars, or contextual information that makes creating a golden answer impractical.

For those cases, PagerDuty uses LLM-as-a-judge evaluations to examine individual parts of the agent journey.

Did the agent select the right tool? Did it provide the right parameters? Did the tool perform as expected? Was the final response relevant to what the user asked?

“Being able to break apart the full journey and provide individual assessments of bits of each of those traces allows us to gain confidence in the whole of a complex agentic system,” Ralph says.

That distinction matters as agents become more complex.

A successful final answer doesn’t necessarily mean every step was performed well. Likewise, when an agent fails, understanding its intermediate decisions makes it easier to identify where the problem started.

Turning user feedback into better evals

Automated evaluations are only one source of truth.

PagerDuty also incorporates feedback from the people actually using its agents.

Both the Insights and Shift Agents live in Slack, where users can give each response a thumbs up or thumbs down and provide additional context about their experience.

That feedback is connected to the span ID for the response and uploaded into Arize, allowing the team to return to the exact trace behind a user’s feedback.

“We can then pull those out and look at those individual traces,” Ralph explains. “Currently we’re able to use those individual human responses to identify areas where it’s performing badly or it’s performing well and build out our datasets from that.”

The same workflow is useful during internal testing. Testers can interact with a system that may include multiple agents and deterministic components, see the complete chain, and quickly annotate what happened.

Ralph says being able to combine that full journey with human feedback “has proved invaluable.”

Critical user journeys define what good looks like

PagerDuty has also learned that better agents don’t necessarily come from making them capable of doing everything.

Instead, the team identifies the critical user journeys that matter most and optimizes around them.

The Shift Agent provides a good example. One recurring question is: Who can cover my next on-call shift?

The agent could solve that problem through a fully agentic process. But because PagerDuty saw the same journey happening repeatedly, the team built a dedicated tool to handle that specific task more efficiently.

That made the experience faster while preserving what appears to the user as one seamless agent interaction.

The same critical user journeys become the foundation for evaluation.

“I think for me the most important thing is understanding what you want your agent to do and building out your critical user journeys,” Ralph says.

From there, PagerDuty works with product teams to define evals around quality, latency, and cost. Ralph adds, “Once we have those, you have something you can build against.”

Knowing when an agent starts getting worse

Offline testing can tell PagerDuty whether an agent performs well before a release. But once an agent is serving users, the team needs to know when that changes.

That became an important requirement when PagerDuty evaluated observability platforms.

“What Arize brings that we didn’t find in any other tooling out there was the ability to run online metrics,” Ralph says. “So it’s not just tracing. It’s the ability to evaluate how your agent is performing in real time.”

PagerDuty can use those evaluations with Arize monitors and its own incident response platform. If an output-quality metric crosses a threshold, the workflow can create a PagerDuty incident, notify the agent team’s on-call engineer, and create a Jira ticket so the issue has clear ownership.

That closes an important gap between AI evaluation and operational response.

“If the performance degrades, we need to know about it and we need to know about it now,” Ralph says.

Without real-time evaluation, the team could discover hours later that an agent had been experiencing a quality problem or potentially being attacked.

“So that real-time online evaluation and monitoring has been crucial to us as we’ve looked to develop our agents.”

Closing the loop from AI failure to engineering action

PagerDuty takes that real-time evaluation a step further by connecting AI quality signals directly to incident response.

With Arize monitors and custom metrics, the team can bring alerts into PagerDuty when an output-quality threshold is breached. The agent team’s on-call engineer receives a low-urgency page, while an incident workflow automatically creates a Jira ticket, giving the issue clear visibility and ownership.

PagerDuty is taking that closed loop further with its SRE Agent. The agent can triage AI incidents by classifying whether escalation to a human is needed and automatically bringing Arize failure-pattern context into the incident, giving responders more information to understand what happened and determine the next step.

The result is a closed feedback loop from detecting an AI quality issue to getting it in front of the right engineer with the context and ownership needed to take action.

Agent engineering is becoming a team sport

PagerDuty’s experience building agents has also changed who participates in AI development.

The work began within a focused AI team, but the company quickly started educating engineers across the organization on how agents work and how to build them. Product managers are increasingly participating as well, reviewing agent behavior and helping tighten the feedback loop between product requirements and performance.

Ralph sees agent engineering as a combination of disciplines.

Teams need to be comfortable with ambiguity and nondeterminism, but they also need rigorous software engineering practices to manage long-running processes, tool calls, and interactions between systems.

The result is increasingly a team effort.

PagerDuty’s specialized agent engineers act more like internal subject matter experts, solving difficult problems and building common tooling, while software engineers across the company can use those foundations to develop agents themselves.

As PagerDuty continues improving its agents, that combination of shared tooling, critical user journeys, evaluation, human feedback, and real-time monitoring gives the team a framework for understanding systems that are inherently nondeterministic.

The goal isn’t to make agents perfectly predictable. It’s to know what good looks like, see when behavior changes, and have the feedback loops in place to keep making them better.