Atropos Health is working to close the evidence gap in healthcare by making it easier for physicians to access observational studies on demand. The company conducts hundreds of studies each month, helping physicians get clinical evidence when they need it. But every study also creates another task: a physician needs to summarize the results and put them into context, a process that can take about 10 minutes per study.
To help, Atropos Health developed Auto Summary, an LLM-based tool that generates those summaries. Physicians can review and edit the output, and the tool was already running in production. But as Auto Summary scaled, the team faced a harder question: how well was it actually performing across hundreds of studies?
“We’ve been running it in production. Physicians can edit those summaries, but we didn’t really know how it was performing at scale,” says Rebecca Hyde, Principal Data Scientist at Atropos Health.
That question led Atropos Health to begin building a measurement framework in Arize AX, giving the team a way to move beyond individual physician edits and develop a clearer view of Auto Summary’s performance in production.
Challenge: Understanding performance at scale
One of the biggest challenges wasn’t building Auto Summary. It was creating the momentum to start measuring a system that was already working.
In a startup environment, there is always something else to build. Once an application appears to work well enough, going back to determine exactly how to measure its performance can easily fall down the priority list.
“I would say one of the challenges with monitoring was kind of getting the initial inertia to monitoring in a startup environment,” Rebecca says, “There’s always more things to do and more things to create, and once people feel like something’s good enough, they often don’t want to dig back into figuring out the right metrics to monitor it.”
For Atropos Health, the question became how to move beyond “good enough” and develop a clearer understanding of Auto Summary’s performance at scale.
The shift: Starting before the framework is perfect
Once the team got past that initial hurdle, it took an intentionally iterative approach rather than trying to define every metric upfront.
“Once we got over that hump, it was a relatively smooth process to actually just build an iterative and scrappy approach to figuring out what questions we wanted to answer and a few metrics that answered those questions,” Rebecca says.
That gave the team a practical starting point: begin with the questions they wanted measurement to answer, then identify a small set of metrics that could help answer them.
Instead of waiting until the team had the perfect measurement framework, Atropos Health could start measuring and refine its approach as it learned more.
Approach: Finding the metrics that matter
The framework is still evolving. Rather than continuing to add measurements, Atropos Health’s next step is to narrow its monitoring down to the one or two metrics that matter most.
“I think the next step that we’re going to take from here is refining that monitoring to narrow down to the one or two metrics that really matter to us,” Rebecca says.
The team also plans to explore how to use LLM-as-a-judge effectively “to get a very accurate read on what’s going on,” while continuing to build internal buy-in around the approach.
For Atropos Health, the process is deliberately iterative: start with the questions, test a few measurements, and refine the framework as the team learns which signals are most useful.
Results and impact: A clearer path to measuring with Arize
Atropos Health’s measurement framework is still developing, but what has changed is that the team has moved past the initial inertia around monitoring and established a practical process for figuring out what to measure. Rebecca describes the process as relatively smooth once the team got started.
The next phase is about refinement: narrowing the framework to the one or two metrics that matter most, determining how LLM-as-a-Judge can provide an accurate view of performance, and building broader buy-in within the company.
