TheFork’s AI team on catching regressions early, finding hidden latency, and building the feedback loops needed to improve LLM-powered experiences at scale.
TheFork is one of Europe’s leading restaurant discovery and booking platforms, connecting millions of diners with tens of thousands of restaurants.
AI is increasingly woven into those experiences. TheFork is applying generative AI across review understanding and summarization, restaurant content, customer service, and new ways for diners to find the right restaurant.
That includes Ask TheFork, a retrieval and semantic-search experience designed to understand queries like “best tapas near me, open late” and surface relevant restaurants. The team is also applying AI to customer service, where conversations can be evaluated to understand how well an AI-powered assistant is performing.
As these experiences move into production, the challenge becomes bigger than building something that works in development. The team needs to understand how its AI is behaving with real users, where quality is degrading, and what to improve next.
Yesmine Rouis, Data Scientist at TheFork states, “It is crucial to use Arize AX to monitor the behavior of LLM models, as they can suffer from hallucinations and non-deterministic outputs, particularly when powering real-time features exposed directly to our users.”
Production AI needs a tighter feedback loop
For TheFork, small changes in AI quality can have an outsized impact.
Millions of hungry customers are searching across tens of thousands of restaurants. Improvements in retrieval quality or latency can mean a better experience finding where to eat, while regressions can directly affect experiences exposed to customers.
That creates a different set of requirements once an LLM-powered feature reaches production.
The team needs to evaluate the quality of prompts, retrieval, and guardrails; understand latency and cost across complex flows; and detect when changes introduce regressions.
Arize AX gives TheFork a way to bring those signals together.
Luca Temperini, CTO at TheFork adds, “Arize AX on AWS gives us prompt-level tracing, automated evaluations, and drift alerts, so we catch regressions early and meet strict Service Level Objective (SLOs) at scale.”
The result is a tighter feedback loop between what TheFork ships, how it performs in production, and what the team improves next.
"Arize AX on AWS gives us prompt‑level tracing, automated evaluations, and drift alerts, so we catch regressions early and meet strict SLOs at scale.”
Luca Temperini, CTO, TheFork
Evaluating the parts of the AI experience that matter
Rather than treating an AI application as one black box, TheFork uses tracing and evals to examine specific parts of the experience.
Engineers can focus evaluations on particular LLM calls, guardrails, retrieval steps, latency thresholds, token usage, or errors. They can then apply LLM-as-a-judge evaluations and code-based checks to understand how those components are performing.
The same evaluations can be run against historical data, giving the team a way to compare versions and identify regressions as applications change.
At TheFork’s scale, evaluating everything isn’t necessarily the goal. The team can target the slices of traffic that provide the most useful signal, such as spans crossing latency or guardrail thresholds.
That gives engineers a more focused question to answer: Where is the AI experience breaking down, and why?
"It is crucial to use Arize AX to monitor the behavior of LLM models."
Yesmine Rouis, Data Scientist, TheFork
Tracing a latency problem to its source
That visibility has already helped TheFork uncover concrete opportunities to improve its AI systems.
In one flow, tracing surfaced duplicated calls generating embeddings along a critical path.
Without visibility into the individual steps of the application, the team could see latency at the experience level but had less information about where that time was actually being spent.
Tracing exposed the duplicate work. TheFork removed it, eliminating unnecessary computation and materially reducing p95 latency for the flow.
It’s an example of observability moving beyond dashboards and into engineering decisions.
Yann Jouanin, Director of Engineering Strategy and Transformation at TheFork adds, “Arize helped us turn tracing into tangible wins: lower latency, clearer cost signals, and faster iteration.”
The result is a faster path from identifying an AI performance or cost issue to understanding its cause and making an improvement.
Using agent evals to catch regressions before users do
TheFork also uses real-time online evaluations in Arize AX to continuously understand how its AI applications are behaving.
Instead of relying only on testing before release, teams can evaluate production behavior and watch for changes in areas such as guardrail correctness and retrieval quality.
Moving to hourly data uploads, combined with targeted backfills, has helped reduce gaps in evaluation data and give teams faster readouts on how their systems are performing.
That matters because LLM-powered applications are inherently nondeterministic. A feature working well during development doesn’t guarantee that every production interaction will behave the same way.
Continuous evaluation gives TheFork another layer of feedback as those systems encounter real-world traffic.
“Arize helped us turn tracing into tangible wins: lower latency, clearer cost signals, and faster iteration.”
Yann Jouanin, Director of Engineering Strategy and Transformation, TheFork
Making cost another AI quality signal
Quality and latency aren’t the only constraints.
As TheFork’s use of GenAI expands, teams also need to understand what those experiences cost to operate.
By normalizing model and provider metadata, TheFork can analyze cost alongside other production signals and track metrics such as cost per 1,000 queries.
That allows teams to evaluate tradeoffs more clearly. A new model or prompt may improve quality, for example, but teams can also see what that improvement means for latency and spend.
Rather than optimizing those dimensions independently, TheFork can evaluate them together in one central dashboard before deciding what to ship.
Giving product and engineering a shared view of AI quality
The feedback loop isn’t only for AI engineering teams.
TheFork uses shared eval labels, cost tables, latency distributions, and experiment comparisons to give product teams transparency and a common view of how AI experiences are performing.
That visibility makes decisions about what to improve more concrete. Instead of relying on whether one version simply “feels better,” product and engineering teams can review evaluation results and production traces together, making informed tradeoffs across quality, latency, and cost.
As the team grows and evolves its AI stack, they plan to build on these lessons with richer session and multi-span evaluations, deeper cost visibility, and closer connections between technical AI signals and business outcomes.
The goal is a tighter loop between what the team builds, what happens in production, and what the team improves next.
