Arize Phoenix vs. Langfuse for AI observability and agent evaluation

Arize Phoenix vs. Langfuse at a glance

Quick answer: If you are comparing open-source AI observability and evaluation tools, the most useful head-to-head is Arize Phoenix vs. Langfuse. Langfuse gives teams a packaged workflow for tracing, prompt management, datasets, online and offline evaluation, and human review. Arize Phoenix gives engineers an open, OpenTelemetry- and OpenInference-based foundation they can customize around their application, with evaluation workflows that extend from individual spans and traces to agent trajectories and full sessions. Arize AX enters when teams want that same engineering loop with managed online evals, automated failure discovery, monitoring, governance, and production scale.

Both products cover tracing, evaluation, datasets, experiments, and self-hosting. The more useful distinction is their design center. Langfuse packages more of the workflow into a ready-made AI engineering product. Arize Phoenix is designed as an engineer-controlled observability and evaluation foundation that teams compose around the system they are building.

For Arize, that creates two layers. Phoenix is the open-source, self-managed product for tracing, evaluation, datasets, experiments, and prompt workflows. Arize AX is the managed AI engineering platform for teams that need continuous production evaluation, Signal, Alyx, monitoring, governance, and enterprise scale.

Langfuse is strongest as an early AI engineering toolkit with a polished path from instrumentation to traces, prompts, datasets, experiments, evaluators, and review. It also supports production tracing and online evaluation, so the distinction is not development versus production. The question is whether you want to adopt a packaged workflow or build the evaluation and observability layer more directly around your own system.

Teams that want a polished, packaged AI engineering toolkit with tracing, prompt management, evaluation, and review in one product will find Langfuse a strong fit. Teams that want an open-source foundation they can customize around agent behavior, evaluation scope, and their existing engineering workflow will find Phoenix the stronger comparison. When that workflow needs managed monitoring, automated failure discovery, governance, or enterprise scale, Arize AX adds the production layer without changing the overall Arize ecosystem.

How Arize Phoenix customizes the evaluation lifecycle

Phoenix hands engineers composable observability and evaluation primitives instead of requiring the application to fit one prescribed workflow.

• Instrument once, on open standards. OpenTelemetry and OpenInference cover 30+ frameworks and providers with one approach, so the trace record stays portable and no proprietary format sits between you and your data.
• Evaluate at the level the system needs. Spans, traces, and sessions are all first-class evaluation targets, so scoring a single retrieval call, a whole run, or a multi-turn conversation never requires restructuring the data first.
• Customize for agents and long-running work. Path evals measure whether the agent took the optimal route, convergence checks catch loops and unnecessary backtracking, and session evals track coherence across turns. Evaluators are code, so teams tune judge prompts, swap judge models, and encode policies specific to their system.
• Carry the work from CI to production. Datasets built from real traces feed experiments, evaluators version through the Evaluator Hub, and the same checks that gate a release in CI keep running once the system is live, so the eval suite is one artifact across the lifecycle instead of a pre-release copy and a production copy that drift apart.

What Langfuse does well

Langfuse earns its adoption, and the specifics matter:

• Tracing during development is fast to set up and easy to read. Teams can trace requests across prompts, retrievers, tools, and model calls, then inspect runs without building internal tooling first.
• Prompt management sits close to the traces. Teams can version prompts, compare behavior across releases, and connect prompt work to runtime evidence instead of keeping the two separate.
• The evaluation system is broader than a pre-release toolkit. Langfuse supports datasets, experiments, custom scores, LLM-as-a-Judge, code evaluators, human annotation, online evaluation of production traffic, and agent trajectory checks.
• Agents can operate parts of the workflow. Langfuse exposes evaluation workflows through agent-facing tools, MCP, CLI, and APIs, allowing coding agents to find low-scoring traces, add examples to datasets, configure evaluators, and inspect experiments.
• Pricing is friendly to collaborative teams. Core and Pro include unlimited users, and the Hobby tier provides a useful no-cost starting point. Usage then scales with the number of traces, observations, and scores ingested.

The tradeoff appears when teams want more control over how observability and evaluation are composed, or when self-hosting complexity and telemetry depth become larger parts of the architecture.

The real difference: packaged workflows vs. an engineer-controlled foundation

Langfuse packages a large portion of the AI engineering workflow into the product: traces, prompts, datasets, experiments, dashboards, LLM judges, code evaluators, annotation queues, and increasingly agent-operated workflows. For many teams, that is exactly the advantage. They can adopt the product’s workflow and get productive quickly.

Arize Phoenix takes a different approach. The trace and evaluation layer is closer to an engineering framework: teams decide what gets instrumented, what counts as good behavior, which unit gets evaluated, and how those evaluations connect to datasets, experiments, CI, or their own production workflows.

That becomes especially useful for agents because the important unit is often not one LLM response. It is a path through tools and state, or an entire session.

Agent teams eventually need to answer questions like:

• Did the agent take an efficient path, or loop and backtrack?
• Did it select the correct tool and arguments?
• Did a multi-turn session preserve context and reach the intended outcome?
• Did the execution follow a product-specific policy that no generic evaluator knows about?

Both products let teams build evaluations around those questions. The difference is where the customization lives. Langfuse increasingly exposes trajectory data and custom evaluators inside its packaged evaluation system. Phoenix makes programmable evaluation and open telemetry the foundation.

When teams want the Phoenix workflow to become a managed production operation, Arize AX adds continuous online evaluation, monitoring, Signal, Alyx, and enterprise controls.

Head-to-head comparison

Observability: seeing what happened

Arize Phoenix treats the trace as the raw material for evaluation. It accepts OpenTelemetry traces and uses OpenInference instrumentation to represent model calls, retrieval, tools, agents, and other AI-specific operations. Teams can inspect those traces directly, query span data for analysis or evaluation, and group multi-turn behavior into sessions.

Because the instrumentation layer is based on open telemetry standards, it can remain stable as the application moves between frameworks or adds custom services. That gives teams a framework-independent foundation for agent observability.

Langfuse also provides detailed tracing across LLM and non-LLM calls, sessions, users, costs, latency, and agent graphs, with native SDKs, framework integrations, and an OTLP endpoint. Mixed stacks can therefore send standard OpenTelemetry traces into Langfuse.

The portability nuance is what happens after ingestion. Langfuse maps incoming OpenTelemetry attributes into the Langfuse data model. Teams evaluating long-term portability should test not only whether Langfuse can receive OTel, but how the attributes and evaluation metadata they depend on export to another backend.

Evaluability: measuring whether it was good

Langfuse supports a substantial evaluation workflow: datasets, experiments, LLM-as-a-Judge, code evaluators, human review, online production evaluation, and scores attached to observations, traces, sessions, or dataset runs. Its agent-evaluation workflows also cover final-response, trajectory, and single-step evaluation, so it would be inaccurate to position Langfuse as lacking agent evals.

Arize Phoenix overlaps substantially with that workflow. The differentiation is strongest in how evaluation scope and customization are exposed:

• Explicit session evaluation. Phoenix provides workflows for evaluating a multi-turn conversation as a complete unit, including coherence and resolution. Langfuse supports session scores and session annotation, but its LLM-as-a-Judge workflow does not directly target a session. Teams instead evaluate an observation or trace containing the relevant conversation history and associate the result with the broader workflow.
• Agent-path customization. Phoenix includes evaluation patterns for path convergence, unnecessary retries and backtracking, tool use, and complete agent behavior. Langfuse also supports trajectory evaluation, so the useful comparison is not presence versus absence. Phoenix’s advantage is that those checks live in a programmable evaluation framework tied directly to open trace data, making it straightforward to encode system-specific policies and path logic.

This matters because agents can produce what researchers call corrupt success, where the final answer looks correct even though the execution violated an important constraint, used the wrong evidence, or reached the result through a broken path.

• A clean handoff to managed production evaluation. Phoenix evaluators can be developed and tested in code against trace data and experiments. Arize AX then adds the Evaluator Hub, managed continuous online evaluation, reusable evaluators, and production monitoring when the team no longer wants to operate that loop itself.

Actionability: getting from failure to fix

Langfuse has closed some of the automation gap that existed in earlier versions. Engineers can move production examples into datasets, run online evaluators, use code evaluators, and give coding agents access to parts of the workflow. An agent can find low-scoring observations, add representative cases to a dataset, configure evaluation rules, and inspect experiment results.

Phoenix keeps that workflow engineer-controlled: traces can become dataset examples, evaluators can be written and tested in code, and experiments can validate the next change.

Arize AX adds the managed automation layer. Signal continuously identifies recurring failure modes in production traffic, while Alyx investigates traces and helps build evaluators. Agent-as-a-Judge and continuous production review workflows extend the loop by inspecting live traces and turning discovered behavior into evaluation and monitoring signals.

The Arize AX CLI makes the workflow accessible from the terminal, while Arize Skills bring tracing, evaluation, experimentation, and instrumentation workflows into coding agents such as Claude Code and Cursor.

The difference is increasingly about how much of the improvement loop you want the platform to operate for you.

Operability: fitting how your team runs it

Self-hosting effort

Both platforms offer real self-hosting, but the operational footprints differ materially.

Phoenix can start as one application container using SQLite for storage, with PostgreSQL available for production or multi-user deployments.

Langfuse’s self-hosted architecture requires ClickHouse for traces, observations, and scores, PostgreSQL for transactional state, Redis/Valkey for queues and caching, and blob storage for events and large objects.

That does not make Langfuse’s architecture bad. ClickHouse is designed for high-throughput analytical workloads. It does mean a production self-hosted deployment has more stateful infrastructure to provision, upgrade, back up, scale, and troubleshoot.

Teams choosing primarily on self-hosting should include that operational cost in the comparison rather than looking only at software-license price.

Data ownership

Both products give self-hosted teams direct control of their data. The difference to test is the telemetry model.

Phoenix is built on OpenTelemetry transport and OpenInference conventions. Langfuse can ingest OTel as well, but maps incoming telemetry into its own internal trace and observation model.

For teams that consider backend portability a requirement, exporting a realistic application with custom attributes and evaluation results is a better test than checking an “OTel supported” box.

When the loop needs a managed production layer: Arize AX

Phoenix covers the engineering loop, and production adds work that goes beyond it: watching live traffic continuously, discovering failures nobody predefined, giving other teams a view into quality, and running all of it as managed infrastructure. That’s what Arize AX layers on top, with the same traces and evaluators Phoenix teams already have:

• Signal reviews production traces on a recurring schedule, groups related failures into prioritized issues, and surfaces evidence, likely causes, and the next change to test.
• Alyx, the in-platform AI engineering agent, investigates traces, proposes evaluators (with human confirmation before changes apply), and helps debug directly.
• Agent-as-a-Judge lets teams describe what good behavior looks like so an agentic harness can inspect live traces, identify the relevant spans, classify issues, and create labels that feed monitoring, evaluation, and experimentation going forward.
• Managed Agents and Agent Experiments extend the same idea across the platform, putting agentic workers on evaluation and experimentation tasks that teams would otherwise run by hand.
• Monitors, dashboards, and alerting route quality and cost regressions to PagerDuty and Slack, and give product, support, operations, and finance a shared view over the same quality signal engineers debug from.
• Governance and scale come with the managed layer: SOC 2 and GDPR support on the platform, HIPAA, RBAC, and audit controls on Enterprise, VPC and on-prem deployment with the control plane included, and adb underneath handling trillion-span monthly volumes.

The move is additive rather than a migration, since Phoenix and AX share the same open-standards record. Teams prove the loop in Phoenix and graduate when production makes the managed layer worth it.

Compare pricing at your expected production volume

Arize

• Phoenix: Free to self-host, with infrastructure and retention managed by your team.
• AX Free: $0, with 25k spans per month, 1 GB ingestion, 15-day retention, unlimited users and evaluations, and managed features including Alyx and online evals.
• AX Pro: $50/month, with 50k spans per month, higher ingestion and retention allowances, unlimited users and evaluations, and usage overages rather than a hard stop.
• AX Enterprise: Custom pricing for scaled workloads, configurable retention, enterprise support, governance, and SaaS or self-hosted deployment.

The structural difference is that Phoenix does not meter evaluation usage, while Arize AX prices its managed tiers around telemetry volume without per-seat charges. Teams can add more reviewers, evaluators, or agent-quality checks without creating a new seat-based cost dimension.

Langfuse

• Hobby: Free, with 50k units per month, 30-day data access, and 2 users.
• Core: $29/month, with 100k included units, unlimited users, 90-day data access, and additional usage starting around $8 per 100k units with volume discounts.
• Pro: $199/month, with 100k included units, 3-year data access, retention management, unlimited annotation queues, higher rate limits, and security reports.
• Enterprise: Starts at $2,499/month for Langfuse Cloud; self-hosted Enterprise has custom pricing.

The detail worth modeling is the billable unit. Langfuse defines monthly units as the number of traces + observations + scores. A multi-step agent can therefore create one trace, many observations, and multiple evaluation scores from a single user interaction.

Teams should model expected agent depth and evaluation frequency, not only the number of top-level requests. Self-hosted OSS removes Langfuse’s usage bill but replaces it with the infrastructure and operational cost of the deployment.

When Langfuse is the right choice

Langfuse fits when a team wants a packaged, collaborative AI engineering toolkit and values getting tracing, prompt management, datasets, experiments, online evaluators, annotation, and dashboards working quickly inside one product.

Its paid cloud tiers do not charge per seat, and the open-source version gives teams a credible self-hosted path when they have the infrastructure capacity to operate the full stack.

It is also a stronger agent-evaluation product than older comparisons may suggest. Langfuse now supports trajectory evaluation, tool-call checks, production code evaluators, and agentic access to evaluation workflows. Teams should evaluate it on the product that exists now rather than on the narrower tracing tool it was earlier in its development.

When Arize Phoenix is the right choice

Choose Arize Phoenix when:

• You want an engineer-controlled open-source foundation. OpenTelemetry and OpenInference keep instrumentation close to open telemetry standards while letting you customize the evaluation layer around your system.
• You’re building agents or multi-turn applications. Phoenix supports path and convergence evaluation and explicit full-session evaluation workflows, so quality does not have to collapse into one final-response score.
• You want to write evaluation logic around your system rather than shape the system around a fixed evaluation workflow. Phoenix’s client-side evals give teams direct control over evaluator code, judge models, data sources, and pipelines.
• You want a lighter self-hosting footprint. Phoenix can run as one application container with local storage, while PostgreSQL is available when the deployment needs production-grade persistence or multi-user scale.
• You want a production layer waiting when you need it. When managed online evals, Signal, Alyx, monitoring, governance, or enterprise deployment become requirements, Arize AX is the production layer.

A practical migration path

Teams rarely move in one cutover, and they shouldn’t. A safer path keeps the parts that work and moves the evaluation loop first:

• Crawl: Instrument one high-risk agent with Phoenix, confirm that prompts, retrieval, tool calls, latency, cost, and evaluator outputs are visible, and write the first system-specific evaluators.
• Walk: Build datasets from real failures, run experiments against them, and promote the evaluators into CI so releases get gated on the cases that actually broke.
• Run: When production traffic needs continuous review, add Arize AX for Signal, monitors, alerting, and cross-team dashboards on the same record.

During the transition, teams usually keep the Langfuse artifacts that still describe reality: datasets matching current workflows, evaluator logic and thresholds, annotation labels tied to real failures, prompt history that explains current behavior, and trace examples covering important edge cases. Stale prompt experiments and random debugging traces shouldn’t become long-term regression coverage, so a cleanup pass belongs in the plan. Langfuse can remain the historical record while Phoenix carries the active loop.