What Is LLMOps?

LLMOps (large language model operations) is the set of practices for versioning, evaluating, deploying, and monitoring applications built on large language models. It treats prompts, models, retrieval, tools, and data as one changing system, so you can ship a change, measure its effect in production, and roll it back or improve it with evidence.

For example, suppose a support assistant starts answering from an outdated refund policy after a document update. The trace shows the new policy was indexed, but retrieval returned the old chunk first. The fix is a change to chunking and ranking, not to the prompt. That is only visible if each release records the index version along with the prompt and model.

Try Arize AX

Build better agents with Arize

Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.

Get started View AX docs

Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.

Why LLMOps matters

In an application built on a large language model (LLM), many parts can change behavior. A model update, a prompt edit, a new index, or a changed tool definition can each alter answers. LLMOps gives you a way to know which change did what.

LLMOps helps with:

• Reproducing a release: Know exactly which model, prompt, index, and tools produced a given behavior.
• Catching regressions early: Block a release that fails cases you have already fixed once.
• Finding root causes: Trace a bad answer to retrieval, a tool, or the model instead of guessing.
• Weighing cost against quality: Compare configurations on cost, latency, and task success together.
• Improving from real usage: Turn production failures into the next round of test cases.

How LLMOps works

An LLMOps workflow is a loop. You version what can change, evaluate it before release, deploy it carefully, and monitor it in production. Failures you find in production feed back into the next evaluation.

Arize’s LLMOps course on operationalizing LLMs at scale walks through the full lifecycle. The sections below cover the core steps.

Version everything that changes behavior

A prompt-only change can alter tool calls, citation behavior, and refusal rates. So a release record needs more than the prompt.

For each release, record:

• The model identifier and its parameters.
• The prompt template.
• The retriever and index version.
• The tool definitions.
• The guardrail settings.
• The version of the evaluation dataset used to test it.

Here is an illustrative release record. The names and version tags are placeholders.

release: support-assistant-2026-10-24
model:
  id: provider-model-pinned-version
  temperature: 0.2
prompt_template: support-answer@v14
retrieval:
  index_version: policies-2026-10-20
  chunking: section-512
  ranker: hybrid-v3
tools:
  definitions: tools@v6
guardrails: support-guardrails@v2
eval_dataset: support-regressions@v31

Prompt versioning is one part of this record, not the whole release.

Use evaluation as a gate

Before a release, run the candidate against a curated dataset. That dataset should include regressions you found in production, not only hand-written examples.

The operational point is to make the result a decision:

1. Define a threshold for each evaluator.
2. Block or warn when the candidate falls below it.
3. Store the dataset version with the result.

An offline pass does not guarantee production quality, so keep evaluating after deployment too. For the evaluator choices themselves, see LLM evaluation.

Deploy and monitor

Deployment can mean a model endpoint change, a prompt release, or a new index. A common pattern is to send a small share of traffic to the new release first and compare it with the current one. This is often called a canary deployment.

After deployment, watch traces for latency, token use, errors, retrieval quality, and evaluator scores. Distributed tracing makes that inspection possible across model, retrieval, and tool spans.

Feed production failures back into evaluation

The improvement loop has four steps:

1. Sample real failures from production traces.
2. Add them to an evaluation dataset.
3. Fix the cause.
4. Re-evaluate before the next release.

For a detailed walkthrough of this loop, see automating the LLMOps feedback loop from production traces.

Worked example: fixing an outdated refund answer

Here is how the refund policy problem from the opening moves through the loop.

1. Save the failing trace as an evaluation example, with the current policy as the expected answer.
2. Change chunking and ranking, then re-run the dataset offline.
3. Deploy to a small share of traffic and compare evaluator scores and latency against the current release.
4. Promote the release only when the new example and the older regression set both pass.

The fix was not a prompt edit. This is why the release record had to include the index. If it had tracked only prompts, nothing in the record would explain the behavior change.

How to compare cost and quality

Cost, latency, and task success belong together. A cheaper run that fails more often can cost more for each task it actually completes.

Formula:

cost per success = cost per run / success rate

Where:

• cost per run is the average spend for one attempt at the task.
• success rate is the share of runs that complete the task correctly, as a decimal.

The following script compares two hypothetical configurations of the same assistant.

# Hypothetical cost per run and task success rate for two configurations
configs = {
    "A (larger model)": {"cost_per_run": 0.020, "success_rate": 0.90},
    "B (cheaper model)": {"cost_per_run": 0.015, "success_rate": 0.60},
}

for name, c in configs.items():
    cost_per_success = c["cost_per_run"] / c["success_rate"]
    print(f"{name}: ${c['cost_per_run']:.3f} per run, "
          f"{c['success_rate']:.0%} success, ${cost_per_success:.4f} per success")

Output:

A (larger model): $0.020 per run, 90% success, $0.0222 per success
B (cheaper model): $0.015 per run, 60% success, $0.0250 per success

Configuration B is 25% cheaper per run but 12.5% more expensive per successful task. For more on this metric, see cost per successful task.

Limitations of LLMOps

• Weak datasets give false confidence: LLMOps does not create a good evaluation dataset by itself. If the dataset misses real user phrasing, the gate passes releases it should block.
• Provider updates bypass your gates: A provider model update can change behavior without any local change. You still need production monitoring.
• Traces can contain sensitive text: Apply retention and redaction rules before traces become shared datasets.

LLMOps vs. MLOps, EvalOps, and related terms

These terms overlap, so here is how they differ.

• MLOps (machine learning operations): Grew up around training, feature pipelines, and model registries for models a team trains itself. LLMOps often uses a provider model the team does not train. Prompt, context, and retrieval versioning matter more, and training pipelines matter less.
• EvalOps: Focuses on the evaluation pipeline and its behavior in continuous integration (CI). LLMOps also includes deployment, serving, monitoring, and incident response.
• Model serving: Covers only the deployment and inference layer.
• Agent engineering: Adds tool use, planning, and multi-step control on top of an LLM application. LLMOps is the operational discipline that can support either a single-call application or an agent.

Frequently asked questions

Is LLMOps just MLOps with a different model?

Partly. Deployment, monitoring, and rollback carry over. The versioned artifacts differ, though. Prompts, context construction, retrieval indexes, and provider model settings replace much of the focus on training pipelines.

What should I version first if I have nothing today?

Start with the prompt, the model identifier, and a small regression dataset drawn from real failures. Add retrieval and tool versions when those parts exist. A short record kept consistently is more useful than a complete record kept once.

How does LLMOps relate to EvalOps?

EvalOps focuses on running evaluations reliably, often in CI. LLMOps is broader. It uses those evaluations as one gate inside deployment, monitoring, and improvement.

Do I need a separate LLMOps platform?

Not necessarily. You need versioned artifacts, repeatable evaluations, deployment control, and trace inspection. Whether those live in one product or several matters less than whether you can reproduce and compare a release.

Why is a passing offline evaluation not enough?

Production traffic contains phrasing, documents, and edge cases your dataset may not include. Providers can also update models without any change on your side. Monitoring after deployment catches what the offline gate missed.

Putting LLMOps to work

Start with a release record that names the model, prompt, index, tools, guardrails, and evaluation dataset version. Keep it with every deployment.

Set a threshold on a small regression dataset built from real failures, and block releases that fall below it. Review production traces regularly and add new failures to that dataset.

Track cost, latency, and task success side by side. Arize AX can capture the traces, run evaluations, and connect production failures back to datasets. For platform detail, see the Arize AX documentation.