Key takeaways
- AI engineering is application work on foundation models. The model is a component.
- The failure signal is task success, not HTTP 200 and not offline model accuracy.
- Engineers own traces, the harness, and the change. PMs own the rubric, the eval budget, and the ship/kill bar.
- Agents are a subset: more steps, more tools, more ways to fail a session while every individual call looks fine.
- Observability, evaluation, and an improvement loop have to work together. One or two is not enough.
AI engineering is the practice of building and operating products on top of foundation models. The model is one component in a larger system that includes prompts, context, retrieval, tools, agents, evaluation, and production infrastructure. For a practical category map, compare the agent engineering tool stack across building, orchestration, evaluation, observability, and deployment.
AI engineering is the practice of shipping products that use foundation models: specifying the task, assembling context and tools, measuring whether the system actually did the job, and changing the system until it holds up in production.
In this sense, AI engineering focuses on the system around the model: prompts, retrieval, tools, agents, evaluation, and production reliability. Chip Huyen draws a similar distinction in AI Engineering (O’Reilly), describing AI engineering as building applications on readily available models rather than training foundation models in-house.
Carnegie Mellon’s Software Engineering Institute uses “AI engineering” more broadly for engineering AI-enabled systems of many kinds. This page uses the application and foundation-model sense common in LLM and agent development.
What counts as AI engineering?
| This is AI engineering | This is not |
|---|---|
| Composing prompts, retrieval, tools, and a harness around a model you did not train | Foundation-model research or training a foundation model from scratch |
| Measuring task success with traces, code evals, and judges | Prompting in a playground with no traces and no evals |
| Shipping a change to context, tools, or the loop, then re-measuring | Classical ML engineering: features, training, serving a model you own |
| A product that uses a model to complete a job | “AI for engineering” in the mechanical, civil, or CAD sense |
If you searched for AI applied to mechanical or civil engineering, this is a different discipline.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
AI engineering vs machine learning engineering (and software, and agents)
Teams hire for the wrong skills when they treat these as the same job. The cleanest split is by deliverable, failure signal, and typical work.
| Discipline | Deliverable | Failure signal | Typical work |
|---|---|---|---|
| Software engineering | A deterministic product | Bugs, latency, uptime | Services, APIs, UX |
| ML engineering | A trained model | Offline metrics, drift | Data, features, training, serving |
| AI engineering | A product that uses a model | Task success, traces, evals, cost | Context, tools, retrieval, agents, eval harness |
| Agent engineering (often a subset of AI engineering) | A multi-step actor | Task, trajectory, tool, or session failure | Harness, tools, memory, control plane |
An AI engineer vs a software engineer is not “who writes Python.” Both write production code. The AI engineer also has to make a non-deterministic component dependable: specify the task, see what the model actually did, and change the system when “it answered” is not the same as “it completed the job.”
Agent engineering is AI engineering with a loop, tools, and state. If you are building that layer, start with what an agent harness is rather than treating the model as the product.
What an AI engineer actually does
Day to day, AI engineering often looks more like software engineering than model research. The difference is that you are debugging both deterministic application code and non-deterministic model behavior, often through traces that capture prompts, retrieval, tool calls, and agent decisions.
The loop looks like this:
- Spec the task. Define what success means in terms you can observe and score. For example: “Book the requested ride to SFO without changing the destination based on unrelated background speech.” Avoid criteria like “the agent was helpful” unless you have a rubric that makes them measurable.
- Instrument the system. Capture the prompt, retrieved context, tool calls, intermediate decisions, and final outcome. For multi-step agents, you usually need the trajectory rather than the final response alone.
- Read failures. Group failures by what went wrong in the task, not only by infrastructure errors. A trace can execute successfully while the agent still chooses the wrong tool, follows the wrong instruction, or fails to complete the user’s job.
- Write an eval. Use deterministic checks when success can be expressed in code. Use an LLM judge when the criterion requires semantic interpretation, then validate the judge against examples you trust. The how-to lives in LLM evaluation and agent evals from traces.
- Change the smallest component supported by the evidence. That might be the prompt, retrieval strategy, tool schema, permissions, harness, routing logic, or model. A model change makes sense when the evidence points to model capability as the bottleneck.
- Re-measure before rollout. Run the eval against the same failure class and a broader regression set. If the change improves the target behavior without creating unacceptable regressions, it is a candidate for rollout and production monitoring.
The important shift is from prompting to engineering under measurement. Prototyping may start in a playground, but production AI engineering requires a loop for observing behavior, evaluating outcomes, changing the system, and checking whether the change worked.
What the PM owns
PMs do not need to train models. But they do need to operate a non-deterministic product.
The exact ownership model varies by team, but a useful split is for engineering to own instrumentation and system changes while product owns the definition of success and the product risk of getting it wrong. Both should participate in failure review and release decisions.
One workable RACI looks like this:
| Owns | Engineer | PM | Both |
|---|---|---|---|
| Tracing, instrumentation, harness changes | R | C | |
| Task-success definition and rubric | C | R | |
| Eval budget (what to score, how often, human vs code vs judge) | C | R | |
| Human-review policy (what must a person see) | C | R | |
| Cost per successful task | C | R | |
| Weekly failure review | R | ||
| Ship / kill bar | C | A |
R = responsible. A = accountable. C = consulted.
Five things should exist in writing before you scale traffic:
- Task-success definition. The user job in one sentence, plus the failure modes that count even when the model replies fluently.
- Eval budget. Which checks are code, which are judges, which are human, and what share of production you can afford to score.
- Human-review policy. What must a person see (refunds, medical, legal, irreversible actions) versus what can ride on an eval.
- Cost per successful task. Token usage is useful for understanding cost, but it does not tell you whether the system accomplished anything. Track the cost of successful outcomes so expensive failures are visible too.
- Ship / kill bar. The score, regression, and cost thresholds that mean you promote a change, or you roll it back.
Those are operating artifacts, not a job description. For the career version of this work, see the AI product manager role.
A production miss: the agent booked pizza instead of the airport
A useful test of whether you are doing AI engineering: can you point to a failure that looked fine in logs and was wrong on the task?
Uber’s voice agent had one. A rider asked to book a trip to San Francisco International Airport. In the background, a child said, “I want pizza too.” The agent heard both. Instead of treating the pizza line as ambient conversation, it started rerouting toward a restaurant. Offline evaluations had not caught it. HTTP-level success would not have caught it either. The booking path ran. The task failed.
That is the shape of AI engineering work in production:
- Spec: get the rider to the airport they asked for. Ignore background speech that is not a booking change.
- Trace: the agent treated a second utterance as a new destination.
- Eval gap: the offline suite did not include overlapping speech, so the miss never became a failing test.
- Change: follow the traces, tighten how the agent handles requests unrelated to booking, add the case to a living dataset.
- Re-measure: the failure class is now a regression test, not a story someone tells later.
The full account is in how Uber evaluates AI agents at production scale. The broader lesson is that infrastructure success and task success are different signals. For a multi-step agent, evaluating the final response alone may miss the behavior that caused the failure, which is why traces and session-level context matter.
If you only remember one thing from this page: an agent can be green in your APM and still fail the user. AI engineering is the discipline that treats that as the primary bug.
The stack (application, evaluation, observability)
Application. Prompts, tools, retrieval, permissions, and the agent harness when the system takes multiple steps. This is the product.
Evaluation. Code checks, LLM-as-judge, and human review, scored against the task definition. Offline on datasets, online on production samples. See LLM evaluation and the agent evaluation handbook.
Observability. Traces and sessions so you can see what happened, cluster failures, and watch the score after you ship. For agents, that is session-level, not request-level only.
Together, those three layers create the core feedback loop for AI engineering: observe what happened, evaluate whether it worked, change the application, and measure the result.
That is why prototype and production are the same discipline at different scale, not different jobs.
- Local loop: run traces and evals on your laptop while you change the prompt, the tool, or the harness. Phoenix is the open-source path for that.
- Production loop: the same traces and evals, on live traffic, with datasets that update when new failures show up. Arize AX is the production path for that.
Arize organizes its AI engineering platform around observability, evaluation, and improvement because the value comes from connecting them. Traces help identify what happened, evals measure whether it worked, and experiments and production monitoring show whether a change actually improved the system. Phoenix supports this loop in open source, while Arize AX extends it to production workflows.
Why this work exists now
Foundation models dramatically lowered the cost of prototyping AI applications. Production is harder because model behavior is probabilistic, application behavior depends on changing context and tools, and the underlying model is often outside your control.
Software engineering brings tests and observability. ML engineering brings measurement of models and data. AI engineering extends those practices to systems where prompts, retrieval, tools, model behavior, and orchestration all affect whether the user’s task succeeds. Traces and evals make those failures measurable enough to debug and improve.
If your team is still arguing whether this is “just software” or “just ML,” use the table above. If the artifact you ship is a product that uses a model, and the bug you fear is a fluent wrong action, you are in this discipline.
FAQ
Is AI engineering the same as machine learning engineering?
No. Machine learning engineering focuses on building and training models. AI engineering builds applications on top of existing models and focuses on context, tools, retrieval, orchestration, evaluation, and reliability in production.
Do you need to train models to do AI engineering?
Usually not. Most AI engineering uses existing foundation models and concentrates on the system around them: prompts, retrieval, tools, evals, and the improvement loop.
What skills does AI engineering require?
A mix of software engineering and applied evaluation: building and instrumenting systems, designing prompts and retrieval, writing and calibrating evals, reading traces, and shipping changes under measurement.
How is AI engineering different for agents than for simple LLM apps?
Agents add steps, tools, and decisions, which multiplies the ways a system can fail. AI engineering for agents depends more heavily on tracing and session-level evaluation than single-prompt applications do.
What tools support AI engineering?
The core stack is observability, evaluation, and an improvement loop. In practice that means tracing infrastructure, an evaluation harness, and a way to compare versions and watch production behavior over time.
Is LLM engineering the same thing?
The terms overlap heavily, and teams use them differently. LLM engineering usually refers specifically to applications built with large language models, including prompting, context management, retrieval, tools, and evaluation. AI engineering is a broader label that can include LLM applications, multimodal systems, agents, and other applications built on foundation models.
Is this a job title or a practice?
A practice first. Titles lag. Some teams hire “AI engineers.” Some fold the work into software engineering or ML engineering and wonder why the model-training interview loop does not predict production quality. Hire for the loop above. Name the role however your org already works.
What should a PM measure in the first 90 days?
Task success on a defined job, cost per successful task, coverage of the eval (what share of production is scored, and by what), human-review load, and time from a production miss to a regression test. If you cannot measure those areas yet, you probably do not have the feedback loop needed to operate the product reliably at scale.
