Restate

Restate is a durable execution platform that makes AI agents and workflows resilient and resumable. It handles retries, recovery, orchestration, agent-to-agent communication, human-in-the-loop approvals, and task control out of the box. Restate spans and OpenInference spans come from two different processes. The restate-server process exports the execution journal — ingress, invocation start, each attempt, invocation end. Your service process exports the agent’s LLM and tool spans. Wrapping your tracer provider in Restate’s RestateTracerProvider attaches the agent spans to the invocation attempt, so once both processes export to Arize AX you get a single trace covering the durable workflow and the agent that runs inside it.

Both exporters are required. If only the service process exports to Arize AX, RestateTracerProvider still parents the agent spans to a Restate span that never arrives, and the trace lands as orphaned spans with no root. Configure restate-server as well, as shown in Export the Restate journal.

Prerequisites

Python 3.10+ (required by restate-sdk)
Restate Server and CLI 1.7+
An Arize AX account (sign up)
Your Arize AX Space ID and API Key
An OPENAI_API_KEY from the OpenAI Platform

Launch Arize AX

Sign in to your Arize AX account.
From Space Settings, copy your Space ID and API Key. You will set them as ARIZE_SPACE_ID and ARIZE_API_KEY below.

Install

This example uses the OpenAI Agents SDK. Any framework with an OpenInference instrumentor works the same way.

pip install "restate-sdk[openai,tracing]" \
  arize-otel \
  openinference-instrumentation-openai-agents \
  hypercorn

The tracing extra installs the OpenTelemetry API that RestateTracerProvider needs, and the openai extra installs the Agents SDK plus Restate’s integration for it. Installing restate-sdk on its own gives you an ImportError on restate.ext.tracing.

Configure credentials

export ARIZE_SPACE_ID="<your-space-id>"
export ARIZE_API_KEY="<your-api-key>"
export ARIZE_PROJECT_NAME="restate-tracing-example"
export OPENAI_API_KEY="<your-openai-api-key>"

Setup tracing

Wrap the tracer provider returned by register in RestateTracerProvider, then hand that to the instrumentor:

# instrumentation.py
import os

from arize.otel import register
from openinference.instrumentation.openai_agents import (
    OpenAIAgentsInstrumentor,
)
from restate.ext.tracing import RestateTracerProvider

tracer_provider = register(
    space_id=os.environ["ARIZE_SPACE_ID"],
    api_key=os.environ["ARIZE_API_KEY"],
    project_name=os.environ["ARIZE_PROJECT_NAME"],
)

OpenAIAgentsInstrumentor().instrument(
    tracer_provider=RestateTracerProvider(tracer_provider)
)
print("Arize AX tracing initialized for Restate.")

Export the Restate journal

restate-server exports its own spans, so point it at Arize AX too. Arize AX authenticates with the authorization and arize-space-id headers, and reads the project from the openinference.project.name resource attribute. Set all three in the server’s environment:

export RESTATE_TRACING_HEADERS__AUTHORIZATION="$ARIZE_API_KEY"
export RESTATE_TRACING_HEADERS__ARIZE_SPACE_ID="$ARIZE_SPACE_ID"
export OTEL_RESOURCE_ATTRIBUTES="openinference.project.name=$ARIZE_PROJECT_NAME"

Each underscore in the part after RESTATE_TRACING_HEADERS__ becomes a hyphen in the header name, so RESTATE_TRACING_HEADERS__ARIZE_SPACE_ID sends arize-space-id. Naming it RESTATE_TRACING_HEADERS__SPACE_ID instead sends space-id, which Arize AX rejects with HTTP 403. Setting the project matters because restate-server has no flag for it. Reusing $ARIZE_PROJECT_NAME keeps both halves of the trace in one project; give the server a different value and the journal spans and agent spans land in separate projects. Restate also accepts these headers in a config file under a [tracing-headers] table, but the environment form keeps your API key out of a file on disk.

For EU spaces both exporters have to move, or half the trace goes to the wrong region and you get the orphaned spans described above. Pass endpoint=Endpoint.ARIZE_EUROPE (from arize.otel) to register in instrumentation.py, and start the server with --tracing-endpoint otlp+https://otlp.eu-west-1a.arize.com/v1/traces.

Run Restate

Define the agent as a Restate service. durable_function_tool makes each tool call recoverable, and run_typed records the API call in the journal so a retry replays it instead of repeating it:

# example.py

# Importing instrumentation first ensures tracing is set up
# before `agents` is imported.
import instrumentation  # noqa: F401

import restate
from agents import Agent
from restate.ext.openai import (
    DurableRunner,
    durable_function_tool,
    restate_context,
)


@durable_function_tool
async def get_weather(city: str) -> dict:
    """Get the current weather for a given city."""

    async def call_weather_api(city: str) -> dict:
        return {"temp_c": 23, "description": "Sunny and warm."}

    return await restate_context().run_typed(
        "Get weather", call_weather_api, city=city
    )


weather_agent = Agent(
    name="WeatherAgent",
    instructions="You are a helpful agent that provides weather updates.",
    model="gpt-5.4-mini",
    tools=[get_weather],
)

agent_service = restate.Service("agent")


@agent_service.handler()
async def run(_ctx: restate.Context, message: str) -> str:
    result = await DurableRunner.run(weather_agent, message)
    return result.final_output


app = restate.app(services=[agent_service])

Serve the app, start the server, register the service, then send a request. Use a separate terminal for each of the first two commands:

hypercorn --bind 0.0.0.0:9080 example:app

restate-server \
  --tracing-endpoint otlp+https://otlp.arize.com/v1/traces

restate deployments register http://localhost:9080

curl -X POST http://localhost:8080/agent/run \
  -H 'content-type: application/json' \
  -d '"What is the weather in Paris?"'

Expected output

The ingress returns the handler’s JSON-encoded reply, so the degree sign arrives escaped:

"Paris: 23\u00b0C, sunny and warm."

Verify in Arize AX

Open your Arize AX space and select project restate-tracing-example.
Open the newest trace. It is rooted at the Restate ingress span, with the agent’s spans under the invocation attempt:

ingress agent/run
  invocation-start agent/run
    invocation-attempt agent/run
      Agent workflow      (agent span)
      Agent workflow      (chain span)
      WeatherAgent        (agent span)
      turn                (chain span)
      response            (LLM span)
      get_weather         (tool span)
      turn                (chain span)
      response            (LLM span)
    invocation-end agent/run

RestateTracerProvider flattens the agent spans into siblings under the invocation attempt rather than preserving the agent framework’s own nesting, so the LLM and tool spans sit at the same depth. Restate has said a future release will preserve the original hierarchy.

Check from the skill, CLI, or SDK

Confirm spans are actually reaching your Arize AX project. Use whichever fits your workflow — the skill and CLI work for any framework; the SDK check is shown for each language.

Arize skill (agent)
AX CLI
SDK

Install the Arize Skills plugin and let your coding agent check for you:

npx skills add Arize-ai/arize-skills

Then prompt your agent:

Use the arize-trace skill to export and analyze recent traces from my project. Confirm spans are arriving, and summarize any errors or latency issues.

Export recent spans for your project — any rows mean traces are landing:

ax spans export "$ARIZE_PROJECT_NAME" --space "$ARIZE_SPACE_ID" \
  --limit 5 --stdout | jq 'length'

A non-zero count confirms spans reached Arize AX. Run ax auth login first if you have not authenticated. See the ax spans reference.

Query the project’s spans and check that at least one came back.

import os
from arize import ArizeClient

client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
resp = client.spans.list(
    project=os.environ["ARIZE_PROJECT_NAME"],
    space=os.environ["ARIZE_SPACE_ID"],
    limit=5,
)
count = len(resp.spans)
print(
    f"{count} span(s) found" if count else "No spans yet — recheck setup"
)

// Reads ARIZE_API_KEY from the environment.
import { listSpans } from "@arizeai/ax-client";

const { data: spans } = await listSpans({
  project: process.env.ARIZE_PROJECT_NAME!,
  space: process.env.ARIZE_SPACE_ID!,
  limit: 5,
});
const count = spans.length;
console.log(
  count ? `${count} span(s) found` : "No spans yet — recheck setup",
);

client, err := arize.NewClient(
    arize.Config{APIKey: os.Getenv("ARIZE_API_KEY")},
)
if err != nil {
    log.Fatal(err)
}
resp, err := client.Spans.List(ctx, spans.ListRequest{
    Project: os.Getenv("ARIZE_PROJECT_NAME"),
    Space:   os.Getenv("ARIZE_SPACE_ID"),
    Limit:   5,
})
if err != nil {
    log.Fatal(err)
}
fmt.Printf("%d span(s) found\n", len(resp.Spans))

SDK span references: Python · TypeScript · Go.

What Arize AX captures

Restate journal spans for the ingress request, invocation start, every invocation attempt, and invocation end, each carrying restate.invocation.id and restate.invocation.target
OpenInference agent, chain, LLM, and tool spans for the agent that runs inside the attempt
Retries as additional invocation attempt spans in the same trace, so a recovered run shows every attempt rather than only the one that succeeded

Troubleshooting

Spans appear with no root, or the journal spans are missing. restate-server is not exporting. Check its logs for BatchSpanProcessor.Flush.ExportError; a Status(403, ...) means the authorization or arize-space-id header is wrong. The most common cause is naming the variable RESTATE_TRACING_HEADERS__SPACE_ID, which sends space-id rather than arize-space-id.
Journal spans and agent spans are in different projects. OTEL_RESOURCE_ATTRIBUTES was not set for the restate-server process, or it names a different project than ARIZE_PROJECT_NAME.
ImportError on restate.ext.tracing. Install the extras: pip install "restate-sdk[openai,tracing]".
No traces at all. Confirm instrumentation is imported before agents in your service module, and that the service was registered with restate deployments register.

OpenTelemetry

LLM Providers

Agent Frameworks

Coding Agents

Platforms

Orchestration

Evaluation

Prerequisites

Launch Arize AX

Install

Configure credentials

Setup tracing

Export the Restate journal

Run Restate

Expected output

Verify in Arize AX

Check from the skill, CLI, or SDK

What Arize AX captures

Troubleshooting

Resources

Restate AI documentation

Restate server tracing

OpenInference OpenAI Agents Instrumentor

OpenTelemetry context propagation

​Prerequisites

​Launch Arize AX

​Install

​Configure credentials

​Setup tracing

​Export the Restate journal

​Run Restate

​Expected output

​Verify in Arize AX

​Check from the skill, CLI, or SDK

​What Arize AX captures

​Troubleshooting

​Resources

Restate AI documentation

Restate server tracing

OpenInference OpenAI Agents Instrumentor

OpenTelemetry context propagation

Prerequisites

Launch Arize AX

Install

Configure credentials

Setup tracing

Export the Restate journal

Run Restate

Expected output

Verify in Arize AX

Check from the skill, CLI, or SDK

What Arize AX captures

Troubleshooting

Resources