> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Restate

> Trace durable Restate agent executions in Arize AX with Restate's tracer provider and OpenInference.

[Restate](https://restate.dev/) is a durable execution platform that makes AI agents and workflows resilient and resumable. It handles retries, recovery, orchestration, agent-to-agent communication, human-in-the-loop approvals, and task control out of the box.

Restate spans and OpenInference spans come from two different processes. The `restate-server` process exports the execution journal — ingress, invocation start, each attempt, invocation end. Your service process exports the agent's LLM and tool spans. Wrapping your tracer provider in Restate's `RestateTracerProvider` attaches the agent spans to the invocation attempt, so once both processes export to Arize AX you get a single trace covering the durable workflow and the agent that runs inside it.

<Note>
  Both exporters are required. If only the service process exports to Arize AX, `RestateTracerProvider` still parents the agent spans to a Restate span that never arrives, and the trace lands as orphaned spans with no root. Configure `restate-server` as well, as shown in [Export the Restate journal](#export-the-restate-journal).
</Note>

## Prerequisites

* Python 3.10+ (required by `restate-sdk`)
* [Restate Server and CLI](https://docs.restate.dev/get-started/quickstart) 1.7+
* An Arize AX account ([sign up](https://arize.com/sign-up/))
* Your Arize AX **Space ID** and **API Key**
* An `OPENAI_API_KEY` from the [OpenAI Platform](https://platform.openai.com/api-keys)

## Launch Arize AX

1. Sign in to your [Arize AX account](https://app.arize.com/).
2. From **Space Settings**, copy your **Space ID** and **API Key**. You will set them as `ARIZE_SPACE_ID` and `ARIZE_API_KEY` below.

## Install

This example uses the [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/). Any framework with an [OpenInference instrumentor](https://github.com/Arize-ai/openinference/tree/main) works the same way.

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
pip install "restate-sdk[openai,tracing]" \
  arize-otel \
  openinference-instrumentation-openai-agents \
  hypercorn
```

<Note>
  The `tracing` extra installs the OpenTelemetry API that `RestateTracerProvider` needs, and the `openai` extra installs the Agents SDK plus Restate's integration for it. Installing `restate-sdk` on its own gives you an `ImportError` on `restate.ext.tracing`.
</Note>

## Configure credentials

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
export ARIZE_SPACE_ID="<your-space-id>"
export ARIZE_API_KEY="<your-api-key>"
export ARIZE_PROJECT_NAME="restate-tracing-example"
export OPENAI_API_KEY="<your-openai-api-key>"
```

## Setup tracing

Wrap the tracer provider returned by `register` in `RestateTracerProvider`, then hand that to the instrumentor:

```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
# instrumentation.py
import os

from arize.otel import register
from openinference.instrumentation.openai_agents import (
    OpenAIAgentsInstrumentor,
)
from restate.ext.tracing import RestateTracerProvider

tracer_provider = register(
    space_id=os.environ["ARIZE_SPACE_ID"],
    api_key=os.environ["ARIZE_API_KEY"],
    project_name=os.environ["ARIZE_PROJECT_NAME"],
)

OpenAIAgentsInstrumentor().instrument(
    tracer_provider=RestateTracerProvider(tracer_provider)
)
print("Arize AX tracing initialized for Restate.")
```

## Export the Restate journal

`restate-server` exports its own spans, so point it at Arize AX too. Arize AX authenticates with the `authorization` and `arize-space-id` headers, and reads the project from the `openinference.project.name` resource attribute. Set all three in the server's environment:

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
export RESTATE_TRACING_HEADERS__AUTHORIZATION="$ARIZE_API_KEY"
export RESTATE_TRACING_HEADERS__ARIZE_SPACE_ID="$ARIZE_SPACE_ID"
export OTEL_RESOURCE_ATTRIBUTES="openinference.project.name=$ARIZE_PROJECT_NAME"
```

Each underscore in the part after `RESTATE_TRACING_HEADERS__` becomes a hyphen in the header name, so `RESTATE_TRACING_HEADERS__ARIZE_SPACE_ID` sends `arize-space-id`. Naming it `RESTATE_TRACING_HEADERS__SPACE_ID` instead sends `space-id`, which Arize AX rejects with HTTP 403.

Setting the project matters because `restate-server` has no flag for it. Reusing `$ARIZE_PROJECT_NAME` keeps both halves of the trace in one project; give the server a different value and the journal spans and agent spans land in separate projects.

Restate also accepts these headers in a [config file](https://docs.restate.dev/server/configuration) under a `[tracing-headers]` table, but the environment form keeps your API key out of a file on disk.

<Note>
  For EU spaces both exporters have to move, or half the trace goes to the wrong region and you get the orphaned spans described above. Pass `endpoint=Endpoint.ARIZE_EUROPE` (from `arize.otel`) to `register` in `instrumentation.py`, and start the server with `--tracing-endpoint otlp+https://otlp.eu-west-1a.arize.com/v1/traces`.
</Note>

## Run Restate

Define the agent as a Restate service. `durable_function_tool` makes each tool call recoverable, and `run_typed` records the API call in the journal so a retry replays it instead of repeating it:

```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
# example.py

# Importing instrumentation first ensures tracing is set up
# before `agents` is imported.
import instrumentation  # noqa: F401

import restate
from agents import Agent
from restate.ext.openai import (
    DurableRunner,
    durable_function_tool,
    restate_context,
)


@durable_function_tool
async def get_weather(city: str) -> dict:
    """Get the current weather for a given city."""

    async def call_weather_api(city: str) -> dict:
        return {"temp_c": 23, "description": "Sunny and warm."}

    return await restate_context().run_typed(
        "Get weather", call_weather_api, city=city
    )


weather_agent = Agent(
    name="WeatherAgent",
    instructions="You are a helpful agent that provides weather updates.",
    model="gpt-5.4-mini",
    tools=[get_weather],
)

agent_service = restate.Service("agent")


@agent_service.handler()
async def run(_ctx: restate.Context, message: str) -> str:
    result = await DurableRunner.run(weather_agent, message)
    return result.final_output


app = restate.app(services=[agent_service])
```

Serve the app, start the server, register the service, then send a request. Use a separate terminal for each of the first two commands:

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
hypercorn --bind 0.0.0.0:9080 example:app
```

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
restate-server \
  --tracing-endpoint otlp+https://otlp.arize.com/v1/traces
```

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
restate deployments register http://localhost:9080

curl -X POST http://localhost:8080/agent/run \
  -H 'content-type: application/json' \
  -d '"What is the weather in Paris?"'
```

### Expected output

The ingress returns the handler's JSON-encoded reply, so the degree sign arrives
escaped:

```text wrap theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
"Paris: 23\u00b0C, sunny and warm."
```

## Verify in Arize AX

1. Open your Arize AX space and select project **`restate-tracing-example`**.
2. Open the newest trace. It is rooted at the Restate ingress span, with the agent's spans under the invocation attempt:

```text wrap theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
ingress agent/run
  invocation-start agent/run
    invocation-attempt agent/run
      Agent workflow      (agent span)
      Agent workflow      (chain span)
      WeatherAgent        (agent span)
      turn                (chain span)
      response            (LLM span)
      get_weather         (tool span)
      turn                (chain span)
      response            (LLM span)
    invocation-end agent/run
```

<Note>
  `RestateTracerProvider` flattens the agent spans into siblings under the invocation attempt rather than preserving the agent framework's own nesting, so the LLM and tool spans sit at the same depth. Restate has said a future release will preserve the original hierarchy.
</Note>

### Check from the skill, CLI, or SDK

Confirm spans are actually reaching your Arize AX project. Use whichever fits your workflow — the skill and CLI work for any framework; the SDK check is shown for each language.

<Tabs>
  <Tab title="Arize skill (agent)">
    Install the [Arize Skills](https://github.com/Arize-ai/arize-skills) plugin and let your coding agent check for you:

    ```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
    npx skills add Arize-ai/arize-skills
    ```

    Then prompt your agent:

    > Use the `arize-trace` skill to export and analyze recent traces from my project. Confirm spans are arriving, and summarize any errors or latency issues.
  </Tab>

  <Tab title="AX CLI">
    Export recent spans for your project — any rows mean traces are landing:

    ```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
    ax spans export "$ARIZE_PROJECT_NAME" --space "$ARIZE_SPACE_ID" \
      --limit 5 --stdout | jq 'length'
    ```

    A non-zero count confirms spans reached Arize AX. Run `ax auth login` first if you have not authenticated. See the [`ax spans` reference](/docs/api-clients/cli/spans).
  </Tab>

  <Tab title="SDK">
    Query the project's spans and check that at least one came back.

    <CodeGroup>
      ```python Python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
      import os
      from arize import ArizeClient

      client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
      resp = client.spans.list(
          project=os.environ["ARIZE_PROJECT_NAME"],
          space=os.environ["ARIZE_SPACE_ID"],
          limit=5,
      )
      count = len(resp.spans)
      print(
          f"{count} span(s) found" if count else "No spans yet — recheck setup"
      )
      ```

      ```typescript TypeScript theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
      // Reads ARIZE_API_KEY from the environment.
      import { listSpans } from "@arizeai/ax-client";

      const { data: spans } = await listSpans({
        project: process.env.ARIZE_PROJECT_NAME!,
        space: process.env.ARIZE_SPACE_ID!,
        limit: 5,
      });
      const count = spans.length;
      console.log(
        count ? `${count} span(s) found` : "No spans yet — recheck setup",
      );
      ```

      ```go Go theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
      client, err := arize.NewClient(
          arize.Config{APIKey: os.Getenv("ARIZE_API_KEY")},
      )
      if err != nil {
          log.Fatal(err)
      }
      resp, err := client.Spans.List(ctx, spans.ListRequest{
          Project: os.Getenv("ARIZE_PROJECT_NAME"),
          Space:   os.Getenv("ARIZE_SPACE_ID"),
          Limit:   5,
      })
      if err != nil {
          log.Fatal(err)
      }
      fmt.Printf("%d span(s) found\n", len(resp.Spans))
      ```
    </CodeGroup>

    SDK span references: [Python](/docs/api-clients/python/version-8/client-resources/spans) · [TypeScript](/docs/api-clients/typescript/version-1/client-resources/spans) · [Go](/docs/api-clients/go/version-2/client-resources/spans).
  </Tab>
</Tabs>

## What Arize AX captures

* Restate journal spans for the ingress request, invocation start, every invocation attempt, and invocation end, each carrying `restate.invocation.id` and `restate.invocation.target`
* OpenInference agent, chain, LLM, and tool spans for the agent that runs inside the attempt
* Retries as additional invocation attempt spans in the same trace, so a recovered run shows every attempt rather than only the one that succeeded

## Troubleshooting

* **Spans appear with no root, or the journal spans are missing.** `restate-server` is not exporting. Check its logs for `BatchSpanProcessor.Flush.ExportError`; a `Status(403, ...)` means the `authorization` or `arize-space-id` header is wrong. The most common cause is naming the variable `RESTATE_TRACING_HEADERS__SPACE_ID`, which sends `space-id` rather than `arize-space-id`.
* **Journal spans and agent spans are in different projects.** `OTEL_RESOURCE_ATTRIBUTES` was not set for the `restate-server` process, or it names a different project than `ARIZE_PROJECT_NAME`.
* **`ImportError` on `restate.ext.tracing`.** Install the extras: `pip install "restate-sdk[openai,tracing]"`.
* **No traces at all.** Confirm `instrumentation` is imported before `agents` in your service module, and that the service was registered with `restate deployments register`.

## Resources

<CardGroup>
  <Card icon="book-open" href="https://docs.restate.dev/ai" title="Restate AI documentation" horizontal />

  <Card icon="book-open" href="https://docs.restate.dev/server/monitoring/tracing" title="Restate server tracing" horizontal />

  <Card icon="terminal" href="https://github.com/Arize-ai/openinference/tree/main/python/instrumentation/openinference-instrumentation-openai-agents" title="OpenInference OpenAI Agents Instrumentor" horizontal />

  <Card icon="book-open" href="/docs/ax/concepts/otel-openinference/context-propagation" title="OpenTelemetry context propagation" horizontal />
</CardGroup>