> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# NVIDIA NeMo Agent Toolkit

> Trace NVIDIA NeMo Agent Toolkit workflows and send spans to Arize AX with the built-in arize_ax exporter.

[NVIDIA NeMo Agent Toolkit](https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html) is NVIDIA's open-source toolkit for building, running, and observing agent workflows. The toolkit ships with a built-in `arize_ax` OpenTelemetry exporter, so traces — including LLM calls, tool invocations, and full input/output payloads — flow into Arize AX without any extra instrumentation code.

<Note>
  This integration is driven entirely by a YAML configuration file in the NeMo Agent Toolkit repository. There is no standalone Python script to run; you launch your workflow with the toolkit's `nat` CLI and the toolkit handles exporter setup.
</Note>

## Prerequisites

* Python 3.11+
* [`uv`](https://docs.astral.sh/uv/getting-started/installation/) installed
* An Arize AX account ([sign up](https://arize.com/sign-up/))
* The [NeMo Agent Toolkit repository](https://github.com/NVIDIA/NeMo-Agent-Toolkit) cloned locally

## Launch Arize AX

1. Sign in to your [Arize AX account](https://app.arize.com/).
2. From **Space Settings**, copy your **Space ID** and **API Key**. You will set them as `ARIZE_SPACE_ID` and `ARIZE_API_KEY` below.

## Install

Install the OpenTelemetry extra to enable the `arize_ax` exporter:

```bash theme={null}
uv pip install "nvidia-nat[opentelemetry]"
```

## Configure credentials

```bash theme={null}
export ARIZE_SPACE_ID="<your-space-id>"
export ARIZE_API_KEY="<your-api-key>"
export ARIZE_PROJECT_NAME="simple_calculator"
```

## Setup tracing

Add the `arize_ax` exporter to your workflow's YAML configuration file:

```yaml theme={null}
general:
  telemetry:
    tracing:
      arize_ax:
        _type: arize_ax
        space_id: ${ARIZE_SPACE_ID}
        api_key: ${ARIZE_API_KEY}
        project: ${ARIZE_PROJECT_NAME:-simple_calculator}
        # Optional: enable for EU data residency
        # use_eu_region: true
```

This sends OTLP-formatted traces to Arize AX, grouped into the project you specify.

## Run NeMo Agent Toolkit

From the root of the NeMo Agent Toolkit repository, install the example workflow and run it with your Arize AX configuration:

```bash theme={null}
uv pip install -e examples/observability/simple_calculator_observability/

nat run \
  --config_file examples/observability/simple_calculator_observability/configs/config-arize-ax.yml \
  --input "What is 2 * 4?"
```

### Expected output

```text wrap theme={null}
Started exporter 'arize_ax'
The result of 2 * 4 is 8.
```

## Verify in Arize AX

1. Open your Arize AX space and select project **`simple_calculator`** (or whatever value you set for `ARIZE_PROJECT_NAME`).
2. You should see a new trace within \~30 seconds containing the workflow's root span and nested LLM / tool spans with prompts, responses, and token usage attached.
3. If no traces appear, see [Troubleshooting](#troubleshooting).

### Check from the skill, CLI, or SDK

Confirm spans are actually reaching your Arize AX project. Use whichever fits your workflow — the skill and CLI work for any framework; the SDK check is shown for each language.

<Tabs>
  <Tab title="Arize skill (agent)">
    Install the [Arize Skills](https://github.com/Arize-ai/arize-skills) plugin and let your coding agent check for you:

    ```bash theme={null}
    npx skills add Arize-ai/arize-skills
    ```

    Then prompt your agent:

    > Use the `arize-trace` skill to export and analyze recent traces from my project. Confirm spans are arriving, and summarize any errors or latency issues.
  </Tab>

  <Tab title="AX CLI">
    Export recent spans for your project — any rows mean traces are landing:

    ```bash theme={null}
    ax spans export "$ARIZE_PROJECT_NAME" --space "$ARIZE_SPACE_ID" \
      --limit 5 --stdout | jq 'length'
    ```

    A non-zero count confirms spans reached Arize AX. Run `ax auth login` first if you have not authenticated. See the [`ax spans` reference](/docs/api-clients/cli/spans).
  </Tab>

  <Tab title="SDK">
    Query the project's spans and check that at least one came back.

    <CodeGroup>
      ```python Python theme={null}
      import os
      from arize import ArizeClient

      client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
      resp = client.spans.list(
          project=os.environ["ARIZE_PROJECT_NAME"],
          space=os.environ["ARIZE_SPACE_ID"],
          limit=5,
      )
      count = len(resp.spans)
      print(
          f"{count} span(s) found" if count else "No spans yet — recheck setup"
      )
      ```

      ```typescript TypeScript theme={null}
      // Reads ARIZE_API_KEY from the environment.
      import { listSpans } from "@arizeai/ax-client";

      const { data: spans } = await listSpans({
        project: process.env.ARIZE_PROJECT_NAME!,
        space: process.env.ARIZE_SPACE_ID!,
        limit: 5,
      });
      const count = spans.length;
      console.log(
        count ? `${count} span(s) found` : "No spans yet — recheck setup",
      );
      ```

      ```go Go theme={null}
      client, err := arize.NewClient(
          arize.Config{APIKey: os.Getenv("ARIZE_API_KEY")},
      )
      if err != nil {
          log.Fatal(err)
      }
      resp, err := client.Spans.List(ctx, spans.ListRequest{
          Project: os.Getenv("ARIZE_PROJECT_NAME"),
          Space:   os.Getenv("ARIZE_SPACE_ID"),
          Limit:   5,
      })
      if err != nil {
          log.Fatal(err)
      }
      fmt.Printf("%d span(s) found\n", len(resp.Spans))
      ```
    </CodeGroup>

    SDK span references: [Python](/docs/api-clients/python/version-8/client-resources/spans) · [TypeScript](/docs/api-clients/typescript/version-1/client-resources/spans) · [Go](/docs/api-clients/go/version-2/client-resources/spans).
  </Tab>
</Tabs>

## Troubleshooting

* **No `Started exporter 'arize_ax'` log line.** The OpenTelemetry extra isn't installed in the active environment. Re-run `uv pip install "nvidia-nat[opentelemetry]"` and confirm the YAML `_type` is exactly `arize_ax`.
* **No traces in Arize AX.** Confirm `ARIZE_SPACE_ID` and `ARIZE_API_KEY` are exported in the same shell that runs `nat run`. The YAML uses `${ARIZE_SPACE_ID}` / `${ARIZE_API_KEY}` substitution, so unset variables resolve to empty strings and the exporter silently fails to authenticate.
* **EU data residency.** Set `use_eu_region: true` under `general.telemetry.tracing.arize_ax` to route traces to the EU ingest endpoint.
* **Wrong project in Arize AX.** The YAML `project:` value wins over `ARIZE_PROJECT_NAME` unless you use the `${ARIZE_PROJECT_NAME:-default}` substitution shown above.

## Resources

<CardGroup>
  <Card icon="github" href="https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/develop/docs/source/run-workflows/observe/observe-workflow-with-arize-ax.md" title="Observe NeMo Agent Toolkit with Arize AX" horizontal />

  <Card icon="book-open" href="https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html" title="NeMo Agent Toolkit Documentation" horizontal />

  <Card icon="globe" href="https://github.com/NVIDIA/NeMo-Agent-Toolkit" title="NeMo Agent Toolkit GitHub" horizontal />
</CardGroup>