> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# vLLM

> Trace vLLM inference in Arize AX with vLLM OpenTelemetry export and OpenInference OpenAI instrumentation.

[vLLM](https://docs.vllm.ai/) is an inference and serving engine for large language models. You can observe vLLM with Arize AX in two complementary ways:

* Export vLLM server spans over OTLP with `--otlp-traces-endpoint`.
* Trace your application calls to vLLM's OpenAI-compatible server with [`openinference-instrumentation-openai`](https://github.com/Arize-ai/openinference/tree/main/python/instrumentation/openinference-instrumentation-openai).

<Note>
  Use both paths when possible. vLLM's server-side OTLP spans show inference-server timing and request metadata. OpenInference OpenAI instrumentation captures the client-side prompt, response, token, model, and latency metadata for calls made through the OpenAI-compatible API.
</Note>

## Prerequisites

* Python 3.9+
* A running vLLM server
* An Arize AX account ([sign up](https://arize.com/sign-up/))
* Your Arize AX **Space ID** and **API Key**

## Launch Arize AX

1. Sign in to your [Arize AX account](https://app.arize.com/).
2. From **Space Settings**, copy your **Space ID** and **API Key**. You will set them as `ARIZE_SPACE_ID` and `ARIZE_API_KEY` below.

## Option 1: export vLLM server spans over OTLP

vLLM defaults to OTLP/gRPC. Arize's direct HTTP endpoint uses OTLP/HTTP with protobuf encoding, so set the exporter protocol before starting vLLM:

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
export OTEL_SERVICE_NAME="vllm-server"
export OTEL_RESOURCE_ATTRIBUTES="openinference.project.name=vllm-server"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="space_id=<your-arize-space-id>,api_key=<your-arize-api-key>"

vllm serve <model-name-or-path> \
  --otlp-traces-endpoint "https://otlp.arize.com/v1/traces"
```

For EU spaces, use:

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
export OTEL_SERVICE_NAME="vllm-server"
export OTEL_RESOURCE_ATTRIBUTES="openinference.project.name=vllm-server"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="space_id=<your-arize-space-id>,api_key=<your-arize-api-key>"

vllm serve <model-name-or-path> \
  --otlp-traces-endpoint "https://otlp.eu-west-1a.arize.com/v1/traces"
```

<Warning>
  Current vLLM releases bundle the core OpenTelemetry packages needed for server-side tracing. If you use an older or custom vLLM image and it errors on startup after you add `--otlp-traces-endpoint`, install matching OpenTelemetry exporter dependencies in the same environment or container image.
</Warning>

## Option 2: trace OpenAI-compatible client calls

Install the client-side packages:

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
pip install arize-otel openinference-instrumentation-openai openai
```

Configure credentials:

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
export ARIZE_SPACE_ID="<your-space-id>"
export ARIZE_API_KEY="<your-api-key>"
export ARIZE_PROJECT_NAME="vllm-tracing-example"
```

Set up tracing before importing and using `openai`:

```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
# instrumentation.py
import os

from arize.otel import register
from openinference.instrumentation.openai import OpenAIInstrumentor

tracer_provider = register(
    space_id=os.environ["ARIZE_SPACE_ID"],
    api_key=os.environ["ARIZE_API_KEY"],
    project_name=os.environ["ARIZE_PROJECT_NAME"],
)

OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)
print("Arize AX tracing initialized for vLLM OpenAI-compatible calls.")
```

Call the vLLM OpenAI-compatible endpoint:

```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
# example.py
from instrumentation import tracer_provider  # noqa: F401

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="<model-name-served-by-vllm>",
    messages=[
        {
            "role": "user",
            "content": "Name two signals that help debug inference latency.",
        }
    ],
)

print(response.choices[0].message.content)
```

## Verify in Arize

1. Open your Arize AX space and select project **`vllm-tracing-example`** for client-side spans.
2. If you also enabled vLLM server OTLP export directly, select project **`vllm-server`** if you set `OTEL_RESOURCE_ATTRIBUTES` as shown above.
3. You should see spans within \~30 seconds.

### Check from the skill, CLI, or SDK

Confirm spans are actually reaching your Arize AX project. Use whichever fits your workflow — the skill and CLI work for any framework; the SDK check is shown for each language.

<Tabs>
  <Tab title="Arize skill (agent)">
    Install the [Arize Skills](https://github.com/Arize-ai/arize-skills) plugin and let your coding agent check for you:

    ```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
    npx skills add Arize-ai/arize-skills
    ```

    Then prompt your agent:

    > Use the `arize-trace` skill to export and analyze recent traces from my project. Confirm spans are arriving, and summarize any errors or latency issues.
  </Tab>

  <Tab title="AX CLI">
    Export recent spans for your project — any rows mean traces are landing:

    ```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
    ax spans export "$ARIZE_PROJECT_NAME" --space "$ARIZE_SPACE_ID" \
      --limit 5 --stdout | jq 'length'
    ```

    A non-zero count confirms spans reached Arize AX. Run `ax auth login` first if you have not authenticated. See the [`ax spans` reference](/docs/api-clients/cli/spans).
  </Tab>

  <Tab title="SDK">
    Query the project's spans and check that at least one came back.

    <CodeGroup>
      ```python Python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
      import os
      from arize import ArizeClient

      client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
      resp = client.spans.list(
          project=os.environ["ARIZE_PROJECT_NAME"],
          space=os.environ["ARIZE_SPACE_ID"],
          limit=5,
      )
      count = len(resp.spans)
      print(
          f"{count} span(s) found" if count else "No spans yet — recheck setup"
      )
      ```

      ```typescript TypeScript theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
      // Reads ARIZE_API_KEY from the environment.
      import { listSpans } from "@arizeai/ax-client";

      const { data: spans } = await listSpans({
        project: process.env.ARIZE_PROJECT_NAME!,
        space: process.env.ARIZE_SPACE_ID!,
        limit: 5,
      });
      const count = spans.length;
      console.log(
        count ? `${count} span(s) found` : "No spans yet — recheck setup",
      );
      ```

      ```go Go theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
      client, err := arize.NewClient(
          arize.Config{APIKey: os.Getenv("ARIZE_API_KEY")},
      )
      if err != nil {
          log.Fatal(err)
      }
      resp, err := client.Spans.List(ctx, spans.ListRequest{
          Project: os.Getenv("ARIZE_PROJECT_NAME"),
          Space:   os.Getenv("ARIZE_SPACE_ID"),
          Limit:   5,
      })
      if err != nil {
          log.Fatal(err)
      }
      fmt.Printf("%d span(s) found\n", len(resp.Spans))
      ```
    </CodeGroup>

    SDK span references: [Python](/docs/api-clients/python/version-8/client-resources/spans) · [TypeScript](/docs/api-clients/typescript/version-1/client-resources/spans) · [Go](/docs/api-clients/go/version-2/client-resources/spans).
  </Tab>
</Tabs>

## What Arize captures

* **OpenInference client spans:** prompt, response, model, token usage, latency, errors, and tool call metadata supported by the OpenAI instrumentor.
* **vLLM server spans:** server-side inference and request-processing spans emitted by vLLM's OpenTelemetry support.

## Troubleshooting

* **No client-side LLM spans.** Make sure `OpenAIInstrumentor().instrument(...)` runs before your OpenAI client sends requests.
* **No vLLM server spans.** Confirm `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf`, `--otlp-traces-endpoint` points to Arize's OTLP/HTTP traces endpoint, and your OTLP headers contain `space_id` and `api_key`.
* **Server spans appear in the default project.** Confirm `OTEL_RESOURCE_ATTRIBUTES` includes `openinference.project.name=vllm-server`, or add the project name with an OpenTelemetry Collector resource processor.
* **Server spans lack prompt or response text.** Use the OpenInference OpenAI client path as well; vLLM's server export may not include the same application-level input and output detail.

## Resources

<CardGroup>
  <Card icon="book-open" href="https://docs.vllm.ai/en/latest/examples/observability/opentelemetry/" title="vLLM OpenTelemetry example" horizontal />

  <Card icon="book-open" href="https://docs.vllm.ai/en/latest/serving/online_serving/openai_compatible_server/" title="vLLM OpenAI-compatible server" horizontal />

  <Card icon="terminal" href="https://github.com/Arize-ai/openinference/tree/main/python/instrumentation/openinference-instrumentation-openai" title="OpenInference OpenAI Instrumentor" horizontal />
</CardGroup>