Skip to main content
Instead of telling your team “go look at traces from yesterday,” create a labeling queue with the specific spans or traces that need review — a structured list with an annotation schema and progress tracking.

​
Add spans to a queue

Individual: Open a span → click Add to → Labeling Queue → choose or create a queue. Bulk: Select multiple spans in the traces table → Add to → Labeling Queue. [screenshot: add to labeling queue menu from span toolbar]

​
The review workflow

  1. Open Annotation Queues from the left sidebar
  2. Select a queue → review each record’s inputs, outputs, and existing evals
  3. Add annotations using the queue’s annotation config
  4. Move to the next record
[screenshot: annotator queue view with span details and annotation form]
A queue record can be a single span or an entire trace. When you annotate a trace-level record, the label is written as a trace-level annotation rather than on an individual span — use it when the judgment covers the whole request or conversation.

​
From queue to dataset

After labeling, create a dataset from the annotated results for experiments and fine-tuning.
With human review: Traces → Labeling Queue → Annotate → DatasetWithout human review: Traces → Dataset directly

​
Manage queues via API

You can create and manage annotation queues programmatically using the GraphQL API. This is useful for automating queue creation in CI/CD pipelines or building custom tooling.
The Python SDK (v8+) supports creating annotation configs programmatically. Annotation queue management via SDK is planned — use the GraphQL API in the meantime.