Use labeling queues when you want a subject matter expert or third party to label spans without exposing the full traces view. Reviewers get a focused interface showing only what they need to annotate. Once labeled, those examples become the ground truth dataset you use to validate evaluators and run experiments.
By Alyx
By UI
Ask Alyx to create a labeling queue, send data to it, and optionally annotate data. For example:
“Send spans where latency is over 5 seconds to my Slow Response labeling queue”
“Send spans where hallucination eval scored 0 to the Hallucination Review queue”
Send data to a labeling queue with Alyx
You can create a labeling queue from the traces or spans table, from within an individual span, or directly from Labeling Queues in the left nav. In all cases you define review instructions, select annotation configs, and assign team members.
After records have been assigned, users with queue update access can change a record’s assignee from the record’s assignee selector. Reassign records to balance review workloads or route work to another annotator when the current annotator is unavailable.
A ground truth dataset is a curated set of labeled examples that captures the range of behaviors your system should and should not produce. It gives you a stable benchmark for validating automated evaluators and a reusable dataset to run experiments against as your prompts and models evolve.
By Alyx
By UI
Ask Alyx to create a dataset from spans of interest, append spans to an existing dataset, or suggest examples that cover edge cases for your rubric.Example prompts:
“Create a dataset from the spans I filtered in this trace view and include inputs and outputs”
“Append these high-error spans to my regression benchmark dataset”
“Suggest 20 diverse examples for a golden dataset based on my last week’s traces”
Create a golden dataset with Alyx
To build one, filter or search your spans in Tracing to find representative examples. Select rows and use Add to Dataset to create a new dataset or append to an existing one — span fields map to columns you can edit. Open the dataset from the left nav to review rows, add reference output columns, and version the dataset as labels stabilize.
