> ## Documentation Index > Fetch the complete documentation index at: https://arizeai-433a7140.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Investigate and label a failed run > Use trace evidence to explain a failed requirement, then label similar requests consistently. In [Use traces and feedback](https://arizeai-433a7140.mintlify.site/docs/phoenix/learn/find-failures.md), you recorded what looked wrong. A note gives you a starting point, but it doesn't yet explain what happened or give you a consistent way to review other answers. Let's inspect the evidence behind a Wonder Toys recommendation, agree on what counts as a budget failure, and save that decision as a label. ![This lesson: check evidence, define review criteria, configure the annotation, then label each reviewed request.](https://storage.googleapis.com/arize-phoenix-assets/assets/images/phoenix-academy/find-failures-label-light-c756aaf1f1dc.svg) ![This lesson: check evidence, define review criteria, configure the annotation, then label each reviewed request.](https://storage.googleapis.com/arize-phoenix-assets/assets/images/phoenix-academy/find-failures-label-light-mobile-c81f3b57d123.svg) ![This lesson: check evidence, define review criteria, configure the annotation, then label each reviewed request.](https://storage.googleapis.com/arize-phoenix-assets/assets/images/phoenix-academy/find-failures-label-dark-ae95fff823d2.svg) ![This lesson: check evidence, define review criteria, configure the annotation, then label each reviewed request.](https://storage.googleapis.com/arize-phoenix-assets/assets/images/phoenix-academy/find-failures-label-dark-mobile-adf10e7ce0fa.svg) Follow these four steps to turn your observations into consistent labels. ## Before you start Open a request you noted in [Use traces and feedback](https://arizeai-433a7140.mintlify.site/docs/phoenix/learn/find-failures.md). The example below uses the recorded \$25 request, where Wonder Toys recommended a \$27.99 kit. If your run stayed within budget, follow the same checks and label it according to your evidence. ## Work with PXI (optional) [PXI](https://arizeai-433a7140.mintlify.site/docs/phoenix/pxi.md) (pronounced "pixie") is the AI assistant built into Phoenix. You can follow the steps yourself or expand **Ask PXI** under each step and copy its prompt into the assistant. Review its findings against the trace before accepting a label. To try it, follow the [PXI setup guide](https://arizeai-433a7140.mintlify.site/docs/phoenix/pxi.md#get-started), then open PXI with **⌘I** on macOS or **Ctrl+I** on Windows and Linux while viewing the trace. ![Phoenix in light mode with the Wonder Toys trace open beside the PXI assistant panel](https://storage.googleapis.com/arize-phoenix-assets/assets/images/phoenix-academy/investigate-pxi-overview-be6dffb2f857.png) The Wonder Toys trace is open on the left, with PXI in the panel on the right. * Review PXI's criteria and explanations against the trace before accepting them. PXI is in beta; keep **Manual Approval** enabled so you can review proposed changes before they're saved. * Continue in the same PXI chat as you move through the steps. If you define the criteria yourself, share them with PXI before asking it to label a request. ## Step 1: Check what the agent knew Read the request and answer on the top-level **Wonder Toys** span, then inspect the steps behind the recommendation: * Select `search-products` and check the returned product prices in **Output**. * Select a model call after that search, such as `ChatAnthropic`, and find those results in **Input**. * Compare the returned prices with the complete recommendation on the **Wonder Toys** span. In the recorded run, the search returned the kit's \$27.99 price, and that price reached the final model call. The model still recommended the kit while noting that it exceeded the budget. ![Model input containing the search result for the Dinosaur Explorer Kit priced at 27.99](https://storage.googleapis.com/arize-phoenix-assets/assets/images/phoenix-academy/wonder-toys-model-search-results.png) The model received the product's price before it made the recommendation. The evidence rules out a missing price in this run. It points to how the model selected and presented its recommendations, but it doesn't yet show which change will fix the problem. If a needed step or value is missing from your trace, record that gap instead of guessing. > [accordion] Ask PXI to inspect the trace > > ```text wrap theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} > Inspect the trace I have open. Compare the shopper's budget with the product prices returned by search-products and the complete answer on the Wonder Toys span. Cite the spans that support your explanation, and identify anything the trace doesn't establish. Keep the answer brief. Do not change any data. > ``` > > ![PXI's summary comparing the shopper's 25 dollar budget with the 27.99 dollar recommendation, with links to the search and model spans](https://storage.googleapis.com/arize-phoenix-assets/assets/images/phoenix-academy/investigate-pxi-6bdc8d27d34d.png) > > PXI summarizes its investigation of the recorded run and links to the supporting spans. ## Step 2: Define the behavior you want to label The trace tells us what happened. Before labeling it, let's agree on one question: **did the recommendation respect the shopper's budget?** Use the same rules for every request you review: * **pass:** Recommended products are below the limit, or the agent explains that no match was found without recommending a more expensive product. * **fail:** The agent recommends a product at or above the limit without the shopper agreeing to change it. These are the review criteria for this exercise. A budget label doesn't assess age suitability or whether the products match the requested topic. Record those separately if your notes show a problem. Leave uncertain cases unlabeled and explain what needs checking. > [accordion] Ask PXI to define the criteria > > ```text wrap theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} > Draft pass/fail criteria for reviewing whether a recommendation respects the shopper's budget. Products must be below the stated limit unless the shopper agrees to change it. An answer that explains no match was found can pass without recommending a more expensive product. Assess other requirements separately and leave uncertain cases unlabeled. Show the criteria without creating a configuration or saving labels yet. > ``` ## Step 3: Configure the annotation in Phoenix Now let's make those decisions available in Phoenix. An [**annotation**](https://arizeai-433a7140.mintlify.site/docs/phoenix/tracing/concepts-tracing/annotations-concepts.md) stores your assessment on a span. An **annotation configuration** defines its name and available labels. We'll create one called `budget_compliance`. This is our name for the budget review in this exercise. Create it once, then reuse it across the requests you review: * Open **Settings → Annotations** and check whether `budget_compliance` already exists. If its labels and scores match the settings below, reuse it. * Otherwise, select **New Configuration**, enter `budget_compliance` as the name, and choose **Categorical** for the type. * Copy the criteria from Step 2 into **Description** so reviewers can refer to the same rules. * Set the two category values to `pass` with score `1` and `fail` with score `0`. * Select **Maximize** for **Optimization Direction**, since a higher score means the budget requirement was met. Select **Create Annotation Config**. * Open **wonder-toys → Config**. Under **Annotation Configs**, select the checkbox for `budget_compliance` so it appears when you annotate a span. ![Phoenix annotation configuration with budget\_compliance and pass and fail labels](https://storage.googleapis.com/arize-phoenix-assets/assets/images/phoenix-academy/investigate-budget-config-9f4d29139845.png) Define the same labels for everyone reviewing this requirement. See [Annotating in the UI](https://arizeai-433a7140.mintlify.site/docs/phoenix/tracing/how-to-tracing/feedback-and-annotations/annotating-in-the-ui.md) for more about annotation configuration. > [accordion] Ask PXI to configure the annotation > > ```text wrap theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} > Using the budget criteria we just agreed on, check for a categorical annotation configuration named budget_compliance. Reuse it if its labels are pass = 1 and fail = 0, with optimization direction Maximize. If it is missing or differs, show the proposed changes for approval. Associate the configuration with this trace's project. Do not label any spans yet. > ``` ## Step 4: Label the request and explain the decision * Return to the trace and select the top-level **Wonder Toys** span. * Choose **Annotate this span**. In the `budget_compliance` field, select `pass` or `fail` based on your review. Phoenix saves the label when you select it. * Select **explain** beside the field, add your reason with the request and relevant price, then select **Save**. For the recorded \$25 request, the label is `fail`: the final model received the \$27.99 price and still recommended the kit. ![Wonder Toys span with a budget\_compliance fail annotation and an explanation citing the 25 dollar limit and 27.99 dollar recommendation](https://storage.googleapis.com/arize-phoenix-assets/assets/images/phoenix-academy/investigate-budget-label-e285d022920c.png) This example review was added by an AI assistant and is marked LLM. An annotation you add in the UI is recorded as a human review. > [accordion] Ask PXI to label this request > > ```text wrap theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} > Apply our budget criteria to the request in this open trace. Propose a pass/fail label and an explanation citing the shopper's limit, returned prices, and final recommendation. After approval, save budget_compliance on the top-level Wonder Toys span with annotator kind LLM. Preserve existing reviews and leave uncertain cases unlabeled. > ``` **Repeat for each request you reviewed on the previous page:** the \$25, \$35, and \$1 requests. Read each trace before deciding; a different budget can produce a different outcome. * Save one budget assessment on the top-level **Wonder Toys** span for each request, rather than labeling every model and tool span. * Include passing requests as well as failures. If the evidence is insufficient, leave the request unlabeled and add a note about what's missing. * Reuse the configuration from Step 3. You don't need to create it again. To find your failing requests again, open **wonder-toys → Spans** and enter: ```python wrap theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} parent_span is None and annotations['budget_compliance'].label == 'fail' ``` The first condition selects top-level spans; the second selects the label you added. The notes and labels are attached to these spans, so use the **Spans** filter here. Your notes and labels now record how each request handled the budget. Next, [turn those runs into test cases](https://arizeai-433a7140.mintlify.site/docs/phoenix/learn/build-test-cases.md) so you can check whether a change fixes the problem while preserving answers that already worked.