TypeScript Quickstart

The Phoenix TypeScript Evals package makes it easy to evaluate your application end-to-end by defining datasets, tasks, and evaluators that reveal how well your system performs and where it can improve.

@arizeai/phoenix-evals

Don’t have traces yet? Let your coding agent set up tracing for you: start Phoenix, then run npx -y @arizeai/phoenix-cli setup from your app’s root directory. See Agent-Assisted Setup.

Why Should I Use Evaluators?

Simply running an application or agent tells you what it does. Evaluations reveal how well it performs, why it succeeds or fails, and where it needs improvement. Ad-hoc runs make it hard to measure progress or compare changes over time. Evaluations add structure and repeatability: they let you test the agent across a controlled datasets and surface hidden failure patterns that casual testing would miss. With evaluations, you can make informed decisions about improving your agent, verify that updates help without introducing regressions elsewhere, and build confidence in its performance before deployment.

If you’d like to follow along with this evaluation example, you can check out the application here: https://github.com/Arize-ai/phoenix/tree/main/js/examples/apps/demo-document-relevancy-experiment

Set up and Connect to Phoenix

Before running your experiment and evaluations, make sure Phoenix is running and your environment is set up with the required dependencies.

Local
Container
Self-Host

uvx arize-phoenix serve

No uv? pip install arize-phoenix && phoenix serve does the same thing. See Terminal setup for customization.

docker run -p 6006:6006 -p 4317:4317 arizephoenix/phoenix:latest

Images are published to Docker Hub. See Docker for volumes, PostgreSQL, and other options.

Phoenix serves its UI and OTLP HTTP on port 6006, and OTLP gRPC on port 4317. For a local instance that’s http://localhost:6006 — leave it running while you work.

Then install the dependencies this guide uses into your TypeScript project:

npm install @arizeai/phoenix-otel @arizeai/openinference-instrumentation-openai openai @ai-sdk/openai @arizeai/phoenix-client @arizeai/phoenix-evals

Include these in your .env file. PHOENIX_ENDPOINT is what @arizeai/phoenix-client reads for datasets and experiments; PHOENIX_COLLECTOR_ENDPOINT is what @arizeai/phoenix-otel reads when exporting spans — point both at the same Phoenix instance.

OPENAI_API_KEY="your-openai-api-key"
PHOENIX_ENDPOINT="http://localhost:6006"
PHOENIX_COLLECTOR_ENDPOINT="http://localhost:6006/v1/traces"  # OTLP HTTP traces URL
# PHOENIX_API_KEY="..."  # Only if your Phoenix instance has auth enabled

Define a Task

Simply put, the task defines how your application should behave. The task specifies exactly which input fields to pass in and how the application should process that input. By standardizing execution across examples, tasks ensure that evaluations are consistent, repeatable, and comparable as your application evolves.This example assumes the task function is calling the spaceKnowledgeApplication that retrieves context from a knowledge base to answer questions:

async function task(example) {
    const question = example.input.question;
    const result = await spaceKnowledgeApplication(question);
    return result.context || [];
}

Define a Dataset

A meaningful evaluation starts with a well-constructed dataset. This dataset should contain a diverse set of examples that capture both typical success cases and realistic failure modes.Each row in your dataset represents a single scenario the application or agent will encounter, including the input and, when applicable, the expected output. The goal is to build a small but representative slice of the real world your application is meant to handle. A thoughtfully designed dataset ensures that the evaluation results are meaningful and aligned with the application’s capabilities.

const DATASET = [
  "Which moon might harbor life due to its unique geological features?",
  "What theoretical region marks the outer boundary of the Solar System?",
  "Which planet defies the typical rotation pattern observed in most celestial bodies?",
  "Where in the Solar System would you experience the most extreme atmospheric conditions?",
  "How dominant is the Sun's gravitational influence compared to all other objects in our solar system?",
  "What region of the Solar System contains remnants from its early formation beyond the gas giants?",
  "What significant change occurred in our understanding of planetary classification in 2006?",
  "What environmental challenge would explorers face during certain seasons on Mars?",
  "What makes Venus one of the most hostile environments for robotic exploration?",
  "What unique liquid features exist on Saturn's largest moon?",
  "What is the duration of the longest-observed storm in our Solar System?",
  "Which celestial body experiences the most intense geological activity?",
  "Which planet experiences the most dramatic temperature swings between day and night?",
  "What region separates the inner and outer planets in our Solar System?",
  "What unusual orbital characteristic makes Uranus unique among the planets?",
];

import { createDataset } from "@arizeai/phoenix-client/datasets";

// Create a dataset for your experiment
const dataset = await createDataset({
    name: "document-relevancy-eval",
    description: "Queries that are answered by extracting context from the space knowledge base",
    examples: DATASET.map(question => ({
      input: {
        question: question,
      },
    })),
  });

Create an Evaluator

Once a task and dataset are defined, the final piece of the experimentation workflow is the evaluator.Evaluators determine whether the task output for each example is “good,” “bad,” or somewhere in between. You can use Phoenix pre-built evaluators or define custom evaluators that give you full control over the metrics and logic used to judge application behavior. The evaluator you choose should align with the specific quality or capability you want to measure.In this example, we’ll use Phoenix’s pre-built Retrieval Relevance evaluator. This evaluator checks whether retrieved context actually contains the information needed to answer the user’s question—ensuring your application is grounding its responses in the right documents.

import { asExperimentEvaluator } from "@arizeai/phoenix-client/experiments";
import { createRetrievalRelevanceEvaluator } from "@arizeai/phoenix-evals/llm/createRetrievalRelevanceEvaluator";
import { openai } from "@ai-sdk/openai";

const retrievalRelevanceEvaluator = createRetrievalRelevanceEvaluator({
    model: openai("gpt-4o-mini"),
  });

  const retrievalRelevanceCheck = asExperimentEvaluator({
    name: "retrieval-relevance",
    kind: "LLM",
    evaluate: async ({ input, output }) => {
      // Use the retrieval relevance evaluator from phoenix-evals
      const result = await retrievalRelevanceEvaluator.evaluate({
        input: input.question,
        context: output || "",
      });
      return result; 
    },
  });

Run the Experiment

An experiment ties the dataset, task, and evaluator together into an end-to-end process.When you run an experiment, each dataset example is passed through the task, generating outputs that are then automatically scored by the evaluator. Experiments provide a structured, repeatable framework for testing the application’s performance and collecting metrics at scale. Running the experiment produces a full set of scores, explanations, and traces for analysis.

import { runExperiment } from "@arizeai/phoenix-client/experiments";

await runExperiment({
    experimentName: "document-relevancy-experiment",
    experimentDescription: "Evaluate the relevancy of extracted context from the space knowledge base",
    dataset: dataset,
    task,
    evaluators: [retrievalRelevanceCheck],
  });

View Results in Phoenix

After the experiment completes, the results provide a detailed breakdown of how the application performed across all examples. You can quickly identify success cases, pinpoint failure modes, and analyze patterns across the dataset.For LLM-as-a-Judge evaluators, the explanation field is especially valuable—it highlights why the evaluator scored a response a certain way. These explanations often reveal actionable insights, such as missing reasoning steps, misinterpretations, or opportunities to refine prompts. By reviewing these results holistically, you can iteratively improve your application and build confidence in its performance.

Get Started

Tracing

Evaluation

Datasets & Experiments

Prompts

Settings

Concepts

Resources

TypeScript Quickstart

@arizeai/phoenix-evals

Why Should I Use Evaluators?

@arizeai/phoenix-evals

​Why Should I Use Evaluators?

Why Should I Use Evaluators?