CI Eval Tests

​

Install

npm install -D @arizeai/phoenix-client @arizeai/phoenix-evals vitest dotenv
# or, for jest:
npm install -D @arizeai/phoenix-client @arizeai/phoenix-evals jest dotenv

​

Minimal Example

import * as px from "@arizeai/phoenix-client/vitest";
import { expect } from "vitest";

px.describe("generate sql demo", () => {
  px.test(
    "generates select all",
    {
      input: { userQuery: "Get all users from the customers table" },
      expected: { sql: "SELECT * FROM customers;" },
    },
    async ({ input, expected }) => {
      const sql = await myApp(input.userQuery);
      px.logOutput({ sql });
      expect(sql).toEqual(expected?.sql);
    },
  );
});

​

Docs And Source In node_modules

node_modules/@arizeai/phoenix-client/docs/
node_modules/@arizeai/phoenix-client/src/

​

Module Map

​

Phoenix Terminology

​

Configuration

​

Repetitions

​

Dry-Run Mode

• Whole process — PHOENIX_TEST_TRACKING=false.
• One suite — describe(name, fn, { dryRun: true }).
• One test — test(name, { input, dryRun: true }, fn); that case runs as an ordinary local test, with no dataset example and nothing uploaded, even when the rest of the suite syncs.

​

Reporter Output

• A scoreboard across all suites — passed count, the gated metric average, the acceptance verdict, and the experiment link, one row per suite.
• A per-suite results table showing only failures and evaluator misses (a run whose annotation score falls below its acceptance bar), with an AGGREGATE row over the whole suite. Passing rows are hidden behind a … N passing rows hidden footer — their full detail (input, output, annotations) is always written to the JSON artifacts (see PHOENIX_TEST_REPORT_DIR).

​

Acceptance Criteria

• average — gate on overall quality. The mean score across all runs must clear threshold (compared in direction). A few weak runs are tolerated as long as the mean holds.
• passRate — gate on consistency. Each run passes when its passFn predicate returns true, and the suite passes when the fraction of passing runs is at least minPassRate (e.g. minPassRate: 0.9 ⇒ 90% must pass; minPassRate: 1 ⇒ all of them).

px.describe("text-to-sql", () => {
  // each test logs `token_f1` (0–1), `valid_sql` (boolean), and `latency_ms`
}, {
  acceptanceCriteria: [
    // overall quality: the mean token_f1 across the suite must be >= 0.8
    { annotationName: "token_f1", metric: "average", threshold: 0.8 },
    // consistency: at least 90% of runs must score >= 0.7 on token_f1
    {
      annotationName: "token_f1",
      metric: "passRate",
      passFn: (a) => typeof a.score === "number" && a.score >= 0.7,
      minPassRate: 0.9,
    },
    // hard floor: every run must produce valid SQL (boolean must be true)
    {
      annotationName: "valid_sql",
      metric: "passRate",
      passFn: (a) => a.score === true,
      minPassRate: 1,
    },
    // budget: lower is better, so the mean latency must stay <= 800ms
    {
      annotationName: "latency_ms",
      metric: "average",
      threshold: 800,
      direction: "minimize",
    },
  ],
});

​

Direction

• "maximize" (default) — higher is better; the mean clears threshold when it is >= it.
• "minimize" — lower is better; the mean clears threshold when it is <= it. Use it for cost, latency, or error-rate annotations.

​

Scoring details

• passFn flexibility — because passFn receives the whole annotation, a passRate criterion can gate on a score bar (a.score >= 0.7), a range (a.score >= 0.5 && a.score <= 0.9), a label (a.label === "correct"), or any combination. Booleans arrive as a.score === true / false.
• Booleans in average count as 1 (true) / 0 (false) when computing the mean.
• Duplicate annotations — if a run logs the same annotationName more than once, the last one counts.
• Missing annotations — an average criterion with no numeric/boolean scores, or a passRate criterion whose annotation was never logged, fails with a “no … found” reason rather than passing vacuously.
• Skipped vs dry-run — skipped tests are excluded from the aggregate; dry-run tests are included because they still execute locally.

​

Reporter output

​

Where To Start

• CI Eval Tests: Vitest — config, reporter, and the full describe / test / test.each API as it surfaces in Vitest
• CI Eval Tests: Jest — the same API surface in a Jest project
• CI Eval Test Annotations — logAnnotation, evaluate, and the Annotation shape