Use a managed agent to summarize every trace in a project, cluster the summaries into named patterns, and label your traces so patterns become filterable.
Managed agents are only available on Enterprise plans.
A project with thousands of traces holds patterns no single trace shows you. Some are what users keep asking for, some are the ways your agent keeps failing, and neither shows up in a latency chart. A managed agent can read the whole project, summarize each trace, group the summaries into named patterns, and write those patterns back onto the traces as annotations so you can filter by them.Arize AX offers three other ways to turn traces into named patterns: Arize Skills in your coding agent, Alyx in the product, or manual annotation. A managed agent works through traces in one unattended run.
You will start a managed agent session against a tracing project, hand it a prompt that describes a summarize-and-cluster pipeline, and read the patterns it finds. The agent runs four stages: it summarizes each trace in a sentence using its own model, turns those summaries into vectors, reduces the vectors to a few dimensions, and clusters the result. It names each cluster and annotates the traces in it with that name, so you can then filter your traces by pattern.This guide is for builders who already have tracing projects in Arize AX and have managed agents enabled on their account. No prior experience with managed agents is required.You should be comfortable with:
Tracing projects in Arize AX, and reading traces in the UI
Each run happens in a sandbox, an isolated container that Arize AX tears down afterward. What the agent can reach is fixed before the run starts, and the prompts below are written to fit those limits:
Capability
Detail
Your traces
The agent reads spans through the Arize AX API, over a time window you choose rather than a whole project at once.
Project scope
The agent’s credentials cover your entire space, so name the project you want analyzed in the prompt.
Python
A plain interpreter with nothing extra installed.
More packages
The agent can install libraries only when a GitHub repo is attached to the run.
Summarization
The agent’s own model calls. No embedding model is reachable, so the vectors have to come from the text of the summaries.
Writing back
The agent can create an annotation config and label spans for you, but it cannot change a config once one exists.
Deliverables
Files the agent saves as deliverables outlive the sandbox.
Which pipeline you can ask for comes down to one thing: whether a GitHub repo is attached to the run, because that is what opens PyPI.With a repo attached, the agent installs scikit-learn and UMAP and clusters with the libraries built for the job, which produces tighter clusters and handles more traces per run.Without a repo, every step has to be computable with the standard library, and requires no repo setup. It is recommended for batches of up to a few thousand traces. Past a few thousand, narrow the time window or attach a repo.Both routes produce the same summaries, the same annotation config, and the same filters. Only vectorizing and clustering differ.
Three methods carry the pipeline, all computable with the standard library:
TF-IDF weighs each word by how often it appears in one summary against how rare it is across all of them, turning short summaries into vectors with nothing downloaded.
Spectral reduction takes the top eigenvectors of the trace-by-trace similarity matrix, which is classical multidimensional scaling on the cosine kernel. Sparse text vectors cluster poorly at full width, and this is the part UMAP would do if you could install it.
Average-link agglomerative clustering merges the closest pair of clusters until nothing is similar enough to join, so you set a similarity threshold rather than a cluster count.
1
Start a session in Agent Studio
Open Observe → Signal & Agents → + New Agent and choose Start from scratch. Then:
Select the tracing project you want analyzed. This is the project the agent reads and annotates.
Leave the preset as None.
Configuring a run in Agent Studio: selecting a project, and the preset, skill, and repo options this pipeline leaves alone
2
Give it the pipeline as a prompt
Paste this as the task prompt. It describes the pipeline in stages, and each stage names its own output so the agent cannot quietly skip one. Replace <project> with your project’s name first; $ARIZE_SPACE_ID is already set inside the sandbox, so leave it as written.
Find the recurring patterns in this project's traces, then label the traces with them.Sandbox constraints: PyPI is not reachable, so do not try to install anything.Use only the Python standard library; /usr/bin/python3 is 3.13 with nothird-party packages. There is no embedding API reachable from here either. Ifsomething you need is genuinely unavailable, stop and tell me rather thansubstituting a different method.1. Export spans with `ax spans export <project> --days 30 --limit 5000 --stdout`. Do not use `--all`: bulk export needs Arrow Flight, which this sandbox blocks. `--limit` defaults to 100, so always set it high enough to cover the window, and if the result comes back exactly at the limit, narrow `--days` and export in several passes rather than accepting a truncated set. Group the spans by `context.trace_id`. For each trace keep the root span's `context.span_id`, `attributes.input.value`, `attributes.output.value`, its `start_time`, the `attributes.tool.name` of any child tool spans, and `status_message` for any span whose `status_code` is ERROR. Report how many spans you exported and how many distinct traces they cover, and work through all of them rather than a sample.2. Summarize each trace in under 25 words: what the user wanted, what the agent did, and how it ended. Give intent and outcome equal weight, and state a success as plainly as a failure. No order IDs, emails, or other incidental detail, and no boilerplate tail such as "no errors occurred".3. Turn the summaries into vectors with tf-idf: - lowercase, split on non-letters, drop tokens under 3 characters - drop tokens appearing in only one summary, and tokens appearing in more than half of them - weight each token as (1 + log(count in summary)) * log(total summaries / summaries containing it) - L2-normalize every vector4. Reduce before clustering. Compute the trace-by-trace cosine similarity matrix, double-center it, then take its top 10 eigenvectors by power iteration with deflation, each scaled by the square root of its eigenvalue. Re-normalize the resulting 10-dimensional rows.5. Cluster in that reduced space with average-link agglomerative merging on cosine similarity: merge the two closest clusters while their average-link similarity is at least 0.30. Break ties by lowest trace index so repeat runs are identical.6. Any cluster holding fewer than 3 traces is not a pattern. Give those traces the slug `unclustered`, and report how many there are.7. Name each cluster in under 10 words. Give each one a slug to use as a label: lowercase, hyphenated, 30 characters or less.8. Write /workspace/.artifacts/patterns.json with one object per cluster: slug, name, trace count, earliest and latest trace timestamp, and three example trace summaries. Then report in your reply: total traces, cluster count, unclustered count, and the largest and smallest cluster size.9. Set up the annotation config. Check first with `ax annotation-configs get pattern --space $ARIZE_SPACE_ID`. - If it does not exist, create it with `ax annotation-configs create --name pattern --space $ARIZE_SPACE_ID --type CATEGORICAL` and one `--value <slug>` per cluster, plus `--value unclustered`. - If it already exists, use the slugs it already lists. Match each cluster to the closest existing slug and reuse it, and label any cluster with no good match `unclustered`. Do not create a second config under another name, and do not try to amend this one: config names are unique per space, and the `ax` build here cannot update a config's values. Report any cluster you had to leave unlabelled so I can widen the config myself. The type must be uppercase, labels are capped at 30 characters, and a categorical config needs between 2 and 100 values.10. Annotate the traces. Build a JSON array of records, one per root span: {"record_id": "<root span id>", "values": [{"name": "pattern", "label": "<slug>", "text": "<that trace's summary>"}]} then run `ax spans annotate <project> --space $ARIZE_SPACE_ID --file <path> --days 31`. Send at most 1000 records per request. Every span id has to fall inside the day window or the whole request is rejected, and the window cannot exceed 31 days, so keep the export window inside that limit too.Finally, tell me which clusters are ordinary demand, which are failures, andwhich are requests the agent has no way to satisfy.
The diagnostics in steps 6 and 8 are what let you audit the result. Compare the reported trace count against your project’s actual volume, because an agent that quietly worked on a subset produces labels that look complete and undercount every pattern. A cluster confined to a single hour is an incident rather than a pattern, which is why each one carries a time range.Click Run session.
Attaching a repo opens PyPI, which lets the agent install the clustering libraries. The repo’s contents are never used: attaching one is only what changes the sandbox’s egress rules, so any repo you can access works.Three methods carry the pipeline, each from a library the agent installs:
TF-IDF weighs each word by how often it appears in one summary against how rare it is across all of them, turning short summaries into vectors. scikit-learn also drops common English words, which sharpens the contrast between summaries.
UMAP reduces those vectors to a handful of dimensions, keeping summaries that share vocabulary near each other. Sparse text vectors cluster poorly at their full width.
HDBSCAN groups the reduced vectors by density. You never say how many patterns to expect, and it labels traces that fit no cluster as noise rather than forcing them into one. min_cluster_size sets the smallest group it will call a cluster.
1
Connect GitHub and set the repo
Open Observe → Signal & Agents → + New Agent, and choose Start from scratch, then:
Select the tracing project you want analyzed. This is the project the agent reads and annotates.
Leave the preset as None, or choose a preset with a repo attached.
Authorize GitHub next to the Repo field. If an Install GitHub App button is offered, use it and then pick your repository from the list, which attaches the integration for you. Otherwise click + Add skill, choose GitHub, and paste a personal access token as GH_TOKEN with repo read scope.
If you went the token route, set Repo to org/repo.
The repo field is ignored unless a GitHub skill is attached.
Already have a preset with a GitHub skill and repo? Pick it from the preset menu instead and skip this step. Binding a preset hides the ad-hoc skills picker, because the run inherits the preset’s skills and repo.
2
Give it the pipeline as a prompt
Paste this as the task prompt. Replace <project> with your project’s name first; $ARIZE_SPACE_ID is already set inside the sandbox, so leave it as written.
Find the recurring patterns in this project's traces, then label the traces with them.A repo is attached to this run, so PyPI is reachable. Create a virtualenv with`uv venv` and install scikit-learn and umap-learn into it, and use thatvirtualenv for every script below. Use scikit-learn's own HDBSCAN(sklearn.cluster.HDBSCAN) rather than the standalone hdbscan package, whichneeds a compiler. If an install fails, stop and tell me rather than falling backto a different method.1. Export spans with `ax spans export <project> --days 30 --limit 5000 --stdout`. Do not use `--all`: bulk export needs Arrow Flight, which this sandbox blocks. `--limit` defaults to 100, so always set it high enough to cover the window, and if the result comes back exactly at the limit, narrow `--days` and export in several passes rather than accepting a truncated set. Group the spans by `context.trace_id`. For each trace keep the root span's `context.span_id`, `attributes.input.value`, `attributes.output.value`, its `start_time`, the `attributes.tool.name` of any child tool spans, and `status_message` for any span whose `status_code` is ERROR. Report how many spans you exported and how many distinct traces they cover, and work through all of them rather than a sample.2. Summarize each trace in under 25 words: what the user wanted, what the agent did, and how it ended. Give intent and outcome equal weight, and state a success as plainly as a failure. No order IDs, emails, or other incidental detail, and no boilerplate tail such as "no errors occurred".3. Vectorize the summaries with TfidfVectorizer, sublinear_tf=True, min_df=2, max_df=0.5, and stop_words="english".4. Reduce with UMAP: metric="cosine", n_components=5, n_neighbors of roughly the square root of the trace count, and a fixed random_state so repeat runs match.5. Cluster the reduced vectors with HDBSCAN, min_cluster_size=5 and min_samples=1.6. Traces HDBSCAN labels -1 are noise. Give them the slug `unclustered`, and report how many there are.7. Name each cluster in under 10 words. Give each one a slug to use as a label: lowercase, hyphenated, 30 characters or less.8. Write /workspace/.artifacts/patterns.json with one object per cluster: slug, name, trace count, earliest and latest trace timestamp, and three example trace summaries. Then report in your reply: total traces, cluster count, unclustered count, and the largest and smallest cluster size.9. Set up the annotation config. Check first with `ax annotation-configs get pattern --space $ARIZE_SPACE_ID`. - If it does not exist, create it with `ax annotation-configs create --name pattern --space $ARIZE_SPACE_ID --type CATEGORICAL` and one `--value <slug>` per cluster, plus `--value unclustered`. - If it already exists, use the slugs it already lists. Match each cluster to the closest existing slug and reuse it, and label any cluster with no good match `unclustered`. Do not create a second config under another name, and do not try to amend this one: config names are unique per space, and the `ax` build here cannot update a config's values. Report any cluster you had to leave unlabelled so I can widen the config myself. The type must be uppercase, labels are capped at 30 characters, and a categorical config needs between 2 and 100 values.10. Annotate the traces. Build a JSON array of records, one per root span: {"record_id": "<root span id>", "values": [{"name": "pattern", "label": "<slug>", "text": "<that trace's summary>"}]} then run `ax spans annotate <project> --space $ARIZE_SPACE_ID --file <path> --days 31`. Send at most 1000 records per request. Every span id has to fall inside the day window or the whole request is rejected, and the window cannot exceed 31 days, so keep the export window inside that limit too.Finally, tell me which clusters are ordinary demand, which are failures, andwhich are requests the agent has no way to satisfy.
min_cluster_size is the number to scale with volume. Leaving it low on a large project fragments patterns that belong together.Click Run session.
If your account restricts annotation config creation, create the config yourself before the run and paste its allowed values into step 9 in place of the create instruction:
Follow the transcript from the session detail slide-over, opened from the Agent List tab under Signal & Agents. The run exports spans, writes summaries, reduces and clusters them, then reports a named pattern for every cluster with the number of traces in it. The example below comes from a customer support agent:
Pattern
Traces
billing-escalation-failed
32
order-billing-limited-access
30
damaged-gear-needs-info
16
order-not-found
14
legacy-pwd-reset-failure
12
defective-item-no-tools
11
item-resolved-with-tools
9
cancel-modify-order-failed
6
unclustered
1
Read past the labels to the counts and examples. In this example the largest patterns are the ones the agent exists to handle, and the interesting ones are smaller: a dependency that fails for one class of input, and requests the agent has no tool to satisfy. That last kind never shows up in an error rate, because from the agent’s side those runs succeeded. It just quietly could not help.Notice that these labels lean on outcome rather than intent: one kind of request split into separate patterns according to whether it succeeded. That is the pipeline working as intended, and it is worth knowing before you read your own result, because patterns divide on whatever the summaries make salient.
2
Filter traces by pattern
Open any trace in the project and check the Annotations panel on its root span. The pattern label and the trace summary appear there, which confirms the write landed.Patterns are now a filter:
Because the agent annotates root spans, this returns one span per matching trace rather than every span in those traces.From there a pattern behaves like any other slice: curate it into a dataset for an experiment, put it in a dashboard to watch its volume, or send it to a labeling queue for review.
3
Run it again as traffic changes
Patterns drift, so a single run describes the window it read and nothing later. Re-run the same prompt against a newer window whenever you want a current picture.Step 9 is what makes a second run safe. Each run clusters from scratch and would otherwise invent fresh slug names, which silently breaks saved filters, so the prompt tells the agent to read the existing config and reuse the slugs already in it. Config names are unique per space, so a run that tries to create pattern again fails outright, and an agent left to improvise may route around that error by creating a differently named config.Because a run inside the sandbox cannot amend a config’s values, genuinely new patterns come back reported rather than labelled. Widen the config yourself when that happens:
--value replaces the whole list rather than adding to it, so pass every value you want to keep, or existing annotations will point at labels the config no longer allows.
The numbers in either prompt are starting points, not settled values. The two knobs pull in opposite directions: a higher similarity threshold splits clusters apart, while a higher min_cluster_size merges them by discarding the small ones. On the no-repo route the threshold is the lever; on the repo route it is min_cluster_size.Clusters that mix unrelated patterns. Raise the threshold above 0.30, or on the repo route lower min_cluster_size. A threshold set too low produces one large cluster that has swallowed several unrelated intents. Ask for more clusters than feels natural and merge them yourself, since over-merged clusters hide exactly the small patterns worth finding.Too many traces landing in unclustered. Lower the threshold. The unclustered share climbs steeply once the threshold is too high, so move it in small steps. On the repo route, lower min_cluster_size instead.One pattern split across several clusters. Expect some of this and merge by hand. TF-IDF matches shared words rather than shared meaning, so “asking for a refund” and “want my money back” need not meet, and a single intent can end up spread across two or three clusters however you tune it. On the no-repo route, the number of eigenvectors is a secondary lever: more of them tends to fragment, fewer tends to merge.Everything in one cluster. The summaries are the problem rather than the parameters. Incidental detail like order IDs gives TF-IDF spurious signal, and a shared boilerplate phrase in every summary flattens the contrast between them. Tighten the summary instruction.
Do not read a low unclustered count as proof the clustering was good. Reduction pulls outliers toward the mass, so genuinely one-off traces are more likely to be absorbed into a cluster than set aside. The count is useful for spotting a threshold set too high; it is not outlier detection.
Fewer traces annotated than you expected. The most common cause is the export, not the clustering: --limit defaults to 100, and --all cannot be used in the sandbox. Check the span and trace counts the agent reported, and if the span count sits exactly on the limit it truncated the window. Partial labels look complete and undercount every pattern.
The agent tries to install packages and then improvises. PyPI is unreachable without a repo attached, and a blocked install rarely stops a run: the agent invents its own method instead, which no longer matches any guidance here. The no-repo prompt’s opening constraint exists to prevent this, so keep it.
The install fails on the repo route. Check that GitHub is authorized and not only that a repo is named, since the repo field is ignored without it. If umap-learn pulls in a numba that will not build, pin umap-learn and numba to recent versions with prebuilt wheels for Python 3.13.
ax spans export fails with a proxy or connection error on --all. Bulk export uses Arrow Flight, which the sandbox blocks. Use --days and --limit instead.
ax spans annotate rejects the whole batch. Every span ID has to fall inside the lookup window, and the window cannot exceed 31 days, so keep the export window inside that limit rather than widening --days past it. Spans older than 31 days cannot be annotated this way at all. Batches are also capped at 1000 records.
Annotations are rejected with label '<x>' is not one of the allowed values. The agent invented a slug that the annotation config does not list. Have it create the config from its own slugs, or paste the allowed values into the prompt.
no annotation config with name 'pattern' exists. The config must exist before any span is annotated. Confirm the name in the prompt matches it exactly.
Labels cannot exceed 30 characters while creating the config. The agent used a cluster summary as a value. Ask for a separate short slug and keep the summary in the annotation’s text field.
Everything lands in one cluster or all in noise. See Tuning the clusters. Check the summaries first: if they all end in the same phrase, or all name the same tool, they will cluster into a single blob no matter what the parameters say.
You now have a repeatable pipeline: one prompt that summarizes, clusters, and labels a whole project, and a filter that turns any pattern into a slice you can act on.Clustering is one job you can hand a managed agent, not the limit of what they do. The same agent reads your traces, runs code, and reports back, so the pattern generalizes:
Evaluating agents
Point a run at a slice of traces and have it draft labels and an evaluator prompt from what it finds.
Skills and permissions
With a GitHub skill attached, a session can trace a failure to its source, write a test, and open a PR.
Agent Swarms
Run any of the above on a cron schedule or a metric threshold instead of by hand.
Surface issues with Signal
Signal scans a project continuously and surfaces ranked issues, with no prompt required.
What these share is the shape of the request: a goal, a bound project, and permission to use tools. Anything you could work out by reading traces and running code, you can hand to an agent instead, once as a session or on a schedule.
