> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Devin
> Trace Devin CLI interactions, per-generation model calls, token usage, and tool calls in Arize AX using the Arize Coding Harness Tracing.
> Trace Devin CLI interactions, model calls, token usage, and tool calls in Arize AX for full observability.
[Devin](https://devin.ai/) is Cognition's autonomous software engineer. The [Arize Coding Harness Tracing](https://github.com/Arize-ai/coding-harness-tracing) instruments the Devin CLI and exports [OpenInference](https://github.com/Arize-ai/openinference) spans to Arize AX. Each agent interaction emits its own trace: a root AGENT span, per-generation LLM spans with real token counts, and TOOL spans for tool calls.
Devin's hook payloads are thin, carrying no session ID and no token or model data, so the rich content is read from Devin's live SQLite session database instead. The harness registers a `Stop` hook that fires at the end of **each** agent response: it resolves the session, reads the generations that have appeared since the last emission, and emits one self-contained trace for that interaction. Traces therefore appear as each interaction completes, not only after the session exits. A `SessionEnd` hook is also registered as a final flush for an interrupted last turn.
## Launch Arize AX
To get started, sign up for a free [Arize AX account](https://app.arize.com/auth/join) and get your Space ID and API Key:
1. Log in at [app.arize.com](https://app.arize.com)
2. Click **Settings** and copy the **Space ID**
3. Open the **API Keys** tab and create or copy an API key
## Install
### Curl installer
**macOS / Linux:**
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
curl -sSL https://raw.githubusercontent.com/Arize-ai/coding-harness-tracing/main/install.sh | bash -s -- devin
```
**Windows (PowerShell):**
```powershell theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
iwr -useb https://raw.githubusercontent.com/Arize-ai/coding-harness-tracing/main/install.bat -OutFile $env:TEMP\install.bat
& $env:TEMP\install.bat devin
```
### Local clone
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
git clone https://github.com/Arize-ai/coding-harness-tracing.git
cd coding-harness-tracing
./install.sh devin # macOS / Linux
install.bat devin # Windows
```
The installer writes credentials to `~/.arize/harness/config.json` and registers `Stop` and `SessionEnd` command hooks under the top-level `hooks` key in Devin's user config: `~/.config/devin/config.json` on macOS and Linux, `%APPDATA%\devin\config.json` on Windows.
Devin allows comments in its config files. The installer reads them, but writes the file back as plain JSON, so comments in a config it has to modify are not preserved. A config it cannot parse is left untouched and the install aborts with an error rather than overwriting your settings.
The installer runs a short interactive setup. Every harness in the Arize Coding Harness Tracing repo asks the same questions, in the same order.
### Setup walkthrough
#### 1. Backend selection
Choose where spans are sent:
* **Phoenix** — your own [Phoenix](https://github.com/Arize-ai/phoenix) instance.
* **Arize AX** — the hosted Arize platform.
#### 2. Credentials
The prompts depend on the backend you picked.
* **API key** — create one on the [API keys](/docs/ax/security-and-settings/api-keys) tab.
* **Space ID** — shown on the same settings tab as your API keys.
* **OTLP endpoint** — defaults to `otlp.arize.com:443`. Override it only for a hosted or dedicated instance.
* **Endpoint** — defaults to `http://localhost:6006`.
* **API key** — optional. Leave it blank when Phoenix runs without auth.
If you have already configured another harness against the same backend, the installer offers a copy-from menu so you can reuse those credentials instead of retyping them.
#### 3. Project name
The project that this harness's spans are grouped under. Defaults to the harness name.
#### 4. User ID (optional)
A free-form identifier attached to every span as `user.id`. Useful when teammates share one backend. Leave it blank to skip.
#### 5. Content logging
Three `[Y/n]` opt-outs that apply to **every** harness, not only the one you are installing:
* Log user prompts?
* Log what tools were asked to do (commands, file paths, URLs)?
* Log what tools returned (file contents, command output)?
You are asked these only the first time you install any harness. Later installs reuse the existing `logging` block in `~/.arize/harness/config.json`, which you can edit at any time.
### Install flags
| Flag | Effect |
| :------------------------ | :--------------------------------------------------------------------------------------------------------------------------------------- |
| `--with-skills` | Symlink this harness's management skill into `.agents/skills/`, so a coding agent in the workspace can manage the tracing config for you |
| `--non-interactive`, `-y` | Ask nothing, and read every value from the environment instead |
| `--branch NAME` | Install from a specific branch instead of `main` |
| `--wheel-dir DIR` | Install from local wheels in `DIR`: no network access and no remote code execution |
### Non-interactive install
Pass `--non-interactive` (or `-y`) to skip every prompt above and take each value from the environment instead. Nothing is asked, and a missing required value is an error rather than a prompt, which makes this the mode to use from a script, from CI, or when a coding agent is driving the install itself.
Values come from the environment, or from a dotenv file named explicitly with `ARIZE_ENV_FILE`. Naming a file keeps the API key out of the command line and your shell history.
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
ARIZE_ENV_FILE=~/.arize/onboarding.env ./install.sh --non-interactive
```
| Variable | Default | Description |
| :------------------------------------ | :---------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `ARIZE_API_KEY` + `ARIZE_SPACE_ID` | — | Arize AX credentials. Both are required for the Arize backend. |
| `PHOENIX_ENDPOINT`, `PHOENIX_API_KEY` | `http://localhost:6006` | Phoenix endpoint and optional API key. |
| `ARIZE_BACKEND` | inferred | `arize` or `phoenix`. A space ID implies Arize AX and a Phoenix endpoint implies Phoenix. When both are present the install stops and asks you to set this rather than guess, since guessing would discard one backend's credentials. |
| `ARIZE_PROJECT_NAME` | harness name | Project that spans are grouped under. **Read from the dotenv file only.** An installed harness exports its own project name into every session, so an inherited environment value is ignored here. |
| `ARIZE_USER_ID` | — | Optional `user.id` on every span. |
| `ARIZE_OTLP_ENDPOINT` | `otlp.arize.com:443` | Override for a hosted or dedicated Arize instance. |
| `ARIZE_LOG_PROMPTS` | `false` | Set `true` to capture prompt text. |
| `ARIZE_LOG_TOOL_DETAILS` | `false` | Set `true` to capture tool commands, file paths, and URLs. |
| `ARIZE_LOG_TOOL_CONTENT` | `false` | Set `true` to capture tool output. |
| `ARIZE_ENV_FILE` | — | Dotenv file to read. No file is read unless this is set, and a path that is not a readable file is an error rather than a fallback to the environment. |
| `ARIZE_WHEEL_DIR` | — | Same as `--wheel-dir`. Install from local wheels instead of downloading the repo. |
A named `ARIZE_ENV_FILE` outranks the environment, and there is deliberately **no automatic `./.env` search**. Reading the working directory would let a cloned repository's dotenv choose `ARIZE_OTLP_ENDPOINT` or `PHOENIX_ENDPOINT` while your real credentials came from the environment, installing a config that ships spans and a bearer API key to an endpoint the repo picked, for every later session on that machine. Name the file you mean.
Content logging is **off by default** in this mode, unlike the interactive wizard where each question defaults to yes. A `[Y/n]` default is a person declining to change an answer they were shown; the same default unattended would capture prompts, commands, and file contents that nobody agreed to. Set the `ARIZE_LOG_*` variables you want to `true`.
The API key is never echoed. The installer reports only that it found one and where it came from, and every resolved value is reported with its source, so a wrong-credentials install stays diagnosable:
```
[arize] Backend: Arize AX at otlp.arize.com:443 (from default)
[arize] space ID: my-space (from /path/to/.env)
[arize] API key: found (from /path/to/.env)
[arize] Project name: codex (from default)
```
### Check what's installed
`status` reports which harnesses are configured and whether their hooks are actually wired into each harness's own settings file. Both have to be true for traces to appear.
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
./install.sh status
./install.sh status --json # machine-readable
```
`hooks: NOT registered` means credentials are saved but the harness was never wired up, or something removed the hooks. Re-run the install for that harness.
Use `--json` from a script or a coding agent to gate on the exit code without parsing output: `0` means every configured harness is wired up, `1` means nothing is configured, and `2` means at least one harness's hooks are missing. The payload contains no secrets — an API key appears only as `"api_key_present": true` — so it is safe to paste into a bug report.
### Keep it up to date
`update` pulls the latest code and re-registers every harness already in `config.json`.
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
./install.sh update
```
Re-registering runs each harness's installer, so in a terminal it still asks for each project name. With no terminal to answer on, in CI or a cron job, it takes the stored values instead of failing: credentials are not re-read on that path, and the project name keeps whatever is in `config.json`.
## Configuration
Credentials live in `~/.arize/harness/config.json`. Environment variables override values in `config.json` and can be set in your shell profile so they apply to every Devin session.
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
export ARIZE_API_KEY=""
export ARIZE_SPACE_ID=""
export ARIZE_PROJECT_NAME="devin"
export ARIZE_TRACE_ENABLED="true"
```
### Redaction controls
Each `ARIZE_LOG_*` flag accepts `"true"` or `"false"` and defaults to `"true"`. Set to `"false"` to opt out per category:
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
export ARIZE_LOG_PROMPTS="false"
export ARIZE_LOG_TOOL_DETAILS="false"
export ARIZE_LOG_TOOL_CONTENT="false"
```
| Flag | Redacts |
| :----------------------- | :-------------------------------------- |
| `ARIZE_LOG_PROMPTS` | User prompt and assistant response text |
| `ARIZE_LOG_TOOL_DETAILS` | Tool names and arguments |
| `ARIZE_LOG_TOOL_CONTENT` | Tool call output content |
### Default settings
| Setting | Default |
| :--------------------- | :--------------------------------------------------------------------- |
| Harness key | `devin` |
| Project name | `devin` |
| Arize AX endpoint | `otlp.arize.com:443` |
| Phoenix endpoint | `http://localhost:6006` |
| Hook config file | `~/.config/devin/config.json` (Windows: `%APPDATA%\devin\config.json`) |
| Hook events registered | `Stop`, `SessionEnd` |
| Data source | `~/.local/share/devin/cli/sessions.db`, opened read-only |
| State directory | `~/.arize/harness/state/devin/` |
| Log file | `~/.arize/harness/logs/devin.log` |
On Windows only the config file moves, to `%APPDATA%`. The sessions database stays home-relative at `%USERPROFILE%\.local\share\devin\cli\sessions.db`, the same layout as macOS and Linux. If traces do not appear, check `~/.arize/harness/logs/devin.log` for the database path the hook tried.
## Observe
Once tracing is enabled, Devin activity is streamed to Arize AX. Each agent interaction is captured as a trace.
### Spans Captured
* **Interaction traces** — one root AGENT span per agent response, carrying the user prompt as input, the final assistant text as output, the model name, and interaction token totals
* **LLM spans** — one per real model generation, with that generation's prompt, completion, and cache token counts, model name, and reasoning content
* **Tool spans** — one per tool call, parented to the LLM span that issued it, with the serialized tool arguments as input
* **Session grouping** — interactions from the same session grouped by `session.id`
Real generations are deduped by `metadata.request_id`. Devin rebuilds the message chain as the conversation grows, so the same generation reappears under new node IDs, and a per-session, per-request watermark ensures each generation is emitted exactly once. The database is opened read-only in WAL-respecting mode, so the newest turn's rows are visible without disturbing Devin's writers.
### Span shape
The root AGENT span carries `session.id`, `input.value`, `output.value`, `llm.model_name`, the `llm.token_count.*` totals (including `prompt_details.cache_read` and `prompt_details.cache_write`), `project.name`, an optional `user.id`, and `devin.backend` for the agent backend.
### What does not carry over
* **LLM spans do not carry `input.value`.** The per-generation prompt messages sent to the model are not reconstructed from the database, so the interaction's user prompt lives on the root AGENT span instead. A generation that issued only tool calls, with no assistant text or reasoning, has an empty `output.value`; its visible output appears on the later generation that answers the user. No output is lost, it is attributed to the generation that produced it.
* **Tool output is best-effort.** Tool results are not reliably present in the live database at the moment `Stop` fires, so a TOOL span may carry only its input arguments.
## Reference
For the full list of environment variables, default file paths, and troubleshooting steps, see the [Devin tracing README](https://github.com/Arize-ai/coding-harness-tracing/blob/main/tracing/devin/README.md).
Errors always land in `~/.arize/harness/logs/devin.log`. Set `export ARIZE_VERBOSE=true` before launching Devin to also log routine hook activity.
## Uninstall
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
curl -sSL https://raw.githubusercontent.com/Arize-ai/coding-harness-tracing/main/install.sh | bash -s -- uninstall devin
```
Uninstall removes only the `Stop` and `SessionEnd` hook entries the installer added, leaving any hooks you added yourself untouched.
## Resources