Cost Tracking

Cost tracking is essential for understanding and managing the spend of your LLM-powered applications. Arize AX provides a flexible, powerful, and easy-to-configure system to track model usage costs across providers and model variants — whether you're using default pricing or defining custom rates.

How Cost Tracking Works

Cost tracking works by ingesting token usage metrics and applying the correct cost configuration based on the model and provider.

Lookup Logic

To determine cost:

1. We extract the model name from your trace using the following fallback order:

   • llm.model_name (Primary)

   • llm.invocation_parameters.model (Fallback 1)

   • metadata.model (Fallback 2)

2. Optionally, if you provide a provider, we’ll match that as well (e.g., differentiating OpenAI vs Azure OpenAI for gpt-4).

3. Each token type (e.g., prompt, completion, audio) is matched against the configuration, and the cost is calculated per million tokens (1M token unit basis).

Important: Cost is not retroactive. To track costs, you must configure pricing before ingesting traces.

Supported Token Types and Semantic Conventions

You can send any token types using OpenInference semantic conventions. Below are the supported fields:

Prompt Tokens

Token Type	Field Name
Prompt (Includes all input subtypes to LLM)	llm.token_count.prompt
Prompt Details	llm.token_count.prompt_details
Audio	llm.token_count.prompt_details.audio
Image	llm.token_count.prompt_details.image
Cache Input	llm.token_count.prompt_details.cache_input
Cache Read	llm.token_count.prompt_details.cache_read
Cache Write	llm.token_count.prompt_details.cache_write

Completion Tokens

Token Type	Field Name
Completion (Includes all output subtypes from LLM)	llm.token_count.completion
Audio	llm.token_count.completion_details.audio
Reasoning	llm.token_count.completion_details.reasoning
Image	llm.token_count.completion_details.image

Total Tokens (Optional)

• llm.token_count.total

Custom Token Types

You can also define custom token types under either prompt_details or completion_details. Just make sure to:

• Use semantic naming

• Include a matching token type and cost in your configuration

Each token sent will have a cost calculated provided a matching token type is defined in your configuration.