Haystack Tracing

Instrument LLM applications built with Haystack

Arize AX provides auto-instrumentation for Haystack applications, allowing you to trace and observe your Haystack Pipelines.

API Key Setup

Before running your application, ensure you have the following environment variables set:

export ARIZE_SPACE_ID="YOUR_ARIZE_SPACE_ID"
export ARIZE_API_KEY="YOUR_ARIZE_API_KEY"
export OPENAI_API_KEY="YOUR_OPENAI_API_KEY" # Needed for the OpenAIGenerator example
# Add other LLM provider API keys if used by Haystack components

You can find your Arize Space ID and API Key in your Arize account settings.

Install

Install Haystack, the OpenInference instrumentor for Haystack, Arize OTel, and supporting OpenTelemetry packages:

pip install haystack-ai openinference-instrumentation-haystack arize-otel

Setup Tracing

Connect to Arize AX using arize.otel.register and apply the HaystackInstrumentor.

import os
from arize.otel import register
from openinference.instrumentation.haystack import HaystackInstrumentor

# Setup OTel via Arize's convenience function
tracer_provider = register(
    space_id=os.getenv("ARIZE_SPACE_ID"),    # or directly pass your Space ID
    api_key=os.getenv("ARIZE_API_KEY"),      # or directly pass your API Key
    project_name="my-haystack-app"         # Choose a project name
)

# Instrument Haystack
HaystackInstrumentor().instrument(tracer_provider=tracer_provider)

print("Haystack instrumented for Arize.")

Arize's auto-instrumentor primarily collects traces from Haystack Pipelines. If you are using Haystack components outside of a Pipeline, you may need to manually instrument those parts or ensure the components themselves are instrumented if they make calls to instrumented LLMs (e.g., an instrumented OpenAI client).

Run Haystack Example

Here's how to set up and run a simple Haystack Pipeline. The instrumentor will capture traces from this pipeline.

# Ensure OPENAI_API_KEY is set in your environment for this example
from haystack import Pipeline
from haystack.components.generators import OpenAIGenerator
from haystack.components.builders.prompt_builder import PromptBuilder # Added for a more complete example

# Define a prompt template
prompt_template = """
Answer the following question based on your knowledge.
Question: {{question}}
Answer:
"""

# Initialize the pipeline
pipeline = Pipeline()

# Initialize Haystack components
prompt_builder = PromptBuilder(template=prompt_template)
llm = OpenAIGenerator(model="gpt-3.5-turbo") # Uses OPENAI_API_KEY

# Add components to the pipeline
pipeline.add_component(name="prompt_builder", instance=prompt_builder)
pipeline.add_component(name="llm", instance=llm)

# Connect the components
pipeline.connect("prompt_builder.prompt", "llm.prompt")

# Define the question
question_to_ask = "What is the location of the Hanging Gardens of Babylon?"

# Run the pipeline
response = pipeline.run({
    "prompt_builder": {"question": question_to_ask}
})

# Print the response from the LLM
if response and "llm" in response and "replies" in response["llm"]:
    print(f"Question: {question_to_ask}")
    print(f"Answer: {response['llm']['replies'][0]}")
else:
    print(f"Failed to get a response or response format is unexpected: {response}")

Observe in Arize AX

After running your Haystack Pipeline, traces will be sent to your Arize project. Log in to Arize AX to:

• Visualize the execution of your Haystack Pipeline, including each component.

• Inspect the inputs, outputs, and parameters of each component.

• Analyze latency and identify bottlenecks.

• Trace errors and exceptions through the pipeline.

Resources

• Haystack Documentation

• OpenInference Haystack Instrumentor

• Haystack Examples on OpenInference GitHub (includes more complex examples like RAG)