What “self-hosted” actually means in AI observability

Direct answer: Self-hosted AI observability means the systems that collect, store, process, evaluate, and display AI telemetry run in infrastructure your organization controls. The label alone is insufficient: teams should verify where evaluations, the UI, identity, usage telemetry, licensing, and support access operate (and what data crosses the network boundary).

Why the label causes confusion

Your security team asks you a straightforward question: does any production trace leave our network?

The observability vendor says its platform is self-hosted. Then the architecture review reveals that only the trace collector and database run in your VPC. The evaluation service, UI, identity layer, and usage telemetry still communicate with the vendor’s cloud.

That architecture may be acceptable, but it’s not the deployment your team thought it was evaluating.

“Self-hosted” is not a precise architectural term. This is something we encounter at Arize all the time when talking with engineering leaders who are evaluating solutions (no pun intended). In our experience, we see vendors use it to describe everything from a Docker container running on a developer laptop to a fully private deployment operated inside a restricted Kubernetes cluster.

Instead of asking whether a platform is self-hosted, you should ask four more useful questions:

1. Where does each component run?
2. What data crosses the network boundary?
3. Who can access and operate the system?
4. What stops working when the vendor’s cloud is unavailable?

The answers will tell you more than the deployment label ever will.

Why we have a point of view on self-hosting

At Arize, we develop AI observability platforms across two very different self-hosted models:

• Arize AX is our enterprise platform, available as a private deployment in a customer-managed cloud, data center, or air-gapped environment.
• Phoenix is our open-source platform, which teams can run locally or deploy in their own infrastructure using Docker, Kubernetes, Helm, and other options.

Building and supporting both has given us a close view of the questions teams encounter as they move from local experimentation to production: what runs where, which data crosses a network boundary, who owns upgrades and reliability, and what enterprise controls must be in place.

This article reflects what we have learned helping teams evaluate those tradeoffs, rather than treating “self-hosted” as a single, well-defined architecture.

Why AI observability data deserves a closer look

Traditional logs and traces are not inherently harmless. They can contain request bodies, query parameters, user identifiers, database values, and other application data.

AI observability adds several new data types to that mix:

• Prompts and model responses
• Retrieved documents and search results
• Tool-call arguments and outputs
• Agent state and execution steps
• Prompt templates and system instructions
• Evaluation inputs, scores, labels, and comments
• User, account, session, and tenant identifiers

Consider an agent that handles refund requests. A single trace might include the customer’s original message, account information retrieved from a CRM, the prompt used to determine eligibility, arguments sent to a refund API, the API response, and an evaluation label indicating whether the decision followed company policy.

Keeping the trace database inside your VPC does not automatically mean every one of those fields stays there. A hosted evaluation service might process the customer message. A hosted UI might render the API response. Diagnostic telemetry might include tenant identifiers, error messages, or schema information.

Some agent frameworks also record planner output, scratchpad content, or intermediate application state. That is not necessarily a model’s private chain of thought, but it can still reveal proprietary logic and sensitive data.

This is why deployment reviews need to go beyond the location of the primary database.

Start with the components, not the label

Before comparing deployment models, you should map the major parts of the platform.

An AI observability system may include:

• Instrumentation libraries or SDKs
• A collector or ingestion endpoint
• Queues and processing workers
• Trace and span storage
• Object storage
• Evaluation workers
• Prompt and dataset storage
• A web application and API
• Authentication and authorization services
• Audit logging
• Licensing and usage-metering services
• Update and package distribution services
• Connections to model providers

Vendors may group these components into a “data plane” and a “control plane,” but those terms are not standardized. One vendor may run evaluations in the data plane while another may treat them as a hosted control-plane service.

You should always ask for a component-level architecture diagram rather than relying on those labels.

For every component, document:

• Where it runs
• What data it reads
• What data it stores
• Which outbound connections it requires
• Who is responsible for operating it

It also helps to treat four commonly conflated properties as separate questions:

• Data residency: Where is the data stored?
• Processing location: Where are traces, evaluations, and derived metrics computed?
• Administrative access: Who can view or operate the system?
• Network egress: What information is transmitted outside the environment?

A deployment can satisfy one of these requirements without satisfying the others.

The common AI observability deployment models

Managed SaaS

In a managed SaaS deployment, the vendor runs most of the platform: ingestion, storage, evaluation services, the UI, authentication, upgrades, and supporting infrastructure.

Your application may send telemetry directly to the vendor, or it may send data through a collector running in your environment.

This is often the fastest deployment model. It avoids database administration, cluster upgrades, backup management, capacity planning, and most application maintenance.

Before choosing it, you should verify the following things:

• Which fields the SDK exports
• Whether data can be redacted before transmission
• Where traces and backups are stored
• Where evaluations run
• Whether evaluation jobs call additional model providers
• What retention and deletion controls are available
• How tenant access and vendor support access are audited

Managed SaaS is a reasonable choice when the organization permits the relevant data to leave its environment and the cost of operating another production system would outweigh the benefits of local control.

Open-source, self-managed

This is the model many teams start with: run an open-source observability tool locally, connect an application, and inspect traces within a few minutes.

That is useful for prototyping and development. It also proves that the software can run without a vendor-managed application.

It does not, by itself, answer whether the system is ready for production.

Before using an open-source deployment as shared infrastructure, you should verify its support for:

• High availability
• Authentication and authorization
• SSO integration
• Service accounts
• Tenant isolation
• Audit events
• Backups and restoration
• Database and schema migrations
• Upgrade and rollback procedures
• Data retention and deletion
• Capacity planning
• Security updates

Some open-source projects provide many of these capabilities. Others require additional infrastructure, custom engineering, or commercial components.

The practical question is whether your team can operate it at the reliability and security level the application requires.

Hybrid: traces stay local, but some services do not

In a hybrid deployment, the collector and trace store typically run in your environment while some combination of the UI, evaluation scheduler, identity service, management API, or analytics services runs in the vendor’s cloud.

The primary benefit is that raw trace storage can remain inside your network and the maintenance overhead of the control plane is managed by the vendor. The tradeoff is that locally stored data may still be processed, summarized, or displayed by external services.

For example, a hosted evaluation service may need to receive an input, output, or retrieved document to score a trace. A hosted UI may receive selected trace fields when a user opens a session. The vendor may also collect usage metrics, error reports, tenant identifiers, or compatibility information.

Before selecting a hybrid model, you should ask:

• Which trace fields are sent to hosted services?
• Do evaluation inputs or outputs leave the environment?
• Where are prompt templates and evaluation datasets stored?
• Does the hosted UI receive raw values or only derived metrics?
• What is included in diagnostic and usage telemetry?
• Can the local services operate when the vendor’s cloud is unavailable?
• How long can the local and hosted components remain on different versions?
• Which component owns authentication, authorization, and audit history?

Hybrid deployment can work well when the main requirement is to keep raw trace storage inside the organization and some external processing is permitted.

It should not be treated as equivalent to a deployment in which every application component operates locally.

Commercial platform, self-deployed

In a self-deployed commercial model, most or all platform components run inside infrastructure controlled by your organization.

These platforms are often packaged for Kubernetes because it provides a familiar model for scheduling, storage, networking, and upgrades. Kubernetes, however, is a deployment mechanism and doesn’t offer proof that the application meets your security or reliability requirements.

Your team will usually own:

• Cluster and node capacity
• Storage configuration
• Ingress and certificates
• Secrets management
• Identity-provider integration
• Database backups
• Disaster recovery
• Application upgrades
• Version compatibility
• Monitoring the observability platform itself
• Incident response

This model provides the most direct control over data storage, application processing, and network access. It also transfers substantial operational responsibility to your platform or infrastructure team.

Before choosing it, determine whether the platform can operate under your actual network policy. A product that runs in your cluster but requires continuous access to a licensing server, hosted control plane, package registry, or telemetry endpoint may not satisfy a restricted-egress requirement.

Ask the vendor to document every required outbound endpoint, then test the system with those connections blocked.

Private deployment, vendor-provisioned

A vendor-provisioned private deployment has a similar target architecture to a self-deployed installation: the application, storage, and evaluation infrastructure run in your environment.

The difference is that vendor engineers assist with installation, sizing, validation, upgrades, or ongoing operations.

This can reduce deployment risk, especially in environments with custom networking, private registries, strict change controls, or limited Kubernetes experience. It can also introduce additional access paths that need to be reviewed.

Ask any vendor that claims to offer private deployments the following questions:

• Does the vendor receive persistent or temporary access?
• Who approves each support session?
• Are credentials time-limited?
• Is vendor activity logged?
• Can access be restricted to specific namespaces or services?
• Can the vendor troubleshoot without accessing trace content?
• Are diagnostic bundles reviewed or scrubbed before upload?
• Does the vendor operate through your identity and access-management system?
• Can the deployment function without vendor access?

The data-residency architecture may be the same as a self-deployed installation, but only when support access, diagnostic data, telemetry, and outbound dependencies are constrained accordingly.

Deployment models at a glance

These are typical patterns, not fixed definitions. The architecture diagram and network behavior should be the source of truth.

Questions to take into a vendor review

1. What does the platform collect?

Do not accept “telemetry” as a complete answer. Ask for the actual schema or an example payload.

Determine whether the platform collects:

• Full prompts and responses
• Retrieved documents
• Tool inputs and outputs
• Attachments
• Agent state
• Prompt templates
• Evaluation criteria and results
• User or customer identifiers
• Session and tenant identifiers
• Model-provider metadata

Ask whether you can allowlist fields or redact them before ingestion. Redacting sensitive content after it has crossed the network boundary does not solve an egress requirement.

2. Where does every component run?

Request a diagram that includes storage, processing, evaluations, authentication, the UI, licensing, telemetry, and update services.

For every component, ask what data it can access. A locally hosted database offers limited protection when a remote service has unrestricted access to query it.

3. What crosses the network boundary?

Separate raw data from derived data.

The platform may transmit:

• Full traces
• Redacted trace fields
• Evaluation inputs
• Evaluation outputs
• Aggregate metrics
• Prompt templates
• Error messages
• Tenant metadata
• Usage information
• Support bundles

A vendor can truthfully say that “raw traces remain in your VPC” while still exporting sensitive derived data or identifying metadata.

4. Which outbound dependencies remain?

Document dependencies on:

• Licensing endpoints
• Vendor control planes
• Container registries
• Package repositories
• Usage-telemetry services
• Identity providers
• Update services
• Model providers

Then ask what happens when each one is unavailable.

Can the platform still ingest traces? Can users access the UI? Can evaluations run? Is there a grace period for licensing failures? Can upgrades be performed through a private registry?

These questions matter in ordinary outages, not only in air-gapped environments.

5. Who can access the deployment?

Access is separate from residency. Data can remain in your account while still being accessible to an external operator.

Review:

• Employee access controls
• Support access
• Approval workflows
• Time-limited credentials
• Session logging
• Read-only versus administrative access
• Break-glass procedures
• Access to backups and diagnostic data

For sensitive deployments, vendor access should be explicit, limited, auditable, and revocable.

6. Who owns reliability?

Clarify responsibility for:

• Backups and restores
• High availability
• Database maintenance
• Capacity planning
• Security patches
• Version upgrades
• Rollbacks
• Disaster recovery
• Incident investigation
• Support escalation

“Supported by the vendor” can mean anything from access to documentation to a contractual recovery-time commitment. Get the support boundary in writing.

7. Can you leave?

Deployment decisions change. Make sure the platform does not turn your historical data into a migration obstacle.

Ask whether you can export:

• Traces and spans
• Evaluation datasets
• Evaluation results
• Prompt versions
• Annotations
• Custom metrics
• Dashboards
• Audit history

The export should preserve relationships between traces, spans, evaluations, datasets, and prompt versions. A collection of disconnected JSON files may technically be an export while still being difficult to use.

The product decision is not only about compliance

Security requirements may determine which architectures are allowed, but they do not determine which permitted architecture is the best product choice.

Product managers and engineering leads should also evaluate:

Time to value

How long will it take to instrument the application, provision infrastructure, configure access, and make the platform available to the team?

A self-deployed system that takes months to approve and operate may delay the evaluation loop the team was trying to improve.

Feature parity

Does the self-hosted version support the same evaluation workflows, dashboards, APIs, and integrations as the managed service?

Some vendors ship features to SaaS first. Others maintain the same application across deployment models. Ask directly rather than assuming.

Release cadence

How frequently must your team upgrade? Are security updates independent from feature releases? How much version skew is supported between collectors, storage, and the UI?

A weekly vendor release cadence can become an operational burden when every upgrade requires an internal change-management process.

Total cost

Infrastructure is only part of the cost.

Include:

• Platform engineering time
• Security-review time
• Storage
• Database operations
• Backups
• On-call responsibility
• Upgrade testing
• Incident response
• Vendor support

A license that appears cheaper can become more expensive once the organization owns the operating model.

Reversibility

Can the team move from SaaS to private deployment, or from self-hosted back to SaaS, without reinstrumenting every application?

Compatible SDKs, stable APIs, open telemetry formats, and usable exports reduce the cost of changing the architecture later.

A practical decision guide

Use managed SaaS when your organization permits the relevant data to leave its environment, fast adoption matters, and the team does not want to operate another production platform.

Use an open-source local deployment for experimentation, development, and early validation. Evaluate production requirements separately before treating it as shared infrastructure.

Use a hybrid model when raw trace storage must remain local but external processing, identity, or application services are permitted.

Use a self-deployed commercial platform when data, processing, and administrative control need to remain inside your environment and your organization has the capacity to operate it.

Use a vendor-provisioned private deployment when you need a private architecture but want installation, validation, or ongoing operational help from the vendor.

For strict-egress environments, do not rely on documentation alone. Deploy the platform in a test environment, block outbound connections, and verify which functions continue to work.

How Arize approaches self-hosting

Arize develops AI observability platforms across two different self-hosted models. This gives our team a close view of the questions organizations encounter as they move from local experimentation to production.

Arize AX: The enterprise platform is available as a private deployment in a customer-managed cloud, data center, or air-gapped environment. Data, authentication, APIs, evaluations, and the UI can run inside the customer’s infrastructure.

Phoenix: The open-source platform can run locally or in a team’s own infrastructure using Docker, Kubernetes, Helm, and other deployment options.

The implementation details still matter. Teams should validate optional outbound integrations, model-provider calls, image and update distribution, support access, and operational ownership against their own policy.

When to revisit the decision

A deployment model does not have to be permanent.

Teams commonly revisit it when:

• A new use case introduces more sensitive data
• A customer contract adds storage, processing, or access requirements
• Internal security policy changes
• The platform becomes important enough to require a stronger SLA
• Self-hosting consumes more engineering time than expected
• The team needs features available only in another deployment model
• An acquisition or organizational change alters infrastructure ownership
• A restricted-egress or disconnected environment becomes necessary

Plan for those transitions before production data accumulates. Standard instrumentation, documented data schemas, and tested exports make future migrations much less painful.

Ask what runs where instead of asking whether a solution is self-hosted

“Self-hosted” is a starting point for an architecture review, not the conclusion of one.

Before choosing a deployment, map every platform component and document:

• Where it runs
• What data it reads and stores
• What it sends outside your environment
• Who can access it
• Who owns upgrades and recovery
• Which external services it depends on

Then choose the architecture whose security properties and operational burden your organization can actually support.

A managed platform may be the right answer when speed and low operational overhead matter most. A private deployment may be necessary when data, application logic, and administrative access must remain inside a controlled environment. A hybrid model may be sufficient when local trace storage is the primary requirement.

The label matters less than the data flow. Get that architecture (and its outbound dependencies) in writing before production traces start moving through it.

Frequently asked questions

What is self-hosted AI observability?

Self-hosted AI observability is a deployment model in which the systems that collect, store, process, evaluate, and display AI telemetry run in infrastructure controlled by the organization using the platform. Teams should still verify licensing, updates, telemetry, model calls, and support access.

Is BYOC the same as fully self-hosted?

Not necessarily. Bring your own cloud often means the data plane runs in the customer’s cloud account while a vendor-hosted control plane provides the UI, identity, management, or updates. The exact data flows and access paths determine whether the architecture meets a private-deployment requirement.

Does local trace storage mean no AI data leaves the network?

No. A hosted evaluation service, web UI, model provider, diagnostic endpoint, or support workflow may still receive prompts, responses, retrieved documents, tool outputs, derived metrics, or tenant metadata even when the primary trace database is local.

Can AI observability run in an air-gapped environment?

Yes, if the application, storage, identity, evaluations, licensing behavior, update distribution, documentation, and required dependencies can operate without public network access. Test this by blocking outbound connectivity in a representative environment.

What should teams test before choosing a self-hosted platform?

Map every component, inspect the telemetry schema, inventory outbound endpoints and payloads, test vendor-cloud outages, review support access, restore a backup, perform an upgrade and rollback, and export connected traces, evaluations, datasets, and prompts.