Skip to main content
adb ingests both large file uploads and continuous streams of trace events at scale, terabytes a day across billions of spans. The goal for streaming is simple: a record should be queryable from the moment it arrives in memory, not after some batch job runs.

​
The ingestion flow

Incoming records accumulate in memory, where they immediately become available to queries. Periodically they’re flushed to local disk, and from there synchronized to cloud blob storage for durable, long-term storage. A record is queryable at the first stage of that journey, so freshness doesn’t wait on the flush to object storage.

​
File formats: in flight vs. at rest

adb deliberately uses different columnar formats for data in motion and data at rest:
  • Arrow IPC, in flight. Uncompressed columnar format optimized for zero-copy deserialization, so records move between memory and clients with no translation cost.
  • Parquet, at rest. Column-compressed format for durable, space-efficient long-term storage in blob storage.
  • Arrow Flight, transport. Built on gRPC (HTTP/2), Flight lets clients read data copy-free and serialization-free.

​
Dynamic schemas in flight

Trace ingestion is a stream of individual records whose schemas vary, a requirement standard Arrow and Flight implementations don’t handle. Arize reengineered the low-level Arrow primitives to support dynamic schemas in flight while keeping the data Arrow-compatible when persisted, so variable-shape events can stream in without a rigid predefined schema.

​
Delivery guarantees

adb offers two levels of guarantee for streamed records:
  • At-least-once delivery. adb streams records with an at-least-once guarantee: a record is acknowledged only after it has been durably persisted, and if a client doesn’t receive an acknowledgment it retries, so no record is silently lost.
  • Effectively-once ingest. adb keeps track of what it has already saved, so if part of the pipeline retries after a failure, it picks up where it left off instead of writing the same record twice. You get no lost records and no accidental duplicates.
A distributed write-ahead log (WAL) sits behind the stateless network edge. It lets clients meet their retry and persistence requirements before records are fully ingested into adb, keeping the ingestion edge stateless without sacrificing durability.

Next: Performance & benchmarks