Sync traces with Data Fabric

​

Why is this better than a standard data export?

• No lock-in: Your data is always available to you. You can move it to data warehouse of your choice so you can use it in other tools you already use.
• Single source of truth, always: No need to maintain separate copies or manage export jobs
• Automatic updates: Your data, up-to-date. Any updates via evaluations or annotations, even on months-old data, are captured in the next sync regardless of timestamp. Data syncs every 60 minutes.
• Query-ready format: Data is stored in Iceberg format for direct querying in BigQuery, Snowflake, and other data warehouses
• Time-partitioned: Leverages Hive standard storage for efficient time-based queries

​

Query and analyze synced traces

1. Cost. Join against your cloud billing export to see which prompt templates and sessions drive spend.
2. Infrastructure. Join against your warehouse’s INFORMATION_SCHEMA to tell model latency apart from infrastructure contention, so you tune the right layer.
3. Customer impact. Join against CRM and revenue data to measure AI spend relative to account value and correlate evaluation scores with outcomes like CSAT.

​

How does Data Fabric work?

• Data Fabric is only enabled for Enterprise accounts. Reach out to Arize Support (support@arize.com) if you’d like to trial access.
• To set up Data Fabric, you must have write permissions to your target cloud storage bucket and at least one tracing project in your space
• Connectors are a connection to a filepath within your bucket. When you create a connector, you’ll be able to specify a bucket and namespace, as well as projects to sync. You can add any number of projects or create one connector per project. Each connector must have a unique filepath.
• Once you’ve created your connectors and added your projects, your data will sync automatically every 60 minutes. This includes updates to historical data that may have changed.
• Project syncs can be paused, resumed or deleted as needed.
• Supported Blobstore Providers: Google Cloud Storage (GCS) and Amazon S3
  • Coming Soon: Azure Storage
• Supported Big Table Providers: BigQuery (GCS), Databricks Unity Catalog
  • Coming Soon: Snowflake

​

Setting up Data Fabric

​

Step 1: Create a Data Connector

1. Navigate to Settings > Data Fabric in your space
2. Click New Connector
3. Fill out the basic connector information:
   1. Connector Name: A descriptive name for your connector
   2. Select Projects: Choose which tracing projects to sync. You can modify these projects later

​

Step 2: Configure Cloud Storage

1. Select Data Storage: Choose your cloud storage provider.
2. File Path: Enter your bucket path in the format shown below.

​

Step 3: Set Up Permissions

gcloud iam roles create arizeDataFabric \
    --project=YOUR_PROJECT_ID \
    --title="Arize Data Fabric Role" \
    --description="Custom IAM role for Arize Data Fabric" \
    --permissions=storage.buckets.get,storage.objects.get,storage.objects.list,storage.objects.create,storage.objects.update,storage.objects.delete \
    --stage=ALPHA

gsutil iam ch serviceAccount:arize-data-fabric@production-269901.iam.gserviceaccount.com:projects/<YOUR_PROJECT_ID>/roles/arizeDataFabric <YOUR_FILEPATH>

aws s3api put-bucket-tagging \
    --bucket YOUR_BUCKET_NAME \
    --tagging 'TagSet=[{Key=arize-ingestion-key,Value=YOUR_VALUE_FROM_SETUP_DIALOG}]'

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:GetBucketTagging"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR_BUCKET_NAME",
        "arn:aws:s3:::YOUR_BUCKET_NAME/*"
      ]
    }
  ]
}

aws iam put-role-policy \
    --role-name arize-data-fabric \
    --policy-name arize-data-fabric-s3-access \
    --policy-document file://policy.json

​

Step 4: Validate and Start Sync

1. Validate: Click Validate to verify your configuration
2. Start Syncing: Once validated, click Start Job to begin syncing. Your first sync will begin immediately and then continue every 60 minutes

​

Step 5: Set Up Query Tables

1. Allow the initial sync to complete: Allow the first sync to complete. Sync time depends on data size and shape, and may vary.
2. Create Table: Once your data is syncing, create external tables to query the data directly. For each project being synced, create an external table using Iceberg or Delta format:

CREATE EXTERNAL TABLE `your-project.your-dataset.your-table`
OPTIONS (
   format = 'ICEBERG',
   uris = ['gs://your-bucket/path/namespace/project-name/metadata/latest.metadata.json']
);

​

Frequently Asked Questions

• How often does data sync? Data syncs every 60 minutes automatically.
• Can I sync multiple spaces to the same bucket? Yes, you can configure multiple connectors to write to the same bucket using different namespaces or prefixes. For stronger access isolation, we recommend using separate buckets or prefixes per space. See Access Control Best Practices for details.
• What happens if I delete a project that’s being synced? The sync will stop for that project, but existing data in your storage will remain.
• Can I change the sync frequency?
  Currently, the sync frequency is fixed at 60 minutes and cannot be customized. Customization is coming soon.\
• Is historical data included in the sync? Yes, all historical data is included in the initial sync, and any changes to historical data between syncs will be included in the next sync.
• What’s the difference between Data Fabric and manual exports? Data Fabric provides automatic, continuous syncing with evaluations and annotations, while exports are manual snapshots at a point in time.