File Importer API

Use the file importer API for a direct path between your cloud storage and the Arize AX platform. While the Arize AX platform has an intuitive user interface for data ingestion workflows, the object store integration provides a direct way to easily set up jobs and automate job creation. For a brief overview of GraphQL itself, please consult our introduction.

Creating Import Jobs

To create an import job: you must complete the setup steps outlined for GCS here or AWS here to grant Arize access to your bucket.

Import jobs belong to a Space — which has a globally unique ID. To create an import job, you’ll need to retrieve your space ID from the platform URL: /organizations/:organization_id/spaces/:space_id.

Import Job Tips:

Create a reusable mutation to create a file import job that will read files from your bucket
Validate your file import job before you begin your file import job. Using two additional parameters - dryRun and fileName - as inputs when creating the import job mutation.
- dryRun(boolean): If true, the import job will be attempted but no changes will be written. The validationResult field may be requested to check the validation status.
- fileName(string): The file name to perform a dry run on. If a file name is not selected, the dry run will be performed on the first file.

Dry Run File Import Job
Create File Import Job Mutation

mutation dryRunMyImportJob($input: CreateFileImportJobInput!) {
  createFileImportJob(input: $input) {
    validationResult {
      validationStatus
      filePath
      error {
        code
        message
        row
      }
    }
  }
}

variables:

{
  "input": {
    "spaceId": "space_id",
    "modelName": "My Test Model",
    "modelType": "categorical",
    "modelEnvironmentName": "production",
    "blobStore": "S3",
    "bucketName": "testing-folder",
    "prefix": "myfolder/mysubfolder/",
    "schema": {
      "predictionId": "prediction_id",
      "predictionLabel": "PREDICTION",
      "actualLabel": "ACTUAL",
      "shapValues": "shap/",
      "tags": "tag/",
      "timestamp": "prediction_ts",
      "predictionScore": "PREDICTION_SCORE",
      "actualScore": "ACTUAL_SCORE"
    },
    "dryRun": true
  }
}

Alternatively, you may pass in the inputs directly:

mutation {
  createFileImportJob(input: {
    spaceId: "space_id",
    modelName: "My Test Model",
    modelType: categorical,
    modelEnvironmentName: production,
    blobStore: S3,
    bucketName: "testing-folder",
    prefix: "myfolder/mysubfolder/",
    schema: {
      predictionId: "prediction_id",
      predictionLabel: "PREDICTION",
      actualLabel: "ACTUAL",
      shapValues: "shap/",
      tags: "tag/",
      timestamp: "prediction_ts",
      predictionScore: "PREDICTION_SCORE",
      actualScore: "ACTUAL_SCORE"
    },
    dryRun: true
  }) {
    validationResult {
      validationStatus
      filePath
      error {
        code
        message
        row
      }
    }
  }
}

mutation createMyNewImportJob($input: CreateFileImportJobInput!) {
  createFileImportJob(input: $input) {
    fileImportJob {
      id
    }
  }
}

variables

{
  "input": {
    "spaceId": "space_id",
    "modelName": "My Test Model",
    "modelType": "categorical",
    "modelEnvironmentName": "production",
    "blobStore": "S3",
    "bucketName": "testing-folder",
    "prefix": "myfolder/mysubfolder/",
    "schema": {
      "predictionId": "prediction_id",
      "predictionLabel": "PREDICTION",
      "actualLabel": "ACTUAL",
      "shapValues": "shap/",
      "tags": "tag/",
      "timestamp": "prediction_ts",
      "predictionScore": "PREDICTION_SCORE",
      "actualScore": "ACTUAL_SCORE"
    }
  }
}

Alternatively, you may pass in the inputs directly:

mutation {
  createFileImportJob(input: {
    spaceId: "space_id",
    modelName: "My Test Model",
    modelType: categorical,
    modelEnvironmentName: production,
    blobStore: S3,
    bucketName: "testing-folder",
    prefix: "myfolder/mysubfolder/",
    schema: {
      predictionId: "prediction_id",
      predictionLabel: "PREDICTION",
      actualLabel: "ACTUAL",
      shapValues: "shap/",
      tags: "tag/",
      timestamp: "prediction_ts",
      predictionScore: "PREDICTION_SCORE",
      actualScore: "ACTUAL_SCORE"
    }
  }) {
    fileImportJob {
      id
    }
  }
}

The variables provided are an example of one particular mapping.For more information on mapping your file, please consult the file schema documentation located on within each data connector page or here on the available fields for all queries and mutations.

https://storage.googleapis.com/arize-phoenix-assets/assets/images/phoenix-docs-images/gc.ico

Google Colab

Querying for Import Jobs and Files

You can query for import jobs off of a Space node.

query {
  node(id: "space_id") {
    ... on Space {
      importJobs(first: 50) {
        edges {
          node {
            id
            modelName
            totalFilesCount
            schema {
              predictionId
              predictionLabel
            }
          }
        }
      }
    }
  }
}

Query fields within an import job given specific criteria using the FileImportJobConnection from a Space. If you have a large number of import jobs, you will have to use pagination to pull the complete list. This data can then be viewed or used in other queries. To view files that belong to an import job, query for files using an import job node since the files belong to an import job. For instance, to view failing files and their error message, use the FileImportJobFileConnection which returns files with a passed status or all files if no status is provided. You may additionally filter files based on a date range, providing an RFC3339 compliant date time shown below.

query {
  node(id: "import_job_id") {
    ... on FileImportJob {
      files(
        first: 25
        status: FAILED
        startTime: "2023-01-02T01:00:00Z"
        endTime: "2023-02-01T01:00:00Z"
      ) {
        edges {
          node {
            id
            filePath
            error {
              message
            }
          }
        }
      }
    }
  }
}

Google Colab

Deleting Import Jobs

Deleting import jobs using a mutation simply requires you to know the ID of the import job you want to delete.

mutation {
  deleteFileImportJob(input: {jobId: "job_id"}) {
    space {
      id
    }
  }
}

Examples

API Use Cases	Example Colabs
Creating a File Import Job	Colab
Querying for Failed Files of an Import Job	Colab

API reference

API clients

Creating Import Jobs

Import Job Tips:

Querying for Import Jobs and Files

Deleting Import Jobs

Examples

​Creating Import Jobs

​Import Job Tips:

​Querying for Import Jobs and Files

​Deleting Import Jobs

​Examples

Creating Import Jobs

Import Job Tips:

Querying for Import Jobs and Files

Deleting Import Jobs

Examples