Import your data

• Import Inference data (CV, NLP)

• Import Prompt and Response (LLM) data

• Import Retrieval data

• Import Corpus (Vector Store) data

Manage the App

• Define your dataset(s)

• How to launch the app

• How to view the UI

• How to close the app

How to export data

How to export your data for labeling, evaluation, or fine-tuning

• How to export embeddings

• How to export a cluster

• How to export all clusters

How to generate embeddings

• Auto-embeddings

• How to enable auto-embeddings

• How to implement auto-embeddings

Exporting Embeddings

Embeddings can be extremely useful for fine-tuning. There are two ways to export your embeddings from the Phoenix UI.

Export Selected Clusters

To export a cluster (either selected via the lasso tool or via a the cluster list on the right hand panel), click on the export button on the top left of the bottom slide-out.

Export All Clusters

To export all clusters of embeddings as a single dataframe (labeled by cluster), click the ... icon on the top right of the screen and click export. Your data will be available either as a Parquet file or is available back in your notebook via your session as a dataframe.

session = px.active_session()
session.exports[-1].dataframe

Phoenix ships with a collection of examples so you can quickly try out the app on concrete use-cases. This guide shows you how to download, inspect, and launch the app with example inferences.

View Available Inferences

To see a list of inferences available for download, run

px.load_example?

This displays the docstring for the phoenix.load_example function, which contain a list of inferences available for download.

Download Your Inference set of Choice

Choose the name of an inference set to download and pass it as an argument to phoenix.load_example. For example, run the following to download production and training data for our demo sentiment classification model:

inferences = px.load_example("sentiment_classification_language_drift")
inferences

px.load_example returns your downloaded data in the form of an ExampleInferences instance. After running the code above, you should see the following in your cell output.

ExampleInferences(primary=<Inferences "sentiment_classification_language_drift_primary">, reference=<Inferences "sentiment_classification_language_drift_reference">)

Inspect Your Inferences

Next, inspect the name, dataframe, and schema that define your primary inferences. First, run

prim_ds = inferences.primary
prim_ds.name

to see the name of the inferences in your cell output:

'sentiment_classification_language_drift_primary'

Next, run

prim_ds.schema

to see your inferences' schema in the cell output:

Schema(prediction_id_column_name='prediction_id', timestamp_column_name='prediction_ts', feature_column_names=['reviewer_age', 'reviewer_gender', 'product_category', 'language'], tag_column_names=None, prediction_label_column_name='pred_label', prediction_score_column_name=None, actual_label_column_name='label', actual_score_column_name=None, embedding_feature_column_names={'text_embedding': EmbeddingColumnNames(vector_column_name='text_vector', raw_data_column_name='text', link_to_data_column_name=None)}, excluded_column_names=None)

Last, run

prim_ds.dataframe.info()

to get an overview of your inferences's underlying dataframe:

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 33411 entries, 2022-05-01 07:00:16+00:00 to 2022-06-01 07:00:16+00:00
Data columns (total 10 columns):
 #   Column            Non-Null Count  Dtype
---  ------            --------------  -----
 0   prediction_ts     33411 non-null  datetime64[ns, UTC]
 1   reviewer_age      33411 non-null  int16
 2   reviewer_gender   33411 non-null  object
 3   product_category  33411 non-null  object
 4   language          33411 non-null  object
 5   text              33411 non-null  object
 6   text_vector       33411 non-null  object
 7   label             33411 non-null  object
 8   pred_label        33411 non-null  object
 9   prediction_id     0 non-null      object
dtypes: datetime64[ns, UTC](1), int16(1), object(8)
memory usage: 2.6+ MB

Launch the App

Launch Phoenix with

px.launch_app(inferences.primary, inferences.reference)

Follow the instructions in the cell output to open the Phoenix UI in your notebook or in a separate browser tab.

View Available Traces

Phoenix supports LLM application Traces and has examples that you can take a look at as well.\

px.load_example_traces?

# Load up the LlamaIndex RAG example
px.launch_app(trace=px.load_example_traces("llama_index_rag"))

For the Retrieval-Augmented Generation (RAG) use case, see the Retrieval section.

Dataframe

Below shows a relevant subsection of the dataframe. The embedding of the prompt is also shown.

Schema

See Retrieval for the Retrieval-Augmented Generation (RAG) use case where relevant documents are retrieved for the question before constructing the context for the LLM.

primary_schema = Schema(
    prediction_id_column_name="id",
    prompt_column_names=EmbeddingColumnNames(
        vector_column_name="embedding",
        raw_data_column_name="prompt",
    )
    response_column_names="response",
)

Inferences

Define the inferences by pairing the dataframe with the schema.

primary_inferences = px.Inferences(primary_dataframe, primary_schema)

Application

session = px.launch_app(primary_inferences)

In Retrieval-Augmented Generation (RAG), the retrieval step returns from a (proprietary) knowledge base (a.k.a. ) a list of documents relevant to the user query, then the generation step adds the retrieved documents to the prompt context to improve response accuracy of the Large Language Model (LLM). The IDs of the retrieval documents along with the relevance scores, if present, can be imported into Phoenix as follows.

Dataframe

Below shows only the relevant subsection of the dataframe. The retrieved_document_ids should matched the ids in the data. Note that for each row, the list under the relevance_scores column have a matching length as the one under the retrievals column. But it's not necessary for all retrieval lists to have the same length.

Schema

Both the retrievals and scores are grouped under prompt_column_names along with the embedding of the query.

Inferences

Define the inferences by pairing the dataframe with the schema.

Application

In , a document is any piece of information the user may want to retrieve, e.g. a paragraph, an article, or a Web page, and a collection of documents is referred to as the corpus. A corpus can provide the knowledge base (of proprietary data) for supplementing a user query in the prompt context to a Large Language Model (LLM) in the Retrieval-Augmented Generation (RAG) use case. Relevant documents are first based on the user query and its embedding, then the retrieved documents are combined with the query to construct an augmented prompt for the LLM to provide a more accurate response incorporating information from the knowledge base. Corpus inferences can be imported into Phoenix as shown below.

Inferences

Below is an example dataframe containing Wikipedia articles along with its embedding vector.

Schema

Below is an appropriate schema for the dataframe above. It specifies the id column and that embedding belongs to text. Other columns, if exist, will be detected automatically, and need not be specified by the schema.

Inferences

Define the inferences by pairing the dataframe with the schema.

Application

The launcher accepts the corpus dataset through corpus= parameter.

This guide shows you how to define Phoenix inferences using your own data.

Once you have a pandas dataframe df containing your data and a schema object describing the format of your dataframe, you can define your Phoenix dataset either by running

ds = px.Inferences(df, schema)

or by optionally providing a name for your dataset that will appear in the UI:

ds = px.Inferences(df, schema, name="training")

As you can see, instantiating your dataset is the easy part. Before you run the code above, you must first wrangle your data into a pandas dataframe and then create a Phoenix schema to describe the format of your dataframe. The rest of this guide shows you how to match your schema to your dataframe with concrete examples.

Predictions and Actuals

Let's first see how to define a schema with predictions and actuals (Phoenix's nomenclature for ground truth). The example dataframe below contains inference data from a binary classification model trained to predict whether a user will click on an advertisement. The timestamps are datetime.datetime objects that represent the time at which each inference was made in production.

Dataframe

Schema

schema = px.Schema(
    timestamp_column_name="timestamp",
    prediction_score_column_name="prediction_score",
    prediction_label_column_name="prediction",
    actual_label_column_name="target",
)

This schema defines predicted and actual labels and scores, but you can run Phoenix with any subset of those fields, e.g., with only predicted labels.

Features and Tags

Phoenix accepts not only predictions and ground truth but also input features of your model and tags that describe your data. In the example below, features such as FICO score and merchant ID are used to predict whether a credit card transaction is legitimate or fraudulent. In contrast, tags such as age and gender are not model inputs, but are used to filter your data and analyze meaningful cohorts in the app.

Dataframe

Schema

schema = px.Schema(
    prediction_label_column_name="predicted",
    actual_label_column_name="target",
    feature_column_names=[
        "fico_score",
        "merchant_id",
        "loan_amount",
        "annual_income",
        "home_ownership",
        "num_credit_lines",
        "inquests_in_last_6_months",
        "months_since_last_delinquency",
    ],
    tag_column_names=[
        "age",
        "gender",
    ],
)

Implicit Features

If your data has a large number of features, it can be inconvenient to list them all. For example, the breast cancer dataset below contains 30 features that can be used to predict whether a breast mass is malignant or benign. Instead of explicitly listing each feature, you can leave the feature_column_names field of your schema set to its default value of None, in which case, any columns of your dataframe that do not appear in your schema are implicitly assumed to be features.

Dataframe

Schema

schema = px.Schema(
    prediction_label_column_name="predicted",
    actual_label_column_name="target",
)

Excluded Columns

You can tell Phoenix to ignore certain columns of your dataframe when implicitly inferring features by adding those column names to the excluded_column_names field of your schema. The dataframe below contains all the same data as the breast cancer dataset above, in addition to "hospital" and "insurance_provider" fields that are not features of your model. Explicitly exclude these fields, otherwise, Phoenix will assume that they are features.

Dataframe

Schema

schema = px.Schema(
    prediction_label_column_name="predicted",
    actual_label_column_name="target",
    excluded_column_names=[
        "hospital",
        "insurance_provider",
    ],
)

Embedding Features

Embedding features consist of vector data in addition to any unstructured data in the form of text or images that the vectors represent. Unlike normal features, a single embedding feature may span multiple columns of your dataframe. Use px.EmbeddingColumnNames to associate multiple dataframe columns with the same embedding feature.

Embedding Vectors

To define an embedding feature, you must at minimum provide Phoenix with the embedding vector data itself. Specify the dataframe column that contains this data in the vector_column_name field on px.EmbeddingColumnNames. For example, the dataframe below contains tabular credit card transaction data in addition to embedding vectors that represent each row. Notice that:

• Unlike other fields that take strings or lists of strings, the argument to embedding_feature_column_names is a dictionary.

• The key of this dictionary, "transaction_embedding," is not a column of your dataframe but is name you choose for your embedding feature that appears in the UI.

• The values of this dictionary are instances of px.EmbeddingColumnNames.

• Each entry in the "embedding_vector" column is a list of length 4.

Dataframe

Schema

schema = px.Schema(
    prediction_label_column_name="predicted",
    actual_label_column_name="target",
    embedding_feature_column_names={
        "transaction_embeddings": px.EmbeddingColumnNames(
            vector_column_name="embedding_vector"
        ),
    },
)

Embeddings of Images

If your embeddings represent images, you can provide links or local paths to image files you want to display in the app by using the link_to_data_column_name field on px.EmbeddingColumnNames. The following example contains data for an image classification model that detects product defects on an assembly line.

Dataframe

Schema

schema = px.Schema(
    actual_label_column_name="defective",
    embedding_feature_column_names={
        "image_embedding": px.EmbeddingColumnNames(
            vector_column_name="image_vector",
            link_to_data_column_name="image",
        ),
    },
)

Local Images

For local image data, we recommend the following steps to serve your images via a local HTTP server:

1. In your terminal, navigate to a directory containing your image data and run python -m http.server 8000.

2. Add URLs of the form "http://localhost:8000/rel/path/to/image.jpeg" to the appropriate column of your dataframe.

For example, suppose your HTTP server is running in a directory with the following contents:

.
└── image-data
    └── example_image.jpeg

Then your image URL would be http://localhost:8000/image-data/example_image.jpeg.

Embeddings of Text

If your embeddings represent pieces of text, you can display that text in the app by using the raw_data_column_name field on px.EmbeddingColumnNames. The embeddings below were generated by a sentiment classification model trained on product reviews.

Dataframe

Schema

schema = px.Schema(
    actual_label_column_name="sentiment",
    feature_column_names=[
        "category",
    ],
    tag_column_names=[
        "name",
    ],
    embedding_feature_column_names={
        "product_review_embeddings": px.EmbeddingColumnNames(
            vector_column_name="text_vector",
            raw_data_column_name="text",
        ),
    },
)

Multiple Embedding Features

Sometimes it is useful to have more than one embedding feature. The example below shows a multi-modal application in which one embedding represents the textual description and another embedding represents the image associated with products on an e-commerce site.

Dataframe

Schema

schema = px.Schema(
    tag_column_names=["name"],
    embedding_feature_column_names={
        "description_embedding": px.EmbeddingColumnNames(
            vector_column_name="description_vector",
            raw_data_column_name="description",
        ),
        "image_embedding": px.EmbeddingColumnNames(
            vector_column_name="image_vector",
            link_to_data_column_name="image",
        ),
    },
)

Define Your Inferences

To define inferences, you must load your data into a pandas dataframe and create a matching schema. If you have a dataframe prim_df and a matching prim_schema, you can define inferences named "primary" with

prim_ds = px.Inferences(prim_df, prim_schema, "primary")

If you additionally have a dataframe ref_df and a matching ref_schema, you can define a inference set named "reference" with

ref_ds = px.Inferences(ref_df, ref_schema, "reference")

See Corpus Data if you have corpus data for an Information Retrieval use case.

Launch the App

Use phoenix.launch_app to start your Phoenix session in the background. You can launch Phoenix with zero, one, or two inference sets.

Open the UI

You can view and interact with the Phoenix UI either directly in your notebook or in a separate browser tab or window.

session.url

session.view()

Close the App

When you're done using Phoenix, gracefully shut down your running background session with

px.close_app()

Phoenix supports any type of dense embedding generated for almost any type of data.

But what if I don't have embeddings handy? Well, that is not a problem. The model data can be analyzed by the embeddings Auto-Generated for Phoenix.

What are Auto-Embeddings?

Generating embeddings is likely another problem to solve, on top of ensuring your model is performing properly. With our Python , you can offload that task to the SDK and we will generate the embeddings for you. We use large, pre-trained models that will capture information from your inputs and encode it into embedding vectors.

We support generating embeddings for you for the following types of data:

• CV - Computer Vision

• NLP - Natural Language

• Tabular Data - Pandas Dataframes

We extract the embeddings in the appropriate way depending on your use case, and we return it to you to include in your pandas dataframe, which you can then analyze using Phoenix.

Auto-Embeddings works end-to-end, you don't have to worry about formatting your inputs for the correct model. By simply passing your input, an embedding will come out as a result. We take care of everything in between.

How to enable Auto-Embeddings?

If you want to use this functionality as part of our Python SDK, you need to install it with the extra dependencies using pip install arize[AutoEmbeddings].

Supported models

You can get an updated table listing of supported models by running the line below.

from arize.pandas.embeddings import EmbeddingGenerator

EmbeddingGenerator.list_pretrained_models()

How do they work?

Auto-Embeddings is designed to require minimal code from the user. We only require two steps:

1. Create the generator: you simply instantiate the generator using EmbeddingGenerator.from_use_case() and passing information about your use case, the model to use, and more options depending on the use case; see examples below.

2. Let Arize generate your embeddings: obtain your embeddings column by calling generator.generate_embedding() and passing the column containing your inputs; see examples below.

Use Case Examples

df = df.reset_index(drop=True)

from arize.pandas.embeddings import EmbeddingGenerator, UseCases

df = df.reset_index(drop=True)

generator = EmbeddingGenerator.from_use_case(
    use_case=UseCases.CV.IMAGE_CLASSIFICATION,
    model_name="google/vit-base-patch16-224-in21k",
    batch_size=100
)
df["image_vector"] = generator.generate_embeddings(
    local_image_path_col=df["local_path"]
)

from arize.pandas.embeddings import EmbeddingGenerator, UseCases

df = df.reset_index(drop=True)

generator = EmbeddingGenerator.from_use_case(
    use_case=UseCases.NLP.SEQUENCE_CLASSIFICATION,
    model_name="distilbert-base-uncased",
    tokenizer_max_length=512,
    batch_size=100
)
df["text_vector"] = generator.generate_embeddings(text_col=df["text"])

from arize.pandas.embeddings import EmbeddingGenerator, UseCases

df = df.reset_index(drop=True)
# Instantiate the embeddding generator
generator = EmbeddingGeneratorForTabularFeatures(
    model_name="distilbert-base-uncased",
    tokenizer_max_length=512
)

# Select the columns from your dataframe to consider
selected_cols = [...]

# (Optional) Provide a mapping for more verbose column names
column_name_map = {...: ...}

# Generate tabular embeddings and assign them to a new column
df["tabular_embedding_vector"] = generator.generate_embeddings(
    df,
    selected_columns=selected_cols,
    col_name_map=column_name_map # (OPTIONAL, can remove)
)