AI & agent engineering glossary

A

A/B testing for LLMs

A/B testing for LLMs splits live traffic between two or more versions of an AI system and compares what happens on real user outcomes.

Access control (agents)

Access control for agents defines which users, agents, tools, data sources, and actions are allowed in a given context.

Accuracy

Accuracy is the measure of the number of correct predictions a model makes out of all predictions it produces.

Adaptive Knowledge Graph Memory

Adaptive knowledge graph memory is a shared, hierarchical knowledge graph that several LLM agents read from and write to instead of keeping separate, isolated transcripts.

AdaptThink

AdaptThink is a reinforcement learning framework that trains an LLM when to think deeply and when to respond immediately.

Agent

An agent is a software system in which a model decides what to do next, not just what to say.

Agent architecture

Agent architecture is the structural design of the system around the model: which components exist, what each one owns, how they connect, and how state moves between them.

Agent control loop

An agent control loop is the repeated cycle of observe state, decide the next action, execute that action, update state, and continue until a stopping condition is met.

Agent debugging

Debugging agents is the practice of tracing and evaluating an agent’s full execution path to find why it failed.

Agent drift

Agent drift is a measurable change in an agent’s behavior over time.

Agent engineering

Agent engineering is the discipline of building, evaluating, and operating systems in which a model decides what to do next.

Agent failure modes

Agent failure modes are the recurring ways agents break in production: wrong tool selection, invalid tool arguments, missing context, bad planning, retrieval failure, policy violation, loops, unsafe actions, and a weak final answer.

Agent feedback loop

An agent feedback loop is the operating cycle that turns production behavior into measurable improvement: capture traces, run evals, analyze failures, curate datasets, test changes, compare results, and deploy only when the evidence supports it.

Agent governance

Agent governance is the program that decides who is accountable for an AI agent, what it is permitted to do, what must be approved before it reaches production, what gets reviewed while it runs, and what record exists of all of it.

Agent lifecycle (dev → prod → improve)

The agent lifecycle is the full path from building an agent to running it in production and improving it over time.

Agent orchestration

Agent orchestration is the coordination layer that decides what runs next.

Agent policy layer

The agent policy layer defines what an agent is allowed to do and under what conditions, and it evaluates that question at runtime.

Agent reliability

Agent reliability is whether an agent completes the task it was given, correctly and within its budget, under ordinary conditions.

Agent security

Agent security is the practice of keeping an AI agent from taking unauthorized action or leaking data under adversarial conditions.

Agent state management

Agent state management is the handling of the information an agent needs across steps, turns, sessions, or workflows.

Agent supervision

Agent supervision is the set of controls used to observe, guide, constrain, and review agent behavior.

Agent workflow

An agent workflow is the defined sequence of steps an agent follows to complete a task.

Agent Workflow Memory (AWM)

Agent workflow memory is a technique where an agent extracts reusable routines from its own past successful runs and applies them to later tasks.

Agent-native evaluation

Agent-native evaluation is evaluation designed around how agents actually run, rather than evaluation designed for single model calls and then pointed at an agent.

Agent-run evaluation

Agent-run evaluation is the practice of scoring a single complete execution of an agent: everything that happened between the triggering input and the terminal state, including tool calls, retries, intermediate reasoning, and the state the agent carried forward.

Agent-to-agent evaluation

Agent-to-agent evaluation measures the interactions between agents in a multi-agent system.

Agent-to-agent protocol (A2A)

Agent-to-Agent Protocol (A2A) is an open communication framework that allows autonomous AI agents to talk to each other in a standardized, predictable way.

AgentBench

AgentBench is a benchmark suite that measures how well large language models perform as agents, meaning models that take actions in an environment over multiple turns rather than producing a single block of text.

Agentic Memory (A-MEM)

Agentic memory is long-term memory for an LLM agent that organizes itself instead of sitting in a flat buffer.

Agentic RAG

Agentic RAG is retrieval augmented generation where an agent decides how retrieval happens instead of following a fixed retrieve-then-generate path.

Agents that evaluate agents

Agents that evaluate agents are evaluators built as agents rather than as a single model call.

AI Evaluation

AI evaluation is the practice of measuring whether an AI system produces the output you need, on data you care about, at a rate you can defend.

AI improvement loop

An AI improvement loop is the process of using real system behavior to improve an AI application.

AI observability

AI observability is the practice of instrumenting an AI system so that you can determine, from recorded data, why it produced the output it did.

AI product quality

AI product quality is whether an AI feature is actually good for the people using it.

AlphaEvolve

AlphaEvolve is a coding agent that improves algorithms by evolving programs instead of training a model.

AM Thinking v1

AM-Thinking-v1 is a 32-billion-parameter open-source LLM that achieves strong reasoning ability by combining supervised fine-tuning and reinforcement learning in its training.

ARC-AGI-2

ARC-AGI-2 is the second generation of the Abstraction and Reasoning Corpus, a benchmark built to test whether a system can work out a rule it has never seen before and apply it correctly.

Audit logs

Audit logs are records of important system actions, decisions, and access events.

AutoAgents

AutoAgents is a research framework designed to automatically generate and coordinate multiple bespoke AI agents to collaboratively solve complex tasks.

Autoencoder

Typically used for tasks like sentence completion, sentence or token classification, autoencoding models are like a jigsaw puzzle solver.

Autonomous evaluation systems

An autonomous evaluation system runs, interprets, and acts on evaluations with limited human involvement.

Autoregressive Model

Autoregressive models like GPT-2 use the previous words (context) to predict the next word in a sentence.

B

Baseline

A baseline is the reference data or benchmark used to compare model performance against for monitoring purposes.

Baseline Distribution

A baseline distribution is a reference dataset used to compare against a model’s current production distribution.

Benchmark vs production evaluation

Benchmark evaluation measures performance on standardized tasks or public datasets.

Bias (AI evaluation)

Bias in AI evaluation refers to systematic differences in behavior or outcomes across groups, topics, languages, dialects, or contexts.

Binary Classification Model

Binary classification refers to machine learning models trained for tasks with exactly two class labels.

Binning

Binning is a way to group a number of continuous values together into smaller cohorts or “bins.” The technique helps reduce the cardinality of data by representing the points in intervals.

BLEU Score

BLEU is a metric that scores generated text by how much of its word sequence overlaps with one or more human reference texts.

C

Canary Deployment

A canary deployment is a method of testing a new model or model version where only a small subset of production data flows through the candidate before making a complete cutover.

Canary evaluation (AI systems)

Canary evaluation routes a small share of real traffic, or a limited set of users, through a new AI system version and evaluates the result before rolling out further.

Cascading failures

Cascading failures occur when one failure triggers additional failures across an agent workflow or a multi-agent system.

Checkpointing (agents)

Checkpointing is the practice of saving an agent’s state at defined points so the workflow can resume, audit, retry, or roll back after an interruption.

Chunking strategy

Chunking strategy is the way source documents are split into retrievable units for embedding, indexing, and RAG.

Classification Model

Classification models are used to predict categories or assign a class label.

Closed-loop agents

Closed-loop agents are agents wired into a feedback cycle: observe what happened, evaluate whether it was good, change the system, and deploy the change under policy.

Completeness

Completeness measures whether a single answer or agent run includes all the information or steps the task required.

Compliance (AI systems)

Compliance for AI systems is the practice of making AI behavior, data handling, access, monitoring, and documentation meet legal, regulatory, contractual, and organizational requirements.

Concept Drift

Concept drift is the shift in the statistical properties of the target or dependent variable(s), i.e.

Confusion Matrix

A confusion matrix provides a summary of all prediction results of a classification problem.

Context

Context is the information available to a model at the moment it generates a response, makes a decision, or chooses the next action.

Context relevance

Context relevance measures whether the retrieved or supplied context is useful for answering the query.

Continual learning

Continual learning is training a model on a sequence of tasks or data distributions over time so that it acquires new capability without losing what it already learned.

Continuous evaluation

Continuous evaluation means running evals as an always-on part of development and production rather than only before a release.

Continuous improvement for AI systems

Continuous improvement for AI systems is the practice of improving AI quality through ongoing measurement rather than one-time launch testing.

Correctness

Correctness measures whether an output is factually or logically right for the task.

Cosine Similarity

Cosine similarity measures how closely two vectors point in the same direction.

Coverage

Coverage measures how much of the expected behavior space an evaluation suite actually exercises.

Current Distribution

Current distribution refers to the statistical distribution, or shape, of the dataset being generated by a machine learning model in production.

D

Data and datasets layer

The data and datasets layer is the part of an evaluation system that stores and serves the examples used for testing, scoring, experimentation, and improvement.

Data Drift

Data drift, feature drift, covariate drift and input drift all refer to a shift in the statistical properties of the independent variable(s), i.e.

Data drift (for LLMs)

Data drift for an LLM system occurs when the inputs change.

Data leakage

Data leakage occurs when information crosses a boundary it was supposed to stay behind, and the result is a score that reflects access rather than ability.

Data Quality

Data quality is whether a dataset is fit for the job you are asking it to do.

Dataset curation

Dataset curation is the practice of selecting, cleaning, labeling, organizing, and maintaining the examples used for evaluation.

Deep Explainer (Deep SHAP)

Deep SHAP is a fast way to approximate SHAP values for neural networks.

Deep Learning

Machine learning is a subset of AI, and it consists of the techniques that enable computers to figure things out from the data and deliver AI applications.

Deep Learning Model

A deep learning model normally refers to a neural network, typically with more than two layers.

Deployment gating (for AI)

Deployment gating for AI is the practice of requiring quality checks to pass before a model, prompt, tool, retrieval, or orchestration change reaches users.

Disparate Impact

Disparate impact is a quantitative measure of the adverse treatment of protected classes that compares the pass rate, or positive outcome, of one group versus another.

Distributed tracing

Distributed tracing is the practice of following one request as it crosses process and service boundaries, by propagating a shared identifier alongside each call so that work done in one service can be attached to work done in another.

Drift

Drift is defined as the change in the data over time.

Durable execution

Durable execution is the ability for a workflow to survive time, retries, failures, restarts, and long-running operations without losing state.

E

EfficientLLM

EfficientLLM is a benchmarking initiative focused on measuring how resource-efficient different LLMs are, beyond just accuracy.

Embeddings

Embeddings are dense, low-dimensional representations of high-dimensional data.

Embeddings / vector search

Embeddings are numerical representations of text, images, or other data that capture semantic similarity.

Error analysis

Error analysis is the process of grouping failures into categories so a team knows what to fix.

Eval maturity model

An eval maturity model describes how a team’s evaluation practice becomes more systematic, automated, and connected to production workflows over time.

EvalOps (CI/CD for agents)

EvalOps is the operational practice of running evaluations continuously across the agent development and deployment lifecycle.

Evals as APIs

Evals as APIs means exposing evaluation results and workflows through programmable interfaces rather than only through dashboards or reports.

Evaluation as API

Evaluation as an API is the API-first form of evaluation infrastructure.

Evaluation as infrastructure

Evaluation as infrastructure means treating evaluation as a core system dependency, in the same category as logging, tracing, testing, and CI/CD.

Evaluation dataset

An evaluation dataset is a collection of examples used to test an AI system.

Evaluation drift

Evaluation drift occurs when an evaluator stops measuring the behavior the team actually cares about.

Evaluation gating

Evaluation gating is the use of eval results to allow, block, or require review before a change moves forward.

Evaluation harness

An evaluation harness is the operational system that turns evals into repeatable workflows and actions.

Evaluation metrics

Evaluation metrics are the numerical or categorical measures used to judge AI system behavior.

Evaluation pipeline

An evaluation pipeline is the repeatable workflow that takes evaluation inputs, runs scoring logic, stores results, and triggers follow-up actions.

Evaluation rubric

An evaluation rubric is a structured set of criteria used to score AI outputs or agent behavior.

Evaluation Store

An evaluation store, also sometimes referred to as an inference store, is a machine learning infrastructure tool used to monitor and improve model performance.

Evaluation Window

In machine learning monitoring and observability, the evaluation window is a plot of the period or duration of time against the metric being calculated.

Evaluation-driven development

Evaluation-driven development is the practice of writing the evals for an LLM or agent feature before or alongside the feature itself, then letting those eval results decide what you change next.

Evaluations (evals)

Evals, or evaluations, are structured tests for measuring the quality of a system, process, or outcome.

Expected Gradients

Expected gradients are a fast explainability technique useful for differentiable models.

Explainability

Explainability is the ability to give a human-readable account of why a model produced a particular output, and which inputs moved it.

F

F-Score

F-Score is a measure of the harmonic mean of precision and recall.

Faithfulness (vs hallucination)

Faithfulness measures whether an answer accurately reflects the provided context without adding unsupported claims.

False Negative

A false negative is a case that is actually positive but the model predicted as negative.

False Positive

When a model mistakenly predicts a positive class, when the value belongs to the negative class, that outcome is a false positive (FP).

False Positive Parity

Commonly used as a model fairness metric, false positive parity measures whether a model incorrectly predicts something as more likely for a sensitive group than for the base group.

Feature Drift

Feature drift, data drift, covariate drift and input drift all refer to a shift in the statistical properties of the independent variable(s), i.e.

Feature Importance

Feature importance is a score assigned to each input feature that reflects how much that feature contributed to a model’s predictions or to its accuracy.

Feature Importance Heatmap

A feature performance heat map is a visual representation of the performance of each feature in a given model.

Feature Store

A machine learning infrastructure tool that handles offline and online feature transformations.

G

Grounding

Grounding means an AI output is supported by the context, data, or sources available to the system.

H

Harness engineering

Harness engineering is the discipline of building and maintaining the agent harness: the code around the model that assembles context, defines and executes tools, enforces limits, handles retries and errors, manages state between steps, and decides when the run is over.

Human evaluation

Human evaluation uses people to judge AI outputs, traces, or sessions.

Human labeling

Labeling is the process of attaching judgments to examples.

Human-in-the-loop

Human-in-the-loop means a human actively participates in an AI workflow before a decision or action is completed.

Human-on-the-loop

Human-on-the-loop means a human supervises an automated system without approving every action in advance.

I

Individual Conditional Expectation (ICE)

Individual conditional explanation (ICE) plots visualize one line per instance to show how the instance’s prediction changes when a feature changes.

Integrated Gradients

Integrated gradients are a technique for attributing the predictions of a classification model to input features.

J

Jailbreaking

Jailbreaking is an attempt to bypass a model or system’s safety rules.

Jensen-Shannon Divergence

Jensen-Shannon divergence, usually shortened to JS divergence, measures how different two probability distributions are by comparing each of them to their average.

JS Distance

JS distance is a symmetric derivation of KL divergence, and it is used to measure drift.

K

Kernel SHAP

Kernel SHAP is a slow, perturbation-based Shapley approach that theoretically works for all types of models but is rarely used by teams in production at full scale.

KNN Algorithm

The K Nearest Neighbor (KNN) algorithm is an uncomplicated, non-parametric machine learning technique employed for classification and regression tasks.

Kolmogorov-Smirnov test

Useful in drift monitoring, the Kolmogorov-Smirnov test (KS test) is an efficient and general way to measure if two distributions significantly differ from one another.

Kullback-Leibler Divergence (KL Divergence)

The Kullback-Leibler Divergence metric is calculated as the difference between one probability distribution from a reference probability distribution.

L

LGTM@K

LGTM@K is a joke metric name from a Vespa.ai talk on evaluating information retrieval systems alongside LangChain.

LIME

LIME, or Local Interpretable Model-Agnostic Explanations, is an explainability method that attempts to provide local ML explainability.

LLM evaluation

LLM evaluation is the practice of measuring whether a large language model, or an application built on one, behaves as intended.

LLM Jailbreaking

LLM jailbreaking refers to escaping the guardrails and safeguards of an LLM application or foundation model.

Logarithmic Loss

Tracks incorrect labelling of the data class by the model and penalizes the model if deviations in probability occur into classifying the labels.

Long-running agents

Long-running agents are agents that operate across extended timeframes rather than a single request-response turn.

M

Mean Absolute Error (MAE)

Mean absolute error (MAE) is a regression loss measure that looks at the absolute value difference between a model’s predictions and ground truth, averaged across the dataset.

Mean Absolute Percentage Error (MAPE)

Mean absolute percentage error (MAPE) is one of the most common metrics of model prediction accuracy and the percentage equivalent of mean absolute error (MAE).

Mean Square Error (MSE)

Mean square error (MSE) is the average of the squared differences between what a model predicted and what actually happened.

Memory Injection Attack (MINJA)

Memory Injection Attack (MINJA) is a security vulnerability identified in LLM or AI agents that possess persistent memory capabilities.

METEOR Score

Metric for Evaluation of Translation with Explicit Ordering (METEOR) score is a metric that measures the quality of generated text based on the alignment between the generated text and the reference text.

Misguided Attention Evaluation

Misguided Attention is a benchmark designed to test an LLM’s reasoning when faced with misleading or irrelevant context.

Model Context Protocol (MCP)

Model Context Protocol (MCP) is an open standard from Anthropic for connecting AI assistants to external data, content, and tools in a uniform way.

Model Performance

Model performance is how well a model does the job it was built for, measured with a metric that suits the task and computed on data that resembles what the model sees in production.

Model Store

A model store is a machine learning infrastructure tool that serves as a central model registry and tracks experiments.

Monitor Threshold

Monitor threshold refers to the value set for a model monitor, beyond which the model’s monitoring status will be triggered accordingly.

Monitoring Embeddings

Embeddings are not static, as new concepts appear in the real world all the time.

MRR (mean reciprocal rank)

Mean reciprocal rank, or MRR, measures how high the first relevant result appears in a retrieved ranking.

Multi Turn LLM Degradation

It has been observed that many LLMs get lost in extended conversations, showing a significant performance drop as the number of dialogue turns increases.

Multi-Agent Post-Co-Training RL (MAPoRL)

Multi agent post-co training reinforcement learning (MAPoRL) is a training methodology that enhances the collaborative capabilities of multiple AI agents.

Multi-Agent Reinforcement Fine-Tuning (MARFT)

Multi-Agent Reinforcement Fine-Tuning (MARFT) is a training paradigm that applies reinforcement learning techniques to fine-tune multiple AI agents simultaneously.

Multi-agent systems

A multi-agent system is an arrangement of several agents working on parts of the same task, each with its own instructions, tools, and context.

Multi-Turn Semantic Drift

Multi-turn semantic drift is what happens when a conversation or agent session gradually stops serving the request it started with.

Multimodal Model

Multimodal models process and relate information from different types of inputs, like text and images.

N

Natural Language Processing (NLP)

Natural language processing (NLP) is the branch of machine learning focused on text and speech: reading language, representing it numerically, and producing labels, spans, translations, summaries, or answers.

NDCG (ranking quality)

NDCG, or normalized discounted cumulative gain, measures ranking quality when results can have graded relevance.

O

Offline vs online evaluation

Offline evaluation runs against a fixed dataset outside the production request path.

One-Shot Reinforcement Learning Using Verifiable Rewards (RLVR)

RLVR stands for reinforcement learning with verifiable rewards, sometimes written as reinforcement learning from verifiable rewards.

OpenInference

OpenInference is an open specification of semantic conventions for tracing LLM and agent applications, along with a set of instrumentation libraries that emit traces following those conventions.

P

Pass/fail criteria

Pass/fail criteria define the conditions under which an AI output, eval run, experiment, or deployment is considered acceptable.

Performance Impact Score

Performance impact score is a measure of how much worse your metric of interest is on the slice compared to the average.

Performance Slice

A performance slice, also known as a cohort or segment, is a subset of model values of interest in performance analysis and troubleshooting.

Planning (agent planning / task decomposition)

Agent planning is how an agent decides on a sequence of steps to reach a goal, either before it starts acting or as it goes.

Policy adherence

Policy adherence measures whether an AI system follows defined rules.

Policy-driven agents

Policy-driven agents are agents whose behavior is governed by explicit policies rather than prompt instructions alone.

Population Stability Index (PSI)

Population Stability Index (PSI) measures how much a variable’s distribution has shifted between two samples, usually a reference window (expected) and a current window (actual).

Precision

Precision is the fraction of values that actually belong to a positive class out of all the values which were predicted to belong to that class: precision = true positives / (predicted true positives + predicted false positives).

Precision Recall (PR) Curve

The precision-recall curve is the correlation between precision and recall at particular cutoff values, with the cutoffs set according to the model.

Precision@K

Precision@K measures how many of the top k retrieved results are relevant.

Prediction Drift Impact

The product of feature importance and drift (population stability index, PSI) measures what impact a specific feature has on the prediction drift.

Principal Component Analysis (PCA)

Principal component analysis (PCA) is a common way to obtain embeddings that does not rely on neural networks.

Prompt drift

Prompt drift is the accumulated change to a prompt’s text over time.

Prompt evaluation

Prompt evaluation is the practice of measuring whether a prompt does its job, by running it over a fixed set of inputs and scoring the outputs against criteria defined in advance.

Prompt injection

Prompt injection is an attack where malicious or untrusted text attempts to override system instructions, change tool behavior, leak data, or manipulate an agent’s decisions.

Prompt Management System

A prompt management system is the version control, testing, and deployment layer for the prompts running inside an AI application.

Prompt versioning

Prompt versioning is the practice of treating a prompt as a versioned artifact: every distinct state of it gets an immutable identifier, and that identifier travels with every output it produces.

Q

Quantile

A quantile is a cutpoint that divides a distribution into intervals containing equal shares of the data.

Question Answering Document Retrieval LLM

Question Answering Document Retrieval with LLMs is a technique designed to pull specific answers from large documents based on user questions.

R

RAG evaluation

RAG evaluation measures how well a retrieval-augmented generation system retrieves relevant context and generates grounded answers from it.

RAGEN (LLM Agent Training System)

RAGEN is a research system for training large language model agents with multi-turn reinforcement learning.

Recall

Recall is the fraction of values predicted to be of a positive class out of all the values that truly belong to the positive class (including false negatives).

Recall Parity

Often used as a model fairness metric, recall parity measures how sensitive the model is for one group compared to another, or a model’s ability to predict true positives correctly at equal rates across groups.

Reference Distribution

In the context of ML observability, the reference distribution can be a number of different options.

Regression

Regression analysis is a fundamental concept in data science and machine learning.

Relevance

Relevance is whether an AI output or a retrieved result addresses the user’s actual request.

Retrieval Augmented Generation and Dense Passage Retrieval

Retrieval Augmented Generation and Dense Passage Retrieval use the context of a question to retrieve relevant passages from a large corpus of documents and extract answers.

Retrieval failure

Retrieval failure happens when the system does not fetch the information needed to answer or act correctly.

Retrieval quality

Retrieval quality measures whether the system found the right information for a user query.

Retrieval-augmented generation (RAG)

Retrieval-augmented generation, or RAG, is an architecture where a system retrieves external context and provides it to a model before generation.

ROC-AUC

The Receiver Operating Characteristics (ROC) is a probability curve plotted between true positive rate (TPR) and false positive rate (FPR).

Root Mean Square Error (RMSE)

Root mean square error (RMSE) is the square root of the average squared prediction error.

S

Safety evaluation

Safety evaluation measures whether an AI system avoids harmful, unsafe, unauthorized, or policy-violating behavior.

Score Models

Score models generate a numeric value as their prediction or output.

Scoring function

A scoring function is the logic that turns an input, output, context, trace, or trajectory into a score or label.

Self-healing agents

Self-healing agents are agents that detect certain failures and recover from them without waiting for a developer to intervene.

Self-improving agents

Self-improving agents are agents that use observed behavior and evaluation results to change their own prompts, policies, tool use, retrieval, or workflows over time.

Self-improving harness

A self-improving harness is an agent harness that changes in response to what it observes about its own failures.

Semantic search

Semantic search retrieves results based on meaning rather than exact keyword matching.

Sensitivity

Sensitivity is the share of actual positive cases that a model correctly identifies.

Sequence To Sequence Model

Sequence-to-sequence models take an input sequence, encode it into an internal representation, and then decode that into an output sequence.

Shadow Deployment

A shadow deployment runs a candidate model or agent version against live production traffic while the existing version keeps serving users.

Shadow testing (LLMs / agents)

Shadow testing runs a new model, prompt, retriever, or agent version alongside production on the same inputs, without showing its output to anyone.

SHAP Values

On each feature of an ML model, a Shapley value can be computed to explain how that feature contributed to the difference between the model’s prediction for an example and the baseline (average or expected) model prediction.

Shapley Additive Explanations (SHAP)

SHAP stands for Shapley Additive Explanations, a framework derived from cooperative game theory for explaining machine learning predictions.

Singular Value Decomposition (SVD)

Singular Value Decomposition (SVD) is a common way to obtain embeddings that does not rely on neural networks.

Specificity

Specificity is the fraction of actual negative cases that the model correctly predicts as negative.

StarPO

StarPO (State-Thinking-Actions-Reward Policy Optimization) is a training paradigm for LLM-based agents that optimizes entire interaction trajectories rather than stepwise decisions.

Surrogate Model

A surrogate model is a simple, readable model trained to imitate the behavior of a more complex or inaccessible one.

Swarm Management

Swarm management is the control-plane layer that owns, tracks, and steers a fleet of long-running agents.

T

t-SNE

t-SNE (t-distributed Stochastic Neighbor Embedding) is a dimension reduction technique for data visualization.

Tabular Data

Data in a table format, with columns and rows.

Tags

Often used alongside model features, tags enable metadata support for slicing and cohorting.

Test set and test cases

A test set is a collection of test cases used to evaluate behavior.

Tokenization

Tokenization is a crucial step in language models as it breaks down text data into smaller units called tokens, such as words or characters.

Tool calling (function calling)

Tool calling, also called function calling, is the mechanism that lets a model invoke external functions, APIs, databases, retrievers, code execution environments, or other systems.

Tool failure

Tool failure happens when an external function, API, retriever, database, browser, code runner, or other tool call fails or returns bad data.

Tool N1

Tool-N1 refers to a class of approaches where language models learn to use external tools through trial and error, without explicit step-by-step demonstrations.

Toxicity

Toxicity measures whether an output contains abusive, hateful, harassing, or otherwise harmful language.

Trace

A trace is the record of one request or run through a system.

Training data vs evaluation data

Training data is used to teach or adapt a model.

TreeSHAP

TreeSHAP is the algorithm that computes SHAP values for tree-based models exactly and fast.

True Negative

A true negative (TN) is when a model correctly predicts the negative class: the example truly belongs to the negative class, and the model said negative.

True Positive

When a model correctly predicts a positive class, when the value belongs to the positive class, that outcome is a true positive (TP).

U

UMAP (Uniform Manifold Approximation and Projection for Dimension Reduction)

Uniform Manifold Approximation and Projection (UMAP) is a technique for visualizing the embedding representation of a dataset using dimension reduction.

Unstructured Data

According to multiple estimates, a large share of data generated by businesses today is unstructured data such as text, images, or audio.

W

Wasserstein Distance

Wasserstein distance measures how far apart two probability distributions are by asking a physical question.

Weak supervision

Weak supervision uses noisy, indirect, or programmatically generated labels to create training or evaluation signal at scale.

What is a Vector DB in LLMOps?

A vector database, or vector db, is a specialized type of database that stores and processes data in vector form.