Embeddings are numerical representations of text, images, or other data that capture semantic similarity. Vector search uses those embeddings to retrieve the items sitting closest to a query embedding in vector space. Together they are the machinery underneath most meaning-based retrieval, including the retrieval layer of a typical RAG pipeline.
The useful mental model is that an embedding model is a compression function with a deliberate bias: it discards surface form and keeps whatever it learned to treat as meaning, so two passages that say the same thing in different words land near each other. The search half is a geometry problem. Given a query point, find the nearest document points, fast, in a space with hundreds or thousands of dimensions and possibly millions of points. Almost every practical decision here is a tradeoff inside that problem.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- An embedding is a fixed-length vector produced by a model. Vectors are only comparable when they came from the same model and version.
- The distance metric must match what the model was trained for. For unit-normalized vectors, cosine similarity, dot product, and Euclidean distance all produce the same ranking.
- Exact nearest-neighbor search scales linearly with corpus size, so production indexes use approximate search that trades measurable recall for large speed gains.
- Changing the embedding model invalidates the entire index. There is no partial migration, because old and new vectors do not share a space.
- Vector search returns the nearest items, not the items that answer the question. Distance is a proxy for relevance, not a measure of it.
What an embedding actually is
An embedding model takes an input and returns a list of numbers of fixed length, typically a few hundred to a few thousand. That length is the dimensionality, set by the model. Higher dimensionality generally carries more information at the cost of storage, memory, and compute per comparison.
Two properties matter more than the architecture:
The same model must embed both sides. Query and documents have to pass through the same model and version. Vectors from two different models are not comparable, and comparing them produces plausible-looking distances that mean nothing.
The geometry is learned, not designed. No dimension corresponds to a concept you can name. Closeness reflects whatever the training objective rewarded, usually topical and contextual similarity. That is why an embedding happily places “how to cancel a plan” next to “how to upgrade a plan”: they are similar in every way the model was taught to notice.
How closeness is measured
Three metrics cover nearly all usage:
- Cosine similarity measures the angle between vectors and ignores their length. Higher is closer.
- Dot product multiplies the vectors element-wise and sums, responding to both angle and magnitude. Higher is closer.
- Euclidean distance is the straight-line distance between the points. Lower is closer.
There is a simplification worth knowing. If every vector is normalized to unit length, the three rank results identically: cosine and dot product become the same computation, and Euclidean distance decreases exactly as the dot product increases. Many embedding models return normalized vectors already, which is why swapping metrics on such a model changes the scores you see but not the order of results.
When vectors are not normalized the choice matters, and the rule is to use whatever the model’s documentation specifies. A model trained with a cosine objective but queried with dot product will let long documents outrank better short ones on magnitude alone.
How vector search scales
Comparing a query against every vector is exact and gives the true nearest neighbors. It is also linear in corpus size, which stops being viable somewhere between hundreds of thousands and millions of vectors depending on your latency budget.
Production systems therefore use approximate nearest neighbor search. The families differ in how they avoid a full scan:
- Graph-based indexes build a navigable network of vectors and walk it toward the query, checking a small fraction of the corpus.
- Cluster-based indexes partition vectors into groups, then search only the groups nearest the query.
- Quantization compresses vectors into smaller codes so more of the index fits in memory, accepting some precision loss.
All of them expose parameters that trade recall against latency and memory. Recall here has a precise meaning: the fraction of the true nearest neighbors the approximate search returned, measurable by running exact search on a sample and comparing. Teams that never measure it can spend weeks tuning a reranker while the index quietly drops correct candidates first.
Metadata filtering interacts with this awkwardly. Restricting results to a tenant or a date range can happen before or after the vector search, and filtering afterward can return fewer results than requested, or none, when the nearest neighbors all fail the filter.
What breaks in production
Swapping the embedding model. A new model means every document must be re-embedded and the index rebuilt. Mixing generations produces silent nonsense rather than an error, so the migration needs the same version discipline as any other model change, one reason retrieval components belong in embedding versioning and drift monitoring rather than being treated as static infrastructure.
Chunking. Chunks that are too small lose the surrounding context that gave them meaning. Chunks that are too large average several topics into one vector that matches many queries weakly and none strongly. This is usually a bigger lever on retrieval quality than the embedding model, and it is more often left at a default.
Domain and input mismatch. A model trained on general web text will underperform on legal citations, medical codes, or internal jargon, and rare identifiers are represented poorly because the model saw them rarely if at all. These and related traps are covered in measuring embedding drift in production, and why getting started with embeddings is easier than most teams assume.
Index staleness. Documents change, get deleted, and get added. An index that is not kept current returns confident results pointing at text that no longer exists.
FAQ
What is the difference between an embedding and a vector?
A vector is the general mathematical object, an ordered list of numbers. An embedding is a vector produced by a model to represent a specific piece of data. Every embedding is a vector, and not every vector is an embedding.
Do I need a dedicated vector database?
Not necessarily. Dedicated vector databases specialize in approximate search at scale, but most mainstream relational databases and search engines now support vector indexes. At modest corpus sizes, exact search over an in-memory array is often easier to reason about than adding another system.
Which distance metric should I use?
The one the embedding model’s documentation specifies. If the model returns normalized vectors, cosine similarity and dot product are equivalent and the choice does not affect result order.
What happens if I change the embedding model?
You must re-embed the entire corpus and rebuild the index. Query vectors from a new model cannot be compared against document vectors from an old one, and nothing raises an error if you try. Plan the swap as a full reindex with a fallback.
Why does vector search return irrelevant results?
Because it returns the nearest vectors, and nearest is not the same as useful. A passage on the same topic that answers a different question is genuinely close in the space. The index also has no way to signal that nothing suitable exists, so asking for ten results always returns ten. Score thresholds and reranking are the usual mitigations.
