Embedding Dimensions and Cost: Bigger Vectors, Better RAG?

You picked an embedding model by scrolling to the top of a leaderboard, saw a 3072-dimension option, and assumed bigger vectors would retrieve better answers. That instinct is about to over-charge your vector database. Embedding dimensions are a storage-and-latency knob, not a quality dial, and past a point extra dimensions buy you bytes, not better retrieval.

What do embedding dimensions actually represent?#

An embedding is a list of numbers, a vector, and the dimension count is how long that list is. Each number is a coordinate in a space where closer means more similar in meaning. More embedding dimensions give a model more room to separate meanings, but only up to the point its training actually fills with real signal.

More axes can encode finer distinctions in theory. In practice a model only produces as much usable signal as it was trained to produce; beyond that, extra dimensions add bytes, not discrimination. This is exactly why the curve of retrieval quality against dimension count flattens fast. The first few hundred numbers do most of the work of telling meanings apart, and the long tail after them contributes little you would ever notice at query time.

It helps to picture the trade at the extremes. Two dimensions could not separate the meanings in a real corpus; everything would collapse onto a plane and unrelated passages would sit next to each other. But once a model has enough room to place distinct concepts far apart, adding more axes mostly refines distances that were already good enough for retrieval. Your ranker pulls the same top chunks either way, so the extra coordinates change the bill without changing the answer.

Does adding dimensions actually improve retrieval?#

Only up to a point, and model generation matters far more than raw size. Per OpenAI's published figures, text-embedding-3-large scores 64.6 on MTEB at its full 3072 dims versus ada-002's 61.0. Truncated all the way down to 256 dims it still scores 62.0, above ada-002 at six times the length.

From OpenAI's published figures. Going 256 to 3072 dims (12x the storage) moves MTEB about 2.6 points; the curve flattens fast.

The shape is what matters here. Going from 256 to 3072 dims, a 12x jump in storage, moves the MTEB score roughly 2.6 points on OpenAI's own scale. On multilingual MIRACL the same model reports 54.9 versus ada-002's 31.4, so a newer generation reshapes retrieval quality in a way that no amount of extra dimensions on an older model would.

Treat those benchmark numbers with care. MTEB is built on public datasets, so scores can be inflated by training-data overlap, and the project added a zero-shot percentage column precisely to flag that risk. A model that has seen a large slice of the benchmark during training will look stronger than it performs on text it has never met, which is exactly your situation. A leaderboard average also does not transfer to your corpus, your query style, or your domain, because MTEB aggregates dozens of tasks that may weight nothing like your retrieval problem. Use MTEB to build a shortlist, then measure recall on your own questions before you trust a single figure. Our comparison of RAG embedding models in n8n walks through that shortlist-then-test discipline on a live pipeline.

What do dimensions cost in storage, memory, and latency?#

Storage scales linearly with dimensions, and you pay it on every vector forever. In pgvector a float32 vector takes 4 × dims + 8 bytes, so about 6,148 B at 1536 dims and 12,296 B at 3072. For a million chunks that is roughly 5.7 GB against 11.5 GB of raw vectors, before any index sits on top.

Raw vector storage for 1M chunks, modeled from pgvector's 4 x dims + 8 bytes per vector, before the HNSW index.

That raw number is only the floor. HNSW indexes are held in RAM for speed and can add roughly 1.5 to 2x the raw data size on top, so bigger vectors mean a bigger index, slower builds, slower queries, and more bytes on every API call. The cost is linear in dimensions and it recurs on each vector you store and each query you run, which is where a leaderboard-driven choice quietly becomes a line item.

Bigger helps at the margin and on hard corpora; it always costs bytes, RAM, and latency. Match the size to the need.

Two levers cut that bill far more than extra dimensions ever add quality. The first is Matryoshka truncation. The second is quantization: storing int8 instead of float32 is about 4x smaller and binary is about 32x smaller. Voyage reports int8 at 1024 dims sitting only about 0.31% below float at 2048 dims while using roughly 8x less storage, per its own published figures. Supabase's adaptive-retrieval writeup found a 512-dim first pass plus a full-dimension re-rank reached about 99% of full-vector accuracy at high throughput, again as that benchmark's own result rather than a universal law. If you are choosing where those vectors live, our pgvector vs Chroma vs Qdrant guide covers how each store handles this.

Matryoshka is why text-embedding-3, Voyage, Cohere v4, and Gemini let you pick a size. Truncate, then L2-normalize.

Why can you shrink a vector without breaking it?#

Because of how modern embedding models are trained. Matryoshka Representation Learning packs the core semantics into the earliest dimensions and adds detail in later ones, so you can slice off a shorter prefix and re-normalize, keeping most of the retrieval quality. This is trained-in structure, not post-hoc PCA, which is why truncation holds up.

One practical detail trips people up. When you truncate a unit-length vector, the shorter slice is no longer normalized, so you must apply L2 normalization again before you compare it. Skip that step and your similarity scores drift. Matryoshka is why text-embedding-3, Voyage, Cohere Embed v4, and Gemini all let you request a smaller size from one model instead of paying to re-embed.

How do you pick an embedding dimension you won't regret?#

Pick the model for your domain and languages first, then start at a mid tier and only raise dimensions if recall on your own queries is genuinely short. A code model suits code, a multilingual model suits non-English, and MTEB comes second, not first. Most flexible models expose tiers, so you rarely need to start at the maximum.

Start around 512 to 1024 dims. Voyage exposes 256, 512, 1024, and 2048; Cohere Embed v4 offers 256, 512, 1024, and 1536; OpenAI's 3-large runs up to 3072. Build a small evaluation set of real questions with known-correct chunks, check recall at k, and raise dimensions one tier only if the gap is real. Fifty labelled queries is enough to catch a bad choice; you are looking for a clear break, not a decimal place.

Reach for a reranker or quantization before you reach for more dimensions, since int8 or binary usually buys more cost relief than extra dimensions buy quality. A reranking model rescoring your top 50 candidates often recovers more recall than doubling the vector length, and it does not touch your storage bill at all. Order the levers by cost: model generation first, then a reranker, then quantization, and only then more dimensions. That sequence keeps you from paying for bytes you will never notice in the answers.

Start from Matryoshka support and corpus size; measure recall on your own queries, and quantize before adding dimensions.

So how big should your vectors be?#

For most builders a current-generation model at 768 to 1024 dims, or Matryoshka-truncated to that range, covers the job. Reserve 1536 to 3072 dims for genuinely hard, diverse, or high-stakes corpora, and confirm the extra bytes earn their keep with a recall test on your own data before you standardize.

The honest summary is that dimensions trade storage and latency for a thin slice of quality that flattens quickly. Model choice, Matryoshka truncation, and quantization move the needle far more than raw vector length. If you want a starting configuration for a working stack, our n8n vector store and embeddings walkthrough sets sane defaults you can measure against and adjust once you have your own recall numbers.