Choosing an Embedding Model: Dimensions, Cost and Quality

The Decision Nobody Warns You About

When I first built a RAG pipeline, I grabbed the first embedding model I saw in the docs — OpenAI's text-embedding-ada-002 — without thinking twice. It worked fine. But when I started scaling the thing up, the bill crept up in ways I didn't expect, and I kept wondering: am I even using the right model for this?

Turns out choosing an embedding model is one of those decisions that feels trivial until it bites you. Unlike swapping out a language model (which you'd obviously notice), switching embedding models is quiet — the difference shows up in retrieval quality, latency, and monthly cost in ways that are easy to miss until you're deep in production.

So let's talk about the tradeoffs that actually matter, and build a simple way to compare candidates before you commit.

What Embedding Dimensions Actually Mean

Every embedding model converts text into a vector — a list of numbers. The length of that list is called the dimensionality. A model might output 384 dimensions, 768, 1536, or even 3072.

Here's the intuition: more dimensions means more "room" to encode nuance. A 3072-dimensional vector can capture more subtle semantic detail than a 384-dimensional one. But more dimensions also means:

• More storage per document (your vector database gets big fast)
• Slower similarity searches at scale
• Higher cost if you're paying per token to generate embeddings

The surprise for most people: the relationship between dimensions and quality is not linear. A well-trained 768-dimensional model can absolutely outperform a poorly-trained 3072-dimensional one. Dimensions are a hint about capacity, not a guarantee of quality.

The Three Tradeoffs That Actually Matter

When I compare embedding models now, I focus on three axes. Everything else is noise until you've nailed these.

1. Retrieval quality on your data

Benchmarks like MTEB (Massive Text Embedding Benchmark) are useful starting points, but they're not your data. A model that tops the leaderboard on general English text might underperform on your medical PDFs or your internal code comments. Quality is always domain-specific.

2. Cost to embed

Hosted models charge per token. Open-source models running locally are "free" but cost you compute time and infrastructure. Neither is free — the question is which cost you're more comfortable with. For small projects, hosted is almost always cheaper. For high-volume production, local can flip the equation fast.

3. Latency and throughput

How fast do embeddings generate? This matters at two points: when you're first indexing a large corpus (a slow model makes ingestion painful) and at query time if you're embedding the user's question on every request. A model that takes 500ms per query at scale is going to hurt.

A Simple Comparison Framework You Can Actually Run

Here's the workflow I now use whenever I'm evaluating a new embedding model. It takes about an hour and gives you real signal, not just vibes.

Step 1: Build a small golden test set

Take 20–30 questions that a real user might ask your system, and manually identify which chunks in your corpus should be the correct answer. This is your ground truth. Boring to build, invaluable to have.

Step 2: Embed with each candidate model

Step 3: Measure hit rate @ k

For each test question, retrieve the top-5 chunks and check whether the correct chunk made it in. That percentage is your hit rate at 5 (HR@5). It's a simple metric but it directly measures what you care about: does the right information surface when a user asks?

Step 4: Add cost and latency to the scorecard

Once you have quality numbers, layer in the practical constraints. A 15% quality improvement might not be worth a 10x cost increase depending on your use case.

The Models Worth Knowing Right Now

The embedding model landscape shifts fast, but here's a practical lay of the land as of mid-2024:

• OpenAI text-embedding-3-small — 1536 dims, cheap at $0.02/million tokens, Matryoshka-enabled, great default for hosted use
• OpenAI text-embedding-3-large — 3072 dims, pricier, meaningfully better on hard retrieval tasks
• BAAI/bge-large-en-v1.5 — Open source, 1024 dims, punches well above its weight class, runs fine on a MacBook
• Cohere embed-english-v3.0 — Strong for retrieval specifically, interesting if you're already in the Cohere ecosystem
• nomic-embed-text — 768 dims, open weights, good balance of quality and size

The Gotcha Nobody Talks About: Model Switching Costs

Here's the part that burned me once and I want to save you from: once you embed your corpus with Model A, you can't just swap in Model B and query against the old vectors. The vector spaces are completely different. It's like translating a library into Spanish and then trying to search it with a French query — the languages don't mix.

Switching embedding models means re-embedding your entire corpus. If you have 500,000 documents, that's not free or fast. This is exactly why your initial model choice matters more than it might seem.

My rule of thumb: if you're in early experimentation, try several models quickly on a small slice. Once you're building toward production, pick one and stick with it until you have a compelling reason to re-embed everything. Version your embeddings in your vector database so you know which model generated which vectors.

My Actual Decision Process

When someone asks me what embedding model to use, here's the honest answer I give them:

If you're experimenting or building something small: text-embedding-3-small from OpenAI. Low friction, good quality, Matryoshka flexibility, and the cost won't matter until you're at real scale.

If you want to avoid vendor lock-in or run locally: BAAI/bge-large-en-v1.5 via SentenceTransformers. It's genuinely impressive for free software and the community support is solid.

If you're in production and quality is paramount: run the comparison framework above on your actual data and let the numbers decide. Don't let a benchmark someone else ran on someone else's data make the call for you.

The key insight I had too late: the best embedding model is the one that retrieves the right chunks for your specific corpus and your specific questions. That's a measurement problem, not a specs problem. Build the test set, run the numbers, trust what you see.