The Search That Made Me Feel Dumb
A while back I built a little internal docs search for a project I was working on. Classic keyword search — user types something, I scan the text, return matches. Simple. Worked fine until a teammate searched for "how do I cancel a subscription" and got zero results. The actual doc was titled "Terminating a membership plan." Same concept, completely different words. My search had no idea.
That's the keyword matching trap. The computer doesn't know that "cancel" and "terminate" mean the same thing in this context. It's just pattern matching characters. Semantic search fixes that by working on meaning rather than spelling. And once I understood how it worked, I couldn't stop using it everywhere.
What Semantic Search Actually Is
At its core, semantic search converts text into vectors — long lists of numbers — called embeddings. These numbers represent the meaning of the text in a mathematical space. The trick is that text with similar meanings ends up with vectors that are numerically close to each other.
So "cancel subscription" and "terminate membership" end up near each other in this space, even though they share zero words. Meanwhile "cancel subscription" and "banana smoothie" end up very far apart. The distance between vectors is the search engine's way of measuring relevance.
When you run a semantic search, you:
- Convert your query into an embedding
- Convert all your documents into embeddings (usually done ahead of time)
- Find the documents whose embeddings are closest to your query embedding
- Return those as results
The math involved is usually cosine similarity — measuring the angle between two vectors rather than the raw distance. Closer angle = more similar meaning.
Building a Simple Semantic Search Pipeline
Let me show you a concrete example using OpenAI's embeddings API and Python. This is a minimal version you can actually run.
# Install dependencies
pip install openai numpy# semantic_search.py
import numpy as np
from openai import OpenAI
client = OpenAI() # uses OPENAI_API_KEY env var
# Your document corpus
docs = [
"How to terminate a membership plan",
"Setting up two-factor authentication",
"Updating your billing information",
"How to export your account data",
"Contacting customer support",
]
# Step 1: Embed all documents upfront
def embed(texts):
response = client.embeddings.create(
model="text-embedding-3-small",
input=texts
)
return np.array([r.embedding for r in response.data])
doc_embeddings = embed(docs)
# Step 2: Semantic search function
def search(query, top_k=3):
query_embedding = embed([query])[0]
# Cosine similarity = dot product of normalized vectors
norms = np.linalg.norm(doc_embeddings, axis=1)
similarities = doc_embeddings @ query_embedding / (norms * np.linalg.norm(query_embedding))
top_indices = np.argsort(similarities)[::-1][:top_k]
return [(docs[i], round(float(similarities[i]), 3)) for i in top_indices]
# Step 3: Try it out
results = search("cancel subscription")
for doc, score in results:
print(f"Score: {score} → {doc}")
# Output:
# Score: 0.821 → How to terminate a membership plan
# Score: 0.612 → Updating your billing information
# Score: 0.589 → Contacting customer supportNotice how "cancel subscription" correctly surfaces "terminate a membership plan" at the top, with a high similarity score. Keyword search would have returned nothing. That's the magic moment right there.
ChatGPT's Role: Embedding + Generating
You can use ChatGPT (via the API) in two ways here. First, you can use the embeddings API (as above) to convert text to vectors. Second, you can pipe your semantic search results into a ChatGPT prompt to generate a natural language answer — this is basically what RAG (Retrieval-Augmented Generation) is.
# Combine semantic search + ChatGPT for a full answer
def answer_question(question):
top_docs = search(question, top_k=2)
context = "\n".join([doc for doc, _ in top_docs])
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Answer using only the context provided."},
{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}
]
)
return response.choices[0].message.content
print(answer_question("How do I cancel my account?"))
# → "To cancel your account, you can terminate your membership plan..."Cache Your Embeddings
Don't re-embed your documents on every search. Compute embeddings once, save them to a file or vector database like Chroma or Pinecone, and load them at startup. Re-embedding kills your latency and burns API credits unnecessarily.
Where Semantic Search Wins
Semantic search genuinely shines in specific situations. I've seen it make a real difference in all of these:
- Synonym-heavy domains — medical, legal, customer support. "Heart attack" finds "myocardial infarction."
- Casual queries against formal docs — users write naturally, docs are written professionally.
- Intent matching — "I want to leave the platform" should find your cancellation page even if "leave" doesn't appear anywhere.
- Cross-language similarity — multilingual embeddings can find French docs with an English query.
- FAQ-style lookups — users rephrase the same questions endlessly, semantic search handles all variants.
The short version: when the gap between how users talk and how content is written is large, semantic search closes that gap.
Where Plain Text Matching Still Wins
I want to be honest here because I've seen people overcomplicate things by reaching for semantic search when good old LIKE '%keyword%' would have been fine.
Keyword search beats semantic search when:
- Exact matches matter — searching for an error code like ERR_CONNECTION_REFUSED or an order number. Semantic embeddings will fuzzy this out and potentially return irrelevant results.
- Proper nouns and names — searching for "Dan Byers" shouldn't return "Daniel Buyers" just because the vectors are similar.
- Speed and infrastructure constraints — vector similarity search requires more compute than a simple text index. For small datasets or tight latency budgets, keyword search is faster and cheaper.
- Keyword-dense technical queries — developers searching for a specific function name or package. Exact token matching is exactly what you want.
- Debuggability — keyword search is easy to explain. "Your doc contains the word 'cancel'." Semantic search scores are harder to debug when results seem wrong.
The Hybrid Approach
Most production systems combine both. Run BM25 (keyword) and vector similarity in parallel, then merge the rankings. Elasticsearch and OpenSearch both support this natively now. You get exact match precision and semantic recall without having to pick one.
Practical Gotchas I Learned the Hard Way
A few things that tripped me up when I first started building with embeddings:
Chunk size matters a lot. If you embed entire pages as one chunk, the embedding averages out all the meaning and becomes a blurry representation. Short, focused chunks (a paragraph, a FAQ item) embed much better. I generally aim for 200-500 tokens per chunk.
Your query and your docs should be in the same "register." If your docs are formal and your users are casual, the embeddings will still find good matches — but you can also embed a rewritten, formal version of the query before searching. This is called query expansion and it helps a lot.
Similarity scores aren't absolute. A score of 0.82 doesn't mean "82% relevant." It's relative to your corpus. Always show top-K results rather than thresholding by score, unless you've calibrated thresholds on your specific data.
The model you use matters. text-embedding-3-small is cheap and fast. text-embedding-3-large is more accurate for nuanced queries. For most beginner projects, small is totally fine.
When to Reach for a Vector Database
The NumPy approach above works fine for a few hundred documents. Once you're into the thousands, you want a proper vector database to handle indexing and approximate nearest neighbor search efficiently.
The easiest one to get started with locally is Chroma:
pip install chromadb
# Then in Python:
import chromadb
client = chromadb.Client()
collection = client.create_collection("my_docs")
# Add documents (Chroma can handle embedding via OpenAI with a plugin)
collection.add(
documents=docs,
ids=[f"doc_{i}" for i in range(len(docs))]
)
# Query
results = collection.query(query_texts=["cancel subscription"], n_results=3)
# Returns ranked results with distancesThe Bottom Line
Semantic search isn't magic — it's a specific tool with specific strengths. Use it when the meaning gap between queries and documents is real and you need flexibility. Stick with keyword search when you need precision, speed, or explainability. And in production, seriously consider running both.
What I love about this technology is how approachable it's become. A few years ago building this kind of search required deep ML knowledge. Now you can call an API, get embeddings back, and have a working semantic search in an afternoon. The hard part has been abstracted away — your job is just knowing when to use it.
Start with a small corpus of docs you care about. Try both approaches. See which one surprises you more. That's honestly how I learned what semantic search was actually good at — by watching it succeed where keyword search failed, and fail where keywords excelled.
More tutorials in this category, or explore the full field guide.