Shipping Text Embeddings in Production

The demo is easy, the system is not

Embedding a hundred documents and querying them takes an afternoon. Keeping millions of embeddings fresh, cheap and actually relevant is a different job. This is a tour of the decisions that determine whether your retrieval system works in production — chunking, storage, and the evaluation loop that most teams skip.

1. Chunking is a retrieval decision, not a preprocessing detail

An embedding compresses a passage into a single vector. Too long and the vector becomes an average of several topics that matches nothing precisely. Too short and it loses the context that makes it interpretable. Chunk boundaries determine what can ever be retrieved.

• Split on structure first — headings, sections, list items — before falling back to token counts.
• Overlap adjacent chunks so a fact spanning a boundary is not lost. Ten to fifteen percent is usually enough.
• Prepend the document title and section path to every chunk. It costs a few tokens and sharply improves matching.
• Store the parent document id and character offsets so you can expand a hit back into context at answer time.
• Keep tables and code blocks intact; splitting them mid-structure destroys their meaning.

def chunk_with_overlap(text, size=400, overlap=60):
    """Character-based chunker that prefers sentence boundaries."""
    chunks, start = [], 0
    while start < len(text):
        end = min(start + size, len(text))
        if end < len(text):
            window = text.rfind(". ", start + size // 2, end)
            if window != -1:
                end = window + 1
        chunks.append(text[start:end].strip())
        if end >= len(text):
            break
        start = max(end - overlap, start + 1)
    return chunks

doc_title = "Refund Policy"
section = "Late Refunds"
body = ("Refunds are processed within five business days of approval. "
        "If you have not received a refund after five days, first check your bank account. "
        "Contact your card issuer, as processing times vary. "
        "If you have done all of this and still have not received your refund, contact support. "
        "Sale items are only refundable if faulty.")

for i, chunk in enumerate(chunk_with_overlap(body, size=160, overlap=30)):
    contextualised = "%s > %s\n%s" % (doc_title, section, chunk)
    print("--- chunk %d (%d chars) ---" % (i + 1, len(chunk)))
    print(contextualised)
    print()

2. Vector store trade-offs

• Postgres with pgvector: one database, transactional consistency, easy metadata filters. Excellent up to a few million vectors; HNSW indexing is essential past a hundred thousand.
• Dedicated vector databases: better recall/latency at very large scale and richer hybrid search, at the cost of another system to operate and keep in sync.
• In-memory brute force: unbeatable simplicity below roughly fifty thousand vectors. Exact results, no index tuning, no drift.
• Approximate indexes trade recall for speed. Measure that recall — the default parameters are a guess about your data, not a promise.

Two operational details matter more than the choice of store: dimensionality (storage and latency scale with it, and many modern models support truncation to a shorter prefix) and metadata filtering, because most real queries are scoped to a tenant, a language or a date range.

3. Hybrid search beats pure vectors

Dense embeddings capture meaning but miss exact tokens — product codes, error numbers, rare names. Keyword search does the opposite. Run both and fuse the ranked lists; reciprocal rank fusion needs no score calibration and is hard to beat for the effort involved.

def reciprocal_rank_fusion(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc in enumerate(ranking, start=1):
            scores[doc] = scores.get(doc, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)

dense   = ["doc_refunds", "doc_shipping", "doc_returns", "doc_faq"]
keyword = ["doc_returns", "doc_refunds", "doc_terms", "doc_faq"]

for doc, score in reciprocal_rank_fusion([dense, keyword]):
    print("%-14s %.5f" % (doc, score))
print()
print("Documents both retrievers agree on rise to the top without any score normalisation.")

4. The evaluation loop nobody talks about

Retrieval quality is measurable, and if you do not measure it you are tuning blind. Build a small golden set — fifty to two hundred real queries with the passage that should be retrieved — and track recall@k and MRR on every change to the chunker, the model or the index.

def recall_at_k(results, relevant, k):
    hits = sum(1 for q in results if any(d in relevant[q] for d in results[q][:k]))
    return hits / float(len(results))

def mrr(results, relevant):
    total = 0.0
    for q, ranked in results.items():
        for rank, doc in enumerate(ranked, start=1):
            if doc in relevant[q]:
                total += 1.0 / rank
                break
    return total / float(len(results))

relevant = {"how do refunds work": {"doc_refunds"},
            "where is my order":    {"doc_shipping"},
            "faulty sale item":     {"doc_returns"}}
results  = {"how do refunds work": ["doc_refunds", "doc_faq", "doc_terms"],
            "where is my order":   ["doc_faq", "doc_shipping", "doc_terms"],
            "faulty sale item":    ["doc_faq", "doc_terms", "doc_refunds"]}

print("recall@1: %.3f" % recall_at_k(results, relevant, 1))
print("recall@3: %.3f" % recall_at_k(results, relevant, 3))
print("MRR:      %.3f" % mrr(results, relevant))
print()
print("The third query never retrieves its answer — that is a chunking bug, not a model bug.")

5. Operating the thing

• Version your embeddings. Store the model name and version with every vector; a model upgrade means a full re-index and you need to be able to run both during the migration.
• Batch and cache. Embedding calls dominate cost; hash the normalised text and skip anything unchanged.
• Re-rank the top 50 with a cross-encoder when precision matters. It is the single largest quality win after hybrid search.
• Watch p99 latency, not the mean. Approximate indexes have long tails under concurrent writes.
• Log every query with its retrieved ids. Those logs become next quarter's golden set.