Vector Search Accuracy Starts With What You Put Into The Vector
· 41 min read · Views --

Vector Search Accuracy Starts With What You Put Into The Vector

Author: Alex Xiang

Engineering

Vector Search Accuracy Starts With What You Put Into The Vector

When vector search is first added to a system, it is easy to feel that keywords are finally no longer a problem. Turn documents, products, APIs, and knowledge-base entries into embeddings, turn the user’s natural-language query into another embedding, run nearest-neighbor search, and the answer should appear.

After launch, the cracks show up slowly.

A user searches for “real-time US stock quotes” and gets pages about financial data vendors. Another searches for “company registration data” and sees the weather API from a generic data platform. Someone searches for “invoice OCR” and gets every OCR API under a provider instead of the specific invoice-recognition endpoint. Each result is not completely unrelated, because the vendor, page, or document did mention finance, data, or OCR. But the user was not looking for something that merely touched the topic. They wanted something usable.

This article uses a fictional API marketplace to discuss these problems. The setting is fictional; the failure modes are very real.

A Common Mistake

Suppose each API tool has these fields:

  • Tool name, such as “minute-level stock quotes”
  • Tool description
  • Category, such as finance, stock, market data
  • Endpoint URL, such as https://api.example.com/v1/market/quotes
  • Provider name
  • Provider description, such as “a global data intelligence platform covering finance, enterprise, geospatial, weather, content moderation, and more”

The simplest indexing strategy is to concatenate everything:

minute-level stock quotes
returns minute-level trade price, volume, and order-book summary for a given stock
finance stock market data
https api example com v1 market quotes get
Example Data
We are a global data intelligence platform covering finance, enterprise, geospatial, weather, content moderation, and more...

Then the whole blob goes into the embedding model.

It looks complete, but it pollutes the vector. An embedding model is not a database field filter. It compresses the whole text into a fixed-dimensional vector. The more mixed the text is, the more the vector becomes a stew of tool capability, vendor marketing, URL fragments, broad industry terms, and HTTP method names.

Provider descriptions are especially dangerous. A provider may own dozens or hundreds of tools, all sharing the same “global, leading, reliable, multi-domain” paragraph. Those tools then become artificially close in vector space even if one is stock quotes, another is weather, and another is ID-card recognition.

When the user searches for “minute stock candles,” the system is no longer asking “which tool best matches this need.” It is asking “which mixed text blob is closest overall.” Those are very different questions.

Embeddings Need A Semantic Main Course

Full-text search often benefits from breadth. Aliases, abbreviations, product names, and path tokens can improve recall. Vector search is different. It needs a clean semantic expression.

The embedding text for a tool should answer three questions:

  • What does this tool do?
  • What are the inputs and outputs?
  • Under what intent would a user look for it?

It should not try to answer every surrounding question: how impressive the provider is, what the URL looks like, whether the endpoint uses GET or POST, or how many business lines the company covers.

I now prefer separating text for vectors from text for full-text search.

Vector text can look like this:

Minute-level stock quotes
Query minute-level trade price, volume, percentage change, and order-book summary for a single stock over a specified time range.
Suitable for market dashboards, quantitative backtests, and trading monitors.
Categories: stock, quotes, financial market data.

Full-text text can be wider:

Minute-level stock quotes stock minute line K-line quote market candle
Query minute-level trade price, volume, percentage change, and order-book summary for a single stock over a specified time range.
Path keywords: market quotes candles
Provider: Example Data

They serve different purposes. Vector text keeps semantic space clear; full-text text creates more keyword entry points.

Noise Is Often Good Information In The Wrong Place

Many retrieval failures are not caused by wrong data. They are caused by placing information in the wrong index.

URLs are not useless. Users may search for tokens such as quote, candles, or invoice, so keeping path terms in full-text search makes sense. But URLs also contain many meaningless fragments: https, api, v1, com, get. Once these enter embeddings, they become nearly identical background noise for REST APIs.

Provider descriptions are useful too. When a user searches for “Example Data stock API,” the provider name matters. When users compare vendors, provider descriptions matter. But if every tool embedding contains the same long provider marketing copy, tools under the same provider become too similar.

Categories also need care. Short, stable categories are often good in vector text. Extremely broad categories such as “data,” “tool,” “API,” or “enterprise service” add little and may pull unrelated tools together.

A better framing is not “delete this field.” It is “put each field in the right place”:

  • Tool name, core description, capability tags: suitable for embeddings
  • Resource names, aliases, abbreviations, English terms: suitable for full-text search, sometimes lightly useful in embeddings
  • URL, method, host, version: usually better for filtering, display, or debugging
  • Provider name: can appear briefly in embeddings
  • Long provider description: better for provider-level search, not repeated in every tool vector

A simple rule: the more a text fragment is shared by many records, the more cautious you should be about putting it into every record’s vector.

One Vector Will Not Solve Every Search Problem

Vector search is good at semantic similarity, but it has blind spots.

When the user searches for “SEC 10-K filing,” 10-K is a strong keyword. When they search for “ISIN lookup,” the abbreviation itself matters. When they search for “v2 invoice parse,” version and endpoint words may be the clue. Pure vector search may understand “annual report,” “securities disclosure,” and “invoice recognition,” but it is less sensitive to exact tokens than full-text search.

Full-text search has the opposite weakness. It does not know that “company registration lookup” and “business entity profile query” are close intents unless you explicitly supply synonyms.

Production systems are usually more stable with hybrid retrieval:

  • Vector retrieval handles semantic recall
  • Full-text retrieval handles exact words, abbreviations, paths, and proper nouns
  • Structured filters handle hard constraints such as status, region, permission, and category
  • Reranking or rank fusion reorders candidates from both paths

Hybrid retrieval does not have to be complicated. A simple RRF, Reciprocal Rank Fusion, can work well. It avoids comparing vector scores and full-text scores directly; it combines rankings instead. It is not perfect, but it is often robust.

Still, fusion is not magic. If the vector text is polluted, RRF can reduce the damage but cannot repair the semantic space at the root.

Thresholds Are Product Decisions

Many systems set a vector similarity threshold, such as “do not recall anything below 0.45.” It is easy to treat this as a technical parameter. In practice, it is a product decision.

A higher threshold gives cleaner results but may miss usable answers. A lower threshold recalls more results but may flood ranking with noise.

Worse, thresholds are not portable across models, corpora, or text templates. If you change embedding text from “all fields concatenated” to “clean capability description,” the score distribution changes. If you change embedding models, it changes again. If your corpus moves from mostly Chinese to mixed Chinese and English, it changes again.

Every change to indexing text, model, dimension, or retrieval strategy should be followed by a new score-distribution check. At minimum, look at:

  • Clearly relevant query-document pairs
  • Clearly irrelevant query-document pairs
  • Confusing near neighbors, such as different tools from the same provider or different capabilities under the same category

The threshold should come from these samples, not from a default configuration or a number copied from a blog post.

TopK Also Affects Accuracy

TopK is not only a performance setting. If K is too small, reranking has nothing to work with. If K is too large, the candidate pool contains too much noise and ranking gets harder and slower.

If the system has a reranker, retrieval usually needs to fetch more candidates than the final page displays. For example, show 10 results, recall 50 or 100, then rerank. But more is not always better. If the pool is full of junk, even an LLM reranker or cross-encoder reranker is forced to find gold in a pile of noise.

I prefer separating the K values:

  • Retrieval TopK: somewhat wider, so relevant results enter the pool
  • Post-fusion TopK: controls how many candidates reach reranking
  • Final TopK: driven by the product surface

These are not the same number and should not share one configuration.

Index Text Needs A Version

Many teams record the embedding model version but forget to record the text-template version used to generate the embedding.

That creates a debugging nightmare. The database may contain vectors from the same model, but half were generated from an old template that included long provider descriptions, while the other half came from a clean capability template. They live in the same vector space but have different semantic distributions. Search quality becomes inconsistent and hard to diagnose from individual records.

Alongside the model version, record the embedding-text version:

embedding_model = text-embedding-xxx
embedding_template = tool-capability-v3

When the template changes, explicitly decide whether to rebuild everything or rebuild gradually in the background. Do not let old and new templates remain mixed for a long time without labels.

Evaluation Sets Beat Feelings

“Search is inaccurate” is hard to discuss because everyone has a different idea of “accurate.”

A better practice is to maintain a small, high-quality retrieval evaluation set. It does not have to be large at first. Thirty to fifty valuable queries can reveal many problems. For each query, record:

  • The original user query
  • Expected results, such as top 1, top 3, or top 10
  • Obviously wrong results that must not appear
  • Query type, such as synonym, abbreviation, long natural-language intent, exact proper noun, or cross-language query

For the fictional API marketplace, examples could be:

query: "query A-share minute K-line"
expected: minute quotes, K-line API
not expected: provider introduction, macroeconomic calendar, weather API

query: "recognize amount and tax ID from invoice image"
expected: invoice OCR, receipt recognition
not expected: generic image compression, ID-card OCR

Every change to indexing text, threshold, fusion weight, or reranker should run against the evaluation set. Fixing a few online bad cases without an evaluation set often fixes one query and breaks three others.

Core Principles

First, define the retrieval unit. Are you searching tools, providers, document chunks, or a specific field? The unit determines the embedding text. Putting provider-level text into tool-level vectors is often the start of quality loss.

Second, short text is not necessarily weak. Clean text matters more. If the name, capability description, and core tags are clear, short text is often more stable than all-field concatenation.

Third, be conservative with shared text. Boilerplate, company descriptions, footers, disclaimers, URL prefixes, and navigation text repeat across many records and distort vector space.

Fourth, let full-text and vector search divide the work. Exact tokens, abbreviations, paths, model names, and version numbers belong to full-text search. Intent, synonymy, and natural-language needs belong to vectors.

Fifth, tune thresholds and TopK with data. A similarity threshold that “looks about right” is not enough. Score distributions, recall, and false positives need sample support.

Sixth, care about ranking, not only recall. A relevant result at rank 50 is almost invisible to users.

Seventh, keep logging bad cases. When a user says “this cannot be found,” do not only patch that query. Classify the problem: template noise, synonym gap, threshold too high, filter too strict, or reranker mistake. Bad cases are the most valuable training material for a retrieval system.

A Practical Refactoring Order

If an existing system already has inaccurate vector search, I would not start by changing the model. Model changes are expensive and can hide the real problem.

I usually start by sampling current embedding text. Read a few dozen records manually. If the text does not look like a searchable object to you and instead looks like a field dump, the model is unlikely to rescue it.

Then remove noise. Remove long shared text first, such as provider descriptions, template footers, and irrelevant navigation. Then remove URL, method, and host noise. Keep the name, core description, and a small number of high-quality tags.

Next, rebuild vectors for a small sample and run the evaluation set. Do not rebuild the whole corpus immediately. Compare old and new indexes on data that covers typical problems.

Then tune fusion. Once vector retrieval is cleaner, full-text weights, RRF parameters, and rerank pool sizes may all need adjustment. Rules that were added to compensate for noisy vectors may now become side effects.

Only after that should you consider changing models. Models matter, but a polluted input template only lets a better embedding model encode noise more elegantly.

Closing

Vector search accuracy is not about whether you use embeddings. It is about what you ask embeddings to express.

Concatenating every field is easy for data engineering, but it is often a slow poison for retrieval quality. A good vector index is more like editing: remove what is irrelevant, suppress what is repeated, and keep the semantics that truly represent the retrieval object. Full-text search, structured filters, reranking, and evaluation each need their own place.

Very often, the model is not failing to understand. We are simply showing it too much clutter.