A vector database stores numerical vectors, usually embeddings produced from text, images or other objects, and retrieves nearby vectors under a chosen similarity measure. Production systems also need identifiers, source content or references, metadata filters, access controls, updates and operational guarantees. Some products are dedicated vector databases; general databases and search engines can also provide vector columns and indexes. The useful category is the required retrieval capability, not the vendor label.
A quick demonstration can embed a small clean corpus and return plausible neighbors, then fail when permissions, duplicates, changing documents, language, metadata filters and real latency arrive. Teams may tune index speed while ignoring whether the embedding represents the user’s task. They may also assume semantic similarity proves factual relevance. A vector database retrieves candidates; it does not validate the claim, preserve source authority automatically or make an AI answer grounded.
Start with the retrieval decision and a labeled query corpus. Compare keyword, structured filter, vector and hybrid methods before choosing infrastructure. Keep source identity, version, permissions and chunk relationships beside every vector. Select exact or approximate search from measured recall, latency, scale and cost. Treat the embedding model, chunking, index and reranking as one versioned retrieval system, and evaluate the downstream outcome rather than nearest-neighbor aesthetics.
Mechanics
The vector index is one layer in a retrieval system
An embedding model maps an object into a numerical space where a distance or similarity function is intended to reflect useful relatedness. Exact search compares the query vector against all candidates. Approximate nearest-neighbor indexes reduce search work and trade some recall for speed and scale. HNSW is one influential graph-based approach. Its paper describes the hierarchical navigable small-world method; actual behavior depends on implementation and tuned construction and search parameters.
General databases can add vector capability next to transactional data. The open-source pgvector project, for example, supports exact and approximate nearest-neighbor search in PostgreSQL. Dedicated systems may provide other scaling and operational features. Architecture should follow workload, consistency, filters, tenancy, updates and team competence. A new database is not required merely because the application uses embeddings.
| Component | Purpose | Failure question |
|---|---|---|
| Embedding model | Represent objects and queries | Does the space match the task? |
| Indexed unit | Define retrievable evidence | Is context split or duplicated? |
| Vector index | Find candidates efficiently | What recall is traded for speed? |
| Metadata filter | Enforce scope and narrow candidates | Is filtering permission-correct? |
| Reranker | Refine candidate order | Does added cost improve outcomes? |
Evaluation
Evaluate relevance with user questions, not a handful of demos
Create ground truth from the real decision. For a support search, the relevant item may be a current approved procedure; for proposal work, it may be evidence valid for a customer and date; for product discovery, diversity can matter as much as similarity. Label disagreement should be adjudicated and retained. Segment queries into identifiers, short ambiguous text, paraphrases, multiple languages and questions that need several pieces of evidence.
Measure retrieval separately from generation. Candidate recall asks whether useful evidence appeared; ranking measures where; answer evaluation asks whether the downstream system used it correctly. Compare against a lexical and structured baseline because exact terminology, codes and dates often favor them. Hybrid search can combine signals, but its weights also require evaluation. Re-run the suite after embedding, chunking, metadata, index or corpus changes.
- Label relevance for the actual user task.
- Keep exact and lexical search as baselines.
- Test realistic filters and permission scopes.
- Measure retrieval before generated answers.
- Version corpus, embeddings, index and evaluation.
What good looks like
Useful outcomes from vector database
- Retrieval is tested against representative queries and adjudicated relevant evidence.
- Every vector remains linked to an authoritative source, version and permission scope.
- Keyword, structured, vector and hybrid search have an evidence-based role.
- Index choices balance recall, latency, write behavior, memory and cost.
- Deletes and source changes propagate predictably to searchable content.
- The complete retrieval release can be reproduced, compared and rolled back.
Operating model
How to run the work
- 01
Define the retrieval task
Specify the user question, relevant unit, source authority, freshness and permission requirement. Build a query set with relevant, irrelevant, difficult and multilingual cases. Include exact identifiers and rare terms that semantic search may not handle well.
- 02
Design the indexed object
Choose the embedding model, normalization, chunk or item boundary, overlap, metadata and link to the original. Define how tables, headings, attachments, versions and deleted sources behave. Never make the vector the only surviving representation.
- 03
Compare retrieval methods
Run lexical, structured, exact-vector, approximate-vector and hybrid baselines. Measure candidate recall and ranking on the labeled set with realistic filters. Add reranking only if it improves the target outcome enough to justify latency and cost.
- 04
Operate the index lifecycle
Version the model, preprocessing, schema and index settings. Test inserts, updates, deletes, backup, rebuild and rollback. Monitor relevance, stale or unauthorized candidates, latency and cost as the corpus and queries change.
Evaluation
Questions that change the decision
- What object should be considered relevant to one query?
- Do exact terms, semantic meaning or structured attributes drive retrieval?
- Which distance or similarity measure matches the embedding model?
- Is exact search feasible at the real corpus size and latency target?
- How do metadata filters and access controls interact with approximate search?
- How will a source update or deletion remove every derived vector?
Failure modes
Where teams lose control
Chunking separates a claim from the qualification that changes its meaning.
Approximate indexing loses a relevant candidate under restrictive filters.
A model change mixes incompatible vector spaces in one index.
Near-duplicate content crowds the result list and hides diverse evidence.
Permission filtering occurs after retrieval and leaks sensitive information.
Deleted source content remains available through stale vectors or caches.
Measurement
Measure the finished job
Measure the completed workflow, including review effort and exceptions. Output volume on its own is not evidence of a better process.
- recall at candidate depth on adjudicated queries
- precision and ranking quality by query segment
- permission-correct retrieval and leakage attempts
- index freshness after create, update and delete
- latency by corpus size, filter selectivity and load
- storage, memory, embedding and query cost per accepted outcome
Questions
Common questions
What is a vector database?
It is a database or search capability that stores numerical vectors and retrieves nearby items using a similarity or distance measure. Production use also needs source identity, metadata, permissions, updates and operational controls.
Is a vector database required for RAG?
No. RAG requires a way to retrieve relevant evidence, which may use keyword search, structured queries, exact vectors, approximate vectors or a hybrid. Choose from corpus, queries, permissions, freshness, relevance and operational needs.
What is approximate nearest-neighbor search?
It uses an index to find likely nearby vectors without comparing every candidate, trading some recall for speed and scale. Measure that tradeoff with the real corpus, filters and labeled queries rather than relying on generic benchmarks.
How do you evaluate vector search?
Use representative labeled queries and measure candidate recall, ranking quality, permissions, freshness, latency and cost. Compare lexical, structured and hybrid baselines, then evaluate whether the downstream user outcome actually improves.
Sources
Primary references
About the author
Tony Kim
Tony writes about applied AI, dependable product engineering and the systems that turn complex response work into controlled delivery.
Zeke
AI product engineering for moving a software brief into a reliable production product.
Product leaders, founders and engineering teams. Start with the workflow, constraints and evidence you already have.
See Zeke→