Pathway 3 · Lesson 0375 min

Vector Databases & RAG

Grounding AI in your organisation's knowledge

Learning Objective

By the end of this lesson, you will be able to: Design a RAG pipeline for an organisational use case, explain the retrieve-augment-generate architecture, and evaluate RAG quality using faithfulness and relevance metrics.

74%

of leaders cite AI inaccuracy as their top risk — RAG is the solution

Engage

"What if your AI assistant knew everything in your organisation's knowledge base — every policy, every document, every past decision — and could answer questions about it accurately? That's RAG."

Explore

A clear technical explanation of how RAG works — from the architecture to the implementation.

Explain

The RAG Architecture: Retrieve → Augment → Generate

RAG solves the two biggest problems with using LLMs for organisational knowledge:

1. Knowledge cutoff: The model doesn't know about your recent documents

2. No proprietary data: The model wasn't trained on your internal knowledge

How RAG works:

1. Index: Your documents are split into chunks, embedded as vectors, and stored in a vector database

2. Retrieve: When a user asks a question, the query is embedded and the most semantically similar document chunks are retrieved

3. Augment: The retrieved chunks are injected into the model's prompt as context

4. Generate: The model generates a response grounded in the retrieved context

The result: AI that can accurately answer questions about your specific documents, policies, and knowledge — without retraining.

✅

Chunking Strategy Matters More Than You Think

How you split documents into chunks dramatically affects RAG quality. Too small: chunks lose context. Too large: irrelevant content dilutes the signal. Best practice: chunk by semantic unit (paragraph or section), not by fixed character count. Overlap chunks by 10-20% to preserve context across boundaries.

Vector Database Options

Database	Best For	Hosting	Key Feature
Pinecone	Production RAG at scale	Cloud (managed)	Fully managed, fast
Weaviate	Complex semantic search	Cloud or self-hosted	GraphQL interface, multi-modal
Chroma	Development and prototyping	Local or cloud	Simple Python API, open source
pgvector	Existing PostgreSQL users	Self-hosted	SQL + vectors in one database
Qdrant	High-performance production	Cloud or self-hosted	Rust-based, very fast

Explain

Evaluating RAG Quality

Three metrics for evaluating RAG performance:

Faithfulness: Is the generated answer grounded in the retrieved context? Does it add information not in the retrieved documents (hallucination)?

Relevance: Are the retrieved documents actually relevant to the query? Irrelevant retrieval leads to irrelevant or wrong answers.

Groundedness: Can every claim in the answer be traced back to a specific retrieved document? This is the gold standard for high-stakes RAG applications.

Tools like RAGAS (open source) automate these evaluations — essential for production RAG systems.

Elaborate · Worked Example

RAG Design: Internal Policy Assistant

Context

An HR team wants to build an AI assistant that accurately answers employee questions about company policies.

The Situation

Without RAG: The LLM answers from training data — giving generic HR advice that may contradict company-specific policies. RAG design: - Documents: All HR policies, employee handbook, benefits guides (updated quarterly) - Chunking: By policy section, ~500 tokens per chunk, 50-token overlap - Vector DB: pgvector (already have PostgreSQL) - Retrieval: Top 5 most relevant chunks per query - Augmentation: 'Answer based only on the following company policies: [retrieved chunks]. If the answer is not in the policies, say so.' - Evaluation: Weekly faithfulness check on 50 random queries

The Outcome

The RAG assistant gives accurate, policy-specific answers and says 'I don't have information about this in our policies' when appropriate — dramatically more useful and trustworthy than a generic LLM.

Try It · Reflect & Apply

Design a RAG system: Choose an organisational knowledge base (policies, documentation, FAQs). Define: What documents would you index? How would you chunk them? What vector database would you use? How would you evaluate quality?

Your notes are saved in your browser for this session.

Evaluate · Knowledge Check

1 / 3

What problem does RAG primarily solve?

Builder Engineering Studio

Production RAG Engineering Studio

Architecture blueprint

Specify every RAG stage: ingest → clean → chunk → embed → index → retrieve → rerank → augment → generate → cite → evaluate → observe → improve.

Engineering trade-off

Higher retrieval depth improves recall but raises latency and distracts the generator; more aggressive reranking can improve precision but adds compute. Optimise for the actual decision the user must make.

Quality gate

Require evidence-backed answers: the assistant must cite retrieved sources, abstain when support is missing, and route high-consequence questions to a human owner.

Evidence to measure

Monitor faithfulness, answer relevance, citation coverage, retrieval relevance, abstention quality, latency, cost per answer, and harmful-answer escape rate.

Build task

Draft a RAG acceptance test with five grounded questions, two deliberately unanswerable questions, and one adversarial prompt-injection attempt.

External video references

Keep Learning on YouTube

Open in a new tab

RAG Explained — Retrieval Augmented GenerationEmbedded lesson videoA clear technical explanation of how RAG works — from the architecture to the implementation.Watch · Watch on YouTube RAG in ActionDeepLearning.AIA production-oriented introduction to retrieval, vector databases, evaluation, and grounded generation.Watch · Tutorial series

Human-centred work design

Human Oversight Check

Use this original practice tool to test whether an AI-enabled workflow supports real human judgment, rather than merely placing a person at the end of an automated process.

Self-check

0/5

Start by making the work and decision rights explicit

Calibrate autonomy

Match AI freedom to consequence. Low-consequence drafting may run quickly; high-consequence decisions need explicit human approval.

Keep context visible

Make the objective, inputs, constraints, source material, and current system state easy to inspect before anyone approves an output.

Design active review

Review should require informed judgment, a plausible override, and the authority to act—not a symbolic sign-off after the fact.

Route exceptions clearly

Specify what uncertainty, conflict, missing evidence, or risk looks like—and who receives the escalation with enough context to decide.

Maintain human capability

Protect deliberate practice in the work that people may one day need to verify, challenge, or perform without AI assistance.

Apply it to one workflow

NASA-TLX workload reference Effective human oversight reference

Ready to move on?

Mark this lesson complete and continue to "Context Engineering at Scale".