Build a Knowledge Base with AI: Answers Your Team Can Actually Cite

The goal

When you are done, you should have: a knowledge base that answers a plain question and links the answer back to the source. A new hire asks "what's the expense limit," and it gives the number with the paragraph from the policy document. Support asks "did this feature ship," and it gives the answer with the dated product announcement. Most importantly, when the documents genuinely don't contain the answer, it says so instead of inventing one.

This path covers internal reference knowledge bases: policies and processes, product documentation, support scripts and FAQs, project history, sales material, research notes. It does not cover three things — compliance work needing strict permission tiers (legal opinions, medical diagnosis, financial audit), real-time business data lookups (inventory, order status), and cross-system automation with write access. The first is a liability question, the second should query a database rather than documents, and the third is a different project.

The tool stack

Figure 1: The order of operations. Note the direction: questions first, documents second. Doing it the other way round — dumping every document in and then wondering what you can ask — is the most common starting point for failure.

Why this stack

Why the question list comes before the documents. "Turn all our company documents into a knowledge base" is a goal with no acceptance criteria; when you finish, nobody can say whether it worked. Rewrite it as "this must answer these 20 questions" and everything becomes decidable: which documents are required, which are noise, when tuning is good enough, and how to measure it after launch. Two hours here decides whether the following two days were wasted.

Why to start on a hosted tool that enforces citations. NotebookLM and similar tools give you something nothing else does: every sentence carries a source you can click, so you see exactly which passage it read. That immediately exposes the real problem — very often retrieval isn't inaccurate, your documents simply don't contain the answer. A self-hosted RAG stack hides this, because the model fills the gap from general knowledge and hands you a plausible answer that has nothing to do with your files.

When self-hosting is genuinely necessary. Four situations only: the data cannot leave your network; you need to join documents with live systems (look up the doc and the order); the corpus is larger than a hosted tool will hold; you need permissions by department or role. Otherwise the benefit rarely covers the maintenance. If you do build it, understand RAG, embeddings and vector databases first, and read Why RAG Answers Are Wrong: 10 Engineering Causes and Fixes for the engineering detail.

Why an evaluation set is mandatory. Without it, all your tuning is vibes. You shrink the chunk size, it feels better, tomorrow a different phrasing fails, and you cannot tell whether you improved anything. Twenty-five fixed questions, rerun after every change, is the only thing that turns this from folklore into engineering.

The full steps

Step 1: Write down the twenty questions

Don't invent them at your desk. Ask real users: what did new hires ask in their first week, which questions repeat most in the support queue, what were you yourself looking for last time you dug through files. Collect 20 real questions and sort them by frequency.

Also write three to five questions this knowledge base will not answer, to make the boundary explicit.

Step 2: Filter documents against the questions

Go looking for documents against that list rather than dumping the shared drive in. Every candidate answers three things: which questions on the list it addresses, whether it is the currently valid version, and who owns it.

Three kinds must be dealt with first: stale (last year's price list, a retired process), duplicated (three versions of the same policy in different folders), and drafts (anything titled "for discussion" or "v2 in progress"). You will usually delete more than you keep. That is normal.

Step 3: Preprocess

• Scans and image-only PDFs: run text recognition first, or you have loaded a blank page. The test is simple — can you select text inside the PDF?
• Very long documents: a several-hundred-page manual loaded whole retrieves badly because the topics blur together. Split by chapter into separate files.
• Tables: raw tables retrieve poorly. Add a paragraph describing what question the table answers and what each column means.
• Meeting recordings and training video: transcribe them, and prepend the date, participants and agenda.
• The parsing pitfalls are covered in How AI Reads a PDF: From OCR to Parsing to RAG.

Step 4: Standardise document headers

Give every document a uniform header: scope, effective date, last reviewed, owner, expiry condition. Those five lines pay off twice — during retrieval they are a strong signal that helps the model prefer the newer document, and during maintenance they turn "is this still valid" into something you can see at a glance.

Step 5: Load the documents

For hosted tools, this is an upload. For self-hosting, get the whole chain working with the plainest possible configuration first: fixed-length chunks with overlap, a general-purpose embedding model, top-N results. Do not add reranking, multi-route retrieval or query rewriting in version one — you have no baseline, so you cannot tell whether they helped. The chunking trade-offs are in RAG Chunking Strategy: A Practical Chunk Size Comparison.

Step 6: Build the evaluation set

Twenty-five questions in four categories:

1. Direct: the answer is stated explicitly in one document.
2. Synthesis: requires combining two or three documents.
3. Trap: the documents contain no answer, and refusing is the correct behaviour. At least twenty percent of the set.
4. Recency: there are old and new versions, and only the newer answer is correct.

For each, write the expected answer and the document it should cite, then rerun the whole set after every tuning change.

Step 7: Tune

Work in order, one change at a time, rerunning the set after each. First check whether retrieval surfaced the right passage — if not, adjust chunking and retrieval. Then check whether generation stayed faithful to what was retrieved — if not, change the system prompt to enforce citations and permit refusal. Only if the right passage is retrieved but ranked low should you add reranking; see What Is Reranking, and Why RAG Often Needs It.

Step 8: Define the maintenance mechanism

Launch is not the end. Settle three things: who owns each category of document, how often it is reviewed (policies maybe twice a year, product docs maybe every release), and how a document is retired. A knowledge base with no maintenance mechanism starts giving wrong answers within three months, and nobody notices.

Figure 2: The tuning triage. Decide first whether the problem is "not found" or "found but answered wrong" — the fixes are completely different, and tuning both at once is the slowest possible approach.

Prompts

1. Derive the document list from the questions

You are helping me plan an internal knowledge base. Below is the list of questions it
must be able to answer.

For each question, output:
1. What kind of document is needed to answer it (type and likely title, e.g.
   "travel expense policy").
2. Which specific fields or numbers that document must contain for the answer to
   be complete.
3. If we don't have that document, what to do — write it first, or move the question
   to the "this base does not answer" list.

Finish with a consolidated "required documents" list, sorted by how many questions
depend on each one.

Questions:
{paste your 20 real questions}

2. Standardise document metadata

Read the document below and produce a standard header to place at the top of it.
Fixed format:

Scope: (who it applies to, in which situations)
Effective date: (use the document's own wording; if absent, output "not stated" —
  do not guess)
Last reviewed: not reviewed
Owner: (the department or role named in the document; if absent, "to be assigned")
Expires when: (under what circumstances this document stops applying)
One-line summary: (under 30 words, what problem it solves)
Key terms: (3–8 terms specific to this document)

Hard requirement: every value must come from the document text. Where the text does
not say, write "not stated" or "to be assigned" — never fill in from common sense.
Finally, list separately: what critical information you think this document is missing.

Document:
{paste the document body}

3. The system prompt for knowledge base Q&A

You are the Q&A assistant for this team's internal knowledge base. Your answers must
follow these rules:

1. Use only the retrieved document passages. Do not complete gaps from your general
   knowledge.
2. Attach a source to every conclusion: document name plus section or paragraph.
3. If the retrieved passages are not sufficient, say "the current documents do not
   contain this information" and state what material would be needed. Do not offer a
   likely answer.
4. If passages contradict each other, present the conflict as-is, prefer the one with
   the more recent effective date, and warn the user that versions disagree.
5. For amounts, deadlines, percentages, names and approval levels, quote the source
   verbatim. Do not paraphrase them.
6. When asked about a process, answer as numbered steps and name the responsible role
   for each step.

Do not widen the scope of an answer in order to seem helpful. Saying "I don't know"
is correct behaviour.

4. Generate the evaluation set and scoring guide

Using the document list below, generate a knowledge base evaluation set of 25 questions
in four categories:

- Direct, 10: the answer is stated explicitly in a single document.
- Synthesis, 5: requires combining 2–3 documents.
- Refusal, 5: plausible-sounding questions with no answer in the current documents;
  refusing is correct.
- Recency, 5: content that exists in old and new versions, where only the newer
  answer is correct.

For each: question / expected answer / document that should be cited / scoring notes
(what counts as a pass). For refusal questions the expected answer is always
"should refuse and state what material is missing."

Document list:
{paste document titles with one-line summaries}

Worked example: one citation, one version conflict, one refusal

Test setup: macOS with the Claude Code CLI 2.1.226, --model sonnet, August 2026. The documents are fictional demo material; the commands and output were actually run.

Before you start: install the CLI

The demos use the Claude Code command line. Three commands to install and verify:

# 1. Node.js 22 or newer is required
node --version

# 2. Install globally
npm install -g @anthropic-ai/claude-code

# 3. Verify
claude --version        # should print something like 2.1.226 (Claude Code)
claude doctor           # checks that the installation is healthy

The first run needs a login: type claude in a terminal to open the interactive interface and follow the prompts to authorise your account (claude auth manages the login state afterwards). Once that is done, the one-shot -p calls below run directly.

Every command here passes --model sonnet so that you reproduce against the same model tier used for these runs; without it you get your account's default model and the output will differ.

Step one: create the material (copy and paste)

Build the smallest possible knowledge base — three documents chosen to reproduce the three problems described above. Paste the whole block into a terminal:

mkdir -p ~/demo/kb/docs && cd ~/demo/kb

cat > docs/travel-expense-policy-v3.md <<'EOF'
Scope: all employees, domestic travel
Effective date: 2026-03-01
Last reviewed: 2026-03-01
Owner: Administration
Expires when: superseded by v4

# Travel expense policy v3

## 2. Accommodation
Tier-one cities: 600 per night. Other cities: 400 per night.
Anything above requires written approval from the department head.

## 3. Submission deadline
Claims must be submitted within 15 working days of the end of the trip.
EOF

cat > docs/travel-expense-policy-v2-retired.md <<'EOF'
Scope: all employees, domestic travel
Effective date: 2025-06-01
Expires when: superseded by v3, kept for history only

# Travel expense policy v2 (retired)

## 2. Accommodation
Tier-one cities: 500 per night. Other cities: 350 per night.

## 3. Submission deadline
Claims must be submitted within 30 working days of the end of the trip.
EOF

cat > docs/remote-work-guidance.md <<'EOF'
Scope: employees on a permanent contract for at least 6 months
Effective date: 2026-01-15
Owner: People team

# Remote work guidance

Up to 4 remote days per month, requested at least 2 working days in advance.
Core collaboration hours are 14:00–18:00 on working days.
EOF

The three documents play different roles: one current version, one retired version whose numbers deliberately disagree with it, and one entirely unrelated document to see whether it gets dragged in. All three carry the standard header from step 4: scope, effective date, owner, expiry condition.

Next, save the third prompt above as the system prompt with the three questions appended:

cat > system.txt <<'EOF'
You are the Q&A assistant for this team's internal knowledge base. Your answers must
follow these rules:
1. Use only the retrieved document passages. Do not complete gaps from general knowledge.
2. Attach a source to every conclusion: document name plus section.
3. If the passages are not sufficient, say "the current documents do not contain this
   information" and state what material would be needed.
4. If passages contradict each other, present the conflict as-is and prefer the more
   recent effective date.
5. For amounts, deadlines and approval levels, quote the source verbatim.

Answer these three questions, each as its own paragraph:
Q1: What is the maximum I can claim for one night in a tier-one city?
Q2: What is the deadline for submitting a claim?
Q3: How do I claim local transport during a trip?
EOF

The three questions map to three categories in the evaluation set: Q1 is answerable but the two versions disagree; Q2 has the same conflict; Q3 is not covered by any of the three documents.

Step two: run it

Join the system prompt and the three documents into one input file, then make a single call:

{ cat system.txt; echo; for f in docs/*.md; do echo "===== Document: $f ====="; cat "$f"; done; } > input.txt
claude --model sonnet -p "$(cat input.txt)"

View full-size image

×

Terminal window: listing the three demo documents, joining the system prompt and documents into input.txt, calling claude, with Q1 flagging the version conflict and quoting both amounts verbatim and Q3 answering that the current documents do not contain the information

Figure 3: The actual terminal output of the commands in this section. Look at Q3 — it did not invent anything, it named the missing material.

Step three: read the result

Excerpt from the output:

Q1: The current policy, Travel expense policy v3 (effective 2026-03-01), § 2 Accommodation,
states: "Tier-one cities: 600 per night." However, the retired Travel expense policy v2
(effective 2025-06-01), § 2 Accommodation, states a different figure: "Tier-one cities:
500 per night." These two documents disagree — v2 is explicitly marked as retired and
superseded by v3, so 600 per night should be treated as current, but note that versions
disagree and confirm v3 is the active policy before relying on this figure.

Q3: The current documents do not contain this information. Neither Travel expense policy v3,
Travel expense policy v2 (retired), nor Remote work guidance has any section addressing
local transport claims during a trip. To answer this you would need a transport or ground
travel section of the travel expense policy that is not present in the current excerpt.

Three things happened as intended.

The conflict was surfaced rather than smoothed over. It did not silently pick one. It quoted both versions, explained why the newer one wins, and warned that they disagree. That is what the "effective date" line in the header bought — remove it and the model can only guess which is current.

Amounts and deadlines are quoted verbatim. 600, 500, 15 working days, each with the document and section, so you can click back and check. The system prompt line "for amounts, deadlines and approval levels, quote the source verbatim" is not decoration.

Q3 is a refusal, and it names the missing material. This is the valuable one. Local transport is a perfectly reasonable question, and the model is entirely capable of producing a smooth standard answer about receipts and reimbursement that reads flawlessly. It didn't, because the prompt explicitly permitted it to say it doesn't know. The twenty percent of refusal questions in your evaluation set exist to test exactly this behaviour.

Worth noting in passing: the unrelated remote work document was never pulled into any answer. At three documents that proves little — but at several hundred, this is precisely what "loading everything in" destroys.

What you have to supply

• Twenty or more real questions, from new hires, support logs, or your own search history
• Candidate documents, with the currently valid version already identified
• The owner for each category of document
• A glossary: internal jargon, product code names, department abbreviations — a major cause of retrieval failure
• A permissions boundary: which documents must not go into an all-staff base
• Meeting recordings or training video, if a lot of knowledge only exists verbally
• One real user willing to spend two hours testing it with you

What you end up with

• A knowledge base that answers with sources and refuses when material is missing
• An evaluation set and this round's score, as the baseline for future changes
• A "required documents" list marking what's missing and who will write it
• A uniform document header standard, so new documents enter the base consistently
• A maintenance table: document category, owner, review cycle, retirement rule
• A stated boundary of what the base does not answer, posted at the entry point

Time

Estimated for 100–300 documents at small-team scale:

The real time goes into filtering and preprocessing — the parts that look least like AI. Every demo that skips them was working with material that was already clean.

Cost

• Hosted route: most teams spend close to nothing — the Q&A capability is usually already inside a subscription you have, and lightweight tools have free tiers. The cost here is labour.
• Self-hosted route: three variable costs — a one-off embedding cost proportional to corpus size (usually the cheapest line item), ongoing vector storage, and generation cost per question, proportional to usage. On top of those sits one machine or managed service as fixed overhead.
• The real expense is maintenance labour: reviewing documents, loading new ones, retiring dead ones. Budget half a day to a day per month. Account for it at project approval, or the knowledge base will quietly die within six months.
• Where to save: separate "Q&A" from "summarisation" — route frequent simple questions to a cheap small model and reserve the expensive one for real synthesis, and cache repeated questions; see prompt caching for the background.

How it fails

Figure 4: Seven typical failures and the action that stops each. The first three happen before loading, the next two during tuning, the last two after launch — all three phases need someone watching.

1. Loading everything. Intuition says more documents is better; the opposite is true. Irrelevant documents take up slots in the retrieved results and push the right passage down. A base with 3,000 documents of which 200 are relevant usually performs worse than one holding only those 200.

2. Three versions of the same policy. The most universal problem in internal knowledge bases. The symptom is asking the same question twice and getting two different amounts. This isn't the model's fault — your material genuinely contradicts itself. Identify the canonical version before loading and remove the old ones rather than keeping them "just in case."

3. Scans that were never recognised. An image-only PDF uploads and looks fine, but retrieval never hits it. Test by trying to select text in a reader; if you can't, there is no text layer.

4. Never refusing. The most dangerous failure because it looks the most like success — fluent, well formatted, entirely invented. Two defences: explicitly permit and require refusal in the system prompt, and put twenty percent unanswerable questions in the evaluation set to test that behaviour specifically. Background in the hallucination entry.

5. Tuning without an evaluation set. The symptom is endless adjustment that feels better some days and worse others, ending with no idea which configuration to keep. Twenty-five fixed questions solve it.

6. Peaking at launch. Unlike a website, a knowledge base does not error when it breaks — it just starts giving stale answers, and the person asking assumes they are correct. That is more dangerous than not launching. The "last reviewed" line and the named owner exist to make this visible.

7. Permissions not separated. Salary tables, unreleased strategy documents and client contracts get dragged into the all-staff base, and then anyone can retrieve them with a question. The fix is separate bases, not tags or prompt instructions — a prompt is not a permission system.

Alternatives

If you only have a few dozen documents: you don't need a knowledge base. Put the relevant files straight into a long-context conversation; it often works better because the model reads the whole thing rather than fragments. The limits of that approach are in Are Long-Context Models Really Better? Where They Fail in Practice.

If the questions are highly concentrated: count them. If eighty percent of questions are the same dozen, a hand-written thirty-item FAQ is worth more than any retrieval system and costs far less to maintain. Do that first and consider a knowledge base only for the long tail.

If what you need is structured data: inventory, orders and reports should not be asked of documents. Have the AI write a query against the database — accuracy and freshness are not in the same league.

If the data absolutely cannot leave your network: go with local models and a self-hosted vector store. The trade-off is generally lower quality than cloud options plus a long-term need for someone who genuinely knows operations.

This is the first published entry in the AI Task Playbook series; the others (reading and organizing PDFs, product research, building a website, and more) will follow. For a deeper look at retrieval trade-offs in the meantime, see Vector Search, BM25, or Hybrid: How to Choose and Why RAG Answers Are Wrong: 10 Engineering Causes and Fixes.