AI knowledge systems: how to make your company's documents answer questions

A new employee asks how many days' notice a supplier contract requires. The answer is in a PDF signed two years ago, in an amended version on the shared drive and in the head of procurement's memory. Most businesses already hold the answers to most of their questions. They are scattered across policies, contracts, manuals, price lists and email threads, and finding them takes a person who knows where to look.

An AI knowledge system lets people ask those documents directly, in plain language, and get an answer with the source attached. This guide explains how it works, which questions it handles well and which it does not, the four reasons answers go wrong and the guardrail for each, what to prepare, whether to build, buy or own one, and how to tell whether it works. It is written for owners and operators, not engineers.

What is an AI knowledge system, in plain words?

An AI knowledge system is a search engine and a writer working as a pair. The search finds the passages in your documents that bear on a question. The language model then writes an answer using only those passages and cites where each point came from. Ask “what is our refund window for annual contracts?” and you get a short answer with a link to the clause in the policy, instead of a list of files to open.

The technique is called retrieval-augmented generation (RAG), a name that comes from a 2020 paper by Lewis and colleagues presented at NeurIPS. The authors started from a known weakness of language models: the facts they learn in training are stored inside the model's parameters, and for such models, showing where an answer came from and updating what they know remained open research problems. RAG adds a second, external memory: a searchable index of documents that the model consults at the moment of each question.

That external memory is what makes the approach practical for a business, and the same paper tested it by swapping the index. Asked about 82 world leaders whose posts changed between 2016 and 2018, the system answered correctly 70% of the time with a 2016 Wikipedia index for 2016 leaders, and 68% of the time with a 2018 index for 2018 leaders. Given the index from the wrong year, it answered correctly only 4–12% of the time. Nothing was retrained. The answers followed the documents, which is both the strength of RAG and its main risk.

How a question becomes an answer

1. Prepare. Documents are collected from where they live (a shared drive, a wiki, a help centre), split into short passages and indexed. Each passage is tagged with its source, its date and who is allowed to see it.
2. Retrieve. When someone asks a question, the system searches the index for the passages most likely to contain the answer, using only the documents that person is allowed to read.
3. Generate. The language model receives the question and those passages, writes an answer from them and cites the passages it used.
4. Decline or hand over. When the passages do not contain the answer, the system says so and, where it matters, passes the question to a person.

Why not train a model on your documents, or paste them all in?

Training a model on your documents puts the knowledge back inside its parameters, where, as the RAG paper notes, it is hard to update and hard to trace to a source. Pasting every document into one long prompt fails in a different way. A study of how language models use long inputs, Lost in the Middle, found that accuracy was usually highest when the relevant information sat at the beginning or end of the input and dropped when it sat in the middle, even for models built for long contexts. Retrieval keeps the input short and relevant, and updating the knowledge becomes a matter of updating a document.

Which questions does a knowledge system answer well, and which does it not?

A knowledge system is strongest when the answer is written down somewhere, in a passage or two, and the person asking needs it quickly. It is weak when the answer has to be calculated across many records, decided by someone, or has not been written down yet.

Typical first uses follow from this: an internal assistant that answers staff questions on HR policies and operating procedures, a support assistant that answers customers from product documentation, and a sales assistant that finds the right specification or contract clause. Each starts with one document set and one audience. If the job you have in mind means acting in other systems, such as booking appointments or updating records, you are describing an AI agent, and the knowledge system becomes one of its tools.

Why do AI knowledge systems give wrong answers?

Wrong answers have four common causes, and only one of them is the model inventing things. Each has a guardrail you can design in before launch.

1. Stale or conflicting documents

A knowledge system repeats what its documents say. If last year's price list is still in the index next to this year's, or two policies disagree, the model has to choose, and it tends to trust what it is given. In ClashEval, a Stanford benchmark of more than 1,200 questions presented at NeurIPS 2024, six leading models, GPT-4o among them, adopted incorrect retrieved content over their own correct knowledge more than 60% of the time. The further the content strayed from the truth, the less likely the models were to adopt it, which means small, plausible errors such as an old price or a superseded notice period are the ones most likely to pass unnoticed.

The business answers for the result. In Moffatt v. Air Canada, a 2024 decision of British Columbia's Civil Resolution Tribunal, the airline's website chatbot told a passenger he could claim a bereavement fare after travelling, while the airline's own policy page said the opposite. Air Canada argued that the chatbot was a separate legal entity responsible for its own actions. The tribunal called this a remarkable submission, held the airline responsible for all the information on its website, whether it came from a static page or a chatbot, and found that it “did not take reasonable care to ensure its chatbot was accurate”.

Guardrail: give every document set a named owner and a review date. Keep one source of truth per topic, and remove superseded versions from the index instead of leaving them beside the current ones. Sync the index from the system where documents are maintained rather than uploading copies by hand, and show each document's date next to its citation so readers can see how fresh the answer is.

2. Missing permissions

A knowledge system that can read the salary sheet can repeat it to anyone who asks the right question. The OWASP Top 10 for LLM applications lists sensitive information disclosure as a core risk, covering personal data, financial details, health records, confidential business data and legal documents. It warns that instructions in the system prompt telling the model not to reveal something may not always be honoured and can be bypassed through prompt injection. Its entry on vector and embedding weaknesses, new in the 2025 list and aimed at retrieval systems, recommends fine-grained access controls and permission-aware stores, with strict logical and access partitioning so that different groups of users cannot reach each other's data.

Guardrail: enforce permissions in the search step, before the model sees anything. Each passage carries the access rights of the document it came from, and each search is filtered by the identity of the person asking. Microsoft's documentation for Azure AI Search, for example, describes storing the groups allowed to see each document in the index and filtering every query by the groups the user belongs to. Test this with accounts at different access levels before launch, and keep sensitive document sets out of the index until their permissions are mapped.

3. Retrieval misses

Sometimes the answer exists but never reaches the model. Researchers at Deakin University who built RAG systems for research, education and biomedicine documented seven failure points. Two of them start in the search step. With missing content, the documents cannot answer the question: the system should say it does not know, but it can be drawn into answering anyway. With missed top-ranked documents, the answer is in a document that does not rank high enough to be passed to the model. Their conclusion is practical: a RAG system can only be fully validated in operation, and its robustness evolves over time rather than being designed in at the start.

Guardrail: test the search on its own before judging the answers. For each real question, check whether the right passage was retrieved. Fix the documents first: clear headings, one topic per section, tables and scanned pages converted to readable text, and the terms your staff and customers actually use, in every language they use them. A question written in Arabic about a policy stored in English has to find that policy too.

4. Confabulation: the model fills the gap

Even with the right passages in hand, a model can state something the passages do not say. NIST's Generative AI Profile calls this confabulation: a system confidently presenting erroneous or false content, often called hallucination. It explains that the statistical prediction behind these models can produce accurate output, and also output that is inaccurate or internally inconsistent. Retrieval reduces the problem without removing it. A peer-reviewed Stanford study of AI legal research tools from LexisNexis and Thomson Reuters, which retrieve from legal sources, found that each hallucinated between 17% and 33% of the time. That was less than general-purpose GPT-4, and the authors judged the vendors' claims of hallucination-free answers to be overstated. The study counted an answer as a hallucination when it was incorrect or misgrounded, meaning it cited a source that does not support the claim.

Guardrail: require a cited source in every answer, linked to the passage rather than just the file, and check during testing that each citation actually supports the claim beside it. Instruct the system to answer only from the retrieved passages and to say “I don't know” when they do not contain the answer, then test that it does, using questions the documents cannot answer. Where an answer carries weight, such as a price, a legal term or anything touching health or safety, route the question to a person.

What should you prepare before building a knowledge system?

Most of the work that decides quality happens before any code is written, and most of it is about documents and people.

1. Choose one document set and one audience. HR policies for staff, product documentation for customers or contracts for the finance team. A narrow first scope is easier to clean, to permission and to test.
2. Take an inventory. For each document, record where it lives, its format, its owner, when it was last reviewed and who may see it. Scanned PDFs and images need converting to text before they can be searched.
3. Clean up. Remove duplicates and old versions, resolve contradictions and mark which document is the source of truth for each topic. Where answers exist only in people's heads, write them down first.
4. Name an owner for each set. The owner approves what goes in, fixes errors found in review and updates the documents when the business changes. Without an owner, the system drifts out of date while still sounding sure of itself.
5. Decide how it stays current. Connect the index to the system where documents are maintained so that changes flow through, and set a review cycle for each set. A policy change is not finished until the document has changed.
6. Collect real questions. Pull them from email, support tickets, chat logs and the people who answer them today, with the correct answer and its source for each. Include awkward ones and questions the documents cannot answer. This becomes your test set.

Should you build, buy or own a knowledge system?

Buy when your documents already live in one workplace suite or help-desk product that offers an AI assistant over its own content, that assistant respects the permissions you have already set, and the questions are internal and routine. It is the quickest way to learn whether people use the answers at all.

Build, or commission a build, when at least one of these is true:

• The documents are spread across several systems that no single product searches well.
• Answers have to reach people where they already are, such as WhatsApp, your website or an internal tool.
• Questions arrive in more than one language, such as Arabic and English, or in dialect.
• Permissions are specific to your organisation and must be enforced exactly.
• You need control over where documents and logs are stored and which model writes the answers.

Whichever route you take, make sure the knowledge itself stays yours: the cleaned documents, the index, the test set, the prompts and the logs, in formats you can move. One limit applies either way. The passages retrieved for each question are sent to the provider of the language model that writes the answer, so that provider belongs on your list of processors and its terms on data retention and training need checking. Our guide to owning versus renting an AI system covers hosting, data location and contract terms in detail.

Before you sign with any supplier, ask:

• Where are the documents, the index and the conversation logs stored, and who can read them?
• Are permissions enforced in the search step for each user, and how is that tested?
• Does every answer cite the passage it came from, and what happens when the documents have no answer?
• Which model writes the answers, and can it be changed without rebuilding the index?
• Can we export the documents, the index and the test set, and what happens if we stop paying?

Flowramo designs and builds AI knowledge systems as projects, alongside AI agents, business automation and custom AI applications. Each system is installed on hosting the client chooses and owns and connected to the client's own accounts, so the documents, the index and the logs stay with the business that relies on them. See what we build.

How do you measure whether a knowledge system works?

Start before launch, with the test set of real questions. The RAGAS framework, presented at EACL 2024, offers three useful lenses: faithfulness, whether the claims in an answer can be inferred from the retrieved passages; answer relevance, whether the answer addresses the question that was asked; and context relevance, whether the retrieved passages are focused rather than padded with irrelevant text. Tools can score these automatically with a language model, but grade a sample by hand as well, because an automated judge can be wrong too.

Keep measuring after launch. As the Deakin team concluded, a RAG system can only be fully validated in operation: new questions arrive, documents change and models are updated. Re-run the test set after every change to the documents' structure, the search settings, the prompts or the model, add the questions that went wrong to it, and review a sample of real conversations with the document owners every week.

An example: Lina answers only from clinic-approved knowledge

Lina, the AI receptionist we built for dental clinics on WhatsApp and the clinic's website, is built on the same idea for one narrow job. She answers patients only from knowledge the clinic has approved, retrieving from that knowledge for each conversation, and she does not diagnose. Urgent, clinical and sensitive questions go to the clinic team with the full conversation, so the patient never has to start again. She runs on hosting the clinic chooses and owns, connected to the clinic's own accounts.

A knowledge system for your own documents follows the same pattern with a different audience: approved sources, cited answers, permissions, an owner and a clear route to a person. If you are weighing one, tell us which questions it should answer.