A knowledge base AI assistant is a system that answers employee questions in the words of your own internal documents and shows which document each answer came from. The knowledge is not baked into the model: for every question the system finds the relevant passages in your policies, manuals, and contracts, then builds an answer from those alone. The value here is not the generated text as such but the discipline behind it: how closely the assistant sticks to the source, and how honestly it declines to answer when the data is not there.
Below is a method for taking a module like this into production: what to check before you start, how to assemble a set of evaluation questions, and what criteria to use when deciding that answer quality is good enough. This is one of the scenarios for AI modules for business, and it is the one where client and developer expectations diverge most often. The subject here is answering employees from internal documents. Answering customers in external correspondence is a separate scenario, built differently.
When do you need an assistant, and when is plain search enough?
An assistant pays off where an answer has to be assembled from several documents and the employee does not know the exact wording or file names. If the knowledge base is small and well structured, plain full-text search is cheaper and more predictable.
| Criterion | Full-text search | Knowledge base AI assistant |
|---|---|---|
| What it returns | A list of documents | A written answer with links to sources |
| Question wording | Needs exact words and terms | Handles synonyms, slang, typos |
| Answer from 3-4 documents | The employee pieces it together by hand | Pieces it together itself, but you have to be able to check the answer |
| Contradictions between documents | Shows both, a person sorts it out | May pick one version and hide the other |
| Cost per query | Close to zero | Billed for every question |
| Transparency | Full: the source file is visible | Depends on how source links are implemented |
| Where it wins | Dozens of documents, disciplined naming | Hundreds and thousands of pages, many new hires |
A practical sign: if the same question keeps showing up in team chat because searching takes longer than asking, an assistant pays for itself. If employees find what they need in a minute, you probably do not need an assistant.
How is this different from plain ChatGPT?
A public model does not know your internal policies, cannot tell a current version of a document from a cancelled one, and does not take an employee’s access rights into account. A knowledge base assistant is a separate system built around the model, and nearly all of the engineering sits in that surrounding layer.
Differences that matter in day-to-day operation:
- Source of the answer. The answer is built only from the passages retrieved from your documents, not from the model’s general knowledge.
- Link to the document and version. An employee can open the original and check the wording.
- Access rights. The assistant answers within what that person is allowed to see.
- Controlled refusal. The system is set up to say “this is not in the documents” rather than invent plausible text.
- Knowledge updates. Change a policy, reindex, and the answers change. Nothing needs retraining.
- Query log. You can see what people ask, what the system fails to answer, and where the knowledge base is empty.
How does the assistant arrive at an answer?
A question travels through five steps, and quality can be lost at each of them.
Splitting this into two phases is not a formality. When the assistant answers incorrectly, the investigation starts with the first phase: whether the right passage was found and whether it made it into the context. Investigations reach the second phase noticeably less often.
In the industry this approach is called RAG: generating an answer grounded in retrieved documents. In plain terms, the model does not recall the answer, it retells what was placed in front of it. Hence the main takeaway for acceptance: most errors are retrieval and document preparation errors, not a sign of a “dumb model”.
Which documents can be connected?
Almost anything stored as text can be connected. The question is what will be useful without a lot of preparation up front.
There is a third group between these two: needs preparation. This covers scans with no text layer (they need text recognition, and the output has to be checked), complex tables and presentations (their structure most often breaks during parsing), and long documents with no clear sections. They do get connected, but with separate work budgeted for it. Sources from the right column are taken selectively and only after cleanup.
Technically, sources are connected to the company’s existing environment: file storage, intranet portal, task tracker, accounting and corporate systems. Integration with internal systems is covered separately on the integrations page.
Why do you have to put the documents in order first?
Because the assistant does not improve the content of the knowledge base. It only speeds up access to it. If the base holds three versions of the same policy, the system will honestly answer from one of them, and half the time it will be the wrong one.
Minimum hygiene before you start:
- every document has an owner and a date of last review;
- current versions are physically separated from cancelled and archived ones;
- duplicates are deleted, not renamed;
- the file name shows what the document is, rather than “Policy_final_2_rev”.
This is not an abstract demand for tidiness. The system needs version and status labels for technical reasons: without them an outdated passage cannot be filtered out at the retrieval stage.
How should access rights be set up?
Rights have to be designed before launch and enforced at the retrieval stage, rather than by asking the model not to say too much. An instruction in a prompt is not a security mechanism.
A workable scheme:
- Roles, not personal lists. Access levels are tied to roles (employee, department head, HR, finance) and taken from the company’s existing user directory.
- A filter at the index level. Passages the user cannot access should not reach the answer context at all.
- Separate sections of the knowledge base. HR and finance documents sit in their own spaces with their own rules.
- Logging. Who asked what, and which sources they received. Needed both for incident review and for quality assessment.
- Processing environment. Documented separately: where document passages go during answer generation, and how long they are kept.
More on security and the data processing environment
What if the answer is not in the documents?
The right behavior is an explicit refusal. An assistant that produces plausible text when the case is unclear is more dangerous than no assistant at all: the employee cannot tell a confident correct answer from a confident wrong one.
How this works in practice:
- If retrieval does not find enough relevant passages, no answer is generated.
- The user sees wording along the lines of “the connected documents do not contain an answer”, plus a list of the closest materials.
- The system offers to pass the question to a person: the responsible expert, support, or a request queue.
- The unanswered question goes into a report. The accumulated list is a ready-made development plan for the knowledge base.
In the first months that report often turns out to be more useful than the assistant itself: it shows which documents the company simply does not have.
How do you prepare a set of evaluation questions?
The evaluation set is the main acceptance tool. Without it, an argument about quality turns into an exchange of impressions.
How to build it:
- The source of questions is employees, not developers. Take real requests from chats, tickets, and new hires’ questions over the past few months.
- Volume: 50-100 questions for the first launch. A smaller set does not provide enough coverage, and a larger one is hard to maintain by hand.
- Each question gets a reference answer and a reference source (document and section) written in advance. A subject matter expert does this, not the development team.
The set should cover different types:
| Question type | What it checks |
|---|---|
| Factual | There is an unambiguous answer in a single document |
| Composite | The answer is assembled from several documents |
| Trap | The knowledge base has no answer, a refusal is expected |
| Outdated | Both an old and a new version exist, the new one is needed |
| Access-based | The employee should not see this document |
| Conversational | Asked in slang, with typos, without the proper terms |
The set is frozen and rerun after every significant change: new documents, a different model, changed retrieval logic. It is a regression test, not a one-off exercise.
How do you assess answer quality?
Quality breaks down into six measurable characteristics. Assess them against the evaluation set, with two independent reviewers and an expert acting as arbiter when the two disagree. The company sets its own target for each characteristic, based on the cost of an error.
Substantive correctness of the answer
Whether the system answers the question that was asked, and whether the meaning matches the reference answer. A three-point scale works better than percentages: full answer, partial, wrong. The share of partial answers is counted separately: they are more dangerous than outright errors, because they look convincing.
Faithfulness to the source
Every statement in the answer has to be present in the attached passages. Anything the model adds on its own counts as an error, even if it happens to be factually true. This is the key trust metric: it is what separates a knowledge base assistant from a general model.
Correctness of the link
The link should lead to the document and section the answer actually came from, and it has to be the current version. A common failure, and an easy one to miss: the answer text is right, but the link points to the clause next door. The employee opens the source, does not find the wording, and stops trusting the system.
Ability to refuse
Two numbers are counted here. Correct refusals: the share of trap questions where the system properly answered that this is not in the documents. False refusals: the share of questions that did have an answer in the base but got a refusal instead. You push the first as high as it will go and keep the second within reasonable limits, or people stop using the assistant.
Handing a question to a person
The whole route is checked, not just the text: the question reached the right expert or the right queue, it carries the context of the conversation, and the employee can see the status. This is also where you measure what share of requests ends up with a person, and whether that share falls as the base grows.
Speed and cost of a single query
Speed is measured with two numbers: the time until the first words of the answer appear, and the total time to a finished answer with links. The first has a stronger effect on how the system feels to use.
Work out the cost of one query in advance and multiply it by a realistic volume: the number of employees, how often they ask, peak days. What goes into it: the amount of context passed, the model chosen, repeated follow-up questions, and regular reindexing of documents. If nobody worked out the cost per query before launch, it will make itself known in the first month of production load.
Cloud or your own environment: what does the choice affect?
The choice affects the speed of getting started, the cost structure, and the demands on the company’s internal resources more than it affects answer quality.
| Criterion | Cloud model | Model inside the company environment |
|---|---|---|
| Time to start | Weeks | Longer: hardware and setup are needed |
| Quality on hard questions | Usually higher | Depends on the size of the available model |
| Cost structure | Payment per volume of queries | Up-front investment plus maintenance |
| Data | Passages go to an external provider | Do not leave the perimeter |
| Model updates | Automatic, sometimes changes behavior | Controlled by you, takes resources |
| Infrastructure requirements | Minimal | Servers with accelerators, administration |
| Dependence on connectivity | Needs stable external access | Works in isolation |
A practical compromise for sensitive data: keep the index and retrieval inside the environment, and send only de-identified passages to the external model, just the ones needed to word the answer. Individual sections of the base can be closed off entirely and served by a local model. The decision follows from your requirements for the data, not from general considerations.
Why start with a pilot on 20-50 documents?
A pilot on a limited section of the knowledge base is a cheap way to answer the main question: whether you can reach acceptable quality on your documents. Testing that across the whole base is slow and expensive, and the result is the same.
How to choose a section for the pilot:
- a high frequency of questions (visible in chats and tickets);
- documents in acceptable shape, with clear owners;
- an expert available to review answers;
- an error is not critical for the client or for accounting.
Agreed before the start: the evaluation set, the acceptance threshold, the length of the pilot, the reviewers. Assessed at the end: results across the six characteristics above, the real cost per query, and the volume of work to put documents in order, extrapolated to the whole base.
A pilot is useful in one more way: it shows how much time experts spend reviewing answers. This is a hidden cost that companies usually remember too late.
How the stages of work and the production launch are organized
What makes up the cost?
It is worth naming a specific figure only after reviewing the documents and the requirements. Below are the factors that move it the most.
- Size and condition of the knowledge base. A thousand tidy pages cost less than three hundred assorted scans.
- Source formats. Scans, complex tables, and presentations need separate processing and checking.
- Number of source systems. Each integration is a separate piece of work: access, synchronization, handling updates.
- The access rights model. One shared section for everyone and a role-based rights matrix are jobs of very different size.
- Cloud or your own environment. Local hosting adds infrastructure and administration.
- Quality requirements. A high bar for faithfulness to the source and for refusals takes more iterations of retrieval tuning.
- Volume of queries. Affects operating expenses, not development.
- Languages. A multilingual base and answers in several languages make both retrieval and review harder.
- Ongoing support. Reindexing, quality control on new documents, extending the evaluation set.
How we estimate the cost of AI projects
When should you not build an AI assistant?
There are two different kinds of situation here, and they often get confused. In some, an assistant is not needed at all. In others it is needed, but not as the first step, and that is a different conversation.
An assistant will not solve the problem
- The knowledge base is small. A few dozen pages with a clear table of contents are covered by search and structure. An assistant costs more here than it gives back.
- The questions call for a calculation, not text. “How much did we ship to this client in June” is a query to the accounting system and its reports, not to documents. Scenarios like that are covered by corporate systems and reporting.
- A high cost of error with no resource to check. If the answer feeds a legally significant or financial decision and there is nobody to verify it, automating the answer gains nothing.
- No owner on the business side. A project with nobody accountable for the content of the base, and for working through the report of unanswered questions, fades out within a couple of months however good the technology is.
An assistant is needed, but it should not be the starting point
- The knowledge base is not maintained. Documents are out of date, updated once a year, nobody is accountable for the content. The assistant will simply hand out stale answers faster. Owners and regular review first, then automated access.
- Documents contradict each other. Until there is an owner who decides which version is current, the system cannot pick the right one. It will pick one at random. Sorting out versions is the first stage of the project, not an obstacle to it.
- The knowledge lives in people’s heads. If the key processes are not written down anywhere, the assistant has nothing to search. Describing the processes is exactly the work such a project starts with.
The difference is a practical one. In the first case the assistant will not help, and it is more honest to say so up front. In the second the task is solvable; the first stage just goes to the documents rather than the model.
How to tell whether the company is ready: a checklist
Tick the items that are already done.
The first two groups determine where the project starts. If they are covered, the next step is a pilot on one section. If there are gaps, documents come first: owners, versions, access rights. The third group comes together during the pilot.
Where to start
We build modules like this around a company’s processes: connecting sources, retrieval that respects access rights, refusal monitoring, and acceptance against a set of evaluation questions. What comes next depends largely on how you did on the checklist.
If the first two groups are covered. We take the section with the highest frequency of questions, assemble the evaluation set, run a pilot on 20-50 documents, and agree the acceptance criteria before we start. The results show answer quality, the real cost per query, and the scope of work for the rest of the base.
If items are missing. We start with the documents rather than the model: what exists, who is accountable for it, which versions are current, and where the duplicates are. The output is a map of what needs putting in order and an estimate of how long it will take. A review like that is worth doing on its own merits, even if the assistant is a long way off. The format of the work is described on the pre-project audit page.
Both cases start with the same conversation: what kind of questions, how many of them, and who answers them today.
Discuss your project or look at the AI module scenarios for business.
FAQ
Do we need to fine-tune a model on our documents?
In most cases, no. The assistant retrieves passages from your documents and answers from them, so updating the knowledge comes down to reindexing. Fine-tuning is discussed separately and only for narrow tasks, for example specific industry terminology.
How quickly do answers update after a policy is edited?
After the changed document is reindexed. This is usually set up automatically: the system tracks changes in the source and updates the index without human involvement.
What should we do if the assistant gives a wrong answer?
Work through the cause step by step: whether the right passage was found, whether it reached the context, whether the answer matches the passage. More often than not the problem is at the retrieval stage or in the documents themselves, rather than in the model. That question is added to the evaluation set so the error does not come back.
Can we avoid sending data to external services?
Yes, by hosting the model inside the company environment. That raises the infrastructure requirements and the time to launch. An intermediate option is also possible, where the index and retrieval run internally and only de-identified passages go out.
How many documents are needed to start?
For a pilot, 20-50 documents from a single section with a high frequency of questions is enough. That is sufficient to assess answer quality and to gauge the volume of work for the rest of the base.
Who should check answer quality: the developers or the client?
Reference answers and sources are prepared by an expert on the client side, who also acts as arbiter in disputed cases. The development team is responsible for running the evaluation set, analyzing errors, and tuning retrieval.
Will the assistant replace the support team or a mentor for new hires?
No. It handles a portion of repetitive questions that have an unambiguous answer in the documents. Complex and non-standard cases still go to a person, and the route for that handover is designed together with the assistant.
