An internal AI assistant can help people find an answer in company documents, but uploading a folder is only the start. The useful work is deciding which documents are current, how the system finds the relevant passage and what it should do when the material doesn't answer the question.
Start with one repeated question family, such as an internal product-support guide. Make answers traceable to maintained sources before connecting more information.
Give the assistant a manageable knowledge set
Imagine a fictional equipment supplier whose staff repeatedly ask which accessories fit a particular product. Its first knowledge set contains approved compatibility lists and current product manuals. It excludes old offers, private customer records and staff discussions about possible future products.
Give each source an owner, version or review date, and a clear status. Remove superseded instructions from the active set or mark their limits in a way the search actually respects. A filename containing “final” is not a version policy.
If staff cannot decide which of two documents is authoritative, the assistant should not make that business decision for them.
Understand what retrieval adds
A common design searches your permitted documents for relevant passages and supplies those passages to an AI model when it answers. This is often called retrieval-augmented generation, or RAG.
Microsoft describes this retrieval pattern as bringing relevant source material into the model's input. It is a way to ground an answer in selected information, not proof that the final answer is correct.
You don't need to pick a search technology to write the brief. Tell the builder which questions need answering, which source should support each answer and what counts as a useful result.
Make the answer easy to check
For the fictional supplier, a good answer would name the compatible accessory, identify any relevant product version and link to the exact section of the current manual. It would not merely say “according to company documents”.
Ask for the source title, useful passage or page reference and a link the staff member is permitted to open. A citation to the right document can still point at a passage that does not support the claim, so check the connection between answer and evidence.
If the documents disagree, the answer should show the conflict and refer it to the owner. If the information is missing, it should say that the available material does not establish an answer. Don't ask it to improvise a compatibility claim to avoid an empty result.
Separate finding from writing
Test the retrieval and the answer as two steps. If the system finds the wrong product manual, rewriting the final answer is unlikely to solve the underlying problem.
Prepare questions covering exact product names, everyday descriptions, common abbreviations and misspellings. Include a discontinued item and a question your documents genuinely cannot answer.
For each, record:
- the document and passage that should be found, if any.
- the essential fact the answer must preserve.
- any condition or product version it must mention.
- the expected handoff when evidence is absent or conflicting.
Keep a few examples out of prompt development and use them for a fresh check. Otherwise the demonstration can become good at the examples its builder already knows.
Keep source changes connected to the service
Ask how long it takes for a corrected document to reach the assistant, and demonstrate it with a harmless test change. Check deletion as well as addition. A document removed from its original folder might still exist in a search index or copied knowledge store.
Name the person who receives reports of wrong answers and who can remove a faulty source. Keep enough information to reproduce a problem without publishing private documents in a general support channel.
Access matters too. The assistant should only retrieve information the current user is allowed to see. Check that boundary separately from answer quality. Correct facts can still be the wrong facts to disclose to a particular person.
Decide what a useful first release does
Begin with internal lookup and visible sources. Automatic customer replies or updates to business records add another set of decisions and tests.
Review unanswered questions to find gaps in the knowledge itself. Sometimes the best improvement is a clearer manual, a named owner or one reliable compatibility list. The AI connection is valuable when it makes that maintained information easier to use.