Summary: When building knowledge Q&A agents on internal documents, three problems appear most often: irrelevant retrieval, outdated knowledge, and unauthorized information leakage. This article gives engineering-level solutions in a problem-cause-fix format (demo sample post).
Problem 1: Retrieval Misses the Question
Cause: chunks split by fixed length lose context; users ask colloquially while documents are written formally. Fix: split by semantic sections, keep heading lineage in each chunk, and expand queries with synonyms.
Problem 2: Outdated Knowledge
Cause: documents are updated without re-indexing. Fix: index pipelines with version metadata and scheduled re-crawls; expire stale chunks automatically.
Problem 3: Information Leakage
Cause: retrieval ignores document-level permissions. Fix: enforce permission filtering at retrieval time, not at generation time, and log every answer with its sources.