What this MCP tool does (high-level):
This MCP tool exposes a local, project-bound knowledge base to an LLM agent. Concretely, it turns a directory tree of documentation files (Markdown/AsciiDoc/plain text) into a queryable knowledge database and returns short, relevant passages as LLM context.
- Input: a local folder (
root) containing project documentation (technical + domain knowledge). - Query: keywords or short natural-language queries generated by the agent.
- Output: a budgeted set of passages (not full documents) that can be inserted into the LLM prompt without blowing up context size.
In other words: it is “passage retrieval for a document-based knowledge DB” so the agent can ground answers in the project’s real docs while keeping token cost under control.
TD;DNR: This tool retrieves relevant text passages (not full docs) from a local knowledge base (<1000 files) for LLM context. It uses lexical retrieval with external DF statistics, optimized for low latency and strict token budgets. Trade-off: Precision over recall, simplicity over SOTA RAG
Flow: Files → Tokenize → SimpleDoc → Query → Score → Smooth → Threshold → Passages
Date: 2026-01-03
Scope: This document explains the design decisions behind the local “knowledge base” retrieval implementation under com.qaware.mcp.tools.knowledge (especially McpKnowledgeTool, SimpleDoc, Dictionary, and the NLP pipeline in Linguistic). It is written as an engineering rationale: it records why the system is built this way, what alternatives were considered, and why they were rejected.
The intent is to make the system maintainable, tunable, and defensible even though it favors simplicity, locality, and performance over “state of the art” RAG architectures.
We need a local, project-bound knowledge base that can be queried by an LLM agent via MCP. The knowledge base is a directory tree of text files (primarily *.md, *.adoc, *.txt). The output of retrieval is used as context for an LLM.
The crucial constraint is that LLM context is expensive:
- Every additional token increases cost and latency.
- LLMs can be distracted by irrelevant context.
- Some documents (e.g., manuals) are much larger than the context window.
Therefore, the retrieval system must return short, information-dense passages rather than entire documents.
- Budgeted passage retrieval: return relevant text blocks under a strict output budget.
- Precision-leaning behavior: slightly favor precision over recall. A small miss-rate can be acceptable if it substantially reduces context size in the common case.
- Local-first: no external services, no persistent databases, no network dependencies.
- Low latency and low memory footprint: retrieval should be fast (< typical LLM runtime by orders of magnitude) and lightweight.
- Mixed-language support: documents can be German and English; sometimes both languages exist within the same document.
- Maintainability through documented trade-offs: choices should be explainable and testable.
- Multi-user / shared service deployment.
- High-throughput concurrent search.
- Full-featured search engine semantics (phrase queries, fielded search, boolean operators, etc.).
- Fully unsupervised semantic retrieval as a primary strategy.
The design is explicitly shaped by the real-world usage scenario:
- Corpus size: typically ≤ 500 documents; worst case ≤ 1000.
- Query size: up to ~20 tokens; queries are primarily keyword lists (generated by the LLM as instructed). (The pipeline is also robust to short natural-language queries; see Section 6.)
- Query frequency: about one query every ~20 seconds.
- Execution mode: single user, local machine, not a shared service.
- Updates: knowledge base is checked into the project; updates mainly happen via edits and branch switches (~ once per hour).
The tool scans a configurable directory root (env var root). Correct usage is to set root to the knowledge base directory (or a small project subtree where KB docs live).
This is a deliberate operational assumption: the tool is optimized for project-bound KBs and is not intended to be pointed at arbitrary large directory trees.
The implementation does not currently enforce file size limits, exclude lists, or .git/build/node_modules heuristics.
This is intentional under the operating assumptions above:
- Supported file types are limited to
*.md,*.adoc,*.txt. - In typical projects, large generated trees do not contain huge text documents of these types.
- If scanning becomes slow or risky in practice, operational mitigation is to narrow
root.
Clarification (practical note): .git rarely contains *.md/.adoc/.txt content that matters for knowledge retrieval. If a project violates that assumption (e.g., large text artifacts checked into .git or generated directories), the intended mitigation is to set root to the actual KB subtree.
Guardrails (size limits, excludes) are documented as an optional extension (see Section 15.6) and should be introduced only if practice shows the need.
This is a local developer tool. It does not assume adversarial user input or deliberate collision / DoS attacks.
If this assumption changes (shared service, untrusted corpora), the design would need additional hardening (scan guardrails, resource limits, collision checks, etc.).
These assumptions justify decisions that would be questionable at 50k+ documents or high QPS.
At a high level, the system follows a lexical passage retrieval pipeline:
- Scan and read text files under a configurable root.
- Tokenize and normalize document text via an NLP pipeline.
- Build an in-memory per-document token structure (
SimpleDoc). - At query time:
- tokenize/normalize the query the same way.
- compute a weighted score per query term.
- apply scores to documents efficiently based on token occurrences.
- smooth scores to produce contiguous regions (“passages”).
- compute a global threshold from a histogram to meet a budget.
- output the best passages with source file references.
The system is synchronized per query to keep internal logic simple and safe under the single-user assumption.
Return contiguous text passages rather than ranking documents and dumping top documents.
- Manuals and long documents frequently exceed the LLM context window.
- Classic document retrieval forces either:
- sending an entire document (budget blow-up), or
- introducing chunking (risking wrong boundaries).
Passage retrieval solves this directly:
- We score positions in the original document.
- We extract only regions above a threshold.
- We keep a global output budget (via
maxcontent).
- Rejected: overly expensive in terms of context tokens; irrelevant content dominates.
- Rejected: chunking introduces a failure mode: important information may be split across chunks, or chunks may have unnatural boundaries.
- Also increases tuning complexity: chunk size, overlap, header handling, etc.
- Deferred (possible future option): more reliable than fixed-size chunking, but still adds complexity and ambiguity (headings may not reflect actual semantic boundaries).
The current approach deliberately avoids an extra indexing-time “cut point” and instead selects passages dynamically.
The primary retrieval behavior is lexical (token-based) using normalization, stopword filtering, and stemming.
- Budget efficiency and controllability: Keyword-driven scoring tends to be sharp and predictable.
- Debuggability: When a query misses, it’s possible to inspect normalization, stopwords, stem variants, and the dictionary mapping.
- Runtime simplicity: No vector database, no embedding model runtime, no index persistence.
- Small corpus: Semantic search is not “free” at small scales; it still needs chunking, embedding computation, and a similarity index.
A common criticism is that global statistics (e.g., Wikipedia DF) are “not domain-adapted”. That is true.
However, embedding-based semantic search typically uses globally trained representations as well:
- Most teams do not fine-tune large embedding models or LLMs for each domain.
- Therefore embeddings also reflect global training, not the project’s specific language distribution.
In other words:
- If domain adaptation is a requirement, both approaches need additional work.
- This system explicitly chooses a simple, locally controlled method with strong empirical performance under the target scenario.
- Rejected for now due to:
- extra operational complexity (embedding pipeline, chunking, persistent index),
- harder debugging, and
- unclear benefit relative to a tuned lexical approach under small corpora.
- Deferred: strong approach in large systems, but adds complexity and typically assumes larger data and stable infra.
Use a fast token stream abstraction (Tokens / Filter) with a mixed-language normalization pipeline:
TokenizerSimple: splits on non-alphanumeric boundaries in a conservative way.FilterToLowerFilterEnglishPossessiveFilterGermanNormalizationFilterStopWords(hash-based)FilterCombine+ Snowball stemming for German and EnglishFilterDeduplicateFilterDisambiguate(precision-biased selection among variants using DF statistics)
- Mixed-language within one document requires per-token handling. A per-document language setting would fail for bilingual text.
- Stemming improves recall of morphological variants.
- Stopword filtering reduces noise and saves budget (a key system goal).
- Disambiguation is used intentionally to favor precision: only emit variants that are plausible based on DF statistics.
Clarification (stopwords): The stopword set is intentionally simple and combined (DE+EN). It is derived as a classic “top-N by DF in Wikipedia” list. This favors robustness and speed over linguistic completeness.
Clarification (DE/EN stemming + DF-based disambiguation): Both German and English stem variants may be produced, and the resulting variants are scored/filtered via DF. If DF strongly indicates one dominant language/variant, that variant is kept. For genuinely ambiguous words (e.g., tokens that are common in both languages), multiple variants may be kept. In practice, DF is an excellent signal for mixed-language detection at token level.
Although the recommended query form is a keyword list, the pipeline is robust to short natural-language queries as well (stemming + stopword filtering removes most function words).
- Rejected because the corpus contains domain keys / identifiers like
A123Bwhere splitting can be harmful. - Tokenization is designed to be conservative and stable.
Per query token, compute a weight using an IDF-like function:
score = 5 * log(3_000_000 / (1 + df))
Then apply it to documents by traversing occurrences and setting position scores.
- We want rare, informative terms to dominate the signal.
- Query input is keyword-based; term weighting needs to reflect global informativeness.
- Term frequency is handled indirectly by scoring all positions where the term appears.
3_000_000 is an intentionally rounded estimate of the reference-corpus size used to build the background DF statistics.
- DF statistics were generated from Wikipedia dumps for English and German.
- The term acts like a corpus-size constant in the IDF-like transform.
Clarification (robustness): The exact value of this constant is not critical. Empirically, even large changes (e.g., multiplying it by 10) have negligible effect on MAP under the target scenario.
The constant factor (5) and passage-smoothing constants (e.g. MAX_DIST, slope) were tuned empirically to optimize retrieval quality (see Section 13).
Clarification (configuration philosophy): Most of these “internal” parameters are intentionally not exposed as user configuration. The intended user-facing knobs are root and maxcontent; exposing DF/scoring internals would complicate usage for minimal practical benefit in the intended environment.
A minimal threshold floor (e.g. 0.0001f) ensures that terms with extremely low weight (or edge cases in threshold selection) do not lead to “zero contribution” behavior. In practice it stabilizes scoring for very weak queries.
- Rejected as a primary method because:
- BM25 relies on a meaningful corpus-derived DF and document length normalization.
- With very small corpora (down to a single document), local DF and length stats become unreliable or degenerate.
- The objective is passage selection, not document ranking.
- Rejected: overly rewards frequent terms in long manuals, which acts against the budget goal.
DF values come from a large external reference corpus (Wikipedia DE/EN), stored as:
- a Count-Min Sketch
- plus quantization (bucketed mapping)
and are loaded as resources (e.g. df-de-en-freq-count-min-sketch.dat).
- A local KB can be as small as 1 document; local DF is then not meaningful.
- Even at 500–1000 documents, local DF can be unstable and too domain-idiosyncratic.
- Wikipedia provides stable, large-scale statistics that work well as global informativeness priors.
- DF is static and not domain-adapted.
- Rare Wikipedia terms might be over-weighted.
- Still, the approach proved empirically effective after tuning.
Clarification (DE/EN DF handling): DF statistics for German and English are stored in a combined representation. After stemming/disambiguation, DF(token) is used as a practical signal for which language/variant is dominant; this works well for mixed-language corpora.
Clarification (resource generation): The repository includes the generated DF resources, but not the full Wikipedia dumps or the local generator scripts used to produce them. This is a deliberate scope choice: dumps are large and not suitable for version control here.
- Rejected due to instability for small corpora (especially the “single manual” case).
- Deferred: potentially valuable, but adds complexity. It is not necessary for the current performance/quality envelope.
- Rejected (as discussed): embeddings are also globally trained and typically not domain-adapted.
Decision: Use a compact dictionary storing words in a single char[] and mapping a 64-bit hash to an integer ID.
Rationale:
- Very memory efficient.
- Fast lookup and addition.
- Suitable for repeated tokenization of documents.
Trade-off:
- Hash collisions are theoretically possible.
- With a high-quality 64-bit hash and realistic dictionary sizes, collisions are negligibly unlikely. If a collision occurs, two different strings would map to the same ID.
Clarification: Under the expected scale (≈100k distinct tokens) and a 64-bit hash space (2^64), the birthday-paradox probability of a collision is negligible for the intended use.
Threat model note: This tool does not assume adversarial inputs. Therefore it does not implement collision detection / collision hardening. If the threat model changes, explicit collision checks should be added.
Decision: Each document stores:
- the token IDs per position,
- begin/end offsets into the original source,
- and a reverse occurrence chain (
lastPos+previous) to iterate occurrences of a token without scanning the whole doc.
Rationale:
- Efficient scoring: for a token, traverse only its occurrences.
- Supports passage retrieval by scoring positions, not documents.
Decision: Do not build a global inverted index.
Rationale:
- Corpus size is small (≤1000 docs) and query rate is low.
- Current approach is O(#docs) per term but extremely fast (single map lookup per doc to determine presence).
- Inverted index would add:
- memory overhead,
- update complexity,
- more moving parts.
Alternative: Lucene
- Rejected: Great general solution, but too heavy for the target scenario and would require chunking or custom passage extraction anyway.
Raw hits create scattered spikes. We want coherent passages.
Decision: Apply smoothing in SimpleDoc.smooth using:
- neighborhood accumulation up to a maximum distance
- plus a linear “slope” fill between peaks
Rationale:
- Encourages contiguous regions to be selected.
- Helps capture supporting context around a key term.
Decision: Build a histogram of positive scores across documents and pick a threshold such that the result stays under a global budget (maxcontent).
Rationale:
- Enforces a budget for LLM context.
- Avoids returning “everything above a fixed score” which would vary wildly by corpus.
Trade-off:
- A long passage that happens to contain many query terms can dominate the global distribution.
- This is acceptable because the primary objective is a global context budget.
Why we don’t mitigate dominance prematurely:
- Dominance is not necessarily harmful: if one document contains most of the relevant information for a query, spending most of the budget there can be the best outcome.
- Diversity / anti-dominance constraints add complexity and tuning burden.
- Therefore, the system only introduces such constraints if real usage shows this to be a practical failure mode (see Section 15.5).
Passages are extracted as raw substrings of the original file text.
- The system guarantees cutting on token (word) boundaries.
- It does not attempt to preserve Markdown/AsciiDoc structural integrity (e.g., fenced code blocks or lists might be cut in the middle).
This is acceptable for LLM context. Making extraction markup-aware is possible but adds substantial complexity with unclear practical benefit.
- Rejected: could enforce fairness but is not aligned with cost control and “best overall context” objectives.
Indexing is incremental at file level:
- Scan the root directory for supported file types.
- For each file, compare
lastModified. - Only re-tokenize/rebuild
SimpleDocif the file changed. - Remove deleted files from the in-memory document map.
- The document map is incrementally updated.
- The global dictionary is monotonic (never evicted/cleaned).
Rationale:
- The process is frequently restarted, and memory is sufficient.
- Branch switches typically share most tokens; retaining dictionary entries is a net benefit (warm dictionary) and not harmful under assumptions.
- Rejected: would waste CPU, and is unnecessary given the update pattern.
- Rejected for now: complexity outweighs benefits under current lifecycle and corpus constraints.
query() is synchronized.
- The system is designed for a single local user.
- Serializing queries prevents accidental concurrency from the MCP environment.
- With one query every ~20 seconds, contention is effectively zero.
This enables:
- internal mutation without additional locks,
- predictable performance,
- simpler mental model.
Within a single query execution, work can be parallelized safely (e.g. per-document operations like smoothing/scoring) as a low-risk performance optimization.
What the design explicitly avoids is multiple concurrent queries, because that would introduce additional complexity around shared mutable state (“context handling”) without being needed for the intended scenario.
The simplest way to reason about correctness is:
- Only one query at a time mutates global state (due to
synchronized query()). - During a query, the document set / map is stable after
scan()completes. - Parallel work is only done on per-document state (e.g., smoothing and scoring arrays).
- Shared structures used during scoring (e.g., the dictionary mapping) are treated as effectively read-only for the duration of a query.
If these invariants are violated in future refactorings (e.g., concurrent scans, concurrent dictionary mutation while scoring), the current concurrency model must be revisited.
The codebase intentionally does not make every class fully thread-safe. Instead, it relies on the invariants above.
McpKnowledgeTool: query-level thread-safe (serializes requests viasynchronized query()); not designed for concurrent queries.SimpleDoc: not thread-safe in general, but safe under the current model because each query mutates per-document arrays and does not mutate a singleSimpleDocconcurrently from multiple threads.Dictionary: partially thread-safe (insertion is synchronized; lookup is lock-free). Correctness relies on not mutating the dictionary concurrently with query-time scoring.Linguisticresources (DF sketch, stopword set): effectively thread-safe after static initialization (immutable data structures / read-only usage).- NLP token streams / filters (
Tokens,Filter,TokenizerSimple, etc.): not thread-safe (they are stateful and are mutated while iterating). The current design is safe because each query/thread uses its own filter instance and does not share filter objects across threads.
This classification is meant as a guardrail for future changes: if you introduce concurrent scans, concurrent queries, shared filter instances, or dictionary mutation during scoring, you must revisit these assumptions.
- Rejected: not required by the scenario; would add complexity and risk.
Parameters (e.g., smoothing distance) and scoring behavior were tuned empirically using Mean Average Precision (MAP).
- TREC7-SDR (English)
- TREC8-SDR (English)
- Two private German corpora
The corpora used for evaluation cannot be checked into this repository:
- TREC data is under a license and requires a paid/approved source.
- The German corpora were created privately and are not distributable.
Therefore, this repository can share the methodology and evaluation code structure, but not a fully reproducible benchmark run.
- Define a set of candidate parameter settings (e.g., smoothing window).
- Run retrieval evaluation with MAP.
- Adopt settings that improve MAP consistently.
- Parameter tuning improved MAP by approximately 40%.
Clarification (baseline): This “~40%” figure is relative to a deliberately naïve, untuned baseline ("throw some reasonable constants in" / defaults before systematic tuning). It is meant as an order-of-magnitude indicator that tuning matters; it is not a claim of a rigorously benchmarked, publishable comparison against a strong IR baseline.
- It captures ranking quality over many queries.
- It provides a stable optimization target while iterating on heuristics.
Because the output is used as LLM context, additional quality dimensions matter:
- output budget efficiency (information per token)
- catastrophic miss rate (“the one crucial paragraph is missing”)
It is planned to be evaluated on the real internal document set.
This system first optimizes retrieval quality (robust ranking/selection). Afterwards, determine the practical token budget required to achieve a target recall (e.g. “95% of queries include the relevant passage”), and expose this budget as a user-tunable parameter (maxcontent).
maxcontent is a budget in (approximate) tokens, not characters.
The retrieval pipeline operates on a token stream where:
- tokens are essentially “words” after normalization,
- stopwords are removed,
- and stemming / disambiguation may reduce variants.
As a result, maxcontent is more directly tied to the cost driver of LLM context than a character-count limit.
Although the final output is emitted as raw substrings of the original text (including whitespace and punctuation), the budget is enforced on the token/position model:
- The histogram threshold is chosen to approximately meet the configured budget in token-units.
- The system aims to keep the total amount of selected text windows within that token budget; it is not a cryptographic/strict guarantee.
- Whitespace and punctuation differences are assumed to be second-order effects for German/English corpora.
If exact LLM token accounting is required, a model-specific tokenizer can be added as a secondary budgeting step (see Section 15.4).
This is intentionally not exact. A plausible heuristic is:
- stopwords often correspond roughly to 1 LLM token,
- “normal words” may average ~1.2 LLM tokens,
- if ~50% of raw words are stopwords, the emitted context might be on the order of
maxcontent * (1 + 1.2)LLM tokens.
The key point: this is a better proxy than characters for the expected input languages (German/English). Languages like Chinese are out-of-scope for this tool.
Clarification (practical tolerance): Minor deviations between the internal token budget and the final emitted raw substring size are acceptable. The primary goal is having a stable, explicit budget knob at all; classical “top-N documents” RAG approaches often lack this form of budget control and can fail catastrophically when a single top document is extremely large.
This section lists plausible extensions without committing to them.
- Structural segmentation (headings-based) as an optional boundary generator.
- Global+local DF interpolation when corpus size grows or domain drift becomes important.
- Hybrid retrieval: lexical candidate generation + optional neural reranking.
- Budget model refinement: estimate LLM token count more precisely if needed.
- Minimum diversity / anti-dominance constraints: prevent a single long passage with many keyword hits from consuming most of the budget (e.g., max passages per file, per-file quota, or diversity-aware selection).
- Scan guardrails (optional): file size limits, directory excludes, or other safety measures to protect against accidental scanning of huge trees.
Each of these adds complexity and should be justified by measured failures on the real KB.
The current knowledge base retrieval system is intentionally:
- local, fast, and memory-efficient,
- optimized for budgeted passage retrieval for LLM context,
- built around a tuned lexical pipeline with external DF priors,
- and engineered under explicit assumptions (≤1000 docs, ≤20 query tokens, single local user, low query volume).
Within this envelope, the design avoids heavy infrastructure (Lucene, vector DB), avoids brittle chunking, and achieves strong empirical quality after MAP-based tuning.