feat(llm): pack chunks by token budget, parallelise, retry on truncation

[jasonm4130]

Two commits, both improvements to extract_corpus_parallel. Reviewable independently.

Summary

Commit 1: token-budget chunking, parallelism, optional tiktoken

• Replace chunk_size=20 static packing with greedy _pack_chunks_by_tokens(token_budget=60_000), grouped by parent directory
• Add tiktoken to the [kimi] extra; _estimate_file_tokens uses cl100k_base when available, falls back to chars/4 when not
• Run chunks via ThreadPoolExecutor capped at max_concurrency=4. on_chunk_done(idx, total, result) fires in completion order with the original submission idx so progress UIs work unchanged. max_concurrency=1 skips the pool to preserve sequential semantics
• Catch per-chunk exceptions, log to stderr, continue. One bad chunk no longer aborts the run
• token_budget=None falls back to legacy chunk_size-based packing for backwards compatibility

Commit 2: adaptive retry on finish_reason == "length"

• Plumb finish_reason out of _call_openai_compat and _call_claude (Anthropic's stop_reason == "max_tokens" is normalised to "length")
• Add _extract_with_adaptive_retry: when a chunk's response is truncated, split in half and recurse on each half. Recursion bounded by max_retry_depth (default 3)
• Single-file chunks that truncate can't recover — surface a warning rather than infinite-loop
• extract_corpus_parallel routes every chunk through the retry wrapper; recursive splits are invisible to callers (callback still fires once per top-level chunk with merged result)

Why

extract_corpus_parallel had three issues that compounded on real corpora:

#	Issue	Concrete failure
1	chunk_size=20 static packing has unbounded per-chunk cost	A 162-file mixed code/docs/images repo (~125k words) packed unevenly. One PNG-heavy chunk hit 282k input tokens and got 400'd by Moonshot's 262k context limit
2	Function name says "parallel" but body is a sequential for loop	Same 162-file repo took ~36 minutes wall-clock
3	A single chunk raising aborts the whole run	Lose all preceding chunks' work to one transient API error

After fixing 1-3, a fourth issue surfaces: chunks too dense to fit their JSON output in max_completion_tokens=8192 are silently truncated and contribute nothing. Adding a hard max_files_per_chunk cap reintroduces the "tune a static constant" problem the chunking commit set out to fix. The finish_reason signal is what the API gives us — acting on it is the principled fix.

Test plan

• Packer: small-file packing into one chunk
• Packer: starts new chunk when next file would exceed budget
• Packer: groups files from same directory contiguously
• Packer: oversized single file gets its own chunk
• Packer: rejects non-positive budget
• Tokenizer: uses tiktoken when available (mocked)
• Tokenizer: falls back to chars/4 when tiktoken absent
• Parallel: 4 chunks × 0.3s sleep finishes in <1s with max_concurrency=4
• Sequential: max_concurrency=1 preserves call order
• Resilience: simulated chunk failure logs to stderr, other chunks still merge
• Legacy: token_budget=None reverts to fixed-count chunking (45 files / chunk_size=20 = [20, 20, 5])
• Token-budget default: 50 tiny files pack into 1 chunk
• Adaptive retry: pass-through when finish_reason="stop"
• Adaptive retry: single-level split on finish_reason="length"
• Adaptive retry: recursive split when halves are still truncated (8 → 4+4 → 2+2+2+2)
• Adaptive retry: max_depth bounds recursion (no infinite loop)
• Adaptive retry: single-file truncation surfaces warning instead of recursing
• Adaptive retry: integration with extract_corpus_parallel — on_chunk_done fires once per top-level chunk
• Full existing test suite (459 tests total) passes with zero regressions

Companion

#623 — kimi-k2.6 reasoning fix. Independent, can land in either order.

[Qodo-Free-For-OSS]

Hi, extract_corpus_parallel() logs per-chunk exceptions and skips them, but the returned merged dict provides no structured indication that the run was partial. Callers cannot reliably detect missing chunks without scraping stderr.

Severity: remediation recommended | Category: reliability

How to fix: Return failure metadata to caller

Agent prompt to fix - you can give this to your LLM of choice:

Issue description

extract_corpus_parallel() continues after chunk errors but does not expose failures in the returned value.

Issue Context

This is a behavioral change from aborting-on-error to best-effort. Best-effort is fine, but callers need a programmatic signal that results may be incomplete.

Fix Focus Areas

• graphify/llm.py[349-446]

Recommended change

Add structured failure reporting, e.g.:

• Maintain failed_chunks: list[dict] with {idx, error} (and maybe chunk_files), and include it in the returned dict.
• Optionally add a fail_fast: bool = False parameter to restore old semantics when desired.
• Consider incrementing/returning chunks_succeeded/chunks_failed counts for easy UI reporting.

---

Found by Qodo. Free code review for open-source maintainers.

[Qodo-Free-For-OSS]

Hi, When tiktoken is available, token-budget packing reads and tokenizes each file, and extraction then reads the same files again to build the prompt. This increases I/O and can noticeably slow startup on large corpora.

Severity: informational | Category: performance

How to fix: Cache content during estimation

Agent prompt to fix - you can give this to your LLM of choice:

Issue description

With tiktoken installed, files are read once to estimate tokens and again to build the prompt.

Issue Context

This can add significant overhead for large corpora.

Fix Focus Areas

• graphify/llm.py[208-269]
• graphify/llm.py[80-93]

Suggested approaches

• Cache per-file truncated content during packing (e.g., dict[Path, str]) and allow extract_files_direct() / _read_files() to accept pre-read content.
• Or, estimate using st_size only as a fast heuristic even when tiktoken exists (but then accept less accurate packing).
• Or, add a flag to disable tiktoken-based estimation when throughput matters.

---

Found by Qodo code review