fix(store): write index atomically and recover from corrupted index files#269
Open
rusel95 wants to merge 2 commits into
Open
fix(store): write index atomically and recover from corrupted index files#269rusel95 wants to merge 2 commits into
rusel95 wants to merge 2 commits into
Conversation
…iles os.Create truncates index.gob in place before encoding, so any unclean shutdown mid-write (reboot, OOM kill, signal) leaves a truncated file behind. On the next start gob decoding fails with 'unexpected EOF' and grepai has no recovery path: the corrupted file persists and every watch/search/status attempt fails forever. Two changes, both mirroring the pattern already used by trace/store.go and rpg/store_gob.go: - persistUnlocked now encodes into a temp file in the same directory, fsyncs it, and atomically renames it over index.gob via fileutil.ReplaceFileAtomically, so a crash mid-write can never produce a truncated index. - loadUnlocked treats a gob decode failure as a rebuildable-cache problem instead of a fatal error: the corrupted file is quarantined to index.gob.corrupt, a warning is logged, and the store starts empty so the next scan rebuilds it. Fixes yoanbernabeu#178 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tine failures Review findings on the original commit: - quarantine now uses plain os.Rename: ReplaceFileAtomically's remove-then-rename fallback could delete a .corrupt file freshly quarantined by a concurrent process (Load holds only a shared lock) - a loser of the quarantine race adopts the winner's recovery instead of resurfacing the fatal decode error - a genuine quarantine failure (e.g. read-only dir) is no longer swallowed: the decode error now wraps the rename error - tests: concurrent recovery, failed-persist-preserves-index, zero-byte index, stale .corrupt replacement Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #178.
GOBStore.Persistwrote the index withgob.NewEncoder(file).Encode(...)directly intoindex.gob. Any crash, OOM-kill, or power loss mid-write left a truncated file behind, and every subsequent command failed forever with:There was no self-heal: the only way out was to manually delete
.grepai/index.gobin the project and in every linked worktree (the watcher indexes them all). Multi-worktree setups made this a recurring, hard-to-diagnose breakage for both humans and AI agents using grepai via MCP.Fix (two independent layers)
1. Atomic persist — the root cause.
persistUnlockednow writes to aCreateTempfile in the same directory, thenSync → Close → renameoverindex.gob. A crash mid-write can no longer produce a truncated index; the previous good index survives any failed persist. This is the exact pattern already used bytrace/store.goandrpg/store_gob.go— the main index was the only store missing it.2. Corruption recovery — heals existing damage. If
loadUnlockedfails to decode (indexes truncated by previous versions, disk issues), it quarantines the bad file toindex.gob.corrupt(evidence preserved for debugging, never silently deleted), logs a loud warning with the path and the next step, and starts with an empty index so the next scan rebuilds it.The recovery path is race-safe:
Loadholds only a shared flock, so two processes (thewatchdaemon retry loop plus asearch/MCP call — the exact #178 scenario) can hit the recovery branch concurrently. The quarantine uses plainos.Rename; a loser whose rename fails withIsNotExistadopts the winner's recovery instead of resurfacing the fatal error, and can never delete the winner's.corruptevidence. If the quarantine genuinely fails (e.g. read-only directory), the returned error now wraps both the decode and rename errors so a failed self-heal is diagnosable.Tests
.corruptquarantinedLoads on the same corrupt index → all recover, evidence survives (-race).corruptfrom an earlier recovery → replaced by the newest evidenceindex.gob.tmp-*files behindKnown limitations (intentionally out of scope)
search,status, MCP) recover to an empty index and log a warning, but don't trigger a rebuild themselves — only the watcher rebuilds. A follow-up could suggestgrepai watchin the CLI output when recovery fired..corruptand can be restored manually.index.gob.tmp-*from a crash mid-persist are not swept on startup (same behavior astrace/store.goandrpg/store_gob.go).🤖 Generated with Claude Code