Skip to content

fix(store): write index atomically and recover from corrupted index files#269

Open
rusel95 wants to merge 2 commits into
yoanbernabeu:mainfrom
rusel95:fix/gob-atomic-persist-corrupt-recovery
Open

fix(store): write index atomically and recover from corrupted index files#269
rusel95 wants to merge 2 commits into
yoanbernabeu:mainfrom
rusel95:fix/gob-atomic-persist-corrupt-recovery

Conversation

@rusel95

@rusel95 rusel95 commented Jul 2, 2026

Copy link
Copy Markdown

Problem

Fixes #178. GOBStore.Persist wrote the index with gob.NewEncoder(file).Encode(...) directly into index.gob. Any crash, OOM-kill, or power loss mid-write left a truncated file behind, and every subsequent command failed forever with:

failed to load index: failed to decode index: unexpected EOF

There was no self-heal: the only way out was to manually delete .grepai/index.gob in the project and in every linked worktree (the watcher indexes them all). Multi-worktree setups made this a recurring, hard-to-diagnose breakage for both humans and AI agents using grepai via MCP.

Fix (two independent layers)

1. Atomic persist — the root cause. persistUnlocked now writes to a CreateTemp file in the same directory, then Sync → Close → rename over index.gob. A crash mid-write can no longer produce a truncated index; the previous good index survives any failed persist. This is the exact pattern already used by trace/store.go and rpg/store_gob.go — the main index was the only store missing it.

2. Corruption recovery — heals existing damage. If loadUnlocked fails to decode (indexes truncated by previous versions, disk issues), it quarantines the bad file to index.gob.corrupt (evidence preserved for debugging, never silently deleted), logs a loud warning with the path and the next step, and starts with an empty index so the next scan rebuilds it.

The recovery path is race-safe: Load holds only a shared flock, so two processes (the watch daemon retry loop plus a search/MCP call — the exact #178 scenario) can hit the recovery branch concurrently. The quarantine uses plain os.Rename; a loser whose rename fails with IsNotExist adopts the winner's recovery instead of resurfacing the fatal error, and can never delete the winner's .corrupt evidence. If the quarantine genuinely fails (e.g. read-only directory), the returned error now wraps both the decode and rename errors so a failed self-heal is diagnosable.

Tests

  • garbage index → recovers empty, .corrupt quarantined
  • realistically truncated index (torn write) → recovers, then persists/reloads cleanly
  • zero-byte index (the most common artifact of the old bug) → recovers
  • 8 concurrent Loads on the same corrupt index → all recover, evidence survives (-race)
  • failed persist (read-only dir) → previous index survives untouched
  • stale .corrupt from an earlier recovery → replaced by the newest evidence
  • overwrite path leaves no index.gob.tmp-* files behind

Known limitations (intentionally out of scope)

  • Read-only consumers (search, status, MCP) recover to an empty index and log a warning, but don't trigger a rebuild themselves — only the watcher rebuilds. A follow-up could suggest grepai watch in the CLI output when recovery fired.
  • A decode failure caused by a transient read error (NFS hiccup, failing disk) is treated as corruption; the data is preserved in .corrupt and can be restored manually.
  • Orphaned index.gob.tmp-* from a crash mid-persist are not swept on startup (same behavior as trace/store.go and rpg/store_gob.go).

🤖 Generated with Claude Code

ruslanpopesku-ops and others added 2 commits July 2, 2026 12:53
…iles

os.Create truncates index.gob in place before encoding, so any unclean
shutdown mid-write (reboot, OOM kill, signal) leaves a truncated file
behind. On the next start gob decoding fails with 'unexpected EOF' and
grepai has no recovery path: the corrupted file persists and every
watch/search/status attempt fails forever.

Two changes, both mirroring the pattern already used by
trace/store.go and rpg/store_gob.go:

- persistUnlocked now encodes into a temp file in the same directory,
  fsyncs it, and atomically renames it over index.gob via
  fileutil.ReplaceFileAtomically, so a crash mid-write can never
  produce a truncated index.
- loadUnlocked treats a gob decode failure as a rebuildable-cache
  problem instead of a fatal error: the corrupted file is quarantined
  to index.gob.corrupt, a warning is logged, and the store starts
  empty so the next scan rebuilds it.

Fixes yoanbernabeu#178

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tine failures

Review findings on the original commit:
- quarantine now uses plain os.Rename: ReplaceFileAtomically's
  remove-then-rename fallback could delete a .corrupt file freshly
  quarantined by a concurrent process (Load holds only a shared lock)
- a loser of the quarantine race adopts the winner's recovery instead
  of resurfacing the fatal decode error
- a genuine quarantine failure (e.g. read-only dir) is no longer
  swallowed: the decode error now wraps the rename error
- tests: concurrent recovery, failed-persist-preserves-index,
  zero-byte index, stale .corrupt replacement

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GOB index corruption on unclean shutdown — crash loop with 'unexpected EOF'

2 participants