Skip to content

Avoid rebuilding the full entity graph for every bench commit #17

Description

@Iron-Ham

Summary

inspect bench rebuilds the full repository entity graph once per analyzed commit. On large repos this makes benchmark runtime scale linearly with a large repeated fixed cost. Because graph construction currently uses the current checkout, this is also coupled to inaccurate historical graph metadata.

Observed behavior

On a large polyglot repo with roughly:

{
  "file_count": 4601,
  "graph_entity_count": 52295,
  "graph_build_ms": 20636,
  "total_ms": 20719
}

A single release inspect diff run took about 20-23s, dominated by graph construction. Running:

inspect bench --repo <repo> --limit 3

took 73.91s, approximately one full graph build per commit.

Expected behavior

bench should avoid repeated full graph builds where possible, or make the cost explicit and configurable. Options:

  • Cache graph builds by tree SHA.
  • Build graph once when analyzing a commit range against a stable checkout/tree where that is semantically valid.
  • Add a fast mode that skips graph-dependent scoring and reports entity/classification-only metrics.
  • Emit timing/progress per commit so the command does not look hung on large repos.

Impact

The current behavior makes benchmark runs expensive on real production repositories and discourages using bench as a regression/performance tool.

Likely cause

bench loops over commits and calls analyze(&repo, DiffScope::Commit { ... }); analyze rebuilds context and graph every time.

Relevant code:

  • crates/inspect-cli/src/commands/bench.rs: per-commit analyze loop
  • crates/inspect-core/src/analyze.rs: build_context always lists files and builds a graph

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions