Skip to content

feat(trace): tree-sitter symbol extraction for 16 new languages#261

Open
kryptt wants to merge 4 commits into
yoanbernabeu:mainfrom
kryptt:pr/treesitter-languages
Open

feat(trace): tree-sitter symbol extraction for 16 new languages#261
kryptt wants to merge 4 commits into
yoanbernabeu:mainfrom
kryptt:pr/treesitter-languages

Conversation

@kryptt

@kryptt kryptt commented May 29, 2026

Copy link
Copy Markdown

Depends on / stacks on top of #260 (which wires up --mode dispatch). This PR adds the actual language coverage.

Problem

Even after the dispatch fix in #260, tree-sitter symbol extraction only covers the original 9 grammars that landed via #152 (F#) and earlier (Go, JS/JSX, TS/TSX, Python, PHP, C#). For everything else — Ruby, Rust, Java, Scala, C, C++, Bash, Lua, Kotlin, Swift, SQL, Protobuf, HCL, Elm, TOML, Emacs Lisp — symbol extraction falls back to regex. Regex is usable but loses precision (Go method receivers, Python class docstrings, Rust trait impls, …) and silently misses syntax it doesn't know about.

We've already paid the CGo + grammar-linkage cost via #260. The marginal cost of linking more grammars and writing per-language queries is small in code and reviewable as one slice of LangSpec entries.

Solution

Single registry, one entry per language

trace/languages.go already holds the registry from #260. This PR turns it into the single source of truth for both grammar registration and symbol extraction:

type LangSpec struct {
    Name        string
    Extensions  []string
    GetLanguage func() *sitter.Language
    Queries     []NamedQuery   // ← new
}

type NamedQuery struct {
    Kind  string  // free-form, propagates to Symbol.Kind
    Query string  // tree-sitter S-expression with a @name capture
}

Adding a language is:

  1. Import the grammar package (smacker subpackage, or vendor at repo root).
  2. Add one LangSpec to treeSitterLanguages.
  3. Write trace/queries_<lang>.go with the S-expression queries.
  4. Drop a fixture in trace/testdata/.

No edits to extractor_ts.go. No new method per language. The dispatch picks up the new entry automatically.

New query-based extraction path

TreeSitterExtractor.extractViaQueries runs each NamedQuery against the parsed tree using sitter.NewQuery + NewQueryCursor. It applies #eq? / #match? predicates via cursor.FilterPredicates. Emits one Symbol per @name capture with the Kind taken verbatim from the corresponding NamedQuery.

The legacy nine languages keep their hand-walked extractGoSymbol / extractPythonSymbol / … — Queries is empty on their LangSpec and the extractor falls through to the existing walk. Zero behavior change for the original languages.

Vendored Emacs Lisp grammar

smacker/go-tree-sitter doesn't ship an elisp grammar, so we vendor Wilfred/tree-sitter-elisp (MIT) at repo root in elisp/, mirroring the pattern set by fsharp/. Pinned at the last ABI-14-compatible upstream commit (3a8f590) since smacker's bindings top out at ABI 14. elisp/README.md documents the provenance + re-vendoring procedure.

Covered macros: defun, defmacro, defvar, defconst, defcustom, defface, define-{minor,derived,generic,globalized-minor}-mode, cl-defun, cl-defmethod, cl-defgeneric.

Languages added (16)

Tier Languages Extensions
Top-8 priority (rich queries: classes/traits/structs/enums/methods/...) Ruby, Rust, Java, Scala .rb .rs .java .scala .sc
Medium (functions + main types) C, C++, Bash, Lua, Kotlin, Swift .c .h .cpp .cc .cxx .hpp .hh .hxx .sh .bash .zsh .lua .kt .kts .swift
Minimal (top-level definitions) SQL, Protobuf, HCL/Terraform, Elm, TOML .sql .proto .hcl .tf .elm .toml
Vendored (top-8 priority) Emacs Lisp .el

Test plan

New TestExtractor_LanguageFixtures is a table-driven test that exercises every language end-to-end:

```go
{name: "ruby", fixturePath: "ruby.rb", want: []string{
"class:Hello", "class:Standalone", "method:say",
"module:Greeter", "singleton_method:banner",
}},
```

Each fixture has ~5 known symbols asserted set-wise — extras allowed because grammars sometimes surface more than the bare minimum. 16 sub-tests, all passing.

The PR-1 compound_test.go cases that used .lua as the canonical regex-only extension are updated to .zig (still regex-only after this PR — no smacker Zig grammar).

`go test ./... -count=1` is clean. `go vet ./...` is clean.

Numbers

  • Binary size: 48 MB (post-PR-feat(trace): wire up --mode dispatch and ship tree-sitter by default #260) → ~70 MB. Each grammar is roughly 1–3 MB compiled.
  • Lines: +7,700 / -57 across 43 files. ~6,500 lines of that is the generated `elisp/parser.c` from Wilfred's grammar.
  • Hand-written code: ~600 lines (queries_*.go averaging 10–20 lines each + LangSpec entries + tests).

What's intentionally not in this PR

  • defalias (elisp) — its target is a quote node, not a bare symbol; needs query-time descent into quote nodes.
  • Elixir def / defp / defmodule / defmacro — these parse as (call ...) whose discriminator is the leading identifier; would need a query predicate of the form `(#eq? @kind "def")`-style on a deeply-nested structure. Doable but noisy enough to defer.
  • YAML, CSS/SCSS, HTML, Markdown, Svelte — selectors / headings / mapping keys are dubious as "symbols" in the navigation sense; the regex extractor doesn't cover them either, and the value is lower.
  • Erlang, R — no smacker grammar for either at this version. Could vendor in a follow-up.
  • Field-level extraction for Java / Kotlin — useful but two nodes deep; deferred to a follow-up to keep this PR focused on the main definition kinds.

Compatibility

  • Single-project mode (grepai watch without --workspace): behaviour for the original 9 languages unchanged. New languages get tree-sitter-quality symbols where regex was previously used.
  • Workspace mode (grepai watch --workspace …): same — the existing dispatch in feat(trace): wire up --mode dispatch and ship tree-sitter by default #260 picks up the new languages automatically via HasTreeSitterGrammar.
  • --mode fast still forces regex for everything, including the new languages, as an escape hatch.

kryptt added 4 commits May 29, 2026 09:33
Pre-existing bug: the `treesitter` build tag gated all tree-sitter code,
so stock `go build` produced a binary that could only do regex symbol
extraction. The `--mode fast`/`--mode precise` flag was a cosmetic
label on TraceResult.Mode that never actually selected an extractor —
both code paths called `trace.NewRegexExtractor()` regardless. F# (yoanbernabeu#152)
landed via tree-sitter but the dispatch to it was dead.

This change ships tree-sitter in the default build and makes the flag
mean what it says.

  * Drop `//go:build treesitter` from extractor_ts.go and its tests.
    `go build` now compiles grammars for Go, JS, JSX, TS, TSX, Python,
    PHP, C#, F# always. Binary grows from ~23 MB to ~48 MB.

  * Introduce `trace.CompoundExtractor`, which implements `SymbolExtractor`
    by routing per-file between the tree-sitter and regex extractors
    according to a Mode (auto / fast / precise). Tests in compound_test.go
    cover the dispatch matrix.

  * Add `trace.ParseMode` (normalizes user input; unknown values fall
    back to auto with a one-line warning to stderr) and `trace/languages.go`
    (`HasTreeSitterGrammar`, `TreeSitterExtensions` — single source of
    truth for which extensions have a compiled-in grammar).

  * `cli/watch.go` (both single-project and workspace runners) replaces
    `trace.NewRegexExtractor()` with `trace.NewCompoundExtractor(...)`.
    The relevant function signatures change `*trace.RegexExtractor` to
    `trace.SymbolExtractor` to widen to the interface.

  * `cli/trace.go` default for `--mode` flips from `fast` to `auto`; help
    text expanded to describe all three modes.

  * `config.TraceConfig.Mode` default flips from `fast` to `auto`.

No behavior change for the 9 already-supported languages at the symbol
level — they go from regex-quality to tree-sitter-quality, which is
strictly more accurate (e.g. Go method receivers, Python class
docstrings) without removing anything regex caught. Other extensions
keep their existing regex behavior.

Refs: extends yoanbernabeu#152 (F#) and yoanbernabeu#176 (Lua fast-mode) by making the dispatch
actually live.
Five small follow-ups from a self-review of the previous commit:

  * Consolidate the tree-sitter language registry. The extension list in
    languages.go and the extension→grammar map inside NewTreeSitterExtractor
    were two sources of truth that could drift. Merge into a single
    treeSitterLanguages map[string]func() *sitter.Language at the package
    level in languages.go. HasTreeSitterGrammar, TreeSitterExtensions, and
    NewTreeSitterExtractor all derive from it. Adding a new language is
    now one entry in that map plus the switch arm in walkNodeForSymbols.

  * Lazy tree-sitter init. NewCompoundExtractor no longer constructs the
    TreeSitterExtractor up front. ModeFast users (and ModeAuto users who
    only touch regex-only extensions) never pay the parser-allocation
    cost. Construction is deferred to the first file that needs it.

  * Drop the (always-nil) error from NewCompoundExtractor's return
    signature. The error claim was misleading; the only possible failure
    mode (lazy TS init) now surfaces from per-file route() calls.

  * Make ParseMode a pure normalizer. It now returns (Mode, ok bool)
    instead of writing the "unknown value" warning to os.Stderr itself.
    The CLI call sites in cli/watch.go log the warning where the I/O
    concern belongs and where the project name is in scope (helpful for
    workspace mode).

  * Library-friendly error message. The route() error no longer leaks
    the "--mode precise:" CLI flag name. Replaced with "trace: precise
    mode requires a tree-sitter grammar for extension %q (file %s); none
    compiled in" so downstream library consumers see a meaningful message.

Also: expanded compound_test.go with TestCompound_FastMode_NoTreeSitterAllocation,
TestCompound_AutoMode_LazyTreeSitter, TestHasTreeSitterGrammar_FromRegistry,
and TestTreeSitterExtensions_SortedAndDerived to lock in these invariants.
Test fixture switched from .pas to a more durable .lua/.rs pair.

No external behavior change. Same 48 MB binary, same dispatch behavior,
just less duplication and a smaller surface to maintain when PR 2 lands
the language additions.
A focused duplication-detection pass with `dupl -t 50` surfaced two
clone groups introduced by the previous commits. Both are now
collapsed:

  * cli/watch.go:1006-1010 vs cli/watch.go:2769-2773 — the two
    NewCompoundExtractor call sites were independently doing
    ParseMode → log warning on unknown value → NewCompoundExtractor.
    Factored into buildSymbolExtractor(rawMode, context), which keeps
    the warning helpful (mentions the project root or workspace
    project name + lists valid values) and reduces each call site to
    a one-liner. The fact that the workspace site now also lists the
    valid values in its warning is a small UX improvement that fell
    out of the unification.

  * trace/compound_test.go:170-175 vs 258-263 — two copies of the
    "is this slice strictly ascending?" check. Factored into a
    t.Helper-marked assertStrictlySorted(t, label, got). The strict
    inequality is the right contract for both call sites because
    both inputs come from set-then-sort and cannot contain duplicates.

Other dupl hits are either pre-existing in upstream code (the
RPG-init blocks in cli/watch.go, the per-language extractGoSymbol /
extractJSSymbol / etc. shapes in trace/extractor_ts.go) or
structurally forced by the SymbolExtractor interface (the route+
delegate body of ExtractSymbols / ExtractReferences / ExtractAll
on CompoundExtractor — different return types preclude a clean
helper without generics). Those stay as-is.

Verified: go vet ./... clean, go test ./... clean,
dupl -t 30 on the three new files reports only the structurally
forced interface-shaped clone in compound.go.
Builds on the dispatch wiring from yoanbernabeu#260. The new query-based extraction
path lets one LangSpec + one S-expression query file cover an entire
language; per-language Go code stays at ~10–30 lines.

Architecture changes:
  * `treeSitterLanguages` becomes a []LangSpec slice in languages.go,
    carrying Name, Extensions, GetLanguage, and optional Queries. The
    original 9 languages keep their hand-walked extractors (Queries nil);
    new languages declare Queries and route through extractViaQueries.
  * `extractViaQueries` runs each NamedQuery against the parsed tree
    using smacker's sitter.NewQuery / NewQueryCursor, applies #eq? /
    #match? predicates via cursor.FilterPredicates, and emits one Symbol
    per @name capture. The Kind string is taken verbatim from the
    NamedQuery — free-form taxonomy per the PR-1 design.

New languages (16):
  Top-8 rich queries:    Ruby, Rust, Java, Scala
  Medium coverage:       C, C++, Bash/Zsh, Lua, Kotlin, Swift
  Long-tail minimal:     SQL, Protobuf, HCL/Terraform, Elm, TOML
  Vendored grammar:      Emacs Lisp (Wilfred/tree-sitter-elisp, MIT)

Vendored elisp/ package mirrors fsharp/ — parser.c + parser.h + LICENSE
+ binding.go + README.md noting the provenance commit. Pinned at the
last ABI-14-compatible upstream commit (3a8f590); smacker's bindings
top out at ABI 14.

Tests:
  * New TestExtractor_LanguageFixtures table-driven test exercises every
    new language against a small fixture under trace/testdata/. Each
    fixture has ~5 known symbols asserted set-wise (extras allowed —
    grammars may extract more than the bare minimum we care about).
  * Updated the PR-1 compound_test.go to use .zig instead of .lua as
    the canonical regex-only extension — Lua moved to the tree-sitter
    path in this commit.

Binary size: 48 MB → 70 MB on x86_64-linux. The grammars themselves
account for roughly 1–3 MB each.

defalias and Elixir's def/defp/defmodule are intentionally not covered
yet — they need either grammar-aware predicate work (Elixir's call/def
shape) or a query that descends into quote/sharp-quote nodes
(defalias' target). Regex still picks these up.

Refs: yoanbernabeu#260 (PR 1 — wires up the --mode dispatch that this PR's languages
ride on).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant