feat(trace): tree-sitter symbol extraction for 16 new languages#261
Open
kryptt wants to merge 4 commits into
Open
feat(trace): tree-sitter symbol extraction for 16 new languages#261kryptt wants to merge 4 commits into
kryptt wants to merge 4 commits into
Conversation
Pre-existing bug: the `treesitter` build tag gated all tree-sitter code, so stock `go build` produced a binary that could only do regex symbol extraction. The `--mode fast`/`--mode precise` flag was a cosmetic label on TraceResult.Mode that never actually selected an extractor — both code paths called `trace.NewRegexExtractor()` regardless. F# (yoanbernabeu#152) landed via tree-sitter but the dispatch to it was dead. This change ships tree-sitter in the default build and makes the flag mean what it says. * Drop `//go:build treesitter` from extractor_ts.go and its tests. `go build` now compiles grammars for Go, JS, JSX, TS, TSX, Python, PHP, C#, F# always. Binary grows from ~23 MB to ~48 MB. * Introduce `trace.CompoundExtractor`, which implements `SymbolExtractor` by routing per-file between the tree-sitter and regex extractors according to a Mode (auto / fast / precise). Tests in compound_test.go cover the dispatch matrix. * Add `trace.ParseMode` (normalizes user input; unknown values fall back to auto with a one-line warning to stderr) and `trace/languages.go` (`HasTreeSitterGrammar`, `TreeSitterExtensions` — single source of truth for which extensions have a compiled-in grammar). * `cli/watch.go` (both single-project and workspace runners) replaces `trace.NewRegexExtractor()` with `trace.NewCompoundExtractor(...)`. The relevant function signatures change `*trace.RegexExtractor` to `trace.SymbolExtractor` to widen to the interface. * `cli/trace.go` default for `--mode` flips from `fast` to `auto`; help text expanded to describe all three modes. * `config.TraceConfig.Mode` default flips from `fast` to `auto`. No behavior change for the 9 already-supported languages at the symbol level — they go from regex-quality to tree-sitter-quality, which is strictly more accurate (e.g. Go method receivers, Python class docstrings) without removing anything regex caught. Other extensions keep their existing regex behavior. Refs: extends yoanbernabeu#152 (F#) and yoanbernabeu#176 (Lua fast-mode) by making the dispatch actually live.
Five small follow-ups from a self-review of the previous commit:
* Consolidate the tree-sitter language registry. The extension list in
languages.go and the extension→grammar map inside NewTreeSitterExtractor
were two sources of truth that could drift. Merge into a single
treeSitterLanguages map[string]func() *sitter.Language at the package
level in languages.go. HasTreeSitterGrammar, TreeSitterExtensions, and
NewTreeSitterExtractor all derive from it. Adding a new language is
now one entry in that map plus the switch arm in walkNodeForSymbols.
* Lazy tree-sitter init. NewCompoundExtractor no longer constructs the
TreeSitterExtractor up front. ModeFast users (and ModeAuto users who
only touch regex-only extensions) never pay the parser-allocation
cost. Construction is deferred to the first file that needs it.
* Drop the (always-nil) error from NewCompoundExtractor's return
signature. The error claim was misleading; the only possible failure
mode (lazy TS init) now surfaces from per-file route() calls.
* Make ParseMode a pure normalizer. It now returns (Mode, ok bool)
instead of writing the "unknown value" warning to os.Stderr itself.
The CLI call sites in cli/watch.go log the warning where the I/O
concern belongs and where the project name is in scope (helpful for
workspace mode).
* Library-friendly error message. The route() error no longer leaks
the "--mode precise:" CLI flag name. Replaced with "trace: precise
mode requires a tree-sitter grammar for extension %q (file %s); none
compiled in" so downstream library consumers see a meaningful message.
Also: expanded compound_test.go with TestCompound_FastMode_NoTreeSitterAllocation,
TestCompound_AutoMode_LazyTreeSitter, TestHasTreeSitterGrammar_FromRegistry,
and TestTreeSitterExtensions_SortedAndDerived to lock in these invariants.
Test fixture switched from .pas to a more durable .lua/.rs pair.
No external behavior change. Same 48 MB binary, same dispatch behavior,
just less duplication and a smaller surface to maintain when PR 2 lands
the language additions.
A focused duplication-detection pass with `dupl -t 50` surfaced two
clone groups introduced by the previous commits. Both are now
collapsed:
* cli/watch.go:1006-1010 vs cli/watch.go:2769-2773 — the two
NewCompoundExtractor call sites were independently doing
ParseMode → log warning on unknown value → NewCompoundExtractor.
Factored into buildSymbolExtractor(rawMode, context), which keeps
the warning helpful (mentions the project root or workspace
project name + lists valid values) and reduces each call site to
a one-liner. The fact that the workspace site now also lists the
valid values in its warning is a small UX improvement that fell
out of the unification.
* trace/compound_test.go:170-175 vs 258-263 — two copies of the
"is this slice strictly ascending?" check. Factored into a
t.Helper-marked assertStrictlySorted(t, label, got). The strict
inequality is the right contract for both call sites because
both inputs come from set-then-sort and cannot contain duplicates.
Other dupl hits are either pre-existing in upstream code (the
RPG-init blocks in cli/watch.go, the per-language extractGoSymbol /
extractJSSymbol / etc. shapes in trace/extractor_ts.go) or
structurally forced by the SymbolExtractor interface (the route+
delegate body of ExtractSymbols / ExtractReferences / ExtractAll
on CompoundExtractor — different return types preclude a clean
helper without generics). Those stay as-is.
Verified: go vet ./... clean, go test ./... clean,
dupl -t 30 on the three new files reports only the structurally
forced interface-shaped clone in compound.go.
Builds on the dispatch wiring from yoanbernabeu#260. The new query-based extraction path lets one LangSpec + one S-expression query file cover an entire language; per-language Go code stays at ~10–30 lines. Architecture changes: * `treeSitterLanguages` becomes a []LangSpec slice in languages.go, carrying Name, Extensions, GetLanguage, and optional Queries. The original 9 languages keep their hand-walked extractors (Queries nil); new languages declare Queries and route through extractViaQueries. * `extractViaQueries` runs each NamedQuery against the parsed tree using smacker's sitter.NewQuery / NewQueryCursor, applies #eq? / #match? predicates via cursor.FilterPredicates, and emits one Symbol per @name capture. The Kind string is taken verbatim from the NamedQuery — free-form taxonomy per the PR-1 design. New languages (16): Top-8 rich queries: Ruby, Rust, Java, Scala Medium coverage: C, C++, Bash/Zsh, Lua, Kotlin, Swift Long-tail minimal: SQL, Protobuf, HCL/Terraform, Elm, TOML Vendored grammar: Emacs Lisp (Wilfred/tree-sitter-elisp, MIT) Vendored elisp/ package mirrors fsharp/ — parser.c + parser.h + LICENSE + binding.go + README.md noting the provenance commit. Pinned at the last ABI-14-compatible upstream commit (3a8f590); smacker's bindings top out at ABI 14. Tests: * New TestExtractor_LanguageFixtures table-driven test exercises every new language against a small fixture under trace/testdata/. Each fixture has ~5 known symbols asserted set-wise (extras allowed — grammars may extract more than the bare minimum we care about). * Updated the PR-1 compound_test.go to use .zig instead of .lua as the canonical regex-only extension — Lua moved to the tree-sitter path in this commit. Binary size: 48 MB → 70 MB on x86_64-linux. The grammars themselves account for roughly 1–3 MB each. defalias and Elixir's def/defp/defmodule are intentionally not covered yet — they need either grammar-aware predicate work (Elixir's call/def shape) or a query that descends into quote/sharp-quote nodes (defalias' target). Regex still picks these up. Refs: yoanbernabeu#260 (PR 1 — wires up the --mode dispatch that this PR's languages ride on).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on / stacks on top of #260 (which wires up
--modedispatch). This PR adds the actual language coverage.Problem
Even after the dispatch fix in #260, tree-sitter symbol extraction only covers the original 9 grammars that landed via #152 (F#) and earlier (Go, JS/JSX, TS/TSX, Python, PHP, C#). For everything else — Ruby, Rust, Java, Scala, C, C++, Bash, Lua, Kotlin, Swift, SQL, Protobuf, HCL, Elm, TOML, Emacs Lisp — symbol extraction falls back to regex. Regex is usable but loses precision (Go method receivers, Python class docstrings, Rust trait impls, …) and silently misses syntax it doesn't know about.
We've already paid the CGo + grammar-linkage cost via #260. The marginal cost of linking more grammars and writing per-language queries is small in code and reviewable as one slice of
LangSpecentries.Solution
Single registry, one entry per language
trace/languages.goalready holds the registry from #260. This PR turns it into the single source of truth for both grammar registration and symbol extraction:Adding a language is:
LangSpectotreeSitterLanguages.trace/queries_<lang>.gowith the S-expression queries.trace/testdata/.No edits to
extractor_ts.go. No new method per language. The dispatch picks up the new entry automatically.New query-based extraction path
TreeSitterExtractor.extractViaQueriesruns eachNamedQueryagainst the parsed tree usingsitter.NewQuery+NewQueryCursor. It applies#eq?/#match?predicates viacursor.FilterPredicates. Emits oneSymbolper@namecapture with theKindtaken verbatim from the correspondingNamedQuery.The legacy nine languages keep their hand-walked
extractGoSymbol/extractPythonSymbol/ … —Queriesis empty on theirLangSpecand the extractor falls through to the existing walk. Zero behavior change for the original languages.Vendored Emacs Lisp grammar
smacker/go-tree-sitterdoesn't ship an elisp grammar, so we vendor Wilfred/tree-sitter-elisp (MIT) at repo root inelisp/, mirroring the pattern set byfsharp/. Pinned at the last ABI-14-compatible upstream commit (3a8f590) since smacker's bindings top out at ABI 14.elisp/README.mddocuments the provenance + re-vendoring procedure.Covered macros:
defun,defmacro,defvar,defconst,defcustom,defface,define-{minor,derived,generic,globalized-minor}-mode,cl-defun,cl-defmethod,cl-defgeneric.Languages added (16)
.rb.rs.java.scala.sc.c.h.cpp.cc.cxx.hpp.hh.hxx.sh.bash.zsh.lua.kt.kts.swift.sql.proto.hcl.tf.elm.toml.elTest plan
New
TestExtractor_LanguageFixturesis a table-driven test that exercises every language end-to-end:```go
{name: "ruby", fixturePath: "ruby.rb", want: []string{
"class:Hello", "class:Standalone", "method:say",
"module:Greeter", "singleton_method:banner",
}},
```
Each fixture has ~5 known symbols asserted set-wise — extras allowed because grammars sometimes surface more than the bare minimum. 16 sub-tests, all passing.
The PR-1
compound_test.gocases that used.luaas the canonical regex-only extension are updated to.zig(still regex-only after this PR — no smacker Zig grammar).`go test ./... -count=1` is clean. `go vet ./...` is clean.
Numbers
What's intentionally not in this PR
quotenode, not a bare symbol; needs query-time descent into quote nodes.def/defp/defmodule/defmacro— these parse as(call ...)whose discriminator is the leading identifier; would need a query predicate of the form `(#eq? @kind "def")`-style on a deeply-nested structure. Doable but noisy enough to defer.Compatibility
grepai watchwithout--workspace): behaviour for the original 9 languages unchanged. New languages get tree-sitter-quality symbols where regex was previously used.grepai watch --workspace …): same — the existing dispatch in feat(trace): wire up --mode dispatch and ship tree-sitter by default #260 picks up the new languages automatically viaHasTreeSitterGrammar.--mode faststill forces regex for everything, including the new languages, as an escape hatch.