Skip to content

Releases: go-again/sqlite

v0.6.0

12 Jun 23:42

Choose a tag to compare

Highlights

  • SESSION extension typed API — record changes on a primary, serialize as a changeset blob, apply onto a replica with typed conflict handlers and table filters; invert and concat included.
  • WAL / snapshot / progress / db-config control — typed (*Conn).WALCheckpoint, RegisterWALHook, GetSnapshot / OpenSnapshot, SetProgressHandler, SetDBConfig / QueryDBConfig.
  • R-Tree custom-geometry / custom-query callbacks(*Conn).RegisterRTreeGeometry and RegisterRTreeQuery let Go code drive the R-Tree MATCH operator; ext/rtree.Table is the typed handle on top.
  • Six new typed vtab handlesspellfix1.Vocab, bloom.Filter, closure.Graph, csv.Table, lines.Table, rtree.Table. All follow the vec.Table / fts.Index shape: Create / Open / domain methods / Drop, WithIfNotExists + ErrAlreadyExists.
  • sqlitex/ utility packageMigrate(ctx, db, fsys) over an fs.FS of NNNN_name.sql files; Save (savepoint with deferred-error closure); Transaction / ImmediateTransaction; ExecScript; ResultInt / ResultText / ResultFloat / ResultBool single-value helpers.

New features

SESSION extension (session.go, snapshot.go)

  • (*Conn).CreateSession(schema)*Session with Attach (empty = all tables), Enable / IsEnabled / IsEmpty, Changeset / Patchset serializers, Diff(fromSchema, table) for two-DB diffing, Close finalizer.
  • (*Conn).InvertChangeset(cs), ConcatChangesets(a, b) — top-level transforms.
  • (*Conn).ApplyChangeset(cs, opts...) with WithConflictHandler(h) (typed ConflictType + ConflictAction enums) and WithTableFilter(f). Default (no handler) aborts and rolls back.
  • (*Conn).GetSnapshot(schema) / OpenSnapshot(schema, snap) / SnapshotRecover(schema) for WAL point-in-time reads; (*Snapshot).Cmp / Close.
  • Coverage matrix: docs/coverage-session.md.

WAL + progress + db-config (wal.go, control.go)

  • (*Conn).WALCheckpoint(schema, mode) returns (logFrames, checkpointed) with typed CheckpointMode (Passive / Full / Restart / Truncate).
  • (*Conn).WALAutoCheckpoint(frames), RegisterWALHook(hook).
  • (*Conn).SetProgressHandler(n, handler) for periodic-VM-instruction interrupt callbacks.
  • (*Conn).SetDBConfig(op, enable) / QueryDBConfig(op) over sqlite3_db_config with typed DBConfigOp enum — including the security-only DEFENSIVE / TRUSTED_SCHEMA / WRITABLE_SCHEMA flags that have no PRAGMA equivalent.

Introspection (introspect.go, stmt.go)

  • (*Conn).TableColumnMetadata(schema, table, column) returns ColumnMetadata{decltype, collation, notnull, pk, autoinc}.
  • (*Conn).Status(op DBStatus, reset) — per-conn cache / lookaside counters.
  • (*Conn).TxnState(schema) — typed TxnState (None / Read / Write).
  • (*Stmt).Readonly() — read/write routing via sqlite3_stmt_readonly.
  • (*Stmt).Status(op StmtStatus, reset) — per-statement VM-step / sort / fullscan counters.
  • Coverage matrix: docs/coverage-conn.md.

R-Tree custom callbacks (rtree.go)

  • RTreeGeometryFunc(coords, params) (overlap, err) — pruning callback.
  • RTreeQueryFunc(*RTreeQueryInfo) — richer per-node form with tree depth + parent classification.
  • Typed RTreeWithin enum (NotWithin / PartlyWithin / FullyWithin); RTreeQueryInfo carries Coords, Params, Level, MaxLevel, Rowid, ParentScore, ParentWithin.

Typed vtab handles

Six new entry points, each Create / Open / domain ops / Drop with WithIfNotExists + ErrAlreadyExists:

  • spellfix1.Vocab — Add / AddMany / Size / Correct / CorrectSQL with WithMaxDistance / WithLimit. UNIQUE index on word so re-adding dedupes via INSERT OR IGNORE.
  • bloom.Filter — Add / AddMany / Contains with WithSize / WithFalsePositiveRate / WithHashes.
  • closure.Graph — Descendants / DescendantsSQL with WithMaxDepth / WithExactDepth / OverGraph per-query retarget; Reversed(over) for ancestor walks.
  • csv.Table — Columns / Rows over the CSV vtab; WithFilename / WithData / WithHeader / WithComma / WithComment / WithColumns.
  • lines.Table — Columns / Rows (lineno + line) over the one-row-per-line vtab.
  • rtree.Table — Insert / InsertPoint / Delete / Search / SearchCircle with WithDimensions(n) / WithInt32Coords().

sqlitex/ utility helpers

  • LoadMigrations(fsys) — parse NNNN_name.sql files (strict numeric prefix; bounded to user_version's int32 range).
  • Migrate(ctx, db, fsys) — apply pending migrations idempotently against a schema_migrations tracking table.
  • Save(ctx, conn) — savepoint with a deferred-error release closure so defer release(&err) works.
  • Transaction(ctx, db, fn) and ImmediateTransaction(ctx, conn, fn) — auto commit/rollback wrappers (Immediate takes the write lock up front).
  • ExecScript(ctx, e, script) — multi-statement SQL.
  • ResultInt / ResultText / ResultFloat / ResultBool — collapse the QueryRow + Scan dance for single-value queries.

New ext sub-packages

  • ext/seriesgenerate_series-style eponymous vtab.
  • ext/texttext_reverse, text_repeat, text_lpad, text_rpad, text_split scalars absent from core.
  • ext/encodeencode(data, format) / decode(text, format) covering hex / base32 / base64 + URL-safe variants.

Improvements

  • (*conn).Close now drains the rtree-geom, rtree-query, progress-handler, and WAL-hook registries alongside the prior six (update / authorizer / trace / pre-update / commit / rollback). The expanded dropHookHandlers doc enumerates every per-conn registry and notes that future ones MUST be drained here.
  • The CREATE VIRTUAL TABLE skeleton in each typed vtab handle moves to internal/vtabx.Create / Drop — name validation, IF NOT EXISTS, parameter assembly, ErrAlreadyExists mapping in one place.
  • A generic callbackTable[T] lifts the {mu, m, ids} shape every per-conn C-callback registry rolled by hand; changeset apply, rtree geometry, and rtree query all use it.
  • internal/sqlid grows QuoteString (single-quoted SQL literal) consolidated from ext/csv and ext/lines; QuoteIdent / QuoteIdentBacktick / QuoteString now spell out their NUL-byte precondition in the docstring.

Bug fixes

  • NUL-byte safety on SQL emissionfts.New rejects NUL bytes in any tokenizer field (Tokenchars, Separators, Categories) before they truncate the generated SQL at the libc.CString boundary; wrapTokenize doubles embedded single quotes to close the outer tokenize='…' string-literal breakout. ext/csv falls back to the positional column name when a CSV header cell contains a NUL.
  • pre-update accessor honesty(*SQLitePreUpdateData).value zeros the output slot before sqlite3_preupdate_new / _old (Xmalloc does not zero), honors the returned rc instead of dereferencing whatever sat in memory, and surfaces an absent OLD/NEW column as (nil, nil).
  • vtab xCreate NOMEM cleanup — when sqlite3_malloc for the vtab struct fails, the Go-side Table's Destroy() runs explicitly so any shadow tables / handles allocated in Create are released. Previously SQLite never got a handle to drive xDestroy against on that branch.
  • (*Conn).OpenBlob and (*Blob).Reopen output slots live in C memory, not on the Go stack. A Go stack address passed as untracked uintptr could go stale when the runtime moves the stack (reentrant calls deep in the graph), leaving sqlite3_blob_open writing through a stale pointer. Same fix applied to (*conn).loadExtension.
  • UDF embedded-NUL TEXT pathfunctionArgs and the pre-update TEXT case drop libc.GoString for Xsqlite3_value_text + Xsqlite3_value_bytes + an explicit length-bounded copy, so "foo\x00bar" survives intact instead of truncating at the first NUL. Accessor order also swapped to the documented-safe form (text/blob content accessor before value_bytes).
  • UDF result > 2 GiB rejected with sqlite3_result_error_toobig instead of silently wrapping int32(len) to a negative size.
  • sqlitex.parseMigrationName rejects leading +/- in the version prefix and bounds the parsed version to math.MaxInt32 (the PRAGMA user_version range) so a truncated round-trip doesn't make Migrate re-apply the migration on every run.

Internal / dev

  • internal/vtabx consolidates the typed-vtab CREATE/DROP skeleton; callback_table.go consolidates the per-conn C-callback registry shape; internal/sqlid grows QuoteString.
  • docs/coverage-conn.md and docs/coverage-session.md are new coverage matrices covering the connection-level and SESSION surfaces.
  • The drift-discipline note in CLAUDE.md and vfs/crypto/doc.go is sharpened: named-field struct literals catch renames / removals at compile time, but not additions or reordering — safety against added fields comes from hard-coding FiVersion / iVersion below any unforwarded field.

Dependencies

  • modernc.org/sqlite bumped (carries the SQLite engine pin).
  • golang.org/x/crypto, golang.org/x/text, golang.org/x/sys carried by go mod tidy.
  • modernc.org/libc stays on the version the new sqlite release pins.

Full Changelog: v0.5.0...v0.6.0

v0.5.0

05 Jun 20:03

Choose a tag to compare

Highlights

  • Loadable Go extensions — 17 sub-packages under ext/, each independent, each with a Register(*Conn) error entry plus a blank-import <name>/auto variant.
  • Four new VFS sub-packagesvfs/crypto (encryption at rest), vfs/cksm (page checksum trailer), vfs/mvcc (in-memory snapshot isolation), vfs/memdb (plain in-memory). Plus vfs.NewReader for io.ReaderAt-backed databases.
  • Incremental BLOB API(*Conn).OpenBlob returns a *Blob that implements io.ReaderAt / io.WriterAt with Reopen for rebinding without realloc.
  • Modern Config short-handssqlite.OpenInMemory(), OpenWAL(path), OpenReadOnly(path), OpenShared(name) plus matching gorm-side helpers. Typed enums (JournalMode, Synchronous, TempStore, CacheMode) replace stringly-typed Pragma values.
  • VFS chainingvfs/crypto and vfs/cksm accept Options.WrapVFS, enabling the checksum-then-encrypt stack with one open database.

New features

ext/ catalog

Scalars / aggregates / collations: regexp, uuid, hash, ipaddr, zorder, stats, unicode. Virtual tables: array, csv, lines, statement, closure, pivot. Specialised stores: bloom (persistent Bloom filter), spellfix1 (fuzzy match). I/O: blobio, fileio. Plus ext/regexp/gorm exposing WhereRegex for GLOB-prefix + REGEXP LIKE-optimization on gorm queries.

Root-package additions

  • (*Conn).OpenBlob(schema, table, column, rowid, write)*Blob
  • (*Conn).EnableChecksums(schema) — one-call cksm activation (reserved_bytes=8 + VACUUM)
  • (*Conn).IsInterrupted() — poll the SQLite interrupt flag for vtab Go-side loops
  • (*Conn).StmtCacheStats(){Hits, Misses, Evictions} counters
  • (*Conn).FileControlReserveBytes(dbName, n)SQLITE_FCNTL_RESERVE_BYTES wrapper
  • (*Conn).CreateModule, CreateEponymousModule, DeclareVTab — Go-implemented vtab modules with xCreate / xConnect split for persistent shadow tables
  • sqlite.RegisterAutoHook(reg) — chains a Register hook onto the default driver's ConnectHook, preserving prior. Underpins the ext/<name>/auto pattern.
  • sqlite.InMemory constant + OpenInMemory / OpenWAL / OpenReadOnly / OpenShared
  • sqlite.Pointer now recognises time.Time as a primitive (no Pointer wrap)
  • stmt.ColumnCount / ColumnName / ColumnDeclType / BindCount / BindName

Hybrid search

  • fusion.RRF and RRF2 — Reciprocal Rank Fusion with WithK / WithWeights / WithLimit options.

VFS surface

  • vfs/crypto: Adiantum (32-byte key, default) or AES-XTS-256 (64-byte key) with Options.Cipher; per-IO Recorder observability; crypto.DeriveKey(passphrase, salt, cipher) for Argon2id derivation.
  • vfs/cksm: 8-byte Fletcher trailer per page; on-disk compatible with SQLite's cksumvfs.
  • vfs/mvcc: snapshot-isolated reads + atomic-publish writes; shared (file:/name) vs private (file:name) DBs.
  • vfs/memdb: smaller-surface in-memory VFS — direct per-page store under sync.RWMutex, no snapshot copy on commit.
  • vfs.NewReader(io.ReaderAt, size) — wraps a backing reader as a one-file read-only VFS.

Improvements

Performance

  • fts.Index pre-renders Insert/Delete SQL once at New/Open (skip per-row fmt.Sprintf + strings.Repeat).
  • fts.assignSQLType fast-paths skip []byte(v) / string(v) copies on matching T (hot Search path).
  • fts.SearchSlice and vec.KNNSlice pre-size from parsed WithLimit, clamped to [0, 1024].
  • pointerRegistry mutex → RWMutex — concurrent loadPointer no longer serialises on UDF hot paths.
  • vec.Table pre-renders insertSQL / updateSQL / deleteSQL at Create/Open.

Ergonomics

  • internal/testhelp.OpenPinned / RawConn / RegisterOn / WithConnectHook as the canonical pinned-conn fixture set.
  • sqlite.DriverName / DriverNameMattn constants.
  • convert.go error wrapping switched from %v to %w (errors.Is / errors.As friendly).
  • SQLiteContext type alias = FunctionContext so mattn UDF callback signatures compile after import-path swap.

Observability

  • (*Conn).StmtCacheStats() per-conn cache telemetry.
  • docs/perf.md baseline numbers for Pointer bind, stats aggregates, CSV scan, Bloom membership.

Bug fixes

  • Hook handler maps drained on (*conn).Close — previously leaked closures + *libc.TLS for the process lifetime; a stale callback could fire on a recycled uintptr handle.
  • UDF / collation / aggregate ids reclaimed on Close — same drain pattern, applied to xFuncs / xCollations / xAggregateFactories.
  • Deserialize no longer leaks pBuf on a non-OK Xsqlite3_deserialize error.
  • bindText / bindBlob reject payloads > math.MaxInt32 instead of truncating to a negative length (which sqlite3_bind_* interpreted as "use strlen" → silent corruption).
  • OpenBlob / Reopen refuse blobs with negative int32 size (a >2 GiB blob would sign-extend to negative int64 and break every bounds check).
  • (*Backup).Commit and Finish are idempotent across both orders — Commit zeros the handle so a follow-up Finish/Close is a no-op rather than a double sqlite3_backup_finish.
  • fts.Insert / fts.Delete / vec.BatchInsert use errors.Join(workErr, tx.Rollback()) so a rollback failure doesn't disappear behind the original error.
  • ext/pivot cursor's Close no longer leaks the scan stmt's psql CString per Filter call.
  • vfs/crypto and vfs/cksm New error paths now free cvfs + cname + tls when initIoMethods fails (previously leaked per retry under degraded memory).
  • vfs/mvcc xClose drops any held writeMu on abnormal close paths (context cancel mid-tx) to prevent next-writer deadlock on shared DBs.
  • ext/blobio openblob() restricts its write flag to {0, 1} (the previous v != 0 silently treated typo -1 as "open for write").
  • ext/csv uintArg widened from 15 bits (max 32767) to 31 (math.MaxInt32).
  • ext/spellfix1 createCtor rejects module arguments instead of silently dropping them.
  • ext/csv empty sourceheader=on errors clearly; header=off declares a single TEXT column and SELECT yields zero rows.
  • vec.Open uses full ValidIdent admission instead of empty-string check (closes string-builder injection).
  • vec.KNNSlice and Observable.KNNSlice clamp negative k so make([]T, 0, k) doesn't panic.
  • gorm/migrator.ColumnTypes deferred rows.Close no longer clobbers a real failure with a happy Close return.
  • In-memory connection survives sqlite3_interrupt — port of modernc.org/sqlite #196. A cancelled QueryContext no longer destroys the entire :memory: database.
  • vec/gorm and fts/gorm soft-delete discovered by type (reflect.TypeFor[gorm.DeletedAt]()) instead of field name, so RemovedAt gorm.DeletedAt / ArchivedAt gorm.DeletedAt participate in sidecar sync.

Breaking changes

  • crypto.DeriveKey now returns ([]byte, error) instead of []byte; a too-short salt becomes a returned error instead of a panic. Migration: add if err != nil handling at every call site (the three bundled examples show the shape).
  • fusion.RRF and fusion.RRF2 return ([]Result[K], error) instead of []Result[K]; mismatched WithWeights length returns an error instead of panicking. Migration: capture the error return.
  • unicode.RegisterLike package-level bool removed. Migration: pass unicode.WithLike() as an option to unicode.Register(c, unicode.WithLike()).
  • cksm.EnableChecksums(c, schema) free function removed. Migration: use (*sqlite.Conn).EnableChecksums(schema) on the connection directly.
  • fts.Column(name, q) no longer panics on a name that fails ValidIdent. FTS5 will reject the malformed query at exec time instead. Migration: validate column names with fts.ValidIdent at the API boundary if they come from untrusted input.

Internal / dev

  • internal/cabi consolidates the Go↔C function-pointer dance: FuncPointer, AsFunc, Registry[T], PtrMap[T], UniqueName, plus the CallX* family for Tsqlite3_io_methods slot dispatch.
  • internal/sqlid consolidates SQL-identifier toolkit: NamedArg, Unquote, QuoteIdent, QuoteIdentBacktick, ValidIdent, ToNamedValues, IsAlreadyExistsErr, AsString, AsInt64, AsFloat.
  • internal/gormbridge consolidates reflect/gorm plumbing shared by vec/gorm and fts/gorm (IndirectType, FindDeletedAtField, IterateRows, MaterializeByRowid[T], ActivePool, etc.).
  • internal/obs consolidates slog level-dispatch for the vec and fts Observable wrappers.
  • internal/raceskip and internal/testhelp lifted from per-package duplicates.
  • ext/internal/filevtab consolidates the file-backed-vtab scaffolding shared by ext/csv and ext/lines (UTF8BOM, OSFS, OpenSource, FullScanBestIndex).
  • vfs/internal/memio consolidates page-overlap copy between vfs/memdb and vfs/mvcc.
  • CI matrix tests both supported Go releases.
  • compat_mattn.go renamed to compat_sqlite3.go; aliases preserved.
  • Composite actions/setup for shared CI Go setup.

Dependencies

  • modernc.org/sqlite bumped (carries the SQLite engine pin); modernc.org/libc rides along.
  • golang.org/x/crypto, golang.org/x/text, golang.org/x/sys carried by go mod tidy.
  • github.com/google/uuid and golang.org/x/text moved from indirect to direct as ext/uuid and ext/unicode import them at the leaf.

Acknowledgements

ncruces/go-sqlite3 credit added to LICENSE.ncruces + README. Several ext/ sub-packages are Go-native ports of upstream loadable extensions — re-implemented against this driver's (*Conn).RegisterFunc / vtab helper APIs rather than copied verbatim, with a credit header in each ported package's doc.go.

**Full Chan...

Read more

v0.4.0

29 May 13:56

Choose a tag to compare

Highlights

  • vfs/crypto — pure-Go encryption at rest. Page-level encryption of main DB + journal + WAL + temp files via Adiantum or AES-XTS-256. No CGo, no plaintext fallthrough on disk. Domain-separated tweak per file kind blocks cross-file ciphertext substitution.
  • sqlite.Config — modern Go-typed open shared by database/sql and gorm. sqlite.Open(Config) returns *sqlite.DB; sqlitegorm.OpenConfig(Config) returns *sqlitegorm.DB. One struct, no DSN string assembly, encryption inline, single defer db.Close() for both pool and VFS lifecycle.
  • fusion — Reciprocal Rank Fusion (Cormack 2009) for combining vec.KNN and fts.Search rankings into a hybrid-search result list. Includes a RRF2 convenience for the two-slice case.
  • Custom window functions(*Conn).RegisterWindowFunction exposes Go-implemented window aggregates with optional WindowFinalizer. Mirrors RegisterAggregator shape.
  • FTS5 external-content sync triggersfts.SyncTriggers materializes the AFTER INSERT / UPDATE / DELETE triggers so external-content indexes stay in step with the source table without manual SQL.
  • Multi-field vec KNN + bridge SQL hatchesvecgorm.WithField selects a specific embedding column on a model with several; vec.KNNSQL / fts.SearchSQL (and gorm-bridge versions) return raw SQL + bindings for callers wiring the typed result into broader queries.

New packages

github.com/go-again/sqlite/vfs/crypto

Pure-Go encryption-at-rest VFS. Transparent page-level cipher of every SQLite file (main DB, journal, WAL, temp). Two cipher choices:

  • Adiantum (default) — 32-byte key, tweakable wide-block, fast on every platform.
  • AESXTS — 64-byte key, AES-XTS-256, the disk-encryption standard.

crypto.New(Options{Key, Cipher, PageSize, Recorder}) registers a per-instance VFS the driver routes through; crypto.DeriveKey(passphrase, salt, cipher) runs Argon2id with cipher-sized output. Recorder exposes per-IO observability (slog-shaped helper included).

github.com/go-again/sqlite/fusion

Rank fusion helpers for hybrid search. Pure Go, no SQLite dependency:

  • RRF[K](rankings ...[]K, opts ...Option) []Hit[K]
  • RRF2[K](a, b []K, opts ...Option) []Hit[K] — two-slice convenience.

Deterministic tiebreak; tunable k constant. Designed to compose vec.KNN + fts.Search keys into a single ordered hybrid result.

Added

Modern Config open

  • sqlite.Config struct: Path, Mode, Pragmas, Encryption, VFS, MaxOpenConns, MaxIdleConns, ConnMaxLifetime.
  • sqlite.Pragmas: JournalMode, BusyTimeout, Synchronous, ForeignKeys, CacheSize, TempStore, Extra.
  • sqlite.RecommendedPragmas() returns the production preset (WAL + busy_timeout=5s + foreign_keys).
  • sqlite.Open(Config) (*DB, error) — pool + VFS lifecycle in one wrapper.
  • (*DB).VFSName(), (*DB).Close() — idempotent close, bundles encryption VFS unregistration.
  • sqlite.BuildDSN(Config) string — render the full DSN (including PRAGMAs) for legacy-DSN integrations.
  • sqlite.ApplyPragmas(*sql.DB, Pragmas) error — apply typed PRAGMAs to a pool opened via bare sql.Open(dsn).
  • sqlite.Cipher / sqlite.Adiantum / sqlite.AESXTS re-exported so root consumers don't need a separate crypto import.
  • sqlitegorm.OpenConfig(sqlite.Config, ...*gorm.Config) (*DB, error) — same Config, gorm-flavored return.

PRAGMAs ride in as DSN _pragma= URL flags, so per-connection settings (busy_timeout, foreign_keys, synchronous, cache_size, temp_store) propagate to every conn in the pool, not just the one database/sql happens to dispatch the first Exec to.

Vector search

  • vec.Encode(embeddings) helper for the binary float32 format.
  • (*Table).KNN accepts WithFilter (renamed from WithWhere) and WithField for multi-field models.
  • vec.KNNSQL[T] returns raw SQL + bindings for callers stitching KNN into broader queries.
  • Idempotent (*Table).Create — re-running CREATE on an existing matching schema is a no-op.
  • vecgorm.KNNSQL[T] / WithField mirror the typed surface at the gorm bridge.
  • vecgorm.Embedding wrapper type so models can declare Embedding vecgorm.Embedding \vec:"dim=384;metric=cosine"`without a siblinggorm:"-"` tag.

Full-text search

  • fts.WithFilter query option for column-aware filtered search.
  • fts.SyncTriggers materializes external-content AFTER INSERT / UPDATE / DELETE triggers.
  • fts.Column with UNINDEXED modifier for content columns excluded from the FTS5 inverted index but kept for snippet / highlight retrieval.
  • fts.SearchSQL[K, V] raw-SQL escape hatch.
  • Idempotent (*Index).Create.
  • ftsgorm.SearchSQL[T] at the gorm bridge.
  • fts.Hit[K, V] is now the canonical result type (renamed from fts.Match; old name kept as a deprecated alias for one release).

gorm bridges

  • Migrator.DropTable cascades through any plugin implementing DropTableHook so vec/fts sidecars are torn down with their parent in one call.
  • vec/gorm and fts/gorm Plugin() registers the cascade hook automatically.
  • gorm.Transaction interaction with sidecar writes is now safe: sidecar inserts ride the outer transaction instead of opening a private one.

Custom window functions

  • (*Conn).RegisterWindowFunction(name string, fn func() WindowAccumulator, deterministic bool) error.
  • Optional WindowFinalizer interface for accumulators that need a finalize step distinct from Value.

Examples

  • examples/sqlite-configsqlite.Open(Config) plain + encrypted.
  • examples/gorm-configsqlitegorm.OpenConfig plain + encrypted with an FK relation.
  • examples/gorm-crypto — comprehensive stack: gorm + vec/gorm + fts/gorm + fusion + crypto + slog Recorder.
  • examples/vfs-crypto — bare encryption-at-rest demo.
  • examples/window-functionRegisterWindowFunction round-trip.
  • examples/fusion-hybrid-search — vec KNN + fts Search combined via fusion.RRF2.

Changed

  • vec.WithWhere renamed to vec.WithFilter; old name removed (no external consumers as of v0.3).
  • vec.Match / vec.Item consolidated to vec.Neighbor / vec.Row for naming consistency with the gorm bridge.
  • fts.Match renamed to fts.Hit; deprecated alias kept for one release.
  • vec/encoding.go reuses scan targets across rows in FTS5 Search for fewer allocations on large result sets.
  • vec KNN k pre-allocation now capped to avoid OOM on adversarial inputs.
  • vfs/crypto io-methods table bumped to iVersion=2 with xShm trampolines forwarding to the default unix VFS, so PRAGMA journal_mode = WAL now succeeds through the encryption path (previously fell back to DELETE silently).

Fixed

  • vfs/crypto: cross-file ciphertext substitution blocked via 1-byte file-kind tag in the tweak (Adiantum) / high 8 bits of the XTS sector index.
  • vfs/crypto: Adiantum concurrency race on the shared hashBuf — every cipher op now holds a mutex matching the upstream library's documented single-threaded contract.
  • vfs/crypto: defensive key copy in newCipher so callers can zero or mutate Options.Key after New returns.
  • gorm: DATETIME columns scanned into any / map now return time.Time (matches mattn behavior; previously returned RFC3339 strings).
  • vec/gorm: nested gorm.Transaction rolls back sidecar writes correctly.
  • vec/gorm: Migrator.DropTable cascades sidecar cleanup so orphans no longer accumulate across AutoMigrate cycles.
  • fts/gorm: external-content triggers stay aligned after AlterColumn / recreateTable.

Compatibility

  • Backward-compatible with every v0.3 entry: sql.Open("sqlite", dsn), sql.Open("sqlite3", dsn), sqlitegorm.Open(dsn), sqlitegorm.New(Config{DSN}), and crypto.New(Options) all keep working unchanged.
  • Supported Go versions: the two most recent releases (1.26 + 1.25).
  • Supported platforms: same matrix as modernc.org/sqlite. All 15 darwin / freebsd / linux / windows targets pass cross-build.
  • Bundled SQLite: 3.53.1 (from modernc.org/sqlite). libc pin carried automatically via go mod tidy.

Acknowledgements

vfs/crypto: Adiantum from lukechampine.com/adiantum; AES-XTS from golang.org/x/crypto/xts; Argon2id from golang.org/x/crypto/argon2. Threat model and io-methods drift discipline live in vfs/crypto/doc.go.

Full Changelog: v0.3.0...v0.4.0

v0.3.0

27 May 17:36

Choose a tag to compare

Transactions

vec/gorm and fts/gorm callbacks route writes through the active gorm.ConnPool exposed on db.Statement — the live *sql.Tx when inside gorm.Transaction, the *sql.DB otherwise. Sidecar writes commit and roll back with the parent gorm transaction. ftsgorm.Search sees read-your-own-writes inside a tx.

vec/gorm.KNN[T] reads through *sql.DB and is therefore not supported inside a gorm.Transaction (it would deadlock under the typical pinned-conn SQLite fixture). Use tx.Raw against the <table>_vec sidecar when you need to query uncommitted state.

Regression tests cover Create / batch-Create / Update / Delete / soft-delete rollback in vec/gorm, plus in-table / contentless / external-mode rollback and search-sees-uncommitted-writes in fts/gorm.

API

  • vec.Neighbor — KNN result. vec.Row — input to BatchInsert.
  • fts.Hit[K, V] — search hit returned by Index.Search / Index.SearchSlice.
  • vecgorm.Hit[T] / ftsgorm.Hit[T] — typed gorm-bridge hit pairing the model with the distance / rank.
  • vec.WithFilter(sql, args...) / vecgorm.WithFilter(sqlFragment, args...) / ftsgorm.WithFilter(sqlFragment, args...) — uniform filter option across all three layers. The fragment is caller-trusted raw SQL (parameters bind via args, fragment interpolates as-is — same contract as gorm.Where). Validate identifiers via vec.ValidIdent / fts.ValidIdent before interpolating. Each variant carries an explicit # Trust model docstring section.
  • vecgorm.ErrNotInstalled / ftsgorm.ErrNotInstalled — sentinel errors returned by helper APIs when the plugin hasn't been installed via db.Use(...). errors.Is against either works.

Performance

  • fts.Index.Search: scanTargets / holders slices hoisted above the per-row loop. Drops per-row allocations from two to zero.
  • vec.Table.KNNSlice: caps result-slice pre-allocation at min(k, 1024). A buggy or malicious caller passing huge k no longer pre-allocates gigabytes before the query runs.
  • registerSchema in both gorm bridges: tightened double-checked locking so concurrent first-access on the same model type only pays the schema.Parse cost once.
  • Per-row reflection in the bridge callbacks uses a cached field-index chain (parsed once per model type).

Benchmarks

New benchmarks (deterministic PCG-seeded corpora, dim=384, 10k docs):

  • BenchmarkKNN, BenchmarkKNN_WithFilter, BenchmarkBatchInsert_JSON, BenchmarkBatchInsert_Binary in vec/.
  • BenchmarkSearch, BenchmarkSearch_WithRanking, BenchmarkInsert in fts/.
  • Wired into the justfile as bench-vec / bench-fts.

Concurrency

  • vec/gorm/concurrent_test.go: 16 goroutines × 8 calls hammering the schema cache under go test -race. Pins the double-checked locking pattern.

Tooling

  • just lint chain expanded to fmt-check + vet + staticcheck + golangci-lint + gopls modernize. Local just lint mirrors CI byte-for-byte.
  • gopls modernize added to CI as well. Forked modernc/glebarez files are filtered out; our code is held to the current-Go-syntax bar.
  • Codebase swept for the modernizations gopls catches: range-over-int, reflect.TypeFor, strings.SplitSeq, sync.WaitGroup.Go, atomic.Int32, errors.Join.

Docs

  • New llms.txt (llmstxt.org-format index) and llms-full.txt (consumer guide: driver-name picker, DSN flag whitelist, sub-package map, pitfalls, migration recipes, what-NOT-to-do). Aimed at LLM agents in downstream projects.
  • README: new "Supported Go versions" section codifying the rolling-window policy.
  • CLAUDE.md "Always" PR checklist enumerates every doc that must move together (doc.go, README, llms.txt, llms-full.txt, docs/coverage-*.md) to prevent doc drift.
  • plan-doc-sweep.md inventory lists the new llms files and reminds the sweeper to mirror DSN-flag / migration-recipe / pitfall changes into llms-full.txt.

Supported Go versions

Two most recent Go releases only: today Go 1.26 (current) and Go 1.25 (previous). Older Go drops out within one release cycle of each new minor. Modern syntax is treated as a feature — generics, iter.Seq2, log/slog, generic type aliases, range over int, sync.WaitGroup.Go, strings.SplitSeq, reflect.TypeFor are all in use.

Stability

Still pre-v1.0. Public API is settling but not frozen. Pin the version you use in go.mod.


Full Changelog: v0.2.0...v0.3.0

v0.2.0

26 May 20:34

Choose a tag to compare

Highlights

  • Tag-driven gorm integration for vec and FTS5. Annotate a field on your gorm model, register the plugin, and the sidecar table or external-content FTS5 index maintains itself — including CreateInBatches in a single transaction, soft-delete filtering, and DropTable cascade.
  • License is now Apache 2.0, up from BSD 3-Clause. Upstream attributions preserved verbatim in LICENSE.modernc / LICENSE.mattn / LICENSE.glebarez plus a NOTICE file. Existing usage is unaffected; redistributors should refresh their bundled license files.
  • Three new CI-enforced upstream-suite lanes. The full gorm.io/gorm/tests integration suite, plus vendored subsets of modernc.org/sqlite's and mattn/go-sqlite3's own test suites, run against this driver on every push.
  • SQL conformance suite under tests/sql/ — a methodical, feature-by-feature matrix of the raw SQL surface (SELECT clauses, joins, CTEs, window functions, JSON1, datetime modifiers, every constraint type, triggers, UPSERT, RETURNING, PRAGMAs, ATTACH, VACUUM, generated columns, WITHOUT ROWID, STRICT).

New packages

github.com/go-again/sqlite/vec/gorm

Tag-driven bridge between gorm models and vec.Table sidecars.

type Document struct {
    ID        uint              `gorm:"primaryKey"`
    Embedding vecgorm.Embedding `vec:"dim=384;metric=cosine"`
}

db.Use(vecgorm.Plugin())
vecgorm.Migrate(db, &Document{})
db.Create(&doc)                                              // sidecar populated
results, _ := vecgorm.KNN[Document](ctx, db, queryVec, 5)    // returns []Result[Document]
db.Migrator().DropTable(&Document{})                         // cascades to sidecar

Supported tag keys: dim, metric, encoding, table, column. Field type can be vecgorm.Embedding (recommended) or []float32 with gorm:"-". Options: WithFilter(sql, args...), IncludeDeleted().

github.com/go-again/sqlite/fts/gorm

Tag-driven bridge for FTS5 search.

type Article struct {
    ID    uint   `gorm:"primaryKey"`
    Title string `fts5:"tokenize=porter+unicode61"`
    Body  string `fts5:"tokenize=porter+unicode61"`
}

db.Use(ftsgorm.Plugin())
ftsgorm.Migrate(db, &Article{})                              // table + triggers
db.Create(&a)                                                // trigger updates FTS5
results, _ := ftsgorm.Search[Article](ctx, db, fts.Term("fox"))

Three FTS5 modes selectable via tag: external (default; triggers-driven), external=false (in-table), contentless=true (index only — Snippet/Highlight rejected at search time). Tag keys: tokenize, prefix, column, table, detail, external, contentless. Options: WithLimit, WithOffset, WithRanking, WithSnippet, WithHighlight, IncludeDeleted.

tests/sql/

Standalone SQL conformance test suite organized by SQLite Language Reference category. Runs on linux / macos / windows in CI. See docs/coverage-sql.md.

API additions

  • vec.Metric.Keyword() string — returns the vec0() constructor keyword (l2 / cosine / l1).
  • vec.Encoding.Encode([]float32) any and vec.Encoding.Placeholder() string — replace the previously-private encodeValue / matchPlaceholder helpers; the old names remain as internal aliases for backwards compat.
  • vec.QuoteIdent / vec.ValidIdent / fts.QuoteIdent / fts.ValidIdent — previously private; exported so sub-packages and advanced users can reuse them.
  • fts.Query.Build() string — renamed from the previously-unexported build() so the gorm bridge (and anyone else) can render MATCH expressions.
  • sqlite.gorm.DropTableHook interface — gorm plugins can implement this to cascade sidecar cleanup when db.Migrator().DropTable(&Model{}) runs.

Bug fixes

  • gorm Table(...).Find(&map) returns time.Time for DATETIME columns instead of an RFC3339 string. Fixed by auto-injecting _texttotime=1 on DSNs opened through the gorm dialector. Aligns map-mode reads with mattn behavior and fixes the upstream TestFind/FirstMapWithTable/Birthday assertion.
  • gorm Migrator.DropTable now cascades. Calling db.Migrator().DropTable(&Model{}) cleans up the vec0 sidecar / FTS5 table + triggers without an explicit second call. Implemented via the new DropTableHook interface.
  • CI gorm-upstream lane now reports correct PASS/FAIL counts. GitHub Actions writes per-line timestamps into the tee'd log, which broke our anchored grep -cE '^--- PASS:' and aborted the script with pipefail+errexit. Drop the ^ anchor, capture go test's real exit via ${PIPESTATUS[0]}, survive zero-match grep with || true.

Compatibility notes

  • License: BSD-style → Apache 2.0. NOTICE attributions added for modernc, mattn, glebarez, sqlite-vec, and SQLite. Adopters redistributing this package should refresh their bundled license disclosures.
  • vec-tagged gorm fields must use either vecgorm.Embedding OR []float32 with gorm:"-". The plugin emits a clear error if both escape hatches are missing — gorm's schema.Parse rejects bare []float32 before any plugin hook can intervene.
  • Existing direct users of vec.Table / fts.Index / the gorm dialector are unaffected by the bridge packages — they're additive.

Internals

  • Removed metricKeyword / encodeValue / matchPlaceholder / quote / validIdent duplication across vec, fts, vec/gorm, fts/gorm. Single source of truth on the vec.Metric / vec.Encoding methods and the exported QuoteIdent / ValidIdent helpers.
  • Per-file copyright headers stripped from authored files; modernc-derived files keep their upstream copyright lines as required by BSD 3-Clause preservation.

Documentation

  • docs/coverage-sql.md — methodical SQL surface matrix.
  • docs/coverage-gorm.md — "Deep integration" section covering both bridges' tag syntax, lifecycle, and supported modes.
  • docs/gorm-upstream.md / docs/modernc-upstream.md / docs/mattn-upstream.md — reproduction recipes for the three CI-enforced upstream suites.
  • examples/gorm-vec-tagged/ and examples/gorm-fts-tagged/ — runnable end-to-end demos of the tag-driven flow.
  • README gained "Deep gorm integration" + "Sponsors" sections.

Full Changelog: v0.1.1...v0.2.0

v0.1.0

26 May 05:55

Choose a tag to compare