feat(kg): add LanceDB unified storage backend#3380
Conversation
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 04546be925
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@danielaskdd Both issues have been addressed:
|
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 63c8361440
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Systematic self-review pass (1690c43)To converge faster than one bot finding per round, I ran a systematic multi-angle review of the whole branch (line-by-line scan, removed-behavior audit, cross-file contract tracing against Fixed in 1690c43Concurrency hardening (same family as the two earlier review comments):
Known limitation #2 from the PR description is now closed: the Correctness/UX: Efficiency on the query hot path: Maintainability: the gunicorn multi-worker guard is now driven by a Checked and deliberately unchanged (sibling parity)For reviewer reference, these were examined and confirmed to match the established behavior of existing backends, so they were left as-is: All 77 LanceDB tests + tools suite pass; ruff clean. |
70b094a to
2977709
Compare
Implement all four LightRAG storage types on embedded LanceDB in a single local directory with no external services: - LanceDBKVStorage / LanceDBVectorStorage / LanceDBGraphStorage / LanceDBDocStatusStorage in lightrag/kg/lancedb_impl.py, built on the native async API (connect_async) with a ref-counted shared connection, per-table write locks, and strong read consistency - Deferred-embedding vector upserts (batched at index_done_callback), cosine query with similarity threshold, embedding-model-suffixed table names for model isolation - CJK-friendly BM25 full-text search on content via ngram bigram tokenizer (LanceDBVectorStorage.full_text_search) - Canonical undirected edge storage, client-side BFS knowledge graph, label search; passes the cross-backend graph conformance suite - Guard lightrag-gunicorn against multi-worker use with LanceDB - Tests (tests/kg/lancedb_impl, 59 offline tests on real tmp-dir LanceDB), runnable demo, docs, env.example and packaging updates generated by Claude Code
Add LanceDB support to the three KV maintenance tools so the backend is no longer excluded from vector-db rebuild and LLM cache maintenance: - LanceDBKVStorage.get_all_keys() for id enumeration (the API the tools need but BaseKVStorage lacks) - rebuild_vdb / clean_llm_query_cache / migrate_llm_cache: add LanceDBKVStorage branches to every storage dispatch, mirroring the existing OpenSearch/PG implementations - tests on real tmp-dir LanceDB for the enumeration method and all three tools' LanceDB paths - docs/lancedb.md Limitations and tool READMEs updated generated by Claude Code
JSON-encode the sorted endpoints before hashing so pairs like
("A-B", "C") and ("A", "B-C") no longer map to the same edge ID.
generated by Claude Code
generated by Claude Code
delete_entity_relation deleted persisted rows before pruning the pending buffer, so an in-flight index_done_callback flush (which holds _buffer_lock throughout) could merge_insert a pending relation for the same entity after the table.delete ran and pop it before the prune saw it, leaving an orphaned relation vector. Prune the pending buffer first and nest table_lock inside _buffer_lock so the delete and flush are serialized. Apply the same nesting to delete() for consistency. Lock order stays buffer -> table, matching the flush path, so no deadlock. generated by Claude Code
…kend Concurrency/validation hardening: - Serialize drop_and_recreate_table with in-flight writers via table_lock (nested inside creation_lock ordering; no reverse acquisition exists) - Hold _buffer_lock across vector drop()'s clear+recreate so an in-flight flush cannot re-persist cleared pending docs - Validate LANCEDB_WORKSPACE env override with validate_workspace, same as the constructor path (closes known limitation HKUDS#2) - Guard maybe_optimize after flush when the client was finalized Correctness/UX: - Parse LANCEDB_ENABLE_FTS with the same truthy set as get_env_value ("1"/"yes"/"t"/"on" no longer silently disable FTS) - full_text_search now fails fast with a clear error when FTS is disabled Efficiency: - _edges_touching selects only id/src/tgt, keeping edge payload JSON out of degree counting and BFS (query hot path) - _fetch_rows_by_ids runs id-chunk scans concurrently Cleanup: - Replace the gunicorn LanceDB name-prefix check with a registry-driven MULTIPROCESS_UNSAFE_STORAGES set in lightrag/kg - Drop four hand-written dataclass __init__s; coerce workspace in __post_init__ (milvus/redis parity) - Simplify table_lock to a single setdefault Tests: regression tests for the delete-vs-flush race (verified to fail on the pre-fix ordering), drop-vs-flush serialization, and env workspace validation. 77 LanceDB tests + tools suite pass; ruff clean. generated by Claude Code
2977709 to
8745842
Compare
2 participants
Closes #3379
What
Implements all four LightRAG storage types on embedded LanceDB in a single local directory, with no external services:
LanceDBKVStorage/LanceDBVectorStorage/LanceDBGraphStorage/LanceDBDocStatusStorage, on the native async API (connect_async) with a ref-counted shared connection, per-table write locks, strong read consistencyrebuild_vdb/clean_llm_query_cache/migrate_llm_cache) extended to support LanceDBlightrag-gunicornmulti-worker guard; docs,env.example, packagingTesting
tests/kg/lancedb_impl) on real tmp-dir LanceDBruffcleanKnown limitations & follow-up enhancements
Deliberately deferred, tracked for future work:
get_docs_paginatedloads the full table then paginates in Python — fine for embedded-scaledoc_statustables (hundreds–thousands of rows), but could push pagination into the DB layer for very large sets.LANCEDB_WORKSPACEenv override bypassesvalidate_workspace— the value still passes through_sanitize_table_name(no safety risk), but validation should be consistent with the constructor path.asyncio.sleep(1.1)— could mock time or use a shorter sleep + tolerance to speed up the suite.Notes
lightrag-gunicornrefuses to start with multiple workers when a LanceDB backend is selected.