Skip to content

fix: quote OpenClaw FTS query terms#2043

Merged
CarltonXiang merged 17 commits into
MemTensor:dev-v2.0.23from
RerankerGuo:fix/quote-openclaw-fts-query-terms
Jul 8, 2026
Merged

fix: quote OpenClaw FTS query terms#2043
CarltonXiang merged 17 commits into
MemTensor:dev-v2.0.23from
RerankerGuo:fix/quote-openclaw-fts-query-terms

Conversation

@RerankerGuo

Copy link
Copy Markdown

Description

Related Issue (Required): Fixes #1247

Quotes each sanitized OpenClaw FTS token before sending it to SQLite FTS5 MATCH.

The reported query shape includes message-id tokens such as message_id and om_x... mixed with CJK text. Previously those terms were passed as bare MATCH tokens after punctuation stripping. This change keeps the existing sanitization and reserved-word filtering, then wraps every remaining token as an FTS5 phrase and escapes internal quotes. That makes the generated MATCH expression safer for ids, underscores, and mixed-language text while preserving the implicit AND behavior.

A regression case was added for the reported message_id om_x... 我要测试 query shape.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

  • Unit Test
  • Test Script Or Test Steps (please provide)
  • Pipeline Automated API Test (please provide)

Commands run:

node - <<'NODE'
const fs = require('fs');
const source = fs.readFileSync('apps/memos-local-openclaw/src/storage/sqlite.ts', 'utf8');
const test = fs.readFileSync('apps/memos-local-openclaw/tests/storage.test.ts', 'utf8');
if (!source.includes('return tokens.map((t) => `"${t.replace(/"/g, \'""\')}"`).join(" ");')) {
  throw new Error('quoted FTS sanitizer not found');
}
if (!test.includes('message_id om_x100b544a390604b8c3e1b7d8641f08e 我要测试')) {
  throw new Error('message_id CJK regression test fixture not found');
}
console.log('static FTS sanitizer regression passed');
NODE

Result: static FTS sanitizer regression passed.

Also attempted:

cd apps/memos-local-openclaw
npm test -- --run tests/storage.test.ts
npm run build

Both are blocked in this local environment because vitest and tsc are not installed.

Checklist

  • I have performed a self-review of my own code | 我已自行检查了自己的代码
  • I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释
  • I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常
  • I have created related documentation issue/PR in MemOS-Docs (if applicable) | 我已在 MemOS-Docs 中创建了相关的文档 issue/PR(如果适用)
  • I have linked the issue to this PR (if applicable) | 我已将 issue 链接到此 PR(如果适用)
  • I have mentioned the person who will review this PR | 我已提及将审查此 PR 的人

@Memtensor-AI Memtensor-AI changed the base branch from main to dev-v2.0.22 July 2, 2026 10:57
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

Automated Test Results: QUEUED

This PR was retargeted to dev-v2.0.22; cloud test-engine rerun has been queued.

  • Run: tr-9bbe4d82-a61 on cloud test-engine 10011
  • Branch tested: refs/pull/2043/head

I will post the final pass/fail result when the run completes.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

Automated Test Results: PASSED

Cloud test-engine rerun against dev-v2.0.22 completed successfully.

  • Run: tr-9bbe4d82-a61 on cloud test-engine 10011
  • memos_local_openclaw/unit: 26 passed, 0 failed, 0 skipped

Manual code review is still required before merge.

@CarltonXiang CarltonXiang deleted the branch MemTensor:dev-v2.0.23 July 3, 2026 07:25
@syzsunshine219 syzsunshine219 reopened this Jul 3, 2026
@syzsunshine219 syzsunshine219 added the needs-audit Requires manual audit before merge label Jul 3, 2026
@syzsunshine219 syzsunshine219 changed the base branch from dev-v2.0.22 to main July 3, 2026 08:20
harvey_xiang and others added 5 commits July 3, 2026 16:34
## Description

Please include a summary of the change, the problem it solves, the
implementation approach, and relevant context. List any dependencies
required for this change.

Related Issue (Required):  Fixes #issue_number

## Type of change

Please delete options that are not relevant.

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Refactor (does not change functionality, e.g. code style
improvements, linting)
- [ ] Documentation update

## How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. Please also list any relevant details
for your test configuration

- [ ] Unit Test
- [ ] Test Script Or Test Steps (please provide)
- [ ] Pipeline Automated API Test (please provide)

## Checklist

- [ ] I have performed a self-review of my own code | 我已自行检查了自己的代码
- [ ] I have commented my code in hard-to-understand areas |
我已在难以理解的地方对代码进行了注释
- [ ] I have added tests that prove my fix is effective or that my
feature works | 我已添加测试以证明我的修复有效或功能正常
- [ ] I have created related documentation issue/PR in
[MemOS-Docs](https://github.com/MemTensor/MemOS-Docs) (if applicable) |
我已在 [MemOS-Docs](https://github.com/MemTensor/MemOS-Docs) 中创建了相关的文档
issue/PR(如果适用)
- [ ] I have linked the issue to this PR (if applicable) | 我已将 issue
链接到此 PR(如果适用)
- [ ] I have mentioned the person who will review this PR | 我已提及将审查此 PR
的人

## Reviewer Checklist
- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] Made sure Checks passed
- [ ] Tests have been provided
Translated an untranslated JSON `description` field in
`docs/cn/open_source/modules/dream.md` from English to Chinese.
@RerankerGuo RerankerGuo marked this pull request as ready for review July 8, 2026 05:42
@RerankerGuo

Copy link
Copy Markdown
Author

This draft is now Ready for review. Cloud test-engine rerun on dev-v2.0.22 already returned PASSED (memos_local_openclaw/unit or memos_local_plugin/unit). Friendly ping @Memtensor-AI / @MemTensor/maintainers — could a maintainer take a look when convenient?

@RerankerGuo

Copy link
Copy Markdown
Author

Hi @Memtensor-AI / @CarltonXiang — friendly ping on this draft.

Quick observation: this PR still has base branch main, but MemTensor/MemOS's current merge flow is into dev-v2.0.22 (every recent PR #2065 / #2058 / #2047 / #2046 / #2042 / #2034 / #2033 / #2026 lands on dev-v2.0.22). The bot already retargeted the base SHA for testing and cloud test-engine returned PASSED against dev-v2.0.22 on every rerun.

Could a maintainer please change the base branch from maindev-v2.0.22 (Edit → base branch dropdown) so this can enter the normal merge queue? The head commit is a single focused change and does not depend on any main-only commits.

Thanks!

@RerankerGuo

Copy link
Copy Markdown
Author

Hi @syzsunshine219 — looks like you've merged 35 of the last 50 PRs on this repo. Friendly ping on this one.

This draft (and the other 21 OPEN PRs I have here) is small, single-commit, has cloud test-engine PASSED, and is fully mergeable. The only thing blocking it is that the base branch dropdown still says main — every recent merge (e.g. #2038, #2037, #2034, #2032) landed on dev-v2.0.22.

When you have a moment, would you mind one of:

  1. Flipping the base branch on this PR from maindev-v2.0.22 (Edit → base branch dropdown), or
  2. Telling me which base branch you'd prefer and I'll re-push accordingly.

Both fix and feature PRs in this batch are independent — each can be reviewed/merged on its own. Happy to rebase / split / drop any that don't fit the direction.

Thanks!

@RerankerGuo RerankerGuo force-pushed the fix/quote-openclaw-fts-query-terms branch from c064a14 to ffcf966 Compare July 8, 2026 07:45
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2043
Task: manual-ocr:MemTensor/MemOS#2043
Base: main
Head: fix/quote-openclaw-fts-query-terms

OpenCodeReview: No comments generated. Looks good to me.

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI Memtensor-AI changed the base branch from main to dev-v2.0.23 July 8, 2026 08:51
@Memtensor-AI Memtensor-AI self-requested a review July 8, 2026 11:07
@CarltonXiang CarltonXiang merged commit d9046d6 into MemTensor:dev-v2.0.23 Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-audit Requires manual audit before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: FTS query failed

5 participants