Describe the bug
When a dataclip is wiped (per-run wipe with save_dataclips: false, or the project data-retention job), its body/request are cleared to NULL and wiped_at is stamped, but its full-text search_vector is left intact. The work-order body search matches that stale vector with no wiped_at filter, so a wiped dataclip stays searchable by the exact content that was meant to be erased — a data-retention hole.
Version number
Reproduced on main (commit 6331486e8d) and on the defer-dataclip-search-vector branch (8fc24bab42).
I have reproduced this locally on main:
On main the AFTER INSERT trigger builds the vector synchronously; the wipe leaves it stale. On the deferred-indexing branch the worker builds it instead, but the wipe is the same — and the worker only re-sweeps WHERE search_vector IS NULL, so it never re-indexes a wiped row.
To Reproduce
- Create a work order whose input dataclip body contains a distinctive token, e.g.
{"secret": "zzzuniquetoken …"}.
- In the work-order list, search that project for
zzzuniquetoken with the body filter → the work order is found (expected).
- Wipe the dataclip — either let the project's data-retention period elapse, or trigger a run with
save_dataclips: false. The dataclip's body becomes NULL and wiped_at is set.
- Search again for
zzzuniquetoken (body filter).
- The wiped work order is still returned, even though its content is gone.
Expected behavior
Once a dataclip is wiped, searching for terms that were in its (now-erased) body should return nothing — the indexed content should be erased alongside the body.
Screenshots
N/A (confirmed via automated test rather than UI).
Additional context
- Code paths:
- Wipe sets
body/request/wiped_at only — Invocation.Query.wipe_dataclips/1 (lib/lightning/invocation/query.ex), used by Runs.wipe_dataclips/1 (lib/lightning/runs.ex) and Projects.wipe_dataclips_for/1 (lib/lightning/projects.ex).
- Read side has no
wiped_at filter — Invocation.build_search_fields_where/2, :body branch (lib/lightning/invocation.ex), reached from the main UI search Invocation.search_workorders/3.
- Pre-existing, not a regression: the original DB trigger was
AFTER INSERT only (priv/repo/migrations/20240329123804_…), so it never re-ran on the wipe UPDATE.
- A guard already exists, just not wired in:
Invocation.exclude_wiped_dataclips/1 filters is_nil(d.wiped_at), but is only applied to search_workorders_for_retry/2 and cancel — not the main search_workorders/3.
- Fix options: (1) add
search_vector: nil to the wipe update_all (on the deferred branch the worker then re-indexes a NULL body to an empty vector → matches nothing) — fixes it at source for every reader; or (2) apply the wiped_at filter to search_workorders/3. Recommend (1), optionally + (2) for defence in depth. Low risk either way.
Describe the bug
When a dataclip is wiped (per-run wipe with
save_dataclips: false, or the project data-retention job), itsbody/requestare cleared toNULLandwiped_atis stamped, but its full-textsearch_vectoris left intact. The work-order body search matches that stale vector with nowiped_atfilter, so a wiped dataclip stays searchable by the exact content that was meant to be erased — a data-retention hole.Version number
Reproduced on
main(commit6331486e8d) and on thedefer-dataclip-search-vectorbranch (8fc24bab42).I have reproduced this locally on main:
On
maintheAFTER INSERTtrigger builds the vector synchronously; the wipe leaves it stale. On the deferred-indexing branch the worker builds it instead, but the wipe is the same — and the worker only re-sweepsWHERE search_vector IS NULL, so it never re-indexes a wiped row.To Reproduce
{"secret": "zzzuniquetoken …"}.zzzuniquetokenwith the body filter → the work order is found (expected).save_dataclips: false. The dataclip'sbodybecomesNULLandwiped_atis set.zzzuniquetoken(body filter).Expected behavior
Once a dataclip is wiped, searching for terms that were in its (now-erased) body should return nothing — the indexed content should be erased alongside the body.
Screenshots
N/A (confirmed via automated test rather than UI).
Additional context
body/request/wiped_atonly —Invocation.Query.wipe_dataclips/1(lib/lightning/invocation/query.ex), used byRuns.wipe_dataclips/1(lib/lightning/runs.ex) andProjects.wipe_dataclips_for/1(lib/lightning/projects.ex).wiped_atfilter —Invocation.build_search_fields_where/2,:bodybranch (lib/lightning/invocation.ex), reached from the main UI searchInvocation.search_workorders/3.AFTER INSERTonly (priv/repo/migrations/20240329123804_…), so it never re-ran on the wipeUPDATE.Invocation.exclude_wiped_dataclips/1filtersis_nil(d.wiped_at), but is only applied tosearch_workorders_for_retry/2and cancel — not the mainsearch_workorders/3.search_vector: nilto the wipeupdate_all(on the deferred branch the worker then re-indexes a NULL body to an empty vector → matches nothing) — fixes it at source for every reader; or (2) apply thewiped_atfilter tosearch_workorders/3. Recommend (1), optionally + (2) for defence in depth. Low risk either way.