match index: include_original is silently accepted and ignored — honour it or reject it

[coderdan]

Summary

The match index option include_original is accepted by the encrypt config and has no effect on the emitted bloom filter. Encrypting the same plaintext with include_original: true and include_original: false yields the same bloom bits — measured on 0.24 and 0.29, under both EQL v2 and v3. The bloom is trigram-only either way, and a value shorter than the tokenizer's token_length blooms to nothing under both settings.

Silently accepting dead config is what made a downstream bug plausible: a claim that include_original: true appends a whole-value token to the needle's bloom (so a substring contains can never match) propagated into a comment, docs, a changeset, and a "confirming" test whose fake built the extra token itself. It cost a bug report (cipherstash/stack#610) and a limitation callout in published docs. Nothing tied the fiction to ffi because ffi neither honours nor rejects the flag — it just ignores it.

Expected

One of:

• Honour it — include_original: true actually adds the whole untokenized value as a term, so whole-value match works through the match index; or
• Reject it — error at newClient/config-normalization time if include_original is set, so callers cannot depend on a no-op.

Either removes the ambiguity. Today the flag reads as meaningful and is not.

Reproduce

import { newClient, encrypt } from '@cipherstash/protect-ffi'

const cfgWith = (includeOriginal) => ({
  v: 1,
  tables: { t: { col: {
    cast_as: 'text',
    indexes: { match: {
      tokenizer: { kind: 'ngram', token_length: 3 },
      token_filters: [{ kind: 'downcase' }],
      k: 6, m: 2048,
      include_original: includeOriginal,
    } },
  } } },
})

const on  = await encrypt(await newClient({ encryptConfig: cfgWith(true),  eqlVersion: 3 }), { plaintext: 'ada@example.com', table: 't', column: 'col' })
const off = await encrypt(await newClient({ encryptConfig: cfgWith(false), eqlVersion: 3 }), { plaintext: 'ada@example.com', table: 't', column: 'col' })

// on.bf and off.bf carry the same bit SET (order is nondeterministic; sort before comparing).

Context

Debunked and worked around in cipherstash/stack (PR that retracts the substring fiction): the stack's v3 match domains now emit include_original: false — the value a substring-search domain wants if a future release ever honours the flag — and a live suite (match-bloom-live.test.ts) pins the current inert behaviour against real ffi as a change-detector, so if ffi starts honouring the flag, that test fails and flags the divergence.

[freshtonic]

Resolution: reject. PR #135 implements the second option — newClient (native and wasm) now rejects a match index with include_original: true, failing fast before any network I/O:

ProtectError UNSUPPORTED_CONFIG
Unsupported config for column 't.col': the match index option include_original: true is not supported — the emitted bloom filter never includes the whole value. Remove include_original (or set it to false). For whole-value equality, add a unique index to the column.

• include_original: false remains accepted, since it describes the actual behaviour (and is what cipherstash/stack's v3 match domains emit).
• The TypeScript MatchIndexOpts type now admits only false, so TS callers fail at compile time; plain-JS callers hit the runtime rejection above.
• If a future release implements the flag, lifting the rejection is a non-breaking change — callers can no longer have come to depend on the no-op.

Heads-up for cipherstash/stack: match-bloom-live.test.ts pins the inert behaviour by encrypting with include_original: true; once stack picks up a release with this change, that construction will error at newClient instead — the divergence signal that test exists for.

The repro from this issue now throws at the first newClient call rather than yielding identical blooms.