Skip to content

Add DEW (Dual-Embedding Watermarking) algorithm - #71

Open
jonasschaefer wants to merge 1 commit into
THU-BPM:mainfrom
jonasschaefer:add-dew-watermark
Open

Add DEW (Dual-Embedding Watermarking) algorithm#71
jonasschaefer wants to merge 1 commit into
THU-BPM:mainfrom
jonasschaefer:add-dew-watermark

Conversation

@jonasschaefer

Copy link
Copy Markdown
Contributor

Summary

DEW (Dual-Embedding Watermarking) is a semantic watermarking method. At each
generation step it biases the token logits by the similarity between keyed
random projections of (a) the preceding context, embedded by a sentence
encoder, and (b) the candidate token embeddings. Both sides are projected into
a shared latent space with secret-key-seeded random matrices, and the per-token
score is standardized so that detection reduces to a one-sided Gaussian test on
the mean score.

Paper: https://arxiv.org/abs/2606.31602

What's included

  • watermark/dew/DEWConfig, DEWUtils, DEWLogitsProcessor, DEW,
    following the existing MarkLLM algorithm interfaces.
  • config/DEW.json — default configuration.
  • Registered in the AutoWatermark and AutoConfig mappings.
  • Token-side whitening (StandardScaler + PCA) is fit on demand from the
    generation model's input embeddings and cached to disk (the cache directory
    is gitignored, so no large binaries are committed).
  • requirements.txt — adds scikit-learn and joblib for the whitening
    pipeline.
  • README.md — DEW added to the "Currently Supported Algorithms" table.
  • test/test_method.pyDEW accepted by the test harness.

Testing

Tested on a GPU (RTX A5000, CUDA 12.4) with facebook/opt-1.3b and the
sentence-transformers/paraphrase-multilingual-mpnet-base-v2 context encoder:

  • watermarked text detected at p ≈ 9e-17 over 228 tokens;
  • unwatermarked and human/natural text not flagged (p ≈ 0.04 and 0.84);
  • the whitening pipeline auto-fits, caches to disk, and reloads on the next run;
  • existing algorithms are unaffected (verified end-to-end with KGW).

DEW is a semantic watermarking method that biases token logits by the
similarity between keyed random projections of the preceding context
embedding and the candidate token embeddings, projected into a shared
latent space. Detection runs a one-sided Gaussian test on the mean
per-token latent score.

* watermark/dew/: DEWConfig, DEWUtils, DEWLogitsProcessor, DEW
* config/DEW.json: default configuration
* Register DEW in the AutoWatermark / AutoConfig mappings
* Token-side whitening (StandardScaler + PCA) is fit on demand from the
  generation model's embeddings and cached to disk (gitignored)
* requirements: add scikit-learn and joblib (whitening pipeline)
* README: add DEW to the supported-algorithms table
* test_method: allow running the harness with 'DEW'

Paper: https://arxiv.org/abs/2606.31602

We used Claude Opus 4.8 to polish the code, simplify the DEW
implementation, and align it with the current MarkLLM interfaces before
preparing this PR.
@GaryFuzheyu

Copy link
Copy Markdown
Collaborator

The watermarking algorithm DEW you proposed is very valuable. Thank you for your interest in the MarkLLM repository and for your pull request. However, there is one improvement that we would like you to consider, and we hope you can make the following modification: DEWLogitsProcessor.call crashes with RuntimeError: scatter(): Expected self.dtype to be equal to src.dtype when the base model runs in fp16/bf16 (reproduced on OPT-1.3B). The projection matrices R_T / R_C are fp32 so the bias tensor is fp32; the mixed-precision add is upcast to fp32 and then fails to scatter into the fp16/bf16 new_scores. Casting back with .to(new_scores.dtype) before scatter_ fixes it. Verified end-to-end in fp16 (p=1e-8 vs 0.065) and bf16 (p=3e-11 vs 0.9).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants