Add DEW (Dual-Embedding Watermarking) algorithm - #71
Open
jonasschaefer wants to merge 1 commit into
Open
Conversation
DEW is a semantic watermarking method that biases token logits by the similarity between keyed random projections of the preceding context embedding and the candidate token embeddings, projected into a shared latent space. Detection runs a one-sided Gaussian test on the mean per-token latent score. * watermark/dew/: DEWConfig, DEWUtils, DEWLogitsProcessor, DEW * config/DEW.json: default configuration * Register DEW in the AutoWatermark / AutoConfig mappings * Token-side whitening (StandardScaler + PCA) is fit on demand from the generation model's embeddings and cached to disk (gitignored) * requirements: add scikit-learn and joblib (whitening pipeline) * README: add DEW to the supported-algorithms table * test_method: allow running the harness with 'DEW' Paper: https://arxiv.org/abs/2606.31602 We used Claude Opus 4.8 to polish the code, simplify the DEW implementation, and align it with the current MarkLLM interfaces before preparing this PR.
Collaborator
|
The watermarking algorithm DEW you proposed is very valuable. Thank you for your interest in the MarkLLM repository and for your pull request. However, there is one improvement that we would like you to consider, and we hope you can make the following modification: DEWLogitsProcessor.call crashes with RuntimeError: scatter(): Expected self.dtype to be equal to src.dtype when the base model runs in fp16/bf16 (reproduced on OPT-1.3B). The projection matrices R_T / R_C are fp32 so the bias tensor is fp32; the mixed-precision add is upcast to fp32 and then fails to scatter into the fp16/bf16 new_scores. Casting back with .to(new_scores.dtype) before scatter_ fixes it. Verified end-to-end in fp16 (p=1e-8 vs 0.065) and bf16 (p=3e-11 vs 0.9). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DEW (Dual-Embedding Watermarking) is a semantic watermarking method. At each
generation step it biases the token logits by the similarity between keyed
random projections of (a) the preceding context, embedded by a sentence
encoder, and (b) the candidate token embeddings. Both sides are projected into
a shared latent space with secret-key-seeded random matrices, and the per-token
score is standardized so that detection reduces to a one-sided Gaussian test on
the mean score.
Paper: https://arxiv.org/abs/2606.31602
What's included
watermark/dew/—DEWConfig,DEWUtils,DEWLogitsProcessor,DEW,following the existing MarkLLM algorithm interfaces.
config/DEW.json— default configuration.AutoWatermarkandAutoConfigmappings.generation model's input embeddings and cached to disk (the cache directory
is gitignored, so no large binaries are committed).
requirements.txt— addsscikit-learnandjoblibfor the whiteningpipeline.
README.md— DEW added to the "Currently Supported Algorithms" table.test/test_method.py—DEWaccepted by the test harness.Testing
Tested on a GPU (RTX A5000, CUDA 12.4) with
facebook/opt-1.3band thesentence-transformers/paraphrase-multilingual-mpnet-base-v2context encoder: