Fix exponential complexity evaluation via NameState memoization - #270
Open
fym-rgb wants to merge 1 commit into
Open
Fix exponential complexity evaluation via NameState memoization#270fym-rgb wants to merge 1 commit into
fym-rgb wants to merge 1 commit into
Conversation
MachineComplexityEvaluator.evaluate recursed into next NameStates once per ByteMatch, i.e. once per exact-match list value. A rule whose sorted key chain carries value lists of sizes L1..Lk therefore re-walked the last key's ByteMachine L1*...*Lk times. With realistic CloudTrail-style lists (2 x 50 x 21) ahead of a key holding four leading-star anything-but-wildcard patterns, one evaluateComplexity call walked the wildcard NFA 2,100 times: ~25 seconds and ~26 GB of transient allocation for a machine whose complexity is 7 and whose construction costs 73 ms / 3.6 MB. Two changes, each sufficient for the list-chain case, together covering convergent NameState graphs as well: - Dedupe recursion targets by next NameState instead of by ByteMatch, so a value list collapses to its single downstream NameState. - Memoize per-NameState results for the duration of a single evaluation: each entry point creates a fresh identity-keyed cache and threads it through the recursion, so any NameState reachable via multiple paths is evaluated once per evaluation. The cache lives exactly as long as its validity window - one evaluation of a machine, which cannot change mid-evaluation - so evaluator instances remain stateless: safe to reuse across evaluations, including after rule changes, and safe to share between threads, exactly as before. Results are unchanged: complexity of a NameState is a pure function of the machine reachable from it, recursion results depend only on the target NameState (ByteMatch equality is (pattern, nextNameState), and max is insensitive to duplicates), and a result capped at maxComplexity caches as that same capped value. The synthetic reproduction above now evaluates in ~90 ms / ~15 MB with identical complexity values. New coverage: MachineComplexityEvaluatorMemoizationTest pins the per-ByteMachine walk counts (each distinct ByteMachine walked once per evaluation, including hand-built convergent graphs) with deterministic count assertions rather than wall-clock bounds; the production-shaped case walks 4 machines where the old evaluation walked 2,203. It also pins statelessness: repeat evaluation recomputes rather than serving evaluator state, one evaluator reflects rule changes across evaluations, and a shared evaluator returns correct results under concurrent use. MachineComplexityEvaluatorEquivalenceTest asserts, across every value pattern type, pairwise key combinations, chain/nested/$or/multi-rule shapes, both additionalNameStateReuse configurations, capped and uncapped maxComplexity values, and a 250-machine seeded random corpus, that memoized evaluation returns exactly the value unmemoized evaluation returns, and that repeat evaluation is idempotent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available
Fixes #269 (MachineComplexityEvaluator re-walks downstream machines once per exact-match
list value)
Description of changes
MachineComplexityEvaluator.evaluate(ByteState)recursed into next NameStates once perByteMatch— once per exact-match list value — with no memoization. All values of one key'slist lead to the same next NameState, so a rule with exact-match lists of sizes L1..Lk ahead
of a wildcard-bearing key re-walked the wildcard key's ByteMachine L1×...×Lk times. For a
CloudTrail-style rule with 2×50×21 lists ahead of four leading-
*anything-but-wildcardpatterns, one
evaluateComplexitycall walked the wildcard NFA 2,100 times: ~25 s and ~26 GBof transient allocation to report complexity 7 for a machine whose construction costs 73 ms /
3.6 MB.
Two changes, each sufficient for the list-chain case, together also covering convergent
NameState graphs:
list to its single downstream NameState before recursing.
(
NameState.evaluateComplexity,ByteMachine.evaluateComplexity) creates a freshidentity-keyed cache and threads it through the recursion as a parameter, so a NameState
reachable via multiple distinct parents is still evaluated once per evaluation.
The cache's lifetime equals its validity window — one evaluation of a machine, which cannot
change mid-evaluation — so evaluator instances remain stateless: no public signature
changes, and reuse across evaluations (including after rule changes) and sharing between
threads behave exactly as before. The cache-threaded forms are package-private overloads.
Why results are unchanged
its ByteMachines' walk sizes and its descendants' values), so caching a computed value and
serving it to later visitors within one evaluation cannot change any result.
NameState;
ByteMatchequality is(pattern, nextNameState), so the set of distinct targetsis the same either way, and
Math.maxover duplicates is insensitive to multiplicity.caches maxComplexity, exactly what re-computation would return.
graphs built by rule compilation are acyclic, so it is always overwritten before it can be
read.
Benchmarks (single shot, wall time + thread-allocated bytes, Corretto 17)
*wildcards)The "wildcard key only" row bounds the honest cost of walking that NFA once; the fix makes the
full rule cost that, instead of 2,100 times that. Complexity values are identical on every
variant measured.
Test coverage
MachineComplexityEvaluatorMemoizationTest(new, 13 tests): pins per-ByteMachine walkcounts with a counting evaluator subclass — deterministic count assertions, no wall-clock
bounds. Covers exact-match list chains (4 walks where pre-fix behavior does 33), list-size
independence, the production-shaped rule (4 walks vs 2,203), a 10-key deep chain (11 walks
vs 2,047), nested keys,
$orsub-rules, and hand-built convergent NameState graphs thatisolate the memoization from the dedupe (4 walks vs 5; wide convergence 7 vs 11). It also
pins statelessness: repeat evaluation recomputes rather than serving evaluator state, one
evaluator reflects rule changes across evaluations, evaluator reuse across distinct
machines, and a shared evaluator returns correct results under concurrent use (4 threads).
MachineComplexityEvaluatorEquivalenceTest(new, 8 tests): property-style equivalenceagainst a cache-disabled evaluator subclass that reproduces pre-fix values — every supported
value pattern type singly, ordered pairs of the complexity-relevant types, chain/nested/
$or/multi-rule shapes, bothadditionalNameStateReuseconfigurations, capped (1, 3) anduncapped maxComplexity, plus a 250-machine seeded random corpus (deterministic in CI;
failure messages carry the full generated rules). Also asserts idempotence of repeat
evaluation.
tests with exactly the predicted walk counts (33, 2,203, 2,047, 15, 7, 5, 11), and injecting
a corrupted cache value fails all 8 equivalence tests.
mvn verifygreen (800 tests, 0 failures; checkstyle + spotbugs clean) and the CIbenchmark step
mvn test -Dtest="Benchmarks#CL2*"green — complexity evaluation is not onthe event-matching path, so match throughput is structurally unaffected.
Non-goal noted for maintainer judgment (pre-existing, unchanged)
While validating equivalence we observed that an
anything-butwildcardset under-scoresrelative to the same patterns as plain wildcards (the four exclusions above score 7 as an ABW
set vs 12 as four plain
wildcardpatterns, measured on this branch), because a values-setshares one
Patternsobject and complexity counts distinct patterns. That behavior ispre-existing, is preserved exactly by this PR (the equivalence suite pins it), and is
deliberately not changed here — it deserves its own discussion if it matters.
Related issues
deterministic regression protection for this defect class; a time-based benchmark as Benchmark test for MachineComplexityEvaluator #26
proposes would complement them and would have surfaced this pathology directly.
evaluateComplexityat rule-creation time, so evaluator cost is part of effective creationlatency even though
addRuleitself is fast.By submitting this pull request, I confirm that my contribution is made under the terms of the
Apache-2.0 license.