feat(evaluators): add FuzzyEquals deterministic evaluator

[lifelmy]

Description

Adds FuzzyEquals, a configurable deterministic evaluator for comparing structured outputs where exact matching is too brittle. Addresses strands-agents/harness-sdk#3727.

The existing deterministic evaluators (Equals, Contains, StartsWith) only do exact matching, which fails for SQL result sets, JSON objects, tabular data, and numeric computations where row/key order, numeric precision, or whitespace shouldn't decide a pass. FuzzyEquals compares actual_output against an expected value recursively across lists and dicts, applying configurable knobs at every level.

Knobs (matching the issue proposal)

• ignore_order — treat lists as multisets so element order does not matter
• ignore_key_order — dict key/value pairs must match regardless of insertion order (default True)
• numeric_tolerance — relative tolerance for numeric values (0.01 accepts +/-1%)
• case_sensitive — case-insensitive string comparison when False
• ignore_whitespace — strip and collapse internal whitespace before comparing
• subset_mode — pass if the expected value is contained in actual (actual ⊇ expected)
• type_coercion — numeric/boolean strings compare equal to numbers/booleans ("42" == 42)

from strands_evals.evaluators.deterministic import FuzzyEquals

evaluator = FuzzyEquals(
    ignore_order=True,
    numeric_tolerance=0.01,
    case_sensitive=False,
    ignore_whitespace=True,
    type_coercion=True,
)

Follows the existing Evaluator[InputT, OutputT] pattern and returns an EvaluationOutput with a detailed diff path in the reason field (e.g. at $.rows[0].total: numbers differ ...). A module sentinel distinguishes an explicit value=None from the "compare against expected_output" default — something Equals cannot currently do.

Testing

• hatch fmt --formatter --check — 362 files already formatted
• hatch fmt --linter — all checks passed
• mypy -p src (hatch-static-analysis env) — Success: no issues found in 236 source files
• pytest tests/strands_evals/evaluators/deterministic/ — 114 passed (44 new FuzzyEquals tests, no regressions)

Notes

I use a coding assistant to help implement, and I review and take responsibility for the final change.

Marking as Draft for maintainer feedback on the API surface and whether the recursive implementation (vs. the deepdiff alternative mentioned in the issue) is preferred before finalizing.

Related Issue

strands-agents/harness-sdk#3727

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.