Skip to content

Guard wildcard similarity for large bodies - #1572

Merged
maurosoria merged 1 commit into
masterfrom
scanner-wildcard-similarity-guard
Jun 8, 2026
Merged

Guard wildcard similarity for large bodies#1572
maurosoria merged 1 commit into
masterfrom
scanner-wildcard-similarity-guard

Conversation

@maurosoria

@maurosoria maurosoria commented Jun 8, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add a conservative content-size guard before the ambiguous wildcard similarity fallback.
  • Skip the expensive body similarity calculation when either response body is larger than 256 KiB.
  • Add regression tests proving very large bodies return before content_similarity is called while medium-size bodies still use the fallback.

Why

The ambiguous wildcard fallback is useful for dynamic soft-404 pages, but full body similarity can become unnecessarily expensive on very large responses. For large bodies, this PR keeps the heuristic conservative and avoids the costly comparison without disabling the fallback for ordinary and moderately large HTML error pages.

Validation

  • python -m unittest tests.core.test_scanner
  • python -m unittest tests.core.test_scanner tests.core.test_wordlist_backend tests.core.test_dictionary_templates
  • python dirsearch.py -u https://example.com -w tests/static/wordlist.txt -q
  • git diff --check

@maurosoria
maurosoria force-pushed the scanner-wildcard-similarity-guard branch from 59b5f76 to 054134a Compare June 8, 2026 05:26
@maurosoria
maurosoria force-pushed the scanner-wildcard-similarity-guard branch from 054134a to c739138 Compare June 8, 2026 05:31
@maurosoria
maurosoria marked this pull request as ready for review June 8, 2026 05:39
@maurosoria
maurosoria merged commit 06c1be6 into master Jun 8, 2026
7 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant