Skip to content

exemptUserAgents should support substring matching, not just prefix #80

@pavelforever

Description

@pavelforever

Problem

The exemptUserAgents config option only matches user agent prefixes, but most real-world bots (including Yandex) use a user agent string that starts with Mozilla/5.0, not with the bot name itself.

For example, YandexBot user agent looks like:
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

Setting exemptUserAgents: YandexBot does nothing because the string starts with Mozilla/5.0, not YandexBot.

Expected behavior

exemptUserAgents should support substring matching (i.e. strings.Contains instead of strings.HasPrefix), so that setting exemptUserAgents: YandexBot would correctly exempt any user agent containing YandexBot.

Current workaround

The only workarounds are:

  1. Use the full prefix: exemptUserAgents: Mozilla/5.0 (compatible; YandexBot — fragile and ugly
  2. Add all Yandex IP ranges to exemptIps — requires maintaining a separate IP list

Suggested fix

Change the matching logic in exemptUserAgents from strings.HasPrefix to strings.Contains (case-insensitive).

This would be backward compatible — existing configs using short prefixes like edge would still work since Contains is a superset of HasPrefix behavior for non-prefix patterns.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions