Problem
The exemptUserAgents config option only matches user agent prefixes, but most real-world bots (including Yandex) use a user agent string that starts with Mozilla/5.0, not with the bot name itself.
For example, YandexBot user agent looks like:
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Setting exemptUserAgents: YandexBot does nothing because the string starts with Mozilla/5.0, not YandexBot.
Expected behavior
exemptUserAgents should support substring matching (i.e. strings.Contains instead of strings.HasPrefix), so that setting exemptUserAgents: YandexBot would correctly exempt any user agent containing YandexBot.
Current workaround
The only workarounds are:
- Use the full prefix:
exemptUserAgents: Mozilla/5.0 (compatible; YandexBot — fragile and ugly
- Add all Yandex IP ranges to
exemptIps — requires maintaining a separate IP list
Suggested fix
Change the matching logic in exemptUserAgents from strings.HasPrefix to strings.Contains (case-insensitive).
This would be backward compatible — existing configs using short prefixes like edge would still work since Contains is a superset of HasPrefix behavior for non-prefix patterns.
Problem
The
exemptUserAgentsconfig option only matches user agent prefixes, but most real-world bots (including Yandex) use a user agent string that starts withMozilla/5.0, not with the bot name itself.For example, YandexBot user agent looks like:
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Setting
exemptUserAgents: YandexBotdoes nothing because the string starts withMozilla/5.0, notYandexBot.Expected behavior
exemptUserAgentsshould support substring matching (i.e.strings.Containsinstead ofstrings.HasPrefix), so that settingexemptUserAgents: YandexBotwould correctly exempt any user agent containingYandexBot.Current workaround
The only workarounds are:
exemptUserAgents: Mozilla/5.0 (compatible; YandexBot— fragile and uglyexemptIps— requires maintaining a separate IP listSuggested fix
Change the matching logic in
exemptUserAgentsfromstrings.HasPrefixtostrings.Contains(case-insensitive).This would be backward compatible — existing configs using short prefixes like
edgewould still work sinceContainsis a superset ofHasPrefixbehavior for non-prefix patterns.