Check cross-submits for sitemaps

[sebastian-nagel]

Sitemaps are automatically detected in the robots.txt but not checked for cross-submits. From time to time this leads to spam-like injections of URLs not matching the news genre. Recently, via one of their periodicals a publishing company "injects" their entire publishing program including landing pages for books and other media. This also happened for real estate ads before.
Note that the sitemaps must follow the news sitemap format which is the barrier for most cross-submits but not always.

[sebastian-nagel]

Further scenario: a news site redirects on of their news articles to a page on another site as kind of an advertisement. We need to check the robots.txt of the target site, of course. But we should ignore the sitemap directives.

[silentninja]

@sebastian-nagel I made a draft PR #68 which contains fix for the issue but have a question.

Sitemap https://www.example.com/rss/sitemaps/2025/sitemap.xml detected from https://www.example.com/robots.txt contains https://video.example.com/politics/abc123 as one of the URL.
But https://video.example.com/robots.txt does not contain https://www.example.com/ as one of its sitemaps.

1️⃣ https://www.example.com/robots.txt (Links to sitemap)
2️⃣ https://www.example.com/rss/sitemaps/2025/sitemap_02_19_items.xml (Contains URL from video subdomain)
3️⃣ https://video.example.com/politics/government/president-statement/abc123 ❓ But...
4️⃣ https://video.example.com/robots.txt ❌ (Does NOT list https://www.example.com/ as its sitemap)

Should we consider this as a cross-submit?

[sebastian-nagel]

4️⃣ https://video.example.com/robots.txt ❌ (Does NOT list https://www.example.com/ as its sitemap)

Maybe we can make this configurable and optionally allow submits on the level of the registered (pay-level) domain or the private domain?

• the sitemaps protocol cross-submit section defines cross-submits on the level of hosts
• however, my experience is that quite a few sites do it wrong and a strict cross-submit check would reject all pages listed in a sitemap
• a safe compromise is to allow cross-submits within the private domain (on level below a suffix in the public suffix list, including both ICANN and "private" suffixes)
• cross-submits within the pay-level domain are definitely not safe for large hosting domains (blogspot.com, github.io, etc.) and would allow to inject spam links

[sebastian-nagel]

Maybe one more comment about cross-submits: there is a long-standing open issue crawler-commons/crawler-commons#85 for this. Could be worth to implement it there, too. This would make it possible to simplify the code in the future.