Crawler stops after www-version redirect to non-www version

[buismaarten]

While crawling a page and a link starting with 'www' is found. And the website redirects from 'www' to non-'www', the crawler simply stops. This is caused by the code below.

if (! $this->crawler->getCrawlProfile() instanceof CrawlSubdomains) {
    if ($crawlUrl->url->getHost() !== $this->crawler->getBaseUrl()->getHost()) {
        return;
    }
}

Note: the Guzzle option allow_redirects must be set to true to follow redirects.