-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Closed
Labels
bugSomething isn't working.Something isn't working.
Description
Which package is this bug report for? If unsure which one to select, leave blank
@crawlee/cheerio (CheerioCrawler)
Issue description
I use proxyConfiguration to set the proxy IP and try to access the target website. I use the curl command to ensure that the proxy IP is valid. The error message is as follows. It seems that the IP authentication of username and password has not been passed.
WARN CheerioCrawler: Reclaiming failed request back to the list or queue. Detected a session error, rotating session...
Proxy responded with 407 Proxy Authentication Required: 29 bytes
Code sample
import {CheerioCrawler,ProxyConfiguration} from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
proxyUrls: [
'http://username:password@ip:port',
],
});
const crawler = new CheerioCrawler({
proxyConfiguration,
requestHandler: async ({request, body, response, proxyInfo}) => {
console.log(`Response body: ${body}`);
},
additionalMimeTypes: ['text/plain'], // 允许 text/plain 类型的内容
});
await crawler.run(['https://cip.cc/']);Package version
├── crawlee@3.10.0 ├── got@14.3.0 ├── http-proxy-agent@7.0.2 ├── https-proxy-agent@7.0.4 └── tunnel@0.0.6
Node.js version
v18.14.2
Operating system
macos 14.3.1 (23D60)
Apify platform
- Tick me if you encountered this issue on the Apify platform
I have tested this on the next release
No response
Other context
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't working.Something isn't working.