Skip to content

Third-party proxy IP does not take effect #2488

@zy783282949

Description

@zy783282949

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/cheerio (CheerioCrawler)

Issue description

I use proxyConfiguration to set the proxy IP and try to access the target website. I use the curl command to ensure that the proxy IP is valid. The error message is as follows. It seems that the IP authentication of username and password has not been passed.

WARN CheerioCrawler: Reclaiming failed request back to the list or queue. Detected a session error, rotating session...
Proxy responded with 407 Proxy Authentication Required: 29 bytes

Code sample

import {CheerioCrawler,ProxyConfiguration} from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: [
        'http://username:password@ip:port',
    ],
});

const crawler = new CheerioCrawler({
    proxyConfiguration,
    requestHandler: async ({request, body, response, proxyInfo}) => {
        console.log(`Response body: ${body}`);
    },
    additionalMimeTypes: ['text/plain'], // 允许 text/plain 类型的内容
});

await crawler.run(['https://cip.cc/']);

Package version

├── crawlee@3.10.0 ├── got@14.3.0 ├── http-proxy-agent@7.0.2 ├── https-proxy-agent@7.0.4 └── tunnel@0.0.6

Node.js version

v18.14.2

Operating system

macos 14.3.1 (23D60)

Apify platform

  • Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions