Third-party proxy IP does not take effect

[zy783282949]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/cheerio (CheerioCrawler)

Issue description

I use proxyConfiguration to set the proxy IP and try to access the target website. I use the curl command to ensure that the proxy IP is valid. The error message is as follows. It seems that the IP authentication of username and password has not been passed.

WARN CheerioCrawler: Reclaiming failed request back to the list or queue. Detected a session error, rotating session...
Proxy responded with 407 Proxy Authentication Required: 29 bytes

Code sample

import {CheerioCrawler,ProxyConfiguration} from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: [
        'http://username:password@ip:port',
    ],
});

const crawler = new CheerioCrawler({
    proxyConfiguration,
    requestHandler: async ({request, body, response, proxyInfo}) => {
        console.log(`Response body: ${body}`);
    },
    additionalMimeTypes: ['text/plain'], // 允许 text/plain 类型的内容
});

await crawler.run(['https://cip.cc/']);

Package version

├── crawlee@3.10.0 ├── got@14.3.0 ├── http-proxy-agent@7.0.2 ├── https-proxy-agent@7.0.4 └── tunnel@0.0.6

Node.js version

v18.14.2

Operating system

macos 14.3.1 (23D60)

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[B4nan]

Are you sure you are not encoding the password wrongly? We cant do much about this without being able to actually reproduce it. Proxies are working fine on our end, including authentication.

[zy783282949]

My proxy format is as follows: 'http://username:password@ip:port'. What do you mean by 'encoding the password'? Does it need to be encoded separately?

[B4nan]

Imagine you have special characters in your password like @, such symbols need to be URL encoded, otherwise your URL wouldn't be valid - you couldn't tell what part is password and where the hostname starts.

[zy783282949]

I have tried various cases and seem to have found the cause of the problem. The proxy I use is http protocol and the target website is https protocol. If I use the http IP proxy to access the http target website, it is normal, but using the http IP There will be a problem when the proxy accesses the https target website (RequestError: Proxy responded with 407 Proxy Authentication Required: 29 bytes). How should I solve this problem?

[barjin]

We've added logging for the actual proxy error in got-scraping@4.0.6 - try updating your got-scraping version to 4.0.6 by running npm i got-scraping@latest and running your script again.

With the new package version, you should see the parsed response in the error, which should give you a better idea about what is happening.

[zy783282949]

I have tried various cases and seem to have found the cause of the problem. The proxy I use is http protocol and the target website is https protocol. If I use the http IP proxy to access the http target website, it is normal, but using the http IP There will be a problem when the proxy accesses the https target website (RequestError: Proxy responded with 407 Proxy Authentication Required: 29 bytes). How should I solve this problem?

[barjin]

As I mentioned above, try updating got-scraping to >4.0.6. Since that version, got-scraping is logging the actual proxy errors, you'll get your error message (and find out what's the actual problem).

[zy783282949]

/usr/local/bin/node /Users/zy/PycharmProjects/crawlee/got-scraping-4-0-6-demo.js
node:internal/process/promises:289
triggerUncaughtException(err, true /* fromPromise */);
^

RequestError: Proxy responded with 407 Proxy Authentication Required: 29 bytes.

Below is the first 100 bytes of the proxy response body:
Proxy Authentication Required
at Request._beforeError (file:///Users/zy/PycharmProjects/crawlee/node_modules/got/dist/source/core/index.js:184:21)
at Request.flush (file:///Users/zy/PycharmProjects/crawlee/node_modules/got/dist/source/core/index.js:173:18)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at ClientRequest. (file:///Users/zy/PycharmProjects/crawlee/node_modules/got-scraping/dist/index.js:449:18)
... 7 lines matching cause stack trace ...
at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
input: undefined,
code: 'ERR_GOT_REQUEST_ERROR',
timings: undefined,
options: Options {
_unixOptions: { insecureHTTPParser: true },
_internals: {
request: undefined,
agent: {
http: TransformHeadersAgent {
agent: HttpRegularProxyAgent {
_events: [Object: null prototype],
_eventsCount: 2,
_maxListeners: undefined,
defaultPort: 80,
protocol: 'http:',
options: [Object: null prototype],
requests: [Object: null prototype] {},
sockets: [Object: null prototype] {},
freeSockets: [Object: null prototype] {},
keepAliveMsecs: 1000,
keepAlive: false,
maxSockets: Infinity,
maxFreeSockets: 256,
scheduling: 'lifo',
maxTotalSockets: Infinity,
totalSocketCount: 0,
proxy: URL {},
[Symbol(shapeMode)]: false,
[Symbol(kCapture)]: false
}
},
https: TransformHeadersAgent {
agent: HttpsProxyAgent {
_events: [Object: null prototype],
_eventsCount: 2,
_maxListeners: undefined,
defaultPort: 443,
protocol: 'https:',
options: [Object: null prototype],
requests: [Object: null prototype] {},
sockets: [Object: null prototype] {},
freeSockets: [Object: null prototype] {},
keepAliveMsecs: 1000,
keepAlive: false,
maxSockets: Infinity,
maxFreeSockets: 256,
scheduling: 'lifo',
maxTotalSockets: Infinity,
totalSocketCount: 0,
maxCachedSessions: 100,
_sessionCache: [Object],
proxy: URL {},
[Symbol(shapeMode)]: false,
[Symbol(kCapture)]: false
}
},
http2: Http2OverHttp {
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
sessions: {},
queue: {},
timeout: 0,
maxSessions: Infinity,
maxEmptySessions: 0,
_emptySessionCount: 0,
_sessionCount: 0,
settings: { enablePush: false, initialWindowSize: 33554432 },
tlsSessionCache: QuickLRU {
maxSize: 100,
onEviction: undefined,
cache: Map(0) {},
oldCache: Map(0) {},
_size: 0
},
proxyOptions: {
raw: true,
url: URL {},
rejectUnauthorized: false,
headers: {}
},
[Symbol(shapeMode)]: false,
[Symbol(kCapture)]: false
}
},
h2session: undefined,
decompress: true,
timeout: {
connect: undefined,
lookup: undefined,
read: undefined,
request: 60000,
response: undefined,
secureConnect: undefined,
send: undefined,
socket: undefined
},
prefixUrl: '',
body: undefined,
form: undefined,
json: undefined,
cookieJar: undefined,
ignoreInvalidCookies: false,
searchParams: undefined,
dnsLookup: undefined,
dnsCache: undefined,
context: {
headerGenerator: HeaderGenerator {
globalOptions: {
browsers: [Array],
operatingSystems: [Array],
devices: [Array],
locales: [Array],
httpVersion: '2',
browserListQuery: '',
strict: false
},
browserListQuery: undefined,
inputGeneratorNetwork: BayesianNetwork {
nodesInSamplingOrder: [Array],
nodesByName: [Object]
},
headerGeneratorNetwork: BayesianNetwork {
nodesInSamplingOrder: [Array],
nodesByName: [Object]
},
uniqueBrowsers: [
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object], [Object], [Object],
[Object], [Object], [Object], [Object],
... 119 more items
],
headersOrder: {
safari: [Array],
chrome: [Array],
firefox: [Array],
edge: []
},
relaxationOrder: [
'locales',
'devices',
'operatingSystems',
'browsers',
'browserListQuery'
]
},
useHeaderGenerator: true,
insecureHTTPParser: true,
proxyUrl: 'http://AAAA:BBBB@CCCC:DDDD',
sessionData: undefined
},
hooks: {
init: [
[Function: optionsValidationHandler],
[Function: customOptionsHook]
],
beforeRequest: [
[Function: insecureParserHook],
[Function: sessionDataHook],
[Function: http2Hook],
[AsyncFunction: proxyHook],
[AsyncFunction: browserHeadersHook],
[Function: tlsHook]
],
beforeError: [],
beforeRedirect: [ [Function: refererHook] ],
beforeRetry: [],
afterResponse: []
},
followRedirect: true,
maxRedirects: 10,
cache: undefined,
throwHttpErrors: false,
username: '',
password: '',
http2: false,
allowGetBody: false,
headers: { 'accept-encoding': 'gzip, deflate, br' },
methodRewriting: false,
dnsLookupIpVersion: undefined,
parseJson: [Function: parse],
stringifyJson: [Function: stringify],
retry: {
limit: 0,
methods: [ 'GET', 'PUT', 'HEAD', 'DELETE', 'OPTIONS', 'TRACE' ],
statusCodes: [
408, 413, 429, 500,
502, 503, 504, 521,
522, 524
],
errorCodes: [
'ETIMEDOUT',
'ECONNRESET',
'EADDRINUSE',
'ECONNREFUSED',
'EPIPE',
'ENOTFOUND',
'ENETUNREACH',
'EAI_AGAIN'
],
maxRetryAfter: undefined,
calculateDelay: [Function: calculateDelay],
backoffLimit: Infinity,
noise: 100
},
localAddress: undefined,
method: 'GET',
createConnection: undefined,
cacheOptions: {
shared: undefined,
cacheHeuristic: undefined,
immutableMinTimeToLive: undefined,
ignoreCargoCult: undefined
},
https: {
alpnProtocols: undefined,
rejectUnauthorized: false,
checkServerIdentity: undefined,
certificateAuthority: undefined,
key: undefined,
certificate: undefined,
passphrase: undefined,
pfx: undefined,
ciphers: undefined,
honorCipherOrder: undefined,
minVersion: undefined,
maxVersion: undefined,
signatureAlgorithms: undefined,
tlsSessionLifetime: undefined,
dhparam: undefined,
ecdhCurve: undefined,
certificateRevocationLists: undefined
},
encoding: undefined,
resolveBodyOnly: false,
isStream: false,
responseType: 'text',
url: URL {},
pagination: {
transform: [Function: transform],
paginate: [Function: paginate],
filter: [Function: filter],
shouldContinue: [Function: shouldContinue],
countLimit: Infinity,
backoff: 0,
requestLimit: 10000,
stackAllItems: false
},
setHost: true,
maxHeaderSize: undefined,
signal: undefined,
enableUnixSockets: false
},
_merging: false,
_init: [
{
handlers: [ [Function: fixDecompress] ],
mutableDefaults: true,
http2: true,
https: { rejectUnauthorized: false },
throwHttpErrors: false,
timeout: { request: 60000 },
retry: { limit: 0 },
headers: { 'user-agent': undefined },
context: {
headerGenerator: HeaderGenerator {
globalOptions: [Object],
browserListQuery: undefined,
inputGeneratorNetwork: [BayesianNetwork],
headerGeneratorNetwork: [BayesianNetwork],
uniqueBrowsers: [Array],
headersOrder: [Object],
relaxationOrder: [Array]
},
useHeaderGenerator: true,
insecureHTTPParser: true
},
agent: {
http: TransformHeadersAgent { agent: [Agent] },
https: TransformHeadersAgent { agent: [Agent] }
},
hooks: {
init: [
[Function: optionsValidationHandler],
[Function: customOptionsHook]
],
beforeRequest: [
[Function: insecureParserHook],
[Function: sessionDataHook],
[Function: http2Hook],
[AsyncFunction: proxyHook],
[AsyncFunction: browserHeadersHook],
[Function: tlsHook]
],
beforeRedirect: [ [Function: refererHook] ]
}
},
{ url: 'https://cip.cc', http2: false },
{ method: 'get' }
]
},
[cause]: ProxyError: Proxy responded with 407 Proxy Authentication Required: 29 bytes.

Below is the first 100 bytes of the proxy response body:
Proxy Authentication Required
at ClientRequest. (file:///Users/zy/PycharmProjects/crawlee/node_modules/got-scraping/dist/index.js:449:18)
at Object.onceWrapper (node:events:634:26)
at ClientRequest.emit (node:events:519:28)
at Socket.socketOnData (node:_http_client:580:11)
at Socket.emit (node:events:519:28)
at addChunk (node:internal/streams/readable:559:12)
at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)
at Readable.push (node:internal/streams/readable:390:5)
at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
[cause]: 'Proxy Authentication Required'
}
}

Node.js v21.7.3

Process finished with exit code 1