Scraping Glassdoor in CDP or UC mode gives me the impression of being censored.

[Patrick2ooo]

Hello everyone,

I recently tried extracting data from various job search sites. I succeeded on LinkedIn and a Swiss site. Now, I'm trying to extract data from Glassdoor and I'm encountering a strange problem: the number of job postings displayed varies depending on the method used. I open the page with a specific job search and use different SeleniumBase methods. Below are screenshots illustrating the results obtained with each method where I have highlighted the number of job offers published on the website. The Cloudflare filter is almost always bypassed.

The code is a function called like this:

if __name__ == '__main__':
    glassdoor_scraper()

CDP_mode:

activate_cdp_mode:

def glassdoor_scraper():
    url = "https://fr.glassdoor.ch/Emploi/software-engineer-emplois-SRCH_KO0,17.htm"
    with SB(uc=True, headless=False, incognito=True) as sb:
        sb.activate_cdp_mode(url)
        sb.sleep(10)

This method is the only one that has allowed me to find more than 1000 jobs on several occasions. I have not obtained the same result with other methods.

sb_cdp.Chrome(url):

def glassdoor_scraper():
    url = "https://fr.glassdoor.ch/Emploi/software-engineer-emplois-SRCH_KO0,17.htm"
    sb = sb_cdp.Chrome(url)
    sb.sleep(10)

UC_mode:

The main difference between UC mode and CDP mode is that, sometimes, UC mode does not seem to be able to bypass Cloudflare everytime.

driver.uc_open:

def glassdoor_scraper():
    url = "https://fr.glassdoor.ch/Emploi/software-engineer-emplois-SRCH_KO0,17.htm"
    driver = Driver(uc=True, headless=False, incognito=True)
    driver.uc_open(url)
    driver.sleep(10)

driver.get:

def glassdoor_scraper():
    url = "https://fr.glassdoor.ch/Emploi/software-engineer-emplois-SRCH_KO0,17.htm"
    driver = Driver(uc=True, headless=False, incognito=True)
    driver.get(url)
    driver.sleep(10)

As you can see, I've tried different methods, but I don't understand why Glassdoor is reacting this way. Could someone help me?

[mdmintz]

This method is the only one that has allowed me to find more than 1000 jobs on several occasions. I have not obtained the same result with other methods.

That's because everyone should be using CDP Mode for stealth: Either with UC + CDP Mode, or Pure CDP Mode.

UC Mode alone for stealth has been deprecated for over 2 years, which includes uc_open(url) and driver.get(url).

For stealth today, either call activate_cdp_mode() from UC Mode, or use sb_cdp.Chrome(url) from Pure CDP Mode.

[Patrick2ooo]

Hello,
Sorry, I think I misspelled what I meant to say. "This method is the only one that has allowed me to find more than 1000 jobs on several occasions. I have not obtained the same result with other methods."

What I meant was that the number of job postings displayed changes each time the code is run, regardless of the method used. I don't know if this is due to Glassdoor or if the bot is detected, in which case I would be censored.

[mdmintz]

Yes, but you used non-stealthy methods in your mix, so if they're tracking IP addresses, then your IP could have already been marked down as bot-traffic. If you're going to scrape a site, use only CDP Mode. Otherwise, you may need to wait until they reset IP lists (up to 24 hours depending on the site), which is done because otherwise all users at a coffee shop, library, etc, would get blocked just because one person on that IP used a bot on a site that was detected.

Also, randomized search results can be the case, so it would then be OK to see different results on each search query made.

[Patrick2ooo]

Okay, thank you for your help and time. I'll try modifying my code so that it only uses CDP mode from now on.