OF: No longer able to scrape usernames?

[lordoffools]

Seems to be timing out. Flipping over to web shows that the account is temporarily blocked/throttled.

Started today-ish. Anyone else seeing this?

Trying to tweak to see if I can diagnose, but I'm coming up empty.

[lordoffools]

Ping

Anyone else experiencing this? I have it set to 1 thread, and have ~700 subscriptions.

[jp1151419]

Same. It seems like they implemented another rate limiting "feature"

[lordoffools]

They're killing me with these annoying rate limits.

I just suspended all my subscriptions as a result. Spend too much money with them to not be able to have content for offline viewing.

[lordoffools]

Yeah... everything triggers the rate limiting lockout now. I'm running just a single thread.

Scraping profiles triggers it (though sometimes it passes). Once you pick a creator profile to scrape, it hits the limit and locks out just during the initial post scrape.

:(

[lordoffools]

Anyone have a workaround to slow down the scraping to not trigger this?

Occasionally, it does scrape the usernames. Pick one, and it stalls when scraping Posts. It never goes past that (and thus creator doesn't have a lot of posts to begin with).

[jp1151419]

It seems to run ok if given only a single thread and only parsing a single user at a time. I enter a couple (3-4) at a time spaced by line breaks and run short batches like that with longer pauses between them. Definitely not feasible at scale, just enough to run out a couple subscriptions.

ex:

4
5
6

[lordoffools]

Single thread seems to bring up a list of users half the time for me, half the time it triggers the lockout just while scraping the list of users (~600 total).

I can't get past that, however. It locks out while scraping main posts of a single user after that, even users with ~200 posts or so.

[naodrej]

The only way I've gotten it to work so far is wait 2-3 minutes between each action and doing individual models one at a time. So I run it, it shows the list of models (idk about 600, i follow about 200), I wait 2-3 minutes, select the model, scrape/download. When that finishes, wait 2-3 minutes, and repeat.

[lordoffools]

The only way I've gotten it to work so far is wait 2-3 minutes between each action and doing individual models one at a time. So I run it, it shows the list of models (idk about 600, i follow about 200), I wait 2-3 minutes, select the model, scrape/download. When that finishes, wait 2-3 minutes, and repeat.

Good advice. Will give that a shot and report back. Thank you!

[naodrej]

The only way I've gotten it to work so far is wait 2-3 minutes between each action and doing individual models one at a time. So I run it, it shows the list of models (idk about 600, i follow about 200), I wait 2-3 minutes, select the model, scrape/download. When that finishes, wait 2-3 minutes, and repeat.

Good advice. Will give that a shot and report back. Thank you!

You may have to extend the wait times to 5-6 mins depending how much content the model has.

[jp1151419]

I was getting a few accounts to scrape by going very slowly as we discussed but as of today it seems the paying accounts don't appear in the list anymore, no matter how long the subscription still has before it expires.