Incomplete downloads - X / Twitter

[Dark-Obsidian]

Issue description

I have recently noticed gallery-dl only downloading a portion of the media expected. For example, trying to download 2,000 images/videos from an account, but gallery-dl exits after downloading <1,000.

Technical summary

The download seems to progress normally, but after a certain point it encounters a 404 error and prompts to re-try the download with a specific cursor reference. Doing this then results in:

• gallery-dl pausing for 15 minutes due to being rate limited
• once the timeout expires
  • re-attempting the download (skipping previously downloaded-files)
  • within a few minutes, reaching the same point and throwing another 404 error
  • advising trying the download again with a slightly different cursor value

Debug log

[urllib3.connectionpool][debug] https://x.com:443 "GET /i/api/graphql/4fpceYZ6-YQCx_JSl_Cn_A/SearchTimeline?variables=%7B%22rawQuery%22%3A%22from%3ABoyfriend__AI+max_id%3A1797372622532903014+filter%3Alinks%22%2C%22count%22%3A20%2C%22querySource%22%3A%22typed_query%22%2C%22product%22%3A%22Latest%22%2C%22withGrokTranslatedBio%22%3Afalse%2C%22cursor%22%3Anull%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22payments_enabled%22%3Afalse%2C%22rweb_xchat_enabled%22%3Afalse%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22responsive_web_grok_show_grok_translated_post%22%3Afalse%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D HTTP/1.1" 404 0
[twitter][debug] API error: 'Unspecified'
[twitter][debug] 
Traceback (most recent call last):
  File "gallery_dl\job.py", line 158, in run
  File "gallery_dl\job.py", line 209, in dispatch
  File "gallery_dl\extractor\twitter.py", line 86, in items
  File "gallery_dl\extractor\twitter.py", line 869, in tweets
  File "gallery_dl\extractor\twitter.py", line 1981, in _pagination_tweets
  File "gallery_dl\extractor\twitter.py", line 1854, in _call
gallery_dl.exception.AbortExtraction: 404 Not Found ()
[twitter][error] 404 Not Found ()
[twitter][info] Use '-o cursor=2_1798656188026167507/DAADDAABAAgAAgAAAAIIAAMAAAAACAAEAAAAAAoABRw4pi8NwCcQCgAGHDimLw2_2PAAAA' to continue downloading from the current position

Other notes

I have also observed this happening on downloads from civitai but believe this may be due to a separate issue. Will create a separate post for this.