Skip to content

Conversation

@firecrawl-spring
Copy link
Contributor

Summary

This PR adds comprehensive documentation for handling API errors and implementing retry logic with exponential backoff. The documentation was created in response to customer support tickets showing confusion about rate limiting (429 errors) and how to properly handle them.

Problem

When customers encounter 429 (Too Many Requests) errors or other transient errors from the Firecrawl API, they had no clear guidance on:

  • What specific error codes mean (429, 401, 402, 5xx)
  • How to implement exponential backoff retry logic
  • Best practices for rate-limited scenarios
  • Production-ready error handling patterns

Solution

Created a comprehensive "Error Handling and Retries" guide that includes:

New Documentation

  • developer-guides/advanced-guides/error-handling-and-retries.mdx - Complete guide with practical code examples

Updated Documentation

  • rate-limits.mdx - Added section linking to error handling guide
  • api-reference/introduction.mdx - Added handling guidance for rate limit section
  • sdks/node.mdx - Enhanced error handling section with examples
  • sdks/python.mdx - Enhanced error handling section with examples

Key Features

The new guide covers:

  1. Understanding HTTP Status Codes - Explains each error code and when it occurs
  2. Rate Limiting (429 Errors) - How to detect and handle rate limits
  3. Exponential Backoff Explained - Concept and formula
  4. Implementation Examples - Complete, working code for:
    • Node.js with exponential backoff
    • Python with exponential backoff
    • Handling specific error types
    • Queue-based rate limiting for bulk operations
  5. Best Practices - Do's and don'ts for production
  6. Troubleshooting - Common issues and solutions

Code Examples

Included practical examples for:

  • Basic retry logic with exponential backoff
  • Detecting retryable vs non-retryable errors
  • Using Retry-After headers
  • Queue-based rate limiting

Testing

  • Reviewed existing documentation for gaps
  • Cross-referenced with SDK implementations
  • Verified code examples are syntactically correct
  • Ensured links are properly formatted for documentation system

Impact

This documentation update will:

  • Reduce support tickets related to rate limiting
  • Help customers implement robust error handling
  • Provide production-ready code patterns
  • Improve overall API experience

Add missing documentation for handling API errors and implementing retry logic
with exponential backoff. This addresses customer confusion about rate limiting
and how to properly handle 429 errors.

Changes:
- Create new Error Handling and Retries guide with code examples for Node.js and Python
- Add sections on exponential backoff, queue-based rate limiting, and monitoring
- Update Rate Limits documentation with link to error handling guide
- Improve API reference with error handling guidance
- Enhance Node.js and Python SDK documentation with error handling examples

The new guide includes practical code examples for:
- Basic exponential backoff retry logic
- Handling specific error types (429, 401, 402, 5xx)
- Queue-based rate limiting for bulk operations
- Best practices and troubleshooting tips

Co-Authored-By: devhims <himanshu@sideguide.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants