Skip to main content

RPS Rate Limiting Guide

Requests Per Second (RPS) and Requests Per Minute (RPM) rate limiting in CrawlDesk is a security feature that controls the volume of requests to your AI search functionality, including the "Ask AI" widget and API endpoints. Rate limiting prevents abuse, denial-of-service (DoS) attacks, and ensures fair usage by throttling excessive requests from a single origin or API key, returning HTTP 429 (Too Many Requests) errors when limits are exceeded. This protects your indexed data and maintains system performance.

info

To request custom limits or monitor usage, contact the CrawlDesk support team. Limits reset on a sliding window (e.g., per second, minute, hour, or day for widgets; per minute for APIs).

Default Rate Limits:

  • Ask AI Widget:
    • 10 questions per second (RPS)
    • 60 questions per minute
    • 300 questions per hour
    • 1000 questions per day
  • API:
    • 25 requests per minute (RPM) per project

Key Benefits of Rate Limiting:

  • Abuse Prevention: Caps requests to mitigate spam or malicious queries.
  • Performance Optimization: Ensures stable response times for legitimate users.
  • Scalability: Supports predefined limits suitable for various use cases, with customization available via support.

Managing Rate Limits​

Rate limits in CrawlDesk are managed by the Crawldesk team, Follow these steps to work with rate limits:

Requesting Custom Limits​

  1. Contact Support
    Submit a request for custom limits (e.g., higher widget RPS for high-traffic sites) via https://crawldesk.com/demo-request. Provide details like expected traffic, use case, and project ID.

  2. Monitor Adjustments
    After approval, updated limits will be applied by the CrawlDesk team. Contact support to request usage metrics or confirm new limits.

Monitoring Throttled Requests​

  1. Check with Support for Logs
    Since rate limit logs are not accessible in the dashboard, reach out to the CrawlDesk support team to obtain 429 error or RPS limit reached:

  2. Analyze Impact
    Work with support to review metrics and identify patterns (e.g., frequent throttling during peak hours). Request adjustments if needed.

info

Exceeding limits returns a 429 response with headers like X-RateLimit-Remaining and Retry-After. Implement exponential backoff in your client code to handle retries gracefully.