Skip to main content

CORS Security

Cross-Origin Resource Sharing (CORS) is a critical security mechanism in CrawlDesk that controls which external domains can access your AI search functionality and API endpoints. By implementing a domain whitelist, CrawlDesk ensures that only trusted origins (e.g., your website or app domains) can make cross-origin requests, preventing unauthorized access from malicious sites. This protects sensitive data, such as indexed documentation or API responses, from being exploited in cross-site scripting (XSS) or other attacks.

Key Benefits of CORS Whitelisting:

  • Access Control: Restricts API calls to verified domains, reducing exposure to cross-origin attacks.
  • Compliance: Aligns with security standards like OWASP by enforcing origin validation.
  • Flexibility: Easily add or remove domains as your applications evolve.

Prerequisites:

  • Active CrawlDesk account with admin access.
  • Browser access to the dashboard.
  • List of trusted domains (e.g., https://yourapp.com).

Managing Domain Whitelist​

Follow these steps to configure and maintain your CORS domain whitelist in CrawlDesk.

Adding a New Domain​

  1. Access the Security Settings
    Log in to the dashboard at https://app.crawldesk.com/dashboard/. Select your project, then navigate to Project Setting.

  2. Initiate Addition
    In the Domain Whitelist section, enter the full domain (including protocol, e.g., https://example.com) in the Add New Domain field.

  3. Save the Domain
    Click Add. The domain will be validated and added to the active list. You'll see a confirmation, and the total count (e.g., "Whitelisted Domains (1)") will update.

Viewing and Removing Domains​

  1. Review the List
    The "Whitelisted Domains" table displays:

    • Count: Total active domains (e.g., "1 Active").
    • Domain List: E.g., https://www.abc.com.
    • Actions: For each domain (e.g., "Domain #1"), click Remove domain to delete it.

  2. Remove a Domain
    Confirm the removal. Only requests from remaining whitelisted domains will access the AI search API. Removed domains will get the CORS error.

info

Only requests from these whitelisted domains will be able to access your AI search API. Non-whitelisted origins will trigger browser errors like "No 'Access-Control-Allow-Origin' header is present."

Best Practices for CORS Security​

To maximize security while using CORS in CrawlDesk:

  • Use HTTPS Only: Always whitelist HTTPS domains (e.g., https://example.com) to prevent mixed-content issues and man-in-the-middle attacks. Avoid HTTP origins.
  • Be Specific with Domains: Whitelist exact domains (e.g., https://app.example.com) rather than wildcards (e.g., https://*.example.com), as wildcards can be exploited (e.g., https://evil.example.com).
  • Regularly Audit Whitelists: Review and remove unused domains periodically to minimize attack surface. Use the dashboard's count and list for quick audits.
  • Combine with Authentication: Don't rely on CORS alone—pair it with API keys, OAuth, or tokens for credentialed requests (set Access-Control-Allow-Credentials: true if needed).
  • Monitor Logs: Check CrawlDesk logs for rejected CORS requests to detect potential probes or misconfigurations.
  • Test Thoroughly: After updates, test API calls from whitelisted domains using tools like Postman or browser dev tools to ensure no disruptions.
  • Avoid Null Origin: Never whitelist null origins, as they can be spoofed in sandboxed or redirected requests.

Following these practices helps prevent common CORS misconfigurations, such as over-permissive policies that expose your AI search to unauthorized access.