Handle Errors, Retries, and Rate Limits

All Gluecrawl API errors use the same JSON envelope:

{
  "error": {
    "code": "error_code_string",
    "message": "Human-readable description."
  }
}

Use both the HTTP status and error.code to decide what your client should do next.

For the complete API code reference, see Error Codes.

Poll status; do not guess

The normal polling workflow is itself a retry-safe way to wait for asynchronous work:

• Poll a job until it reaches ready or failed.
• Poll a run until it reaches completed or failed.
• Stop polling when you receive a terminal state. A failed resource needs investigation, not an endless request loop.

Use a bounded interval and a shared request budget when several jobs run concurrently.

Respect the global rate limit

All /v1/ endpoints share a limit of 60 requests per minute per API key. A rate-limited response includes a Retry-After header, expressed in seconds.

HTTP/1.1 429 Too Many Requests
Retry-After: 15

When you receive 429:

1. Read Retry-After.
2. Wait at least that many seconds.
3. Retry the request after the wait.

Coordinate polling workers behind the same API key. For example, a two-second polling interval is 30 requests per minute for one active poller before any create, list, or item requests are counted.

Treat write requests carefully

Creating a job, run, or schedule changes state. Do not blindly resend a write request after a client-side timeout or lost connection: the API may already have processed it even though your client did not receive the response.

Instead, inspect the relevant resource or list before creating another one. Once you have a job or run ID, use its status endpoint to continue the workflow.

Retry only after the cause is clear

• Retry 429 after its server-provided wait period.
• Correct the credentials, credit balance, input body, or job state for 401, 402, 422, and 409 rather than automatically retrying them.
• Treat 404 as an ID or ownership problem, not a transient failure.
• For a job or run with status: failed, inspect the returned error and decide whether a new job or rerun makes sense for your target.

This keeps retries from creating duplicate work, hiding configuration errors, or exhausting the shared rate limit.