Schedule a Recurring Scrape

Schedules create fresh runs for an existing job on a cadence you choose. They are useful for catalog monitoring, inventory refreshes, directories, and other data that changes over time.

1. Create and prepare the job

Create the job normally, then poll it until its status is ready. You cannot create a schedule for a job that is still mapping or has failed.

POST /v1/jobs
GET  /v1/jobs/JOB_ID  → wait for status: ready

The schedule reuses the ready job's mapping. It does not repeat the job processing step on every trigger.

2. Attach the schedule

Send PUT /v1/jobs/{id}/schedule with the cadence and the maximum listing pages for each scheduled run.

curl -X PUT https://api.gluecrawl.ai/v1/jobs/JOB_ID/schedule \
  -H "Authorization: Bearer $GLUECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "interval_type": "days",
    "every": 1,
    "time": "9:00 AM",
    "max_pages": 10
  }'

The job response includes the saved schedule configuration:

{
  "id": "JOB_ID",
  "status": "ready",
  "schedule": {
    "interval_type": "days",
    "every": 1,
    "time": "9:00 AM",
    "enabled": true
  }
}

Choose a cadence

interval_type accepts one-time, minutes, hours, days, weeks, or months.

• Use every for a repeating cadence, such as every 2 hours or every 7 days. It is ignored for one-time schedules.
• Include time for days, weeks, months, and one-time schedules, for example "9:00 AM".
• For weeks, pass weekdays, where 0 is Monday and 6 is Sunday.
• For months, pass day_of_month from 1 to 31.
• Use start_date in YYYY-MM-DD format when you need to choose the first run date. For a one-time schedule, it is the execution date.

The Set a Schedule reference has the complete parameter list and an interactive request builder.

What happens on each trigger

Each trigger creates a separate run with its own items and billing. Read its results through List Runs, Get a Run, and Get Items, exactly as you would for a manually created run.

Scheduled runs consume credits when they execute. Make sure your account can cover the expected work; if the balance is insufficient when a trigger runs, the run does not start.

Update or remove the schedule

Send PUT /v1/jobs/{id}/schedule again to change a cadence or set its enabled value. To remove it entirely, call DELETE /v1/jobs/{id}/schedule. Removing a schedule does not affect runs already in progress.