Gluecrawl Docs

Getting Started

Create a job, wait for it to be ready, run it, and retrieve structured data.

The Gluecrawl API turns a listing or directory page into structured data. This quickstart creates a job, waits for Gluecrawl to map the page, starts a run, and retrieves the resulting items.

Base URL: https://api.gluecrawl.ai/v1/

Authentication

All /v1/ endpoints require an API key passed as a Bearer token.

API access is included from the Starter plan up. Free accounts can use the dashboard, but cannot generate an API key — the button below is replaced by an upgrade notice until you subscribe.

  1. Log in to your Gluecrawl dashboard
  2. Click your avatar → API Key → Generate API Key
  3. Copy your key — it is shown only once

Include the key in every request:

Authorization: Bearer glue_yourApiKeyHere

Set it as an environment variable before using the examples below:

export GLUECRAWL_API_KEY="glue_yourApiKeyHere"

The workflow

Every scrape follows the same sequence:

  1. Create a job with a URL and extraction input.
  2. Poll the job until its status is ready or failed.
  3. Create a run for the ready job.
  4. Poll the run until its status is completed or failed.
  5. Retrieve JSON items or download CSV.

The examples use a product listing URL, but the same flow works for directories, job boards, marketplaces, and other repeating-page collections.

The request and response bodies below are illustrative. Replace JOB_ID and RUN_ID with the IDs returned by your own API calls, and replace the example URL and extraction goal with your target.

cURL quickstart

1. Create a job

Use a goal when you want Gluecrawl to infer the output fields from a plain-language instruction.

curl -X POST https://api.gluecrawl.ai/v1/jobs \
  -H "Authorization: Bearer $GLUECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products",
    "input": {
      "type": "goal",
      "value": "Extract all product names, prices, and product URLs"
    },
    "max_pages": 10
  }'

The response includes the job id and a mapping status. Keep the id; you will use it in later requests. A job progresses through in_progress, ready, or failed.

{
  "id": "JOB_ID",
  "status": "in_progress"
}

2. Wait for the job to be ready

Request the job again until its status is ready. A ready job includes the inferred columns and protection_level.

curl "https://api.gluecrawl.ai/v1/jobs/JOB_ID" \
  -H "Authorization: Bearer $GLUECRAWL_API_KEY"
{
  "id": "JOB_ID",
  "status": "ready",
  "columns": {
    "listing": [
      { "name": "product_name", "type": "text" },
      { "name": "price", "type": "number" },
      { "name": "product_url", "type": "url" }
    ],
    "detail": []
  },
  "protection_level": "light"
}

3. Create and wait for a run

Only create a run after the job is ready.

curl -X POST "https://api.gluecrawl.ai/v1/jobs/JOB_ID/runs" \
  -H "Authorization: Bearer $GLUECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "max_pages": 10 }'

The new run starts as queued. Poll GET /v1/runs/RUN_ID until its status is completed or failed.

{
  "id": "RUN_ID",
  "job_id": "JOB_ID",
  "status": "queued",
  "credits_used": null
}
curl "https://api.gluecrawl.ai/v1/runs/RUN_ID" \
  -H "Authorization: Bearer $GLUECRAWL_API_KEY"

When the run completes, its response includes item_count, page_count, credits_used, and settled billing details.

{
  "id": "RUN_ID",
  "status": "completed",
  "item_count": 120,
  "page_count": 5,
  "credits_used": 5
}

4. Retrieve the data

curl "https://api.gluecrawl.ai/v1/runs/RUN_ID/items?limit=50&offset=0" \
  -H "Authorization: Bearer $GLUECRAWL_API_KEY"
{
  "items": [
    {
      "data": {
        "product_name": "Wireless Keyboard",
        "price": 49.99,
        "product_url": "https://example.com/products/wireless-keyboard"
      },
      "page_number": 1,
      "item_index": 0
    }
  ],
  "total": 120,
  "limit": 50,
  "offset": 0
}

For a file download instead, call the CSV export endpoint.

Node.js quickstart

This example uses Node.js with its built-in fetch. Set GLUECRAWL_API_KEY in your environment before running it.

const baseUrl = 'https://api.gluecrawl.ai/v1'
const headers = {
  Authorization: `Bearer ${process.env.GLUECRAWL_API_KEY}`,
  'Content-Type': 'application/json',
}
const sleep = (milliseconds) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds))

async function request(path, init = {}) {
  const response = await fetch(`${baseUrl}${path}`, {
    ...init,
    headers: { ...headers, ...init.headers },
  })
  const body = await response.json()

  if (!response.ok) {
    throw new Error(`${response.status}: ${body.error?.message ?? 'Request failed'}`)
  }

  return body
}

async function waitForStatus(path, completedStatus) {
  for (;;) {
    const resource = await request(path)

    if (resource.status === 'failed') {
      throw new Error(`Gluecrawl processing failed: ${resource.error ?? 'Unknown error'}`)
    }
    if (resource.status === completedStatus) return resource

    await sleep(2000)
  }
}

const job = await request('/jobs', {
  method: 'POST',
  body: JSON.stringify({
    url: 'https://example.com/products',
    input: {
      type: 'goal',
      value: 'Extract all product names, prices, and product URLs',
    },
    max_pages: 10,
  }),
})

const readyJob = await waitForStatus(`/jobs/${job.id}`, 'ready')
const run = await request(`/jobs/${readyJob.id}/runs`, {
  method: 'POST',
  body: JSON.stringify({ max_pages: 10 }),
})
const completedRun = await waitForStatus(`/runs/${run.id}`, 'completed')
const items = await request(`/runs/${completedRun.id}/items?limit=50&offset=0`)

console.log(items.items)
console.log(`Used ${completedRun.credits_used} credits`)

Python quickstart

Install requests, set GLUECRAWL_API_KEY, then run this script.

import os
import time

import requests

BASE_URL = "https://api.gluecrawl.ai/v1"
HEADERS = {
    "Authorization": f"Bearer {os.environ['GLUECRAWL_API_KEY']}",
    "Content-Type": "application/json",
}

def request(method, path, **kwargs):
    response = requests.request(method, f"{BASE_URL}{path}", headers=HEADERS, **kwargs)
    response.raise_for_status()
    return response.json()

def wait_for_status(path, completed_status):
    while True:
        resource = request("GET", path)
        if resource["status"] == "failed":
            raise RuntimeError(f"Gluecrawl processing failed: {resource.get('error', 'Unknown error')}")
        if resource["status"] == completed_status:
            return resource
        time.sleep(2)

job = request(
    "POST",
    "/jobs",
    json={
        "url": "https://example.com/products",
        "input": {
            "type": "goal",
            "value": "Extract all product names, prices, and product URLs",
        },
        "max_pages": 10,
    },
)

ready_job = wait_for_status(f"/jobs/{job['id']}", "ready")
run = request("POST", f"/jobs/{ready_job['id']}/runs", json={"max_pages": 10})
completed_run = wait_for_status(f"/runs/{run['id']}", "completed")
items = request("GET", f"/runs/{completed_run['id']}/items?limit=50&offset=0")

print(items["items"])
print(f"Used {completed_run['credits_used']} credits")

Polling, retries, and credits

Polling

  • Poll a job until it is ready or failed; only a ready job can create a run.
  • Poll a run until it is completed or failed. Items can appear progressively while a run is in progress, but a completed run contains final counts and settled billing.
  • The two-second interval in the examples is illustrative. Keep the total requests for an API key within the 60 requests-per-minute limit, especially when you run several workflows at once.

Retries

  • For 429 rate_limited, wait for the response's Retry-After value before making another request.
  • Do not retry 401, 402, 403, 404, 409, or 422 blindly. Fix the API key, credits, plan, resource/state, or request body first. See Error Codes.
  • If your client loses a response to a write request, first check the relevant job or run list before creating another resource. This avoids treating an unknown outcome as a safe duplicate.

Credits

  • Creating a job costs 10 credits. The charge is refunded automatically if job processing fails.
  • Runs are charged after completion for the work actually performed. Cost depends on the selected protection level, listing pages, and any detail-page visits.
  • Set max_pages deliberately, then use a completed run's credits_used and billing fields to understand its final cost. See Credits for the rate table.

Extract a Product Catalog

Turn a category or collection page into structured product data.

On this page

AuthenticationThe workflowcURL quickstart1. Create a job2. Wait for the job to be ready3. Create and wait for a run4. Retrieve the dataNode.js quickstartPython quickstartPolling, retries, and creditsPollingRetriesCredits