Work with Protected and Dynamic Sites
Understand automatic protection selection, runtime escalation, and costs.
Gluecrawl automatically chooses how to access a target site. You do not select a protection level in the API request.
Let Gluecrawl map the site first
Create a job with the target listing page, then poll it until it is ready or failed.
POST /v1/jobs
GET /v1/jobs/JOB_ID → wait for status: ready or failedDuring job processing, Gluecrawl determines the appropriate protection_level and returns it on the ready job. A dynamic site may use a full browser environment; a protected site may need external routing.
| Level | Typical use | Listing-page credits |
|---|---|---|
| Light | Content is served directly | 1 |
| Moderate | Content loads after page scripts run | 2 |
| Heavy | The site blocks simple automated requests | 3 |
| Strict | The site needs the most thorough approach | 6 |
See Protection Levels for the full explanation, including detail-page rates.
Read the ready job before running it
Once a job is ready, use its mapped columns and protection_level to understand the planned extraction. Then create a run and monitor it through GET /v1/runs/{id}.
If a site's defenses change between runs, Gluecrawl can escalate to a higher protection level during a run and update the job for future runs. Higher protection levels can increase the cost per page, so use max_pages to keep the first run deliberately bounded.
Handle failures deliberately
- A job in
failedstate did not complete processing. Check its error message, correct the target URL or extraction request if needed, and create a new job. - A run in
failedstate did not complete scraping. Check the run response before deciding whether a later rerun is appropriate. - Do not treat an access or mapping failure as a reason to increase
max_pages; that limit only controls the number of listing pages to crawl. - Follow the target site's terms and your applicable legal obligations when using scraped data.
Estimate before scaling up
Job creation is a fixed 10-credit charge, while run cost depends on actual listing pages, detail-page visits, and protection level. Start with a small max_pages value, inspect the completed run's credits_used and billing, then increase scope if the result meets your needs. See Credits.