Jobs

A job is a saved scraping configuration. It combines a starting URL with a description of the data you want to extract, so Gluecrawl knows exactly what to look for.

Creating a Job

When you create a job you provide two things:

1. A URL — a page on the site. It does not need to be the exact listing page: if it isn't one already, Gluecrawl's discovery agent navigates the site — following categories or using its own search — to find the listing that matches your extraction definition.
2. An extraction definition — what data to pull from that listing.

There are two ways to define the extraction:

• Goal-based — describe what you need in plain language, e.g. "Extract all product names, prices, and ratings." Gluecrawl figures out the rest.
• Columns-based — list the exact columns you want (name, type, and an optional description for each). This gives you precise control over the output schema.

You can also set a max pages limit to control how far Gluecrawl paginates.

Processing

After a job is created, Gluecrawl's discovery agent visits the URL and, if it isn't already the listing you described, navigates the site to find it. It then analyzes the listing's structure. During processing, the agent:

• Resolves the listing page when the starting URL is a homepage, category page, or other entry point
• Identifies where each piece of data lives on the page
• Figures out how pagination works (next-page links, infinite scroll, etc.)
• Detects whether the site links to detail pages with additional data
• Determines the appropriate protection level for the site

Processing costs a fixed 10 credits and typically takes 30–90 seconds.

Job States

A job moves through the following states:

Once a job reaches ready, it stays ready permanently. You can run it as many times as you like without repeating the processing step.