Choose Goal-Based or Columns-Based Extraction

Every job needs an input definition. Choose goal-based extraction when you want to describe the result in plain language; choose columns-based extraction when your downstream system needs an explicit field list.

Goal-based: describe the result

Goal-based input is the recommended starting point. Give Gluecrawl a clear statement of the data you want, including the important fields and the kind of item to extract.

{
  "type": "goal",
  "value": "Extract all job listings with title, company name, location, salary, and application URL"
}

Use a goal when:

• You are exploring an unfamiliar listing or directory page.
• You want Gluecrawl to infer the output schema from a natural-language outcome.
• The required fields are easier to describe than to model up front.

After mapping succeeds, inspect the job's columns.listing and columns.detail to see the inferred output fields and types.

Columns-based: define the schema

Columns-based input names the fields you want and can assign a type to each field. Supported types are text, number, url, array, and date.

{
  "type": "columns",
  "value": [
    { "name": "product_name", "type": "text" },
    { "name": "price", "type": "number" },
    { "name": "product_url", "type": "url" },
    { "name": "available_sizes", "type": "array" }
  ]
}

Use columns when:

• A database, spreadsheet, or workflow expects fixed names and types.
• You need a repeatable schema across several similar jobs.
• You already know the fields your application consumes.

Decide before you create the job

The input belongs to the job's mapping configuration. Choose the mode before calling POST /v1/jobs, then poll the job until it reaches ready before creating a run.

Both modes can discover detail-page fields and pagination. Both also use the same run, item, and CSV endpoints after the job is ready.

Practical checklist

• Start with a goal when the desired outcome is clear but the site's structure is not.
• Use columns when field names and data types are part of your integration contract.
• Keep field names meaningful and stable for the system that will consume item.data.
• Set max_pages separately: it controls how many listing pages Gluecrawl may crawl, regardless of which input mode you choose.