Why “load more” and infinite scroll break product intelligence—and how to capture everything

A collection page that shows 24 cards tells you almost nothing about coverage. Look for a result count, page count, category filters, search terms, regional settings, and the point at which new items stop appearing. Write down the intended scope before you run the job.

This matters because a catalogue may be divided across category pages, internal search, collections, or product variants. A dataset can look clean while silently omitting the pages a client cares about most.

Traditional pagination exposes a sequence of page URLs. A load-more button often appends cards to the existing page. Infinite scroll may request the next set only after someone moves through the page. Filters can change either the URL, the page content, or both.

Do not assume that a button means the rest of the catalogue is unavailable. Inspect how the site behaves and test the last page or final load. Search crawlers generally rely on crawlable links rather than triggering interactive controls, which is also a useful reminder to make your own coverage rule explicit.

Use listing or search pages to discover every product URL, then use product pages for fields that exist only at the detail level. Trying to collect every field from cards usually produces incomplete specifications, missing variants, or a mix of product and category data.

Keep the discovery URL, product URL, and page position when they help explain a result. These fields make it easier to rerun the same scope and diagnose an unexpected row count later.

A product with several purchasable variants can be one product for an assortment view but many rows for a pricing view. Choose before collecting. If the price, stock, or URL changes by size, color, or pack count, a parent-only row hides information that may matter.

Record the parent product and selected variant separately when possible. It keeps the analysis flexible: you can report at product level while retaining the detail needed to investigate an outlier.

Use at least three checks: compare the row count with the displayed result count when one exists, deduplicate canonical product URLs or identifiers, and inspect samples near the first, middle, and final part of the collection. If the source shows 300 results and the export has 24, the job is not finished.

For a recurring collection, track count changes over time. A sudden drop may be a genuine assortment change, but it can also be a new filter, layout, or loading behavior. Flag it for review rather than silently accepting a smaller dataset.