What people build with a web scraping API

Price and availability monitoring

Tracking competitor prices, stock levels or shipping terms across a set of product pages, on a schedule. The hard part is almost never parsing a price — it is that the same URL shows a different price depending on where the request came from, that commerce pages are the most aggressively bot-protected pages on the web, and that a monitor which silently starts returning null looks identical to a product that went out of stock.

Country targeting is doing real work here, not garnish: "country": "de" and "country": "us" on the same product URL are two different answers, and which one you meant is a business question. Retail pages are also the heaviest thing we serve, which makes block_resources the single best flag on this job — a rendered product page without its images extracts identically, returns faster, and bills 1 unit instead of 5.

For the null problem, use selectors and read field_errors. A price field that comes back null with no field_errors means the page genuinely has no price element; the same null with a field_errors entry means the site changed its markup and your selector no longer compiles. Alerting on the second is how a monitor tells you it broke instead of quietly reporting that everything is free.

A practical shape: POST /v1/batch with up to 20 product URLs, a shared fields map, "country" set to the market you are pricing, and "block_resources": true if the pages need rendering. Each URL bills as its own delivered request. Blocked attempts cost nothing, which matters a lot on this job specifically, because the retry rate against commerce sites is the highest of anything here.

Lead and company data

Enriching a list of companies from their own websites: what the business does, where it is, which technologies or job titles it mentions, whether it is hiring. The input is usually a domain and the output is a row in a CRM.

This is the job where selectors are the wrong tool and it is worth being blunt about why. Ten thousand company sites have ten thousand different layouts. There is no h1 convention, no .about class, nothing to select against — writing a parser per site is the entire cost of the project. A prompt describing the fields you want works across all of them without knowing any of their markup, which is exactly the case AI extraction exists for. It needs the Scale plan or above and bills 2 extra request-units on the fast tier.

Send a schema alongside the prompt to pin the output shape, so every row has the same keys and your database insert does not need to defend itself. For people and company profiles specifically, extract_type: "profile" applies a curated prompt and schema without you writing either. If a company site is a JavaScript app — increasingly they all are — add "render": true with "block_resources": true, which keeps the render at 1 unit.

Two boundaries to design around. The API is stateless and does not crawl: it fetches the URL you give it and nothing else, so discovering a company's About page is your job — "links": true returns every link on the homepage resolved to an absolute URL, which is usually enough to pick the next fetch. And personal data carries obligations independent of how you collected it; see our Acceptable Use Policy and Terms before building a pipeline around it.

Search results

Finding which pages exist for a query, monitoring how a set of terms resolves over time, or feeding a discovery step that decides what to fetch next.

POST /v1/search takes a query and returns organic results — title, the real destination URL with the engine's redirect wrapper already unwrapped, and a snippet. It bills as one delivered request. Be aware of the shape of it before you plan around it: count is capped at 10, because that is one result page from the engine, and asking for more returns 10 anyway. There is no pagination parameter, and adding one would not be a small change — it is a different product.

What it is genuinely good for is discovery inside a larger pipeline: resolve a query to ten URLs, then fetch and extract each one properly with the rest of the API. What it is not is a rank-tracking product. If you need deep result pages, per-engine parity or position tracking across hundreds of keywords, this endpoint is not that, and we would rather say so here than have you find out after integrating.

Market and competitive research

One-off or periodic sweeps that answer a question rather than fill a table: how a segment positions itself, which features competitors advertise, how pricing pages have changed, what a category's job postings imply about where it is investing.

The distinguishing feature of this job is that you often do not know the schema in advance. You are not extracting a known field from a known page — you are asking a question of a page you have not seen. "structured": false is built for exactly that: send a prompt like "what does this company sell and who to?" and get a plain-language answer back, with no data key and no invented field names for you to guess at. It costs the same as structured extraction, because it is the same page and the same model call.

When you do know the shape, the presets cover most of the ground without a schema: article for editorial and press coverage, job for hiring signals, event, real_estate, discussion for forum and community threads. "metadata": true is underrated on this job — it returns parsed schema.org JSON-LD, OpenGraph tags, canonical and publish dates with no selectors and no model call at all, at one request-unit. A surprising amount of what people write prompts for is already sitting in the page's own structured data.

Research sweeps are bursty by nature: nothing for a week, then several thousand pages in an afternoon. Rollover suits that pattern — unused requests carry into the next period rather than being forfeited — and a burst that outruns your rate limit gets a 429 with a Retry-After rather than a failure, so a backoff loop is all the handling it needs.

Training and grounding data for LLMs

Building a corpus for fine-tuning, or a retrieval index that a model reads from at query time. Volume is high, per-page value is low, and the quality bar is entirely about what the text looks like after cleaning.

"format": "markdown" is the endpoint for this. Navigation, headers, footers, asides, scripts and styles are stripped; headings, lists, links, code blocks and tables survive; and relative links and images are resolved to absolute URLs, which is the detail that matters once a chunk is separated from the page it came from and nobody can tell what /img/3.png referred to. It costs one request-unit — the same as raw HTML, with no Markdown or AI surcharge — and shipping HTML tags into a context window means paying for them as tokens twice, once to us and once to your model provider.

At corpus scale, three things stop mattering in theory and start mattering on the invoice. Blocked requests being free changes the economics of crawling long tails of unfamiliar domains, where the block rate is high and unpredictable. block_resources keeps rendered pages at 1 unit, which at a million pages is the difference between one bill and five. And the response size cap is a guard rather than a limit — the median response we serve is 40.6 KB, so what it actually catches is the accidental fetch of a video or an archive that would have contributed nothing to a text corpus anyway.

For throughput, POST /v1/batch takes 20 URLs per call and POST /v1/jobs runs work asynchronously with a webhook callback, so a large sweep does not need you to hold thousands of open connections. Rate limits scale with the plan, up to 1,000 requests per second. What the API deliberately does not do is crawl for you: it has no frontier, no queue and no politeness scheduler, because those are decisions about your corpus that we would only get wrong. You bring the URL list; we turn each URL into clean text.

Something else

All five patterns are the same primitive with different parameters, which means most jobs that are not on this list are also just a different combination. The free tier is 1,000 requests with no card, which is enough to find out whether yours is.