How the scraping API works, request by request

The one request you send

Everything is one POST with an X-Api-Key header. POST /v1/extract with a url and a fields map is the request most people use most of the time; POST /v1/fetch returns the whole page instead, as HTML, visible text, Markdown or raw bytes. There is no session to establish, no crawler to configure and no state to keep between calls. Every parameter below is a key in that same JSON body.

There is also a raw forward proxy, if what you want is HTTP access to the page yourself rather than structured data. Point any client at it with your API key as the username and the same rotation, country and session controls apply. The rest of this page describes both, because they share one engine.

Step 1: choosing an exit

Every request goes out through a different IP address by default. You do not manage a proxy list, rotate anything, or handle a pool going stale — you send a URL and the address is chosen for you, fresh, per request.

Which addresses you are offered depends on your plan. Paid plans are offered premium ISP and datacenter exits first, then a deeper bench of continuously health-checked addresses, then a last-resort attempt so a paid request does not fail simply because no exit was free. The Free plan is served from a high-latency public pool — same rotation, same features, slower addresses that the harder targets block more often. That is what the free tier is for: prototyping and testing.

Country targeting

Add "country": "us" — any ISO country code — and the request leaves from that country. On the raw proxy it is -country-us appended to your key. This matters more than it sounds: prices, availability, language, and sometimes the entire page differ by where the request appears to come from, and a US price scraped from a European IP is quietly wrong rather than obviously broken.

One implementation detail that is worth stating because it is not the obvious approach: we resolve each address's country directly, in batches, rather than by making a request through it. Geolocating a proxy by using it means only the proxies you have recently used have a known country, so the targetable share of the pool is always a fraction of the live pool. Resolving directly keeps effectively the whole live pool geo-targetable.

Sticky sessions

Rotation is right up until it isn't. A login, a cart, a multi-step form or anything behind a session cookie needs the same IP across several requests, because the site is watching for exactly the pattern where a session hops between countries mid-checkout. Pass "session": "some-name" and that name is pinned to one exit IP for about ten minutes, then rotates. Use a distinct session name per logical user or job. Sticky sessions work on every plan, Free included, and cost nothing extra.

Step 2: the retry, and why blocks are free

This is the part worth understanding properly, because it is where a scraping API either earns its price or quietly bills you for its own failures.

When a response comes back, we decide whether it is an answer from the target or a symptom of the exit we picked. A 403, a 429, a 407, a timeout, a 5xx, a CDN shield error — those say "this IP is not welcome", not "this is what the page contains". So we discard the response entirely, roll to a different exit, and try again. You are never handed one of those, and you are never billed for one.

The other direction is just as deliberate. A 400, 401, 404, 410 or 451 is a genuine answer from the target about your request, so we hand it straight through rather than burning three more exits discovering that the page really is missing. Target-side statuses are passed through unchanged; we do not translate the site's answers into ours.

Two limits are worth knowing. Non-idempotent requests — a POST, a PUT — are never retried, because retrying one could submit a form or place an order twice; a failed exit on a POST still is not billed, we simply do not try again automatically. And if every exit we tried failed, you get an unbilled error rather than the last piece of junk we saw. Billing on delivery means there is nothing to hand you and nothing to charge for.

Learning from blocks

When a target blocks a request with a 403, 429 or 451, we record which domain blocked which exit country for your account, and route that domain through other countries next time. It happens asynchronously and never slows the request that triggered it. Over a long-running job against a site that dislikes one region, this is the difference between a retry loop and a route that settles.

Step 3: rendering, when the page needs a browser

Plenty of pages are a shell that assembles itself in JavaScript. Fetching one plainly gets you an empty div and a bundle reference. Add "render": true and we drive a real Chrome, let the page run, then read the DOM and apply your selectors to what the page actually became.

Pair it with "wait_for" set to a CSS selector to hold until that element exists before reading — far more reliable than guessing a sleep duration, because it waits for the thing you care about rather than for a number you picked. For content that only appears after interaction, actions runs scripted steps first: scroll, click, wait, wait_for and fill, which covers infinite scroll and content behind a button. device: "mobile" renders as a phone if the mobile layout is the one you want, and fingerprint: true presents a browser identity coherent with your chosen country — timezone, locale, navigator.languages — so the render reads as one real client from that region.

A render is not a fetch with a flag set. It downloads everything the page asks for — scripts, stylesheets, images, fonts, trackers — and holds a browser slot for the duration. Measured over 14 pages in a real browser, a render pulls between roughly 1 and 300 times the bytes of the same page fetched plainly, median around 17 times, with image-heavy commerce at the top of that range. That is why it bills 5 request-units instead of 1, and 10 if you also want a screenshot.

It is also why block_resources costs 1. Skipping images, fonts and media leaves the DOM your selectors read completely unchanged and removes around 40% of the transfer on a heavy page. If you are extracting fields rather than looking at pictures, it is the default you want. Renders are capped at 5 per second per account on every plan; for anything slower or bulkier, POST /v1/jobs runs it asynchronously with a five-minute budget and can call a webhook when it finishes.

Step 4: turning the page into fields

By this point there is a document. What comes back to you depends on what you asked for, and there are three ways to ask.

A fields map of CSS or XPath selectors returns a data object with one key per field, on every plan including Free — and when a selector will not compile, the field is null and a field_errors entry names it, so "the page has no price" and "your selector is broken" are never the same answer. A prompt describes the fields in plain English instead, adapting to markup you have never seen, on Scale and above. And POST /v1/fetch hands back the document itself as html, text, markdown or raw bytes, at one request-unit.

What comes back, and what it costs

One JSON response, with the target's own status passed through and the exit country reported in a header. Up to 20 URLs can share a single call via POST /v1/batch, each billed as its own delivered request. Anything long-running goes to POST /v1/jobs, which returns an id immediately and either polls or posts to a webhook you nominate — signed with HMAC-SHA256 so you can verify it came from us. Refusals caused by your plan all answer with a stable machine-readable code and a fallback telling you what to do instead, so you write one branch rather than matching English sentences.