How to cut web scraping costs with block_resources

Most of a scraping bill is decided by a few choices you make once: whether a page really needs a browser, whether that browser downloads images, whether you fetch the same page twice, and which plan you are on. This guide explains how request units are counted and how to cut web scraping costs with block_resources, caching and the right plan. The numbers come from our pricing and docs.

How request units are counted

Every plan bills in one unit: the request unit. A delivered page is 1 unit, and everything about the fetch is included in it: a fresh exit IP, country targeting, sticky sessions, retries past blocks, and conversion to text or Markdown. Two things cost more, because they cost more to run:

A /v1/batch call bills each URL as its own request, so batching saves round trips, not units.

What you are never billed for

You pay for pages that come back, not for attempts. When an exit gets a 403, a 429 or a 5xx from the site, we switch exits and retry, and none of those attempts is billed. A render that fails, a failed AI extraction, a 429 for going over your rate limit, a 503 when every browser slot is busy, and a response refused with 413 for being too large are all free.

The one thing that is billed and sometimes surprises people: a genuine error page from the site, such as a 404 for a URL that no longer exists. That is the site's real answer and it took a request to get it. Clean dead URLs out of your lists instead of retrying them.

block_resources: a render for 1 unit instead of 5

A plain fetch downloads one HTML document. A render runs the page in a real browser, which downloads everything the page asks for: scripts, styles, images, fonts, trackers. We measured renders at 1 to 300 times the bytes of the same page fetched plainly, with a median around 17 times. That is why a render is 5 units.

Most of those bytes are images, fonts and media that your selectors never read. Add "block_resources": true and the browser skips them. The scripts still run, so the DOM your selectors read is the same, and the render drops from 5 units to 1: the same as a plain fetch.

curl https://scrape.land/v1/extract \
  -H "X-Api-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://shop.example/product/42",
       "render": true,
       "block_resources": true,
       "wait_for": ".price",
       "fields": {"title": "h1", "price": ".price"}}'

The response has the same shape as any extraction: url, status and your data object. On image-heavy pages, blocking removed a median of 42% of the transfer across the 14 pages we measured (77% at best), so renders also tend to finish sooner.

When not to use it: when you want a screenshot that shows the images, or when the data you need is in an image URL that only appears after the image loads. Blocking also applies alongside screenshot, so a screenshot taken with it will be missing its pictures.

Renders have one more limit worth planning around: 5 renders per second per account, the same on every plan. For large render jobs, submit them to POST /v1/jobs and collect the results asynchronously.

Do you need a render at all?

The cheapest render is the one you do not do. Many sites that look like JavaScript apps still ship their content in the HTML, or in a JSON-LD block. Before you add render, run the same extraction without it:

• If your fields come back filled, you are done at 1 unit per page and a fraction of the time.
• If they come back null (and there is no field_errors entry, so the selectors are fine), the values are added by JavaScript. Add "render": true and "block_resources": true.

Cache your own results

Fetching a page you already fetched an hour ago costs a unit for data you have. If your pipeline might ask for the same URL twice (retries after a crash, several jobs sharing inputs, a dashboard refresh), keep a local cache. A few lines of Python with the standard library's SQLite are enough:

import hashlib
import json
import sqlite3
import time

import requests

db = sqlite3.connect("scrape-cache.db")
db.execute("CREATE TABLE IF NOT EXISTS cache (k TEXT PRIMARY KEY, t REAL, body TEXT)")


def cached(endpoint, payload, max_age=24 * 3600):
    k = hashlib.sha256((endpoint + json.dumps(payload, sort_keys=True)).encode()).hexdigest()
    row = db.execute("SELECT t, body FROM cache WHERE k = ?", (k,)).fetchone()
    if row and time.time() - row[0] < max_age:
        return json.loads(row[1])          # free: no request sent
    r = requests.post("https://scrape.land" + endpoint,
                      headers={"X-Api-Key": "YOUR_KEY"}, json=payload, timeout=160)
    r.raise_for_status()
    body = r.json()
    if body.get("status") == 200:          # only cache real pages
        db.execute("INSERT OR REPLACE INTO cache VALUES (?, ?, ?)",
                   (k, time.time(), json.dumps(body)))
        db.commit()
    return body

The key is the endpoint plus the whole request body, so the same URL with different fields is cached separately. In our test the first call took 1.9 seconds and the second returned from the cache instantly without sending a request. Set max_age to how fresh the data really needs to be: a price monitor might want an hour, a list of company pages a week.

Watch your usage from code

GET /v1/account returns your plan and what is left. This is the real response for a Free key:

{
  "included_requests_remaining": 714,
  "key_budget": {
    "max_requests": 0,
    "requests_used": 230,
    "unlimited": true
  },
  "plan": "free",
  "prepaid_credit_cents": 0,
  "rate_limit_rps": 5
}

Check it before a big job. key_budget shows the per-key limit: give each job or client its own key with a request budget, and a runaway loop stops with a 402 (key-budget-exhausted) instead of spending your whole allowance.

Pick the right plan

Paid plans include a monthly allowance, and the rate per 1,000 falls as the plan grows: $0.165 on Starter ($99 for 600,000), $0.156 on Growth ($249 for 1.6M), $0.147 on Scale ($499 for 3.4M), $0.139 on Business ($999 for 7.2M), down to $0.100 on Business Ultra ($9,999 for 100M). Annual billing is ten months' price for twelve.

Past your allowance, requests draw on prepaid credit at $0.20 per 1,000, and stop with a 402 when the credit runs out, so there is no surprise bill. Unused included requests roll over to the next period. At 1,000,000 requests a month, Starter plus 400,000 requests of credit is $99 + $80 = $179, less than Growth's $249. Growth becomes cheaper once you go past about 1.35 million a month, and it also doubles your rate limit, from 50 to 100 requests per second.

Two other things decide the plan, whatever the volume:

• AI extraction starts at the Scale plan. If you need it, that is your floor.
• Pool is $19 a month with no request count at all, capped at 10 requests per second on non-priority exits. It includes CSS and XPath extraction, Markdown and raw tunneling, and the lead generation tools, but no browser rendering and no AI extraction. For steady, plain-fetch workloads it can be the cheapest option by far.

What it costs

1 unit per plain fetch, extraction or blocked-resources render; 5 per full render; 10 with a screenshot; AI extraction from +2 on Scale and up. Failures are free. The Free plan includes 1,000 requests a month. Full details on the pricing page.

Next steps

To find out whether a site needs a browser, read how to scrape a JavaScript-heavy page. For large render jobs, see async jobs and webhooks. Every render option is in the docs.