Documentation

Data extraction API

Give us a URL and a set of fields. We fetch the page, apply your CSS selectors, and hand back clean JSON. No browser to run, no parsing code, no IP rotation to manage. The base URL for every endpoint is https://scrape.land.

Authentication

Send your API key in the X-Api-Key header on every request:

X-Api-Key: pb_live_YOURKEY

An Authorization bearer header works too, if that fits your stack better:

Authorization: Bearer pb_live_YOURKEY

curl https://scrape.land/v1/extract \
  -H "X-Api-Key: pb_live_YOURKEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example-product.com/item/42",
       "fields":{"title":"h1","price":".price"}}'

import requests

r = requests.post("https://scrape.land/v1/extract",
    headers={"X-Api-Key": "pb_live_YOURKEY"},
    json={"url": "https://example-product.com/item/42",
          "fields": {"title": "h1", "price": ".price"}},
    timeout=60)
print(r.json()["data"])

from scrapeland import ScrapelandClient

client = ScrapelandClient("pb_live_YOURKEY")   # or $SCRAPELAND_API_KEY
data = client.extract("https://example-product.com/item/42",
                      {"title": "h1", "price": ".price"})
print(data)

const res = await fetch("https://scrape.land/v1/extract", {
  method: "POST",
  headers: { "X-Api-Key": "pb_live_YOURKEY", "Content-Type": "application/json" },
  body: JSON.stringify({
    url: "https://example-product.com/item/42",
    fields: { title: "h1", price: ".price" },
  }),
});
console.log((await res.json()).data);

package main

import ("bytes"; "encoding/json"; "fmt"; "net/http")

func main() {
    body, _ := json.Marshal(map[string]any{
        "url":    "https://example-product.com/item/42",
        "fields": map[string]string{"title": "h1", "price": ".price"},
    })
    req, _ := http.NewRequest("POST", "https://scrape.land/v1/extract", bytes.NewReader(body))
    req.Header.Set("X-Api-Key", "pb_live_YOURKEY")
    req.Header.Set("Content-Type", "application/json")
    resp, _ := http.DefaultClient.Do(req)
    defer resp.Body.Close()
    var out map[string]any
    json.NewDecoder(resp.Body).Decode(&out)
    fmt.Println(out["data"])
}

$ch = curl_init("https://scrape.land/v1/extract");
curl_setopt_array($ch, [
  CURLOPT_POST => true,
  CURLOPT_HTTPHEADER => ["X-Api-Key: pb_live_YOURKEY", "Content-Type: application/json"],
  CURLOPT_POSTFIELDS => json_encode([
    "url" => "https://example-product.com/item/42",
    "fields" => ["title" => "h1", "price" => ".price"],
  ]),
  CURLOPT_RETURNTRANSFER => true,
]);
$data = json_decode(curl_exec($ch), true)["data"];

{
  "url": "https://example-product.com/item/42",
  "status": 200,
  "data": {
    "title": "Vintage desk lamp",
    "price": "49.00"
  }
}

How fields map to selectors

Each entry in fields is one of these forms:

• A CSS selector string. We return the text of the first matching element. "title": "h1" returns the text inside the first h1.
• A selector@attr shorthand to read one attribute. "image": "img.gallery@src" returns the src; use @text or no param for the text.
• An XPath shorthand: a string whose first character is / or ( is treated as XPath, anything else as CSS. "title": "//h1/text()", "link": "//a/@href", "first": "(//p)[1]".
• An object with css plus optional keys: attr (read an attribute instead of text), all (return every match as an array), or xpath (an XPath expression instead of css, for top-level fields). They combine.
• A group: an object with css plus its own fields. The css picks the repeating container and the inner fields are read inside each one, so you get a list of objects instead of parallel arrays that can drift apart. Use "." for the container itself. "articles": {"css": ".card", "fields": {"title": "h3", "url": {"css": "a", "attr": "href"}}} returns [{"title": ..., "url": ...}, ...], and a field missing from one container is null in that row rather than shifting every row after it.

Where XPath works. XPath is supported for top-level fields, as the shorthand or the xpath key, and an XPath that does not compile is a 400 naming the field. A group's container must be CSS: {"xpath": ..., "fields": {...}} is a 400. Inside a group's fields, XPath is not supported: an XPath sub-field comes back null in every row, so use CSS there.

Set "group": true on the request to zip the list fields into one object per row under items, instead of parallel arrays you line up yourself. Rows are built to the longest list, and a field with no value at that index is null. If the lists matched different numbers of elements the response says so, because the padding goes at the end: when the real gap was in the middle, every column after it is offset. Where rows can legitimately have gaps, prefer the nested group above, which reads every value of a row from that row’s own container and cannot drift.

So this field map:

{
  "title": "h1",
  "price": ".price",
  "image": "img.gallery@src",
  "tags":  {"css": ".tag", "all": true}
}

produces this data object, field by field:

When a field comes back null

A field is null either because the page genuinely has no such element, or because the selector itself could not be used. Those mean very different things to you, so we tell them apart: if a selector could not be compiled, the response carries a field_errors object naming the field and why. The field is still present in data as null (an empty array for an "all": true field), the request is still a 200, and your other fields are unaffected: one unusable selector never sinks the rest of the extraction.

{
  "data": {"title": "Example Domain", "price": null, "tags": null},
  "field_errors": {
    "tags": "selector did not compile (unknown pseudoclass or pseudoelement :is) — this field is null because the selector could not be used, not because the page lacks the element"
  }
}

No field_errors key means every selector compiled, so a null there is a real "not on the page". XPath is validated up front instead and returns 400.

Request options

These work on every endpoint that fetches a page: /v1/extract, /v1/fetch, /v1/batch, /v1/rank, and inside jobs. Browser options need render.

JavaScript pages: render & wait_for

If the page builds its content with JavaScript, add "render": true. We drive a real browser, let the page run, then read the DOM and apply your selectors. Pair it with "wait_for" set to a CSS selector to hold until that element appears before reading. It is the reliable way to wait for late-loading content:

curl https://scrape.land/v1/extract \
  -H "X-Api-Key: pb_live_YOURKEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example-product.com/item/42",
    "render": true,
    "wait_for": "h1.title",
    "fields": {"title": "h1.title", "price": ".price"}
  }'

AI extraction (describe fields in plain English)

Don't want to write or maintain selectors? Send a prompt instead of fields: we fetch the page (all the params above still apply), convert it to Markdown, and a language model returns exactly the fields you described as JSON. Selectors break when a site's markup changes; a prompt adapts. Add an optional schema (a JSON object of key→type) to pin the exact output shape. Pass model to pick a strength tier. "fast" is the default: a median 1.5s, and the right choice for almost every extraction, since pulling stated fields off a page is a reading task. "smart" takes a median of about 10 s and is better when the answer has to be derived from the page rather than read off it, combining a start time, a duration and a timezone, say. "max" runs our strongest model (a median of about 3 s) and is for extractions where every field must be right. Each tier bills from a minimum on top of the fetch unit (fast 2, smart 4, max 25 request-units), and a very large page or a long answer bills more, in proportion to what the model call actually cost. An unrecognised name is a 400 listing the valid ones. If the tier you asked for is unavailable we answer on the other one rather than failing, bill the tier that actually served, and tell you by returning requested_model alongside model.

Asking a question instead of listing fields? A prompt like "what's this business about?" is a question, not a field list, but the model still has to invent key names for it (business, category, competitors_mentioned…) and you have to guess what it picked. Send "structured": false and you get a plain-language reply under answer instead, with no data key at all:

curl https://scrape.land/v1/extract \
  -H "X-Api-Key: pb_live_YOURKEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://scrape.land/",
    "prompt": "what's this business about?",
    "structured": false
  }'

{
  "url": "https://scrape.land/",
  "status": 200,
  "extracted_by": "ai",
  "model": "fast",
  "answer": "scrape.land is a web data-extraction API: you send a URL and it returns clean structured data, handling proxy rotation, retries, geo-targeting and JavaScript rendering for you. Pricing is flat per 1,000 delivered requests."
}

data and answer are never both present, so a typed client branches on which one it got rather than type-switching a single field. structured defaults to true, so nothing you already send changes. It costs exactly the same request-units as structured extraction: it is the same page and the same model call. A schema or an extract_type pins a JSON shape and therefore implies structured output; sending either together with "structured": false is a 400 naming both fields rather than us silently guessing which you meant. It has no effect on selector-based fields, which never reach a model.

Need a list? Ask for "every product on the page" and you get the array back under data.items. If the page has no text (a JS-only page fetched without render), you get a 422 telling you to retry with "render": true rather than a response full of nulls.

curl https://scrape.land/v1/extract \
  -H "X-Api-Key: pb_live_YOURKEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://books.toscrape.com/",
    "prompt": "the page title, and the title and price (as a number) of the first book"
  }'

{
  "url": "https://books.toscrape.com/",
  "status": 200,
  "extracted_by": "ai",
  "model": "fast",
  "data": {
    "page_title": "Books to Scrape",
    "first_book_title": "A Light in the Attic",
    "first_book_price": 51.77
  }
}

Auto-extract (one word instead of a schema)

Auto-extract is AI extraction under the hood, so it needs the same Scale or above plan.

For common page types you don't even need a prompt. Pass extract_type and we apply a curated prompt + schema so you get a normalized object back: product, article, job, discussion, event, recipe, real_estate, and profile. Your own prompt/schema still win if you also send them.

Presets run on the default fast tier and typically answer in a couple of seconds. Pass "model": "smart" if a page is genuinely ambiguous. It takes about twice as long and costs more request-units. Synchronous requests get a 150s budget; past that you get a 504, so submit a slow one as an async job (POST /v1/jobs, 5-minute budget).

curl https://scrape.land/v1/extract \
  -H "X-Api-Key: pb_live_YOURKEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "extract_type": "product",
    "model": "fast"
  }'

Getting the response headers

Sometimes you need the response headers, not just the body: a redirect Location, a Set-Cookie, a Content-Type. Add "headers": true and we include a headers object alongside data. Header names come in canonical form (Content-Type, Set-Cookie), and a repeated header is joined with ", ". The object also carries two headers of ours: X-Proxy-Country, the exit country the request left from, and X-Proxy-Tier, the exit tier that carried it (premium or pool). Neither is a header on the API's own HTTP response.

{
  "url": "https://example-product.com/item/42",
  "status": 200,
  "data": {"title": "Vintage desk lamp"},
  "headers": {
    "Content-Type": "text/html; charset=utf-8",
    "Set-Cookie": "sid=abc123; Path=/; HttpOnly",
    "Cache-Control": "max-age=300",
    "X-Proxy-Country": "US",
    "X-Proxy-Tier": "premium"
  }
}

More endpoints

curl https://scrape.land/v1/fetch \
  -H "X-Api-Key: pb_live_YOURKEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example-product.com/item/42", "format": "html"}'

{
  "url": "https://example-product.com/item/42",
  "status": 200,
  "html": "<!doctype html><html>…"
}

When the page that came back is an anti-bot challenge or refusal (for example a Cloudflare "Just a moment" page) rather than the content, the response also carries "blocked": true and the header X-Scrapeland-Blocked: 1, even if the target answered 200. Retry it: blocked responses are never billed.

LLM-ready Markdown

"format": "markdown" returns the page as clean Markdown under a markdown key, the shape RAG pipelines and LLM context windows actually want, so you don't ship raw HTML tags to a model and pay for them as tokens. Site chrome (nav, header, footer, aside, scripts and styles) is stripped, headings, lists, links, code blocks and tables are kept, and relative links and images are resolved to absolute URLs so a chunk still works once it's separated from its source page. It composes with every other option (country, session, render, wait_for, actions, fingerprint) and it works in /v1/batch too. It costs exactly one request-unit, the same as any other fetch; there is no Markdown or AI surcharge.

curl https://scrape.land/v1/fetch \
  -H "X-Api-Key: pb_live_YOURKEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/blog/post", "format": "markdown", "render": true}'
# -> {"url":"...","status":200,"markdown":"# Post title\n\nBody text..."}

curl https://scrape.land/v1/batch \
  -H "X-Api-Key: pb_live_YOURKEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://a.example/", "https://b.example/"], "format": "text"}'

{
  "results": [
    {"url": "https://a.example/", "status": 200, "text": "…"},
    {"url": "https://b.example/", "error": "…"}
  ]
}

curl https://scrape.land/v1/search \
  -H "X-Api-Key: pb_live_YOURKEY" \
  -H "Content-Type: application/json" \
  -d '{"q": "best web scraping api", "count": 5}'

{
  "query": "best web scraping api",
  "count": 2,
  "results": [
    {"title": "…", "url": "https://…", "snippet": "…"},
    {"title": "…", "url": "https://…", "snippet": "…"}
  ]
}

curl https://scrape.land/v1/rank \
  -H "X-Api-Key: pb_live_YOURKEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/blog", "query": "in-depth technical tutorials", "top_k": 10}'

{
  "url": "https://example.com/blog",
  "query": "in-depth technical tutorials",
  "status": 200,
  "total_links": 84,
  "ranked": [
    {"url": "https://example.com/blog/go-generics",
     "text": "A deep dive into Go generics",
     "score": 0.93, "reason": "long-form technical tutorial"}
  ],
  "ranked_count": 10,
  "extracted_by": "ai",
  "model": "fast"
}

curl https://scrape.land/v1/jobs \
  -H "X-Api-Key: pb_live_YOURKEY" -H "Content-Type: application/json" \
  -d '{"type":"batch","urls":["https://a.example","https://b.example"],
       "fields":{"title":"h1"},"webhook_url":"https://your.app/hook"}'
# -> {"id":"job_ab12","status":"queued"}

# poll it:
curl https://scrape.land/v1/jobs/job_ab12 -H "X-Api-Key: pb_live_YOURKEY"
# -> {"id":"job_ab12","type":"batch","status":"done","result":{}}

# list recent jobs (newest first, no result blobs):
curl "https://scrape.land/v1/jobs?limit=20" -H "X-Api-Key: pb_live_YOURKEY"

When a job finishes we POST {"id","status","result"} to your webhook_url, which must be a public https URL (internal and loopback addresses are rejected). Jobs run for up to a few minutes; each is metered exactly like the synchronous call.

Verifying the webhook. Every delivery carries an X-Scrapeland-Signature header of the form t=<unix>,v1=<hex>, where the hex is HMAC-SHA256("<t>.<raw body>") keyed with your webhook secret. Sign the RAW body bytes (not a re-serialised copy), compare in constant time, and reject anything whose t is more than a few minutes old: that is what stops a captured delivery being replayed at you later. Ask support for the secret if you use webhook_url.

import hmac, hashlib, time

def verify(raw_body: bytes, header: str, secret: str, tolerance=300) -> bool:
    parts = dict(p.split("=", 1) for p in header.split(",") if "=" in p)
    ts, sig = parts.get("t", ""), parts.get("v1", "")
    if not ts or not sig or abs(time.time() - int(ts)) > tolerance:
        return False
    expected = hmac.new(secret.encode(),
                        ts.encode() + b"." + raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, sig)

curl https://scrape.land/v1/places \
  -H "X-Api-Key: pb_live_YOURKEY" \
  -H "Content-Type: application/json" \
  -d '{"category": "dentist", "location": "Austin, TX", "website": "yes"}'

{
  "attribution": "© OpenStreetMap contributors, ODbL (openstreetmap.org/copyright)",
  "bbox": [30.0985133, -97.9367663, 30.5166255, -97.5605288],
  "category": "dentist",
  "count": 10,
  "matched": 79,
  "preview": true,
  "preview_note": "Free plan: the first 10 matches. Every paid plan returns up to 200 per call.",
  "results": [
    {"name": "Aloha Dental", "category": "dentist",
     "address": "1614 East Riverside Drive, 78741", "city": "Austin",
     "lat": 30.245304, "lon": -97.730394,
     "website": "https://www.aloha-dental.com/", "phone": "+1-512-707-7300",
     "email": "riverside@aloha-dental.com",
     "opening_hours": "Mo 08:00-17:00; Tu 09:00-18:00; We 08:00-17:00; Th 09:00-18:00; Fr 08:00-17:00",
     "osm_url": "https://www.openstreetmap.org/node/3856445942"},
    {"name": "Austin Dental", "category": "dentist",
     "address": "2206 West Parmer Ln, 78727", "city": "Austin",
     "lat": 30.417672, "lon": -97.698822,
     "website": "https://www.lifetimedental.com/", "phone": "+1-512-982-4006",
     "opening_hours": "Mo-Fr 07:30-17:00",
     "osm_url": "https://www.openstreetmap.org/way/383188243"}
  ],
  "units": 1
}

{
  "formats": ["html", "text", "markdown", "raw"],
  "render": {"enabled": true, "screenshot": true, "actions": true,
             "device": true, "block_resources": true, "fingerprint": true},
  "ai": {"enabled": true, "configured": true, "min_plan": "scale",
         "plan_scoped": true, "models": ["fast", "smart", "max"], "default": "fast"},
  "extract_types": ["product", "article", "job", "discussion",
                    "event", "recipe", "real_estate", "profile"],
  "endpoints": ["/v1/fetch", "/v1/extract", "/v1/rank", "/v1/batch",
                "/v1/search", "/v1/jobs", "/v1/places", "/v1/account",
                "/v1/capabilities", "/mcp"],
  "limits": {"batch_max_urls": 20, "links_max": 2000,
             "search_max_count": 10, "raw_max_bytes": 7340032,
             "max_response_bytes": 5242880,
             "max_response_bytes_scope": "key"}
}

{
  "plan": "scale",
  "included_requests_remaining": 2871400,
  "prepaid_credit_cents": 0,
  "rate_limit_rps": 200,
  "key_budget": {"max_requests": 0, "requests_used": 1234, "unlimited": true}
}

MCP server (Claude, Cursor and other AI clients)

scrape.land is also a remote Model Context Protocol server, so an AI assistant can fetch pages, extract data, search the web and find local businesses as tools. Endpoint: https://scrape.land/mcp (Streamable HTTP, JSON-RPC over POST). Authenticate with your API key as Authorization: Bearer YOUR_KEY. Each tool call runs the matching endpoint on your account, so it is billed, rate limited and plan gated exactly like the API.

Add it to your AI tool. Replace YOUR_KEY with an API key from your dashboard; the one-click buttons install it with that placeholder, so open the server's settings afterwards and paste your key.

Add to Cursor Add to Kiro

Claude (claude.ai, Desktop, mobile): Settings, Connectors, Add custom connector. Name it scrape.land, set the URL to https://scrape.land/mcp, and add the request header Authorization with the value Bearer YOUR_KEY.

Claude Code:

claude mcp add --transport http scrape-land https://scrape.land/mcp --header "Authorization: Bearer YOUR_KEY"

Cursor, Claude Desktop config, Cline and other clients that take an mcpServers JSON config:

{
  "mcpServers": {
    "scrape-land": {
      "type": "http",
      "url": "https://scrape.land/mcp",
      "headers": { "Authorization": "Bearer YOUR_KEY" }
    }
  }
}

VS Code (GitHub Copilot), in .vscode/mcp.json; VS Code asks for the key once and stores it securely:

{
  "inputs": [
    { "type": "promptString", "id": "scrapeland-key", "description": "scrape.land API key", "password": true }
  ],
  "servers": {
    "scrape-land": {
      "type": "http",
      "url": "https://scrape.land/mcp",
      "headers": { "Authorization": "Bearer ${input:scrapeland-key}" }
    }
  }
}

Gemini CLI, in ~/.gemini/settings.json:

{
  "mcpServers": {
    "scrape-land": {
      "httpUrl": "https://scrape.land/mcp",
      "headers": { "Authorization": "Bearer YOUR_KEY" }
    }
  }
}

Windsurf, in mcp_config.json:

{
  "mcpServers": {
    "scrape-land": {
      "serverUrl": "https://scrape.land/mcp",
      "headers": { "Authorization": "Bearer YOUR_KEY" }
    }
  }
}

Zed, in settings.json:

{
  "context_servers": {
    "scrape-land": {
      "url": "https://scrape.land/mcp",
      "headers": { "Authorization": "Bearer YOUR_KEY" }
    }
  }
}

Your own agent, OpenAI Agents SDK (Python):

from agents import Agent, Runner
from agents.mcp import MCPServerStreamableHttp

async with MCPServerStreamableHttp(params={
    "url": "https://scrape.land/mcp",
    "headers": {"Authorization": "Bearer YOUR_KEY"},
}) as scrapeland:
    agent = Agent(name="researcher", mcp_servers=[scrapeland])
    result = await Runner.run(agent, "Find three dentists in Austin without a website")
    print(result.final_output)

Your own agent, Claude API (MCP connector, beta header anthropic-beta: mcp-client-2025-11-20):

{
  "model": "claude-sonnet-5",
  "max_tokens": 2048,
  "messages": [{ "role": "user", "content": "Summarise https://example.com" }],
  "mcp_servers": [{
    "type": "url",
    "url": "https://scrape.land/mcp",
    "name": "scrape-land",
    "authorization_token": "YOUR_KEY"
  }],
  "tools": [{ "type": "mcp_toolset", "mcp_server_name": "scrape-land" }]
}

Python SDK & migrating from Zyte

Our Python client wraps both the extraction API and the raw tunnel, and ships a drop-in adapter for the Zyte API client. Already on Zyte? Change the import and your key (the query and response shapes match).

pip install scrapeland

from scrapeland import ScrapelandClient

client = ScrapelandClient("pb_live_YOURKEY")          # or $SCRAPELAND_API_KEY

# structured extraction:
res = client.extract("https://example-product.com/item/42", {"title": "h1"})

# raw tunnel fetch through a fresh exit IP:
r = client.get("https://api.ipify.org", country="us", session="job42")

# or hand the tunnel mapping to your existing requests/httpx code:
import requests
requests.get("https://api.ipify.org", proxies=client.proxies(country="de"))

Raw tunnel (advanced)

Most people want structured data and should use the extraction API above. If instead you need raw HTTP access to the page yourself, the raw tunnel is a forward proxy: point any HTTP client at it with your API key as the username, and every request exits from a fresh IP, with optional country, session, and protocol controls. Host: gateway.scrape.land:8080.

curl -x http://pb_live_YOURKEY:@gateway.scrape.land:8080 https://api.ipify.org
# -> an exit IP, different on each request

Control parameters

Append -name-value pairs to the key in the username, or send X-Proxy-* control headers. You always connect to the gateway as an HTTP proxy (http:// scheme); protocol picks which kind of exit proxy carries the request onward: http, https or socks5.

Tunnel URL builder

Pick your options and copy the ready-to-run command. Replace YOURKEY with a key from your dashboard.

Using the tunnel from code

import requests

KEY = "pb_live_YOURKEY"
proxy = f"http://{KEY}-country-us:@gateway.scrape.land:8080"
r = requests.get("https://api.ipify.org?format=json",
                 proxies={"http": proxy, "https": proxy}, timeout=30)
print(r.json())   # {'ip': '...'}  a US exit IP

import { ProxyAgent, fetch } from "undici";   // npm i undici

const KEY = "pb_live_YOURKEY";
// The key is the proxy username and the password is empty. undici only builds
// Proxy-Authorization from a URL that has both, so pass it as a token.
const dispatcher = new ProxyAgent({
  uri: "http://gateway.scrape.land:8080",
  token: "Basic " + Buffer.from(`${KEY}-country-gb:`).toString("base64"),
});
// Node's fetch ignores an `agent` option: the tunnel goes in `dispatcher`.
const res = await fetch("https://api.ipify.org?format=json", { dispatcher });
console.log(await res.json());   // { ip: '...' }  a GB exit IP

p, _ := url.Parse("http://pb_live_YOURKEY-country-us:@gateway.scrape.land:8080")
client := &http.Client{Transport: &http.Transport{Proxy: http.ProxyURL(p)}}
resp, _ := client.Get("https://api.ipify.org")

import { chromium } from "playwright";

const browser = await chromium.launch({
  proxy: {
    server: "http://gateway.scrape.land:8080",
    username: "pb_live_YOURKEY-country-us",
    password: "",
  },
});
const page = await browser.newPage();
await page.goto("https://api.ipify.org");

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapeland.scrapy.ScrapelandMiddleware": 740,
}
SCRAPELAND_API_KEY = "pb_live_YOURKEY"
SCRAPELAND_COUNTRY = "us"            # optional default for every request

# per-request override:
yield scrapy.Request(url, meta={"scrapeland": {"country": "de", "session": "job42"}})

Errors

{
  "error": "AI extraction (prompt, schema, extract_type) is not included on the \"free\" plan (it needs \"scale\" or above). Upgrade on the Plans page of your dashboard, or use CSS/XPath \"fields\" extraction, which every plan includes.",
  "code": "plan-upgrade-required",
  "plan": "free",
  "required_plan": "scale",
  "fallback": "use CSS/XPath \"fields\" extraction, which every plan includes",
  "capability": "AI extraction (prompt, schema, extract_type)"
}

Target-site statuses (the site's own 200/403/404) are passed through unchanged. The exit country is not a header on the API's own response: send "headers": true and it is X-Proxy-Country inside the headers object. On the raw tunnel, X-Proxy-Country is on the CONNECT response for https:// targets and on the response itself for http:// targets.

Plan limits

Four things change with your plan: how fast you may send requests, how large a single response may be, whether AI extraction is available, and whether you get priority exits. They apply identically to the extraction API and the raw tunnel. GET /v1/capabilities with your key returns your own entitlements under entitlements, and GET /v1/account your own numbers.

Pool is $19/month ($190/year) and unmetered: there is no monthly request count, and its 10 req/s rate limit is the only ceiling. It runs on non-priority exits, does not include AI extraction or browser rendering, and does include the lead generation tools (POST /v1/places).

Rate limit. Sustained requests per second, enforced per account across every key you own (a per-key limit would be no limit at all: keys are free to mint); an individual key can be throttled below it. Short bursts above it are tolerated. Over it you get a 429 with a Retry-After header: back off and continue, nothing is lost and nothing is billed. Each plan's rate is sized so a flat-out run against your whole monthly allowance lasts hours rather than minutes.

Max response. The largest single response we will return, exposed as limits.max_response_bytes in GET /v1/capabilities. Real pages are nowhere near it (the median response we serve is about 40 KB and 99% are under 1.2 MB) so it only catches a video, an archive or a dataset dump. Anything larger is refused whole with a 413 that names the size, the limit and your plan. We never return a truncated page, and a refused response is not billed.

Billing never exceeds what you've paid. Included requests are spent first. Past them, requests draw on prepaid credit at $0.20 / 1,000: the balance counts down and floors at zero, at which point requests return 402 instead of continuing. There is no invoice after the fact and no way to run up a charge you did not authorise in advance.

Guides

Rotating vs sticky sessions

By default every request exits from a fresh IP (great for spreading load and avoiding rate limits). When you need the same IP across requests (logins, carts, multi-step flows) add a session param (or -session-NAME on the tunnel); we pin that session to one exit IP for about 10 minutes, then it rotates. Use a unique session name per logical user.

Handling rate limits & errors

• Target returns 429/403: retry. A new request rotates to a fresh IP. Add a small backoff.
• 402 from us: your quota/balance is exhausted. Upgrade or top up.
• 401/407 from us: the API key is missing or wrong (header for the API, username for the tunnel).
• 502 from us: no upstream matched your filters. Loosen country/maxlatency and retry.

import requests, time

KEY = "pb_live_YOURKEY"
def get(url, tries=4):
    for i in range(tries):
        proxy = f"http://{KEY}:@gateway.scrape.land:8080"   # fresh IP each try
        try:
            r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=30)
            if r.status_code < 400:
                return r
        except requests.RequestException:
            pass
        time.sleep(2 ** i)        # exponential backoff
    raise RuntimeError("all retries failed")

FAQ

Is there a free tier?

Yes (1,000 requests, no card required). The cheapest paid plan is Pool at $19/mo (unmetered at 10 req/s, without AI extraction or rendering). Metered plans start at $99/mo (600k requests = $0.165 / 1k) and the rate improves as the tier grows: $0.139 / 1k on Business ($999 / 7.2M) and $0.100 / 1k on Business Ultra ($9,999 / 100M). See Plan limits for the full ladder.

How am I billed?

Per 1,000 delivered requests. If a proxy is blocked or returns a 403/429, we retry another IP and never bill for it. You only pay once the page comes back.

Your plan's included requests are spent first. Past them, requests come out of prepaid credit at $0.20 / 1,000: you top up in advance, the balance counts down, and it floors at zero. There is no invoice after the fact and no way to run up a bill you didn't authorise. Without a subscription, prepaid credit costs the same $0.20 / 1,000. Pay-as-you-go as a plan of its own is coming in 2027; it is not sold today.

Can I target a specific country?

Yes. On the extraction API add "country": "us" to the request body. On the raw tunnel add -country-us (any ISO code) to your key. See the tunnel URL builder.

How do I scrape pages that need JavaScript?

Add "render": true to your extract request, and "wait_for" with a CSS selector if content loads late. See render & wait_for.

Do you offer residential IPs?

Today the pool is rotating datacenter/ISP-grade IPs, self-validated continuously. Residential/mobile pools are on the roadmap (premium plans).

Is this legal?

These are standard tools; you're responsible for using them lawfully and respecting destination sites' terms. See our Terms.