How to route Python requests through scrape.land

If your scraper is built on Python's requests library, you do not need to rewrite it to get rotating exit IPs. Pass the scrape.land gateway in the proxies argument and every request leaves from a fresh address, from the country you pick, or from the same address for as long as you need one. This guide covers the URL format, the documented options, how to check which IP you actually got, and what the errors look like.

The gateway URL

The gateway is a forward tunnel at gateway.scrape.land:8080. Your API key is the username and the password is empty, so the key is followed by a bare colon:

import requests

KEY = "YOUR_KEY"
tunnel = f"http://{KEY}:@gateway.scrape.land:8080"
proxies = {"http": tunnel, "https": tunnel}

r = requests.get("https://api.ipify.org?format=json", proxies=proxies, timeout=30)
print(r.json())

Set both the http and https keys, or requests to the other scheme go out directly from your own machine. Keep http:// at the start of the gateway URL even for HTTPS sites: it describes how requests talks to the gateway, not the site. For an HTTPS page, requests opens an encrypted connection to the site through the tunnel, so the page content stays between you and the site and no certificate settings change.

If your network filters port 8080, as many office, university and hotel networks do, use gateway.scrape.land:443 instead. Behaviour and billing are identical.

Options go in the username

Options are -name-value pairs appended to the key. These are the ones the docs list:

They combine in any order: YOUR_KEY-country-de-session-cart-42 is a German exit that stays fixed for the cart-42 session. A session name may contain hyphens, so a UUID works. The docs also list header equivalents (X-Proxy-Country and friends), but with requests those only reach the gateway on plain http:// URLs: on HTTPS your headers travel inside the encrypted connection to the site. Use the username form and it works for both.

A small helper, and checking the exit IP

This script builds the gateway URL from keyword arguments, retries a failed attempt on a fresh exit, and then checks three things: that the IP rotates, that the country is right, and that a session holds one IP.

import os
import time
import requests

KEY = os.environ["SCRAPELAND_KEY"]
GATEWAY = "gateway.scrape.land:8080"


def tunnel(**opts):
    """Proxy URL: your key, then -name-value options, then an empty password."""
    user = KEY + "".join(f"-{k}-{v}" for k, v in opts.items())
    url = f"http://{user}:@{GATEWAY}"
    return {"http": url, "https": url}


def get(url, tries=4, **opts):
    """GET through the tunnel; a failed attempt is retried on a fresh exit."""
    for i in range(tries):
        try:
            return requests.get(url, proxies=tunnel(**opts), timeout=30)
        except (requests.exceptions.ProxyError, requests.exceptions.SSLError,
                requests.exceptions.ConnectionError, requests.exceptions.Timeout):
            time.sleep(2 ** i)
    raise RuntimeError(f"{url}: all {tries} attempts failed")


# 1. rotating: a new exit IP per request
for _ in range(3):
    print("rotating:", get("https://api.ipify.org?format=json").json()["ip"])

# 2. country: ask a geo-IP service where the exit really is
r = get("http://ip-api.com/json/?fields=countryCode,query", country="de")
print("country:", r.json(), "X-Proxy-Country:", r.headers.get("X-Proxy-Country"))

# 3. sticky: same session name, same exit IP
for _ in range(3):
    print("sticky:", get("https://api.ipify.org?format=json", session="cart-42").json()["ip"])

Output from a real run, with the last two parts of each address masked:

rotating: 146.190.x.x
rotating: 85.8.x.x
rotating: 85.8.x.x
country: {'countryCode': 'DE', 'query': '103.237.x.x'} X-Proxy-Country: DE
sticky: 97.74.x.x
sticky: 97.74.x.x
sticky: 97.74.x.x

The rotating lines show three different exits (the second and third share a network but not an address). The country check asks an independent geo-IP service, which is the honest test: it reports where the request arrived from, not what we say. Because that check uses a plain http:// URL, the gateway can also add its X-Proxy-Country response header naming the exit country it used; on HTTPS the response comes straight from the site, so that header is not there. The sticky lines are one address three times.

A sticky session holds its exit IP for about 10 minutes, then rotates. Use one session name per logical user or flow (one per login, one per checkout) and pick a new name when you want a new IP. Our sticky sessions guide goes into when that matters.

Environment variables and sessions

Requests also reads HTTP_PROXY and HTTPS_PROXY from the environment, which is handy for a script you cannot edit:

export HTTPS_PROXY="http://YOUR_KEY-country-us:@gateway.scrape.land:8080"
export HTTP_PROXY="$HTTPS_PROXY"
python your_script.py

One catch if you use a requests.Session: when those variables are set, they win over session.proxies. Passing proxies= on each call, as the helper above does, always wins, so it is the safer habit. A Session is still worth using for its cookies and connection reuse.

What the errors look like

• A wrong or missing key is a 407. On an HTTPS URL, requests raises ProxyError with Tunnel connection failed: 407 Proxy Authentication Required in the message. On a plain HTTP URL you get a response with status 407 and the body scrape.land: authenticate with your API key as the proxy username.
• 429 means you went over your plan's rate limit, which the tunnel shares with the extraction API. Back off and continue; nothing over the limit is billed.
• 402 means your quota or prepaid balance is used up. Stop and top up rather than retrying.
• 502 means no exit matched your options. Loosen country or maxlatency and try again.
• An SSL or connection error mid-request is an exit that dropped the connection. Retry: the next attempt goes out from a different IP.

On HTTPS these refusals arrive as the tunnel failing to open, so requests shows them as a ProxyError with the status in the message. On plain HTTP the gateway answers in plain text and names the reason in an X-Scrapeland-Error response header, using the same codes as the API (rate-limited, quota-exhausted and so on).

If the target site itself answers 403 or 429, that is the site blocking an exit, not us. Retry after a short pause: the new request rotates to a fresh IP.

What it costs

Each delivered request through the tunnel is 1 request unit, the same as a plain fetch on the API; failed attempts are not billed. Raw tunneling is included on every plan, including Free (1,000 requests a month) and the unmetered Pool plan. See pricing.

Next steps

The tunnel options and a URL builder are in the docs. If you would rather get parsed fields than raw HTML, see rotating IPs and picking a country for when to switch to the extraction API. Create a free account to get a key.