How to use scrape.land in Puppeteer and Playwright

Tunneling and IPs14 September 20265 min read

Headless Chrome is often the part of a scraper that gets blocked first, because every page load comes from your own server's IP. Both Puppeteer and Playwright can send their traffic through the scrape.land tunnel instead: Puppeteer with the --proxy-server launch flag and page.authenticate(), Playwright with its proxy launch option. This guide shows both, run for real, and ends with when you should skip your own browser and let the API render the page.

Puppeteer

Chrome takes the tunnel address as a launch flag, but it cannot read credentials from that flag. Puppeteer answers the gateway's login challenge with page.authenticate(): your API key, with any options, is the username, and the password is an empty string.

Node.js

import puppeteer from "puppeteer";

const KEY = process.env.SCRAPELAND_KEY;
const SKIP = new Set(["image", "font", "media"]);

const browser = await puppeteer.launch({
  args: ["--proxy-server=http://gateway.scrape.land:8080"],
});
const page = await browser.newPage();
await page.authenticate({ username: `${KEY}-country-us`, password: "" });

await page.setRequestInterception(true);
page.on("request", (req) => (SKIP.has(req.resourceType()) ? req.abort() : req.continue()));

await page.goto("https://api.ipify.org?format=json");
console.log("exit IP:", await page.evaluate(() => document.body.innerText));

await page.goto("https://quotes.toscrape.com/js/", { waitUntil: "networkidle2" });
const quotes = await page.$$eval(".quote .text", (els) => els.map((e) => e.textContent));
console.log(quotes.length, "quotes; first:", quotes[0]);

await browser.close();

Output from a real run (exit address masked). The second page, quotes.toscrape.com/js/, builds its quotes with JavaScript, so this also shows the browser running scripts as usual behind the tunnel:

shell

exit IP: {"ip":"107.150.x.x"}
10 quotes; first: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”

Call page.authenticate() before the first goto. If you forget it, the gateway answers 407 and the navigation fails. Keep http:// in the flag even though the sites are HTTPS: it is how Chrome talks to the gateway. HTTPS pages are tunnelled end to end, so there are no certificate warnings to switch off.

Playwright

Playwright accepts the server, username and password together in its proxy launch option, so there is no separate login step. This example also adds a sticky session, so every request from this browser leaves from one IP:

Node.js

import { chromium } from "playwright";

const KEY = process.env.SCRAPELAND_KEY;

const browser = await chromium.launch({
  proxy: {
    server: "http://gateway.scrape.land:8080",
    username: `${KEY}-country-us-session-run1`,
    password: "",
  },
});
const page = await browser.newPage();
await page.route("**/*", (route) =>
  ["image", "font", "media"].includes(route.request().resourceType())
    ? route.abort()
    : route.continue()
);

for (let i = 0; i < 2; i++) {
  await page.goto("https://api.ipify.org?format=json");
  console.log("exit IP:", await page.textContent("body"));
}

await page.goto("https://quotes.toscrape.com/js/");
const quotes = await page.locator(".quote .text").allTextContents();
console.log(quotes.length, "quotes; first:", quotes[0]);

await browser.close();

shell

exit IP: {"ip":"159.89.x.x"}
exit IP: {"ip":"159.89.x.x"}
10 quotes; first: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”

The same code works with firefox or webkit in place of chromium, and Playwright for Python takes the same proxy dictionary in launch().

Rotation inside a browser

Without a session, every new connection through the gateway gets a fresh exit IP. A browser opens several connections for one page load (the page, its scripts, its API calls, often on different hosts), so a single page view can arrive at the site from several addresses. Most sites do not care. Sites that tie a login, a cart or an anti-bot cookie to your IP do, and for those add -session-NAME to the username as in the Playwright example. A session holds its IP for about 10 minutes.

To switch to a new IP, launch a new browser with a new session name. One browser per session also keeps cookies from leaking between the identities you are running.

Keep the request count down

The tunnel bills delivered requests, and a browser makes far more of them than a script does. Both examples above abort images, fonts and media, which your code almost never reads. That cuts the number of connections a page opens and the data it pulls through the tunnel. Leave stylesheets and scripts alone: the page may need them to build its content.

Browsers cannot set a per-request tunnel from inside a web page, so run this from a backend or a script, not from front-end JavaScript.

Your own browser or the API's render mode?

The extraction API can run the browser for you. Add "render": true and it loads the page in a real browser on our side, waits for a selector if you ask, and returns your fields as JSON:

curl

curl https://scrape.land/v1/extract \
  -H "X-Api-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://quotes.toscrape.com/js/",
       "render": true,
       "wait_for": ".quote",
       "block_resources": true,
       "fields": {"quotes": {"css": ".quote", "fields": {"text": ".text", "author": ".author"}}}}'

The response has the same shape as any extraction: url, status and a data object holding the quotes list. Render mode also takes actions (scroll, click, wait, wait_for, fill) for pages that need a few steps first.

Use render mode when you want data out of a page and would rather not run, update and scale Chrome yourself. One HTTP call per page, from any language, and a render that fails is not billed.
Use your own browser through the tunnel when the flow is long or stateful (logging in with your own account, stepping through a multi-page form, reacting to what the page shows), when you already have a Puppeteer or Playwright test suite, or when you need full control of the page's scripts and events.

Render mode is not included on the Pool plan; the tunnel is.

What it costs

Through the tunnel, each delivered request is 1 request unit, and failed attempts are not billed. A render on the API is 5 units, 1 with block_resources, and 10 with a screenshot. The Free plan includes 1,000 requests a month. See pricing.

Next steps

All tunnel options are in the docs, and render mode is covered in scraping JavaScript-heavy pages. For scripted steps inside the API's browser, see filling a form and clicking before scraping. Sign up free to get a key.

Start free with 1,000 requests Read the docs