How to compare prices across countries

E-commerce and prices10 September 20265 min read

Many online shops charge different prices depending on where the visitor is: a different currency, a different number, sometimes a product that is simply not sold in one market. To compare prices across countries you need to load the same product page as a shopper in each of those countries would. This guide does it with the country field, turns each price into a currency code and a number, and explains how to read the result.

How country targeting works

Add "country" to any extraction request with a two-letter ISO code, lower case: "us", "gb", "de", "jp" and so on. The request then leaves from an exit IP in that country, so the shop sees a visitor from there. Add "headers": true and the response includes a headers object; in it, X-Proxy-Country names the exit country that was actually used, which is how you confirm the targeting worked.

curl

curl https://scrape.land/v1/extract \
  -H "X-Api-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
       "country": "de",
       "headers": true,
       "fields": {"price": ".product_main .price_color",
                  "stock": ".product_main .availability"}}'

The real response, with the target's other headers trimmed:

JSON

{
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "status": 200,
  "data": {
    "price": "£51.77",
    "stock": "In stock (22 available)"
  },
  "headers": {
    "Content-Type": "text/html",
    "X-Proxy-Country": "DE",
    …
  }
}

Turning local price text into numbers

Before prices from different countries can be compared, each one needs a currency and a number. Shops format both differently by market: $1,049.00 in the US, 1.049,00 € in Germany, CHF 1'049.00 in Switzerland, ¥ 12,800 in Japan. The function below handles those cases. Its one rule: a final dot or comma followed by one or two digits is the decimal mark, and every other dot or comma is a thousands separator.

Python

import re
import requests

SYMBOLS = {"£": "GBP", "€": "EUR", "¥": "JPY", "$": "USD"}  # "$" is also CAD, AUD, MXN...

def parse_price(text):
    """'1.049,00 €' -> ('EUR', 1049.0). Returns (None, None) for empty text."""
    if not text:
        return None, None
    code = re.search(r"\b[A-Z]{3}\b", text)
    currency = code.group() if code else next((c for s, c in SYMBOLS.items() if s in text), None)
    num = re.sub(r"[^\d.,]", "", text)
    m = re.fullmatch(r"(.*?)[.,](\d{1,2})", num)  # a final . or , with 1-2 digits is the decimal mark
    whole, frac = m.groups() if m else (num, "0")
    whole = re.sub(r"[.,]", "", whole)
    return currency, float(f"{whole or 0}.{frac}")

URL = "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
FIELDS = {"price": ".product_main .price_color", "stock": ".product_main .availability"}

for country in ["us", "gb", "de"]:
    r = requests.post(
        "https://scrape.land/v1/extract",
        headers={"X-Api-Key": "YOUR_KEY"},
        json={"url": URL, "country": country, "headers": True, "fields": FIELDS},
        timeout=90,
    )
    if r.status_code == 502:
        print(country, "no exit available right now, try again later")
        continue
    r.raise_for_status()
    body = r.json()
    hdrs = {k.lower(): v for k, v in body.get("headers", {}).items()}
    currency, amount = parse_price(body["data"]["price"])
    print(country, hdrs.get("x-proxy-country"), currency, amount, body["data"]["stock"], sep="\t")

We ran it against the practice shop. Every exit country matched the request, and every market saw the same thing:

shell

us	US	GBP	51.77	In stock (22 available)
gb	GB	GBP	51.77	In stock (22 available)
de	DE	GBP	51.77	In stock (22 available)

That is the correct result for this site. Books to Scrape is a static sandbox that serves one page to everyone, so it makes a good control: it proves the requests left from three countries, and that any difference you see on a real shop comes from the shop, not from your script. Keep one static page like this in your test set.

What varies by country, and what does not

On a real shop, loading the same product from several countries can change:

Currency. The most common difference. Some shops convert at display time; others set a separate price per market.
The number itself. Beyond conversion, prices often include local VAT or sales tax, regional promotions, or deliberate market pricing.
Availability. "In stock" in one country, "not available in your region" in another, or a redirect to a different regional store.
The page you land on. Many shops redirect by location to /de/, /uk/ or a country domain. Check url and status in each response, and compare the product title as well as the price so you know it is the same item.

Results depend entirely on the site. Some shops pick the currency from a cookie or an account setting rather than the visitor's IP, and show the same price everywhere to a new visitor. Some read the browser language. If the IP alone does not change what you see, send an Accept-Language header with send_headers (see custom headers and cookies), or render the page with "render": true and "fingerprint": true, which gives the browser a timezone, locale and language that match the country.

Making the comparison fair

Fix the list of countries and keep it the same on every run, so your history compares like with like.
Store the original text next to the parsed number. When a shop changes its format, the raw text shows you why a parse failed.
Convert to one currency last, with exchange rates from a source you trust and the date you used. The shop's own prices are the data; the conversion is your analysis.
Expect the odd failure. If no exit in a country is available at that moment, the API returns a 502 and you are not billed. Retry that country later rather than dropping it from the comparison.

What it costs

Choosing a country adds nothing: each request above is 1 request unit, and you only pay for responses that land. One product checked from five countries is 5 units; 100 products from five countries every week is about 2,000 units a month. See pricing.

Next steps

For how country targeting works in general, and on the tunnel, read getting geo-targeted results by country. To run the comparison daily and log changes, see monitoring competitor prices on a schedule. Every request option is in the docs, and you can sign up free to try it.

Start free with 1,000 requests Read the docs