How to monitor competitor prices on a schedule

E-commerce and prices30 September 20264 min read

A price monitor is three small pieces: a list of product pages, a script that reads the current price from each, and a scheduler that runs the script every day. This guide builds all three with cron, a short Python script and POST /v1/extract, and keeps the price history in a CSV you own.

Who does what

The scrape.land API is stateless by design. It fetches the URL you send, returns the data, and forgets it. It has no built-in scheduler, watch list or history. That is deliberate: you choose how often to check, where the history lives and what counts as an alert, and you never pay for checks you did not ask for. In short: you schedule, it fetches.

Step 1: the watch list

Create watch.csv with one product per line: a name you recognise, the URL, and the CSS selector for the price on that site.

watch.csv

name,url,price_selector
lamp-shop-a,https://shop.example/item/42,.price
lamp-shop-b,https://example.com/products/desk-lamp,span[itemprop=price]

To find a selector, open the product page, right-click the price, choose Inspect, and pick a class or attribute that looks stable. itemprop=price is a good choice when a site has it, because it is part of the site's structured data and rarely changes with a redesign.

Step 2: one request per product

Each check is a single extraction. Test one from the command line first:

curl

curl https://scrape.land/v1/extract \
  -H "X-Api-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://shop.example/item/42",
       "country": "us",
       "fields": {"price": ".price", "title": "h1"}}'

JSON

{
  "url": "https://shop.example/item/42",
  "status": 200,
  "data": {
    "price": "$49.00",
    "title": "Vintage desk lamp"
  }
}

"country" matters for pricing work: many shops show a different price, currency or tax depending on where the visitor is. Set it to the market you are tracking and keep it fixed, or your history will mix two different prices.

Step 3: the script

Python

#!/usr/bin/env python3
"""check_prices.py: read watch.csv, append today's prices to history.csv, report changes."""
import csv
import datetime
import os
import requests

HERE = os.path.dirname(os.path.abspath(__file__))
WATCH = os.path.join(HERE, "watch.csv")
HISTORY = os.path.join(HERE, "history.csv")

def last_prices():
    last = {}
    if os.path.exists(HISTORY):
        with open(HISTORY, newline="", encoding="utf-8") as f:
            for row in csv.DictReader(f):
                last[row["name"]] = row["price"]
    return last

def check(url, selector):
    r = requests.post(
        "https://scrape.land/v1/extract",
        headers={"X-Api-Key": os.environ["SCRAPELAND_KEY"]},
        json={"url": url, "country": "us", "fields": {"price": selector}},
        timeout=90,
    )
    r.raise_for_status()
    body = r.json()
    if body.get("field_errors"):
        raise ValueError(f"selector broken: {body['field_errors']}")
    return body["data"]["price"]

previous = last_prices()
now = datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds")
new_file = not os.path.exists(HISTORY)

with open(WATCH, newline="", encoding="utf-8") as f, \
     open(HISTORY, "a", newline="", encoding="utf-8") as out:
    w = csv.writer(out)
    if new_file:
        w.writerow(["checked_at", "name", "price"])
    for item in csv.DictReader(f):
        try:
            price = check(item["url"], item["price_selector"])
        except Exception as e:
            print(f"ERROR {item['name']}: {e}")
            continue
        if price is None:
            print(f"MISSING {item['name']}: no price on the page (sold out or layout change?)")
        elif previous.get(item["name"]) not in (None, price):
            print(f"CHANGED {item['name']}: {previous[item['name']]} -> {price}")
        w.writerow([now, item["name"], price or ""])

The script separates three situations that a careless monitor mixes up. A field_errors entry means your selector is broken. A null price with no error means the page has no price element, which usually means sold out or a redesign. A changed value is a real price change. Only the last one should reach whoever sets your prices.

Step 4: schedule it with cron

Run crontab -e on any Linux or macOS machine and add a line. This one runs every day at 07:00:

crontab

SCRAPELAND_KEY=YOUR_KEY
0 7 * * * /usr/bin/python3 /opt/prices/check_prices.py >> /opt/prices/check.log 2>&1

Any scheduler works the same way: a systemd timer, Windows Task Scheduler, a scheduled GitHub Actions workflow or your job queue. Pipe the CHANGED lines into email, Slack or a webhook once you trust the output.

How often to check

Daily is enough for most shops, since prices rarely change more than once a day. Hourly checks make sense for flash sales, marketplaces and travel, but they multiply your request count by 24. Start daily, look at how often the history actually changes, and only then decide whether a tighter schedule is worth it. Checking at a random minute rather than exactly on the hour also spreads your load more politely across the sites you watch.

Growing it

Many products on one site: read a whole category page with a group selector instead of one request per product. See extracting prices into a spreadsheet.
Pages that need JavaScript: add "render": true and "block_resources": true. See the JavaScript guide.
Many different shops: an AI prompt such as "the current price and currency" saves writing a selector per site (Scale plan and up).

What it costs

Each check is 1 request unit, and you only pay for responses that land (blocks and retries are free). 30 products checked once a day is about 900 requests a month, inside the Free plan's 1,000; 100 products daily is about 3,000, which fits Starter ($99, 600K a month) many times over. See pricing.

Next steps

The country option and every field form are in the docs. Create a free account, put your key in the crontab, and your first price history starts tomorrow morning.

Start free with 1,000 requests Read the docs