How to use scrape.land from Claude and AI assistants (MCP)

AI assistants are good at reading and summarising web pages, but many of them cannot fetch a page on their own, or get blocked when they try. The Model Context Protocol (MCP) lets an assistant call outside tools. scrape.land runs an MCP server, so Claude and other MCP clients can fetch pages, extract data, search the web and find local businesses with your key.

What the assistant gets

The server lives at https://scrape.land/mcp and offers four tools:

• fetch_page: fetch a URL and return its content, for example as clean Markdown the model can read.
• extract_data: pull specific fields out of a page.
• web_search: search the web and return result titles, URLs and snippets.
• find_local_businesses: list businesses of one category in one place, with the same filters as POST /v1/places.

The assistant decides when to call them. You ask in plain language, and it picks the tool and fills in the arguments.

Claude Code: one command

In a terminal, with your API key in place of YOUR_KEY:

claude mcp add --transport http scrape-land https://scrape.land/mcp --header "Authorization: Bearer YOUR_KEY"

Start a new Claude Code session and the tools are available. Run /mcp inside the session to check that scrape-land is connected. Then ask for something that needs the web:

Fetch https://example.com/pricing and summarise the plans in a table.

Find plumbers in Leeds that have a phone number but no website,
and save them to plumbers.csv.

Other assistants and MCP clients

Any client that supports remote MCP servers over HTTP can connect with the same two pieces of information: the URL https://scrape.land/mcp and the header Authorization: Bearer YOUR_KEY. Where that goes depends on the client: a settings screen for connectors, or a JSON config file. The shape is usually close to this:

{
  "mcpServers": {
    "scrape-land": {
      "type": "http",
      "url": "https://scrape.land/mcp",
      "headers": {"Authorization": "Bearer YOUR_KEY"}
    }
  }
}

Check your client's own documentation for the exact key names; they differ slightly between tools.

Checking the connection with curl

MCP over HTTP is JSON-RPC in a POST request, so you can talk to the server without any assistant, which is useful when a client says it cannot connect. This asks the server for its tool list:

curl https://scrape.land/mcp \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "tools": [
      {"name": "fetch_page", "description": "...", "inputSchema": {"...": "..."}},
      {"name": "extract_data", "description": "...", "inputSchema": {"...": "..."}},
      {"name": "web_search", "description": "...", "inputSchema": {"...": "..."}},
      {"name": "find_local_businesses", "description": "...", "inputSchema": {"...": "..."}}
    ]
  }
}

A 401 means the key is missing or wrong. Each tool's inputSchema lists the arguments it accepts.

From your own Python agent

If you are building an agent, the official MCP Python SDK (pip install mcp) connects to the same endpoint:

import asyncio
from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client

async def main():
    headers = {"Authorization": "Bearer YOUR_KEY"}
    async with streamablehttp_client("https://scrape.land/mcp", headers=headers) as (read, write, _):
        async with ClientSession(read, write) as session:
            await session.initialize()
            tools = await session.list_tools()
            print([t.name for t in tools.tools])
            result = await session.call_tool("fetch_page", {"url": "https://example.com"})
            print(result.content[0].text[:500])

asyncio.run(main())

Hand the tool list to your model, and pass its tool calls through session.call_tool. Your agent then has the same web access as Claude above, without you writing a scraper.

Good habits

• Use a separate key for each assistant, so you can see its usage in the dashboard and revoke it on its own.
• Ask for Markdown when the goal is reading. It is much shorter than HTML, so the assistant spends fewer tokens per page.
• Be specific. "Fetch these three URLs" costs three requests. "Research this topic" lets the assistant decide how many pages to read.

When the assistant does not use the tools

If the assistant answers from memory instead of fetching, say so in the prompt: "use scrape-land to fetch the page". If a tool call fails, the error text comes back to the assistant, and it will usually tell you what went wrong: an invalid key, a page that needs rendering, or a category that does not exist.

What it costs

A tool call costs the same as the API call behind it: a page fetch or CSS extraction is 1 request unit, a local business search is 1 unit per 10 rows, and AI extraction (Scale plan and up) adds +2, +4 or +25 by tier. You only pay for responses that land. The Free plan includes 1,000 requests a month, no card. See pricing.

Next steps

The request options each tool maps to are in the docs. Create a free account, copy your key into the command above, and ask your assistant to read a page.