Web scraping with Python: a working API example

What you will build

A Python script that requests one public page, checks the response, and prints Markdown. The goal is a small, inspectable integration before you build a larger collection job. The request runs through FetchRelay’s hosted extraction engine; Python does not need to launch a browser.

Before you start

Use Python 3, a FetchRelay account, and an API key created in Account. This example uses only Python’s standard library. Set FETCHRELAY_API_KEY in your local environment or secret manager. Do not put a real key in a notebook you share, a browser app, or version control. A successful page uses one account credit.

Make your first request

Save this as scrape.py. It uses Example Domain so the first result is easy to recognize.

import json
import os
import sys
from urllib.request import Request, urlopen
from urllib.error import HTTPError, URLError

payload = {
    "url": "https://example.com",
    "formats": ["markdown"],
    "mode": "http",
    "onlyMainContent": True,
    "timeout": 25000,
}
request = Request(
    "https://fetchrelay.com/api/scrape",
    data=json.dumps(payload).encode("utf-8"),
    headers={
        "Authorization": "Bearer " + os.environ["FETCHRELAY_API_KEY"],
        "Content-Type": "application/json",
    },
    method="POST",
)
try:
    with urlopen(request, timeout=40) as response:
        result = json.load(response)
except HTTPError as error:
    print("Request failed with HTTP", error.code, file=sys.stderr)
    raise SystemExit(1)
except URLError:
    print("Network request failed; check your connection.", file=sys.stderr)
    raise SystemExit(1)

if not result.get("success"):
    raise SystemExit("Extraction failed; inspect the API response.")
for warning in result.get("diagnostics", {}).get("warnings", []):
    print("Review:", warning, file=sys.stderr)
print(result["data"]["markdown"])

Run python scrape.py. Look for the Example Domain heading in the output. Check data.metadata.sourceURL against the target and inspect diagnostics.warnings before treating a result as complete. An HTTP success response alone does not prove that the page contains the content your application needs.

Extract fields instead of prose

For a known layout, add an extract object with named CSS selectors. For the Books to Scrape example, use title: h1, price: .price_color, and availability: .availability. The API returns these under data.extract. These selectors are specific to that test catalog; they are not universal shopping-site selectors. Validate both the field values and their meaning before using them in a report.

When a request fails

An invalid or revoked key returns an authentication error. A rate or allowance limit needs a slower request rate or more available credits, not a tight retry loop. A protected or unavailable source may stay inaccessible after retries. For JavaScript-driven content, try mode: browser after testing a representative page. Browser rendering is not CAPTCHA solving or residential proxy access.

Do not automatically replay a request just because your client timed out: the server may have completed it. For multi-page work, start a saved crawl or batch, retain its job ID, and poll its status instead of restarting the job.

Next step

Try one actual public page you need. Compare the extracted text with the source, then choose a stable output format. Use the OpenAPI reference for the complete request contract and the HTTP versus browser guide to choose rendering deliberately.

Reference: Python urllib.request documentation.