# Web scraping with Python: a working API example

> Extract a webpage as Markdown with Python’s standard library. Handle HTTP errors, check extraction warnings, and keep your FetchRelay API key out of source code.

By FetchRelay. Updated 2026-09-06.
Canonical: https://fetchrelay.com/guides/web-scraping-python

## What you will build

A Python script that requests one public page, checks the response, and prints Markdown. The goal is a small, inspectable integration before you build a larger collection job. The request runs through FetchRelay’s hosted extraction engine; Python does not need to launch a browser.

## Before you start

Use Python 3, a [FetchRelay account](https://fetchrelay.com/account), and an API key created in Account. This example uses only Python’s standard library. Set FETCHRELAY_API_KEY in your local environment or secret manager. Do not put a real key in a notebook you share, a browser app, or version control. A successful page uses one account credit.

## Make your first request

Save this as scrape.py. It uses Example Domain so the first result is easy to recognize.

```python
import json
import os
import sys
from urllib.request import Request, urlopen
from urllib.error import HTTPError, URLError

payload = {
    "url": "https://example.com",
    "formats": ["markdown"],
    "mode": "http",
    "onlyMainContent": True,
    "timeout": 25000,
}
request = Request(
    "https://fetchrelay.com/api/scrape",
    data=json.dumps(payload).encode("utf-8"),
    headers={
        "Authorization": "Bearer " + os.environ["FETCHRELAY_API_KEY"],
        "Content-Type": "application/json",
    },
    method="POST",
)
try:
    with urlopen(request, timeout=40) as response:
        result = json.load(response)
except HTTPError as error:
    print("Request failed with HTTP", error.code, file=sys.stderr)
    raise SystemExit(1)
except URLError:
    print("Network request failed; check your connection.", file=sys.stderr)
    raise SystemExit(1)

if not result.get("success"):
    raise SystemExit("Extraction failed; inspect the API response.")
for warning in result.get("diagnostics", {}).get("warnings", []):
    print("Review:", warning, file=sys.stderr)
print(result["data"]["markdown"])
```

Run python scrape.py. Look for the Example Domain heading in the output. Check data.metadata.sourceURL against the target and inspect diagnostics.warnings before treating a result as complete. An HTTP success response alone does not prove that the page contains the content your application needs.

## Extract fields instead of prose

For a known layout, add an extract object with named CSS selectors. For the [Books to Scrape example](https://fetchrelay.com/use-cases/product-price-extraction), use title: h1, price: .price_color, and availability: .availability. The API returns these under data.extract. These selectors are specific to that test catalog; they are not universal shopping-site selectors. Validate both the field values and their meaning before using them in a report.

## When a request fails

An invalid or revoked key returns an authentication error. A rate or allowance limit needs a slower request rate or more available credits, not a tight retry loop. A protected or unavailable source may stay inaccessible after retries. For JavaScript-driven content, try mode: browser after testing a representative page. Browser rendering is not CAPTCHA solving or residential proxy access.

Do not automatically replay a request just because your client timed out: the server may have completed it. For multi-page work, start a [saved crawl or batch](https://fetchrelay.com/use-cases/bulk-web-scraping), retain its job ID, and poll its status instead of restarting the job.

## Next step

Try one actual public page you need. Compare the extracted text with the source, then choose a stable output format. Use the [OpenAPI reference](https://fetchrelay.com/api/openapi) for the complete request contract and the [HTTP versus browser guide](https://fetchrelay.com/guides/http-vs-browser-scraping) to choose rendering deliberately.

Reference: [Python urllib.request documentation](https://docs.python.org/3/library/urllib.request.html).