Skip to content
FetchRelay

Bulk web scraping

Keep the successful pages. Know which ones failed.

A collection job should give you usable records and visible failures. Use Batch when you already have the URLs, or Crawl when you want to discover pages from a starting point.

Useful for: Developers collecting public research material, documentation, or website inventories for downstream processing.

Start a collection jobWorkbench instructions
The result you are aiming for
{
  "total": 2,
  "completed": 2,
  "failed": 0,
  "data": ["one result per normalized URL"]
}

Illustrative response shape; inspect actual per-URL records

From the first request to a repeatable workflow

  1. 01

    Choose discovery or a known URL list

    Use Map to inspect a site’s sitemaps and links without extracting every page. Use Crawl to follow pages from a starting URL. Use Batch when your application already has the exact URLs it needs.

  2. 02

    Set a bounded request

    For Crawl, choose a page limit, depth, and include/exclude paths. For Batch, enter one URL per line. Direct batch requests accept up to 20 URLs; authenticated background jobs accept up to 1,000.

  3. 03

    Save work that should outlive one request

    Sign in and enable Run in background. The API returns a job ID. Poll its status and read partial results while it runs. The worker saves progress and can resume eligible interrupted jobs; retries after a crash can repeat a page fetch.

  4. 04

    Export and retry selectively

    Download JSON, JSONL, or CSV. Check completed and failed counts instead of assuming the whole job succeeded. Cancellation preserves saved results. Re-run only failed or missing URLs after checking why they failed.

Make the API request

Create a key in Account and set it as FETCHRELAY_API_KEY in your environment. Keep it out of client-side code and source control.

curl --fail-with-body https://fetchrelay.com/api/batch \
  -H "Authorization: Bearer $FETCHRELAY_API_KEY" \
  -H 'Content-Type: application/json' \
  --data '{
  "urls": [
    "https://example.com",
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
  ],
  "formats": [
    "markdown",
    "links"
  ],
  "mode": "auto",
  "maxAge": 0
}'

This is a live request and uses account credits. See the OpenAPI reference for the complete response schema.

Check before you ship

  • Completed and failed counts reconcile with the requested normalized URLs.
  • Exports contain the successful records and expected source URLs.
  • Settled charges reflect successful pages, with unused reservations released.

Know the limits

Background jobs have a 1,000-page maximum and a 20 MB saved-dataset budget. Resource limits can produce partial results. FetchRelay does not promise unlimited concurrency or fixed throughput. Scheduled multi-page crawls are not included; monitors watch individual pages.

Common questions

What is the difference between Map and Crawl?

Map lists discovered URLs. Crawl extracts page content. A successful map request uses one credit; a crawl settles against successfully extracted pages.

Do I pay for a page that failed?

No page credit is charged for a failed extraction. Crawl and batch jobs reserve capacity first and release the unused portion when the job settles.

Can I download a running job?

Yes. Saved jobs expose partial results with paginated JSON, JSONL, and CSV exports. Keep the job ID so you can fetch the remaining records later.

Try it on a page you actually need.

Inspect the output before committing to an integration. Create an account for 1,000 free monthly credits, saved jobs, monitors, and API keys. No credit card required.

Open the workbench See pricing