Web Scraping API
The Web Scraping API built for AI agents and pipelines
Scrape any URL to markdown, HTML, links and tables. Crawl a whole site as a background job. Map every URL a site publishes. Extract schema-defined records from pages that don't hand you a schema. One key, one flat credit per call — $0.30 per 1,000 — with no surcharge for JavaScript rendering or anti-bot handling.
{ "searchTime": 0.17, "totalResults": 85000, "page": 1, "organic": [ { "position": 1, "title": "31.10. Connection Pools and Data Sources", "link": "www.postgresql.org/docs/…", "domain": "postgresql.org", "snippet": "For an environment without an application se…" }, { "position": 2, "domain": "medium.com", … }, { "position": 3, "domain": "stackoverflow.blog", … } ], "credits": 1}What is a web scraping API and what does it cost?
A web scraping API fetches a URL through managed infrastructure — solving JavaScript rendering, anti-bot defenses and proxy rotation for you — and returns clean markdown, HTML, links or structured records instead of raw code you parse yourself. Searlo charges one flat credit per call, $0.30 per 1,000 at the Scale pack, with no surcharge for rendering or anti-bot handling.
That last part is the part every competitor's pricing page quietly contradicts. Firecrawl charges +4 credits (5x) for a JSON-formatted response instead of markdown. ZenRows charges 5 credits (5x) for JavaScript rendering, 10 (10x) for a premium proxy, and 25 (25x) for both together — exactly quantified, on their own pricing page. ScrapingBee and Scrapfly both say plainly that rendering, premium proxies and extraction cost more, without publishing the multiplier. Searlo's backend charges the same 1 credit per /scrape or /map call and 1 credit per successfully-fetched /extract URL regardless of whether a page needed a real browser — confirmed in source, not marketing copy.
- Cost per call
- $0.30 per 1,000 (Scale pack, $74.99 one-time)— our pricing
- 100,000 pages
- $43.97 one-time (~$0.44/1K)— our pricing
- JS-render / anti-bot surcharge
- None — same 1 credit whether or not a browser render was needed
- Free to start
- 3,000 credits, no card— our pricing
- Firecrawl JSON-format surcharge
- +4 credits/page (5x the base rate)— firecrawl.dev/pricing
- ZenRows JS + premium proxy surcharge
- 25 credits/page (25x the base rate)— zenrows.com/pricing
- Scrapfly per-request cost
- Depends on features enabled — no flat multiplier published— scrapfly.io/pricing
Third-party figures on this page last verified against the sources linked above. Prices change; if you find one of these stale, tell us and we will correct it.
Getting structured data off the web usually means stitching together several tools: one to fetch a page past anti-bot defenses, one to render JavaScript, one to discover a site's URLs, one to crawl at depth, and one to turn a page into structured fields. Searlo's Web Data API is one key covering all of it. /scrape returns a single URL as markdown, plain text, raw HTML, links, images, headings, tables and the page's own JSON-LD in one call. /map lists every URL a site publishes, read from its robots.txt and sitemaps. /crawl walks a whole site as a background job with depth and path filters and a webhook on completion. /extract turns a field schema into filled-in records. /search-and-scrape runs a web search and fetches the pages it found, in one call — and for a single quick fetch, GET /search/webpage takes just a URL and returns clean markdown with metadata already parsed.
Every one of those calls costs the same flat 1 credit — $0.30 per 1,000 at the Scale pack — whether the page is static HTML or needs a full browser render to load. That is not how the rest of the category prices scraping: see the section below for exactly what Firecrawl, ScrapingBee, Scrapfly and ZenRows charge extra for, with their own published numbers.
Extract is schema-in, records-out. Give it a field schema and one or more URLs, and it fills the schema from what the page already declares — schema.org JSON-LD, Open Graph, meta tags — before falling back to a language model only for whatever is left blank. A field the page never states comes back null and listed in missingFields rather than guessed, and you are billed per URL fetched, not per field or per model call.
Pages that render client-side fall back to a real browser automatically across every one of these endpoints, so a React or Vue front end returns the same complete markdown as a static HTML page — you don't set a render flag ahead of time to find out which kind of page you're dealing with.
Features
Scrape any URL
Markdown, plain text, raw HTML, links, images, headings, tables and the page's own JSON-LD — one credit, whatever formats you ask for.
Map a site's URLs
Every URL a site publishes, read from robots.txt and its sitemaps, falling back to on-page links when there is no sitemap.
Crawl whole sites as a job
Depth and path filters, a webhook on completion, robots.txt obeyed by default — billed per page as the job actually runs.
Extract schema-defined records
Give a field schema, get records back — sourced first from what the page declares, with an optional LLM fallback for what's left blank.
Search and scrape in one call
Run a web search and fetch the pages it found — one credit for the search, one per page actually scraped.
Quick single-page fetch
GET /search/webpage takes just a URL and returns clean markdown with title, author and publish date — the lightweight mode for an agent's tool call.
Get started in one request
One credit, whichever formats you ask for. The same call works on a static blog post or a JavaScript-rendered app — a real browser kicks in automatically when the page needs one.
curl -X POST "https://api.searlo.tech/api/v1/scrape" \
-H "x-api-key: YOUR_API_KEY" \
-H "content-type: application/json" \
-d '{
"url": "https://example.com/blog/how-we-ship",
"formats": ["markdown", "links", "tables"],
"onlyMainContent": true
}'import requests
r = requests.post(
"https://api.searlo.tech/api/v1/scrape",
headers={"x-api-key": "YOUR_API_KEY"},
json={
"url": "https://example.com/blog/how-we-ship",
"formats": ["markdown", "links", "tables"],
"onlyMainContent": True,
},
)
page = r.json()
print(page["metadata"]["title"])
print(page["markdown"][:500])
print(len(page["links"]), "links found")const res = await fetch("https://api.searlo.tech/api/v1/scrape", {
method: "POST",
headers: {
"x-api-key": "YOUR_API_KEY",
"content-type": "application/json",
},
body: JSON.stringify({
url: "https://example.com/blog/how-we-ship",
formats: ["markdown", "links", "tables"],
onlyMainContent: true,
}),
});
const page = await res.json();
console.log(page.metadata.title);
console.log(page.markdown.slice(0, 500));
console.log(page.links.length, "links found");Searlo vs Firecrawl
Firecrawl is the category's most visible product page and the clearest comparison. Both sell scrape, crawl, map and extract as one API — the difference is what a real page actually costs.
| Capability | Searlo | Firecrawl |
|---|---|---|
| Price per 1,000 pages (basic scrape) | $0.30 | $0.599–$3.20, depending on plan |
| 100,000 pages, total cost | $43.97 one-time (~$0.44/1K) | $83/month (Standard), recurring |
| Structured JSON output | Included, same 1 credit | +4 credits/page (5x) |
| Free credits to start | 3,000, one-time, no card | 1,000/month, only while on the free plan |
| Billing model | One-time credit packs, no expiry pressure | Monthly subscription tiers |
| Site map + full crawl with webhook | Yes | Yes |
The credit-multiplier trap
Every competitor in this category quotes a headline per-1,000 rate, and every one of those rates is the price of the cheapest possible request — plain HTML, no rendering, no anti-bot handling, no structured output. The moment a real page needs any of those three things, which most pages do, the real cost is higher, and "higher" is a published, exact number on at least two vendors' own pricing pages.
ZenRows is the most explicit: JavaScript rendering costs 5 credits instead of 1 (5x), a premium proxy costs 10 credits (10x), and the two combined cost 25 credits (25x) — all stated directly on zenrows.com/pricing. Firecrawl charges +4 credits per page (5x) for a JSON, Question or Highlight-formatted response instead of plain markdown. ScrapingBee and Scrapfly both say plainly that JavaScript rendering, premium proxies and AI extraction cost more credits than a basic request, without publishing the exact multiplier on their pricing page — which is itself worth knowing before you commit to a plan sized on the headline rate.
Searlo's backend charges the same 1 credit per /scrape or /map call and 1 credit per successfully-fetched /extract URL no matter which of those conditions apply — confirmed directly in scrapeService.js and webdataRoutes.js, not asserted in marketing copy. A page that needs a real browser to render costs exactly the same as one that doesn't.
- ZenRows: 5x for JS render, 10x for a premium proxy, 25x for both (published)
- Firecrawl: +4 credits/page (5x) for JSON/Question/Highlight output formats
- ScrapingBee & Scrapfly: acknowledge a render/proxy surcharge, don't publish the number
- Searlo: flat 1 credit per /scrape or /map call, 1 credit per successful /extract URL, always
Extract: schema in, records out, with an honest LLM fallback
Most pages that matter already publish their own structured data — a product page carries schema.org JSON-LD with the price, an article carries the author and publish date, every page carries Open Graph tags. /extract reads what the page declares first, because that is exact, free to compute, and nothing a model can improve on.
What's left after that deterministic pass is optional model fallback — used only for fields the page did not declare, and only when a model is configured on the deployment. A field neither pass can fill comes back null and listed in missingFields, never guessed, and every field in the response records which of the two passes produced it so you can tell a value the site published from one a model inferred.
- JSON-LD, then Open Graph and meta tags, then unambiguous text patterns — model last, and optional
- Billed per URL successfully fetched, not per field and not per model call
- A URL that fails to fetch is never billed
- Every field's source is in the response — json-ld, meta, text, or llm
Crawl a whole site without babysitting a queue
A crawl is queued and returns a job id immediately — you poll it or supply a webhook, and credits are charged per page as the job actually runs, so a job capped at 500 pages that finds only twelve costs twelve. Path filters (includePaths, excludePaths) and a maxDepth keep a crawl from wandering into a site's entire history when you only want the blog or the docs; robots.txt is obeyed by default.
/map answers a narrower, cheaper question first: what exists on this site, before you pay to fetch any of it. It reads robots.txt for declared sitemaps, walks them — including sitemap indexes — and falls back to the homepage's internal links when a site publishes no sitemap at all. Run /map before a big /crawl and you know the real size of the job before you queue it.
- Job id back immediately; poll status or supply a webhook for completion
- includePaths / excludePaths as globs, maxDepth up to 10, concurrency up to 8
- robots.txt obeyed by default
- Cancel a running crawl any time — pages already fetched stay readable, and you stop paying
What teams build on the Web Data API
- RAG pipelines that need a page's clean content, not its raw HTML
- Product catalog and price-monitoring feeds from competitor sites
- Lead enrichment — company sites turned into structured firmographic fields
- Content migrations — mapping and crawling a whole site before a rebuild
- AI agent tool calls that need one quick page fetch mid-conversation
- Search-driven research pipelines that need the pages behind the results, not just the links
FAQ
Frequently asked questions
What is a web scraping API?
A web scraping API is an endpoint that fetches a URL through managed infrastructure — proxies, anti-bot handling and a real browser when a page needs one — and returns clean, structured content instead of raw HTML you'd have to parse and maintain a parser for yourself.
How much does it cost to scrape 100,000 pages?
On Searlo, $43.97 one-time (the cheapest combination of credit packs reaching 100,000 credits), or about $0.44 per 1,000 pages. There is no subscription and no expiry pressure beyond the standard 90-day validity on new-user credits.
Does Searlo charge more for JavaScript-rendered pages?
No. /scrape, /map and /crawl each cost the same flat 1 credit per call whether or not the page needed a real browser render, and /extract bills 1 credit per successfully-fetched URL regardless. Several competitors — ZenRows most explicitly, at 5x for rendering and 10x for a premium proxy — charge more for exactly that.
What's the difference between /scrape, /crawl and /map?
/scrape fetches one URL you already know. /map lists the URLs a site publishes, from its robots.txt and sitemaps, without fetching their content — useful for scoping a job before you pay to run it. /crawl fetches many pages starting from one URL, following links to a set depth, and is billed per page as it runs.
Does /extract use AI?
Only as a fallback. It reads a page's own schema.org JSON-LD, Open Graph tags and meta tags first — deterministic, exact, and free to compute — and only sends a model the fields that pass could not fill. A field neither step can find comes back null and listed in missingFields rather than invented.
Am I billed for a URL that fails to load?
No, for /extract — it is billed per URL successfully fetched, after the work runs, so a URL that could not be reached costs nothing. /scrape and /map are billed on the request itself, the same way every Searlo endpoint is.
Can I crawl a whole site and get notified when it's done?
Yes. POST /crawl returns a job id immediately; poll its status endpoint or supply a webhook URL and it fires when the crawl finishes. Credits are charged per page as the crawl runs, so the limit you set is a ceiling, not a quote.
Is there a free tier?
Yes — 3,000 free credits on signup, no card required, usable across every endpoint on this page.
How is a web scraping API different from a SERP or search API?
A search API answers "which pages exist for this query" — organic results, positions, titles, snippets. A web scraping API answers "what is actually on this page" once you have a URL, whether from a search result, a sitemap, or your own list. Searlo runs both under the same key; /search-and-scrape is the endpoint that combines them in one call.
Can I search the web and get the page content back in one call?
Yes — POST /search-and-scrape runs a web search and fetches the pages it found, returning each result with its scraped content attached. A page that blocks the fetch comes back with scraped: false next to the ones that worked, rather than failing the whole request.
From the team behind Searlo
We also build RenderKit — the browser-rendering API for screenshots, PDFs, and LLM-ready markdown. Same billing model, same developer-first API design.