Web Data (Scrape, Map, Crawl, Extract)

The retrieval half of the API: read a page, list a site's URLs, crawl a site, or pull structured records out of pages. Pages that render client-side are rendered automatically. That automatic rendering is for accounts that have bought credits; on sign-up credits alone, pass render: true to render a page.

Endpoints#

• POST/scrape
• POST/scrape/batch
• POST/map
• POST/crawl
• GET/crawl
• GET/crawl/{jobId}
• GET/crawl/{jobId}/results
• DELETE/crawl/{jobId}
• POST/extract
• POST/search-and-scrape
• GET/search/webpage

Credits:

From 2026-11-01: pages that needed a browser cost 5. On every row above except /map, a page that had to be rendered costs 4 credits more than the table says: 5 instead of 1 per page. That covers render: true, actions, and automatic rendering when a plain fetch returns no usable content, such as a JavaScript-only page. The page's via says which it was: browser or http. A cached copy costs the same as the page it copies; a failed fetch is never charged; the live-fetch surcharge is added on top, not multiplied (a forced live fetch that needed a browser is 2 + 4 = 6). render: false keeps every page at the plain price. Until 2026-11-01 a browser page costs the same as any other.

Freshness and the Live-Fetch Surcharge#

/scrape, /map and /search-and-scrape keep a cache, controlled by two fields:

• Omit both, and any cached copy is used. The call costs 1 whether or not it was cached (from 2026-11-01, 5 for a page that needed a browser, cached or not).
• maxAge (seconds, 0–2592000): use a cached copy only if it is at most this old.
• cache: false is the same as maxAge: 0: always fetch live.

When your maxAge or cache: false makes Searlo fetch the page live, the call costs 2 and the response carries X-Cache-Bypass: CHARGED. If a cached copy satisfied your maxAge, it costs 1. X-Credits-Deducted always shows the total, and a cached answer carries _cacheAgeSec.

URLs#

Every url and urls field takes an http or https URL of up to 2,048 characters. An internationalised domain name works as you write it on scrape, map, crawl, scrape/batch and extract: https://www.löschwagen.com/preise is fetched, cached and billed under its encoded form, https://www.xn--lschwagen-07a.com/preise, and so is a path in any script. (GET /search/webpage takes the encoded xn-- form only.)

A URL that cannot be fetched as written is a 400 VALIDATION_ERROR (url must be an http(s) URL), answered before anything is fetched and never charged: a control character or a space anywhere in it, an escaped one in the host (%00), a scheme other than http or https, or a URL that cannot be parsed. A private or reserved address is 422 TARGET_NOT_ALLOWED.

Scrape#

{  "searchParameters": { "url": "https://example.com/pricing", "type": "scrape", "formats": ["markdown", "links"] },  "url": "https://example.com/pricing",  "requestedUrl": "https://example.com/pricing",  "statusCode": 200,  "via": "http",  "bytes": 48213,  "markdown": "# Pricing\n\nStart free, then pay as you go...",  "metadata": {    "title": "Pricing | Example",    "description": "Plans and prices.",    "author": null,    "publishedDate": null,    "language": "en",    "canonicalUrl": "https://example.com/pricing",    "robots": null,    "wordCount": 812,    "openGraph": { "title": "Pricing", "image": "https://example.com/og.png" },    "twitter": null  },  "links": [{ "url": "https://example.com/signup", "text": "Sign up", "internal": true }],  "source": "searlo.tech(scrape)",  "credits": 1}

url is the final URL after redirects and requestedUrl is what you sent. via is http or browser (serp for a Google results page). A page that comes back empty is returned but not cached.

Map#

Every URL a site publishes, before you pay to crawl it: robots.txt, then its sitemaps (sitemap indexes included), then the homepage's links when there is no sitemap.

{  "searchParameters": { "url": "https://example.com", "type": "map", "limit": 1000 },  "origin": "https://example.com",  "discovery": "sitemap",  "sitemaps": ["https://example.com/sitemap.xml"],  "robots": { "present": true, "sitemapCount": 1 },  "totalResults": 214,  "urls": [    { "url": "https://example.com/pricing", "lastModified": "2026-09-01" },    { "url": "https://example.com/blog/launch", "lastModified": null }  ],  "source": "searlo.tech(map)",  "credits": 1}

discovery is sitemap, links (homepage links, when the site publishes no sitemap) or none.

Crawl#

Queue a crawl of one site and get a job id back at once. Starting needs a minimum balance (configurable); after that, credits are charged per page as the crawl runs, so a job with limit: 500 on a twelve-page site costs 12 (from 2026-11-01, plus 4 for each of those pages that needed a browser). robots.txt is obeyed by default.

The snake_case spellings (max_depth, include_paths, exclude_paths, allow_subdomains, allow_external, respect_robots, use_sitemap, only_main_content, store_content) are accepted too.

{  "success": true,  "jobId": "crawl_0123456789abcdef01234567",  "status": "queued",  "url": "https://example.com/docs",  "options": { "maxDepth": 2, "limit": 50, "includePaths": ["/docs/*"], "excludePaths": [], "formats": ["markdown"], "allowSubdomains": false, "allowExternal": false, "respectRobots": true, "concurrency": 3, "render": null, "onlyMainContent": true, "useSitemap": true, "storeContent": true },  "maxCredits": 50,  "statusUrl": "/api/v1/crawl/crawl_0123456789abcdef01234567",  "resultsUrl": "/api/v1/crawl/crawl_0123456789abcdef01234567/results",  "createdAt": "2026-09-29T10:00:00.000Z"}

The response is 202. maxCredits is a ceiling, not a quote; from 2026-11-01 it allows 5 per page unless the job sets render: false.

Batch Scrape#

Up to 1000 URLs as one job, with no discovery: the job fetches exactly the URLs you send. They are normalised and de-duplicated first, so compare submitted with urlCount. Like a crawl, it needs a minimum balance to start and then bills 1 credit per page fetched (from 2026-11-01, 5 for a page that needed a browser). Status, results, cancellation and webhooks use the crawl endpoints below.

maxAge, cache, includeLinks and excludeSelectors are accepted on a batch but have no effect: every page is fetched live, at 1 credit (5 from 2026-11-01 if it needed a browser).

The 202 response has the crawl shape plus mode: "batch", urlCount (what was queued) and submitted (what you sent).

Job Status and Results#

• GET/crawl/{jobId}
• GET/crawl/{jobId}/results
• GET/crawl
• DELETE/crawl/{jobId}

All free, for crawls and batches alike.

{  "success": true,  "jobId": "crawl_0123456789abcdef01234567",  "status": "completed",  "url": "https://example.com/docs",  "options": { "maxDepth": 2, "limit": 50, "includePaths": ["/docs/*"], "excludePaths": [], "formats": ["markdown"], "allowSubdomains": false, "allowExternal": false, "respectRobots": true, "concurrency": 3, "render": null, "onlyMainContent": true, "useSitemap": true, "storeContent": true },  "progress": { "discovered": 64, "completed": 50, "failed": 2, "skipped": 3 },  "creditsCharged": 50,  "stopReason": "page limit reached",  "webhook": { "url": "https://example.com/hooks/crawl", "deliveredAt": "2026-09-29T10:04:10.000Z", "attempts": 1 },  "createdAt": "2026-09-29T10:00:00.000Z",  "startedAt": "2026-09-29T10:00:02.000Z",  "finishedAt": "2026-09-29T10:04:09.000Z"}

status is queued, running, completed, failed or cancelled. stopReason says how a job ended: page limit reached, all urls processed (a batch), credit balance exhausted, job deadline reached (jobs run for a configurable maximum time) or cancelled by caller. A job that runs out of credits keeps the pages it already fetched.

GET /crawl/{jobId}/results pages through what was fetched:

{  "success": true,  "jobId": "crawl_0123456789abcdef01234567",  "status": "completed",  "totalResults": 50,  "hasMore": false,  "nextCursor": null,  "pages": [    {      "url": "https://example.com/docs/intro",      "finalUrl": null,      "depth": 1,      "parentUrl": "https://example.com/docs",      "status": "completed",      "statusCode": 200,      "via": "http",      "title": "Introduction",      "description": "Get started with Example.",      "language": "en",      "canonicalUrl": "https://example.com/docs/intro",      "wordCount": 640,      "markdown": "# Introduction\n...",      "fetchedAt": "2026-09-29T10:00:05.000Z"    }  ]}

totalResults counts the pages in this response, not the whole job. Skipped pages carry a reason (e.g. disallowed by robots.txt) and failed ones an error. GET /crawl lists your jobs, newest first (limit 1–100, default 20; optional status). DELETE /crawl/{jobId} stops a job; pages already fetched stay billed and readable. Jobs and their pages are kept for a configurable retention period.

Job Webhooks#

When a crawl or batch ends, Searlo POSTs once to webhook.url:

{  "jobId": "crawl_0123456789abcdef01234567",  "event": "crawl.completed",  "status": "completed",  "url": "https://example.com/docs",  "stopReason": "page limit reached",  "progress": { "discovered": 64, "completed": 50, "failed": 2, "skipped": 3 },  "creditsCharged": 50,  "resultsUrl": "https://api.searlo.tech/api/v1/crawl/crawl_0123456789abcdef01234567/results",  "startedAt": "2026-09-29T10:00:02.000Z",  "finishedAt": "2026-09-29T10:04:09.000Z"}

Headers: X-Searlo-Job-Id, X-Searlo-Timestamp, X-Searlo-Event and, when you set a secret, X-Searlo-Signature, signed the same way as Async Tasks webhooks. The URL must be HTTPS and publicly reachable. A job webhook is attempted once and not retried: webhook.attempts and webhook.deliveredAt on the job show whether it landed.

Extract#

URLs plus a field schema in, one record per URL out. The URLs are fetched several at a time inside one overall time budget (75 seconds by default), so a long list takes about as long as its slowest pages, not the sum of them. Pages that arrive in time are served and billed; the rest come back as rows that say why.

Values come first from what the page declares (schema.org JSON-LD, then Open Graph and meta tags), then from unambiguous patterns in the text (prices, currencies, emails, phone numbers), then from the model if it is enabled. That first pass recognises common field names (name, description, price, currency, rating, reviews, image, sku, brand, author, datePublished, availability, url, email, phone, address and their usual aliases); other names are filled only by the model. A field nothing can fill comes back null and is listed in missingFields: it is not guessed. Your balance must cover one credit per URL before the call starts (402 otherwise). Billing is one credit per page actually fetched, or from 2026-11-01, 5 for a row whose via is browser: a row that failed, timed out or was never reached is free.

{  "searchParameters": { "type": "extract", "urls": 3, "fields": ["name", "price", "currency", "availability"] },  "mode": "structured+llm",  "llmAvailable": true,  "totalResults": 3,  "partial": true,  "data": [    {      "url": "https://shop.example.com/p/widget",      "success": true,      "data": { "name": "Widget Pro", "price": 49.99, "currency": "USD", "availability": null },      "_sources": { "name": "json-ld", "price": "json-ld", "currency": "json-ld" },      "missingFields": ["availability"],      "via": "http",      "llmUsed": false,      "llmSkipped": "time budget"    },    {      "url": "https://shop.example.org/p/gadget",      "success": false,      "data": null,      "errorCode": "TARGET_REFUSED",      "error": "target site refused the request (HTTP 403)",      "retryable": false,      "targetStatus": 403    },    {      "url": "https://slow.example.net/p/thing",      "success": false,      "data": null,      "errorCode": "NOT_FETCHED",      "error": "not fetched: the request's time budget ran out before this URL's turn",      "retryable": true    }  ],  "source": "searlo.tech(extract)",  "credits": 1}

_sources gives each filled field's origin: json-ld, meta, text or llm. credits is the number of pages fetched. Every row has url, success and data; the rest depends on the row:

A private or reserved address among the urls fails the whole call with 422 TARGET_NOT_ALLOWED before anything is fetched.

Search and Scrape#

A web search, then the result pages scraped, in one call.

{  "searchParameters": { "q": "vector database benchmarks", "type": "search-and-scrape", "limit": 3, "scrape": true },  "totalResults": 3,  "scrapedCount": 2,  "results": [    {      "position": 1,      "title": "Vector database benchmarks, 2026",      "link": "https://www.example.com/benchmarks",      "snippet": "We ran each database on the same hardware...",      "domain": "example.com",      "scraped": true,      "via": "http",      "statusCode": 200,      "metadata": { "title": "Vector database benchmarks, 2026", "language": "en", "wordCount": 2310 },      "markdown": "# Vector database benchmarks, 2026\n..."    },    { "position": 3, "title": "Benchmark notes", "link": "https://unreachable.example.net/post", "snippet": "...", "domain": "unreachable.example.net", "scraped": false, "error": "scrape failed" }  ],  "source": "searlo.tech(search-and-scrape)",  "credits": 3}

Each row is a web search result (position, title, link, snippet, domain, …) with its page attached. A page that could not be scraped comes back with scraped: false and an error, and is not billed.

Webpage Fetch (GET or POST)#

GET /search/webpage reads one page from a query string, for clients that can only send GET: one URL, one format, the page's metadata. POST /search/webpage takes the same fields as a JSON body, plus excludeSelectors (up to 20 CSS selectors to remove before extraction). 1 credit; from 2026-11-01, 5 when the page needed a browser (via: "browser"). A repeat call can be answered from cache (X-Cache: HIT) and is billed the same; to control freshness, use POST /scrape with maxAge.

{  "searchParameters": { "url": "https://example.com/blog/benchmarks", "type": "webpage", "format": "markdown" },  "url": "https://example.com/blog/benchmarks",  "requestedUrl": "https://example.com/blog/benchmarks",  "domain": "example.com",  "statusCode": 200,  "via": "http",  "bytes": 48213,  "title": "Vector database benchmarks, 2026",  "description": "We ran each database on the same hardware...",  "author": null,  "publishedDate": "2026-03-02",  "openGraph": { "title": "Vector database benchmarks, 2026", "image": "https://example.com/og.png" },  "wordCount": 2310,  "content": "# Vector database benchmarks, 2026\n...",  "source": "searlo.tech(webpage)",  "credits": 1}

url is where the page ended up after redirects; requestedUrl is what you sent. via is http for a plain fetch and browser for a rendered one.

A site that refuses the fetch is 422 TARGET_REFUSED (targetStatus is the site's own status), a private or reserved address is 422 TARGET_NOT_ALLOWED, and a site no attempt could fetch is 422 TARGET_UNREACHABLE, as on POST /scrape. None is charged.

# One page as markdown plus its linkscurl -X POST "https://api.searlo.tech/api/v1/scrape" \  -H "x-api-key: sk_your_api_key" \  -H "Content-Type: application/json" \  -d '{ "url": "https://example.com/pricing", "formats": ["markdown", "links"] }' # Start a crawl, then poll it and page through the resultscurl -X POST "https://api.searlo.tech/api/v1/crawl" \  -H "x-api-key: sk_your_api_key" \  -H "Content-Type: application/json" \  -d '{ "url": "https://example.com/docs", "maxDepth": 2, "limit": 50, "includePaths": ["/docs/*"] }' curl "https://api.searlo.tech/api/v1/crawl/crawl_0123456789abcdef01234567" \  -H "x-api-key: sk_your_api_key" curl "https://api.searlo.tech/api/v1/crawl/crawl_0123456789abcdef01234567/results?limit=50" \  -H "x-api-key: sk_your_api_key" # Structured records from a product pagecurl -X POST "https://api.searlo.tech/api/v1/extract" \  -H "x-api-key: sk_your_api_key" \  -H "Content-Type: application/json" \  -d '{ "urls": ["https://shop.example.com/p/widget"], "schema": { "name": "string", "price": "number", "currency": "string" } }'

const API = "https://api.searlo.tech/api/v1";const headers = { "x-api-key": process.env.SEARLO_API_KEY, "Content-Type": "application/json" }; // Start the crawlconst job = await (await fetch(`${API}/crawl`, {  method: "POST",  headers,  body: JSON.stringify({ url: "https://example.com/docs", maxDepth: 2, limit: 50 }),})).json(); // Poll until it endslet status;do {  await new Promise(r => setTimeout(r, 5000));  status = await (await fetch(`${API}/crawl/${job.jobId}`, { headers })).json();} while (status.status === "queued" || status.status === "running");console.log(status.stopReason, status.progress, status.creditsCharged); // Page through the results by cursorlet cursor = null;do {  const qs = new URLSearchParams({ limit: "200" });  if (cursor) qs.set("cursor", cursor);  const page = await (await fetch(`${API}/crawl/${job.jobId}/results?${qs}`, { headers })).json();  page.pages.forEach(p => console.log(p.url, p.wordCount));  cursor = page.nextCursor;} while (cursor);

import osimport requests API = "https://api.searlo.tech/api/v1"HEADERS = {"x-api-key": os.environ["SEARLO_API_KEY"]} # Accept a cached copy up to an hour old; a live fetch forced by maxAge costs 2page = requests.post(    f"{API}/scrape",    json={"url": "https://example.com/pricing", "formats": ["markdown", "tables"], "maxAge": 3600},    headers=HEADERS,    timeout=90,)print(page.headers.get("X-Cache"), page.headers.get("X-Credits-Deducted"))print(page.json()["markdown"][:500]) out = requests.post(    f"{API}/extract",    json={        "urls": ["https://shop.example.com/p/widget"],        "schema": {"name": "string", "price": "number", "currency": "string", "availability": "string"},    },    headers=HEADERS,    timeout=120,).json()if out.get("partial"):    # The time budget ran out: these rows can be sent again    print("retry:", [r["url"] for r in out["data"] if not r["success"] and r["retryable"]])for row in out["data"]:    print(row["url"], row.get("data"), row.get("_sources"), row.get("missingFields"))