> Searlo API Docs — Web Data (Scrape, Map, Crawl, Extract)
> Source: https://searlo.tech/docs#web-data
> Full docs: https://searlo.tech/docs/full.md

# Web Data (Scrape, Map, Crawl, Extract)

The retrieval half of the API: read a page, list a site's URLs, crawl a site, or pull structured records out of pages. Pages that render client-side are rendered automatically. That automatic rendering is for accounts that have bought credits; on sign-up credits alone, pass `render: true` to render a page.

### Endpoints

```
POST   /scrape
POST   /scrape/batch
POST   /map
POST   /crawl
GET    /crawl
GET    /crawl/{jobId}
GET    /crawl/{jobId}/results
DELETE /crawl/{jobId}
POST   /extract
POST   /search-and-scrape
GET    /search/webpage
```

**Credits:**

| Endpoint | Credits |
|----------|---------|
| `POST /scrape`, `POST /map` | 1; 2 when your `maxAge` or `cache: false` forced a live fetch |
| `POST /scrape/batch`, `POST /crawl` | 1 per page fetched successfully, charged while the job runs |
| `POST /extract` | 1 per page fetched; a URL that failed, timed out or was not reached is free |
| `POST /search-and-scrape` | 1 for the search, plus 1 per page scraped, plus 1 per page fetched live when you set `maxAge` or `cache: false` |
| `GET /search/webpage` | 1 (a cached copy is billed the same) |
| `GET` and `DELETE` on `/crawl` | Free |

**From 2026-11-01: pages that needed a browser cost 5.** On every row above except `/map`, a page that had to be rendered costs 4 credits more than the table says: 5 instead of 1 per page. That covers `render: true`, `actions`, and automatic rendering when a plain fetch returns no usable content, such as a JavaScript-only page. The page's `via` says which it was: `browser` or `http`. A cached copy costs the same as the page it copies; a failed fetch is never charged; the live-fetch surcharge is added on top, not multiplied (a forced live fetch that needed a browser is 2 + 4 = 6). `render: false` keeps every page at the plain price. Until 2026-11-01 a browser page costs the same as any other.

### Freshness and the Live-Fetch Surcharge

`/scrape`, `/map` and `/search-and-scrape` keep a cache, controlled by two fields:

- Omit both, and any cached copy is used. The call costs 1 whether or not it was cached (from 2026-11-01, 5 for a page that needed a browser, cached or not).
- `maxAge` (seconds, 0–2592000): use a cached copy only if it is at most this old.
- `cache: false` is the same as `maxAge: 0`: always fetch live.

When your `maxAge` or `cache: false` makes Searlo fetch the page live, the call costs 2 and the response carries `X-Cache-Bypass: CHARGED`. If a cached copy satisfied your `maxAge`, it costs 1. `X-Credits-Deducted` always shows the total, and a cached answer carries `_cacheAgeSec`.

### URLs

Every `url` and `urls` field takes an http or https URL of up to 2,048 characters. An internationalised domain name works as you write it on `scrape`, `map`, `crawl`, `scrape/batch` and `extract`: `https://www.löschwagen.com/preise` is fetched, cached and billed under its encoded form, `https://www.xn--lschwagen-07a.com/preise`, and so is a path in any script. (`GET /search/webpage` takes the encoded `xn--` form only.)

A URL that cannot be fetched as written is a `400` `VALIDATION_ERROR` (`url must be an http(s) URL`), answered before anything is fetched and never charged: a control character or a space anywhere in it, an escaped one in the host (`%00`), a scheme other than http or https, or a URL that cannot be parsed. A private or reserved address is `422` `TARGET_NOT_ALLOWED`.

### Scrape

```
POST /scrape
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `url` | string | Yes | http(s) URL, max 2048 chars |
| `formats` | string[] | No | Any of `markdown`, `text`, `html`, `links`, `images`, `headings`, `tables`, `json_ld` (default: `["markdown"]`) |
| `onlyMainContent` | boolean | No | Strip navigation, headers and footers (default: true) |
| `render` | boolean | No | `true` always renders the page; `false` never renders. By default a page is rendered only when the plain fetch returns no usable content (an empty JavaScript shell, or a page the site refused), on accounts that have bought credits. From 2026-11-01 a rendered page costs 5 |
| `includeLinks` | boolean | No | `false` removes links from the content and omits `links` unless `formats` asks for it |
| `includeImages` | boolean | No | `false` removes images from the content |
| `excludeSelectors` | string[] | No | CSS selectors to remove before extraction (up to 20) |
| `chunk` | boolean | No | Also split the markdown into `chunks` for embedding |
| `chunkMaxTokens` | number | No | Chunk size (256–16384) |
| `waitForSelector` | string | No | Wait for this selector before reading the page |
| `timeout` | number | No | Milliseconds (1000–60000) |
| `actions` | object[] | No | Up to 20 steps run in the page before it is read; implies `render: true`. `type` is one of `click`, `type`, `press`, `scroll`, `wait`, `waitForSelector`, `evaluate`, with `selector`, `text`, `key`, `ms` (0–15000) or `script` as the step needs. Each step's outcome comes back in `actions` |
| `cache` | boolean | No | Default: true (see Freshness) |
| `maxAge` | number | No | Seconds (see Freshness) |

```json
{
  "searchParameters": { "url": "https://example.com/pricing", "type": "scrape", "formats": ["markdown", "links"] },
  "url": "https://example.com/pricing",
  "requestedUrl": "https://example.com/pricing",
  "statusCode": 200,
  "via": "http",
  "bytes": 48213,
  "markdown": "# Pricing\n\nStart free, then pay as you go...",
  "metadata": {
    "title": "Pricing | Example",
    "description": "Plans and prices.",
    "author": null,
    "publishedDate": null,
    "language": "en",
    "canonicalUrl": "https://example.com/pricing",
    "robots": null,
    "wordCount": 812,
    "openGraph": { "title": "Pricing", "image": "https://example.com/og.png" },
    "twitter": null
  },
  "links": [{ "url": "https://example.com/signup", "text": "Sign up", "internal": true }],
  "source": "searlo.tech(scrape)",
  "credits": 1
}
```

`url` is the final URL after redirects and `requestedUrl` is what you sent. `via` is `http` or `browser` (`serp` for a Google results page). A page that comes back empty is returned but not cached.

| Status | `errorCode` | Meaning | Charged |
|--------|-------------|---------|---------|
| `422` | `TARGET_REFUSED` | The site refused every attempt, or answered with an error of its own. `targetStatus` holds the site's own HTTP status; `retryable` is `true` only when that was a `429` or a `5xx` | No |
| `422` | `TARGET_NOT_ALLOWED` | The URL points to a private or reserved network address (localhost, a private range, link-local, a cloud metadata address). Only pages on the public internet are fetched (`retryable: false`) | No |
| `422` | `TARGET_UNREACHABLE` | The site could not be fetched at all: its server refuses the secure-connection handshake, it redirects in a loop, or the page is larger than the size limit. `targetError` says which (`retryable: false`) | No |

### Map

```
POST /map
```

Every URL a site publishes, before you pay to crawl it: robots.txt, then its sitemaps (sitemap indexes included), then the homepage's links when there is no sitemap.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `url` | string | Yes | The site, max 2048 chars |
| `limit` | number | No | Maximum URLs (1–5000, default: 1000) |
| `search` | string | No | Only URLs containing this text (max 200 chars) |
| `includeSubdomains` | boolean | No | Include subdomains |
| `sitemapOnly` | boolean | No | Sitemaps only, no homepage-link fallback |
| `cache`, `maxAge` | | No | See Freshness |

```json
{
  "searchParameters": { "url": "https://example.com", "type": "map", "limit": 1000 },
  "origin": "https://example.com",
  "discovery": "sitemap",
  "sitemaps": ["https://example.com/sitemap.xml"],
  "robots": { "present": true, "sitemapCount": 1 },
  "totalResults": 214,
  "urls": [
    { "url": "https://example.com/pricing", "lastModified": "2026-09-01" },
    { "url": "https://example.com/blog/launch", "lastModified": null }
  ],
  "source": "searlo.tech(map)",
  "credits": 1
}
```

`discovery` is `sitemap`, `links` (homepage links, when the site publishes no sitemap) or `none`.

### Crawl

```
POST /crawl
```

Queue a crawl of one site and get a job id back at once. Starting needs a minimum balance (configurable); after that, credits are charged per page as the crawl runs, so a job with `limit: 500` on a twelve-page site costs 12 (from 2026-11-01, plus 4 for each of those pages that needed a browser). robots.txt is obeyed by default.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `url` | string | Yes | Start URL |
| `maxDepth` | number | No | Link depth from the start URL (0–10, default: 2) |
| `limit` | number | No | Maximum pages (1–5000, default: 100) |
| `includePaths` | string[] | No | Only follow paths matching these globs, e.g. `/blog/*` (up to 25) |
| `excludePaths` | string[] | No | Skip paths matching these globs (up to 25) |
| `formats` | string[] | No | `markdown`, `text`, `html`, `links`, `json_ld` (default: `["markdown"]`) |
| `allowSubdomains` | boolean | No | Follow links to subdomains (default: false) |
| `allowExternal` | boolean | No | Follow links off the site (default: false) |
| `respectRobots` | boolean | No | Obey robots.txt (default: true) |
| `useSitemap` | boolean | No | Seed the crawl from the site's sitemaps (default: true) |
| `concurrency` | number | No | Pages fetched at once (1–8, default: 3) |
| `render` | boolean | No | `true` always renders the page; `false` never renders. By default a page is rendered only when the plain fetch returns no usable content (an empty JavaScript shell, or a page the site refused), on accounts that have bought credits. From 2026-11-01 a rendered page costs 5 |
| `onlyMainContent` | boolean | No | Default: true |
| `storeContent` | boolean | No | `false` keeps a link graph with no page bodies (default: true) |
| `webhook` | object | No | `{ "url": "https://…", "secret": "…" }`, called once when the crawl ends |

The snake_case spellings (`max_depth`, `include_paths`, `exclude_paths`, `allow_subdomains`, `allow_external`, `respect_robots`, `use_sitemap`, `only_main_content`, `store_content`) are accepted too.

```json
{
  "success": true,
  "jobId": "crawl_0123456789abcdef01234567",
  "status": "queued",
  "url": "https://example.com/docs",
  "options": { "maxDepth": 2, "limit": 50, "includePaths": ["/docs/*"], "excludePaths": [], "formats": ["markdown"], "allowSubdomains": false, "allowExternal": false, "respectRobots": true, "concurrency": 3, "render": null, "onlyMainContent": true, "useSitemap": true, "storeContent": true },
  "maxCredits": 50,
  "statusUrl": "/api/v1/crawl/crawl_0123456789abcdef01234567",
  "resultsUrl": "/api/v1/crawl/crawl_0123456789abcdef01234567/results",
  "createdAt": "2026-09-29T10:00:00.000Z"
}
```

The response is `202`. `maxCredits` is a ceiling, not a quote; from 2026-11-01 it allows 5 per page unless the job sets `render: false`.

### Batch Scrape

```
POST /scrape/batch
```

Up to 1000 URLs as one job, with no discovery: the job fetches exactly the URLs you send. They are normalised and de-duplicated first, so compare `submitted` with `urlCount`. Like a crawl, it needs a minimum balance to start and then bills 1 credit per page fetched (from 2026-11-01, 5 for a page that needed a browser). Status, results, cancellation and webhooks use the crawl endpoints below.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `urls` | string[] | Yes | 1–1000 http(s) URLs |
| `formats` | string[] | No | `markdown`, `text`, `html`, `links`, `json_ld` (default: `["markdown"]`). `images`, `headings` and `tables` are accepted but not stored |
| `render` | boolean | No | `true` always renders the page; `false` never renders. By default a page is rendered only when the plain fetch returns no usable content (an empty JavaScript shell, or a page the site refused), on accounts that have bought credits. From 2026-11-01 a rendered page costs 5 |
| `onlyMainContent` | boolean | No | Default: true |
| `concurrency` | number | No | Pages fetched at once (1–8, default: 3) |
| `storeContent` | boolean | No | `false` keeps only metadata and links (default: true) |
| `webhook` | object | No | `{ "url": "https://…", "secret": "…" }`, called once when the job ends |

`maxAge`, `cache`, `includeLinks` and `excludeSelectors` are accepted on a batch but have no effect: every page is fetched live, at 1 credit (5 from 2026-11-01 if it needed a browser).

The `202` response has the crawl shape plus `mode: "batch"`, `urlCount` (what was queued) and `submitted` (what you sent).

### Job Status and Results

```
GET /crawl/{jobId}
GET /crawl/{jobId}/results
GET /crawl
DELETE /crawl/{jobId}
```

All free, for crawls and batches alike.

```json
{
  "success": true,
  "jobId": "crawl_0123456789abcdef01234567",
  "status": "completed",
  "url": "https://example.com/docs",
  "options": { "maxDepth": 2, "limit": 50, "includePaths": ["/docs/*"], "excludePaths": [], "formats": ["markdown"], "allowSubdomains": false, "allowExternal": false, "respectRobots": true, "concurrency": 3, "render": null, "onlyMainContent": true, "useSitemap": true, "storeContent": true },
  "progress": { "discovered": 64, "completed": 50, "failed": 2, "skipped": 3 },
  "creditsCharged": 50,
  "stopReason": "page limit reached",
  "webhook": { "url": "https://example.com/hooks/crawl", "deliveredAt": "2026-09-29T10:04:10.000Z", "attempts": 1 },
  "createdAt": "2026-09-29T10:00:00.000Z",
  "startedAt": "2026-09-29T10:00:02.000Z",
  "finishedAt": "2026-09-29T10:04:09.000Z"
}
```

`status` is `queued`, `running`, `completed`, `failed` or `cancelled`. `stopReason` says how a job ended: `page limit reached`, `all urls processed` (a batch), `credit balance exhausted`, `job deadline reached` (jobs run for a configurable maximum time) or `cancelled by caller`. A job that runs out of credits keeps the pages it already fetched.

`GET /crawl/{jobId}/results` pages through what was fetched:

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `limit` | number | No | Pages per response (1–200, default: 50) |
| `cursor` | string | No | `nextCursor` from the previous response |
| `status` | string | No | `completed` (default), `failed`, `skipped`, `queued` or `all` |

```json
{
  "success": true,
  "jobId": "crawl_0123456789abcdef01234567",
  "status": "completed",
  "totalResults": 50,
  "hasMore": false,
  "nextCursor": null,
  "pages": [
    {
      "url": "https://example.com/docs/intro",
      "finalUrl": null,
      "depth": 1,
      "parentUrl": "https://example.com/docs",
      "status": "completed",
      "statusCode": 200,
      "via": "http",
      "title": "Introduction",
      "description": "Get started with Example.",
      "language": "en",
      "canonicalUrl": "https://example.com/docs/intro",
      "wordCount": 640,
      "markdown": "# Introduction\n...",
      "fetchedAt": "2026-09-29T10:00:05.000Z"
    }
  ]
}
```

`totalResults` counts the pages in this response, not the whole job. Skipped pages carry a `reason` (e.g. `disallowed by robots.txt`) and failed ones an `error`. `GET /crawl` lists your jobs, newest first (`limit` 1–100, default 20; optional `status`). `DELETE /crawl/{jobId}` stops a job; pages already fetched stay billed and readable. Jobs and their pages are kept for a configurable retention period.

### Job Webhooks

When a crawl or batch ends, Searlo POSTs once to `webhook.url`:

```json
{
  "jobId": "crawl_0123456789abcdef01234567",
  "event": "crawl.completed",
  "status": "completed",
  "url": "https://example.com/docs",
  "stopReason": "page limit reached",
  "progress": { "discovered": 64, "completed": 50, "failed": 2, "skipped": 3 },
  "creditsCharged": 50,
  "resultsUrl": "https://api.searlo.tech/api/v1/crawl/crawl_0123456789abcdef01234567/results",
  "startedAt": "2026-09-29T10:00:02.000Z",
  "finishedAt": "2026-09-29T10:04:09.000Z"
}
```

Headers: `X-Searlo-Job-Id`, `X-Searlo-Timestamp`, `X-Searlo-Event` and, when you set a secret, `X-Searlo-Signature`, signed the same way as [Async Tasks](#async-tasks) webhooks. The URL must be HTTPS and publicly reachable. A job webhook is attempted once and not retried: `webhook.attempts` and `webhook.deliveredAt` on the job show whether it landed.

### Extract

```
POST /extract
```

URLs plus a field schema in, one record per URL out. The URLs are fetched several at a time inside one overall time budget (75 seconds by default), so a long list takes about as long as its slowest pages, not the sum of them. Pages that arrive in time are served and billed; the rest come back as rows that say why.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `urls` | string[] | One of `urls`, `url` | 1–20 http(s) URLs |
| `url` | string | One of `urls`, `url` | A single URL |
| `schema` | object | Yes | 1–30 fields, each name mapped to a type: `string`, `number`, `integer`, `float`, `boolean` or `array`. Names use letters, digits and `_`, do not start with a digit, max 64 chars |
| `useLlm` | boolean | No | Use a language model for fields the page does not declare (default: true). It only has an effect when the deployment has a model configured; `llmAvailable` in the response says whether it does |
| `onlyMainContent` | boolean | No | Default: true |

Values come first from what the page declares (schema.org JSON-LD, then Open Graph and meta tags), then from unambiguous patterns in the text (prices, currencies, emails, phone numbers), then from the model if it is enabled. That first pass recognises common field names (`name`, `description`, `price`, `currency`, `rating`, `reviews`, `image`, `sku`, `brand`, `author`, `datePublished`, `availability`, `url`, `email`, `phone`, `address` and their usual aliases); other names are filled only by the model. A field nothing can fill comes back `null` and is listed in `missingFields`: it is not guessed. Your balance must cover one credit per URL before the call starts (`402` otherwise). Billing is one credit per page actually fetched, or from 2026-11-01, 5 for a row whose `via` is `browser`: a row that failed, timed out or was never reached is free.

```json
{
  "searchParameters": { "type": "extract", "urls": 3, "fields": ["name", "price", "currency", "availability"] },
  "mode": "structured+llm",
  "llmAvailable": true,
  "totalResults": 3,
  "partial": true,
  "data": [
    {
      "url": "https://shop.example.com/p/widget",
      "success": true,
      "data": { "name": "Widget Pro", "price": 49.99, "currency": "USD", "availability": null },
      "_sources": { "name": "json-ld", "price": "json-ld", "currency": "json-ld" },
      "missingFields": ["availability"],
      "via": "http",
      "llmUsed": false,
      "llmSkipped": "time budget"
    },
    {
      "url": "https://shop.example.org/p/gadget",
      "success": false,
      "data": null,
      "errorCode": "TARGET_REFUSED",
      "error": "target site refused the request (HTTP 403)",
      "retryable": false,
      "targetStatus": 403
    },
    {
      "url": "https://slow.example.net/p/thing",
      "success": false,
      "data": null,
      "errorCode": "NOT_FETCHED",
      "error": "not fetched: the request's time budget ran out before this URL's turn",
      "retryable": true
    }
  ],
  "source": "searlo.tech(extract)",
  "credits": 1
}
```

`_sources` gives each filled field's origin: `json-ld`, `meta`, `text` or `llm`. `credits` is the number of pages fetched. Every row has `url`, `success` and `data`; the rest depends on the row:

| Field | Description |
|-------|-------------|
| `success` | `true` when the page was fetched (1 credit), `false` when it was not (free, with `data: null`) |
| `errorCode` | On a failed row: `TARGET_REFUSED` (the site refused; `targetStatus` is its own HTTP status), `TARGET_NOT_ALLOWED` (a private or reserved address), `TARGET_UNREACHABLE` (the site cannot be fetched at all), `INVALID_REQUEST` (the URL was not accepted), `SCRAPE_FAILED` (the page could not be fetched right now), `TIMEOUT` (it did not arrive inside the time budget) or `NOT_FETCHED` (the budget ran out before this URL's turn) |
| `error` | On a failed row: what happened, in a sentence |
| `retryable` | On a failed row: whether the same URL may succeed on a retry. `true` for `SCRAPE_FAILED`, `TIMEOUT`, `NOT_FETCHED` and a refusal that was the site's own `429` or `5xx`; a site's `401` or `403` will not change |
| `partial` | At the top level, `true` when the time budget ran out before every URL was fetched. The rows say which (`TIMEOUT`, `NOT_FETCHED`); the rest are served and billed. Absent otherwise |
| `llmSkipped` | On a fetched row: `"time budget"` when the page came in too late to ask the model for its missing fields. `missingFields` lists them; send that URL again on its own to give the model its time |

A private or reserved address among the `urls` fails the whole call with `422` `TARGET_NOT_ALLOWED` before anything is fetched.

### Search and Scrape

```
POST /search-and-scrape
```

A web search, then the result pages scraped, in one call.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `q` | string | Yes | Search query (1–500 chars) |
| `limit` | number | No | Results to scrape (1–20, default: 5). The search step returns at most 10 |
| `scrape` | boolean | No | `false` returns the search results only, for 1 credit (default: true) |
| `formats` | string[] | No | `markdown`, `text`, `html`, `json_ld` (default: `["markdown"]`) |
| `onlyMainContent` | boolean | No | Default: true |
| `render` | boolean | No | `true` always renders the page; `false` never renders. By default a page is rendered only when the plain fetch returns no usable content (an empty JavaScript shell, or a page the site refused), on accounts that have bought credits. From 2026-11-01 a rendered page costs 5 |
| `gl` | string | No | Country code, max 5 chars |
| `hl` | string | No | Language code, max 5 chars |
| `cache`, `maxAge` | | No | Apply to the scraped pages (see Freshness) |

```json
{
  "searchParameters": { "q": "vector database benchmarks", "type": "search-and-scrape", "limit": 3, "scrape": true },
  "totalResults": 3,
  "scrapedCount": 2,
  "results": [
    {
      "position": 1,
      "title": "Vector database benchmarks, 2026",
      "link": "https://www.example.com/benchmarks",
      "snippet": "We ran each database on the same hardware...",
      "domain": "example.com",
      "scraped": true,
      "via": "http",
      "statusCode": 200,
      "metadata": { "title": "Vector database benchmarks, 2026", "language": "en", "wordCount": 2310 },
      "markdown": "# Vector database benchmarks, 2026\n..."
    },
    { "position": 3, "title": "Benchmark notes", "link": "https://unreachable.example.net/post", "snippet": "...", "domain": "unreachable.example.net", "scraped": false, "error": "scrape failed" }
  ],
  "source": "searlo.tech(search-and-scrape)",
  "credits": 3
}
```

Each row is a web search result (`position`, `title`, `link`, `snippet`, `domain`, …) with its page attached. A page that could not be scraped comes back with `scraped: false` and an `error`, and is not billed.

### Webpage Fetch (GET or POST)

`GET /search/webpage` reads one page from a query string, for clients that can only send GET: one URL, one format, the page's metadata. `POST /search/webpage` takes the same fields as a JSON body, plus `excludeSelectors` (up to 20 CSS selectors to remove before extraction). 1 credit; from 2026-11-01, 5 when the page needed a browser (`via: "browser"`). A repeat call can be answered from cache (`X-Cache: HIT`) and is billed the same; to control freshness, use `POST /scrape` with `maxAge`.

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `url` | string | Yes | An http(s) URL, max 2048 chars |
| `format` | string | No | `markdown` (default), `text` or `html` |
| `render` | boolean | No | `true` always renders the page; `false` never does. By default a page is rendered only when the plain fetch returns no usable content (an empty JavaScript shell, or a page the site refused), on accounts that have bought credits. From 2026-11-01 a rendered page costs 5 |
| `onlyMainContent` | boolean | No | Strip navigation, headers and footers (default: true) |
| `includeLinks` | boolean | No | `false` drops links from the content (default: true) |
| `includeImages` | boolean | No | `false` drops images from the content (default: true) |
| `chunk` | boolean | No | Also return `chunks`: the content split into overlapping pieces for embedding |
| `chunkMaxTokens` | number | No | Chunk size, 256–16384 tokens |
| `timeout` | number | No | Milliseconds, 1000–60000 |

```json
{
  "searchParameters": { "url": "https://example.com/blog/benchmarks", "type": "webpage", "format": "markdown" },
  "url": "https://example.com/blog/benchmarks",
  "requestedUrl": "https://example.com/blog/benchmarks",
  "domain": "example.com",
  "statusCode": 200,
  "via": "http",
  "bytes": 48213,
  "title": "Vector database benchmarks, 2026",
  "description": "We ran each database on the same hardware...",
  "author": null,
  "publishedDate": "2026-03-02",
  "openGraph": { "title": "Vector database benchmarks, 2026", "image": "https://example.com/og.png" },
  "wordCount": 2310,
  "content": "# Vector database benchmarks, 2026\n...",
  "source": "searlo.tech(webpage)",
  "credits": 1
}
```

`url` is where the page ended up after redirects; `requestedUrl` is what you sent. `via` is `http` for a plain fetch and `browser` for a rendered one.

A site that refuses the fetch is `422` `TARGET_REFUSED` (`targetStatus` is the site's own status), a private or reserved address is `422` `TARGET_NOT_ALLOWED`, and a site no attempt could fetch is `422` `TARGET_UNREACHABLE`, as on `POST /scrape`. None is charged.

## Code Examples

### cURL

```bash
# One page as markdown plus its links
curl -X POST "https://api.searlo.tech/api/v1/scrape" \
  -H "x-api-key: sk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com/pricing", "formats": ["markdown", "links"] }'

# Start a crawl, then poll it and page through the results
curl -X POST "https://api.searlo.tech/api/v1/crawl" \
  -H "x-api-key: sk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com/docs", "maxDepth": 2, "limit": 50, "includePaths": ["/docs/*"] }'

curl "https://api.searlo.tech/api/v1/crawl/crawl_0123456789abcdef01234567" \
  -H "x-api-key: sk_your_api_key"

curl "https://api.searlo.tech/api/v1/crawl/crawl_0123456789abcdef01234567/results?limit=50" \
  -H "x-api-key: sk_your_api_key"

# Structured records from a product page
curl -X POST "https://api.searlo.tech/api/v1/extract" \
  -H "x-api-key: sk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{ "urls": ["https://shop.example.com/p/widget"], "schema": { "name": "string", "price": "number", "currency": "string" } }'
```

### JavaScript

```javascript
const API = "https://api.searlo.tech/api/v1";
const headers = { "x-api-key": process.env.SEARLO_API_KEY, "Content-Type": "application/json" };

// Start the crawl
const job = await (await fetch(`${API}/crawl`, {
  method: "POST",
  headers,
  body: JSON.stringify({ url: "https://example.com/docs", maxDepth: 2, limit: 50 }),
})).json();

// Poll until it ends
let status;
do {
  await new Promise(r => setTimeout(r, 5000));
  status = await (await fetch(`${API}/crawl/${job.jobId}`, { headers })).json();
} while (status.status === "queued" || status.status === "running");
console.log(status.stopReason, status.progress, status.creditsCharged);

// Page through the results by cursor
let cursor = null;
do {
  const qs = new URLSearchParams({ limit: "200" });
  if (cursor) qs.set("cursor", cursor);
  const page = await (await fetch(`${API}/crawl/${job.jobId}/results?${qs}`, { headers })).json();
  page.pages.forEach(p => console.log(p.url, p.wordCount));
  cursor = page.nextCursor;
} while (cursor);
```

### Python

```python
import os
import requests

API = "https://api.searlo.tech/api/v1"
HEADERS = {"x-api-key": os.environ["SEARLO_API_KEY"]}

# Accept a cached copy up to an hour old; a live fetch forced by maxAge costs 2
page = requests.post(
    f"{API}/scrape",
    json={"url": "https://example.com/pricing", "formats": ["markdown", "tables"], "maxAge": 3600},
    headers=HEADERS,
    timeout=90,
)
print(page.headers.get("X-Cache"), page.headers.get("X-Credits-Deducted"))
print(page.json()["markdown"][:500])

out = requests.post(
    f"{API}/extract",
    json={
        "urls": ["https://shop.example.com/p/widget"],
        "schema": {"name": "string", "price": "number", "currency": "string", "availability": "string"},
    },
    headers=HEADERS,
    timeout=120,
).json()
if out.get("partial"):
    # The time budget ran out: these rows can be sent again
    print("retry:", [r["url"] for r in out["data"] if not r["success"] and r["retryable"]])
for row in out["data"]:
    print(row["url"], row.get("data"), row.get("_sources"), row.get("missingFields"))
```


---
*From https://searlo.tech/docs — see full docs at https://searlo.tech/docs/full.md*