webaccess API

Overview

webaccess exposes a small REST API. You point it at a domain or a URL, it fetches the content behind SSRF protection and politeness rules, and returns the extracted result. It never invents data: if a page cannot be fetched, the run reports it as failed rather than fabricating content.

Base URL

https://api.webaccess.searchestra.com

Conventions

• All request and response bodies are application/json.
• Successful responses are wrapped in a top level data field. Errors return a top level error field (see Errors).
• Every /v1/* endpoint requires the X-API-Key header. /healthz does not.
• You pass your own clientRef on every request; it scopes and isolates your data (see Authentication).
• Timestamps are UTC, ISO 8601 (for example 2026-08-24T10:12:00Z).

Quickstart

Start a crawl, then poll for its status. Replace wa_live_your_key with the API key issued to your tenant (see Authentication).

curl -X POST https://api.webaccess.searchestra.com/v1/crawl/start \
  -H "X-API-Key: wa_live_your_key" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "godiva.com.tr",
    "scope": "domain",
    "clientRef": "brand-123"
  }'

{
  "data": {
    "runId": "01a0347a-3cd9-70e1-9dce-c2dd5962a4e6",
    "status": "pending"
  }
}

Then poll the run until it is done or failed, and read the extracted pages:

curl "https://api.webaccess.searchestra.com/v1/crawl/pages?clientRef=brand-123&domain=godiva.com.tr" \
  -H "X-API-Key: wa_live_your_key"

Authentication

Every /v1/* request must carry an API key in the X-API-Key header. Keys begin with the wa_live_ prefix and identify a tenant, not a user.

X-API-Key: wa_live_your_key

The server stores only a hash of the key. The raw value is shown once, at creation, and cannot be recovered. If a key is lost, issue a new one.

Multi-tenancy and clientRef

On every request you also pass your own project or brand identifier, clientRef. The service combines it with your authenticated tenant into a composite key, so data isolation is automatic: two different customers using the same clientRef never see each other's data.

Start a crawl

Starts an asynchronous crawl for a domain and returns a runId immediately. The work runs in the background: track it with GET /v1/crawl/{runId} or wait for the webhook.

If the same domain was crawled recently for this project (default 24 hours), the service returns the last run instead of re-crawling. Set force: true to skip that shortcut and re-crawl from scratch.

Request body

{ "data": { "runId": "01a0347a-3cd9-70e1-9dce-c2dd5962a4e6", "status": "pending" } }

Get run status

Returns the current status and progress counters for a crawl run. Poll this endpoint until status is done or failed. Pass your clientRef as a query parameter.

{
  "data": {
    "runId": "01a0347a-3cd9-70e1-9dce-c2dd5962a4e6",
    "status": "running",
    "domain": "godiva.com.tr",
    "scope": "domain",
    "pagesFound": 90,
    "pagesCrawled": 74,
    "pagesFailed": 2
  }
}

status is one of pending, running, done, failed. When failed, an error field explains why.

Get the latest run

Returns the project's most recent crawl run without needing the runId: the answer to "is a crawl active right now, and what was crawled last?" If there is no run yet, data is an empty object ({}).

List crawled pages

Returns the project's crawled pages with their extracted content: title, description, visible text, status code, canonical URL and structured data. Raw HTML is included by default; pass summary=true for a lighter response without it.

Query parameters

{
  "data": [
    {
      "url": "https://godiva.com.tr/urunler",
      "canonicalUrl": "https://godiva.com.tr/urunler",
      "title": "Products",
      "description": "Godiva chocolate collection",
      "textExcerpt": "Handmade Belgian chocolates...",
      "statusCode": 200,
      "fetchedVia": "http",
      "depth": 1,
      "crawledAt": "2026-08-24T10:12:00Z"
    }
  ]
}

fetchedVia is http for a plain fetch, or browser when the page needed a real headless browser.

Index a project

Chunks all of the project's crawled pages at paragraph boundaries for search and RAG, and, if an embedding provider is configured, generates vectors. Without embeddings the chunks are still stored; only classic full-text search works.

{ "clientRef": "brand-123" }

{ "data": { "pagesIndexed": 74, "chunksCreated": 210, "chunksEmbedded": 210 } }

chunksEmbedded is 0 when no embedding provider is configured.

Fetch a single URL

Fetches one address with raw HTML, all HTTP headers and the full redirect chain, plus timing metrics (TTFB, total time) and body size.

{ "url": "https://competitora.com" }

{
  "data": {
    "html": "...",
    "headers": { "Content-Type": ["text/html"] },
    "statusCode": 200,
    "finalUrl": "https://competitora.com/",
    "redirectChain": [ { "url": "http://competitora.com", "status": 301 } ],
    "ttfbMs": 120,
    "totalMs": 480,
    "bodyBytes": 51234,
    "fetchedVia": "http",
    "robotsBlocked": false,
    "fromCache": false
  }
}

Measure Core Web Vitals

Opens a real headless browser and measures a page's performance metrics: LCP (Largest Contentful Paint), CLS (Cumulative Layout Shift) and TBT (Total Blocking Time).

{ "url": "https://competitora.com" }

{ "data": { "lcp": 2100, "cls": 0.04, "tbt": 150, "ttfb": 180, "lcpSupported": true } }

Support flags (lcpSupported, clsSupported, ...) tell you whether each metric could be observed for this page.

Fetch robots.txt

Returns a domain's robots.txt rules as raw text.

{ "domain": "competitora.com" }

{ "data": { "found": true, "rawBody": "User-agent: *\nDisallow: /admin" } }

Fetch sitemap URLs

Lists the addresses in a domain's sitemap. Index sitemaps and gzip-compressed (.xml.gz) sitemaps are supported. You can hint known sitemap paths with hints.

{ "domain": "competitora.com", "hints": [] }

{ "data": { "urls": ["https://competitora.com/product-1", "https://competitora.com/product-2"] } }

Health check

Liveness probe for container orchestration. No authentication. Returns 200 when the service is up.

Scopes

The scope field on crawl/start controls how far a crawl reaches from the seed domain.

Webhook

When a run completes, and if your tenant registered a webhook URL, webaccess sends an HMAC-SHA256 signed event so you do not have to poll.

{
  "event": "crawl.completed",
  "runId": "01a0347a-3cd9-70e1-9dce-c2dd5962a4e6",
  "clientRef": "brand-123",
  "status": "done",
  "pagesCrawled": 74,
  "pagesFailed": 2
}

Verify the signature header against your WEBHOOK_SECRET before trusting the payload.

Errors

Errors return an error object with a code and a message.

{ "error": { "code": "unauthorized", "message": "Invalid API key." } }

Limits and defaults

Crawl behaviour is bounded by configurable defaults. These protect both the target and your budget.

These are deployment-level defaults; maxPages can be overridden per run on crawl/start.