API documentation

Create an account, generate an API key from your dashboard, then make your first call. Most teams are live in under 10 minutes.

curl https://api.trawl.dev/v1/scrape \
  -H "Authorization: Bearer tk_live_..." \
  -d '{"url": "https://example.com"}'

All requests authenticate with a bearer token. Pass your API key in the Authorization header. Keys are shown once at creation — store them securely.

Authorization: Bearer tk_live_xxxxxxxx_xxxxxxxxxxxxxxxxxxxx

Requests without a valid, non-revoked key return 401 unauthorized.

POST /v1/scrape — fetch any URL and return clean markdown or HTML with title and metadata.

curl https://api.trawl.dev/v1/scrape \
  -H "Authorization: Bearer $TRAWL_API_KEY" \
  -d '{"url": "https://example.com", "format": "markdown"}'

{
  "data": {
    "url": "https://example.com",
    "title": "Example Domain",
    "description": null,
    "format": "markdown",
    "content": "Example Domain\n==============\n\nThis domain is for use...",
    "word_count": 24,
    "fetched_status": 200,
    "rendered": false
  },
  "credits": { "cost": 1, "remaining": 499 }
}

Parameters: url (required), format — markdown (default) or html, render — set true to evaluate the page in a real browser before scraping (JS-rendered sites). Empty client-rendered shells are upgraded to a render automatically; the rendered flag reports which path ran.

Caching: results are cached briefly by URL + options across all endpoints; cache hits still cost credits and set credits.cached: true. Pass "cache": false to force a fresh fetch.

Idempotency: send an Idempotency-Key header to safely retry a billed request — the first response is replayed (no re-charge) for repeats, and reusing a key with a different body returns 422.

POST /v1/html — same as Scrape but returns denoised HTML (scripts, nav, footers stripped) instead of markdown.

curl https://api.trawl.dev/v1/html \
  -H "Authorization: Bearer $TRAWL_API_KEY" \
  -d '{"url": "https://example.com"}'

{ "data": { "url": "...", "title": "...", "format": "html", "content": "<h1>...</h1>", "word_count": 24, "fetched_status": 200 } }

POST /v1/images — every <img> plus the og:image, resolved to absolute URLs, de-duplicated, with alt text and a count.

{ "data": { "url": "...", "images": [{ "src": "https://site/logo.png", "alt": "Logo" }], "count": 3 } }

POST /v1/links — all anchors split into internal and external, made absolute and de-duplicated, with counts.

{ "data": { "url": "...", "internal": [{ "href": "...", "text": "About" }], "external": [...], "counts": { "internal": 12, "external": 4, "total": 16 } } }

POST /v1/metadata — title, description, canonical, language, robots, full Open Graph + Twitter Card tags, favicons, charset and an h1–h3 outline.

{ "data": { "title": "...", "canonical": "...", "lang": "en", "robots": "index,follow", "open_graph": { "title": "..." }, "twitter": { "card": "summary" }, "favicons": [...], "headings": [{ "level": "h1", "text": "..." }] } }

GET /v1/sitemap?domain=example.com — reads robots.txt Sitemap: lines and /sitemap.xml, follows sitemap-index files (up to 5 children) and returns URLs capped at 1000.

{ "data": { "domain": "example.com", "sitemaps": ["..."], "urls": ["..."], "total": 240, "truncated": false } }

POST /v1/crawl — breadth-first crawl restricted to the same host, scraping each page to markdown. max_pages defaults to 10 (hard cap 25), depth ≤ 2. Crawls run asynchronously: the call returns 202 with a job_id; poll GET /v1/jobs/{id} for the result. Pass an optional callback_url to be notified on completion. Credits are only charged when the job succeeds.

curl https://api.trawl.dev/v1/crawl \
  -H "Authorization: Bearer $TRAWL_API_KEY" \
  -d '{"url": "https://example.com", "max_pages": 10, "callback_url": "https://you.dev/hook"}'

{ "data": { "job_id": "01J...", "status": "queued" } }

{ "data": { "job_id": "01J...", "status": "succeeded", "result": { "root": "...", "pages": [{ "url": "...", "title": "...", "markdown": "..." }], "count": 10 } }, "credits": { "cost": 0, "remaining": 199990 } }

POST /v1/extract — every application/ld+json block decoded, plus the distinct @type values found on the page.

{ "data": { "url": "...", "jsonld": [{ "@type": "Organization", "name": "..." }], "types": ["Organization", "WebSite"] } }

GET /v1/screenshot?url=https://example.com — returns a ready-to-embed screenshot URL. Optional width, height and full_page.

{ "data": { "url": "...", "screenshot_url": "https://cdn.trawl.dev/screenshots/...png", "provider": "browserless", "width": 1280, "height": 960 } }

Note: screenshots are captured by our own browser pool (provider: "browserless") and served from our CDN. Where no browser pool is configured, the API falls back to WordPress mShots (provider: "mshots"), whose first request for a URL may warm the cache before the image is available.

POST /v1/brand — give it a domain, email or company and get a full brand profile: logos, colors, fonts, description and social links.

curl https://api.trawl.dev/v1/brand \
  -H "Authorization: Bearer $TRAWL_API_KEY" \
  -d '{"domain": "github.com"}'

{
  "data": {
    "domain": "github.com",
    "name": "GitHub",
    "description": "GitHub is where over 100 million developers...",
    "logos": [{ "type": "apple-touch-icon", "url": "https://github.com/..." }],
    "colors": ["#1f2328", "#0969da"],
    "fonts": ["Mona Sans"],
    "socials": { "twitter": "https://twitter.com/github" },
    "theme_color": "#1e2327"
  },
  "credits": { "cost": 10, "remaining": 489 }
}

GET /v1/logo?domain=stripe.com — the frictionless Clearbit replacement. Returns the best logo URL plus all candidates.

{
  "data": {
    "domain": "stripe.com",
    "logo_url": "https://stripe.com/img/v3/home/twitter.png",
    "candidates": [ ... ]
  },
  "credits": { "cost": 10, "remaining": 479 }
}

POST /v1/company/colors — the brand's colour palette plus its theme colour. Accepts domain, email or company.

{ "data": { "domain": "stripe.com", "colors": ["#635bff", "#0a2540"], "theme_color": "#635bff" } }

POST /v1/company/description — the company name and meta description.

{ "data": { "domain": "stripe.com", "name": "Stripe", "description": "..." } }

POST /v1/company/socials — links to the company's social profiles (Twitter/X, LinkedIn, GitHub, …).

{ "data": { "domain": "stripe.com", "socials": { "twitter": "https://twitter.com/stripe" } } }

POST /v1/company/fonts — the typeface families detected on the homepage.

{ "data": { "domain": "stripe.com", "fonts": ["Inter", "Söhne"] } }

POST /v1/company/styleguide — the full visual identity: logos, colours, theme colour and fonts in one call.

{ "data": { "domain": "stripe.com", "logos": [...], "colors": [...], "theme_color": "#635bff", "fonts": [...] } }

POST /v1/company/address — postal address, preferring JSON-LD PostalAddress and falling back to a footer/contact heuristic.

{ "data": { "domain": "stripe.com", "address": { "street": "354 Oyster Point Blvd", "locality": "South San Francisco", "region": "CA", "postal_code": "94080" }, "source": "json-ld" } }

POST /v1/classify/naics — classify a domain or company into a NAICS code.

{ "data": { "input": "stripe.com", "scheme": "naics", "code": "522320", "title": "Financial Transactions Processing...", "confidence": 0.6, "demo": true } }

Note: classification is a keyword heuristic ("demo": true) unless an Anthropic key is configured server-side, in which case it is LLM-backed.

POST /v1/classify/sic — the SIC variant of the classifier.

{ "data": { "input": "stripe.com", "scheme": "sic", "code": "6099", "title": "Functions Related to Depository Banking", "confidence": 0.6, "demo": true } }

POST /v1/transaction — clean a bank-statement descriptor into a merchant name, guess a domain and attempt a brand lookup.

curl https://api.trawl.dev/v1/transaction \
  -H "Authorization: Bearer $TRAWL_API_KEY" \
  -d '{"descriptor": "SQ *BLUE BOTTLE COFFEE OAKLAND CA"}'

{ "data": { "descriptor": "SQ *BLUE BOTTLE COFFEE OAKLAND CA", "merchant": "Blue Bottle Coffee", "guessed_domain": "bluebottlecoffee.com", "brand": { ... } } }

POST /v1/query — ask a natural-language question about any page and get a structured answer.

curl https://api.trawl.dev/v1/query \
  -H "Authorization: Bearer $TRAWL_API_KEY" \
  -d '{"url": "https://stripe.com/pricing", "question": "What is the per-transaction fee?"}'

Note: answers are LLM-backed when an Anthropic key is configured on the server. In demo mode the endpoint returns a labelled extractive answer ("demo": true).

GET /v1/me — your plan, rate limit and credit usage for the current period. This call is not billed.

Errors use a consistent envelope and standard HTTP status codes.

{ "error": { "code": "insufficient_credits", "message": "..." } }

• 401 — missing, invalid or revoked API key.
• 402 — plan credits exhausted for the period.
• 422 — invalid request parameters.
• 429 — rate limit exceeded for your plan.
• 502 — the upstream site could not be fetched.

Rate limits are enforced per API key and scale with your plan:

• Free — 10 requests / minute
• Pro — 300 requests / minute
• Scale — 1,200 requests / minute