Is Botvee Crawler free?

Yes. The website crawler is free with no login required. The developer API is free too, with per-account quotas to prevent abuse.

Does it support JavaScript-rendered websites?

Yes. Pages are fetched statically first for speed; when a page needs JavaScript to show its content, the crawler falls back to a headless browser render automatically.

Do you respect robots.txt?

Yes. robots.txt Disallow rules are honored, and private or loopback hosts and non-http(s) schemes are always rejected.

Yes — a REST API with API keys, short-lived access tokens and per-account quotas. A free account gets you a key.

Turn any website into AI‑ready data.

Crawl, extract and export clean structured text from any public website — built for RAG pipelines, chatbots and LLM training.

API documentation

No login requiredZIP, Markdown, JSON, CSVrobots.txt respectedREST API

botvee-crawler LIVE

$botvee crawl <url>

Paste a website URL above to begin.

Pages stream here in real time as they are crawled,

then export as ZIP, Markdown, JSON or CSV.

Free to use — no account needed for your first crawl.

Platform

Built like AI infrastructure, not a scraper

Everything you need to feed production AI systems with clean website data.

AI-ready output

Clean structured text with navigation, boilerplate and duplicates stripped — ready for RAG, chatbots and LLM training.

Fast, concurrent crawling

Parallel workers with an async job queue, automatic retries and per-site analytics.

Five export formats

ZIP, Markdown, JSON, TXT and CSV — every export includes an index and per-page metadata.

Token-based API

Permanent keys exchange for short-lived access tokens. Keys are stored hashed and shown once.

Safe by default

Rate limits, daily crawl credits, SSRF protection and robots.txt compliance on every request.

Built to scale

Per-account quotas and a worker queue designed to grow from one machine to a distributed fleet.

How it works

Website to AI data in four steps

Paste a URL

Any public website, blog, docs or knowledge base.

Crawl

The sitemap is mapped and a job is queued instantly.

Watch live

Pages stream into the console in real time.

Export

Download as ZIP, Markdown, JSON, TXT or CSV.

For developers

A REST API with tokens and quotas

Exchange your permanent key for a short-lived access token, then crawl. Quotas are enforced per account.

You need an API key to call the API — free account, no credit card. Get your API key Full API reference