Frequently asked questions

Is Botvee Crawler free?

Yes. The website crawler is free with no login required. The developer API is free too, with per-account quotas to prevent abuse. Paid plans will only raise limits.

How many pages can I crawl?

Anonymous website crawls are rate limited by IP. The free API plan gives 1000 crawl credits per day (1 page = 1 credit), up to 1000 pages per crawl at depth 5.

Does it support JavaScript-rendered websites?

Yes. Pages are fetched statically first (fast); when a page needs JavaScript to show its content, the crawler falls back to a headless browser render automatically.

Do you respect robots.txt?

Yes. robots.txt Disallow rules are honored, and private/loopback hosts and non-http(s) schemes are always rejected.

What export formats are supported?

ZIP (plain text), Markdown, JSON, TXT and CSV. Every export includes an index.json and per-page metadata.json.

Is there an API?

Yes — a REST API with API keys, short-lived access tokens and per-account quotas. See the API docs; a free account gets you a key.

How does API authentication work?

Create an account to get a permanent key (bsk_live_…). Exchange it for a 60-minute access token at POST /api/v1/auth/token and send it as a Bearer token. Keys are stored hashed and shown only once.

What are crawl credits?

Every crawled page consumes one credit from your daily quota, so a single API call can’t do unlimited work. Credits reset every 24 hours.

How long are downloads stored?

Downloads remain available for 24 hours after a crawl completes, then the data is deleted automatically.

Can I crawl documentation sites and blogs?

Absolutely — docs, blogs and knowledge bases produce the cleanest AI-ready output.

Can I stop a running crawl?

Yes — from the website UI while it runs, or with POST /api/v1/jobs/{id}/stop via the API.

What do you do with my crawled data?

Nothing. It sits on the server only so you can download it, and it is deleted after 24 hours. See the Privacy page.