How Botvee Crawler works
Everything you need to know about crawling a website and getting AI-ready data out of it.
What the crawler does
You give it one URL. It discovers every reachable page on that website (sitemap first, then internal links), downloads each page, strips navigation, boilerplate and duplicate blocks, and saves the clean text — one file per page plus an index and per-page metadata.
How a crawl runs
- Map — sitemap.xml and robots.txt are read, internal links are discovered up to depth 5.
- Crawl — pages are fetched concurrently by a worker pool; JavaScript-heavy pages fall back to a headless browser render.
- Extract — content is cleaned and de-duplicated; words, links and timings are recorded per page.
- Export — download everything as ZIP, Markdown, JSON, TXT or CSV.
Rate limits
| Context | Limit |
|---|---|
| Anonymous (website) | Per-IP job limit with a cooldown between crawls |
| Free API account | 1000 crawl credits/day · 10 requests/min · 1000 pages/hour · 2 concurrent jobs |
| Per crawl | Up to 1000 pages, depth 5 (free plan) |
One crawled page always costs one credit. Credits reset every 24 hours.
robots.txt & safety
The crawler honors robots.txt Disallow rules, rejects private and loopback hosts, non-http(s) schemes, and skips binary/media files. Each site has a bounded runtime so a crawl can never run forever.
Export formats
| Format | Contents |
|---|---|
| ZIP (txt) | One .txt per page + index.json + metadata.json |
| Markdown | One .md per page with title + source URL header |
| JSON | Single pages.json array: {url, title, crawledAt, text} |
| CSV | pages.csv — url, title, crawled_at, text |
Data retention
Crawled data is temporary: downloads stay available for 24 hours, then the files are deleted automatically. Job records expire shortly after.