Product

Changelog

What's new in Botvee Crawler.

2026-07-18

Public site & developer accounts

  • New public pages: documentation, API reference, pricing, FAQ, changelog, status, privacy and terms.
  • Developer accounts: register/login, dashboard with API key management, live usage and quotas.
  • Complete design refresh — cleaner, faster, no clutter.
2026-07-17

Developer API with tokens & quotas

  • API keys (bsk_live_…) stored as hashes, shown once; 60-minute Bearer access tokens.
  • Per-account rate limits, daily crawl credits, hourly page caps and concurrency limits.
  • Live public stats; per-account job history endpoints.
2026-07-17

Job queue & export formats

  • Async job queue with 4 workers, job states and 24-hour retention.
  • Export as ZIP, Markdown, JSON, TXT or CSV — with index and metadata files.
  • Live crawl console streaming pages in real time.
2026-07-16

Public-service hardening

  • robots.txt compliance, SSRF protection, depth and runtime caps, per-IP rate limiting.
  • Per-site analytics: words, characters, internal/external links, duration.
2026-07-16

Initial release

  • Full-site crawling with sitemap discovery, concurrent fetching and clean text extraction.
  • Headless browser rendering fallback for JavaScript-heavy pages.