API reference
REST API
Authenticate with an API key, exchange it for a short-lived access token, then crawl. Anyone can read these docs — create a free account to generate a key.
Authentication
Your permanent key (bsk_live_…) is exchanged for a 60-minute Bearer token. Keys are stored hashed and shown only once at creation.
# exchange your key for an access token curl -X POST https://crawler.botvee.ai/api/v1/auth/token \ -H "X-API-Key: $BOTVEE_CRAWLER_API_KEY" # → { "access_token": "eyJ…", "expires_in": 3600 }
Endpoints
| Endpoint | Description |
|---|---|
| POST /api/v1/auth/token | Exchange API key → 60-min access token |
| POST /api/v1/scrape | Start an async crawl job — body {"urls":["https://…"]} |
| GET /api/v1/jobs | List your jobs |
| GET /api/v1/jobs/{id} | Job status + per-site progress |
| POST /api/v1/jobs/{id}/stop | Cancel a running job |
| GET /api/v1/jobs/{id}/results/{site} | List extracted pages for a site |
| GET /api/v1/jobs/{id}/results/{site}/{file} | One page: {url, title, crawledAt, text} |
| GET /api/v1/me | Account, plan and live quota usage |
| POST /api/v1/keys/regenerate | Rotate your key (old key revoked, new shown once) |
| GET /download/job/{id}.zip?format=… | Download results (txt · md · json · csv) |
Start a crawl
curl -X POST https://crawler.botvee.ai/api/v1/scrape \ -H "Authorization: Bearer $ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{"urls":["https://example.com"]}' # → 202 { "jobId": "job_…", "statusUrl": "/api/v1/jobs/job_…" }
Quotas & errors
Every crawled page consumes one credit (free plan: 1000/day). When a limit is hit the API answers 429 with a JSON error and a Retry-After header. Invalid credentials answer 401; another account's job answers 403.
Need a key? API keys are generated from your dashboard. Create a free account or log in — no credit card, free plan included.