How to Monitor Which AI Bots Are Crawling Your Website
AI Crawlers in QueryBurst connects to Cloudflare analytics and surfaces bot-by-bot activity for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and others — request volumes, data transferred, status codes, trend charts, and referral tracking for the full site and individual pages. Use it to see which AI bots are visiting, how often, how much data they're consuming, and whether those visits result in referral traffic from AI-generated answers.
Getting Started
1. Create a Cloudflare API Token
You'll need a token with these permissions:
| Permission | Access |
|---|---|
| Analytics | Read |
| Zone | Read |
| AI Crawl Control | Read |
The connect flow links directly to Cloudflare's token creation page with these permissions pre-selected.
2. Select a Zone
After verifying your token, select the Cloudflare zone (domain) that corresponds to this site. Each site gets its own connection and API token.
3. Initial Sync
Once connected, an initial data sync pulls available historical data. The amount of history depends on your Cloudflare plan:
| Plan | Historical Data | Query Window |
|---|---|---|
| Free | ~24 hours | 1 day |
| Pro | ~7 days | 7 days |
| Business | ~30 days | 30 days |
| Enterprise | ~90 days | 90 days |
After the initial sync, data is pulled daily at 06:00 UTC.
Dashboard
Bot Filter
Filter data by AI provider or individual bot. Provider groups include:
- OpenAI — GPTBot, ChatGPT-User, OAI-SearchBot
- Anthropic — ClaudeBot, Claude-SearchBot, Claude-User
- Perplexity — PerplexityBot, Perplexity-User
- Google — Google-CloudVertexBot
- Meta — Meta-ExternalAgent, Meta-ExternalFetcher, FacebookBot
- ByteDance — Bytespider
- Other — CCBot, Applebot, Amazonbot, DuckAssistBot, MistralAI-User
Click a provider to select all its bots. Click individual bot chips to toggle them on/off.
Crawl Activity Over Time
A stacked area chart showing daily crawl requests broken down by bot. Hover for per-bot counts on any given day.
Requests by Bot
A horizontal bar chart showing total requests per bot over the selected period. Hover for detailed stats including status code breakdown (2xx, 3xx, 4xx, 5xx) and data transferred.
Top Crawled Pages
A sortable table of the most-crawled pages on your site. Columns include request count, data transferred, and referrals (Pro+ only). Sort by any column to find your most-visited content.
Referral Traffic (Pro+ only)
For Cloudflare Pro and above, referral traffic shows actual human click-throughs from AI answers. This is broken down by:
- Platform — A pie chart showing which AI platforms (OpenAI, Anthropic, Perplexity, etc.) are driving traffic
- Page — A table showing which pages receive referrals, with conversion rate (referrals ÷ crawls)
Per-Page Crawler Data
When viewing an individual page's Overview tab, crawler activity for that specific page is displayed at the top. This uses the same charts and filters as the main dashboard but scoped to a single path — showing you exactly which bots are visiting that page and how often.
Day Selector
Use the day selector (7d / 14d / 30d / 60d / 90d) to adjust the time range for all charts and data. The default is 30 days. Available range depends on your Cloudflare plan's data retention.
Sync & Connection
- Sync — Trigger an on-demand data pull from Cloudflare (rate-limited to one at a time)
- Disconnect — Remove the Cloudflare connection, optionally deleting all stored crawl data
What Gets Tracked
Bot crawl data is stored per page, per bot, per day. Static assets (JS, CSS, images, fonts) and meta files (robots.txt, sitemap.xml) are filtered out automatically so you only see content page activity.
Tips
- Check after publishing — New content typically gets crawled within hours by active bots
- Watch for 4xx/5xx — Status code errors in the bot breakdown may indicate pages that bots can't access
- Compare bots — Use the filter to compare how different AI providers crawl your site
- Track referrals — If on Cloudflare Pro+, referral traffic shows which pages are actually being cited in AI answers
- Use per-page view — When analysing a specific page, the Overview tab shows crawler data scoped to that page