Can AI crawlers run up your app's bill? How to block them
Ship ItChecked against the official source
Source checked 5 Oct 2026 · Published 21 Oct 2026 · Last reviewed 21 Oct 2026 · By Ctrl+Z Therapy (how we test)
The short answer
If your traffic and your bill jumped overnight, not every visitor may be a person. AI crawlers like OpenAI's GPTBot and Anthropic's ClaudeBot say they follow robots.txt, but Google's docs say the instructions in robots.txt files cannot enforce crawler behavior: it's up to the crawler to obey them. To actually block AI bots, use Vercel's AI Bots ruleset or Cloudflare's AI bot blocking, then set real spending caps.
What to do
- On Vercel, turn on the AI Bots managed ruleset. It's available on all plans and inactive by default. Traffic the firewall denies does not count toward CDN Requests or Fast Data Transfer.
- On Cloudflare, all customers, Free included, can choose to block AI bots and agents based on their behavior.
- Careful with search bots: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Consider blocking only training bots, or start with Log rather than Deny.
- Set real caps. On Vercel (Spend Management, Pro and Enterprise), a spend amount does not stop usage on its own, so turn on Pause Production Deployments. And set a hard spend limit at your AI provider.
Good to know
- The overnight spike is an example scenario, not a documented case.
- Our own tip, not from the docs: don't call the AI API on page load for anonymous visitors.
Source
- Google Search Central: Introduction to robots.txt
- Anthropic: Does Anthropic crawl data from the web?
- OpenAI: Bots
- Vercel: WAF managed rulesets
- Vercel: WAF usage and pricing
- Vercel: Spend management
- Cloudflare: Block AI bots
Checked on 5 Oct 2026. Prices, limits and settings change, so check the source before you rely on them.