
Two stories about AI bot traffic went around this week. A security firm's monthly report said two companies' AI agents made nearly nine in ten of the agent requests it saw in October. And a major airline rewrote its terms so that search engines may index its site while AI agents need written permission first. Both are about very large consumer sites. We wanted to know what the same question looks like on a small business website, so we read our own: every request that reached thinqdifferent.com for 14 days, November 3 to 17. There were 66,392 of them.
One request in thirteen said it was an AI
5,049 requests announced themselves as an AI crawler or assistant, under 28 different names: the well-known ones from OpenAI, Anthropic, Meta, Perplexity, Amazon and Apple, plus a long tail of newer ones. If we had stopped there, the story would be that AI is a real share of our traffic.
Then we looked at what they asked for.
Most of them weren't who they said they were
3,480 of the 5,049, or 69 percent, came from just 11 addresses. Between them, those 11 addresses used 26 different AI names. The same handful of machines called themselves GPTBot, then ClaudeBot, then Perplexity, then a dozen more.
None of them asked for an article. They asked for files with names like .env, cloud credentials, configuration backups and private keys: the places where passwords get left by accident. They got "not found" 2,218 times, and for most of the rest a redirect or the public sign-in page. 1,627 of those requests arrived on a single day, November 16.
This is ordinary password-hunting, the background noise every website gets. What's new is the costume. The name a visitor gives is whatever the sender types, and it costs nothing to borrow. Scanners have started borrowing AI names because so many sites have been told to wave AI bots through.
A bot's name is a claim, not an ID. More than two thirds of the "AI visitors" in our logs were someone else wearing the badge.
What the real ones did
That leaves 1,569 requests in two weeks. Not one of them asked for a secrets file. They behaved the way these crawlers are documented to behave, and they came from hundreds of addresses, not eleven.
The busiest was Meta's search indexer: 529 requests, and it kept coming back to our WordPress security pieces. Anthropic's crawler made 248. It read robots.txt 46 times and the sitemap 45 times, and it read 25 different articles. OpenAI's search crawler made 75 requests, 53 of them for robots.txt, and read one article. OpenAI's training crawler made 70, fetched the sitemap 14 times and the home page four, and didn't read a single article in two weeks. Amazon's crawler read 18 articles and a newer AI search engine's read 27.
Two smaller numbers are worth having. Our llms.txt file was fetched nine times in total, which fits what we said in does your website need an llms.txt: cheap to have, not yet something most crawlers come looking for. And the politeness is real. The genuine crawlers checked the rules far more often than they read pages.
Then there are live fetches, where an assistant opens your page because a person just asked it something. We saw 21 of those from ChatGPT in 14 days, and effectively none from any other assistant. On November 12, one conversation walked through our home page, work, method, contact and privacy pages and several others in eight minutes. We can't see who asked or what they asked, and we shouldn't be able to. But that is what "an AI visited" looks like at our size: one curious conversation, not a crowd.
What we couldn't see at all is the kind of agent the report counted, the kind that drives a real browser for someone. In a log, those look like a person. The report itself says 85 percent of agent requests went to product and search pages and about 60 percent to online stores, so a service firm isn't where they're shopping yet. One more limit: we separated real from fake by behaviour and by address. We did not check every address against the lists the AI companies publish.
What a firewall would have done with this
Most firewalls now offer a switch for AI bots, and it works on the name. Look at what each setting would have done to our two weeks.
Block AI bots by name turns away the 1,569 real requests, the ones that get a business read and cited. The impostors change their name the next morning. We covered how often this is switched on without the owner knowing in is your website blocking AI bots?
Allow AI bots by name does the opposite and it's worse: it waves 3,480 password probes past the checks everyone else has to clear.
On our own site the firewall stopped some of the impostors at the door. The rest reached the site and found nothing, because nothing was there to find. That second part is the real protection.
The bot policy we'd write instead
A block list is a guess about names. A policy is four decisions, written down where the next person can find them.
1. Search crawlers and live fetches are welcome. They're how an assistant learns what you do and brings you up.
2. Training crawlers are a business decision. Make it once, say it in robots.txt, and apply it to all of them. We laid out both sides in let AI search in, keep AI training out. Our own site lets them in.
3. No trust by name. No firewall exception for anything because of what it calls itself. A request for a file that isn't a page gets refused, whoever it claims to be.
4. The rules and the firewall must agree. Check both after any security change, and assume the probing never stops: nothing sensitive belongs in a folder the web can reach.
If you sell or book online, there's a fifth decision the airline just made in public: whether an agent may act for a customer on your site. For most service businesses that isn't a live question yet. It will be.
You can check the first half of this in about twenty seconds. Run the free scan and it shows which AI assistants your site's rules let in and which they turn away, bot by bot. It reads what your site tells each one. Your logs tell you who actually came.


