For the last couple of years, a website owner who didn't want their writing used to train AI models had an uncomfortable choice. Blocking the crawlers that collect training data often meant blocking the ones that power AI search as well — and disappearing from the answers customers now ask for.
That trade-off is going away. Most of the large AI companies now separate the crawler that reads your site to answer questions from the one that collects material for training, and this month Cloudflare turned that split into a single setting for the sites it serves. So the real question is no longer technical. It is which of those two things you actually want.
Two different jobs, now two different crawlers
When an AI company visits your website, it is doing one of two jobs. Either it is reading your pages so it can find, quote or link to you when someone asks a question, or it is collecting text and images to improve the next version of its model. The first is much closer to search. The second is where most of the worry lives.
The major companies increasingly let you tell those jobs apart. OpenAI runs separate crawlers for search and for training, and so does Anthropic. Google uses a separate token, Google-Extended, to control whether your content is used for its AI models, and opting out of it does not affect how you appear in Google Search. Apple does the same with Applebot-Extended. Microsoft's equivalent for Bing is still on its way: robots.txt support for training preferences there is not expected until early next year.
In practice, the preference lives in your robots.txt file, the plain-text instructions a well-behaved crawler reads before it visits. Allow the search crawlers, disallow the training ones, and you have said what you want.
Opting out of AI training no longer has to mean opting out of AI search. The choice is finally yours to make on purpose.
What the new Cloudflare setting does
Cloudflare, which sits in front of a large share of the web, has introduced a setting that publishes a no-training preference in a site's robots.txt while leaving search crawlers alone. It writes the right instruction for each major company, and it treats crawlers that do both jobs according to whether their operator has committed to honouring training opt-outs without penalising you in search. Sites that had previously switched on Cloudflare's blanket AI blocking were moved to the new setting by default — worth knowing if you turned that on some time ago and haven't looked since.
One honest caveat: robots.txt is a request, not a lock. Reputable companies honour it; a scraper that doesn't care will ignore it. Network-level blocking can stop some of those, but no setting turns a public website into a private one.
What we chose, and why
This site currently lets AI crawlers in, for search and for training alike. That is a considered position for a business like ours. Our writing exists to be found, read and cited, and the ideas in it are worth more to us in circulation than kept out of a model's training data. We would rather be well understood by the tools our customers use than invisible to them.
A different business could reasonably decide the other way. If your content is the product — original research, paid courses, a distinctive body of work competitors would love to imitate — opting out of training while staying in search is now a sensible default rather than a sacrifice. The mistake is not picking either option. It is leaving the choice to a setting somebody switched on years ago, the same trap we described in whether your website is blocking AI bots.
How to decide for your own site
Start by reading your robots.txt and your security settings, so you know your current position rather than assuming it. Then ask the one question that matters: is your content how you market the business, or is it the thing you sell? Marketing content usually benefits from being read everywhere. Product content usually deserves protection.
Either way, decide it once, write down why, and revisit it when the standards shift again — which, given how quickly provenance and labelling are moving, they will. Sites built as clean, well-structured sources are the ones assistants can read and, increasingly, connect to directly, whichever way you set this dial. If you would like a second pair of eyes on yours, ask us.
