More and more of the people looking for what you do never reach a search results page. They ask an assistant, and the assistant answers. Which means there is now a second question worth asking about your website, alongside whether people can find it: can the assistants read it at all?
For a surprising number of businesses, the answer is no. Not because anyone decided that — almost nobody decides that — but because something in the stack decided it on their behalf and never mentioned it.
How this happens by accident
The crawlers that feed AI assistants are not the same as the one that feeds Google's search index. They arrive under their own names, and they get blocked in their own ways. Four causes cover most of what we find:
A security or SEO plugin with a tidy-sounding switch — block AI scrapers, protect my content — that writes rules into your robots.txt without telling you which visitors it just turned away. A bot-protection rule at the network edge that challenges anything without a normal browser signature, which an assistant's fetcher does not have. A staging configuration that shipped to production with the whole site marked as off-limits. Or a firewall that treats an unfamiliar visitor as a threat by default.
None of these look like a mistake from the inside. The site is up, the traffic looks normal, and nothing is broken. The only symptom is an absence — you are not in the answer, and there is nothing on your end that says why.
How to check yours in ten minutes
Open yoursite.com/robots.txt in a browser. Read it. You are looking for named disallow rules for the AI fetchers and for any blanket rule that catches everything. If you see the names of specific assistants listed there, someone or something put them there on purpose, and it is worth knowing who and why.
Then check the layer above it. Your CDN or firewall dashboard has a log of blocked and challenged requests; filter it for the last month and see which non-browser visitors are being turned away. A rule that challenges every request without browser-like headers will stop an assistant just as effectively as a disallow line, and it will not show up in your robots.txt at all.
Last, look for the flip side: whether there is anything useful for a machine to read when it does get in. A page built as a pile of scripts that assemble themselves in the browser can be technically reachable and still practically unreadable.
Being invisible to AI assistants is usually not a decision. It is a default nobody reviewed.
Blocking is a legitimate choice — just make it a choice
We are not going to tell you that every business should let every crawler in. There are real reasons to hold some of them off. If your content is the product, or you are in a licensing conversation about it, or you have a genuine reason not to feed a particular company's model, then a deliberate block is a perfectly sound position.
The distinction that matters is between a decision and an accident. A business that has looked at the list and chosen to allow some and refuse others is in good shape. A business that is refusing all of them because a plugin's default said so, while its owner wonders why the assistants never mention them, is not making a choice at all.
What we do about it
We build sites that are legible to both audiences from the start, because retrofitting that is harder than it sounds. That means clean, server-rendered pages rather than content that only exists once a browser has run it; a structured summary of the business that a machine can read directly; and an explicit, reviewed position on which fetchers are welcome, written down rather than inherited from a plugin.
It is also the reason we do not build on a database-and-plugin stack, where a well-meaning extension can change who is allowed to read your site without anyone approving it. And it is the near-term, practical half of a longer shift we have written about separately: assistants are moving from reading pages to connecting to sites directly, and the businesses that are readable now are the ones that will be reachable then. If you want the reasoning behind how we build, it is in our method.
Go and read your own robots.txt today. It takes a minute, and a fair number of people who do it are surprised by what they find.
