You've read the headlines: publishers blocking GPTBot, robots.txt as the new battleground, the web going dark to AI. That may be true of large publishers. It is emphatically not what small-business websites look like from an agent's point of view.
- Of 679 probed Auckland businesses, just 28 deny any AI crawler in robots.txt.
- When we actually fetched each site the way an AI crawler would, only 19 blocked the request.
- That's roughly 96% of small-business sites fully open to AI systems — across restaurants, barbers, plumbers, and retail alike.
The usual caveats apply: this is a June 2026 snapshot with one crawler user-agent tested per site — policies change, and a different bot might be treated differently. And the sample is small-business websites; large publishers blocking AI crawlers is real and well documented. This finding doesn't contradict that story — it just shows the story doesn't describe main street.
So when 60% of these same businesses never surface in any AI engine's answers (the invisibility wall), access isn't the explanation. The engines could read almost everyone. They simply have no reason to: nothing they trust has written those businesses down.
Why this matters for the advice you're given
A lot of "AI readiness" consulting is access-flavoured — crawler permissions, technical SEO for bots, feeds and files. Necessary hygiene, occasionally decisive, but in our data the blocked-door case is rare (and sometimes accidental — a security service swatting bots the owner never configured). The binding constraint is upstream: authority, not access. An open door on a street nobody walks down.
One nuance we did find worth checking on your own site: readability failures that look like content but read as nothing. About one site in eight across our restaurant runs was a JavaScript shell serving a nearly-empty page to a plain fetch. Those aren't blocked — they're mute. Different problem, same result.
Spend five minutes on hygiene: confirm your robots.txt isn't accidentally denying AI crawlers, and that your key pages render actual text without JavaScript. For roughly 96% of you, this is already fine.
Then stop optimising the door and work on the street. Citations in sources engines trust are what generate visits from agents — that's where the invisible 60% and the visible 40% actually differ in our data.