AI search could kill the web without new quality signals and revenue models
https://www.theregister.com/ai-and-ml/2026/07/01/ai-search-could-kill-the-web-without-new-quality-signals-and-revenue-models/5265269
Thinking this through to farther down the chain of events and it gets interesting. The knowledge base of AI will get stale as it kills off the very source of it's knowledge. Then what happens?
Side: Do we know if the AI crawlers are respecting robots.txt? Will every publicist start blocking AI en mass as an act of self-preservation? I could see Google making it really difficult to separate out the SE access from Gemini access.
>AI crawlers are respecting robots.txt?
Reports say they are not, hence all the cloudflare scanning added by sites.
>blocking
We've mentioned it here a little bit, but there is now a growing realization that G should be blocked if it is not sending significant traffic. As a webmaster and/or advertiser that is hard to accept.
That said, the now-evil G may access it anyway.
It's going to take something more than robots.txt. We can assume that AI has not and will not respect robots.txt when it suits them to ignore it.
> Then what happens?
I think I've shared my farmers vs miners rant.
The Vermont or Irish farmers would pick rocks out of the field and end up with ever more productive fields and growing lines of stone walls.
The miners pick rocks out of the ground and end up with big holes in the ground.
I think Google began life as a farmer - that is, it made the web bigger and more useful. But now it is a miner.
>>AI crawlers are respecting robots.txt?
Fortress is a stealth Chromium engine that stops your scrapers and browser agents from getting blocked, with one line of code change
GitHub - tiliondev/fortress: Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change. ยท GitHub
https://github.com/tiliondev/fortress?tab=readme-ov-file
>> Cloudflare Turnstile all read it as human
That has to be a cat and mouse game. In the worst case, it will render Turnstile obsolete
This is why we can't have nice things