https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/
Just an hour ago, I told Louise that I strongly suspected that since retail searchers are a lazy bunch and would go for the low-hanging serp, they would just take the AI overview as the answer to their query.
The AI overviews often contain errors. They are not to be trusted without further checking.
=========
A small way to fight back: beef up your robots.txt with this list to exclude AI crawlers:
https://perishablepress.com/ultimate-ai-block-list/#block-ai-via-robots
Nice that he has Apache and nginx rules that return 403 instead of assuming that his robots.txt directives will be honored.
Periodically I get high CPU warnings and based on the amount of money being poured into AI crawling and what a huge presence they have in logs last I looked, I wonder if that could explain it.
Quote from: buckworks on July 04, 2025, 03:33:32 PMThe AI overviews often contain errors. They are not to be trusted without further checking.
Human-generated overviews, assertions and "facts" often contain errors. As Aristotle wrote, "Don't trust everything you find on Instagram."