Web hostile to upstart search engine crawlers, most websites only allow Google

[rcjordan]

Why the internet needs crawl neutrality
https://www.fastcompany.com/90759792/with-google-dominating-search-the-internet-needs-crawl-neutrality?partner=feedburner

[littleman]

If I ran a search engine I'd definitely try to mask the spider's identity upon denial.