has anyone successfully done this? I'm not sure how workable it is - the proposal is to block everything unless specifically whitelisted (and there's an acknowledgement that some bots will ignore it, of course)
thoughts?
Sounds like a recipe for future disaster to me. Perfect on paper though.
Ain't but two guys to ask, IMO, and I don't know where Ralph is right now. So that leaves incredibill and it looks like he's posted recently on this
https://www.google.com/search?q=site:incredibill.com+whitelist
I'm not much of a techie so consider this endorsement with great caution, but I've been using Incredibill's whitelisting approach for two or three years with no known problems.
aaah, good to hear - thanks for the link RC. Does anyone know if he's every got around to part 2 or expanded/published suggested approved bot list/etc elsewhere, or does anyone have an up to date list of good bots?
I'm concerned about stuff like Google product crawlers (suspect they crawl to check price validity etc and if we block them we might have issues with product feed approval) and whether Yahoo still uses SLURP and what the smaller engines use (while DDG may not be sending us much traffic it seems wrong to exclude it - not only for us but also they're never going to rule the world if we all block them!). I don't have a lot of time to hunt this stuff down and double check it at the moment!
<edit> Ralph of course, but i don't really know him well enough to hit him up for a random freebie!
Unless he's mellowed, or muddled his brain on those damn cigs, you won't get a freebie from Ralph.
>Unless he's mellowed, or muddled his brain on those damn cigs, you won't get a freebie from Ralph.
Hahahah! Probably got that right, however he is a nice guy, lol.
i used to selectively deliver differing IP addresses via DNS requests.
It's effectively normal IP cloaking but at a lower level and I found it much easier to control / manage.