Author Topic: Archive.org changes...  (Read 2392 times)

4Eyes

  • Hero Member
  • *****
  • Posts: 817
    • View Profile
    • Email
Archive.org changes...
« on: January 03, 2012, 05:32:56 PM »
.. .as I understand it, they are now only showing archives for sites that have a current robots.txt.

Not checked it, just what I have been told.

Zwart

  • Inner Core
  • Full Member
  • *
  • Posts: 207
  • SEO Insultant
    • View Profile
Re: Archive.org changes...
« Reply #1 on: January 03, 2012, 05:37:15 PM »
"current" ????

littleman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4635
    • View Profile
Re: Archive.org changes...
« Reply #2 on: January 03, 2012, 06:59:50 PM »
I'm not seeing it on pages I have bookmarked.  Maybe they'll still be live, but removed from search?

I, Brian

  • Inner Core
  • Sr. Member
  • *
  • Posts: 396
    • View Profile
Re: Archive.org changes...
« Reply #3 on: January 03, 2012, 10:26:06 PM »
Am rebuilding a website after the hosting completely actually dropped the database in a server move (wasn't Clook!). Archive.org has a good record of the site from early last year I am using to rebuild the posts, and there's no robots.txt on the site.

Would presume archive.org are looking to only maintain crawls now on sites with robots.txt, if making any changes based on it??

4Eyes

  • Hero Member
  • *****
  • Posts: 817
    • View Profile
    • Email
Re: Archive.org changes...
« Reply #4 on: January 04, 2012, 10:12:22 AM »
http://www.archive.org/post/406187/we-were-unable-to-get-the-robotstxt-document-to-display-this-page
... not very clear. but seems to be implying not showing unless robots.txt is there.

Plenty of other threads in their forum on the general subject, but their forum is such a pain to use that I lost the will to live after 5 minutes.

4Eyes

  • Hero Member
  • *****
  • Posts: 817
    • View Profile
    • Email
Re: Archive.org changes...
« Reply #5 on: January 04, 2012, 10:14:03 AM »
... and there are a few posts on the illogical use of current robots.txt to block historic pages.

Archive.org = A bag of worms IMO