« previous next »
Pages: [1]

Author Topic: Archive.org changes...  (Read 5491 times)

4Eyes

  • Hero Member
  • *****
  • Posts: 817
    • View Profile
    • Email
Archive.org changes...
« on: January 03, 2012, 05:32:56 PM »
.. .as I understand it, they are now only showing archives for sites that have a current robots.txt.

Not checked it, just what I have been told.
Logged

Zwart

  • Inner Core
  • Full Member
  • *
  • Posts: 207
  • SEO Insultant
    • View Profile
Re: Archive.org changes...
« Reply #1 on: January 03, 2012, 05:37:15 PM »
"current" ????
Logged

littleman

  • Administrator
  • Hero Member
  • *****
  • Posts: 6554
    • View Profile
Re: Archive.org changes...
« Reply #2 on: January 03, 2012, 06:59:50 PM »
I'm not seeing it on pages I have bookmarked.  Maybe they'll still be live, but removed from search?
Logged

I, Brian

  • Inner Core
  • Sr. Member
  • *
  • Posts: 397
    • View Profile
Re: Archive.org changes...
« Reply #3 on: January 03, 2012, 10:26:06 PM »
Am rebuilding a website after the hosting completely actually dropped the database in a server move (wasn't Clook!). Archive.org has a good record of the site from early last year I am using to rebuild the posts, and there's no robots.txt on the site.

Would presume archive.org are looking to only maintain crawls now on sites with robots.txt, if making any changes based on it??
Logged

4Eyes

  • Hero Member
  • *****
  • Posts: 817
    • View Profile
    • Email
Re: Archive.org changes...
« Reply #4 on: January 04, 2012, 10:12:22 AM »
http://www.archive.org/post/406187/we-were-unable-to-get-the-robotstxt-document-to-display-this-page
... not very clear. but seems to be implying not showing unless robots.txt is there.

Plenty of other threads in their forum on the general subject, but their forum is such a pain to use that I lost the will to live after 5 minutes.
Logged

4Eyes

  • Hero Member
  • *****
  • Posts: 817
    • View Profile
    • Email
Re: Archive.org changes...
« Reply #5 on: January 04, 2012, 10:14:03 AM »
... and there are a few posts on the illogical use of current robots.txt to block historic pages.

Archive.org = A bag of worms IMO
Logged
Pages: [1]
« previous next »