Author Topic: Google pushing new robots.txt standard  (Read 1211 times)


  • Administrator
  • Hero Member
  • *****
  • Posts: 5116
    • View Profile


  • Inner Core
  • Hero Member
  • *
  • Posts: 5611
    • View Profile
Re: Google pushing new robots.txt standard
« Reply #1 on: July 04, 2019, 12:00:28 AM »
>>Google follows at least five redirect hops.

I know John Mueller has said that in Webmaster office hours, but I don't think I've seen that in help docs before.

>>Google currently enforces a size limit of 500 kibibytes (KiB), and ignores content after that limit.

That's a pretty massive robots.txt. I wonder how many are out there that go beyond that.