Author Topic: Easy/Lazy way to make wget look like a browser  (Read 1526 times)

littleman

  • Administrator
  • Hero Member
  • *****
  • Posts: 6531
    • View Profile
Easy/Lazy way to make wget look like a browser
« on: August 11, 2016, 12:47:51 AM »
Just make a .wgetrc file.

Here's mine:
header = Accept-Language: en-us,en;q=0.5
header = Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
header = Connection: keep-alive
user_agent = Mozilla/5.0 (Windows NT 6.1; rv:47.0) Gecko/20100101 Firefox/47.0
referer = /
robots = off

ergophobe

  • Inner Core
  • Hero Member
  • *
  • Posts: 9255
    • View Profile
Re: Easy/Lazy way to make wget look like a browser
« Reply #1 on: August 11, 2016, 05:10:12 PM »
Brilliant. Pasted into evernote for future reference

BoL

  • Inner Core
  • Hero Member
  • *
  • Posts: 1205
    • View Profile
Re: Easy/Lazy way to make wget look like a browser
« Reply #2 on: August 17, 2016, 10:13:32 PM »
Firebug (a Firefox plugin) has a nice 'copy as cURL' menu option, so you can replicate exactly a previous request. Comes in handy sometimes to reverse engineer what the request should look like.

littleman

  • Administrator
  • Hero Member
  • *****
  • Posts: 6531
    • View Profile
Re: Easy/Lazy way to make wget look like a browser
« Reply #3 on: October 26, 2017, 06:52:33 PM »
I had to download a list of images today from a site.  I didn't want to do it manually, but didn't want to set any red flags.  I put the list of image URLs in a file "t.txt" and wrote a little perl script to do the work for me using wget.  I added a random interval to the script so that the works more human and less bot like.  A few lines of code saving me at least three hours of work.  Posting here just in case someone else could use the idea.

Code: [Select]
open(T,"t.txt");
my @t = <T>;
close T;

foreach $t (@t) {
my $time = int(rand(10)) + 1;
print "waiting $time seconds \n";
sleep $time;
`wget $t`;