Easy/Lazy way to make wget look like a browser

[littleman]

Just make a .wgetrc file.

Here's mine:
header = Accept-Language: en-us,en;q=0.5
header = Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
header = Connection: keep-alive
user_agent = Mozilla/5.0 (Windows NT 6.1; rv:47.0) Gecko/20100101 Firefox/47.0
referer = /
robots = off

[ergophobe]

Brilliant. Pasted into evernote for future reference

[BoL]

Firebug (a Firefox plugin) has a nice 'copy as cURL' menu option, so you can replicate exactly a previous request. Comes in handy sometimes to reverse engineer what the request should look like.

[littleman]

I had to download a list of images today from a site.  I didn't want to do it manually, but didn't want to set any red flags.  I put the list of image URLs in a file "t.txt" and wrote a little perl script to do the work for me using wget.  I added a random interval to the script so that the works more human and less bot like.  A few lines of code saving me at least three hours of work.  Posting here just in case someone else could use the idea.

Code Select Expand


open(T,"t.txt");
my @t = <T>;
close T;

foreach $t (@t) {
my $time = int(rand(10)) + 1;
print "waiting $time seconds \n";
sleep $time;
`wget $t`;