I seem to remember a thread before with someone asking for this. I Am pretty sure Jasons kit does this too when you buy a domain from him :
<Jason to chip in>
But this looks neat. $27 per month :
https://waybackrebuilder.com/
$26 lifetime access....
http://blazingseollc.com/#scraper
used it yesterday.
like it :)
> <Jason to chip in>
Our system does do it and there are alternatives, some of which you've pointed out above.
I won't go into why I believe ours is better (Just believe me.... it is) but they have us beaten on price.
Leave things with me and I'll see if I can sort out something that competes on price for Core members.
Sorry to drag up an old thread, but was wondering whether anyone knew whether any of the above mentioned (or something else) might be best suited for the following:
I want to compile a reasonable sample (something between a few hundred and a few thousand URLs if possible - the domain normally would have had >100k URLs live) from a domain between two dates
Ultimately looking for a list of URLs that were live on the site in that period so I can recrawl them to see how those URLs are now handled (e.g. 301, 200, 404, chained redirects, etc.)
I figure I need a wayback scraper that...
1. starts with the root domain and a date range
2. follows internal links from the root URL
3. outputs a list of URLs found
I can then run a separate process easily to clean up URLs and recrawl to check header status.
Thinking about it now, maybe I could just repurpose a generic scraper to get what I need, but wondering if any of the Archive specific tools are worth exploring for this?