Appears to be an archive scraper

[Rupert]

I seem to remember a thread before with someone asking for this.  I Am pretty sure Jasons kit does this too when you buy a domain from  him :
<Jason to chip in>


But this looks neat.  $27 per month :
https://waybackrebuilder.com/

[Drastic]

$26 lifetime access....

http://blazingseollc.com/#scraper

used it yesterday.

[Rupert]

like it

[JasonD]

> <Jason to chip in>

Our system does do it and there are alternatives, some of which you've pointed out above.

I won't go into why I believe ours is better (Just believe me.... it is) but they have us beaten on price.

Leave things with me and I'll see if I can sort out something that competes on price for Core members.

[Adam C]

Sorry to drag up an old thread, but was wondering whether anyone knew whether any of the above mentioned (or something else) might be best suited for the following:

I want to compile a reasonable sample (something between a few hundred and a few thousand URLs if possible - the domain normally would have had >100k URLs live) from a domain between two dates

Ultimately looking for a list of URLs that were live on the site in that period so I can recrawl them to see how those URLs are now handled (e.g. 301, 200, 404, chained redirects, etc.)

I figure I need a wayback scraper that...

1.  starts with the root domain and a date range
2.  follows internal links from the root URL
3.  outputs a list of URLs found

I can then run a separate process easily to clean up URLs and recrawl to check header status.

Thinking about it now, maybe I could just repurpose a generic scraper to get what I need, but wondering if any of the Archive specific tools are worth exploring for this?