The Core

Why We Are Here => Web Development => Topic started by: jetboy on April 23, 2012, 11:42:15 AM

Title: Spidering tools
Post by: jetboy on April 23, 2012, 11:42:15 AM
I need to report on outbound links on a third-party site (site A). Is there a web-based (or free desktop) tool that will generate a list of all outbound links from site A that point to site B? FYI, I have some affiliates that habitually let their product links get stale, and I'd like to proactively help out.
Title: Re: Spidering tools
Post by: Rumbas on April 23, 2012, 12:01:37 PM
Xenu and some filtering on the link list?
Title: Re: Spidering tools
Post by: Rooftop on April 23, 2012, 12:13:19 PM
+1 for xenu : dump it to excel then filter it.
Title: Re: Spidering tools
Post by: Adam C on April 23, 2012, 12:30:04 PM
Screaming Frog - for an upgrade from Xenu.  Desktop based - £99 pa

Semetrical Deepcrawl - for a hosted spidering service.  Various packages.  Not particularly cheap, but pretty good.
http://www.semetrical.com/products/deepcrawl
Title: Re: Spidering tools
Post by: Rumbas on April 24, 2012, 09:45:52 AM
http://www.seomoz.org/blog/crawler-faceoff-xenu-vs-screaming-frog
Title: Re: Spidering tools
Post by: JasonD on April 24, 2012, 11:12:18 AM
ScreamingFrog is superb and Dan is a damn fine bloke too
Title: Re: Spidering tools
Post by: Rooftop on April 24, 2012, 11:38:51 AM
screaming frog is excellent - it isn't free though.
Title: Re: Spidering tools
Post by: JamesR on April 25, 2012, 11:17:57 PM
I've used 80legs.com on a list of 10 websites to analyze outbound link patterns.  I did have to hand the resulting data off to the Excel ninja to make use of the results but it was pretty cool.  Those desktop apps might be easier to use for a single site though.
Title: Re: Spidering tools
Post by: littleman on July 27, 2013, 08:19:56 AM
Old thread, but I just wanted to add webcheck to the tool list.  It's a Linux crawler that makes a nice HTML report.