Main Menu

Spidering tools

Started by jetboy, April 23, 2012, 11:42:15 AM

Previous topic - Next topic

jetboy

I need to report on outbound links on a third-party site (site A). Is there a web-based (or free desktop) tool that will generate a list of all outbound links from site A that point to site B? FYI, I have some affiliates that habitually let their product links get stale, and I'd like to proactively help out.

Rumbas

Xenu and some filtering on the link list?

Rooftop

+1 for xenu : dump it to excel then filter it.

Adam C

Screaming Frog - for an upgrade from Xenu.  Desktop based - £99 pa

Semetrical Deepcrawl - for a hosted spidering service.  Various packages.  Not particularly cheap, but pretty good.
http://www.semetrical.com/products/deepcrawl

Rumbas


JasonD

ScreamingFrog is superb and Dan is a damn fine bloke too

Rooftop

screaming frog is excellent - it isn't free though.

JamesR

I've used 80legs.com on a list of 10 websites to analyze outbound link patterns.  I did have to hand the resulting data off to the Excel ninja to make use of the results but it was pretty cool.  Those desktop apps might be easier to use for a single site though.

littleman

Old thread, but I just wanted to add webcheck to the tool list.  It's a Linux crawler that makes a nice HTML report.