bulk broken link checker

[Gurtie]

I'm looking for a broken link checker which will allow me to upload a list of domains and will run through the queue providing a standard broken links check for each - xenu or screamingfrog but without having to check everything one at a time.

Anyone seen one? Happy to pay fo a one off or a subscription service if absolutely necessary

[rcjordan]

Found one that says it does multi-site. No idea if it works.

http://www.deadlinkchecker.com/login.asp

[littleman]

I'm pretty sure you could give Xenu a list of URLs from multiple sites from multiple sites and that it can hit them in parallel.

[Gurtie]

Xenu - yup was playing with that but it seems to stop after the first site when I add a list

deadlink - thanks, will check it out

there's a github linkchecker at http://wummel.github.io/linkchecker/index.html if anyone else is looking for this - but I want to try and find something I can deal with myself before launching into trying to get a developer on it!

[rcjordan]

>github linkchecker

I read over that one first, but could find nothing in the doc that indicated multi-domain crawling.  There is reference to a config setting (--stdin : Read list of white-space separated URLs to check from stdin) that has possibilities but it doesn't appear to crawl, just check specified pages.

Anther option that *might* work is regex, but that looks to be mostly for output, not input.

[JasonD]

Ahrefs filter domains by 404

[Adam C]

Deepcrawl.  Get an agency account.  Very good tool IMO.

[Rooftop]

I've always meant to try 80legs for something like this, but never quite found the time to work it out.

[rcjordan]

>80legs for something like this, but never quite found the time to work it out.

Ditto

[Rumbas]

Waddabout Garreth: http://www.brokenlinkbuilding.com/

[JasonD]

I'm playing around with a new code base.

Is there interest in 404s then?

[Gurtie]

my orignal reason for asking was that a client has been contacted by someone asking him to part with £6500 a year for a login to a system which 'checks whether your sites have broken links'

For many reasons this is poor value for money for him, but as he has around 350 sites (all run from single CMS but managed by individual site managers and on seperate urls) I was looking for something easier than setting him up with 350 instances of WT to do the same job - if I could upload a list of the root url's and get back a list of pages on those sites with broken links on them once a quarter that would save him a fortune.

I think he's already actually signed up so less urgency now, but clearly there is a market for it - because he's signed up. I presume it has some type of pretty dashboard too (which I would have no need of)

sstrangely since I posted this a couple of other people have mentioned needing something similar for different reasons, but suspect they'll code their own. Its basically only taking a standard single site version and adding a queue to it, isn't it?

[Rooftop]

If it is for their own site they surely don't need to crawl. That makes the job pretty easy. Extract anything with an http, put it in a table and start curling.

[Gurtie]

yeah, but there's 350 sites.

Whatever way we check the data,  I don't want to sit there and do whatever it is 350 times. I just want to upload a list of domains and come back to a list of some type.