We run quite a few audits for clients and I'm always looking for dupe content or near dupe content. However I could really use a tool to automate the process.
Anyone know of anything worth working with out there?
That would be a very useful tool. You'd think a company like Majestic or another link analysis company would have such a thing because they are already spidering the web anyway.
It's not automated but I sometimes find this useful:
http://www.webconfs.com/similar-page-checker.php
I've used Copyscape's API in the past, a co-worker would look at the results to check whether content was unique or not. iirc they have some kind of similarity/unique metric.
I always think of Copyscape as a tool for catching plagiarists, so I don't know as it's what you're looking for
Copyscape is a 1:many tool - you input the one text you care about and it searches many texts to see if your text is plagiarized
It sounds like you're looking for a many:many tool - something that will look at many texts and see if there are matches among them.
I think to use Copyscape that way you're going to need to pipe stuff in - some sort of
bash>Screaming Frog http://example.com | Copyscape
type of workflow.