Author Topic: checking comment follow-status in bulk  (Read 5567 times)

Rooftop

  • Inner Core
  • Hero Member
  • *
  • Posts: 1915
    • View Profile
checking comment follow-status in bulk
« on: March 15, 2011, 04:43:50 PM »
Does anyone know of / have a tool that can test blogs for follow comments? Ideal workflow would be to import a list of blog homepage urls set it running and get returned a list of just those that show follow links.

Have tried 2 solutions today. Once can't spot a followed link if it slaps it in the face.  The other is paranoid and thinks everything is following - whether it is or not.

Looking to process lists in the high-hundreds / low thousands.

All totally above board of course.  I have a lot of blogs and just can't remember which have follow and no-follow comments.

Drastic

  • Need a bigger hammer...
  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3087
  • Resident Redneck
    • View Profile
Re: checking comment follow-status in bulk
« Reply #1 on: March 15, 2011, 04:47:17 PM »
Scrapebox will do this, it has an addon for it. You need to feed it post urls to check though, it needs to check the links on the url you give it.

However, you could use it to scrape the pages needed, remove duplicate domains, then check for nofollow.

Rooftop

  • Inner Core
  • Hero Member
  • *
  • Posts: 1915
    • View Profile
Re: checking comment follow-status in bulk
« Reply #2 on: March 15, 2011, 05:02:42 PM »
scrapebox is one of the tools that we've tried.  However I think you just pointed out my mistake - we were trimming to root before doing the DF check (as there are a LOT of dupes). 

I guess it is pot luck whether you find posts with comments. I might even not remove dupes until after the DF check to minimise the number that get removed because there is no comment yet.

Thanks drastic.  Think I'm just having a retard day.

Drastic

  • Need a bigger hammer...
  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3087
  • Resident Redneck
    • View Profile
Re: checking comment follow-status in bulk
« Reply #3 on: March 15, 2011, 05:11:01 PM »
No problem. Couple of sb tips:

1 - made sure you're using the latest version of the addon, it had a fix recently

2 - you can remove duplicate urls by domain without trimming to root, just select remove duplicate domains and it will only leave one url per domain.

3- if you choose to not remove dupes, be sure you select "Randomize Comment Poster Blogs List" so when you transfer your harvested urls to the poster/checker area they are randomized. This keeps you from hitting the same domain with multiple request simultaneously. (I've got my settings pretty jacked up, this would look like a DDOS if you do too.)

If you didn't keep your original harvest, you can use site:domain.com + footprint for all of your keywords and harvest again.

Harvest - free proxies (scrape them) or cheap shared proxies
Post - private proxies
check links/sites - no proxy, but randomize

Rooftop

  • Inner Core
  • Hero Member
  • *
  • Posts: 1915
    • View Profile
Re: checking comment follow-status in bulk
« Reply #4 on: March 15, 2011, 05:15:52 PM »
Cheers for the tips.  Very welcome.

I'm going to check for that update, as something is definitely screwy with the DF checker.  Just did a quick test on 400 blogs:  40 NF, 1 DF, remainder unknown.  Does that sound right to you?  Topic is hobby related rather than particularly mercenary / commercial. 

Drastic

  • Need a bigger hammer...
  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3087
  • Resident Redneck
    • View Profile
Re: checking comment follow-status in bulk
« Reply #5 on: March 15, 2011, 05:24:52 PM »
I don't use the nofollow check much, but I've not had that kind of problem with it that I remember.

It sounds like it got hung up, were you using proxies? If so, free or paid?


Rooftop

  • Inner Core
  • Hero Member
  • *
  • Posts: 1915
    • View Profile
Re: checking comment follow-status in bulk
« Reply #6 on: March 15, 2011, 05:30:04 PM »
Free proxies - but I did the same run again without. 

My theory: It checks posts only for DF comments.  Most posts have no comments, so it comes back unknown - even if every other post on the site has big fat follow comments. 

I suspect the answer is "live with it".  Broaden the search, try to return more posts from each site and pick off whatever you can find.  I need to play more.

Drastic

  • Need a bigger hammer...
  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3087
  • Resident Redneck
    • View Profile
Re: checking comment follow-status in bulk
« Reply #7 on: March 15, 2011, 05:39:44 PM »
The addon has several columns
Dofollow Nofollow Assumption Status

So it tells you how many of each and what it assumes the blog is. Sounds like you may be running an older version.

The number of do-follow is very, very low, btw. You may already know that, but depending on what you're trying to do, your efforts may be better focused on Auto Approve.

Rooftop

  • Inner Core
  • Hero Member
  • *
  • Posts: 1915
    • View Profile
Re: checking comment follow-status in bulk
« Reply #8 on: March 16, 2011, 06:39:08 PM »
Our version was old - you were right.  that was part of the issue, but it's thrown up a few interesting things for us.

We're getting somewhere with this now, so I thought I'd update.  The reason that we're struggling is that we're trying to use scrapebox, a carpet bomber of a tool, to assist in some quite surgical work.  The idea is that it helps us find quite a small number of very targeted, ranked blogs with follow comments.  These will be part of a sustained manual commenting campaign rather than auto-post mayhem.

Bit strange maybe, but we're like that.

So, the problem with the DF plugin is that it just checks the page that it finds and most pages don't have comments.  Very quickly large numbers of good blogs fall through the net as "unknown". 

I've been trying to use custom footprints to find the post pages rather than the category or index and increasing the liklihood of finding posts with comments, so that there is actually something to check for follow status.

So, footprints including stuff like -"be first to comment" -"no comments" -"0 comments" etc.

Seems to be working out fairly well.  Been getting 1-1.5% follows from that on the keyword sets we've tried so far.  Other bonus is that it runs fairly quickly as the initial lists are smaller.

Could do with filtering out more category  and home pages from the initial search, but it's getting there.