Author Topic: Can paid site search help fund expanded search engine crawls?  (Read 106 times)

Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 2551
  • What, me worry?
    • View Profile
I'm pretty sure I don't know what I'm talking about here but I thought I'd throw this out there anyway in relation to Mojeek in particular.

My thought goes something like this:  right now DDG provides site search for me via some sort of combination of Bing and Yandex crawlers.  What if I and other webmasters could pay or subscribe to have Mojeek deep crawl our sites for site search purposes, but by design also be subsidizing Mojeek's crawl of the web with deeper content?   WOuld that be worth the effort by Mojeek?  I'm not sure there is even enough demand for paid third-party site search, but with Google dropping older pages from their serp and privacy problems with Google there might be.

Anyone have any thoughts?

ergophobe

  • Inner Core
  • Hero Member
  • *
  • Posts: 4834
    • View Profile
Re: Can paid site search help fund expanded search engine crawls?
« Reply #1 on: July 14, 2019, 02:53:53 PM »
'm not sure there is even enough demand for paid third-party site search

With Google sunsetting their paid CSE, I think there is an opportunity. My company was using CSE. I switched to a paid https://www.addsearch.com/ account. Believe it or not, some properties let their paid CSE go to free, which meant they were showing competitor ads in their site search.

Right now, you have a few alternatives
 - free Google search... which will show competitor ads
 - native search for the app you use... usually bad. Witness the search on The Core or pay attention to the size of your database on a Drupal site of any size.
 - Apache Solr - works well and is the recommended solution for Drupal, but you really need some technical know-how and server foo to run this well.

I think a search engine could step in and offer what Add Search offers and make some money. I will say, I find Add Search very good and not that cheap really... $89/mo for 1,000 to 10,000 docs - I think that's a fair bit more than Google was charging for their paid CSE product.

It gives you fine-grained control of your index and how your search results present (like choosing a feature image and telling it which pages to rank for which term). There might be some conflicts of interest there with a search engine, but if they could apply your filters and knobs just to site search and not the general SERPS, I think it would be a product people would pay for. Add Search seems to be making it. Not sure if they operate in the red or the black. But something like Mojeek would have way more marketing muscle since they already have customers after a fashion.
« Last Edit: July 14, 2019, 02:56:44 PM by ergophobe »

BoL

  • Inner Core
  • Hero Member
  • *
  • Posts: 831
    • View Profile
Re: Can paid site search help fund expanded search engine crawls?
« Reply #2 on: July 16, 2019, 10:10:23 AM »
It's definitely an idea that's been discussed at Mojeek. One potential problem area is that it can become a sort-of pay for inclusion model, a paid for crawling budget that a site might not otherwise have. I do really like the idea though, as it's the bread and butter of the product.

Apparently algolia.com do quite well for themselves.

Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 2551
  • What, me worry?
    • View Profile
Re: Can paid site search help fund expanded search engine crawls?
« Reply #3 on: July 16, 2019, 10:34:56 AM »
>pay for inclusion

I hadn't thought of that but it is a danger. 

I'm just glad I wasn't completely daft.

ergophobe

  • Inner Core
  • Hero Member
  • *
  • Posts: 4834
    • View Profile
Re: Can paid site search help fund expanded search engine crawls?
« Reply #4 on: July 16, 2019, 02:35:38 PM »
it can become a sort-of pay for inclusion model

That's what I was alluding to with Add Search. One of the things we like is that we fully control the index, but since they do not have a public search engine, it's just our site search, so has no impact on their credibility, no conflict of interest.

Although, when it comes right down to it, Google and Bing (and for that matter Expedia and Amazon) are already pay for inclusion. If I'm not doing well in the organic results (and often even if I am), I pay to be included at the very top of the SERPs. I would say it's disclosed and labelled, but one friend of mine not long ago told me "Ads on Google? I don't know what you mean? I don't get any ads when I go to Google. Where is it exactly that you're advertising?"