Author Topic: Roll your own search engine?  (Read 4448 times)

Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 4136
  • What, me worry?
    • View Profile
Re: Roll your own search engine?
« Reply #15 on: September 20, 2018, 08:06:51 PM »
Brad, your feedback was received and greatly appreciated, cheers. It's all taken on board. One of the things I'll be helping out with is to try squeeze a bit more room in indexes to increase overall index size...

Did you notice the knowledge graph stuff? I'd put a bit of that together.

Marc (the founder/owner) has a fair idea of the value of traffic, he's looking simply to get the search numbers up before thinking more about monetisation.

Bol,

It was actually a lot of fun, which is way more than I can say for Gigablast.

Since Mojeek tweets seemed to be popular I wrote a follow-on post sort of to remind the UK that they ought to rally behind a UK search engine with it's own index and a decent algo.  Ya, I'm sure they will listen to me. :-)

"You don't know what you got till it's gone. We paved Paradise and put in a parking lot."

BoL

  • Inner Core
  • Hero Member
  • *
  • Posts: 1205
    • View Profile
Re: Roll your own search engine?
« Reply #16 on: September 20, 2018, 08:29:18 PM »
Getting the word out is much appreciated.

>rally behind

Sometimes American eyes see the value better :-) DDG recently got some more funding, and they're little more than a skin.


Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 4136
  • What, me worry?
    • View Profile
Re: Roll your own search engine?
« Reply #17 on: September 20, 2018, 09:09:13 PM »
>skin

Heck, DDG is doing better promoting Bing than MS is.

I'd love to know DDG's long term strategy.  My two working theories today are: 1. They are building their own index in secret, while learning how to spider and rank, and will roll it out when needed, or 2. out wait MS (which has no idea what to do with Bing other than milk it) and when MS gets tired of running Bing, DDG will buy it and run it as DDG.

>funding
Sounds like there is a Canadian flavored DDG in the works.

>value
Well Marc has shown that you can still build (over 10 years) a good search engine without spending billions of dollars.  (Okay running it is different but still.)  A couple of Brits, working in a bedroom, did what the entire EU could not do.

Now the UK is leaving the EU, but Britain will still have a decent search engine under it's own control, not EU not US not RU, and that will come in very handy someday - if you can keep it.

>appreciated
I appreciated the retweets from Mojeek.  Please convey my thanks, if you can.
« Last Edit: September 20, 2018, 09:12:50 PM by Brad »

BoL

  • Inner Core
  • Hero Member
  • *
  • Posts: 1205
    • View Profile
Re: Roll your own search engine?
« Reply #18 on: October 10, 2018, 10:37:02 AM »
Wee resurrection of this thread

Quote
code up with some iframes and javascript

Brad, I think you mentioned potentially having a use for this. See the attached which I've knocked together for Firefox. Some engines have content-security and iframe policies that require a little tweaking before they work in iframes.


Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 4136
  • What, me worry?
    • View Profile
Re: Roll your own search engine?
« Reply #19 on: October 10, 2018, 11:47:31 AM »
BoL,

Wow!  That's it on steroids and it's really cool.  It's kind of amazing what you see when the different SERPs are side by side.

Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 4136
  • What, me worry?
    • View Profile
Re: Roll your own search engine?
« Reply #20 on: October 11, 2018, 08:05:08 PM »
Wee resurrection of this thread

Quote
code up with some iframes and javascript

Brad, I think you mentioned potentially having a use for this. See the attached which I've knocked together for Firefox. Some engines have content-security and iframe policies that require a little tweaking before they work in iframes.

Silly me.  When I first looked at this I overlooked the .txt doc.  I came back and found it and got it working.  Now I get to play.  Thanks again for coding this up 1. it really works well, 2. it was very kind of you

I've only had a chance to do a few searches, but Mojeek is holding up well.  What I notice is how much Bing tries to emulate Google's results.  I also note the heavy advertising StartPage has mucking up the SERP.

Will use. Thanks again!

BoL

  • Inner Core
  • Hero Member
  • *
  • Posts: 1205
    • View Profile
Re: Roll your own search engine?
« Reply #21 on: October 11, 2018, 09:10:09 PM »
I had to rename the thing as this forum software doesn't like .html attachments.

No worries RE: coding it up, part of my recent remit is some algo tweaking and this fitted in with it. Apart from bit fiddling, I wanted to see any glaringly obvious missing results.

I've grabbed a few thousand from the major engines and hope to compare, but the UI stuff is good for a gut feel.

If you look at the source you shouldn't have much bother customising the engines or adding new ones, let me know if you need a pointer
« Last Edit: October 11, 2018, 09:14:10 PM by BoL »

Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 4136
  • What, me worry?
    • View Profile
Re: Roll your own search engine?
« Reply #22 on: October 24, 2018, 09:15:53 PM »
>Brad, you really should just build yourself a search engine.

Done. Well not a search engine but a directory.  Great fun.  Something to work on over the winter.  It feels strange to be running a directory again like 20 years ago. Thanks for the encouragement.