The Core

Why We Are Here => Hardware & Technology => Topic started by: BoL on August 11, 2013, 01:28:20 AM

Title: For your spidering needs...
Post by: BoL on August 11, 2013, 01:28:20 AM
The entire Gigablast codebase is open source.

https://github.com/gigablast/open-source-search-engine

It's pretty much anything you can do with the gigablast engine itself. All written in C++ and simple to install.

If you have a dedicated server or running a copy of linux somewhere it's well worth a look.
Title: Re: For your spidering needs...
Post by: littleman on August 11, 2013, 02:56:08 AM
Th code should be pretty powerful is they've built a billion page index with it.  Some years back I built a 4 million page index off of DMOZ and I thought that was impressive.
Title: Re: For your spidering needs...
Post by: BoL on August 11, 2013, 01:21:43 PM
After messing around with the UI it could make for a great site search.
Title: Re: For your spidering needs...
Post by: Brad on August 12, 2013, 12:46:16 PM
Quote from: BoL on August 11, 2013, 01:21:43 PM
After messing around with the UI it could make for a great site search.

Yeah Gigablast used to let webmasters set up a free custom site search that was pretty darn good in its day.  I used it on an archive of an old forum I had turned into flat files and it worked well.
Title: Re: For your spidering needs...
Post by: ergophobe on August 12, 2013, 08:09:04 PM
I thought this was open sourced years ago, but obviously I was wrong - http://www.prnewswire.com/news-releases/gigablast-now-an-open-source-search-engine-217624911.html

See here where evreyone is touting different site search options and Matt Wells (search on it) says "I recently open-sourced Gigablast"
http://www.quora.com/Apache-Solr/Who-are-major-competitors-to-Solr