For your spidering needs...

[BoL]

The entire Gigablast codebase is open source.

https://github.com/gigablast/open-source-search-engine

It's pretty much anything you can do with the gigablast engine itself. All written in C++ and simple to install.

If you have a dedicated server or running a copy of linux somewhere it's well worth a look.

[littleman]

Th code should be pretty powerful is they've built a billion page index with it.  Some years back I built a 4 million page index off of DMOZ and I thought that was impressive.

[BoL]

After messing around with the UI it could make for a great site search.

[Brad]

After messing around with the UI it could make for a great site search.

Yeah Gigablast used to let webmasters set up a free custom site search that was pretty darn good in its day.  I used it on an archive of an old forum I had turned into flat files and it worked well.

[ergophobe]

I thought this was open sourced years ago, but obviously I was wrong - http://www.prnewswire.com/news-releases/gigablast-now-an-open-source-search-engine-217624911.html

See here where evreyone is touting different site search options and Matt Wells (search on it) says "I recently open-sourced Gigablast"
http://www.quora.com/Apache-Solr/Who-are-major-competitors-to-Solr