To start a search engine

[Brad]

>See?

I do see. We are still at PPC, Pills, Pron and Casinos! Humans never change. 

[rcjordan]

So, pick your fight wisely. Chop off the long tail. Go for the throat. And I think your hybrid would be do-able.  With admin oversight of a restricted pool of proven authoritative sources, you could probably even trust an elf or two.

[Rooftop]

If I was going to put together a search engine I'd make one for TOR onions rather than clear web:  Less to index (well, public facing ones), less competition and the bar is so low.  Main issue would be the need to frequently re-crawl to see whether each site has been seized by the FBI.

[gm66]

I recently read about this, the guy used less than 20 Amazon EC2 instances :

https://aws.amazon.com/ec2/

plus Common Crawl data :

http://commoncrawl.org/

I can't remember the algo he used or if he wrote his own, will try and find article.

[rcjordan]

>aws

Something has been stirring the AWS-related news lately.  I almost had to filter it out the other day as it was crowding out the other tech articles.

[JasonD]

I aim to build a search engine. I "believe" I have the algo covered and refined enough to be "as good" as G or M, although I hope it would be better.

I've been trying to do it for over 10 years though and still not there so please don't hold your breath!