Common Search: large-scale, nonprofit search engine

[bill]

https://about.commonsearch.org/faq

Noticed this search engine project because they just made a Tor node available.
https://about.commonsearch.org/2016/08/state-of-common-search-august-2016/

They are only using publicly available data sets, mostly Common Crawl.

[BoL]

I'm cynical, mainly because the commoncrawl dataset is in the low billions.  I think the coverage would be too low regardless of the quality of the algo.

[Brad]

At least somebody is spidering the web besides Bing and Google.  I guess we will wait and see.

[rcjordan]

>somebody

They ain't got G maps or Waze.  Anecdotally, I see more & more eyeballs drawn in servitude to G because they are dependent upon these.

[Mackin USA]

I tried to seed a site to them and it DID NOT WORK

whatever

[buckworks]

CommonSearch.com is for sale ...

http://www.afternic.com/forsale/commonsearch.com?utm_source=TDFS&utm_medium=sn_affiliate_click&utm_campaign=TDFS_Affiliate_domainthenet_an&traffic_type=TDFS&traffic_id=SN_AFFILIATE

[littleman]

BOL, well, they have to start somewhere.  In their FAQ they say that the search is still incomplete.  I think there is definitely room, with the market as big as it is and there being so few players these days.   The few tests I did seemed not too bad. 

An interesting thing about the SE business model is that server infrastructure is much more a factor of user level than the size of the data being queried.  So, in theory a good search engine could be started on a budget and then the hardware could be expanded as the user base grows.

[BoL]

There's definitely room littleman, and I'm all for them trying to make it happen. Perhaps if the commoncrawl dataset were to grow a lot then it has the ability to satisfy more queries. One major problem it's going to have is freshness but obviously they can keep knocking down those barriers to entry and keep improving. Not even sure how many 'pages' are on the web nowadays but it seems like their crawl data needs to grow by 10x to 100x.

Majestic's historical index is sitting at 865 billion,

A member from WmW I've spoken to a bit built his own, mojeek.co.uk... they have a similar philosophy to DDG but IIRC are just focusing on the UK.