Just doing some pipe dreaming here, because sometimes I think about this stuff but can't actually do it. What would it take to start a search engine with its own DB dealing with the English language websites?
Of course money, but the learning curve is less now, the wheel has already been invented.
Crawler. Gigablast script is now open source, I suppose one could start with that. The problem with Gigablast is the algo.
Algo: there is the rub IMO, coming up with a really good algo that could compete with Bing and Google for quality. It would have to be better than Gigablast. And I think it would be better to use quality indicators that do not involve linking but I'm not smart enough to know what those might be.
Also, is there room for some sort of search engine directory hybrid either like old NBCi or like the old Google when it still relied heavily on ODP to tell it if a site was quality of not? I guess today one would use ODP and Wikipedia for that.
Any thoughts?
I see DDG frequently being touted by the tech sites, yet it gets no traction. Even with money to overcome the mundane issues of crawling, bandwidth, & storage, developing a userbase would be the big problem.
Majestic is in a good position to have something comparatively good.
Maybe it's possible to have a custom made search engine(s) based on their data, the users tune the algo to what they think is relevant. It'd probably be very expensive per customer but surely something could be worked out.
> algo
Pay people to run a toolbar, that purely tracks time spent on sites, search terms and whatnot... it'd have to be quite clear on what it collects. Maybe give the option to 'like' a site on the SERPs too. All subject to manipulation of course.
>pay people
If minions are involved somebody will eventually pay them for rank, karma, whatever. You need a job 'term limit' of some sort --which would eventually cause an availability problem.
I built a simple search engine some years back that crawled ODP and had about two million page results. I had the thing running from a static DSL line and it was powered by a very slow 533 MHz fanless mini-itx computer sitting in my office. The search engine got about 2k/day traffic -- I had a semi-captive audience at the time. The performance was really great, and I wanted to bring it up to 10 million URLs indexed, but I never did get to do that.
I used Estraier for the index, which produced very fast results, but it doesn't have any provisions for weighing results via a metric like link popularity. The author of Estraier now has an improved engine called Hyper Ester.
http://fallabs.com/hyperestraier/index.html
More:
This is just the engine side of it, not the crawler. For that I was doing something kind of primitive with w-get.
It is my understanding that the size of the database is much less of a factor for hardware requirements as is the amount of traffic the engine gets. For instance, you could probably get away with a low cost servers if you had an index over a billion as long as the engine only receives moderate traffic. This means that one could probably build a good quality, large index search engine without a lot of money invested and scale up as the user base goes.
I've also had the dream of building a large scale and good quality search engine. It is something I'd like to return to someday.
Quote from: rcjordan on November 10, 2014, 05:58:29 PM
I see DDG frequently being touted by the tech sites, yet it gets no traction. Even with money to overcome the mundane issues of crawling, bandwidth, & storage, developing a userbase would be the big problem.
I'm looking at this from a couple of angles: 1. What would it take for someone to produce a search engine for a country in Europe, 2. USA, or whole EU.
Particularly the EU countries where Google is really the only search engine and they complain about it. You have to offer a viable alternative. Nobody will use it if it does not exist. Be prepared to fight for years in the wilderness.
>>DDG
Right now they mainly rely on Bing for DB and algo. But they do have their own crawler, and I bet they are preparing for the day when Bing starts getting jealous or spooked and cuts them off. My suspicion is that DDG is working on their own crawler and algo while they buy time using Bing as a crutch.
I ran a small niche search engine using Fluid Dynamics Search Engine. It worked well but because of FDSE's database size limits I had to limit crawl depth to index page only. Actually it worked rather well.
All this is fascinating to me, I thought spidering would eat up most bandwidth, but I can see how queries and ad serving would dominate on a mature SE. I think as long as everyone thinks Google indexes the Universe no new search engines will get built. It's a shame.
With my rss news reader content-gathering experience as background, I think there are some niches that *might* be able to make some money. Medical research or dietary supplements, for instance. This would probably be easiest with a loosely-curated crawl using terms gathered from a VERY managed list of rss feeds rather than an algo. Curating the list with the niche in mind as a target, I think it would be do-able to come up with authoritative serps as output. In -say- the case of supplements, I could see the hybrid search/directory model working well for the most popular categories (but seo weighs in heavily for the reason to have a directory).
>news reader
What about a subscription model? I'm paying per month for just the NYTimes and I'd dump that and switch to your news reader app.
The place I see curated directories being useful is highly focused local search. Hard to monetize.
>What about a subscription model?
Ewwww! That would involve ...people. (Beat you to it, Dras)
It's not an app, its a series/sequence of php scripts. I haven't even shown it to my daughter who is head of a math department because it would glaze her eyes over --and we communicate very well. Gawd, just thinking about trying to explain the "RC UI" makes me grind my teeth.
>Hard to monetize
Not, say, an good serp for 'diabetes' or 'pancreatic cancer' or 'low-carb diet' ...I haven't looked, but I'd imagine there'd be a feeding frenzy in adwords.
<added>
>>news reader
It does already scrape to a flatfile database.
diabetes.4geeks.com ....hmmmm (also fourgeeks and forgeeks --it's a type-in set of domains)
>adwords
Use it to help refine popularity based on $$$
I think a bunch of old SEO could come up with a search engine algo that would compare well to Google.
Experience and gile beats youth and skill. Heh.
>gile
I'm not sure about gile, but experience says that you can set the depth of category relatively shallow and the users don't really know or care. (How many times have we used the phrase "top 3 pages" in the seo biz?) You can avoid scaling issue by setting tight boundaries, too. What users still CAN'T do is differentiate between spam vs legit sites.
See? Stuff like this:
America's best-selling drug is ... Viagra? Prozac? No ...Abilify. From April 2013, through March 2014, sales of Abilify (official name, aripriprazole) totaled almost $6.9 billion.
http://www.thedailybeast.com/articles/2014/11/09/mother-s-little-anti-psychotic-is-worth-6-9-billion-a-year.html
With sales like that, a lot of people are going to be looking it up, I'd bet.
>See?
I do see. We are still at PPC, Pills, Pron and Casinos! Humans never change. :)
So, pick your fight wisely. Chop off the long tail. Go for the throat. And I think your hybrid would be do-able. With admin oversight of a restricted pool of proven authoritative sources, you could probably even trust an elf or two.
If I was going to put together a search engine I'd make one for TOR onions rather than clear web: Less to index (well, public facing ones), less competition and the bar is so low. Main issue would be the need to frequently re-crawl to see whether each site has been seized by the FBI.
I recently read about this, the guy used less than 20 Amazon EC2 instances :
https://aws.amazon.com/ec2/
plus Common Crawl data :
http://commoncrawl.org/
I can't remember the algo he used or if he wrote his own, will try and find article.
>aws
Something has been stirring the AWS-related news lately. I almost had to filter it out the other day as it was crowding out the other tech articles.
I aim to build a search engine. I "believe" I have the algo covered and refined enough to be "as good" as G or M, although I hope it would be better.
I've been trying to do it for over 10 years though and still not there so please don't hold your breath!