The Core

I kinda miss submitting sites to 1,500 to 5000 "search engines".  My email inbox used to be jammed with spam asking me to submit my sites or buy some software that would submit for me.  It has been a long time since the last one of those emails.

I also miss directory optimization: where we would craft a title and description for Yahoo, dmoz and Looksmart that contained the keywords we needed without sounding or looking commercial or spammy and hope it got past the editors.

But I would kind of like to at least have the option to submit my URL's to a few (maybe 5, please) major search engines besides Google.  That would be rip snorting good fun. 

Good times. 

I had a setup where I would auto-submit to Excite, Altavista, LookSmart, Infoseek, Lycos, WebCrawler etc through an LWP User Agent script I wrote.  I also had a DirectHit script that use to tag links for rank. 

Then there was the cloaked pages that I use to generate on the fly Mad Libs style -- no database needed, just filter via IP and pick the template that worked for the particular engine.

I felt like a wizard back then.

Excite, Altavista, LookSmart, Infoseek, Lycos, WebCrawler

Shockingly, half of those or more are still out there and returning search results.

I kinda miss submitting sites to 1,500 to 5000 "search engines".  My email inbox used to be jammed with spam asking me to submit my sites or buy some software that would submit for me.  It has been a long time since the last one of those emails.

Cryptocoin emails have sort of taken over that roll.

I also miss directory optimization: where we would craft a title and description for Yahoo, dmoz and Looksmart that contained the keywords we needed without sounding or looking commercial or spammy and hope it got past the editors.

Being a Zeal directory editor (which fed into LookSmart, which fed into MSN Search) was fantastic for instant editing your rankings in MSN Search back in the day.

The whole biz model of web directories sort of died off. It's hard to compete with Google on relevancy unless you are very tightly focused & it is hard to generate as much revenue per visitor as Google does in the key commercial areas. The general perception of the broader industry is links from general web directories don't have value unless they are for local search & have NAP data. In addition most directory sites not only can't draw repeat visitors, but they typically can't even get the category pages well indexed in Google. On the off chance they do get deeply indexed they'll likely get hit by the Panda algorithm. Even DMOZ cratered.

I don't think they are actively accepting submissions yet, but Curlie is allegedly to become a rebirth of DMOZ. One thing I think is a mistake is they put AdSense ads on their help forum before opening the new directory up to submissions. And since their site mostly caters to webmasters they probably don't make much from the display ads on Resource Zone.

But I would kind of like to at least have the option to submit my URL's to a few (maybe 5, please) major search engines besides Google.  That would be rip snorting good fun.

I wonder if a new search engine would be able to scale up to create a competitive product with a strong algorithm & a fairly comprehensive web index without getting blocked by firewalls for excessive crawling.

When overseas many websites won't load due to firewall restrictions unless one uses a VPN or some such.

I also think modern search engines use a lot of the end user engagement data for ranking refinement. Competing on the usage data front would be quite hard. Google owns well over 90% of mobile search via owning Android & paying Apple to be the default search engine. And on desktop devices Chrome has massive marketshare, and I believe Google pays Opera, Mozilla Firefox & Vivaldi for default placement. Bing is aggressively pushed by Microsoft in Skype & Edge, but both Yahoo! and Bing are so ad dense they've seen sharp drops in organic search clicks.

Google's dominance is so strong I saw a Microsoft bundleware option where they were trying to install Chromium optimized for Bing browser. The crazy thing there is they are only one Chromium "security" update away from that being reset to promoting Google.

Most web browsers are also moving away from supporting toolbars.

Oh and I miss Searchhippo.  hhh 

I actually used the HipRank API in one of my directories.  Sites were listed  in the categories by HipRank order.  Well use of the API was free.  It was my little joke on the Google Directory, but nobody understood it (plus it wasn't very good) so I eventually ditched it. 

Gigablast is still around & they have an API http://gigablast.com/api.html

If Marissa Meyer had been able to get Yahoo out of their contract with Bing and start their own spidering engine again I had hoped they would revive the AltaVista name.

She did get the agreement amended to where they are able to show ads from Bing, Google or Yahoo!. Yahoo only has to use Bing for (I think it was something like) 51% of desktop searches & gets paid the vast majority of the revenue from the ads (something like 93% without having to incur any of the costs with running a search engine).

But they saw from Gemini data they weren't really gaining ground creating a third parallel network against the first which was already so dominant.

It is also hard to argue that organics would be a differentiating factor for them given how heavy the ad load is on their search results page. Sometimes they have graphical mini banners for other commercial search queries on noncommercial search results. Even when you search for a fairly explicit search topic where they should be able to do good from one of their vertical search services they punt. A few days ago I went to their homepage to search for "Dollar General stock" and there were 4 or 5 ads above the organics, then a local search result panel with map, then the mini insert for their stock price data. I searched again just now and there was only 1 ad above the organics, but sometimes they really weigh the page down with ads.

RKG mentioned in their most recent quarterly report that Yahoo & Bing had their fastest ad click growth in a long time, but that came at the cost of seeing sharp declines in organic clicks.

While Google continues to dominate paid search traffic with 90% click share, Bing Ads and Yahoo Gemini spend have actually increased faster Y/Y than Google in each of the last two quarters.

The growth of paid clicks on Bing and Yahoo appears to have hurt organic search traffic driven by the two engines, as both posted their largest year-over-year (Y/Y) organic visit declines in the history of this report.

While organic search visits driven by Google were roughly flat Y/Y in Q1 2018, Yahoo organic search visits fell 36% and Bing visits fell 20%. Yahoo and Bing’s declines were the largest observed in the history of this report and occurred at the same time that click growth for Bing and Yahoo paid search ads is running near multi-year highs.

Infoseek got borged by the Mouse, but the name has been out of circulation so long only us geezers remember it so you could reuse it and all the kids would think it is new and shiny.

Excite and Lycos are still trying to be portals.  To bad somebody does not try something new in search with them.

If you enter a search into the Lycos search box it will say "There were no results for your search query." Excite is owned by an IAC subsidiary.

I like the new design of HotBot.

Someone tried reviving the Mamma.com domain, but it got hit with an algorithmic penalty as everything on their site seems to rank page 4 or worse in Google.

Many long existing businesses like Cafepress & Overstock.com have been nailed by Google ranking shifts over the past year or two.

Reviving a dead brand might be even harder in the current algorithm given that many active businesses are still getting nailed without going through a shut down period.

CircuitCity.com was relaunched out of bankruptcy once already, but the company which bought the domain more recently now has it redirecting to circuitcitycorporation.com.

Quote from: littleman on May 22, 2018, 12:30:26 AM

Excite, Altavista, LookSmart, Infoseek, Lycos, WebCrawler

Shockingly, half of those or more are still out there and returning search results.

Most search engines beyond Google & Bing are to a large degree retreads of one of those two (at least outside of a few markets like China, Russia & South Korea).

And then some of them are heavily wrapped in ads for one another as the traffic is arbitraged back and forth. IAC still does quite a bit of arb & they're still using domains they've "retired" from an SEO perspective to drive SEM campaigns.

DuckDuckGo has a clean interface.
StartPage.com pitches being privacy focused.
Ecosia plants trees with some portion of your search ad clicks (feeling green, I might just go look for some auto insurance quotes and NYC attorneys real quick .

I have no idea who bought HotBot. Was surprised by the clean search result set though, as typically crawling & indexing is expensive, and most the search engines that could license such data usually want you to eat their ads with the organics. Google even did away with offering a paid ad-free site search service.

>directories ... They had their day and it is largely over.

I think there may be room for  a hybrid directory search engine based on popularity and categories from a site like Reddit.  The idea would be a spidering search engine that uses the sub-reddit data, popularity waves and 'upvote/downvote' scores to affect the ranking algorithm.  There is a lot going on there that could be applied to rank, stuff that DMOZ and sites like DirectHit use to try to do.

The problem with the native Reddit engine is that it only uses its onsite data to display results, but combining that data with actual on page content would make for a great search engine.

>Hotbot

Pure Bing feed.  Just looked side by side.  Except for some things which get inset on bing like photos, the organic results and the order, are the same.

That is not a bad thing.  It's a good clean serp with no advertising.  Win!

>mojeek.com

BoL thanks for posting that.  I had no idea Mojeek was still around.  I do have a dim memory of it from years ago, back when I was paying attention to search in general.  I think Chris Ridings might have interviewed the owner or he stopped into comment on a thread about early ver. 1 mojeek on SearchGuild forums.

Anyway, it is neat somebody else is out there spidering, respects privacy and is doing it from the UK.

I see they even have site search. Nice.