Author Topic: I kinda miss those 5000 search engines.  (Read 719 times)

Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 2142
  • What, me worry?
    • View Profile
I kinda miss those 5000 search engines.
« on: May 21, 2018, 11:11:49 PM »
I kinda miss submitting sites to 1,500 to 5000 "search engines".  My email inbox used to be jammed with spam asking me to submit my sites or buy some software that would submit for me.  It has been a long time since the last one of those emails.

I also miss directory optimization: where we would craft a title and description for Yahoo, dmoz and Looksmart that contained the keywords we needed without sounding or looking commercial or spammy and hope it got past the editors.

But I would kind of like to at least have the option to submit my URL's to a few (maybe 5, please) major search engines besides Google.  That would be rip snorting good fun. 

Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 2142
  • What, me worry?
    • View Profile
Re: I kinda miss those 5000 search engines.
« Reply #1 on: May 22, 2018, 12:13:13 AM »
Oh and I miss Searchhippo.  hhh 

I actually used the HipRank API in one of my directories.  Sites were listed  in the categories by HipRank order.  Well use of the API was free.  It was my little joke on the Google Directory, but nobody understood it (plus it wasn't very good) so I eventually ditched it.  :D

littleman

  • Administrator
  • Hero Member
  • *****
  • Posts: 3862
    • View Profile
Re: I kinda miss those 5000 search engines.
« Reply #2 on: May 22, 2018, 12:30:26 AM »
Good times. 

I had a setup where I would auto-submit to Excite, Altavista, LookSmart, Infoseek, Lycos, WebCrawler etc through an LWP User Agent script I wrote.  I also had a DirectHit script that use to tag links for rank. 

Then there was the cloaked pages that I use to generate on the fly Mad Libs style -- no database needed, just filter via IP and pick the template that worked for the particular engine.

I felt like a wizard back then.

Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 2142
  • What, me worry?
    • View Profile
Re: I kinda miss those 5000 search engines.
« Reply #3 on: May 22, 2018, 01:12:33 AM »
>wizard

Back then you were, LM.

That was a really cool setup.  And I bet it was fun, at least in retrospect.  I never was anywhere near that class. Strictly low tech.

I completely forgot about Direct Hit until you mentioned it.

This was the site I used in the early days to submit URL's.  http://www.selfpromotion.com/  I'm amazed even a static version is still around.  It got me started.

ergophobe

  • Inner Core
  • Hero Member
  • *
  • Posts: 4068
    • View Profile
Re: I kinda miss those 5000 search engines.
« Reply #4 on: May 22, 2018, 07:25:04 PM »
Excite, Altavista, LookSmart, Infoseek, Lycos, WebCrawler

Shockingly, half of those or more are still out there and returning search results.

Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 2142
  • What, me worry?
    • View Profile
Re: I kinda miss those 5000 search engines.
« Reply #5 on: May 22, 2018, 07:58:35 PM »
If Marissa Meyer had been able to get Yahoo out of their contract with Bing and start their own spidering engine again I had hoped they would revive the AltaVista name.

Infoseek got borged by the Mouse, but the name has been out of circulation so long only us geezers remember it so you could reuse it and all the kids would think it is new and shiny.

Excite and Lycos are still trying to be portals.  To bad somebody does not try something new in search with them.

aaron

  • Inner Core
  • Full Member
  • *
  • Posts: 137
    • View Profile
Re: I kinda miss those 5000 search engines.
« Reply #6 on: May 23, 2018, 11:34:51 AM »
I kinda miss submitting sites to 1,500 to 5000 "search engines".  My email inbox used to be jammed with spam asking me to submit my sites or buy some software that would submit for me.  It has been a long time since the last one of those emails.
Cryptocoin emails have sort of taken over that roll.

I also miss directory optimization: where we would craft a title and description for Yahoo, dmoz and Looksmart that contained the keywords we needed without sounding or looking commercial or spammy and hope it got past the editors.
Being a Zeal directory editor (which fed into LookSmart, which fed into MSN Search) was fantastic for instant editing your rankings in MSN Search back in the day.

The whole biz model of web directories sort of died off. It's hard to compete with Google on relevancy unless you are very tightly focused & it is hard to generate as much revenue per visitor as Google does in the key commercial areas. The general perception of the broader industry is links from general web directories don't have value unless they are for local search & have NAP data. In addition most directory sites not only can't draw repeat visitors, but they typically can't even get the category pages well indexed in Google. On the off chance they do get deeply indexed they'll likely get hit by the Panda algorithm. Even DMOZ cratered.

I don't think they are actively accepting submissions yet, but Curlie is allegedly to become a rebirth of DMOZ. One thing I think is a mistake is they put AdSense ads on their help forum before opening the new directory up to submissions. And since their site mostly caters to webmasters they probably don't make much from the display ads on Resource Zone.

But I would kind of like to at least have the option to submit my URL's to a few (maybe 5, please) major search engines besides Google.  That would be rip snorting good fun.
I wonder if a new search engine would be able to scale up to create a competitive product with a strong algorithm & a fairly comprehensive web index without getting blocked by firewalls for excessive crawling.

When overseas many websites won't load due to firewall restrictions unless one uses a VPN or some such.

I also think modern search engines use a lot of the end user engagement data for ranking refinement. Competing on the usage data front would be quite hard. Google owns well over 90% of mobile search via owning Android & paying Apple to be the default search engine. And on desktop devices Chrome has massive marketshare, and I believe Google pays Opera, Mozilla Firefox & Vivaldi for default placement. Bing is aggressively pushed by Microsoft in Skype & Edge, but both Yahoo! and Bing are so ad dense they've seen sharp drops in organic search clicks.

Google's dominance is so strong I saw a Microsoft bundleware option where they were trying to install Chromium optimized for Bing browser. The crazy thing there is they are only one Chromium "security" update away from that being reset to promoting Google.

Most web browsers are also moving away from supporting toolbars.

Oh and I miss Searchhippo.  hhh 

I actually used the HipRank API in one of my directories.  Sites were listed  in the categories by HipRank order.  Well use of the API was free.  It was my little joke on the Google Directory, but nobody understood it (plus it wasn't very good) so I eventually ditched it.  :D
Gigablast is still around & they have an API http://gigablast.com/api.html

If Marissa Meyer had been able to get Yahoo out of their contract with Bing and start their own spidering engine again I had hoped they would revive the AltaVista name.
She did get the agreement amended to where they are able to show ads from Bing, Google or Yahoo!. Yahoo only has to use Bing for (I think it was something like) 51% of desktop searches & gets paid the vast majority of the revenue from the ads (something like 93% without having to incur any of the costs with running a search engine).

But they saw from Gemini data they weren't really gaining ground creating a third parallel network against the first which was already so dominant.

It is also hard to argue that organics would be a differentiating factor for them given how heavy the ad load is on their search results page. Sometimes they have graphical mini banners for other commercial search queries on noncommercial search results. Even when you search for a fairly explicit search topic where they should be able to do good from one of their vertical search services they punt. A few days ago I went to their homepage to search for "Dollar General stock" and there were 4 or 5 ads above the organics, then a local search result panel with map, then the mini insert for their stock price data. I searched again just now and there was only 1 ad above the organics, but sometimes they really weigh the page down with ads.

RKG mentioned in their most recent quarterly report that Yahoo & Bing had their fastest ad click growth in a long time, but that came at the cost of seeing sharp declines in organic clicks.
Quote from: RKG report
While Google continues to dominate paid search traffic with 90% click share, Bing Ads and Yahoo Gemini spend have actually increased faster Y/Y than Google in each of the last two quarters.

The growth of paid clicks on Bing and Yahoo appears to have hurt organic search traffic driven by the two engines, as both posted their largest year-over-year (Y/Y) organic visit declines in the history of this report.

While organic search visits driven by Google were roughly flat Y/Y in Q1 2018, Yahoo organic search visits fell 36% and Bing visits fell 20%. Yahoo and Bing’s declines were the largest observed in the history of this report and occurred at the same time that click growth for Bing and Yahoo paid search ads is running near multi-year highs.

Infoseek got borged by the Mouse, but the name has been out of circulation so long only us geezers remember it so you could reuse it and all the kids would think it is new and shiny.

Excite and Lycos are still trying to be portals.  To bad somebody does not try something new in search with them.
If you enter a search into the Lycos search box it will say "There were no results for your search query." Excite is owned by an IAC subsidiary.

I like the new design of HotBot.

Someone tried reviving the Mamma.com domain, but it got hit with an algorithmic penalty as everything on their site seems to rank page 4 or worse in Google.

Many long existing businesses like Cafepress & Overstock.com have been nailed by Google ranking shifts over the past year or two.

Reviving a dead brand might be even harder in the current algorithm given that many active businesses are still getting nailed without going through a shut down period.

CircuitCity.com was relaunched out of bankruptcy once already, but the company which bought the domain more recently now has it redirecting to circuitcitycorporation.com.

Excite, Altavista, LookSmart, Infoseek, Lycos, WebCrawler

Shockingly, half of those or more are still out there and returning search results.
Most search engines beyond Google & Bing are to a large degree retreads of one of those two (at least outside of a few markets like China, Russia & South Korea).

And then some of them are heavily wrapped in ads for one another as the traffic is arbitraged back and forth. IAC still does quite a bit of arb & they're still using domains they've "retired" from an SEO perspective to drive SEM campaigns.

DuckDuckGo has a clean interface.
StartPage.com pitches being privacy focused.
Ecosia plants trees with some portion of your search ad clicks (feeling green, I might just go look for some auto insurance quotes and NYC attorneys real quick :)).

Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 2142
  • What, me worry?
    • View Profile
Re: I kinda miss those 5000 search engines.
« Reply #7 on: May 23, 2018, 02:20:16 PM »
aaron: thanks for your post.  So much new info and good observations.

>directories

They had their day and it is largely over.  I see two scenarios left for directories that would be run as hobbies not for profit. 1. local geo specific search, especially in secondary tourist markets. Use the owner's local knowledge to blow away anything an algo might try to do. Boots on the ground promotion, fliers in local restaurants sort of thing. Don't be dependent on Google.  2. Indieweb and blog subjects.   The Indieweb movement is not hostile to search engines but they aim to have discovery that does not depend on silos and gatekeepers. I don't think they really care about Google listings or rankings. It's about the fun and quality of words and thoughts and open source code that they care about.  You won't make a dime with either hobby only, but I still believe helping people navigate the web is worthwhile.

>Curilie

This is news to me.  I'm glad somebody is giving it a try if for no other reason than they would make a good starter crawl for new search engines.  Things that provide legit web navigation independent of Google are good.

>new search engine

I was musing about this last night.  http://bradfordenslen.com/2018/05/22/provided-that-i.html  If that reminds you of Duckduckgo, that's because it is what I suspect they are doing. It may only be a contingency plan like how Apple kept an Intel version of OSX secretly in dev for years just in case IBM abandoned chip making.  If I were DDG, I would not be spending all the time and money building a search site and brand upon something as ephemeral as Bing. (DDG is really a meta-search: mostly Bing, some Yandex, some their own small crawl with Amazon, Wikipedia, Wikihow and others rolled in.)  I'd have a plan B and C.  And I just have a suspicion that starting their own index, is either DDG's secret endgame or in their contingency plans.

>Hotbot

I was making notes about Inktomi the other day and I could not remember the name Hotbot.  It's kinda neat they are back. Nice clean SERP for now.  They toned down the color scheme.  hhh  The privacy search niche is growing. Any word on who owns them?

>Gigablast

I thought it was long dead.  I know the owner made the Gigablast script Open Source.




aaron

  • Inner Core
  • Full Member
  • *
  • Posts: 137
    • View Profile
Re: I kinda miss those 5000 search engines.
« Reply #8 on: May 25, 2018, 05:20:21 PM »
I have no idea who bought HotBot. Was surprised by the clean search result set though, as typically crawling & indexing is expensive, and most the search engines that could license such data usually want you to eat their ads with the organics. Google even did away with offering a paid ad-free site search service.

Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 2142
  • What, me worry?
    • View Profile
Re: I kinda miss those 5000 search engines.
« Reply #9 on: May 25, 2018, 06:16:47 PM »
I have no idea who bought HotBot. Was surprised by the clean search result set though, as typically crawling & indexing is expensive, and most the search engines that could license such data usually want you to eat their ads with the organics. Google even did away with offering a paid ad-free site search service.

Hotbot has to be using, Bing, Yandex or Gigablast or a blend, there is nobody else out there.  (They might be getting a feed from DDG but I can't imagine why.)  I haven't really compared the serps with Hotbot so I'm speculating.

littleman

  • Administrator
  • Hero Member
  • *****
  • Posts: 3862
    • View Profile
Re: I kinda miss those 5000 search engines.
« Reply #10 on: May 25, 2018, 06:26:03 PM »
>directories ... They had their day and it is largely over.

I think there may be room for  a hybrid directory search engine based on popularity and categories from a site like Reddit.  The idea would be a spidering search engine that uses the sub-reddit data, popularity waves and 'upvote/downvote' scores to affect the ranking algorithm.  There is a lot going on there that could be applied to rank, stuff that DMOZ and sites like DirectHit use to try to do.

The problem with the native Reddit engine is that it only uses its onsite data to display results, but combining that data with actual on page content would make for a great search engine.

Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 2142
  • What, me worry?
    • View Profile
Re: I kinda miss those 5000 search engines.
« Reply #11 on: May 25, 2018, 08:17:28 PM »
littleman, that is a fantastic idea. Hat's off to you.  I've never registered at Reddit so i really don't pay attention to the vote stuff.

In addition to that, I've always thought there was value in a human reviewed directory of websites.  Curated.  I always thought that there needed to be a way to blend that in to the algo of a search engine along with other factors but the Reddit part never would have come to my mind.

>on-site data

This is the problem with dmoz and it's successors: the way we searched dmoz had only site name, short description, category for data. That is about like using meta tags to search a 3 million site DB. Not very useful.

Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 2142
  • What, me worry?
    • View Profile
Re: I kinda miss those 5000 search engines.
« Reply #12 on: May 25, 2018, 10:00:53 PM »
>Hotbot

Pure Bing feed.  Just looked side by side.  Except for some things which get inset on bing like photos, the organic results and the order, are the same.

That is not a bad thing.  It's a good clean serp with no advertising.  Win!

BoL

  • Inner Core
  • Hero Member
  • *
  • Posts: 752
    • View Profile
Re: I kinda miss those 5000 search engines.
« Reply #13 on: May 25, 2018, 11:36:02 PM »
I have to plug mojeek.com here. Known the owner for 10+ years.

2 billion page index with room for growth. Privacy orientated. a crawler engine with its own results, not just another bing clone.

I'm helping out with the knowledge based stuff.

Some of the results are a bit dated and some things definitely need updated in that regard, but definitely one to watch.

(Having studied their results, I think a bigger index and fresher results solves a lot of the relevancy issues, their ranking algo is pretty sophisticated)

Brad

  • Inner Core
  • Hero Member
  • *
  • Posts: 2142
  • What, me worry?
    • View Profile
Re: I kinda miss those 5000 search engines.
« Reply #14 on: May 26, 2018, 12:17:41 AM »
>mojeek.com

BoL thanks for posting that.  I had no idea Mojeek was still around.  I do have a dim memory of it from years ago, back when I was paying attention to search in general.  I think Chris Ridings might have interviewed the owner or he stopped into comment on a thread about early ver. 1 mojeek on SearchGuild forums.

Anyway, it is neat somebody else is out there spidering, respects privacy and is doing it from the UK.

I see they even have site search. Nice.