deadlock Message Board
[ March Full List | Reply ]

The SEs are going to hell in a handbasket
From David on 12 March '99
There have been several comments on the SEs "getting weird."
They are. Based on some research, and a dash of number crunching:

1. The major SEs have gone to hell in a handbasket, with the
exception of AV -- and AV has been acting a bit weird also.

2. Inktomi-powered sites are now officially powered by Direct
Hit, as of March 3. Actually Inktomi and Direct Hit results were
identical weeks before that.

3. Direct Hit works on a boost-links-that-have-been-hit protocol.
As usual, someone thought they had the final algorithm. Instead
the DH protocols are causing "opportunistic" pages such as porn
or quick money to rise in the rankings.

4. This is the pattern:

SE is taken over by suits.
Shortly after, SE brags about "increasing relevancy."
Weeks later, relevancy goes to hell in a handbasket.

Suspicion: the suits force popularity theories on the techies;
or maybe the suits simply demand a new theory, and techies respond
with any crackpot theory at hand.

With AV moving towards an IPO, and their CTO talking about
increasing relevancy, it too may soon become a disaster.

5. Google is capturing market share. Although they are weighting
for links, the SE works curiously well. Professional researchers
are increasingly using it.

6. (now the Open Directory Project) is outpacing Yahoo
in listings per week, by 280%. At that rate they will have more
listings in about one year. Since Netscape is going to give them
a home page link next week, traffic should rocket. NewHoo is already
1/3 of Yahoo's size, and they only started in mid-1998.

Exact traffic: unknown.

7. While's total 1 million URLs listed are not quite believeable
on a spreadsheet, they are clearly growing faster than Yahoo. At 150
editors vs Yahoo's 40-50, at 2 minutes per addition, and 40 hours per
week, the math stands clear.

Exact traffic: unknown.


All this was predicted some time ago: the SEs may become a) useless, and
b) complete whores. However, the suits are missing some facts. Searching
is still a major draw for visitors, running from 15% to 100%, depending
on the SE. (Google is obviously 100% searchers, since they have no other

Since starting an SE (only) needs code skill, time, access to high-speed
servers, and bandwidth, it is possible that the SE industry is like
the air carrier industry: there will always be a new player.

Google--running on Linux-configured boxes--already claims 60 million URls.
Considering their calculations/second, it's possible; Stanford University
funding seems without limit. But as prices plummet on hardware and
bandwidth, perhaps anyone with a good hypothesis will be able to enter
the game.


Feedback welcome. Please, no debates.

Reply to this message

Required Details
Your name:

Optional Details
Home page title:
Home page URL:
(if you'd like to receive
automatic replies by email)

Your Message