I was an early search engine buff. I was right there using boolean searches on AltaVista to catch plagiarists in my wife’s classes. And I started using Google in 1998.
Google was a miracle in those days. To get anything relevant out of AltaVista you had to be a complete AltaVista kung-fu master. You had to use a few clever booleans and then get to the third page before finding what you really wanted. But finding that result–using your fu and getting it–was a good feeling. Well, Google was different. You would just type in your search and BAM, up pops the most relevant result, right there in the top 3-5 results. It was amazing.
What about now? Where is Google now? Let’s search on a topic that I take a particular interest in: Mac Twitter clients. Try the search “twitter client mac”.
First result: a TechCrunch article on Tweetie. Seems reasonable, no? NO, it isn’t reasonable! Where is the web page for Tweetie itself? You know, the product that we are actually looking for? It comes in seventh, after such giants of the genre as Tweetr (Huh? I’ve never heard of anybody using Tweetr.) And Twit Menulet, where do we come in, about one month after launch? At the top of page 2, in 14th place. Pretty darn good for a site that didn’t exist, basically, until yesterday.
These results may seem OK to you. Perhaps you, too, are a refugee from the AltaVista era and consider any relevant result in the top 30 to be a pretty good achievement.
To those of us who live and breathe search, who literally eat our search results, these results reek. Because they show how the success of Google has actually ruined search on the web. Think about these points:
- TechCrunch shouldn’t be the first result. OK, TechCrunch is a great site and very relevant indeed, so it should be up there, but not first. Why is it first? Because TechCrunch, like Wikipedia, is a PageRank hog, using “nofollow” almost everywhere to hoard its PageRank. None of the opinions of TechCrunch commenters, many of whom are quite astute, are included in computation of the search result. The practice of using “nofollow” everywhere has the dual effect of inflating the importance of internal pages and degrading the influence of thoughtful netizens. Using the input of these netizens is Google’s raison d’etre–and it has been absolutely destroyed by “nofollow”. (BTW, I don’t blame TechCrunch or Google for this problem, I understand spam, but it is a shame.)
- Where are the results from Twitter?! I.e., the status updates pointing to the best Twitter clients for Mac? Where are they?! The most relevant possible results, in other words, are missing. Where could they be? It’s always the same answer, these days: they are hidden behind a “nofollow” wall, in this case Twitter’s wall. Nofollow, in other words, has excluded the most relevant possible results from this search.
- The last problem, and I hate to admit it, is this: why is Twit Menulet listed in the top 20 results? We’ve been on the web for a week, we have very few users, and yet we’re right up there within a few results of Tweetie. I hate to admit it, but this is silly. The reason? We’ve gone out of our way to target the “twitter client mac” search words, putting, for example, those words in the title of our page. We’ve also taken care to link to our home page with those words.
In summary, the magic of PageRank has completely evaporated. Heavy reliance on PageRank no longer produces search results that are almost magically correct. Instead, PageRank produces a bunch of results that are ranked based on the greediness of page authors for one thing: PageRank.
I think that the solution to these problems is surprisingly obvious, and I’ll cover it in my next post!