A few weeks ago, I wrote about why search matters (and the answer is pancakes). Last year, I wrote about all of the things wrong with search.

But there is something that has been troubling me about search, and it has taken some long thinking and reflection to get to it. Search is good for information that is missing, like what was Babe Ruth’s WAR? Google and Bing can tell you who Babe Ruth was, along with what a WAR is (wins above replacement value— a geeky baseball stat). Those are kind of needle in the haystack factual tidbits that can satisfy your query. And I guess that last word, query, is the salient issue.

Search is about queries. Questions, with answers. To a search engine’s thinking, there is an algorithmic way to determine an authoritative response. And yeah, for fact-based stuff there probably is. But what about subjective arenas? The search engines respond to the notion of subjectivity through personalization filters. And these create their own issues with insulation and skewed viewpoints. (A filter essentially tries to guess what you would want to click on based on other stuff you’ve clicked on…). Eventually, personalization becomes a self-reinforcing algorithm which makes your world smaller. I wrote about filters earlier, and  I strongly recommend Eli Pariser’s book, The Filter Bubble: What the Internet Is Hiding from You (affilate link). He gives a clear view of how filters can make your world smaller.

But my real issue with search is that of discovery. Search is good for finding what you are missing, not what you are looking for…

Recently, I was looking for a gift. I didn’t exactly know what I was looking for. I had concepts.: Artisinal. Elegant. Minimal. Maybe Metal. Perhaps finely woodworked. Special.

Using that specific query, Google pointed me to a content farm (wasn’t Panda and the Revenge of Panda supposed to kill that stuff?). A fully unsatisfactory experience. And mind you, I am not blaming Google for the algorithmic disappointment. What they try to do is incredibly difficult and has amazing complexity. It is orders of magnitude more difficult than any technology project ever embarked upon. But the real disconnect is that I cannot express what I am looking for in any way that is truly structured. I have attributes. I have concepts. But I have no language to express what I am looking for because I don’t know what it is. I am looking for an intrinsic value, an essence. It is as if I am looking for a product that embodies the BMW motorcycle in Zen and the Art of Motorcycle Maintenance: An Inquiry into Values (affiliate link, again). I am looking for quality.


Is it possible for Google (or Bing) to infer and understand such a nebulous and subjective an essence as quality? Because the algorithm is machine driven, machine measured and objectively informed by click data, an algorithm cannot assess what it is serving up as a result, but rather only a fact that based on the parameters embodied within the algorithm, that a particular page is our best mathematical guess that it matches your query. But the engine has no idea of the intrinsic value of the result. Search is still in its nascent stages. It is a baby, and it has much to learn. Perhaps it will require us getting to the point where Google’s algorithm is in some state of sentience, perhaps it will evolve with deeper human influence on the algorithm, or perhaps a better way to get nuanced feedback into the algorithm.

Search today is good for things that are lost or need to be found. It doesn’t help you discover. Who wants to make a discovery engine?

2 responses

Do you want to comment?

Comments RSS and TrackBack Identifier URI ?

This is a similar problem I’ve been having with the hot music service, Spotify. At first, I was gobsmacked. All my music, every album I can think of, all right there? I gorged. And then I realized I was listening to different albums from artists I was already familiar with. I wanted to expand my tastes. I tried some of my friend’s playlists, but they did not match any of my tastes. I tried some of Spotify’s suggestions and, while good, they were infrequent.

In the end, I found myself going back to Pandora, for it’s ability to tailor to my tastes, and help me discover new music. It also allows you to set up stations based on concepts, like your example of the gift concepts. And I have purchased more music as a result of the discovery engine.

On the iPhone, I see a similar trend. I use Google Reader for my news reading. Unfortunately, voracious reading means I frequently exhaust the feeds. Now, an app called News360 lets you plug in your interests, and customizes the new based on your preferences, pulling in multiple data sources. Result? A discovery of more news content that I click to, and am exposed to their ads, generating revenue.

Discovery engines are definitely tricky. You mention filters, and that usually becomes an example of what many discovery engines become. You need some way to be able to expose people to new content, without making them feel they are missing out on their core content.

November 14, 2011 7:20 pm

Josh, you are spot on. Prensentation of a mammoth data set (Spotify music) is inevitably difficult. and when overwhelmed with choices, people invariably start to gravitate towards things that they recognize. We are, after all, masters of pattern recognition (which is why you can see faces where they don’t exist). Discovery is amazingly hard. To algorithmically capture the delight of a new discovery is amazingly complex (unimaginable for me, frankly), but in order to truly tame the internet, we need to have the equivalent of a guide, the benefit of an experienced friend. I am not sure if we have the capacity to do that today, but mammoth data requires mammoth thinking. Google is an incredible, learning machine. They mistake happy users with clickthroughs and low bounce rate. Those users may not be happy, but rather beaten down.

I want an engine that understands me. Is that too much to ask for? 😉

November 14, 2011 7:44 pm

Comment now!