Thursday, March 10, 2011

Experiential versus Informational Search

Yesterday Louis Gray posed a set of questions which seem like they should be easy to answer, but aren't. While reading it one of the questions stood out, for obvious reasons.

"2. When was the first time Denton Gentry left a comment on my blog?"

Louis had sent that question via email earlier in the day, and it turned out to be very difficult to answer. My profile on shows comments going back to July 2009. That should be definitive, but unfortunately isn't correct as manual checking had already turned up earlier comments. In the end Louis answered his own question by searching his email for Disqus notifications. It was a full year before the first comment shown on my profile page. The other questions were similar: find the first citation. From a technical perspective, it should be easy to answer questions like this as all of the information is available. That it isn't easy is a reflection of economic reality. There is infrequent demand for it.

I'd like to flip it around, though: why is email able to answer questions like this? You can search email and sort it by date. You can find emails around a particular time. You can find emails which happened at about the same time as some other event which is unrelated, but intertwined in your memories. Why is email structured this way?

I suspect this is a reflection of human psychology. Email is information which we personally experienced. It exists in our own memories, albeit dimly or imperfectly. When we go to search for it, we're searching as an extension of our own memory. Its Experiential search, not Informational, and email services which don't match our expectations in this regard get less traction. This is also what makes services like Evernote so useful, letting us organize and search arbitrary information Experientially.

In comparison when searching for something we never personally experienced we're looking for information which we know must exist, and we just need to find it. Search engines are designed to this expectation.

The disconnect occurs when we want an Experiential search over an Informational dataset. Organizing arbitrary information in a way which maps to what we'd expect had we personally experienced it is an unsolved problem. It has been a rich field of speculation in science fiction, as authors have postulated implanted memories and neural interface.

Will there be developments in this area? Clearly there is at least some demand, as LexisNexis can answer such queries for the subset of publications they handle. Its something we'll need to work on if we're going to make the world even more like science fiction.