Jennifer Zaino
SemanticWeb.com Contributor

Anyone who's been following Microsoft's recent acquisition of Powerset will get to hear more about how the technology is evolving as the semantic search market itself evolves at Jupitermedia's Web 3.0 conference in Santa Clara, Calif., this week. Scott Prevost, general manager and director of product at Powerset will be giving a keynote presentation, on "The Road to Semantic Search."

Prevost chatted with Semanticweb.com about how search on the web has evolved from directories to keywords to semantic search, and about how Powerset aims to show how semantic search can improve relevance and change the user search experience.

Semanticweb.com: Why do we need to refine and improve the search experience?

Prevost: The reason why we should pay attention is because search isn't a solved problem. A lot of queries just aren't answered properly, users get a set of links that don't get them what they want. ... It's hard for users to specify their intent clearly enough for a search engine to make use of it and understand what kind of information to present and how, because basically the user has to take what they need to say and then condense it into disconnected keywords. We let people express themselves more naturally, but the flip side of that is the hard work building a search engine that tries to understand what's in the document -- not just the words but how they fit together and form concepts and relationships and extract from users' query to the meaning expressed in documents. Semantic search vastly improves relevance by having a search engine understand at some level what's in the document. The nice thing is it doesn't require 100% understanding to improve search. We can incrementally improve search over the next year, the next five and ten years, and even beyond that to continually get better and better. That's one way for semantic search to change the search landscape.

It also enables new ways of summarizing information, aggregating information from multiple sources, presenting information and starting to explore that. In Powerset.com. you can see snippets of results that highlight the answer to your query so you don't have to click through to the page -- and it may be highlighted words that weren't even in your query. Keyword search just puts in bold the word you put in the search box. We talk about highlighting the things you didn't put in the search box. Or we can generate a list of answers: if I type into Wikipedia what causes cancer, I can find every sentence in Wikipedia that says that and give you a list. That is only possible with an understanding of what is on the page -- not just picking pages by what words occur on them.

Semanticweb.com: Why does the advertising community need better search?

Prevost: We're not as far along in terms of productizing the technology for that kind of thing. But it very much applies. So just as we would use sematnic techniques to match users' queries to the meaning of what is in a document, we could do the same thing for advertising models. So now you are not bidding on particular keywords but on concepts people express in their queries.

Semanticweb.com: What would you say sets Powerset apart from many of the companies that seem to have jumped into the semantic web search space?

Prevost: There's obviously a lot of different kinds of approaches that people are using in search. In a lot of the smaller search startups -- and when I got into this a few years ago I was shocked to learn how many there were -- most are doing small tweaks around the UI for search but that's not the next big thing. Others do niche verticals -- maybe focus on travel or health. Powerset is in a category with, maybe, Hakia, where we are really trying to change the game for how documents are indexed, and how search engines understand the documents it points you to, and that 's where we make a big impact. We're throwing away the keyword model and replacing it with one that is far more robust because of much deeper analysis of documents.

Semanticweb.com: How dependent are semantic search engines on web site publishers marking up their documents?

Prevost: This is something that differentiates Powerset from other players in the semantic web area. Much of the focus has been about marking things up so they are machine readable, and much of that requires the sites to actually do some work. What Powerset does is, we have automatic techniques for extracting structure from free text, so sites don't need to mark things up semantically. We run the content through our semantic pipeline and produce another representation that we can use that has that markup, but it's done automatically, to identify key concepts on the page, the way words relate to each other on the page. We're taking a much more automated approach to the semantic web, which we think lets people be more natural in the way they author and search content.

Semanticweb.com: Where are we in the maturity of semantic search?

Prevost: Right now in some sense we are obviously at the beginning of a long journey. But in another sense natural language technology has been around for 30 years. But it's only now that we have the computational resources, the scalable computing to do it on a big scale. At Powerset we've shown that by doing semantic analysis on documents and queries we can create better relevance, so we can surpass the keyword technologies on pure relevance. We also have demonstrated how we can change the overall user experience in ways that can delight users, getting them an answer on a search result page or providing an automatic summary of documents or a collection of important relationships about a particular concept drawn from a bunch of different documents. But the journey will involve getting better linguistic models, improving systems' understanding of vocabularies and how words relate to each other.

Semanticweb.com: How much does cross-language support play into the semantic search picture?

Prevost: That is a goal. Powerset works in a linguistic framework that supports many languages, but the idea is there is a representation that all of these languages can point to. So actually quite a bit of work was done on this from some people now at Powerset on using some of this core technology for processing different languages. We haven't released anything yet-- a lot of work is involved in bringing new languages online, on the order of two man years per language.

Semanticweb.com: Has, or how has, being a part of Microsoft changed plans at Powerset?

Prevost: Certainly in terms of the vision for the technology we see the acquisition as an accelerant. It provides us with the human resources, machine resources, and world class infrastructure to support what we're doing. The vision is pretty much the same. Obviously there's more room for exploration but the change is that we always had a long term vision, but as a startup you can only focus on fairly short term accomplishments. ...At Microsoft we have the freedom to maintain that long term view and focus not just on six months out but five years out.