System and Method for Learning Balanced Relevance Functions from Expert and User Judgments

Document Type


Publication Date



The present invention relates to systems and methods for determining a content item relevance function. The method comprises collecting user preference data at a search provider for storage in a user preference data store and collecting expert-judgment data at the search provider for storage in an expert sample data store. A modeling module trains a base model through the use of the expert-judgment data and tunes the base model through the use of the user preference data to learn a set of one or more tuned models. A measure (B measure) is designed to evaluate the balanced performance of tuned model over expert judgment and user preference. The modeling module generates or selects the content item relevance function from the tuned models with B measure as the selection criterion.


United States Patent #2008/0301069 A1