Exploring fact-focused relevance and novelty detection

Purpose - Automated sentence-level relevance and novelty detection would be of direct benefit to many information retrieval systems. However, the low level of agreement between human judges performing the task is an issue of concern. In previous approaches, annotators were asked to identify sentences in a document set that are relevant to a given topic, and then to eliminate sentences that do not provide novel information. This paper aims to explore a new approach in which relevance and novelty judgments are made within the context of specific, factual information needs, rather than with respect to a broad topic. Design/methodology/approach - An experiment is conducted in which annotators perform the novelty detection task in both the topic-focused and fact-focused settings. Findings - Higher levels of agreement between judges are found on the task of identifying relevant sentences in the fact-focused approach. However, the new approach does not improve agreement on novelty judgments. Originality/value - The analysis confirms the intuition that making sentence-level relevance judgments is likely to be the more difficult of the two tasks in the novelty detection framework.