eXtended cumulated gain measures for the evaluation of content-oriented XML retrieval
- 1 October 2006
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 24 (4) , 503-542
- https://doi.org/10.1145/1185877.1185883
Abstract
We propose and evaluate a family of measures, the eXtended Cumulated Gain (XCG) measures, for the evaluation of content-oriented XML retrieval approaches. Our aim is to provide an evaluation framework that allows the consideration of dependency among XML document components. In particular, two aspects of dependency are considered: (1) near-misses, which are document components that are structurally related to relevant components, such as a neighboring paragraph or container section, and (2) overlap, which regards the situation wherein the same text fragment is referenced multiple times, for example, when a paragraph and its container section are both retrieved. A further consideration is that the measures should be flexible enough so that different models of user behavior may be instantiated within. Both system- and user-oriented aspects are investigated and both recall and precision-like qualities are measured. We evaluate the reliability of the proposed measures based on the INEX 2004 test collection. For example, the effects of assessment variation and topic set size on evaluation stability are investigated, and the upper and lower bounds of expected error rates are established. The evaluation demonstrates that the XCG measures are stable and reliable, and in particular, that the novel measures of effort-precision and gain-recall ( ep / gr ) show comparable behavior to established IR measures like precision and recall.Keywords
This publication has 14 references indexed in Scilit:
- Precision recall with user modeling (PRUM)ACM Transactions on Information Systems, 2007
- Binary and graded relevance in IR evaluations—Comparison of the effects on ranking of IR systemsInformation Processing & Management, 2005
- The concept of relevance in IRJournal of the American Society for Information Science and Technology, 2003
- Cumulated gain-based evaluation of IR techniquesACM Transactions on Information Systems, 2002
- Using graded relevance assessments in IR evaluationJournal of the American Society for Information Science and Technology, 2002
- Relevance judgements for assessing recallInformation Processing & Management, 1996
- Variations in relevance judgments and the evaluation of retrieval performanceInformation Processing & Management, 1992
- The pragmatics of information retrieval experimentation, revisitedInformation Processing & Management, 1992
- Relevance assessments and retrieval system evaluationInformation Storage and Retrieval, 1968
- Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systemsAmerican Documentation, 1968