Collection statistics for fast duplicate document detection
Top Cited Papers
- 1 April 2002
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 20 (2) , 171-191
- https://doi.org/10.1145/506309.506311
Abstract
We present a new algorithm for duplicate document detection thatuses collection statistics. We compare our approach with thestate-of-the-art approach using multiple collections. Thesecollections include a 30 MB 18,577 web document collectiondeveloped by Excite@Home and three NIST collections. The first NISTcollection consists of 100 MB 18,232 LA-Times documents, which isroughly similar in the number of documents to theExcite&at;Home collection. The other two collections are both 2GB and are the 247,491-web document collection and the TREC disks 4and 5---528,023 document collection. We show that our approachcalled I-Match, scales in terms of the number of documents andworks well for documents of all sizes. We compared our solution tothe state of the art and found that in addition to improvedaccuracy of detection, our approach executed in roughly one-fifththe time.Keywords
This publication has 7 references indexed in Scilit:
- Accessibility of information on the webNature, 1999
- Finding Near-Replicas of Documents on the WebPublished by Springer Nature ,1999
- Searching the World Wide WebScience, 1998
- Copy detection mechanisms for digital documentsPublished by Association for Computing Machinery (ACM) ,1995
- Discrimination of authorship using visualizationInformation Processing & Management, 1994
- An algorithm for suffix strippingProgram: electronic library and information systems, 1980
- A vector space model for automatic indexingCommunications of the ACM, 1975