Mining comparable bilingual text corpora for cross-language information integration
- 21 August 2005
- proceedings article
- Published by Association for Computing Machinery (ACM)
- p. 691-696
- https://doi.org/10.1145/1081870.1081958
Abstract
Integrating information in multiple natural languages is a challenging task that often requires manually created linguistic resources such as a bilingual dictionary or examples of direct translations of text. In this paper, we propose a general cross-lingual text mining method that does not rely on any of these resources, but can exploit comparable bilingual text corpora to discover mappings between words and documents in different languages. Comparable text corpora are collections of text documents in different languages that are about similar topics; such text corpora are often naturally available (e.g., news articles in different languages published in the same time period). The main idea of our method is to exploit frequency correlations of words in different languages in the comparable corpora and discover mappings between words in different languages. Such mappings can then be used to further discover mappings between documents in different languages, achieving cross-lingual information integration. Evaluation of the proposed method on a 120MB Chinese-English comparable news collection shows that the proposed method is effective for mapping words and documents in English and Chinese. Since our method only relies on naturally available comparable corpora, it is generally applicable to any language pairs as long as we have comparable corpora.Keywords
This publication has 13 references indexed in Scilit:
- A cross-collection mixture model for comparative text miningPublished by Association for Computing Machinery (ACM) ,2004
- Challenges in information retrieval and language modelingACM SIGIR Forum, 2003
- Two-stage language models for information retrievalPublished by Association for Computing Machinery (ACM) ,2002
- A study of smoothing methods for language models applied to Ad Hoc information retrievalPublished by Association for Computing Machinery (ACM) ,2001
- A bootstrapping method for extracting bilingual text pairsPublished by Association for Computational Linguistics (ACL) ,2000
- Parallel Text ProcessingPublished by Springer Nature ,2000
- Information retrieval as statistical translationPublished by Association for Computing Machinery (ACM) ,1999
- A language modeling approach to information retrievalPublished by Association for Computing Machinery (ACM) ,1998
- Resolving ambiguity for cross-language retrievalPublished by Association for Computing Machinery (ACM) ,1998
- Extraction of lexical translations from non-aligned corporaPublished by Association for Computational Linguistics (ACL) ,1996