Language and Task Independent Text Categorization with Simple Language Models

Document Type

Conference Proceeding

Publication Date



We present a simple method for language independent and task independent text categorization learning, based on character-level n-gram language models. Our approach uses simple information theoretic principles and achieves effective performance across a variety of languages and tasks without requiring feature selection or extensive pre-processing. To demonstrate the language and task independence of the proposed technique, we present experimental results on several languages--Greek, English, Chinese and Japanese--in several text categorization problems--language identification, authorship attribution, text genre classification, and topic detection. Our experimental results show that the simple approach achieves state of the art performance in each case.


Presented at the Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, Edmonton, Alberta, Canada, May 27-June 1, 2003.