A lightweight and efficient tool for cleaning Web pages

Conference contribution
(Conference Contribution)


Publication Details

Author(s): Evert S
Publishing place: Marrakech, Morocco
Publication year: 2008
Conference Proceedings Title: Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008)
Language: English


Abstract


Originally conceived as a ”naive” baseline experiment using traditional n-gram language models as classifiers, the NCleaner system has turned out to be a fast and lightweight tool for cleaning Web pages with state-of-the-art accuracy (based on results from the CLEANEVAL competition held in 2007). Despite its simplicity, the algorithm achieves a significant improvement over the baseline (i.e. plain, uncleaned text dumps), trading off recall for substantially higher precision. NCleaner is available as an open-source software package. It is pre-configured for English Web pages, but can adapted to other languages with minimal amounts of manually cleaned training data. Since NCleaner does not make use of HTML structure, it can also be applied to existing Web corpora that are only available in plain text format, with a minor loss in classfication accuracy.



FAU Authors / FAU Editors

Evert, Stefan Prof. Dr.
Lehrstuhl für Korpus- und Computerlinguistik


How to cite

APA:
Evert, S. (2008). A lightweight and efficient tool for cleaning Web pages. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008). Marrakech, Morocco.

MLA:
Evert, Stefan. "A lightweight and efficient tool for cleaning Web pages." Proceedings of the Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008) Marrakech, Morocco, 2008.

BibTeX: 

Last updated on 2018-11-08 at 00:10