Skip to content

khmer_nlp_toolkits

Khmer NLP Toolkits is an open-source Python library that provides practical utilities, text processing functions, and NLP components for the Khmer language.

The project is designed to make Khmer NLP easier for: - Researchers - Students - Developers - Production systems - AI applications

It aims to provide a unified, lightweight, and extensible toolkit for common Khmer language processing tasks.


Installation

Requirements

  • Python 3.10+

Install from PyPI

pip install khmer_nlp_toolkits


Quick Start

from khmer_nlp_toolkits.text import *
from khmer_nlp_toolkits.number import num2text

print(num2text(2026))
# ពីរពាន់ម្ភៃប្រាំមួយ

print(anonymizer(("You can contact us by phone at +855 12 345 678, email us at support@example.com, "
"or visit our website at https://www.example.com anytime.")))
# You can contact us by phone at [TEL], email us at [EML], or visit our website at [URL] anytime.

print(text_cleaner(
    "\u200b Hello\u2014\u2014World!!!??      ខ្មែរ   123\u200b "
))
# "Hello - World !? ខ្មែរ 123"


Features

Khmer NLP Toolkits provides modular utilities and NLP components for Khmer language processing.

Commoncrawl 🔗

  • Content Quality Warning
  • Document Quality Filtering
  • Adult Content Filtering

Text Processing 🔗

  • Text Anonymization
  • Text Normalization
  • Text Cleaning
  • Segmentation

Number Processing 🔗

  • number conversion to text

Deduplication 🔗

  • URL deduplication
  • Content Deduplication

Utilities 🔗

  • End2End Pipeline
  • File Utilities Features


Vision

The long-term goal of Khmer NLP Toolkits is to help build a comprehensive open-source ecosystem for Khmer Natural Language Processing and accelerate Khmer AI/NLP research and development.

We hope this project can contribute to: - Better Khmer language tooling - Educational resources - NLP research accessibility - Open-source AI infrastructure for Khmer


Contributing

Contributions, issues, and feature requests are welcome. Feel free to open a pull request or discussion.

Planned Features

  • Khmerize
  • NER
  • Semantic Similarity
  • Text2Num
  • etc.


Contributors


License

This project is licensed under the MIT License.


References

[1] Hour Kaing, Raj Dabre, Haiyue Song, Van-Hien Tran, Hideki Tanaka, and Masao Utiyama. 2025. PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generation. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1309–1322, Abu Dhabi, UAE. Association for Computational Linguistics.

[2] Martin Hosken, Norbert Lindenberg, and Makara Sok. 2022. Khmer encoding structure. Technical report, The Unicode Technical Committee.

[3] Hoang, P. V. (2020). Khmer Natural Language Processing Toolkit. GitHub repository. https://github.com/VietHoang1512/khmer-nltk

[4] Vichet Chea, Ye Kyaw Thu, Chenchen Ding, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. Khmer word segmentation using conditional random fields. Khmer Natural Language Processing, 2015