Agha Ali Raza

Publications/Natural Language Processing & LLMs for Low-Resource Languages

Conference · 2018

Urdu Word Segmentation using Conditional Random Fields

Haris Bin Zia, Agha Ali Raza, Awais Athar

COLING 2018, Santa Fe, NM, USA · CORE A

Summary

Before a computer can process text, it must know where words begin and end. In Urdu this is surprisingly hard: spaces are often omitted between words and inserted inside them, a consequence of how letters in the Arabic script join. We present a word segmentation system based on conditional random fields with orthographic, linguistic, and morphological features. It learns to predict both word boundaries and sub-word boundaries, achieving F1 scores of 0.97 and 0.85 respectively on a manually annotated corpus. Code and corpus are publicly available.

Cite this paper

Haris Bin Zia, Agha Ali Raza, Awais Athar. Urdu Word Segmentation using Conditional Random Fields. COLING 2018, Santa Fe, NM, USA.

@inproceedings{zia2018urdu,
  author = {Haris Bin Zia and Agha Ali Raza and Awais Athar},
  title = {{Urdu Word Segmentation using Conditional Random Fields}},
  booktitle = {COLING 2018, Santa Fe, NM, USA},
  year = {2018},
  url = {https://aghaaliraza.com/publications/2018--COLING--Urdu_Word_Segmentation_using_Conditional_Random_Fields.pdf}
}

More on this theme

All 18 works on this theme →