Agha Ali Raza

Publications/Efficient Machine Learning: Data Selection, Pruning, Active & Federated Learning

Conference · 2023

Data Pruning for Efficient Model Pruning in Neural Machine Translation

Abdul Hameed Azeemi, Ihsan Ayyub Qazi, Agha Ali Raza

Findings of EMNLP 2023, Singapore · CORE A*

Summary

Large translation models can be compressed by pruning away parameters, but the pruning process itself is computationally heavy, requiring repeated rounds of training. We show that it can be made cheaper by also pruning the training data. Selecting the hardest-to-learn examples, identified by their training loss, works better than other data selection methods when combined with movement pruning on Romanian-English and Turkish-English translation. Training on this subset reduces the steps needed to converge and the overall training time, and our experiments shed light on how data pruning and model pruning interact.

Cite this paper

Abdul Hameed Azeemi, Ihsan Ayyub Qazi, Agha Ali Raza. Data Pruning for Efficient Model Pruning in Neural Machine Translation. Findings of EMNLP 2023, Singapore.

@inproceedings{azeemi2023data,
  author = {Abdul Hameed Azeemi and Ihsan Ayyub Qazi and Agha Ali Raza},
  title = {{Data Pruning for Efficient Model Pruning in Neural Machine Translation}},
  booktitle = {Findings of EMNLP 2023, Singapore},
  year = {2023},
  url = {https://aghaaliraza.com/publications/2024--EMNLP--Data_Pruning_for_Efficient_Model_Pruning_in_Neural_Machine_Translation.pdf}
}

More on this theme

All 8 works on this theme →