Agha Ali Raza

Publications/Preprints & Manuscripts Under Review

Preprint · 2024

Beyond Uniform Query Distribution: Key-Driven Grouped Query Attention

Zohaib Khan, Muhammad Khaquan, Omer Tafveez, Burhanuddin Samiwala, Agha Ali Raza

arXiv:2408.08454, 2024

Summary

Transformer models are powerful but memory-hungry. Grouped Query Attention saves memory by sharing components across fixed groups of attention heads. We propose variants that form these groups dynamically instead: one allocates queries according to the norms of the key heads in each forward pass, and another tracks how those norms evolve during training. In image classification experiments with Vision Transformers on four datasets, the dynamic variants improve on the original, with gains of up to 8% in accuracy for a large Vision Transformer.

Cite this paper

Zohaib Khan, Muhammad Khaquan, Omer Tafveez, Burhanuddin Samiwala, Agha Ali Raza. Beyond Uniform Query Distribution: Key-Driven Grouped Query Attention. arXiv:2408.08454, 2024.

@misc{khan2024beyond,
  author = {Zohaib Khan and Muhammad Khaquan and Omer Tafveez and Burhanuddin Samiwala and Agha Ali Raza},
  title = {{Beyond Uniform Query Distribution: Key-Driven Grouped Query Attention}},
  howpublished = {arXiv:2408.08454, 2024},
  year = {2024},
  url = {https://arxiv.org/abs/2408.08454}
}