Publications/Preprints & Manuscripts Under Review
Preprint · 2024
Beyond Uniform Query Distribution: Key-Driven Grouped Query Attention
Zohaib Khan, Muhammad Khaquan, Omer Tafveez, Burhanuddin Samiwala, Agha Ali Raza
arXiv:2408.08454, 2024
Summary
Transformer models are powerful but memory-hungry. Grouped Query Attention saves memory by sharing components across fixed groups of attention heads. We propose variants that form these groups dynamically instead: one allocates queries according to the norms of the key heads in each forward pass, and another tracks how those norms evolve during training. In image classification experiments with Vision Transformers on four datasets, the dynamic variants improve on the original, with gains of up to 8% in accuracy for a large Vision Transformer.
Cite this paper
Zohaib Khan, Muhammad Khaquan, Omer Tafveez, Burhanuddin Samiwala, Agha Ali Raza. Beyond Uniform Query Distribution: Key-Driven Grouped Query Attention. arXiv:2408.08454, 2024.
@misc{khan2024beyond,
author = {Zohaib Khan and Muhammad Khaquan and Omer Tafveez and Burhanuddin Samiwala and Agha Ali Raza},
title = {{Beyond Uniform Query Distribution: Key-Driven Grouped Query Attention}},
howpublished = {arXiv:2408.08454, 2024},
year = {2024},
url = {https://arxiv.org/abs/2408.08454}
}