Packed QKV and Rotary Embedding Support for sm<80 GQA
Merged
AC
3y ago+216+216+012345678901234567890123456789-64-64-01234567890123456789
Description
Add support for packed qkv input and rotary embedding with sm<80 using memory efficient attention kernel.
Motivation and Context
Allows lower-end gpus to run gqa with packed qkv input and rotary embedding.