Add Paged Attention Op for CUDA SM80 support
Merged
AC
1y ago+2298+2298+0123456789012345678901234567890123456789-182-182-012345678901234567890123456789
Description
Adds Paged Attention Op which enables of Paged KV Cache. Inputs to this op are unpadded (packed / varlen) so Cumulative Sequence Lengths are a required input.
Motivation and Context
Adding this op to ONNXRuntime is necessary to allow the GenAI team to enable a continuous batching server API.