[CUDA Plugin EP] Add provider options: user_compute_stream, do_copy_in_default_stream, use_ep_level_unified_stream, external allocator
Description
Wires up the remaining provider options from OrtCUDAProviderOptionsV2 that were previously missing in the CUDA plugin EP (CudaEpFactory::CreateEpImpl). This closes the "Provider Options Gaps" section (§2) of the plugin EP tracking doc for the stream/allocator options that are not blocked on the tunable ops framework.
The bundled CUDA EP already supports these through OrtCUDAProviderOptionsV2. This PR brings parity to the plugin EP so that users can configure external streams, copy behavior, unified stream mode, and external GPU allocators via session options.
Summary of Changes
Stream and copy options
| File | Change |
|---|---|
onnxruntime/core/providers/cuda/plugin/cuda_ep.h | Add has_user_compute_stream, user_compute_stream, do_copy_in_default_stream, use_ep_level_unified_stream, external_alloc, external_free, external_empty_cache to Config. |
onnxruntime/core/providers/cuda/plugin/cuda_ep_factory.cc | Parse all new options from session config with hex-safe pointer parsing (base 0 + full-string validation). Add validations: incomplete external allocator warning, mutual exclusion of user_compute_stream + external_allocator, mutual exclusion of user_compute_stream + enable_cuda_graph. Auto-force unified stream when user stream or external allocator is configured. |
onnxruntime/core/providers/cuda/plugin/cuda_ep.cc | Use user_compute_stream in CreateSyncStreamForDeviceImpl via new InitHandlesWithUserStream. Disable concurrent runs when unified stream is active. |
onnxruntime/core/providers/cuda/plugin/cuda_stream_plugin.cc/h | New CudaSyncStream::InitHandlesWithUserStream() — wraps user CUDA stream with owned cuBLAS/cuDNN/cuBLASLt handles without taking ownership of the stream itself. |
External allocator
| File | Change |
|---|---|
onnxruntime/core/providers/cuda/plugin/cuda_allocator_plugin.cc/h | New CudaExternalDeviceAllocator class delegating alloc/free/empty_cache to user-provided function pointers. |
onnxruntime/core/providers/cuda/plugin/cuda_ep_factory.h | Extend DeviceCacheEntry with external allocator state, refcount, and UseExternalAllocator() helper. |
onnxruntime/core/providers/cuda/plugin/cuda_ep_factory.cc | CreateAllocatorImpl creates CudaExternalDeviceAllocator when external pointers are configured; ReleaseAllocatorImpl handles its refcount and cleanup. |
Kernel adapter plumbing
| File | Change |
|---|---|
onnxruntime/core/providers/cuda/plugin/cuda_kernel_adapter.h | Add do_copy_in_default_stream to CudaKernelAdapterRuntimeConfig; add CUDAExecutionProvider::DoCopyOnDefaultStream(). |
Tests
| File | Change |
|---|---|
onnxruntime/test/framework/dynamic_plugin_ep_test.cc | New test CudaKernelAdapterRuntimeConfigExposesDoCopyInDefaultStream; fix shared_ptr access pattern in existing tests (auto& → auto, . → ->). |
Testing
- All 33
*Plugin*gtest cases pass (4DynamicPluginEpInfraTest+ 29PluginExecutionProviderTest). - Plugin
.sobuilds clean with no warnings. - Tests under
#if defined(ORT_USE_EP_API_ADAPTERS)guard validate the kernel adapter plumbing when built as a plugin library; they are intentionally excluded from the monolithic test binary (known pre-existing limitation due toORT_API_MANUAL_INITheader requirement).
Motivation and Context
The CUDA plugin EP is a standalone shared library EP using the OrtEpFactory API. It was missing support for several OrtCUDAProviderOptionsV2 options that users rely on in production: external CUDA streams (for framework interop with PyTorch/TensorFlow), copy-stream behavior, unified stream mode, and external GPU memory allocators (for memory pool sharing across frameworks). This PR brings the plugin EP to parity with the bundled EP for these options.
Checklist
- Tests added/updated
- No breaking changes
- CI passes (local build + test verified)