feat(connect): Local AI Cluster IDL and commons validation scaffolding

AD
4d ago
+677-3

Description

Implements the IDL schema and commons validation scaffolding for Issue #541: Local AI Cluster (pipeline-parallel sharding across heterogeneous devices). Note: Transport layers and engine-side shard loading are planned as a follow-up.

Key Changes:

  • IDL Schema (idl/connect.proto): Added \ClusterPeerCapability, \ClusterLayerAssignment\ (with documented [layer_start, layer_end)\ half-open ranges), \ClusterStartRequest/Response, \ClusterJoinRequest, \ClusterActivation, \ClusterStopRequest, and \ClusterState.
  • C ABI Interface (core/include/rac/connect/rac_connect.h): Exported
    ac_connect_cluster_start_proto,
    ac_connect_cluster_join_proto,
    ac_connect_cluster_stop_proto, and
    ac_connect_cluster_validate_activation_proto.
  • C++ Native Core (core/src/connect/rac_connect.cpp): Added thread-safe \ClusterRuntime\ state management, layer contiguity validation with automatic sorting, duplicate peer rejection, start idempotency, conflict rejection, and activation tensor bounds checking with uint64 overflow prevention and minimum 2 bytes/element (FP16/BF16).
  • Export Registry (core/exports/RACommons.exports): Registered exported symbols to prevent ABI drift.
  • Schema Lock & Versions (idl/VERSION, idl/SCHEMA_LOCK): Synced with upstream and regenerated schema lock digest.
  • Test Suite (core/tests/test_connect_proto_abi.cpp): Added unit test assertions in \ est_cluster_orchestration()\ covering happy paths, edge cases, conflict scenarios, and isolation.

Progress toward #541

Summary by CodeRabbit

  • New Features
    • Added support for coordinating cluster sessions, including starting and stopping clusters and joining peers.
    • Cluster setup validates model details and peer layer assignments, including coverage and capacity.
    • Cluster status and operation results are returned in serialized responses. Repeating an identical start returns the current status; conflicting starts and mismatched stop requests are rejected.
    • Activation transfers are checked for a matching active cluster, valid layer boundaries and dimensions, and sufficient data.