fix(trace): restore gRPC server spans and add HTTP server spans

Merged
MI
1mo ago
+202-19

Problem

With tracing enabled, traces rooted at middleware spans (WithCurrentMembershipsMiddleware, WithCurrentAPITokenAndOrgMiddleware, ...) instead of transport-level server spans — for both gRPC and HTTP entrypoints. See #3448.

Root causes

  1. gRPC: kratos' grpc.Options assigns s.grpcOpts instead of appending (kratos/v2@v2.9.2/transport/grpc/server.go:124). NewGRPCServer passed grpc.Options(grpcLib.StatsHandler(otelgrpc.NewServerHandler())) and then, when max_recv_msg_size was configured (Helm controlplane.grpcMaxRecvMsgSize), a second grpc.Options(grpcLib.MaxRecvMsgSize(...)) — silently wiping the stats handler and with it all gRPC server spans.
  2. HTTP: the Kratos HTTP server had no server-side span instrumentation at all (no tracing.Server() middleware, no otelhttp), so transcoded REST requests started their trace at the first middleware span, with no traceparent extraction.

Fix

  • grpc.go: pass all raw grpc-go server options in a single grpc.Options(...) call, with a comment explaining the assignment semantics so it doesn't regress.
  • http.go: prepend kratos tracing.Server() as the outermost HTTP middleware. It uses the global tracer provider and extracts W3C tracecontext by default. gRPC is untouched there (already covered by the stats handler, so no double spans).

Verification

New regression tests in app/controlplane/internal/server/otel_test.go spin up real Kratos servers with an in-memory exporter:

  • TestGRPCServerSpanWithMaxRecvMsgSize — gRPC server span grpc.health.v1.Health/Check is produced with MaxRecvMsgSize set. During development the split-grpc.Options variant was confirmed to produce zero spans, reproducing the bug.
  • TestHTTPServerSpanFromTracingMiddleware — HTTP request produces a server span named after the route (/ping).

go build ./app/controlplane/... ./app/artifact-cas/..., go vet, and go test ./app/controlplane/internal/server/ all pass. artifact-cas checked: only one grpc.Options call, unaffected.

Fixes #3448

🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri