runtime: pass the step span to step processes as TRACEPARENT

SC
3d ago
+439-0

Summary

When OpenTelemetry is enabled for a DAG, each step now gets the W3C trace context of its own step span in TRACEPARENT, and in TRACESTATE when the span has a trace state. A program that the step runs can then continue the trace with any OTel SDK (environment variables as context carriers), or forward the value to the services it calls, so their spans join the trace of the run.

Nothing changes for a DAG without otel.enabled: the step span is not valid, so no variable is added.

Changes

  • runtime: pass the step span to step processes as TRACEPARENT — right after the Step: <name> span starts, telemetry.InjectTraceContext (no caller on main since #2247/#2248) builds the values from the global W3C propagator, and they go into the step env scope. No new import cycle: internal/telemetry does not depend on internal/runtime.
  • runtime: keep a step-local TRACEPARENT over the step span — precedence as asked in the issue. The two names behave like the other initial current-step values of spec 006:
    • a TRACEPARENT or TRACESTATE from the root scope (DAG env, dotenv, params, inherited base env) is replaced by the step span;
    • when the step env, the selected container env/env_file, or a predecessor output sets either name, Dagu leaves both names as they are. So a step can pass its own context, or clear it with TRACEPARENT=, and the step span's trace state is never paired with a TRACEPARENT of another trace.
  • specs: spec 017 gets a "Trace context projection" table and rules (format, availability, handlers excluded, why the DAGU_ prefix rule does not apply to these standard names); spec 006 lists the names as initial current-step values that are neither protected nor secret-reserved.
  • conformance: a DAG with tracing enabled against an endpoint that accepts no connection: two steps share the run's trace ID with distinct span IDs, a root-scope TRACEPARENT is replaced, a step env value (also an empty one) wins; a DAG without otel gets neither name.

Unit tests (internal/runtime/runner_trace_test.go) use an in-memory span recorder and check that the trace ID in TRACEPARENT is the DAG trace ID, that its parent ID is the span ID of the recorded Step: 1 span, that TRACESTATE follows the incoming trace state, and the precedence cases.

Notes:

  • Lifecycle handlers run without a step span, so they get neither name. Giving them the DAG span is possible but I left it out to keep this small.
  • A step that goes back to retrying and runs again gets a new step span, so each run of the step has its own parent span.

Docs: dagucloud/docs#51 (OpenTelemetry page section "Step Processes" and two rows in the runtime variable reference).

Related Issues

Closes #3206

Checklist

  • Code follows the project style guidelines
  • Self-review of the code has been performed
  • Tests have been added or updated as needed
  • Documentation has been updated as needed
  • Changes have been tested locally

Tested in a golang:1.27.2-bookworm container:

  • make test TEST_TARGET="./internal/runtime/... ./internal/cmn/runenv/... ./internal/telemetry/...": 4100 tests, one failure in internal/runtime/builtin/docker (TestCreateContainerKeepAlive_RetriesWithoutHelper_WhenStartFails), which fails the same way on main in that container (it needs a Docker daemon able to mount the helper).
  • make bin then go test -count=1 ./conformance/spec006_env/ ./conformance/spec012_step_outputs/ ./conformance/spec017_built_in_run_context/: pass. The new conformance test fails with a binary built from main.
  • golangci-lint run on the touched packages with GOOS=linux, windows and darwin: 0 issues; go fix -diff and gofmt -l: clean.

Summary by CodeRabbit

  • New Features
    • Executable steps in traced workflows now receive W3C trace context through TRACEPARENT and TRACESTATE. Step-level values take precedence, including an explicitly empty TRACEPARENT; root-scope values do not override the step’s trace context.
    • When tracing is disabled, these trace context variables remain unset.
  • Documentation
    • Clarified when trace context variables are available and how step-level values affect them.
  • Tests
    • Added conformance coverage for trace propagation, value precedence, and behavior when tracing is disabled.