[Geneva] Revert ETW payload framing that broke the agent
Fixes #
Design discussion issue #
Changes
Reverts the EtwDataTransport changes from #4729, which altered the bytes written to ETW and caused the agent to drop all logs and spans emitted over EtwSession=... with Bad forward protocol format.
What broke
#4729 added a byte[] data parameter to InformationalEvent so that the runtime-generated ETW manifest would match the payload, allowing .NET EventSource consumers to subscribe to the provider. In the ETW manifest format, .NET always prepends a synthetic 4-byte length to a declared byte[] field, so SendEvent was changed to write two EventData descriptors instead of one:
before (<= 1.16.0): [ raw payload ]
after (1.17.0): [ 4-byte LE length ][ raw payload ]
WriteEventCore concatenates descriptors into the ETW user-data blob, so every record gained a 4-byte prefix.
The agent reads that blob verbatim as the forward protocol frame, which must begin with the msgpack array marker 0x93. With the prefix in place it saw a length byte instead and rejected the record.
The byte layout is not just inferred from the WriteEventCore contract. Running the new tests against the #4729 implementation shows .NET's own manifest decoder reconstructing [1, 2, 3, 4] from the event, which is only possible if the blob on the wire was [4-byte length][payload].
Blast radius
| Path | Affected |
|---|---|
Logs and traces on Windows via EtwSession=... (MsgPackLogExporter, MsgPackTraceExporter) | Yes |
Metrics (MetricWindowsEventTracingDataTransport) | No, still writes a single descriptor |
| Linux user_events / UDS | No |
TLD path (EventProvider) | No |
1.16.0 is unaffected. The diff between Exporter.Geneva-1.16.0 and Exporter.Geneva-1.17.0 contains exactly one functional file, EtwDataTransport.cs, plus a core version bump.
The fix
Restores the pre-#4729 behaviour: a single descriptor carrying the unframed payload, and a parameterless InformationalEvent. The manifest deliberately declares no fields, and there is now a comment on the event saying why, so this is not "fixed" again by accident.
The only deltas from the pre-#4729 file are the expression-bodied Send/IsEnabled introduced later by #4849, which are unrelated and preserved, and that comment. The #pragma warning disable IDE0060 added by #4849 is dropped, since it existed only to silence the unused parameter this PR removes.
Tests
Two regression tests, both of which fail against the #4729 implementation:
TraceEventManifestDeclaresNoPayloadTemplateassertsEventSource.GenerateManifestemits<event value="100">with notemplateattribute. The manifest is what native consumers read, and a declared field is exactly what triggers the synthetic length.SendEventWritesRawPayloadWithoutDeclaredFieldsasserts the emitted event decodes to zero payload fields.
Verified by checking out the #4729 implementation and running them:
########## BEFORE - PR #4729 implementation ##########
SendEventWritesRawPayloadWithoutDeclaredFields [FAIL] Actual: 1
TraceEventManifestDeclaresNoPayloadTemplate [FAIL] Actual: template="InformationalEventArgs"
Failed! - Failed: 2, Passed: 0
########## AFTER - this PR ##########
Passed! - Failed: 0, Passed: 2
The roundtrip test added by #4729 could not catch this. It used an in-process EventListener, which strips the synthetic length while decoding, so it passed while the bytes on the wire were wrong. Raw ETW bytes are not observable in-process, which is why these tests assert the manifest and the decoded field count instead.
EtwCollection is removed, as it existed only to serialize that roundtrip test. Neither new test needs a real ETW session. Doing so surfaced a latent race in the listener helper, which stored events in a plain List<T> while OnEventWritten can be called from any thread, so it now uses a ConcurrentQueue.
Full suite passes on all eight target frameworks (net10.0, net9.0, net8.0, net48, net472, net471, net47, net462).
Trade-off
This deliberately reinstates the manifest/payload arity mismatch that #4729 set out to remove, so in-process EventListener and TraceEvent consumers still cannot decode these events. That capability and the agent's raw-blob contract are mutually exclusive on event ID 100. If .NET-side subscription is still wanted, it needs a separate event ID rather than a change to the existing one.
Merge requirement checklist
- CONTRIBUTING guidelines followed (license requirements, nullable enabled, static analysis, etc.)
- Unit tests added/updated
- Appropriate
CHANGELOG.mdfiles updated for non-trivial changes - Changes in public API reviewed (if applicable) - no public API change
Addendum: corroboration from consumer-side diagnostics
Added after merge. Rejection diagnostics from the receiving agent confirm the mechanism above and rule out the alternatives.
The agent's rejection reports the msgpack type it decoded for the top-level object, and that type differs from record to record, including between records rejected in the same instant. A constant fault (wrong encoding, wrong table, truncation) would report the same type every time. A type that varies per record means the parser is reading a byte that varies per record.
That follows directly from the framing. Both exporters cap the buffer at BUFFER_SIZE = 65360, so the length always fits in two bytes and the prefix is always:
[size & 0xFF][(size >> 8) & 0xFF][0x00][0x00][0x93]...
Byte 0, the only byte the parser inspects before failing, is the low byte of that record's serialized length. Mapping it onto msgpack type codes:
| First byte range | Decoded as | Share of sizes |
|---|---|---|
0x00-0x7F | positive integer | ~50% |
0xA0-0xBF, 0xD9-0xDB | string | ~13% |
0xE0-0xFF | negative integer | ~12.5% |
0x80-0x8F, 0xDE, 0xDF | map | ~7% |
0xC4-0xC6 | binary | ~1% |
0x90-0x9F, 0xDC, 0xDD | array | ~7% |
Observed rejections span several of these categories, consistent with the distribution above.
Notably, array is never among the rejected types. Those ~7% of records do not fail the array check, so the parser proceeds into the remaining bytes as though they were a valid frame. This is the more damaging branch: for a first byte of 0xDC or 0xDD the parser reads an array16/array32 header and takes its element count from the following bytes, which are [(size >> 8) & 0xFF][0x00][0x00][0x93]. For a ~22 KB record that decodes to an element count in the billions, so a single malformed record can drive an unbounded allocation in the consumer. At 0xDD alone that is roughly 1 record in 256.
Two consequences worth recording:
- The corruption is not intermittent.
SendEventis unconditional, so on an affected build every record is malformed. Partial impact observed in the field comes from configuration and version splits, not from sampling:EtwSession=...combined withPrivatePreviewEnableTraceLoggingDynamic=trueresolves toTransportProtocol.EtwTldand never reachesEtwDataTransport, and metrics, Linux transports, and services still on1.16.0or earlier are untouched. - Silent drops and consumer memory growth share this one root cause, separated only by which branch the first byte happens to select.