chore: NATS broker for micro services

Draft
SA
7d ago
+2189-56

https://rocketchat.atlassian.net/browse/RDI-278

Proposed changes (including videos or screenshots)

This is a proof of concept. It adds NatsBroker, a second network broker for the microservices deployment. It talks to NATS directly instead of going through Moleculer. It uses the NATS services protocol for calls and discovery, and plain subjects for events. Moleculer stays the default. BROKER=nats selects the new broker in every service and in the monolith running in microservices mode.

Broker selection

  • startBroker() (ee/packages/network-broker/src/startBroker.ts) picks MoleculerBroker or NatsBroker from BROKER. The NATS factory (nats.ts) reads NATS_URL, falling back to TRANSPORTER, plus REQUEST_TIMEOUT and BROKER_LOCAL_ROUTING.
  • Api.setBroker() now throws once the API has started, so a broker can no longer be swapped under running services.

What NatsBroker implements

  • Calls: rpc.<service>.<method>, load balanced by a NATS queue group. node.<nodeID>.<service>.<method> pins a call to one instance and backs CallingOptions.nodeID. A service in the same process is called directly, so arguments keep their identity instead of going through EJSON. BROKER_LOCAL_ROUTING=false turns that off.
  • Events:
    • broadcast publishes on event.<name> and reaches every node.
    • emitToOne publishes on emit.<name>, where each service subscribes in a queue group named after itself, so one instance of each listening service receives it.
    • broadcastLocal stays in process.
    • Events and methods live under different prefixes, because some event names collide with <service>.<method>, e.g. accounts.login.
  • Errors: a throwing handler answers with a service error, and the caller rethrows it. MeteorError keeps its error, reason, details and isClientSafe.
  • Discovery: $node.list and $node.services are answered from a $SRV.PING round trip, with a short cache. Each service carries its node id as metadata. License enforcement uses that list, the same way it does under Moleculer.
  • Startup:
    • A service is registered only once its dependencies (settings and license by default) are reachable locally or remotely.
    • Services whose dependencies are met together are registered together, before their created()/started() hooks run.
    • A failing hook is logged instead of rejecting api.start().
  • Retries: a request that reaches no responder (503) is retried with backoff for about 4s, which covers peers that are still booting or being rolled. Nothing else is retried, because a timed-out request may already have run.
  • Streams: a Readable in a call's arguments or its result is replaced by an inbox subject. The other side pulls 256KB chunks from that subject, which provides backpressure. An idle timeout cleans up the subject when the consumer never pulls.
  • Context: hooks, remote handlers and local calls all run inside asyncLocalStorage, so ServiceClass.context is populated. Under NATS, ddp-streamer's login handler depends on this.
  • Shutdown: on SIGTERM/SIGINT every service runs stopped(), then the connection drains and the process exits. SKIP_PROCESS_EVENT_REGISTRATION=true disables this.

Shared with MoleculerBroker

  • The default-dependencies rule now lives in withDefaultDependencies.ts, used by both brokers.
  • startBroker from moleculer.ts is renamed to startMoleculerBroker.

Tests and docs

  • NatsBroker.test.ts runs the broker against an in-memory FakeNatsConnection. It covers calls, events, emitToOne, errors, retries, dependency waiting, lifecycle failures, streams, context and shutdown. nats.test.ts covers the signal handling.
  • docs/service-brokers.md describes all three brokers, everything that had to be rebuilt for NATS, how events reach other instances in each deployment, what is not implemented yet, and the configuration.

Not done yet

[!WARNING]
Tracing, metrics and namespaces are not implemented. Switching to BROKER=nats silently loses all three. See "Not implemented yet" in docs/service-brokers.md.

  • Tracing: the caller's trace context is not sent with a call, and handlers do not run in a tracer span, so a trace stops at the first remote call.
  • Metrics: broker.metrics is undefined. ddp-streamer's rocketchat_subscription, users_connected and users_logged are not recorded, and no Prometheus reporter is exported (no MS_METRICS equivalent).
  • Namespaces: there is no MS_NAMESPACE equivalent. Subjects and service names are global to the NATS server, so two deployments on one server would answer each other's calls, receive each other's events, and discover each other's services. Until this lands, give each deployment its own NATS account.

Also missing:

  • broadcastToServices is a no-op.
  • $node.connected / $node.disconnected are never emitted. Presence relies on onNodeDisconnected to drop the sessions of a node that died.
  • $services.changed and the other broker lifecycle events are never emitted. Nothing in the codebase listens to them today.

Issue(s)

Steps to test or reproduce

  • Unit tests: yarn testunit in ee/packages/network-broker.
  • End to end: the CI e2e workflow has a broker input that is passed to docker compose as BROKER. The workflow checks that the requested broker actually started. The input defaults to nats on this branch so CI exercises the new broker. That default has to go before merge.
  • Locally: start the microservices compose with BROKER=nats.

Further comments

The nats dependency was bumped to ^2.29.1 in every app so the repo keeps a single version.

šŸ¤– Generated with Claude Code