redis: add bitmap, HLL, geo, scripting, server, and stream commands

Merged
RO
3mo ago
+1417-1

Problem

  • Bun.redis / RedisClient has no first-class methods for several command families that ship in both Redis 7+ and Valkey: bitmap ops beyond bitcount/getbit/setbit, HyperLogLog beyond pfadd, scripting beyond script, and nothing at all for geo, streams, or server introspection. Users have to drop to .send("XADD", [...]) and lose types and autocomplete.
  • #21404 asks specifically for the scripting subset (EVAL family).

Fix

  • Adds 45 methods, each a one-line instantiation of an existing command macro in src/runtime/valkey_jsc/js_valkey_functions.rs, registered in valkey.classes.ts:

    CategoryCommands
    Bitmapbitop, bitpos, bitfield
    HyperLogLogpfcount, pfmerge
    Geogeoadd, geodist, geohash, geopos, geosearch, geosearchstore
    Scriptingeval, evalsha, fcall, function
    Serverdbsize, flushdb, flushall, info, time, echo, lastsave, client, config, debug, command
    Genericobject, sort, wait, lcs
    Streamsxadd, xlen, xrange, xrevrange, xread, xreadgroup, xdel, xtrim, xack, xclaim, xautoclaim, xpending, xinfo, xgroup, xsetid

  • The macros themselves are unchanged, and every variadic command here uses cmd_strings_varargs! (the fixed-arity ones use cmd_noargs!, cmd_key! or cmd_key_value!). cmd_key_varargs! is deliberately not used: it skips null/undefined after the key, which would let xdel(key, undefined, id) send a shifted argument list. So an explicit null or undefined anywhere in a new command's arguments throws client-side and nothing is sent (flushdb(undefined) throws, like getex(key, undefined) does today); the only cost is that a missing first argument is reported by the server rather than as a client-side "key" error. The Rust diff against main is only the added invocations.

  • client is bound as js_client in Rust because the bare name collides with the client field; the JS-visible name is unchanged. debug and command use their plain names.

  • packages/bun-types/redis.d.ts: declarations with JSDoc for every method (plus the previously undeclared script()). Where the protocol has a fixed minimum arity it is encoded in the signature (pfcount(key, ...more), xdel(key, id, ...more), xack(key, group, id, ...more), xclaim(.., id, ...rest), xtrim(key, strategy, threshold, ..), and a separate unary overload for bitop("NOT", ..); the multi-key overload also lists the Redis 8.2 DIFF/DIFF1/ANDOR/ONE operations). sort() resolves (string | null)[] | number, since a GET pattern yields null for missing keys. Optional trailing arguments (flushdb/flushall mode, geodist unit) are declared as a zero-argument overload plus a required-argument overload rather than ? parameters, because passing an explicit undefined throws at runtime. Commands whose required arguments sit behind optional prefix modifiers (geoadd, xadd, xread, xreadgroup, geosearch) stay variadic like the existing zadd/scan declarations. The redis.d.ts vs valkey.classes.ts lint from #39271 listed script() as pending on this PR; that entry is removed here and the lint passes.

  • Verification, mostly in test/js/valkey/valkey.test.ts:

    • Docker-backed suite: one describe per command family above, exercising every command (runs on TCP and TLS), including SORT ... GET on missing keys.
    • Argument validation: mset(k, undefined, v), copy(src, undefined, dst), flushdb(undefined), bitop(.., undefined, ..) and xdel(stream, undefined, id) throw, and MGET/GET/XLEN afterwards show nothing reached the server; the no-server RedisClient argument validation block has the same calls plus null variants, holes after the key (bitpos, xdel, pfcount, sort) and psubscribe/punsubscribe.
    • PUB/SUB: punsubscribe(undefined) / psubscribe(undefined) / unsubscribe(undefined) throw and leave the existing subscriptions in place (observed through publish() counts), punsubscribe(pattern) removes only that pattern, and punsubscribe() removes all of them.
    • No-server prototype checks: every new method exists on the prototype (these fail on a build without this change).
    • test/integration/bun-types/fixture/redis.ts calls every new declaration and asserts its return type, with @ts-expect-error negatives for the bitop("NOT", ..) arity, unknown bitop operations, bad flushdb/flushall modes, a bad geodist unit, explicit undefined for those three, a bad xtrim strategy, and missing required arguments; bun-types.test.ts and the redis-client-types source lint pass.
    • Return types were checked against a live RESP3 server (geopos returns numbers, geodist returns a string).

Background

  • Command macros. Every RedisClient method is generated by one of a handful of macro_rules! in js_valkey_functions.rs (cmd_noargs!, cmd_key!, cmd_key_varargs!, cmd_key_value!, cmd_strings_varargs!, ...). Each expands to a #[bun_jsc::host_fn(method)] that converts the JS arguments with from_js (strings, buffers, and numbers-as-strings) and hands them to send_cmd, which queues the RESP command. Adding a command is choosing the macro whose argument shape matches the Redis syntax.
  • .classes.ts. generate-classes.ts reads valkey.classes.ts and emits the C++ prototype table; the fn: name is snake-cased to find the Rust method, which is why the Rust side uses js_client for the JS method client.
  • Test layout. test/js/valkey/valkey.test.ts is the main suite: a docker-gated block with one describe per command family, followed by a few top-level describes that need no server. The new tests follow that split.
History

Originally implemented against the Zig runtime; re-ported to the Rust macros after #30412 and rebased onto current main as a single commit. An earlier standalone valkey.commands.test.ts that ran against whatever was listening on localhost:6379 was removed in favour of the layout above. Earlier revisions changed cmd_strings_varargs! to treat an explicit undefined as an omitted argument (first for every command, then with a psubscribe/punsubscribe opt-out after review noted a bare PUNSUBSCRIBE drops every pattern). Review then pointed out that skipping an interior undefined shifts the arguments after it for the ~55 existing commands on the macro (mset("a", undefined, "b") would send MSET a b), so the macro change was dropped entirely; the PUB/SUB and argument-validation tests written along the way remain as coverage. A second punsubscribe on a pattern-only client trips a pre-existing debug assertion in on_valkey_unsubscribe (fixed separately in #37340), so each test here issues one PUNSUBSCRIBE per client.

Fixes #21404

[review] gate passed · iteration 26 · 6 files touched

fails on main (without fix) 
ASAN without fix: 47 failed, 992 skipped
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/mechgate.xml" test/internal/source-lints/redis-client-types.test.ts test/js/valkey/valkey.test.ts
bun test v1.4.0 (8326d1bd3)

test/js/valkey/valkey.test.ts:
failed to connect to the docker API at unix:///var/run/docker.sock; check if the path is correct and if the daemon is running: dial unix /var/run/docker.sock: connect: no such file or directory
Redis is not enabled, skipping tests
(skip) Valkey Redis Client (tls) > (unnamed)
(skip) Valkey Redis Client (tls) > Basic Operations > should keep process alive when connecting
(skip) Valkey Redis Client (tls) > Basic Operations > should set and get strings
(skip) Valkey Redis Client (tls) > Basic Operations > should test key existence
(skip) Valkey Redis Client (tls) > Basic Operations > should increment and decrement counters
(skip) Valkey Redis Client (tls) > Basic Operations > should increment by specified amount with INCRBY
(skip) Valkey Redis Client (tls) > Basic Operations > should increment by float amount with INCRBYFLOAT
(skip) Valkey Redis Client (tls) > Basic Operations > should decrement by
... (truncated)

release without fix: 2 failed, 992 skipped
bun test v1.4.0-canary.1 (a1a7f7aa7)

test/js/valkey/valkey.test.ts:
failed to connect to the docker API at unix:///var/run/docker.sock; check if the path is correct and if the daemon is running: dial unix /var/run/docker.sock: connect: no such file or directory
Redis is not enabled, skipping tests
(skip) Valkey Redis Client (tls) > (unnamed)
(skip) Valkey Redis Client (tls) > Basic Operations > should keep process alive when connecting
(skip) Valkey Redis Client (tls) > Basic Operations > should set and get strings
(skip) Valkey Redis Client (tls) > Basic Operations > should test key existence
(skip) Valkey Redis Client (tls) > Basic Operations > should increment and decrement counters
(skip) Valkey Redis Client (tls) > Basic Operations > should increment by specified amount with INCRBY
(skip) Valkey Redis Client (tls) > Basic Operations > should increment by float amount with INCRBYFLOAT
(skip) Valkey Redis Client (tls) > Basic Operations > should decrement by specified amount with DECRBY
(skip) Valkey Redis Client (tls) > Basic Operations > should rename a key with RENAME
(skip) Valkey Redis Client (tls) > Basic Operations > should rename a key with RENAME overwr
... (truncated)
passes on PR (with fix) 
ASAN with fix: 992 skipped
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/mechgate.xml" test/internal/source-lints/redis-client-types.test.ts test/js/valkey/valkey.test.ts
bun test v1.4.0 (8326d1bd3)

test/js/valkey/valkey.test.ts:
failed to connect to the docker API at unix:///var/run/docker.sock; check if the path is correct and if the daemon is running: dial unix /var/run/docker.sock: connect: no such file or directory
Redis is not enabled, skipping tests
(skip) Valkey Redis Client (tls) > (unnamed)
(skip) Valkey Redis Client (tls) > Basic Operations > should keep process alive when connecting
(skip) Valkey Redis Client (tls) > Basic Operations > should set and get strings
(skip) Valkey Redis Client (tls) > Basic Operations > should test key existence
(skip) Valkey Redis Client (tls) > Basic Operations > should increment and decrement counters
(skip) Valkey Redis Client (tls) > Basic Operations > should increment by specified amount with INCRBY
(skip) Valkey Redis Client (tls) > Basic Operations > should increment by float amount with INCRBYFLOAT
(skip) Valkey Redis Client (tls) > Basic Operations > should decrement by
... (truncated)

release with fix: 992 skipped
$ bun scripts/build.ts --profile=release
[configured] bun-profile → bun (stripped) in 670ms (unchanged)
ninja: Entering directory `/workspace/bun/build/release'
[1/24] gen generated_host_exports.rs
generated_host_exports.rs: 92 exports (host=3, lazy=10, generic=79, rust=0); 241 extern-C blocks audited
[2/24] gen ZigGeneratedClasses.{cpp,h,rs}
Found 2 classes from /workspace/bun/src/jsc/resolve_message.classes.ts
  - ResolveMessage (15 fields)
  - BuildMessage (10 fields)
Found 1 classes from /workspace/bun/src/runtime/api/Archive.classes.ts
  - Archive (4 fields, 1 class fields)
Found 2 classes from /workspace/bun/src/runtime/api/BunObject.classes.ts
  - ResourceUsage (8 fields)
  - Subprocess (20 fields)
Found 1 classes from /workspace/bun/src/runtime/api/cron.classes.ts
  - CronJob (5 fields)
Found 3 classes from /workspace/bun/src/runtime/api/filesystem_router.classes.ts
  - FileSystemRouter (5 fields)
  - FrameworkFileSystemRouter (2 fields)
  - MatchedRoute (8 fields)
Found 1 classes from /workspace/bun/src/runtime/api/Glob.classes.ts
  - Glob (5 fields)
Found 1 classes from /workspace/bun/src/runtime/api/h2.classes.ts
  - H2FrameParser (32 fields)
Found 9 cl
... (truncated)
diff hotspot 
packages/bun-types/redis.d.ts                      | 652 +++++++++++++++++++++
 src/runtime/valkey_jsc/js_valkey_functions.rs      |  71 +++
 src/runtime/valkey_jsc/valkey.classes.ts           |  52 ++
 test/integration/bun-types/fixture/redis.ts        | 123 ++++
 .../source-lints/redis-client-types.test.ts        |   1 -
 test/js/valkey/valkey.test.ts                      | 519 ++++++++++++++++
 6 files changed, 1417 insertions(+), 1 deletion(-)

gate history · 6 passed · 0 rejected · iteration 26

evidence per changed file 
file                                                   reads  edits  tests
packages/bun-types/redis.d.ts                             16     21     22
src/runtime/valkey_jsc/js_valkey_functions.rs             11     20     15
src/runtime/valkey_jsc/valkey.classes.ts                   1      1     15
test/integration/bun-types/fixture/redis.ts                3      3     15
test/internal/source-lints/redis-client-types.test.ts      1      2      0
test/js/valkey/valkey.test.ts                             13     17     15