fix: hit the volume cache on Azure and keep a volume's lease through its pull

Merged
RU
1d ago
+9-5

Summary

Two volume bugs, both present in 1.875 and still on main, that together make writable volumes on Azure Blob unusable for long jobs.

1. The per-worker volume cache never hits on Azure Blob. A pull trusted a cached file when its size matched and, if the listed ETag "looked like" an MD5, when that MD5 matched. etag_to_md5 only excluded multipart ETags (those containing -), so Azure's opaque ETags (0x8DC…) were taken as MD5s and compared with real content hashes. They never match, so every file was pulled from object storage again on every job, even back-to-back on the same worker. For a 6.7 GB / 180k-file volume that is ~15 minutes added to every job. The same applies to any store whose ETag isn't the content MD5 (e.g. S3 with SSE-KMS).

2. A writable volume's lease is lost whenever its pull takes over 60 seconds, and the job's writes are discarded. The lease is taken with a 60s TTL, but the 10s renewal task was only spawned after every volume had been pulled, and renewal refuses an already-expired lease. Any pull over 60s (always the case with bug 1 on a large volume) let the lease lapse for good, so sync-back ended with failed to sync back: Internal: the volume's lease was lost, not writing it back. The job still reports success, but nothing it wrote to the volume is persisted, and another worker could take the volume in the meantime.

Changes

  • FileEntry gains an optional e_tag: the object's ETag when this content was last read or written. It is serde(default, skip_serializing_if), so manifests and the agent-worker begin response stay compatible both ways.
  • etag_to_md5 only returns an ETag that is actually a 32-hex MD5.
  • EE (companion PR):
    • download_volume decides a cache hit by comparing the stored ETag with the listed one. ETags are normalized, because Azure lists them bare but returns them quoted from GET/PUT. It falls back to comparing MD5s only when the ETag is one, so an S3 cache written by an older worker still hits. On Azure, an old cache misses once and then hits.
    • A download records the ETag of the bytes it received. Sync-back records the ETag the upload returned, and keeps the pulled entry's ETag for unchanged files. This way a job's own writes also hit on the next pull.
    • Lease renewal (SQL workers and HTTP agent workers) starts before the pulls and renews each lease from the moment it is taken. On a setup error the renewal stops before the leases are released.

Test plan

  • Reproduced bug 1 against Azurite with the real download_volume/sync_volume_back: an unchanged volume pulled twice reported 0 from cache on both pulls. After the fix: the second pull hits on every file, a file the job rewrote hits on the next pull, and a same-size rewrite by another writer is downloaded fresh.
  • On a local instance with Azurite as the workspace's object and volume storage, ran bash jobs mounting # volume: agentvol data back to back. Before the fix every file was pulled each time; now the log says 51 total files (51 from cache, 0 pulled from object store).
  • Reproduced bug 2 on the same instance by holding the volume's cache lock for 90s so that setup stalls after taking the lease. Old code: lease_until counted down to -20s and the job ended with the exact error above. New code: lease_until stayed ~50s ahead throughout and sync-back succeeded.
  • cargo test -p windmill-worker-volumes --features private,enterprise, plus new unit test test_cache_hit_with_opaque_e_tags.
  • Not exercised: the HTTP agent-worker path (setup_volumes_http_worker). Its lease change mirrors the SQL path line for line; agent workers keep no local cache, so the ETag change doesn't affect them.

🤖 Generated with Claude Code


Summary by cubic

Fixes two bugs, both present since 1.875, that together made writable Azure Blob volumes unusable for long jobs. The per-worker volume cache never hit on Azure because its opaque ETags (0x8D…) were taken for content MD5s and never matched, so every file was re-pulled from object storage on every job. And a writable volume's lease was lost whenever its pull took over 60 seconds, so sync-back failed and the job's writes were silently discarded while the job still reported success.

  • FileEntry now records the object's ETag alongside that file's MD5, keeping manifests and the agent-worker begin response compatible both ways.
  • etag_to_md5 now only accepts real 32-hex MD5s, so Azure's and multipart S3 ETags are no longer compared against content hashes.
  • The linked EE change decides cache hits by comparing stored and listed ETags, falling back to MD5 when the ETag is a real one so caches written by older workers still hit.
  • Lease renewal now starts before the pulls and renews every lease from the moment it is taken, so a long pull no longer lets it lapse.