fix: hit the volume cache on Azure and keep a volume's lease through its pull
Summary
Two volume bugs, both present in 1.875 and still on main, that together make writable volumes on Azure Blob unusable for long jobs.
1. The per-worker volume cache never hits on Azure Blob. A pull trusted a cached file when its size matched and, if the listed ETag "looked like" an MD5, when that MD5 matched. etag_to_md5 only excluded multipart ETags (those containing -), so Azure's opaque ETags (0x8DC…) were taken as MD5s and compared with real content hashes. They never match, so every file was pulled from object storage again on every job, even back-to-back on the same worker. For a 6.7 GB / 180k-file volume that is ~15 minutes added to every job. The same applies to any store whose ETag isn't the content MD5 (e.g. S3 with SSE-KMS).
2. A writable volume's lease is lost whenever its pull takes over 60 seconds, and the job's writes are discarded. The lease is taken with a 60s TTL, but the 10s renewal task was only spawned after every volume had been pulled, and renewal refuses an already-expired lease. Any pull over 60s (always the case with bug 1 on a large volume) let the lease lapse for good, so sync-back ended with failed to sync back: Internal: the volume's lease was lost, not writing it back. The job still reports success, but nothing it wrote to the volume is persisted, and another worker could take the volume in the meantime.
Changes
FileEntrygains an optionale_tag: the object's ETag when this content was last read or written. It isserde(default, skip_serializing_if), so manifests and the agent-workerbeginresponse stay compatible both ways.etag_to_md5only returns an ETag that is actually a 32-hex MD5.- EE (companion PR):
download_volumedecides a cache hit by comparing the stored ETag with the listed one. ETags are normalized, because Azure lists them bare but returns them quoted from GET/PUT. It falls back to comparing MD5s only when the ETag is one, so an S3 cache written by an older worker still hits. On Azure, an old cache misses once and then hits.- A download records the ETag of the bytes it received. Sync-back records the ETag the upload returned, and keeps the pulled entry's ETag for unchanged files. This way a job's own writes also hit on the next pull.
- Lease renewal (SQL workers and HTTP agent workers) starts before the pulls and renews each lease from the moment it is taken. On a setup error the renewal stops before the leases are released.
Test plan
- Reproduced bug 1 against Azurite with the real
download_volume/sync_volume_back: an unchanged volume pulled twice reported0 from cacheon both pulls. After the fix: the second pull hits on every file, a file the job rewrote hits on the next pull, and a same-size rewrite by another writer is downloaded fresh. - On a local instance with Azurite as the workspace's object and volume storage, ran bash jobs mounting
# volume: agentvol databack to back. Before the fix every file was pulled each time; now the log says51 total files (51 from cache, 0 pulled from object store). - Reproduced bug 2 on the same instance by holding the volume's cache lock for 90s so that setup stalls after taking the lease. Old code:
lease_untilcounted down to -20s and the job ended with the exact error above. New code:lease_untilstayed ~50s ahead throughout and sync-back succeeded. -
cargo test -p windmill-worker-volumes --features private,enterprise, plus new unit testtest_cache_hit_with_opaque_e_tags. - Not exercised: the HTTP agent-worker path (
setup_volumes_http_worker). Its lease change mirrors the SQL path line for line; agent workers keep no local cache, so the ETag change doesn't affect them.
🤖 Generated with Claude Code
Summary by cubic
Fixes two bugs, both present since 1.875, that together made writable Azure Blob volumes unusable for long jobs. The per-worker volume cache never hit on Azure because its opaque ETags (0x8D…) were taken for content MD5s and never matched, so every file was re-pulled from object storage on every job. And a writable volume's lease was lost whenever its pull took over 60 seconds, so sync-back failed and the job's writes were silently discarded while the job still reported success.
FileEntrynow records the object's ETag alongside that file's MD5, keeping manifests and the agent-workerbeginresponse compatible both ways.etag_to_md5now only accepts real 32-hex MD5s, so Azure's and multipart S3 ETags are no longer compared against content hashes.- The linked EE change decides cache hits by comparing stored and listed ETags, falling back to MD5 when the ETag is a real one so caches written by older workers still hit.
- Lease renewal now starts before the pulls and renews every lease from the moment it is taken, so a long pull no longer lets it lapse.