fix(lookup): date every proposable release and admit prereleases only on purpose
Merged
SW
1d ago+1015+1015+0123456789012345678901234567890123456789-234-234-012345678901234567890123456789
Fixes from the lookup and datasource scan. Each one was letting through a release that policy should have held back.
Invariant
Every version the lookup could propose is either dated and flagged by its source, or marked unchecked. A positive waitHours holds an unchecked release instead of treating it as undated. It shows up in the report as a notice, and a later lookup asks about it again. A missing answer can no longer skip the release-age wait.
Fixes
- Release dates cover every version, not the newest N (Go, Maven, NuGet).
- New
Version::unchecked(catalog.rs) and arelease-datesstate (dates.rs). The state keeps each registry answer per version: a date, or "cannot be served". It is keyed by package and, for Maven, by the operator registry entry's scope. - Go
.infoand Maven POMHEADrequests go only to versions with no stored answer, wherever they sort, with a boundedbuffer_unorderedfan-out of 8. - A cold lookup dates every version once. After that, a lookup asks only about versions it hasn't seen. This makes
guide/sources.md's ".info for new versions only" true. - A failed request stores nothing and leaves the version unchecked. This replaces Maven's fail-open
.ok()?. - A 404 from the Go proxy or a missing Maven POM drops the version.
- The old top-N cut (
GO_DATED_RELEASES, Maven'sDATED) is gone, and so arereached_knownand the head-stoppingmerge(). - NuGet reads every registration page (
nuget-registrationv2 lists them all), up to 4 at once.- The flat-container listing and the registration entries are matched without build metadata.
- A version the registration doesn't describe yet is unchecked.
- A page link off nuget.org fails the lookup instead of being ignored.
- I didn't plumb each repository's baseline into the shared catalog fetch. Catalogs are shared across repositories and reused for up to 23 h, so a floor recorded at fetch time could be stale for another repository's lower baseline, which fails open. Dating everything once and storing the answers is simpler, and it fails closed by construction.
- New
- A
permittedfilter admits prereleases only when it asks for them (lookup/src/lib.rs,config/src/matching.rs).Filter::asks_for_prereleaseimplements three rules:- A non-negated
regex:asks when it matches the prerelease but not the same version without its prerelease part (regex:-rc\.andregex:nightlyask;regex:^2\.does not). - A
semver:range asks when it names a prerelease version itself and matches (semver:>=3.0.0-rc.1andsemver:[1.0-beta1,2)ask;semver:[1,2)andsemver:<3do not). - A negated filter never asks.
- A non-negated
- Documented in
guide/lookup-policy.mdandconfiguration.md. - The existing
nightly-pinsexample (regex:-nightly\.) keeps working.
- mise prerelease flag.
mise()now carriesprerelease: trueintoVersion::prerelease, and the lookup treats it like a prerelease version, including for the current version,latest, and dist-tag targets. This matchesgithub-releases. - mise-versions tag dates. When the mirror's
tags.jsonhas a tag thatreleases.jsondoesn't date, thegithub-tagslookup falls back to GitHub's own tag catalog. That path dates annotated tags by their tagger and is cached byTagCatalog/TagObject.guide/lookup-policy.mdalready promised this;guide/sources.mdnow says how. - Go
.infofetches run in parallel with a bounded fan-out, as part of fix 1.
Cache and version bookkeeping:
catalog.rsandconfig/srcare hashed sources, so every derived representation and state that lists them got a version bump, and its row intests/derived-versions.tsv/tests/state-versions.tsvwas updated.release-datesis intests/state-exempt.tsv, since it holds facts about immutable releases.internals/cache.mddescribes the new state.
Tests
datasource/tests/go_proxy.rs:- every version is dated once, and a warm lookup makes no
.inforequests; - a new v0.29.4 below a known v0.30.0 is dated;
- a failed
.infoleaves the version unchecked and is retried.
- every version is dated once, and a warm lookup makes no
datasource/tests/sources.rs:- NuGet reads an older linked page holding an unlisted
6.0.5+build(yanked, dated); - an undescribed version is unchecked;
- an off-site page fails the lookup;
- Maven dates all 13 versions, retries a 503, dates new older-line releases on the next lookup and drops a 404 POM.
- NuGet reads an older linked page holding an unlisted
datasource/tests/versions_host.rs: an annotated tag the mirror leaves undated gets its tagger date through GitHub.datasource/tests/mise_lookup.rs:prereleases_stay_in_the_catalog_for_the_policy_to_decidenow asserts that the flag is carried (jq1.7rc2, prettier alphas), and that it is absent on stable releases.lookup/tests/policy.rs:- npm
!regex:^3\.,regex:^2\.and!semver:>=3admit no rc;regex:-rc\.andsemver:>=2.2.0-rc.1do; - Maven
semver:[1,2)admits no beta;semver:[1.5-beta1,2)andregex:-betado; - PyPI
semver:<3admits no rc;semver:>=2.2.0rc1,<3does; - a flagged stable-looking prerelease is held, unless the current version is flagged;
- an unchecked release waits and is reported, and is proposed with
waitHours = 0.
- npm
- Ran on the VM:
cargo test -p patchula-config -p patchula-datasource -p patchula-lookup -p patchula-engine, and clippy-D warningson those crates pluspatchula-tools. - The corpus snapshots are unchanged: the corpus test passes without regenerating them.
Dropped
Nothing.
Follow-ups
- A cold lookup of a Go module or Maven artifact with hundreds of versions now makes one date request per version, once. Watch cold-run timings on large monorepos.
- Rows the old Go code wrote to the
releasestable (source = 'go') are no longer read. A later cache migration can drop them. I didn't add one here, to avoid colliding with other open cache migrations. - Repositories in mise's registry whose tags are mostly lightweight with no releases now always take the GitHub API path for
github-tags.