feat: [AI-8448] count installs from the shell installers, not just npm

Merged
SA
3mo ago
+918-24

Fixes AI-8448.

The dip was measurement, not installs

first_launch is the only install metric. It fires off a marker file rather than any network call from the installer — and that marker was written in exactly one place, packages/opencode/script/postinstall.mjs. Neither install nor install.ps1 wrote it, so once the advertised path moved from npm to altimate.sh/install, those installs stopped being counted. Nothing changed about how many people were installing.

What this does

Both shell installers now write the same marker postinstall.mjs writes, and first_launch carries a new install_method (curl | powershell | npm | unknown) so the recovered volume is separable from npm instead of folded into one number. altimate upgrade on the curl path re-runs install, so curl upgrades become visible too.

Brand-new installs stay is_upgrade: false — that field probes whether ~/.altimate/machine-id existed before this launch:

count(distinct machine_id) where type = "first_launch" and is_upgrade = false

Expect install_method: "unknown" for the first upgrade after this ships — those markers predate the field.

Details that fail silently rather than loudly

  • Marker path. $XDG_DATA_HOME, default ~/.local/share/altimate-code, on every platform including Windows. welcome.ts resolves the data dir through Node's os.homedir() and never consults %LOCALAPPDATA%; a marker written there would be ignored at read time.
  • -Encoding ascii in install.ps1. The documented entrypoint is powershell -c "irm ... | iex" — Windows PowerShell 5.1, where -Encoding utf8 prepends a BOM. .trim() does strip a leading BOM (U+FEFF is JS whitespace), so this was latent rather than broken, but install_method is matched against a fixed allowlist and shouldn't depend on that.
  • unknown version fallback. An empty marker is deleted unread, so an unresolved version would lose the install outright. That's the state check_version leaves whenever the GitHub API is unreachable.
  • Write happens after the install dispatch. A version already present (check_version exits 0 early) reports no install, and neither does a failed download.
  • Allowlisted install_method. A hand-edited or truncated source file reads unknown rather than minting a new dimension.
  • Source file is consumed on read, including on the empty-marker path, so an orphan can't be attributed to a later install.
  • Non-fatal in both installers. A read-only $HOME costs the event, never the install.

Privacy

No new network call and no new identifier. The installers record a version and their own name to a local file; the CLI's existing opt-out gates (ALTIMATE_TELEMETRY_DISABLED, OPENCODE_DISABLE_TELEMETRY, telemetry.disabled) still decide whether anything is transmitted. docs/docs/reference/security-faq.md and docs/docs/reference/telemetry.md are updated to say so.

Tests

Followed the touchpoint set from #1064 (event union → docs → emitter → unit tests → install-script assertions).

  • test/cli/welcome.test.ts — is_upgrade both ways, install_method attribution, allowlist rejection, source-file consumption, empty-marker path.
  • test/install/install-telemetry.test.ts — marker path/fallback/ordering/non-fatality for both installers, no-BOM, plus the ordering invariant below.
  • test/install/postinstall.test.ts — npm writes .install-source.

The load-bearing test is the ordering invariant. is_upgrade is only correct because src/index.ts fires Telemetry.init() unawaited and doInit() yields at await Config.get() before minting the machine-id, so the synchronous banner call on the next line still sees pre-launch state. An await added ahead of that mint would make every install report is_upgrade: true and silently empty the brand-new-install metric without a single existing test failing. The test asserts the machine-id is absent at that instant and present once the promise resolves, so it can't pass vacuously.

598 pass / 0 fail across test/cli/welcome.test.ts test/install/ test/telemetry/telemetry.test.ts test/branding/; typecheck clean.

Verification

install's marker writer was executed directly: XDG override honored, v prefix stripped, unknown fallback, exit 0 on a read-only $HOME. install.ps1 is asserted at source level only — no pwsh on the dev machine, so its runtime behavior rides on CI.

Not in scope

Installs that never launch the CLI remain uncounted, so download→launch conversion is still unmeasurable. That needs a beacon from the install script itself — a new event plus opt-out handling in bash — and is deliberately deferred.

Two pre-existing things noticed but left alone:

  1. welcome.ts:70 returns before printing the welcome box when isUpgrade is false, so the box only ever shows on upgrades. Plausibly intentional (the TUI has its own first-run flow), but it reads backwards for a "welcome" banner.
  2. test/altimate/review/telemetry.test.ts redirects $HOME to keep the suite from minting a machine-id in the developer's real home — but Bun resolves os.homedir() at startup and ignores later process.env.HOME mutation, so that protection doesn't currently work. This PR's tests use spyOn(os, "homedir"), the convention already used in test/mcp/discover.test.ts.

🤖 Generated with Claude Code


Summary by cubic

Counts installs from the shell installers and the VS Code extension installer, attributing each install's source in telemetry. Previously only npm postinstall.mjs wrote the install marker, so installs via altimate.sh/install, install.ps1, and the VS Code extension went uncounted; now all installers write the same marker and first_launch carries install_method (curl/powershell/npm/vscode-extension/local, or unknown). Closes AI-8448.

  • Both shell installers and postinstall.mjs write .install-source before .installed-version and publish the version atomically (temp+rename), so a CLI reading mid-write can't misattribute or drop the install; install --binary reports "local" and both scripts fall back to "unknown" for an unresolvable version.
  • install.ps1 now computes its data dir inside the try: under $ErrorActionPreference = "Stop" a null USERPROFILE or bad PSDrive previously aborted the installer after the binary was placed but before the PATH write. The marker block became a Write-InstallMarker function so Pester can AST-extract and execute it, with the suite pinned to Stop semantics so the try/catch tests can't silently pass.
  • welcome.ts reads and consumes .install-source (cleared in finally, and recursive so a directory-shaped file can't pin attribution to unknown), allowlists the five methods, and adds install_method to the first_launch event.
  • src/index.ts runs the welcome banner before Telemetry.init() so is_upgrade is structurally correct instead of relying on init's async timing.
  • Docs note the installers send no telemetry themselves and that the telemetry.disabled config key can be bypassed when telemetry startup runs before config resolves — env vars remain the guaranteed opt-out.
  • Rollout: first upgrade after this ships may report install_method: "unknown" for pre-existing markers. Release the CLI before the VS Code extension starts writing markers so the reader exists first.

Summary by CodeRabbit

  • New Features

    • First-launch telemetry now records installation methods including curl, PowerShell, npm, VS Code extension, local binary, or unknown.
    • Installers record version and source information locally for later telemetry transmission.
    • Telemetry respects opt-out settings, and installers do not transmit telemetry directly.
  • Documentation

    • Clarified upgrade detection, delivery limits, installer sources, and fallback behavior.
  • Bug Fixes

    • Improved handling of missing, invalid, or unwritable installation markers.
    • Marker updates are published safely to prevent incomplete telemetry records.

Review fixes (human + 3 bots)

One real bug. install.ps1 computed $dataRoot/$dataDir above the try. With $ErrorActionPreference = "Stop" and Join-Path's provider-qualified path resolution, a null $env:USERPROFILE or an XDG_DATA_HOME naming a bad PSDrive raised a terminating error there — aborting the installer after the binary was placed but before the PATH registry write. Result: installed binary, not on PATH, directly contradicting the block's own "non-fatal" comment. Now everything is inside the try, using [IO.Path]::Combine to keep PSDrive resolution out of it. The old test passed because it only checked that } catch { appeared somewhere; it now pins that the assignments come after try {.

One false claim of mine, corrected. The test header said install.ps1's runtime behaviour "is exercised by the Windows Installer (Pester) CI job". It is not — that job's subprocess tests deliberately stop the installer via -Help/unknown -Version so nothing downloads, and never reach the marker block. It also runs under pwsh, never powershell.exe, so Windows PowerShell 5.1 — the documented entrypoint and the entire reason for -Encoding ascii — is still unexercised anywhere. I wrote that claim from the job name passing, without reading the Pester suite.

To fix the underlying gap, the block is now a Write-InstallMarker function so Pester can AST-extract and execute it the way it already does Test-Checksum. Five new cases run it against a temp profile: byte-exact contents with no BOM, v-strip and unknown fallback, USERPROFILE fallback, no throw on empty USERPROFILE, no throw when the data dir can't be created. That last pair is the coverage that would have caught the bug above.

Other fixes:

  • Companion before trigger, all three writers. .installed-version is the reader's trigger; writing it first let a CLI starting in between report unknown, and since writes truncate first, a reader could observe an empty version file and delete it unread — losing the install, not just its attribution.
  • --binary is attributed local, not curl (that branch sets specific_version="local", so it misreported both source and version). write_install_marker now takes the method as $1 — which also removed an unbound $marker_source I'd introduced mid-edit, a set -u abort waiting to happen.
  • Test isolation: the ordering test now snapshots/clears/restores OPENCODE_DISABLE_TELEMETRY too — doInit() returns before minting if either gate is set.
  • Docs: the security FAQ no longer asserts unconditionally that the opt-out decides transmission. Env vars are always honoured; the telemetry.disabled config key can be bypassed when telemetry startup runs before config is resolvable, so that caveat is now stated with a pointer to use an env var for a guarantee.

Still not fixed: the config-key opt-out itself

Flagged by CodeRabbit, cubic, and the human reviewer, and it stays open deliberately. The gate is shared by every event emitted from CLI middleware — this PR raises how often it's hit (~30x volume), it doesn't introduce it. Both available in-PR routes are wrong: failing closed emits nothing at all on the middleware path (killing the feature), and reading config here means duplicating the merge + JSONC semantics of config/config.ts. The fix belongs in telemetry init — make Config resolvable there, or adopt a module-wide fail-closed policy. Tracked in the expanded FIXME; needs its own ticket.

Release order

vscode-extension has no producer in this repo — it ships in AltimateAI/vscode-altimate-mcp-server#453. Release this CLI first, so the reader exists before the extension starts writing .install-source; otherwise that file is orphaned by pre-#1096 readers and can be misattributed later. A zero vscode-extension share after release means the extension hasn't rolled out yet, not that there are no extension installs — now noted in telemetry.md.

Verification: typecheck clean, 1383 pass / 0 fail across test/cli test/install test/telemetry test/branding. The install.ps1 Pester additions are unverified locally (no pwsh on this machine) and rely on the CI job.


Round 3 (b69530bf0f) — both blocking items, plus four found pre-submit

Blocking, from review:

  • postinstall.mjs write order — fixed. The previous commit message claimed "all three writers"; only two had been changed.
  • Pester test that could not fail — fixed. $ErrorActionPreference = "Stop" in BeforeAll, a no-marker assertion proving the failure path was reached, and a third case asserting the preference itself so drift back to Continue fails visibly.

Found before submitting:

  • Atomic trigger publish in all four writers. Companion-first ordering closes the "attribution lost" window but not truncation: a plain write truncates before filling, so a reader mid-write can observe an empty .installed-version, which welcome.ts deletes unread — losing the install itself. Only the extension published atomically. Now temp+rename in install, install.ps1 and postinstall.mjs too.
  • clearInstallSource could not do what its comment claimed. The comment said a directory in place of .install-source is still cleared; fs.unlinkSync throws EPERM/EISDIR on a directory, so it would have survived every launch and pinned install_method to unknown permanently. Now rmSync with force+recursive, with a test that creates a directory-shaped file and asserts removal.
  • Pester header no longer described the file — read as subprocess-only after AST-extracted execution was added. Rewritten to document both layers.
  • Empty-USERPROFILE Pester case wrote into the git checkout. [IO.Path]::Combine("", ".local", "share") returns the relative path .local\share, so the marker landed under the Pester process's cwd (checkout root in CI) while AfterEach only cleaned the sandbox. Now Push-Location $script:Sandbox.
  • Two test-quality fixes: the bash "unwritable dir" case asserted only exit 0 (passed with the writer deleted — bash twin of the Pester finding), and executed tests leaked temp dirs on assertion failure.
  • Docs: unknown added to the FAQ's install_method list; telemetry.md now states both the boolean schema and the string KQL form of is_upgrade instead of one ambiguously.

Verified: typecheck clean; 1388 pass / 0 fail across test/cli test/install test/telemetry test/branding, two consecutive runs; bash marker writer executed directly for both install methods with no temp residue.

Not verified locally: the install.ps1 and Pester changes. No pwsh on the dev machine (cask needs interactive sudo), so the Windows Installer (Pester) job is their only check — and a passing job confirms the tests pass, not that they can fail. The mutation test (delete the try/catch, confirm the case fails) has not been performed.