fix(pages): stop Claude reaching for an artifact when the user asked for a page

Merged
HA
2mo ago
+399-11

Links

Summary

  • Adds a PreToolUse guard on Claude Code's Artifact tool that denies the first publish of a session inside a Superset terminal, pointing the agent at superset:page. The second call goes through untouched.
  • Teaches the three text surfaces we own (the page skill description and body, the pages_publish MCP tool description, the desktop page-agent prompt) to name artifacts explicitly, so the skill has a lexical reason to win.

Why / Context

From SUPER-2021: "Often i ask for a Superset page and it still generates a Claude artifact."

Artifact publishes a self-contained HTML document to a private URL, which is close enough to a page that the model picks it. It wins on prompt real estate: it is a one-step tool whose system-prompt description frames publishing as how a deliverable gets finished, while superset:page is a skill that must be invoked first. Before this PR nothing in the repo mentioned artifacts at all, so there was no signal pushing the other way.

The two are not interchangeable. A page is listed in the org, mints a version on every publish, and takes pinned comments that route back to an agent that can act on them. An artifact has none of that.

How It Works

The guard (packages/agent-setup/templates/artifact-guard.template.sh) stays completely silent unless all of:

ConditionWhy
SUPERSET_TERMINAL_ID/SUPERSET_TAB_ID setHook configs are global; this must be inert in non-Superset Claude sessions
tool_name is Artifact
action is a publishlist/comments/reply/read_db/write_db operate on an artifact that already exists
no url in tool_inputA url means updating an artifact the user already owns
this session not yet nudgedMarker file keyed by session_id, falling back to terminal id

Then it emits permissionDecision: "deny" with a reason naming superset:page and superset pages publish. SUPERSET_PAGES_NUDGE=off disables it outright.

Registration goes into ~/.claude/settings.json under PreToolUse with matcher Artifact, wrapped in the same SUPERSET_HOME_DIR guard the notify command uses.

Design Decisions

  • Why a separate script instead of another notify.sh event: the guard returns a permission decision on stdout. notify.sh is a fire-and-forget notifier whose only stdout is debug output, and mixing a decision into it would be fragile.
  • Why deny-once instead of a hard block: a hard block removes the choice from someone who genuinely wants a Claude artifact. Deny-once makes Pages the default while leaving a one-retry escape hatch that costs the agent nothing.
  • Why not text-only: a one-line skill description does not reliably out-rank a long imperative tool description in the system prompt. The wording helps; the hook is what actually changes the outcome.
  • Why cleanEntry had to change: it previously recognized managed entries by the notify script path alone. Without teaching it the guard path too, a re-merge would append a duplicate entry every boot and teardown would orphan it.

Manual QA Checklist

  • Provisioned into a temp SUPERSET_HOME_DIR, confirmed ~/.superset/hooks/artifact-guard.sh is written at mode 0755
  • Ran the exact registered command string against a real PreToolUse payload: first call → deny, second call → no output (tool proceeds)
  • PreToolUse entry merges as {"matcher":"Artifact", …} alongside the existing notify events
  • Re-merging over the hook's own output is idempotent (no duplicate Artifact entry)
  • Teardown removes the guard and leaves a user-authored PreToolUse hook in place
  • Guard is silent outside a Superset terminal, and with SUPERSET_PAGES_NUDGE in 0/off/false/no
  • Guard is silent for list/comments/reply/read_db/write_db and for a url-carrying update
  • Guard is silent for every other tool name
  • End-to-end in a real Superset terminal: ask an agent for a page, confirm it lands on Pages rather than an artifact

Testing

  • bun test in packages/agent-setup — 363 pass, 0 fail (10 new guard-script tests, 3 new registration tests)
  • bunx turbo typecheck --filter=@superset/desktop --filter=@superset/mcp --filter=@superset/agent-setup — clean
  • bunx biome check on the touched trees — clean
  • bun run check:plugins — passes
  • bun run check:i18n — no catalog churn (all added text is agent-facing, not interface copy)
  • bunx sherif — no issues

Known Limitations

  • Claude-only by construction. No other harness Superset launches has an Artifact tool, so Codex, Cursor, Droid et al. get the wording change and nothing else.
  • The plugin was not republished. plugins/superset/skills/page/SKILL.md changed without superset plugins publish superset --bump patch. There is no superset@* tag on origin, so check:plugins passes and the edited skill still ships via the bundled plugin dir — but if you want the marketplace release, that is a follow-up commit.
  • The guard's once-per-session marker lives in TMPDIR, so a machine that clears it mid-session gets one extra nudge. Harmless, and preferable to persisting state under ~/.superset.

Risks / Rollout

  • Risk: low, and scoped to Claude sessions launched from Superset terminals. Worst case is one denied tool call per session that the agent immediately retries.
  • Rollout: provisioning is declarative and runs at desktop/host boot; existing installs pick the hook up on next start.
  • Rollback: revert the commit. Next boot rewrites ~/.claude/settings.json without the PreToolUse entry, since teardown recognizes the guard command. SUPERSET_PAGES_NUDGE=off disables it without a deploy.

https://claude.ai/code/session_01R3SdsBKMTAniYr8U78UooQ


Summary by cubic

Stops Claude from reaching for a Claude artifact when the user asked for a Superset page. A PreToolUse hook now denies the first Artifact publish inside a Superset terminal and points the agent to the superset:page skill; the second call in the same session goes through untouched. The page skill, pages_publish tool description, and desktop agent prompt now name artifacts explicitly so the wording also pushes toward pages. The guard can be disabled with SUPERSET_PAGES_NUDGE=off.

Tracks SUPER-2021.

Notes

  • The guard only fires in Superset terminals, for publish actions without an existing url, once per session.
  • Existing installs pick up the hook on next boot; teardown removes it cleanly.
  • The edited SKILL.md ships via the bundled plugin dir; republishing the plugin to the marketplace is a follow-up.

Summary by CodeRabbit

  • New Features

    • Added a reminder that directs eligible artifact publishing to Superset Pages instead.
    • Prevents repeated reminders for the same session while allowing explicitly requested artifacts and opt-out behavior.
    • Pages guidance now highlights organization visibility, version history, pinned comments, and applicable content restrictions.
  • Bug Fixes

    • When the Pages skill is unavailable, the agent no longer substitutes a generic HTML workflow or Claude artifact and instead explains the required setup.
  • Tests

    • Added coverage for artifact publishing, session handling, opt-out behavior, and preserving existing configuration.