feat(coding-agent): clean webfetch HTML with reader mode

Merged
CO
4mo ago
+1300-40

Summary

  • Sync the merged pi-webfetch reader-mode implementation into senpi's vendored webfetch extension.
  • Convert HTML markdown/text responses through @mozilla/readability + jsdom before fallback conversion, so article content is returned without page chrome.
  • Add a regression test for markdown and text output, and refresh package lock/shrinkwrap metadata for the new dependencies.

External upstream work: https://github.com/code-yeongyu/pi-webfetch/pull/1

QA

  • npx tsx ../../node_modules/vitest/dist/cli.js --run test/suite/webfetch-reader-mode.test.ts
    • Evidence: local-ignore/qa-evidence/20260622-webfetch-reader-mode/senpi-webfetch-reader-test-6.txt
  • npm run check
    • Evidence: local-ignore/qa-evidence/20260622-webfetch-reader-mode/senpi-check-4.txt
  • node .agents/skills/senpi-qa/scripts/lib/common.mjs --self-check
    • Evidence: local-ignore/qa-evidence/20260622-webfetch-reader-mode/senpi-qa-common-self-check.txt
  • node .agents/skills/senpi-qa/scripts/mock-loop.mjs --with-tool
    • Evidence: local-ignore/qa-evidence/20260622-webfetch-reader-mode/senpi-qa-mock-loop-with-tool.txt
  • Real CLI webfetch QA with isolated fake model + local article server + --permission webfetch=allow.
    • Evidence: local-ignore/qa-evidence/20260622-webfetch-reader-mode/senpi-webfetch-cli-qa-3.txt
    • Observed: actual tool result contains ## Readable Fixture Article, Alpha fixture paragraph, and Beta fixture paragraph; excludes title/header/nav/aside/footer/script chrome.
  • Pre-commit hook passed: npm run check, node scripts/verify-package-managers.mjs for npm/bun/pnpm install+build, and browser smoke.

Review

  • Code review status: clear/approved after replacing the initial incomplete CLI QA artifact.
  • Audit note: npm audit --omit=dev still reports one high undici advisory under pre-existing @earendil-works/gondolin/node_modules/undici; the new jsdom subtree resolves undici@7.28.0.

Summary by cubic

Run HTML through reader mode before conversion so webfetch returns the main article (not page chrome) when requesting markdown or text. This improves readability and consistency across sites.

  • New Features

    • Use @mozilla/readability with jsdom to extract article content before markdown/text conversion; include the title if missing and avoid duplicates.
    • Pass the fetched URL into the converter for correct base URL handling, and add tests covering markdown and text outputs.
  • Dependencies

    • Add @mozilla/readability, jsdom, and @types/jsdom.
    • Update package-lock.json and npm-shrinkwrap.json.