#579 Playbook Agent: examples and component walkthrough (8/8)

• depends on #572
• depends on #573
• depends on #574
• depends on #575
• depends on #576
• depends on #577
• depends on #578

Summary

Concrete examples and an end-to-end walkthrough that exercises every component through two contrasting scenarios.

Part 8 of 8 — final PR in the stack. Depends on #578 (runner and service).

Examples

File	Scenario
dummy_csam_triage.playbook.json	Hard rule fires: CRITICAL hash -> REPORT_AND_REMOVE
dummy_csam_triage.evidence.json	Sample query request and result
dummy_csam_triage.result.json	Expected PlaybookResult with confidence breakdown
account_takeover_triage.playbook.json	LLM decides: 3 labels, scoring guidance, no hard rules
account_takeover_triage.catalog.sql	SQL template with :account_id and :lookback_days
README.md	Documents backend flow and target UI rendering

componentWalkthrough.test.ts — 8 tests across 2 scenarios

Scenario 1: Account Takeover (LLM decides)

Test	Component exercised
Parses ATO playbook with no hard rules	PlaybookLoader
Parses SQL template, renders with bindings	QueryCatalog
Confirms no rules fire	HardRuleEngine
Extracts verdict from messy LLM text with code fences	JsonExtractor
Verifies C_final = C_ground x (1 - alpha x u_llm) math	ConfidenceEngine
Full pipeline: LLM verdict used, contradictions penalize	PlaybookAgentService

Scenario 2: CSAM Triage (hard rule decides)

Test	Component exercised
CRITICAL hash fires deterministic rule	HardRuleEngine
Hard rule overrides LLM NO_ACTION, rationale preserved	PlaybookAgentService

examples.test.ts — 2 tests

• Validates playbook fixture stays parseable
• Validates evidence and result fixtures stay aligned

Test plan

• npm test -- --testPathPattern=playbookAgentService passes (all 59 tests across 8 suites)

Depends on #578 Runner and service. Merge #572, #573, #574, #575, #576, #577, #578 first.

Summary by CodeRabbit

• New Features

  • Introduced a Playbook Agent framework for automated risk investigations, including playbook parsing, rule evaluation, evidence querying, LLM verdict extraction, and confidence scoring.
  • Added query catalog, deterministic hard-rule evaluation, JSON verdict extraction, and orchestration/runner service.
  • Included example playbooks and sample data for CSAM and account-takeover scenarios.

• Tests

  • Added extensive integration and unit tests covering parsing, rule evaluation, extraction, confidence computation, runner behavior, and end-to-end scenarios.

• Documentation

  • Added example README documenting the dummy CSAM triage example.

Part of #659