Fix generation and self-healing bugs found in hands-on testing

BI
3mo ago
+95-86

Summary

I ran generate-workflow end-to-end against a real site (an apartment listing site's apply-now flow, with a dismissible popup and a nav-driven multi-page task) using Anthropic models, then replayed the generated workflow. This surfaced three bugs blocking the two features that matter most for this project's value proposition — semantic workflow generation and self-healing replay — plus two smaller bugs in dead code that only appeared once the first fix let it actually run.

  • workflow_creation_prompt.md: an unescaped literal {variable} in prose (line 26) broke prompt_content.format(goal=..., actions=...) in HealingService.create_workflow_definition with KeyError: 'variable', since str.format() treats any single-brace token as a substitution placeholder, not just the intentional {goal}/{actions}. Escaped it to {{variable}} to match the rest of the template.

  • healing/service.py: _history_to_workflow_definition always tags the agent's screenshot as image/jpeg regardless of actual format, so Anthropic rejects it: messages.1.content.1.image.source.base64: The image was specified using the image/jpeg media type, but the image appears to be a image/png image. browser-use's get_screenshot() returns PNG; fixed the hardcoded MIME type to match.

  • workflow/service.py: _fallback_to_agent — the self-healing path — was fully implemented but entirely commented out, so any failed deterministic step (e.g. a selector that no longer matches after a page changes) crashed the whole run instead of healing, even though the surrounding log messages ("Attempting fallback with agent") implied it was active. Re-enabled it, and fixed two bugs in that dead code that only surfaced once it actually executed:

    • _run_agent_step(agent_step_config) was missing the required step_index argument.
    • self.steps doesn't exist on Workflow (the attribute is self.schema.steps, per the comment a few lines away at for step_index, step_dict in enumerate(self.schema.steps): # self.steps now holds dictionaries).

Result after all four fixes

Generated a workflow from a live multi-step task (navigate → dismiss popup → click nav → scroll → extract). On replay, two of the five steps' selectors didn't match (expected — they were generated from one page load and matched against a fresh one) and both healed successfully through two different paths: the agent-based fallback fixed here handled one, and the existing semantic-executor hierarchical-selector fallback handled the other. The workflow completed end to end with correct extracted data.

Test plan

  • Ran generate-workflow against a live site with ChatAnthropic as agent/extraction/workflow LLM — completes without the KeyError or image MIME error.
  • Ran Workflow.load_from_file(...).run(...) against the generated workflow — a deterministic step with a stale selector now heals via _fallback_to_agent instead of crashing, and the run completes successfully.
  • No existing automated tests were run against this change (couldn't find a test suite covering workflow/service.py's replay path) — happy to add coverage if there's a preferred pattern for it in this repo.

🤖 Generated with Claude Code

Summary by cubic

Fixes four bugs that blocked semantic workflow generation and self-healing replay in end-to-end testing. Failed deterministic steps previously crashed the entire run; they now fall back to the agent and heal.

Bug Fixes

  • Escaped a literal {variable} in workflow_creation_prompt.md so str.format() no longer raises KeyError: 'variable' during workflow generation.
  • Uses image/png instead of a hardcoded image/jpeg MIME type for screenshots, which Anthropic previously rejected as a format mismatch.
  • Re-enables _fallback_to_agent in workflow/service.py, which was fully implemented but commented out.
  • Fixes two latent bugs in the fallback path: passes the required step_index to _run_agent_step, and references self.schema.steps instead of the non-existent self.steps.

After these fixes, a workflow generated from a live multi-step task replayed end to end with both stale selectors healing successfully via the agent fallback and the existing semantic-executor fallback.