neutral

Phase 23 — Visual Workflow Builder (Full Rewrite)

Replaces the existing flow builder (components/flow/) with an n8n-style visual workflow builder featuring schema-driven node configuration, live execution visualization, expression-based data mapping, Postgres persistence with versioning, dynamic MCP tool palette, and comprehensive test coverage.

Status: Not Started Depends on: Phase 22 (UI Remediation), Phase 20 (React UI), Phase 14 (Production API) Migrations: NNNN_workflow_definitions.sql (new tables) Branch: feature-react-ui

Why Now

Phase 20 delivered a functional flow builder with 8 node types, React Flow canvas, Zustand state, validation, serialization, and GitHub-backed persistence. However, the builder has significant architectural limitations that prevent it from reaching n8n-level usability:

No schema-driven configuration. The PropertyPanel uses hardcoded fields per node type. Adding a new tool requires modifying React code rather than simply registering a JSON schema.
No expression/data mapping engine. Nodes cannot reference output from upstream nodes. The condition field is a plain text input with no parsing, validation, or autocomplete.
No live execution visualization. Preview is a synchronous request/response cycle. There is no real-time SSE streaming of per-node execution states.
No execution history. Only git commit history exists. Users cannot view past workflow runs, drill into per-node results, or replay executions.
No dynamic tool discovery. The palette is static with 3 hardcoded groups. It does not discover tools from the MCP registry or tool registry API.
Missing node types. LLM decision nodes, supervisor nodes (7 patterns), and trigger nodes are absent.
Limited canvas features. No grid snapping, minimap, clipboard (copy/paste), or keyboard shortcuts beyond delete.

Phase 23 performs a full rewrite in a new components/flow-v2/ directory, preserving the existing flow builder until the rewrite is complete.

Architecture

Phase 23 follows a layered architecture with clean separation of concerns:

Canvas Layer (23A): React Flow canvas with custom node/edge components, Zustand state management with undo/redo, drag-and-drop palette, and keyboard shortcuts.
Configuration Layer (23B): Generic SchemaFormRenderer that reads JSON Schema and generates form fields. Per-node configuration panels with tabs (Parameters, Settings, Output). Expression editor with autocomplete.
Execution Layer (23C): Workflow execution via Temporal, SSE streaming of per-node states, visual execution overlay, result inspection drawer.
Persistence Layer (23D): Postgres-backed CRUD, workflow versioning (draft/published), execution history, graph compiler/decompiler for GraphDSL ↔ visual canvas.
Discovery Layer (23E): Dynamic palette populated from tool registry and MCP servers. Template gallery. Node grouping/subgraph support.
Quality Layer (23F): Unit tests (vitest, 80%+), integration tests (Go), E2E tests (Playwright), SonarQube frontend coverage.

Sub-Phases

Sub-Phase	Name	Prompts	Depends On
23A	Canvas Architecture & Node Framework	5	—
23B	Schema-Driven Node Configuration	6	23A
23C	Execution Engine & Live Visualization	5	23A, 23B
23D	Persistence, Versioning & History	5	23A, 23C
23E	Dynamic Palette & MCP Integration	4	23A, 23B
23F	Test Coverage & Quality	5	23A–23E

Total: 6 sub-phases, 30 prompts

Dependency Graph

23A (Canvas Architecture) ─────────┬──→ 23B (Schema Config) ──┬──→ 23C (Execution) ──→ 23D (Persistence)
                                   │                          │                                │
                                   └──→ 23E (Dynamic Palette) ─┘                               │
                                                                                               ▼
                                                                                         23F (Tests)

23A is the foundation. 23B and 23E can start after 23A. 23C needs 23A+23B. 23D needs 23A+23C. 23F is last.

Sub-Phase Details

23A: Canvas Architecture & Node Framework

Builds the new React Flow canvas with infinite pan/zoom, grid snapping, minimap, new node component architecture with typed input/output ports, connection type validation, Zustand state with undo/redo and clipboard, categorized palette sidebar with search and drag-and-drop, keyboard shortcuts, and the new serialization format.

23B: Schema-Driven Node Configuration

Builds the generic SchemaFormRenderer component that reads JSON Schema and generates form fields. Creates per-node configuration panels with tabs (Parameters, Settings, Output). Implements the expression editor with {{nodes.<id>.output.<path>}} syntax and autocomplete. Covers all node types: tool nodes, LLM decision nodes, control flow nodes, approval/HITL nodes, supervisor nodes (7 patterns), and start/end/trigger nodes.

23C: Execution Engine & Live Visualization

Implements workflow execution from canvas (full run and execute-to-node). Adds the backend /api/workflows/\{id\}/execute endpoint that submits the graph to Temporal. Streams per-node execution state via SSE/NATS. Adds visual execution states (idle → pending → running → success/error), result badges, node result drawer, execution log timeline, and cancel button.

23D: Persistence, Versioning & History

Adds Postgres-backed workflow CRUD (create, read, update, delete, duplicate). Implements workflow versioning with immutable published versions and draft editing. Adds execution history view with per-node result drill-down and read-only replay mode. Builds the graph compiler (visual → Graph DSL) and decompiler (Graph DSL → visual). Includes SQL migration.

23E: Dynamic Palette & MCP Integration

Auto-discovers MCP tools from the registry and generates palette entries dynamically. Generates node configuration panels from MCP tool schemas using SchemaFormRenderer. Adds node grouping/subgraph support (collapse selected nodes). Creates template gallery with 5–8 pre-built workflow templates.

23F: Test Coverage & Quality

Achieves 80%+ test coverage across all Phase 23 code. Unit tests for canvas components, SchemaFormRenderer, expression parser, flow compiler/decompiler. Integration tests for all new backend endpoints. E2E tests for the full workflow builder user journey. SonarQube frontend coverage configuration.

Files Overview

New Files

File	Sub-Phase	Description
`cmd/ui/frontend/src/components/flow-v2/`	23A	New flow builder component directory
`cmd/ui/frontend/src/components/flow-v2/Canvas.tsx`	23A	Main React Flow canvas
`cmd/ui/frontend/src/components/flow-v2/Toolbar.tsx`	23A	Top toolbar with actions
`cmd/ui/frontend/src/components/flow-v2/nodes/`	23A	Node components (FlowNode, per-type)
`cmd/ui/frontend/src/components/flow-v2/edges/`	23A	Edge components
`cmd/ui/frontend/src/components/flow-v2/palette/NodePalette.tsx`	23A	Categorized palette sidebar
`cmd/ui/frontend/src/stores/flow-builder.ts`	23A	New Zustand store
`cmd/ui/frontend/src/types/flow-builder.ts`	23A	New type definitions
`cmd/ui/frontend/src/lib/flow-graph-schema.json`	23A	Visual graph JSON schema
`cmd/ui/frontend/src/components/flow-v2/config/SchemaFormRenderer.tsx`	23B	Schema → form field renderer
`cmd/ui/frontend/src/components/flow-v2/config/ExpressionEditor.tsx`	23B	Expression editor with autocomplete
`cmd/ui/frontend/src/components/flow-v2/config/panels/`	23B	Per-node-type config panels
`cmd/ui/frontend/src/lib/schema-to-form.ts`	23B	Schema → field mapping logic
`cmd/ui/frontend/src/lib/expression-parser.ts`	23B	Expression parsing and evaluation
`cmd/ui/frontend/src/components/flow-v2/execution/ExecutionOverlay.tsx`	23C	Visual execution state overlay
`cmd/ui/frontend/src/components/flow-v2/execution/NodeResultDrawer.tsx`	23C	Per-node result inspection
`cmd/ui/frontend/src/components/flow-v2/execution/ExecutionTimeline.tsx`	23C	Execution log timeline
`cmd/ui/frontend/src/api/workflow-execution.ts`	23C	Execution API client
`cmd/ui/workflow_execution_handler.go`	23C	Backend execution handler
`cmd/ui/frontend/src/api/workflow-crud.ts`	23D	Workflow CRUD API client
`cmd/ui/frontend/src/components/flow-v2/history/ExecutionHistoryPanel.tsx`	23D	Execution history view
`cmd/ui/frontend/src/components/flow-v2/history/RunReplayOverlay.tsx`	23D	Read-only run replay
`cmd/ui/frontend/src/lib/flow-compiler.ts`	23D	Visual → Graph DSL compiler
`cmd/ui/frontend/src/lib/flow-decompiler.ts`	23D	Graph DSL → visual decompiler
`cmd/ui/workflow_crud_handler.go`	23D	Backend CRUD handler
`internal/store/workflow_store.go`	23D	Postgres workflow store
`migrations/NNNN_workflow_definitions.sql`	23D	Database migration
`cmd/ui/frontend/src/components/flow-v2/palette/DynamicNodePalette.tsx`	23E	MCP-aware dynamic palette
`cmd/ui/frontend/src/components/flow-v2/palette/TemplateGallery.tsx`	23E	Template browser
`cmd/ui/frontend/src/components/flow-v2/grouping/NodeGrouping.tsx`	23E	Node grouping/subgraph
`cmd/ui/frontend/src/api/tool-schemas.ts`	23E	Tool schema API client
`cmd/ui/tool_schemas_handler.go`	23E	Backend tool schema handler
`cmd/ui/frontend/src/__tests__/flow-v2/`	23F	Unit test directory
`cmd/ui/frontend/e2e/flow-builder-v2.spec.ts`	23F	E2E test spec
`cmd/ui/workflow_execution_handler_test.go`	23F	Backend handler tests
`cmd/ui/workflow_crud_handler_test.go`	23F	Backend CRUD handler tests
`cmd/ui/tool_schemas_handler_test.go`	23F	Backend schema handler tests
`internal/store/workflow_store_test.go`	23F	Postgres store tests

Modified Files

File	Sub-Phase	Change
`cmd/ui/main.go`	23C, 23D, 23E	Register new API endpoints
`cmd/ui/phase20_routes.go`	23C, 23D, 23E	Add new route registrations
`cmd/ui/frontend/src/App.tsx`	23A	Add flow builder v2 route
`cmd/ui/frontend/src/components/layout/Sidebar.tsx`	23A	Update flows nav item to use v2
`cmd/ui/frontend/vitest.config.ts`	23F	Add coverage configuration
`sonar-project.properties`	23F	Add JS/TS coverage paths

Success Metrics

Metric	Target
Node types supported	>= 12 (start, end, tool, route, parallel, join, subgraph, approval, llm_decision, supervisor, trigger, group)
Tool schema coverage	100% of registered tools have palette entries
Schema → form fields	All JSON Schema types mapped (string, number, boolean, enum, object, array, code)
Expression autocomplete	Upstream node outputs resolved with path completion
Execution visualization	Real-time SSE per-node state updates
Persistence	Postgres CRUD with versioning (draft/published)
Graph DSL round-trip	Compiler + decompiler produce equivalent graphs
Frontend unit coverage	>= 80% (flow-v2 components)
Backend handler coverage	>= 80% (new handlers)
E2E coverage	Full create → configure → execute → verify journey
Canvas UX	Grid snap, minimap, undo/redo, copy/paste, keyboard shortcuts
`npm run build`	Passes without errors
`go build ./cmd/ui/`	Passes without errors

Code Quality Requirements

ESLint: All new .tsx files pass existing ESLint config without warnings.
TypeScript: Strict mode — no any types, discriminated unions for node types, explicit return types on API functions.
Vitest: Coverage reporter configured with v8 provider, lcov output for SonarQube.
SonarQube: sonar.javascript.lcov.reportPaths set to frontend coverage output.
Go: All new backend code passes go vet, staticcheck, and existing CI checks.
Bundle size: Flow builder v2 page chunk under 100KB gzipped (canvas + React Flow + Monaco lazy-loaded).
Accessibility: All form fields have labels, keyboard-navigable palette, focus management in panels.

Risk Mitigation

Risk	Mitigation
React Flow performance with large graphs (100+ nodes)	Implement node virtualization, memoize node components, lazy-render off-screen nodes
Monaco editor bundle size	Lazy-load Monaco via dynamic import; only load when code/expression fields are opened
Expression parser complexity	Start with simple path expression syntax; defer function calls to a later phase
Graph DSL round-trip fidelity	23F includes compiler/decompiler round-trip unit tests with property-based testing
Existing flow builder disruption	v2 lives in `components/flow-v2/`; existing builder remains functional until v2 is complete
Database migration risk	Migration is additive (new tables only); no existing tables modified
SSE connection management	Reuse existing `hooks/useSSE.ts` pattern; add reconnection and heartbeat

Relationship to Other Phases

Phase	Relationship
Phase 20 (React UI)	23 replaces 20's flow builder; reuses 20's routing, components, API client patterns
Phase 22 (UI Remediation)	23 builds on 22's test infrastructure and component library improvements
Phase 14 (Production API)	23 backend handlers follow Phase 14 route patterns
Phase 7F (Graph DSL)	23D compiler targets the Graph DSL from Phase 7F
Phase 10 (Supervisor)	23B supervisor node panels reflect Phase 10's 7 patterns
Phase 9 (Tool Registry)	23E dynamic palette consumes Phase 9's tool registry
Phase 12 (Events)	23C execution streaming uses Phase 12's NATS/SSE infrastructure
Phase 16 (Test Coverage)	23F extends Phase 16's coverage patterns to the flow builder

Investigation Notes

docs/notes/phase23-n8n-reference.md — n8n UX patterns and interaction models
docs/notes/phase23-current-ui-audit.md — Existing flow builder inventory
docs/notes/phase23-tool-schemas.md — Tool definitions with JSON schemas
docs/notes/phase23-api-requirements.md — Existing vs needed backend endpoints
docs/notes/phase23-test-baseline.md — Test infrastructure and SonarQube config

Progress Notes

(none yet)

Why Now​

Architecture​

Sub-Phases​

Dependency Graph​

Sub-Phase Details​

23A: Canvas Architecture & Node Framework​

23B: Schema-Driven Node Configuration​

23C: Execution Engine & Live Visualization​

23D: Persistence, Versioning & History​

23E: Dynamic Palette & MCP Integration​

23F: Test Coverage & Quality​

Files Overview​

New Files​

Modified Files​

Success Metrics​

Code Quality Requirements​

Risk Mitigation​

Relationship to Other Phases​

Investigation Notes​

Progress Notes​