scawful/yaze
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b77bd201e262765fdd5dcd3d51491b14f207f081
yaze/docs/z3ed/E6-z3ed-implementation-plan.md

47 KiB
Raw Blame History

z3ed Agentic Workflow Implementation Plan

Last Updated: October 2, 2025 (10:30 PM)
Status: IT-01 Complete ✅ | IT-02 Complete ✅ | E2E Validation Ready 🎯

📋 Quick Start: See README.md for essential links and NEXT_PRIORITIES_OCT2.md for detailed task guides.

Executive Summary

The z3ed CLI and AI agent workflow system has completed major infrastructure milestones:

✅ Completed Phases:

  • Phase 6: Resource Catalogue - Machine-readable API specs for AI consumption
  • AW-01/02/03: Acceptance Workflow - Proposal tracking, sandbox management, GUI review with ROM merging
  • IT-01: ImGuiTestHarness - Full GUI automation via gRPC + ImGuiTestEngine (all 3 phases complete)
  • IT-02: CLI Agent Test - Natural language → automated GUI testing (implementation complete)

🔄 Active Phase:

  • E2E Validation: Testing complete proposal lifecycle with real GUI widgets (window detection debugging in progress)

📋 Next Phases:

  • Priority 1: Complete E2E Validation - Fix window detection after menu actions (2-3 hours)
  • Priority 2: Policy Evaluation Framework (AW-04) - YAML-based constraints for proposal acceptance (6-8 hours)

Recent Accomplishments (October 2, 2025):

  • IT-02 implementation complete with async test queue pattern
  • Build system fixes for z3ed target (gRPC integration)
  • Documentation consolidated into clean structure
  • E2E test script operational (5/6 RPCs working)
  • Menu interaction verified via ImGuiTestEngine

Known Issues:

  • Window detection timing after menu clicks needs refinement
  • Screenshot RPC proto mismatch (non-critical)

Time Investment: 20.5 hours total (IT-01: 11h, IT-02: 7.5h, Docs: 2h)
Code Quality: All targets compile cleanly, no crashes, partial test coverage

Quick Reference

Start Test Harness:

./build-grpc-test/bin/yaze.app/Contents/MacOS/yaze \
  --enable_test_harness \
  --test_harness_port=50052 \
  --rom_file=assets/zelda3.sfc &

Test All RPCs:

./scripts/test_harness_e2e.sh

Create Proposal:

./build/bin/z3ed agent run "Test prompt" --sandbox
./build/bin/z3ed agent list
./build/bin/z3ed agent diff --proposal-id <ID>

Review in GUI:

  • Open YAZE → Debug → Agent Proposals
  • Select proposal → Review → Accept/Reject/Delete

1. Current Priorities (Week of Oct 2-8, 2025)

Status: IT-01 Complete ✅ | IT-02 Complete ✅ | E2E Tests Running ⚡

Priority 0: E2E Test Validation (IMMEDIATE) 🎯

Goal: Validate test harness with real YAZE widgets
Time Estimate: 30-60 minutes
Status: Test script running, needs real widget names

Current Results:

  • ✅ Ping RPC working
  • ⚠️ Tests 2-5 using fake widget names
  • 📋 Need to identify real widget names from YAZE source
  • 🔧 Screenshot RPC needs proto fix

Task Checklist:

  1. ✅ E2E Test Script: Already created (scripts/test_harness_e2e.sh)
  2. 📋 Manual Testing Workflow:
    • Start YAZE with test harness enabled
    • Create proposal via CLI: z3ed agent run "Test prompt" --sandbox
    • Verify proposal appears in ProposalDrawer GUI
    • Test Accept → validate ROM merge and save prompt
    • Test Reject → validate status update
    • Test Delete → validate cleanup
  3. 📋 Real Widget Testing:
    • Click actual YAZE buttons (Overworld, Dungeon, etc.)
    • Type into real input fields
    • Wait for actual windows to appear
    • Assert on real widget states
  4. 📋 Document Edge Cases:
    • Widget not found scenarios
    • Timeout handling
    • Error recovery patterns

Priority 2: CLI Agent Test Command (IT-02) 📋 NEXT

Goal: Natural language → automated GUI testing via gRPC
Time Estimate: 4-6 hours
Blocking Dependency: Priority 1 completion

Implementation Tasks:

  1. Create z3ed agent test command:

    • Parse natural language prompt
    • Generate RPC call sequence (Click → Wait → Assert)
    • Execute via gRPC client
    • Capture results and screenshots

  2. Example Usage:

    z3ed agent test --prompt "Open Overworld editor and verify it loads" \
  --rom zelda3.sfc

# Generated workflow:
# 1. Click "button:Overworld"
# 2. Wait "window_visible:Overworld Editor" (5s)
# 3. Assert "visible:Overworld Editor"
# 4. Screenshot "full"

  3. Implementation Files:

    • src/cli/handlers/agent.cc - Add HandleTestCommand()
    • src/cli/service/gui_automation_client.{h,cc} - gRPC client wrapper
    • src/cli/service/test_workflow_generator.{h,cc} - Prompt → RPC translator

Priority 3: Policy Evaluation Framework (AW-04) 📋

Goal: YAML-based constraint system for gating proposal acceptance
Time Estimate: 6-8 hours
Blocking Dependency: None (can work in parallel)

<EFBFBD> Detailed Guides: See NEXT_PRIORITIES_OCT2.md for complete implementation breakdowns with code examples.

2. Workstreams Overview

This plan decomposes the design additions into actionable engineering tasks. Each workstream contains milestones, blocking dependencies, and expected deliverables.

  1. src/cli/handlers/rom.cc - Added RomInfo::Run implementation
  2. src/cli/z3ed.h - Added RomInfo class declaration
  3. src/cli/modern_cli.cc - Updated HandleRomInfoCommand routing
  4. src/cli/service/resource_catalog.cc - Added rom info schema entry

2. Workstreams Overview

Workstream Goal Status Notes
Resource Catalogue Machine-readable CLI specs for AI consumption ✅ Complete docs/api/z3ed-resources.yaml generated
Acceptance Workflow Human review/approval of agent proposals ✅ Complete ProposalDrawer with ROM merging operational
ImGuiTest Bridge Automated GUI testing via gRPC ✅ Complete All 3 phases done (11 hours)
Verification Pipeline Layered testing + CI coverage 📋 In Progress E2E validation phase
Telemetry & Learning Capture signals for improvement 📋 Planned Optional/opt-in (Phase 8)

Completed Work Summary

Resource Catalogue (RC) ✅:

  • CLI flag passthrough and resource catalog system
  • agent describe exports YAML/JSON schemas
  • docs/api/z3ed-resources.yaml maintained
  • All ROM/Palette/Overworld/Dungeon/Patch commands documented

Acceptance Workflow (AW-01/02/03) ✅:

  • ProposalRegistry with disk persistence and cross-session tracking
  • RomSandboxManager for isolated ROM copies
  • agent list and agent diff commands
  • ProposalDrawer GUI: List/detail views, Accept/Reject/Delete, ROM merging
  • Integrated into EditorManager (Debug → Agent Proposals)

ImGuiTestHarness (IT-01) ✅:

  • Phase 1: gRPC infrastructure (6 RPC methods)
  • Phase 2: TestManager integration with dynamic tests
  • Phase 3: Full ImGuiTestEngine (Type/Wait/Assert RPCs)
  • E2E test script: scripts/test_harness_e2e.sh
  • Documentation: IT-01-QUICKSTART.md

3. Task Backlog

ID Task Workstream Type Status Dependencies
RC-01 Define schema for ResourceCatalog entries and implement serialization helpers. Resource Catalogue Code ✅ Done Schema system complete with all resource types documented
RC-02 Auto-generate docs/api/z3ed-resources.yaml from command annotations. Resource Catalogue Tooling ✅ Done Generated and committed to docs/api/
RC-03 Implement z3ed agent describe CLI surface returning JSON schemas. Resource Catalogue Code ✅ Done Both YAML and JSON output formats working
RC-04 Integrate schema export with TUI command palette + help overlays. Resource Catalogue UX 📋 Planned RC-03
RC-05 Harden CLI command routing/flag parsing to unblock agent automation. Resource Catalogue Code ✅ Done Fixed rom info handler to use FLAGS_rom
AW-01 Implement sandbox ROM cloning and tracking (RomSandboxManager). Acceptance Workflow Code ✅ Done ROM sandbox manager operational with lifecycle management
AW-02 Build proposal registry service storing diffs, logs, screenshots. Acceptance Workflow Code ✅ Done ProposalRegistry implemented with disk persistence
AW-03 Add ImGui drawer for proposals with accept/reject controls. Acceptance Workflow UX ✅ Done ProposalDrawer GUI complete with ROM merging
AW-04 Implement policy evaluation for gating accept buttons. Acceptance Workflow Code 📋 Next AW-03, Priority 1 - YAML policies + PolicyEvaluator (6-8 hours)
AW-05 Draft .z3ed-diff hybrid schema (binary deltas + JSON metadata). Acceptance Workflow Design 📋 Planned AW-01
IT-01 Create ImGuiTestHarness IPC service embedded in yaze_test. ImGuiTest Bridge Code ✅ Done Phase 1+2+3 Complete - Full GUI automation with gRPC + ImGuiTestEngine (11 hours)
IT-02 Implement CLI agent step translation (imgui_action → harness call). ImGuiTest Bridge Code ✅ Done z3ed agent test command with natural language prompts (7.5 hours)
IT-03 Provide synchronization primitives (WaitForIdle, etc.). ImGuiTest Bridge Code ✅ Done Wait RPC with condition polling already implemented in IT-01 Phase 3
IT-04 Complete E2E validation with real YAZE widgets ImGuiTest Bridge Test 🔄 Active IT-02, Fix window detection after menu actions (2-3 hours)
VP-01 Expand CLI unit tests for new commands and sandbox flow. Verification Pipeline Test 📋 Planned RC/AW tasks
VP-02 Add harness integration tests with replay scripts. Verification Pipeline Test 📋 Planned IT tasks
VP-03 Create CI job running agent smoke tests with YAZE_WITH_JSON. Verification Pipeline Infra 📋 Planned VP-01, VP-02
TL-01 Capture accept/reject metadata and push to telemetry log. Telemetry & Learning Code 📋 Planned AW tasks
TL-02 Build anonymized metrics exporter + opt-in toggle. Telemetry & Learning Infra 📋 Planned TL-01

Status Legend: 🔄 Active · 📋 Planned · ✅ Done

Progress Summary:

  • ✅ Completed: 11 tasks (61%)
  • 🔄 Active: 1 task (6%)
  • 📋 Planned: 6 tasks (33%)
  • Total: 18 tasks

3. Immediate Next Steps (Week of Oct 1-7, 2025)

Priority 0: Testing & Validation (Active)

  1. TEST: Complete end-to-end proposal workflow

    • Launch YAZE and verify ProposalDrawer displays live proposals
    • Test Accept action → verify ROM merge and save prompt
    • Test Reject and Delete actions
    • Validate filtering and refresh functionality

  2. Widget ID Refactoring (Started Oct 2, 2025) 🎯 NEW

    • ✅ Added widget_id_registry to build system
    • ✅ Registered 13 Overworld toolset buttons with hierarchical IDs
    • 📋 Next: Test widget discovery and update test harness
    • See: WIDGET_ID_REFACTORING_PROGRESS.md

Priority 1: ImGuiTestHarness Foundation (IT-01) ✅ PHASE 2 COMPLETE

Rationale: Required for automated GUI testing and remote control of YAZE for AI workflows
Decision: ✅ Use gRPC - Production-grade, cross-platform, type-safe (see IT-01-grpc-evaluation.md)

Status: Phase 1 Complete ✅ | Phase 2 Complete ✅ | Phase 3 Planned <20>

Phase 1: gRPC Infrastructure ✅ COMPLETE

  • ✅ Add gRPC to build system via FetchContent
  • ✅ Create .proto schema (Ping, Click, Type, Wait, Assert, Screenshot)
  • ✅ Implement gRPC server with all 6 RPC stubs
  • ✅ Test with grpcurl - all RPCs responding
  • ✅ Server lifecycle management (Start/Shutdown)
  • ✅ Cross-platform build verified (macOS ARM64)

See: GRPC_TEST_SUCCESS.md for Phase 1 completion details

Phase 2: ImGuiTestEngine Integration ✅ COMPLETE

Goal: Replace stub RPC handlers with actual GUI automation
Status: Infrastructure complete, dynamic test registration implemented
Time Spent: ~4 hours

Implementation Guide: 📖 IT-01-PHASE2-IMPLEMENTATION-GUIDE.md

Completed Tasks:

  1. ✅ TestManager Integration - gRPC service receives TestManager reference
  2. ✅ Build System - Successfully compiles with ImGuiTestEngine support
  3. ✅ Server Startup - gRPC server starts correctly on macOS with test harness flag
  4. ✅ Dynamic Test Registration - Click RPC uses IM_REGISTER_TEST() macro for dynamic tests
  5. ✅ Stub Handlers - Type/Wait/Assert RPCs return success (implementation pending Phase 3)
  6. ✅ Ping RPC - Fully functional, returns YAZE version and timestamp

Key Learnings:

  • ImGuiTestEngine requires test registration - can't call test functions directly
  • Test context provided by engine via test->Output.Status not test->Status
  • YAZE uses custom flag system with FLAGS_name->Get() pattern
  • Correct flags: --enable_test_harness, --test_harness_port, --rom_file

Testing Results:

# Server starts successfully
./build-grpc-test/bin/yaze.app/Contents/MacOS/yaze \
  --enable_test_harness \
  --test_harness_port=50052 \
  --rom_file=assets/zelda3.sfc &

# Ping RPC working
grpcurl -plaintext -d '{"message":"test"}' \
  127.0.0.1:50052 yaze.test.ImGuiTestHarness/Ping
# Response: {"message":"Pong: test","timestampMs":"...","yazeVersion":"0.3.2"}

Issues Fixed:

  • ❌→✅ SIGSEGV on TestManager initialization (deferred ImGuiTestEngine init to Phase 3)
  • ❌→✅ ImGuiTestEngine API mismatch (switched to dynamic test registration)
  • ❌→✅ Status field access (corrected to test->Output.Status)
  • ❌→✅ Port conflicts (use port 50052, killall yaze to cleanup)
  • ❌→✅ Flag naming (documented correct underscore format)

Phase 3: Full ImGuiTestEngine Integration ✅ COMPLETE (Oct 2, 2025)

Goal: Complete implementation of all GUI automation RPCs

Completed Tasks:

  1. ✅ Type RPC Implementation - Full text input automation

    • ItemInfo API usage corrected (returns by value, not pointer)
    • Focus management with ItemClick before typing
    • Clear-first functionality with keyboard shortcuts
    • Dynamic test registration with timeout handling

  2. ✅ Wait RPC Implementation - Condition polling with timeout

    • Three condition types: window_visible, element_visible, element_enabled
    • Configurable timeout (default 5000ms) and poll interval (default 100ms)
    • Proper Yield() calls to allow ImGui event processing
    • Extended timeout for test execution

  3. ✅ Assert RPC Implementation - State validation with structured responses

    • Multiple assertion types: visible, enabled, exists, text_contains
    • Actual vs expected value reporting
    • Detailed error messages for debugging
    • text_contains partially implemented (text retrieval needs refinement)

  4. ✅ API Compatibility Fixes

    • Corrected ItemInfo usage (by value, check ID != 0)
    • Fixed flag names (ItemFlags instead of StatusFlags)
    • Proper visibility checks using RectClipped dimensions
    • All dynamic tests properly registered and cleaned up

Testing:

  • Build successful on macOS ARM64
  • All RPCs respond correctly
  • Test script created: scripts/test_harness_e2e.sh
  • See IT-01-PHASE3-COMPLETE.md for full implementation details

Known Limitations:

  • Screenshot RPC not implemented (placeholder stub)
  • text_contains assertion uses placeholder text retrieval
  • Need end-to-end workflow testing with real YAZE widgets
  1. End-to-End Testing (1 hour)
    • Create shell script workflow: start server → click button → wait for window → type text → assert state
    • Test with real YAZE editors (Overworld, Dungeon, etc.)
    • Document edge cases and troubleshooting

Phase 4: CLI Integration & Windows Testing (4-5 hours)

  1. CLI Client (z3ed agent test)

    • Generate gRPC calls from AI prompts
    • Natural language → ImGui action translation
    • Screenshot capture for LLM feedback

  2. Windows Testing

    • Detailed build instructions for vcpkg setup
    • Test on Windows VM or with contributor
    • Add Windows CI job to GitHub Actions
    • Document troubleshooting

IT-01 Quick Reference

Start YAZE with Test Harness:

./build-grpc-test/bin/yaze.app/Contents/MacOS/yaze \
  --enable_test_harness \
  --test_harness_port=50052 \
  --rom_file=assets/zelda3.sfc &

Test RPCs with grpcurl:

# Ping - Health check
grpcurl -plaintext -import-path src/app/core/proto -proto imgui_test_harness.proto \
  -d '{"message":"test"}' 127.0.0.1:50052 yaze.test.ImGuiTestHarness/Ping

# Click - Click UI element
grpcurl -plaintext -import-path src/app/core/proto -proto imgui_test_harness.proto \
  -d '{"target":"button:Overworld","type":"LEFT"}' \
  127.0.0.1:50052 yaze.test.ImGuiTestHarness/Click

# Type - Input text
grpcurl -plaintext -import-path src/app/core/proto -proto imgui_test_harness.proto \
  -d '{"target":"input:Filename","text":"zelda3.sfc","clear_first":true}' \
  127.0.0.1:50052 yaze.test.ImGuiTestHarness/Type

# Wait - Wait for condition
grpcurl -plaintext -import-path src/app/core/proto -proto imgui_test_harness.proto \
  -d '{"condition":"window_visible:Overworld Editor","timeout_ms":5000}' \
  127.0.0.1:50052 yaze.test.ImGuiTestHarness/Wait

# Assert - Validate state
grpcurl -plaintext -import-path src/app/core/proto -proto imgui_test_harness.proto \
  -d '{"condition":"visible:Main Window"}' \
  127.0.0.1:50052 yaze.test.ImGuiTestHarness/Assert

Troubleshooting:

  • Port in use: killall yaze or use --test_harness_port=50053
  • Connection refused: Check server started with lsof -i :50052
  • Unrecognized flag: Use underscores not hyphens (e.g., --rom_file not --rom)

Priority 2: Policy Evaluation Framework (AW-04, 4-6 hours)

  1. DESIGN: YAML-based Policy Configuration

    # .yaze/policies/agent.yaml
version: 1.0
policies:
  - name: require_tests
    type: test_requirement
    enabled: true
    rules:
      - test_suite: "overworld_rendering"
        min_pass_rate: 0.95
      - test_suite: "palette_integrity"
        min_pass_rate: 1.0

  - name: limit_change_scope
    type: change_constraint
    enabled: true
    rules:
      - max_bytes_changed: 10240  # 10KB
      - allowed_banks: [0x00, 0x01, 0x0E]  # Graphics banks only
      - forbidden_ranges:
        - start: 0xFFB0  # ROM header
          end: 0xFFFF

  - name: human_review_required
    type: review_requirement
    enabled: true
    rules:
      - if: bytes_changed > 1024
        then: require_diff_review: true
      - if: commands_executed > 10
        then: require_log_review: true

  2. IMPLEMENT: PolicyEvaluator Service

    • src/cli/service/policy_evaluator.{h,cc}
    • Singleton service loads policies from .yaze/policies/
    • EvaluateProposal(proposal_id) -> PolicyResult
    • Returns: pass/fail + list of violations with severity
    • Hook into ProposalRegistry lifecycle

  3. INTEGRATE: Policy UI in ProposalDrawer

    • Add "Policy Status" section in detail view
    • Display violations with icons: ⛔ Critical, ⚠️ Warning, ℹ️ Info
    • Gate Accept button: disabled if critical violations exist
    • Show helpful messages: "Accept blocked: Test pass rate 0.85 < 0.95"
    • Allow policy overrides with confirmation: "Override policy? This action will be logged."

Priority 3: Documentation & Consolidation (2-3 hours)

  1. CONSOLIDATE: Merge standalone docs into main plan

    • ✅ AW-03 summary → already in main plan, delete standalone doc
    • Check for other AW-* or task-specific docs to merge
    • Update main plan with architecture diagrams

  2. CREATE: Architecture Flow Diagram

    • Visual representation of proposal lifecycle
    • Component interaction diagram
    • Add to implementation plan

Later: Advanced Features

  • VP-01: Expand CLI unit tests
  • VP-02: Integration tests with replay scripts
  • TL-01: Telemetry capture for learning

4. Current Issues & Blockers

Active Issues

None - all blocking issues resolved as of Oct 1, 2025

Known Limitations (Non-Blocking)

  1. ProposalDrawer lacks keyboard navigation
  2. Large diffs/logs truncated at 1000 lines (consider pagination)
  3. Proposals don't persist full metadata to disk (prompt, description, sandbox_id reconstructed)
  4. No policy evaluation yet (AW-04)

5. Architecture Overview

5.1. Proposal Lifecycle Flow

┌─────────────────────────────────────────────────────────────────┐
│ 1. CREATION (CLI: z3ed agent run)                               │
├─────────────────────────────────────────────────────────────────┤
│ User Prompt                                                      │
│      ↓                                                           │
│ MockAIService / GeminiAIService                                 │
│      ↓ (generates commands)                                     │
│ ["palette export ...", "overworld set-tile ..."]                │
│      ↓                                                           │
│ RomSandboxManager::CreateSandbox(rom)                           │
│      ↓ (creates isolated copy)                                  │
│ /tmp/yaze/sandboxes/<timestamp>/zelda3.sfc                      │
│      ↓                                                           │
│ Execute commands on sandbox ROM                                 │
│      ↓ (logs each command)                                      │
│ ProposalRegistry::CreateProposal(sandbox_id, prompt, desc)      │
│      ↓ (creates proposal directory)                             │
│ /tmp/yaze/proposals/proposal-<timestamp>-<seq>/                 │
│   ├─ execution.log (command outputs)                            │
│   ├─ diff.txt (if generated)                                    │
│   └─ screenshots/ (if any)                                      │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│ 2. DISCOVERY (CLI: z3ed agent list)                             │
├─────────────────────────────────────────────────────────────────┤
│ ProposalRegistry::ListProposals()                               │
│      ↓ (lazy loads from disk)                                   │
│ LoadProposalsFromDiskLocked()                                   │
│      ↓ (scans /tmp/yaze/proposals/)                             │
│ Reconstructs metadata from filesystem                           │
│      ↓ (parses timestamps, reads logs)                          │
│ Returns vector<ProposalMetadata>                                │
│      ↓                                                           │
│ Display table: ID | Status | Created | Prompt | Stats           │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│ 3. REVIEW (GUI: Debug → Agent Proposals)                        │
├─────────────────────────────────────────────────────────────────┤
│ ProposalDrawer::Draw()                                          │
│      ↓ (called every frame from EditorManager)                  │
│ ProposalDrawer::RefreshProposals()                              │
│      ↓ (calls ProposalRegistry::ListProposals)                  │
│ Display proposal list (selectable table)                        │
│      ↓ (user clicks proposal)                                   │
│ ProposalDrawer::SelectProposal(id)                              │
│      ↓ (loads detail content)                                   │
│ Read execution.log and diff.txt from proposal directory         │
│      ↓                                                           │
│ Display detail view:                                            │
│   ├─ Metadata (sandbox_id, timestamp, stats)                   │
│   ├─ Diff (syntax highlighted)                                  │
│   └─ Log (command execution trace)                              │
│      ↓                                                           │
│ User decides: [Accept] [Reject] [Delete]                        │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│ 4. ACCEPTANCE (GUI: Click "Accept" button)                      │
├─────────────────────────────────────────────────────────────────┤
│ ProposalDrawer::AcceptProposal(proposal_id)                     │
│      ↓                                                           │
│ Get proposal metadata (includes sandbox_id)                     │
│      ↓                                                           │
│ RomSandboxManager::ListSandboxes()                              │
│      ↓ (find sandbox by ID)                                     │
│ sandbox_rom_path = sandbox.rom_path                             │
│      ↓                                                           │
│ Load sandbox ROM from disk                                      │
│      ↓                                                           │
│ rom_->WriteVector(0, sandbox_rom.vector())                      │
│      ↓ (copies entire sandbox ROM → main ROM)                   │
│ ROM marked dirty (save prompt appears)                          │
│      ↓                                                           │
│ ProposalRegistry::UpdateStatus(id, kAccepted)                   │
│      ↓                                                           │
│ User: File → Save ROM                                           │
│      ↓                                                           │
│ Changes committed ✅                                            │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│ 5. REJECTION (GUI: Click "Reject" button)                       │
├─────────────────────────────────────────────────────────────────┤
│ ProposalDrawer::RejectProposal(proposal_id)                     │
│      ↓                                                           │
│ ProposalRegistry::UpdateStatus(id, kRejected)                   │
│      ↓                                                           │
│ Proposal preserved for audit trail                              │
│ Sandbox ROM left untouched (can be cleaned up later)            │
└─────────────────────────────────────────────────────────────────┘

5.2. Component Interaction Diagram

┌────────────────────┐
│   CLI Layer        │
│  (z3ed commands)   │
└────────┬───────────┘
         │
         ├──► agent run ──────────┐
         ├──► agent list ─────────┤
         └──► agent diff ─────────┤
                                  │
         ┌────────────────────────▼──────────────────────┐
         │         CLI Service Layer                     │
         ├───────────────────────────────────────────────┤
         │  ┌─────────────────────────────────────────┐  │
         │  │ ProposalRegistry (Singleton)            │  │
         │  │  • CreateProposal()                     │  │
         │  │  • ListProposals()                      │  │
         │  │  • GetProposal()                        │  │
         │  │  • UpdateStatus()                       │  │
         │  │  • RemoveProposal()                     │  │
         │  │  • LoadProposalsFromDiskLocked()        │  │
         │  └────────────┬────────────────────────────┘  │
         │               │                               │
         │  ┌────────────▼────────────────────────────┐  │
         │  │ RomSandboxManager (Singleton)          │  │
         │  │  • CreateSandbox()                     │  │
         │  │  • ActiveSandbox()                     │  │
         │  │  • ListSandboxes()                     │  │
         │  │  • RemoveSandbox()                     │  │
         │  └────────────┬────────────────────────────┘  │
         └───────────────┼────────────────────────────────┘
                         │
         ┌───────────────▼────────────────────────────────┐
         │         Filesystem Layer                       │
         ├────────────────────────────────────────────────┤
         │  /tmp/yaze/proposals/                          │
         │    └─ proposal-<timestamp>-<seq>/              │
         │         ├─ execution.log                       │
         │         ├─ diff.txt                            │
         │         └─ screenshots/                        │
         │                                                │
         │  /tmp/yaze/sandboxes/                          │
         │    └─ <timestamp>-<seq>/                       │
         │         └─ zelda3.sfc (isolated ROM copy)      │
         └────────────────────────────────────────────────┘
                         ▲
                         │
         ┌───────────────┴────────────────────────────────┐
         │         GUI Layer                              │
         ├────────────────────────────────────────────────┤
         │  ┌─────────────────────────────────────────┐   │
         │  │ EditorManager                           │   │
         │  │  • current_rom_                         │   │
         │  │  • proposal_drawer_                     │   │
         │  │  • Update() { proposal_drawer_.Draw() } │   │
         │  └────────────┬────────────────────────────┘   │
         │               │                                │
         │  ┌────────────▼────────────────────────────┐   │
         │  │ ProposalDrawer                          │   │
         │  │  • rom_ (ptr to EditorManager's ROM)    │   │
         │  │  • Draw()                               │   │
         │  │  • DrawProposalList()                   │   │
         │  │  • DrawProposalDetail()                 │   │
         │  │  • AcceptProposal() ← ROM MERGE         │   │
         │  │  • RejectProposal()                     │   │
         │  │  • DeleteProposal()                     │   │
         │  └─────────────────────────────────────────┘   │
         └────────────────────────────────────────────────┘

5.3. Data Flow: Agent Run to ROM Merge

User: "Make soldiers wear red armor"
         │
         ▼
┌────────────────────────┐
│ MockAIService          │ Generates: ["palette export sprites_aux1 4 soldier.col"]
└────────┬───────────────┘
         │
         ▼
┌────────────────────────┐
│ RomSandboxManager      │ Creates: /tmp/.../sandboxes/20251001T200215-1/zelda3.sfc
└────────┬───────────────┘
         │
         ▼
┌────────────────────────┐
│ Command Executor       │ Runs: palette export on sandbox ROM
└────────┬───────────────┘
         │
         ▼
┌────────────────────────┐
│ ProposalRegistry       │ Creates: proposal-20251001T200215-1/
│                        │   • execution.log: "[timestamp] palette export succeeded"
└────────┬───────────────┘   • diff.txt: (if diff generated)
         │
         │ Time passes... user launches GUI
         ▼
┌────────────────────────┐
│ ProposalDrawer loads   │ Reads: /tmp/.../proposals/proposal-*/
│                        │ Displays: List of proposals
└────────┬───────────────┘
         │
         │ User clicks "Accept"
         ▼
┌────────────────────────┐
│ AcceptProposal()       │ 1. Find sandbox ROM: /tmp/.../sandboxes/.../zelda3.sfc
│                        │ 2. Load sandbox ROM
│                        │ 3. rom_->WriteVector(0, sandbox_rom.vector())
│                        │ 4. Main ROM now contains all sandbox changes
│                        │ 5. ROM marked dirty
└────────┬───────────────┘
         │
         ▼
┌────────────────────────┐
│ User: File → Save      │ Changes persisted to disk ✅
└────────────────────────┘

5. Open Questions

  • What serialization format should the proposal registry adopt for diff payloads (binary vs. textual vs. hybrid)?
    ➤ Decision: pursue a hybrid package (.z3ed-diff) that wraps binary tile/object deltas alongside a JSON metadata envelope (identifiers, texture descriptors, preview palette info). Capture format draft under RC/AW backlog.
  • How should the harness authenticate escalation requests for mutation actions?
    ➤ Still open—evaluate shared-secret vs. interactive user prompt in the harness spike (IT-01).
  • Can we reuse existing regression test infrastructure for nightly ImGui runs or should we spin up a dedicated binary?
    ➤ Investigate during the ImGuiTestHarness spike; compare extending yaze_test jobs versus introducing a lightweight automation runner.

5. Completed Work Summary

Resource Catalogue Workstream (RC) - ✅ COMPLETE

The Resource Catalogue workstream has been successfully completed, providing a foundation for AI-driven automation:

Implementation Details:

  • Created comprehensive schema system in src/cli/service/resource_catalog.{h,cc}
  • Implemented resource catalog for: ROM, Patch, Palette, Overworld, Dungeon, and Agent commands
  • Each resource includes: name, description, actions, arguments, effects, and return values
  • Built dual-format serialization: JSON (compact) and YAML (human-readable)

Key Fixes:

  • Fixed rom info segfault by creating dedicated RomInfo handler using FLAGS_rom
  • Added rom info action to resource schema with proper metadata
  • Ensured all ROM commands consistently use flag-based dispatch

Generated Artifacts:

  • docs/api/z3ed-resources.yaml - Authoritative machine-readable API reference
  • Both JSON and YAML output formats validated and working
  • Resource filtering capability (--resource <name>) operational

Command Examples:

# View all resources in YAML
z3ed agent describe --format yaml

# Get specific resource as JSON
z3ed agent describe --format json --resource rom

# Generate documentation file
z3ed agent describe --format yaml --output docs/api/z3ed-resources.yaml

Testing Results: All commands tested and verified working:

  • ✅ z3ed rom info --rom=zelda3.sfc - displays title, size, filename
  • ✅ z3ed rom validate --rom=zelda3.sfc - verifies checksum and header
  • ✅ z3ed agent describe --format yaml - outputs complete catalog
  • ✅ z3ed agent describe --format json --resource rom - filters by resource

Acceptance Workflow (AW-01, AW-02) - ✅ CORE COMPLETE

The foundational infrastructure for proposal tracking and review is now operational:

RomSandboxManager Implementation (AW-01):

  • Singleton service managing isolated ROM copies for agent proposals
  • Sandboxes created in YAZE_SANDBOX_ROOT (env var) or system temp directory
  • Automatic directory creation and ROM file cloning
  • Active sandbox tracking for current agent session
  • Cleanup utilities for removing old sandboxes

ProposalRegistry Implementation (AW-02):

  • Comprehensive tracking of agent-generated ROM modifications
  • Stores proposal metadata: ID, sandbox ID, prompt, description, timestamps
  • Records execution diffs in diff.txt within proposal directory
  • Appends command execution logs to execution.log with timestamps
  • Support for screenshot attachments (path tracking)
  • Proposal lifecycle: Pending → Accepted/Rejected
  • Query capabilities: get by ID, list all, filter by status, find latest pending

Agent Run Integration:

  • agent run now creates sandbox + proposal automatically
  • All command executions logged with timestamps and status
  • Success/failure outcomes captured in proposal logs
  • User feedback includes proposal ID and sandbox path for review
  • Foundation ready for agent diff, agent commit, agent revert enhancements

Agent Diff Enhancement (Completed Oct 1, 2025):

  • Reads proposal diffs from ProposalRegistry automatically
  • Displays detailed metadata: proposal ID, status, timestamps, command count
  • Shows diff content from proposal directory
  • Displays execution log (first 50 lines, with truncation for long logs)
  • Provides next-step guidance (commit/revert/GUI review)
  • Supports --proposal-id flag to view specific proposals
  • Fallback to legacy diff behavior if no proposals found

Agent List Command (New - Oct 1, 2025):

  • Enumerates all proposals in the registry
  • Shows proposal ID, status, creation time, prompt, and stats
  • Indicates pending/accepted/rejected status for each proposal
  • Provides guidance on using agent diff to view details
  • Empty state message guides users to create proposals with agent run

Resource Catalog Updates:

  • Added agent list action with returns schema
  • Added agent diff action with arguments (--proposal-id) and returns schema
  • Updated agent resource description to include listing and diffing capabilities
  • Regenerated docs/api/z3ed-resources.yaml with new agent actions

ProposalDrawer GUI Component (Completed Oct 1, 2025):

  • ImGui right-side drawer for proposal review (AW-03)
  • Split view: proposal list (top) + detail view (bottom)
  • List view: table with ID, status, prompt columns; colored status indicators
  • Detail view: collapsible sections for metadata/diff/log; syntax-aware display
  • Action buttons: Accept, Reject, Delete with confirmation dialogs
  • Status filtering (All/Pending/Accepted/Rejected)
  • Integrated into EditorManager with Debug → Agent Proposals menu
  • Accept/Reject updates ProposalRegistry status
  • Delete removes proposal from registry and filesystem
  • TODO: Implement actual ROM merging in AcceptProposal method

Proposal Persistence Fix (Completed Oct 1, 2025):

  • Fixed ProposalRegistry to load proposals from disk on first access
  • Added LoadProposalsFromDiskLocked() method scanning proposal root directory
  • Lazy loading implementation in ListProposals() for automatic registry population
  • Reconstructs metadata from filesystem: ID, timestamps, log/diff paths, screenshots
  • Parses creation time from proposal ID format (proposal-20251001T200215-1)
  • Enables cross-session proposal tracking - agent list now finds all proposals
  • ProposalDrawer can now display proposals created via CLI agent run

CMake Build Integration:

  • Added cli/service/proposal_registry.cc and cli/service/rom_sandbox_manager.cc to all app targets
  • Fixed linker errors by including CLI service sources in:
    • yaze (main GUI app)
    • yaze_emu (emulator standalone)
    • yaze_core (testing library)
    • yaze_c (C API library)
  • All targets now build successfully with ProposalDrawer dependencies

Architecture Benefits:

  • Clean separation: RomSandboxManager (file ops) ↔ ProposalRegistry (metadata)
  • Thread-safe with mutex protection for concurrent access
  • Extensible design ready for ImGui review UI (AW-03)
  • Proposal persistence enables post-session review and auditing
  • Proposal-centric workflow enables human-in-the-loop review
  • GUI and CLI both have full access to proposal system

Next Steps for AW Workstream:

  • Test ProposalDrawer in running application
  • Complete ROM merging in AcceptProposal method
  • AW-04: Policy evaluation for gating mutations
  • AW-05: .z3ed-diff hybrid format design

Files Modified/Created

Phase 6 (Resource Catalogue):

  1. src/cli/handlers/rom.cc - Added RomInfo::Run implementation
  2. src/cli/z3ed.h - Added RomInfo class declaration
  3. src/cli/modern_cli.cc - Updated HandleRomInfoCommand routing
  4. src/cli/service/resource_catalog.cc - Added rom info schema entry
  5. docs/api/z3ed-resources.yaml - Generated comprehensive API catalog

AW-01 & AW-02 (Proposal Tracking): 6. src/cli/service/proposal_registry.h - New proposal tracking service interface 7. src/cli/service/proposal_registry.cc - Implementation with full lifecycle management 8. src/cli/handlers/agent.cc - Integrated ProposalRegistry into agent run workflow

Agent Diff & List Enhancement: 9. src/cli/handlers/agent.cc - Enhanced HandleDiffCommand with proposal reading, added HandleListCommand 10. src/cli/service/resource_catalog.cc - Added agent list/diff actions with schemas 11. docs/api/z3ed-resources.yaml - Regenerated with new agent commands 12. docs/E6-z3ed-cli-design.md - Updated Section 8.1 with list/diff documentation

AW-03 (ProposalDrawer GUI): 13. src/app/editor/system/proposal_drawer.h - Complete drawer interface with Draw/Accept/Reject/Delete 14. src/app/editor/system/proposal_drawer.cc - Full implementation (~350 lines) with list/detail views 15. src/app/editor/editor_manager.h - Added ProposalDrawer member and include 16. src/app/editor/editor_manager.cc - Added menu item and Draw() call in Update loop 17. src/CMakeLists.txt - Added proposal_drawer files to System Editor source group 18. src/app/app.cmake - Added CLI service sources to yaze target (both Apple and non-Apple builds) 19. src/app/emu/emu.cmake - Added CLI service sources to yaze_emu target 20. src/CMakeLists.txt - Added CLI service sources to yaze_core library sources 9. src/cli/z3ed.cmake - Added proposal_registry.cc to build 10. docs/E6-z3ed-implementation-plan.md - Updated progress and task statuses

Agent Diff & List (Oct 1, 2025): 21. src/cli/handlers/agent.cc - Enhanced HandleDiffCommand with proposal reading, added HandleListCommand 22. src/cli/service/resource_catalog.cc - Added agent list and diff actions to schema 23. docs/api/z3ed-resources.yaml - Regenerated with new agent commands

Proposal Persistence Fix (Oct 1, 2025): 24. src/cli/service/proposal_registry.h - Added LoadProposalsFromDiskLocked() declaration 25. src/cli/service/proposal_registry.cc - Implemented disk loading with timestamp parsing and metadata reconstruction 26. src/cli/service/proposal_registry.cc - Modified ListProposals() to lazy-load proposals from disk

ROM Merging Implementation (Oct 1, 2025): 27. src/app/editor/system/proposal_drawer.h - Added SetRom() method and Rom* member for merge operations 28. src/app/editor/system/proposal_drawer.cc - Implemented full ROM merging in AcceptProposal() with sandbox loading 29. src/app/editor/editor_manager.cc - Added SetRom(current_rom_) call before drawing ProposalDrawer 30. src/app/editor/system/proposal_drawer.cc - Added RomSandboxManager include for sandbox path resolution

6. References

Active Documentation:

  • E6-z3ed-cli-design.md - Overall CLI design and architecture
  • NEXT_PRIORITIES_OCT2.md - Current work priorities with detailed implementation guides
  • IT-01-QUICKSTART.md - Test harness quick reference
  • docs/api/z3ed-resources.yaml - Machine-readable API reference (generated)

Source Code:

  • src/cli/service/ - Core services (proposal registry, sandbox manager, resource catalog)
  • src/app/editor/system/proposal_drawer.{h,cc} - GUI review panel
  • src/app/core/imgui_test_harness_service.{h,cc} - gRPC automation server

Historical Documentation (archived):

  • archive/STATE_SUMMARY_*.md - Historical state snapshots
  • archive/IT-01-PHASE*-COMPLETE.md - Phase completion reports
  • archive/*-grpc-*.md - gRPC design decisions and technical notes
  • archive/PROGRESS_SUMMARY_*.md - Daily progress logs

Last Updated: October 2, 2025
Contributors: @scawful, GitHub Copilot
License: Same as YAZE (see ../../LICENSE)

Reference in New Issue View Git Blame Copy Permalink