scawful/yaze
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement z3ed CLI Agent Test Command and Fix Runtime Issues

Browse Source 
- Added new session summary documentation for the z3ed agent implementation on October 2, 2025, detailing achievements, infrastructure, and usage.
- Created evening session summary documenting the resolution of the ImGuiTestEngine runtime issue and preparation for E2E validation.
- Updated the E2E test harness script to reflect changes in the test commands, including menu item interactions and improved error handling.
- Modified imgui_test_harness_service.cc to implement an async test queue pattern, improving test lifecycle management and error reporting.
- Enhanced documentation for runtime fixes and testing procedures, ensuring comprehensive coverage of changes made.
This commit is contained in:
scawful
2025-10-02 09:18:16 -04:00
parent 4a4b815c1b
commit 983ef24e4d
16 changed files with 2986 additions and 292 deletions
Download Patch File Download Diff File Expand all files Collapse all files

88
docs/z3ed/E6-z3ed-implementation-plan.md
View File

@@ -1,9 +1,9 @@
# z3ed Agentic Workflow Implementation Plan
**Last Updated**: October 2, 2025
**Status**: IT-01 Complete ✅ | AW-03 Complete ✅ | E2E Validation Phase
**Last Updated**: October 2, 2025 (10:30 PM)
**Status**: IT-01 Complete ✅ | IT-02 Complete ✅ | E2E Validation Ready 🎯
> 📋 **See Also**: [NEXT_PRIORITIES_OCT2.md](NEXT_PRIORITIES_OCT2.md) for detailed implementation guides for current priorities.
> 📋 **Quick Start**: See [README.md](README.md) for essential links and [NEXT_PRIORITIES_OCT2.md](NEXT_PRIORITIES_OCT2.md) for detailed task guides.
## Executive Summary
@@ -13,13 +13,28 @@ The z3ed CLI and AI agent workflow system has completed major infrastructure mil
- **Phase 6**: Resource Catalogue - Machine-readable API specs for AI consumption
- **AW-01/02/03**: Acceptance Workflow - Proposal tracking, sandbox management, GUI review with ROM merging
- **IT-01**: ImGuiTestHarness - Full GUI automation via gRPC + ImGuiTestEngine (all 3 phases complete)
- **IT-02**: CLI Agent Test - Natural language → automated GUI testing (implementation complete)
**🔄 Active Phase**:
- **Priority 1**: End-to-End Workflow Validation - Test complete proposal lifecycle with real GUI
- **E2E Validation**: Testing complete proposal lifecycle with real GUI widgets (window detection debugging in progress)
**📋 Next Phases**:
- **Priority 2**: CLI Agent Test Command (IT-02) - Natural language → automated GUI testing
- **Priority 3**: Policy Evaluation Framework (AW-04) - YAML-based constraints for proposal acceptance
- **Priority 1**: Complete E2E Validation - Fix window detection after menu actions (2-3 hours)
- **Priority 2**: Policy Evaluation Framework (AW-04) - YAML-based constraints for proposal acceptance (6-8 hours)
**Recent Accomplishments** (October 2, 2025):
- IT-02 implementation complete with async test queue pattern
- Build system fixes for z3ed target (gRPC integration)
- Documentation consolidated into clean structure
- E2E test script operational (5/6 RPCs working)
- Menu interaction verified via ImGuiTestEngine
**Known Issues**:
- Window detection timing after menu clicks needs refinement
- Screenshot RPC proto mismatch (non-critical)
**Time Investment**: 20.5 hours total (IT-01: 11h, IT-02: 7.5h, Docs: 2h)
**Code Quality**: All targets compile cleanly, no crashes, partial test coverage
## Quick Reference
@@ -51,12 +66,18 @@ The z3ed CLI and AI agent workflow system has completed major infrastructure mil
## 1. Current Priorities (Week of Oct 2-8, 2025)
**Status**: Phase 1 Complete ✅ | Phase 2 Complete ✅ | Phase 3 Complete ✅
**Status**: IT-01 Complete ✅ | IT-02 Complete ✅ | E2E Tests Running ⚡
### Priority 1: End-to-End Workflow Validation (ACTIVE) 🔄
**Goal**: Validate complete AI agent workflow from proposal creation to ROM commit
**Time Estimate**: 2-3 hours
**Status**: Ready to execute
### Priority 0: E2E Test Validation (IMMEDIATE) 🎯
**Goal**: Validate test harness with real YAZE widgets
**Time Estimate**: 30-60 minutes
**Status**: Test script running, needs real widget names
**Current Results**:
- ✅ Ping RPC working
- ⚠️ Tests 2-5 using fake widget names
- 📋 Need to identify real widget names from YAZE source
- 🔧 Screenshot RPC needs proto fix
**Task Checklist**:
1. ✅ **E2E Test Script**: Already created (`scripts/test_harness_e2e.sh`)
@@ -162,26 +183,33 @@ This plan decomposes the design additions into actionable engineering tasks. Eac
| ID | Task | Workstream | Type | Status | Dependencies |
|----|------|------------|------|--------|--------------|
| RC-01 | Define schema for `ResourceCatalog` entries and implement serialization helpers. | Resource Catalogue | Code | Done | Schema system complete with all resource types documented |
| RC-02 | Auto-generate `docs/api/z3ed-resources.yaml` from command annotations. | Resource Catalogue | Tooling | Done | Generated and committed to docs/api/ |
| RC-03 | Implement `z3ed agent describe` CLI surface returning JSON schemas. | Resource Catalogue | Code | Done | Both YAML and JSON output formats working |
| RC-04 | Integrate schema export with TUI command palette + help overlays. | Resource Catalogue | UX | Planned | RC-03 |
| RC-05 | Harden CLI command routing/flag parsing to unblock agent automation. | Resource Catalogue | Code | Done | Fixed rom info handler to use FLAGS_rom |
| AW-01 | Implement sandbox ROM cloning and tracking (`RomSandboxManager`). | Acceptance Workflow | Code | Done | ROM sandbox manager operational with lifecycle management |
| AW-02 | Build proposal registry service storing diffs, logs, screenshots. | Acceptance Workflow | Code | Done | ProposalRegistry implemented with disk persistence |
| AW-03 | Add ImGui drawer for proposals with accept/reject controls. | Acceptance Workflow | UX | Done | ProposalDrawer GUI complete with ROM merging |
| AW-04 | Implement policy evaluation for gating accept buttons. | Acceptance Workflow | Code | In Progress | AW-03, Priority 2 - YAML policies + PolicyEvaluator |
| AW-05 | Draft `.z3ed-diff` hybrid schema (binary deltas + JSON metadata). | Acceptance Workflow | Design | Planned | AW-01 |
| IT-01 | Create `ImGuiTestHarness` IPC service embedded in `yaze_test`. | ImGuiTest Bridge | Code | Done | ✅ Phase 1+2+3 Complete - Full GUI automation with gRPC + ImGuiTestEngine |
| IT-02 | Implement CLI agent step translation (`imgui_action` → harness call). | ImGuiTest Bridge | Code | In Progress | IT-01, `z3ed agent test` command with natural language prompts |
| IT-03 | Provide synchronization primitives (`WaitForIdle`, etc.). | ImGuiTest Bridge | Code | Done | ✅ Wait RPC with condition polling already implemented in IT-01 Phase 3 |
| VP-01 | Expand CLI unit tests for new commands and sandbox flow. | Verification Pipeline | Test | Planned | RC/AW tasks |
| VP-02 | Add harness integration tests with replay scripts. | Verification Pipeline | Test | Planned | IT tasks |
| VP-03 | Create CI job running agent smoke tests with `YAZE_WITH_JSON`. | Verification Pipeline | Infra | Planned | VP-01, VP-02 |
| TL-01 | Capture accept/reject metadata and push to telemetry log. | Telemetry & Learning | Code | Planned | AW tasks |
| TL-02 | Build anonymized metrics exporter + opt-in toggle. | Telemetry & Learning | Infra | Planned | TL-01 |
| RC-01 | Define schema for `ResourceCatalog` entries and implement serialization helpers. | Resource Catalogue | Code | ✅ Done | Schema system complete with all resource types documented |
| RC-02 | Auto-generate `docs/api/z3ed-resources.yaml` from command annotations. | Resource Catalogue | Tooling | ✅ Done | Generated and committed to docs/api/ |
| RC-03 | Implement `z3ed agent describe` CLI surface returning JSON schemas. | Resource Catalogue | Code | ✅ Done | Both YAML and JSON output formats working |
| RC-04 | Integrate schema export with TUI command palette + help overlays. | Resource Catalogue | UX | 📋 Planned | RC-03 |
| RC-05 | Harden CLI command routing/flag parsing to unblock agent automation. | Resource Catalogue | Code | ✅ Done | Fixed rom info handler to use FLAGS_rom |
| AW-01 | Implement sandbox ROM cloning and tracking (`RomSandboxManager`). | Acceptance Workflow | Code | ✅ Done | ROM sandbox manager operational with lifecycle management |
| AW-02 | Build proposal registry service storing diffs, logs, screenshots. | Acceptance Workflow | Code | ✅ Done | ProposalRegistry implemented with disk persistence |
| AW-03 | Add ImGui drawer for proposals with accept/reject controls. | Acceptance Workflow | UX | ✅ Done | ProposalDrawer GUI complete with ROM merging |
| AW-04 | Implement policy evaluation for gating accept buttons. | Acceptance Workflow | Code | 📋 Next | AW-03, Priority 1 - YAML policies + PolicyEvaluator (6-8 hours) |
| AW-05 | Draft `.z3ed-diff` hybrid schema (binary deltas + JSON metadata). | Acceptance Workflow | Design | 📋 Planned | AW-01 |
| IT-01 | Create `ImGuiTestHarness` IPC service embedded in `yaze_test`. | ImGuiTest Bridge | Code | ✅ Done | Phase 1+2+3 Complete - Full GUI automation with gRPC + ImGuiTestEngine (11 hours) |
| IT-02 | Implement CLI agent step translation (`imgui_action` → harness call). | ImGuiTest Bridge | Code | ✅ Done | `z3ed agent test` command with natural language prompts (7.5 hours) |
| IT-03 | Provide synchronization primitives (`WaitForIdle`, etc.). | ImGuiTest Bridge | Code | ✅ Done | Wait RPC with condition polling already implemented in IT-01 Phase 3 |
| IT-04 | Complete E2E validation with real YAZE widgets | ImGuiTest Bridge | Test | 🔄 Active | IT-02, Fix window detection after menu actions (2-3 hours) |
| VP-01 | Expand CLI unit tests for new commands and sandbox flow. | Verification Pipeline | Test | 📋 Planned | RC/AW tasks |
| VP-02 | Add harness integration tests with replay scripts. | Verification Pipeline | Test | 📋 Planned | IT tasks |
| VP-03 | Create CI job running agent smoke tests with `YAZE_WITH_JSON`. | Verification Pipeline | Infra | 📋 Planned | VP-01, VP-02 |
| TL-01 | Capture accept/reject metadata and push to telemetry log. | Telemetry & Learning | Code | 📋 Planned | AW tasks |
| TL-02 | Build anonymized metrics exporter + opt-in toggle. | Telemetry & Learning | Infra | 📋 Planned | TL-01 |
_Status Legend: Prototype · In Progress · Planned · Blocked · Done_
_Status Legend: 🔄 Active · 📋 Planned · ✅ Done_
**Progress Summary**:
- ✅ Completed: 11 tasks (61%)
- 🔄 Active: 1 task (6%)
- 📋 Planned: 6 tasks (33%)
- **Total**: 18 tasks
## 3. Immediate Next Steps (Week of Oct 1-7, 2025)