scawful/yaze
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement z3ed CLI Agent Test Command and Fix Runtime Issues

Browse Source 
- Added new session summary documentation for the z3ed agent implementation on October 2, 2025, detailing achievements, infrastructure, and usage.
- Created evening session summary documenting the resolution of the ImGuiTestEngine runtime issue and preparation for E2E validation.
- Updated the E2E test harness script to reflect changes in the test commands, including menu item interactions and improved error handling.
- Modified imgui_test_harness_service.cc to implement an async test queue pattern, improving test lifecycle management and error reporting.
- Enhanced documentation for runtime fixes and testing procedures, ensuring comprehensive coverage of changes made.
This commit is contained in:
scawful
2025-10-02 09:18:16 -04:00
parent 4a4b815c1b
commit 983ef24e4d
16 changed files with 2986 additions and 292 deletions
Download Patch File Download Diff File Expand all files Collapse all files

211
docs/z3ed/README.md
View File

@@ -2,211 +2,28 @@
**Status**: Active Development
**Version**: 0.1.0-alpha
**Last Updated**: October 2, 2025 (IT-01 Complete, E2E Validation Phase)
**Last Updated**: October 2, 2025 (IT-02 Complete, E2E Validation In Progress)
## Overview
`z3ed` is a command-line interface for YAZE (Yet Another Zelda3 Editor) that enables AI-driven ROM modifications through a proposal-based workflow. It allows AI agents to suggest changes, which are then reviewed and accepted/rejected by human operators via the YAZE GUI.
z3ed is a command-line interface for YAZE that enables AI-driven ROM modifications through a proposal-based workflow.
## Documentation Index
### 🎯 Essential Documents (Start Here)
1. **[Next Priorities](NEXT_PRIORITIES_OCT2.md)** - 🚀 **CURRENT WORK** - Detailed Priority 1-3 tasks with implementation guides
2. **[Implementation Plan](E6-z3ed-implementation-plan.md)** - ⭐ **MASTER TRACKER** - Complete task backlog, progress, architecture
3. **[CLI Design](E6-z3ed-cli-design.md)** - 📐 **DESIGN DOC** - High-level vision, command structure, workflows
4. **[IT-01 Quickstart](IT-01-QUICKSTART.md)** - ⚡ **QUICK REFERENCE** - Test harness commands and examples
### Essential Documents
1. [Next Actions](NEXT_ACTIONS_OCT3.md) - Tomorrow's implementation guide
2. [Implementation Plan](E6-z3ed-implementation-plan.md) - Master tracker
3. [CLI Design](E6-z3ed-cli-design.md) - Architecture
4. [IT-01 Quickstart](IT-01-QUICKSTART.md) - Test harness reference
### <EFBFBD> Archive
Historical documentation (design decisions, phase completions, technical notes) moved to `archive/` folder for reference.
### Status Documents
- [Project Status](PROJECT_STATUS_OCT2.md) - Comprehensive overview
- [Work Summary](WORK_SUMMARY_OCT2.md) - Today's accomplishments
- [Test Validation](TEST_VALIDATION_STATUS_OCT2.md) - E2E test results
## Architecture
```
┌─────────────────────────────────────────────────────────┐
│ z3ed CLI │
│ └─ agent subcommand │
│ ├─ run <prompt> [--sandbox] │
│ ├─ list │
│ └─ test <prompt> │
└────────────────────┬────────────────────────────────────┘
│
┌────────────────────▼────────────────────────────────────┐
│ Services Layer (Singleton Services) │
│ ├─ ProposalRegistry │
│ │ ├─ CreateProposal() │
│ │ ├─ ListProposals() │
│ │ └─ LoadProposalsFromDiskLocked() │
│ ├─ RomSandboxManager │
│ │ ├─ CreateSandbox() │
│ │ └─ FindSandbox() │
│ └─ PolicyEvaluator (Planned) │
│ ├─ LoadPolicies() │
│ └─ EvaluateProposal() │
└────────────────────┬────────────────────────────────────┘
│
┌────────────────────▼────────────────────────────────────┐
│ Filesystem Layer │
│ ├─ /tmp/yaze/proposals/<id>/ │
│ │ ├─ metadata.json │
│ │ ├─ execution.log │
│ │ └─ diff.txt │
│ └─ /tmp/yaze/sandboxes/<id>/ │
│ └─ zelda3.sfc (copy) │
└────────────────────┬────────────────────────────────────┘
│
┌────────────────────▼────────────────────────────────────┐
│ YAZE GUI │
│ └─ ProposalDrawer (400px right panel) │
│ ├─ List View (proposals from registry) │
│ ├─ Detail View (metadata, diff, log) │
│ └─ AcceptProposal() → ROM merging │
└─────────────────────────────────────────────────────────┘
```
## Current Status
### ✅ Phase 6: Resource Catalogue (COMPLETE)
- **Resource Catalog System**: Comprehensive schema for all CLI commands
- **Agent Describe**: Machine-readable API catalog export (JSON/YAML)
- **API Documentation**: `docs/api/z3ed-resources.yaml` for AI/LLM consumption
### ✅ AW-01 & AW-02: Proposal Infrastructure (COMPLETE)
- **ProposalRegistry**: Disk persistence with lazy loading
- **RomSandboxManager**: Isolated ROM copies for safe testing
- **Cross-Session Tracking**: Proposals persist between CLI runs
### ✅ AW-03: ProposalDrawer GUI (COMPLETE)
- **ProposalDrawer GUI**: Split view, proposal list, detail panel
- **ROM Merging**: Sandbox-to-main ROM data copy on acceptance
- **Full Lifecycle**: Create (CLI) → Review (GUI) → Accept/Reject → Commit
### ✅ IT-01: ImGuiTestHarness (COMPLETE) 🎉
**All 3 Phases Complete**: gRPC + TestManager + ImGuiTestEngine
**Time Invested**: 11 hours total (Phase 1: 4h, Phase 2: 4h, Phase 3: 3h)
- **Phase 1** ✅: gRPC infrastructure with 6 RPC methods
- **Phase 2** ✅: TestManager integration with dynamic test registration
- **Phase 3** ✅: Full ImGuiTestEngine integration (Type/Wait/Assert RPCs)
- **Testing** ✅: E2E test script operational (`scripts/test_harness_e2e.sh`)
- **Documentation** ✅: Complete guides (QUICKSTART, PHASE3-COMPLETE)
**See**: [IT-01-QUICKSTART.md](IT-01-QUICKSTART.md) for usage examples
### ✅ IT-02: CLI Agent Test Command (IMPLEMENTATION COMPLETE) 🎉
**Implementation Complete**: Natural language → automated GUI testing
**Time Invested**: 6 hours (design + implementation + build fixes)
**Status**: Build successful, runtime issue discovered
**See**: [IMPLEMENTATION_STATUS_OCT2_PM.md](IMPLEMENTATION_STATUS_OCT2_PM.md) for complete details
**Components Completed**:
- ✅ GuiAutomationClient: gRPC wrapper for CLI usage (6 RPC methods)
- ✅ TestWorkflowGenerator: Natural language prompt parser (4 pattern types)
- ✅ `z3ed agent test`: End-to-end automation command
- ✅ Build system integration (gRPC proto generation, includes, linking)
- ✅ Conditional compilation guards for optional gRPC features
**Known Issue**:
- ImGuiTestEngine assertion failure during test cleanup
- Root cause: Synchronous test execution + immediate unregister violates engine assumptions
- Solution: Refactor to use async test queue (see status document)
### 📋 Priority 1: Fix Runtime Issue (NEXT) 🔄
**Goal**: Resolve ImGuiTestEngine test lifecycle issue
**Time Estimate**: 2-3 hours
**Status**: Ready to implement
**Approach**: Refactor RPC handlers to use asynchronous test queue instead of synchronous execution
### 📋 Priority 1: End-to-End Workflow Validation (NEXT)
**Goal**: Test complete proposal lifecycle with real GUI and widgets
**Time Estimate**: 2-3 hours
**Status**: Ready to execute - all prerequisites complete
**Tasks**:
1. Run E2E test script and validate all RPCs
2. Test proposal workflow: Create → Review → Accept/Reject
3. Test GUI automation with real YAZE widgets
4. Validate CLI agent test command with multiple prompts
5. Document edge cases and troubleshooting
**See**: [E2E_VALIDATION_GUIDE.md](E2E_VALIDATION_GUIDE.md) for detailed checklist
### 📋 Priority 3: Policy Evaluation Framework (AW-04)
**Goal**: YAML-based constraint system for gating proposal acceptance
**Time Estimate**: 6-8 hours
**Blocking**: None (can work in parallel)
### Archive
Historical documentation moved to archive/ folder.
## Quick Start
### Build and Test
```bash
# Build z3ed CLI
cmake --build build --target z3ed -j8
# Build YAZE with test harness
cmake -B build-grpc-test -DYAZE_WITH_GRPC=ON
cmake --build build-grpc-test --target yaze -j$(sysctl -n hw.ncpu)
# Run E2E tests
./scripts/test_harness_e2e.sh
```
### Create and Review Proposal
```bash
# 1. Create proposal
./build/bin/z3ed agent run "Test proposal" --sandbox
# 2. List proposals
./build/bin/z3ed agent list
# 3. Review in GUI
./build/bin/yaze.app/Contents/MacOS/yaze
# Open: Debug → Agent Proposals
```
### Test Harness Usage
```bash
# Start YAZE with test harness
./build-grpc-test/bin/yaze.app/Contents/MacOS/yaze \
--enable_test_harness \
--test_harness_port=50052 \
--rom_file=assets/zelda3.sfc &
# Test individual RPCs (see IT-01-QUICKSTART.md for full reference)
grpcurl -plaintext -import-path src/app/core/proto -proto imgui_test_harness.proto \
-d '{"message":"test"}' 127.0.0.1:50052 yaze.test.ImGuiTestHarness/Ping
```
## Key Files & Components
**Core Services**:
- `src/cli/service/proposal_registry.{h,cc}` - Proposal tracking and persistence
- `src/cli/service/rom_sandbox_manager.{h,cc}` - Isolated ROM copies
- `src/cli/service/resource_catalog.{h,cc}` - Machine-readable API specs
**GUI Integration**:
- `src/app/editor/system/proposal_drawer.{h,cc}` - Proposal review panel
- `src/app/core/imgui_test_harness_service.{h,cc}` - gRPC test automation server
**CLI Handlers**:
- `src/cli/handlers/agent.cc` - Agent subcommand (run, list, diff, describe)
- `src/cli/handlers/rom.cc` - ROM commands (info, validate, diff)
**Configuration**:
- `docs/api/z3ed-resources.yaml` - Generated API catalog for AI/LLM
- `.yaze/policies/agent.yaml` - (Planned) Policy rules
## Development Guidelines
See `docs/B1-contributing.md` for general guidelines.
**z3ed-Specific**:
- Use singleton pattern for services (`Instance()` accessor)
- Return `absl::Status` or `absl::StatusOr<T>` for error handling
- Update `NEXT_PRIORITIES_OCT2.md` when starting new work
- Update `E6-z3ed-implementation-plan.md` task backlog when completing tasks
---
**Questions?** Open an issue or see implementation plan for detailed architecture.
See NEXT_ACTIONS_OCT3.md for detailed next steps.