Implement z3ed CLI Agent Test Command and Fix Runtime Issues
- Added new session summary documentation for the z3ed agent implementation on October 2, 2025, detailing achievements, infrastructure, and usage. - Created evening session summary documenting the resolution of the ImGuiTestEngine runtime issue and preparation for E2E validation. - Updated the E2E test harness script to reflect changes in the test commands, including menu item interactions and improved error handling. - Modified imgui_test_harness_service.cc to implement an async test queue pattern, improving test lifecycle management and error reporting. - Enhanced documentation for runtime fixes and testing procedures, ensuring comprehensive coverage of changes made.
This commit is contained in:
scawful
206
docs/z3ed/TEST_VALIDATION_STATUS_OCT2.md
Normal file
206
docs/z3ed/TEST_VALIDATION_STATUS_OCT2.md
Normal file
@@ -0,0 +1,206 @@
|
||||
# Test Validation Status - October 2, 2025
|
||||
|
||||
**Time**: 9:30 PM
|
||||
**Status**: E2E Tests Running | Menu Interaction Verified | Window Detection Issue Identified
|
||||
|
||||
## Current Test Results
|
||||
|
||||
### Working ✅
|
||||
1. **Ping RPC** - Health check fully operational
|
||||
2. **Menu Item Clicks** - Successfully clicking menu items via gRPC
|
||||
- Example: `menuitem: Overworld Editor` → clicked successfully
|
||||
- Example: `menuitem: Dungeon Editor` → clicked successfully
|
||||
|
||||
### Issues Identified 🔍
|
||||
|
||||
#### Issue 1: Window Detection After Menu Click
|
||||
**Problem**: Menu items are clicked successfully, but subsequent window visibility checks fail
|
||||
|
||||
**Observed Behavior**:
|
||||
```
|
||||
Test 2: Click (Open Overworld Editor)
|
||||
✓ Clicked menuitem ' Overworld Editor' (1873ms)
|
||||
|
||||
Test 3: Wait (Overworld Editor Window)
|
||||
✗ Condition 'window_visible:Overworld Editor' not met after 5000ms timeout
|
||||
```
|
||||
|
||||
**Root Cause Analysis**:
|
||||
1. Menu items call `editor.set_active(true)`
|
||||
2. This sets a flag but doesn't immediately create ImGui window
|
||||
3. Window creation happens in next frame's `Update()` call
|
||||
4. ImGuiTestEngine's `WindowInfo()` API may not see newly created windows immediately
|
||||
5. Window title may include ICON_MD prefix: `ICON_MD_LAYERS " Overworld Editor"`
|
||||
|
||||
**Potential Solutions**:
|
||||
- A. Use longer wait time (current: 5s)
|
||||
- B. Check for window with icon prefix: `window_visible: Overworld Editor`
|
||||
- C. Use different condition type (element_visible vs window_visible)
|
||||
- D. Add frame yield between menu click and window check
|
||||
|
||||
#### Issue 2: Screenshot RPC Proto Mismatch
|
||||
**Problem**: Screenshot request proto schema doesn't match client usage
|
||||
|
||||
**Error Message**:
|
||||
```
|
||||
message type yaze.test.ScreenshotRequest has no known field named region
|
||||
```
|
||||
|
||||
**Solution**: Update proto or skip for now (non-blocking for core functionality)
|
||||
|
||||
## Next Steps (Priority Order)
|
||||
|
||||
### 1. Debug Window Detection (30 min)
|
||||
**Goal**: Understand why windows aren't detected after menu clicks
|
||||
|
||||
**Tasks**:
|
||||
- [ ] Check actual window titles in YAZE (with icons)
|
||||
- [ ] Test with exact window name including icon
|
||||
- [ ] Add diagnostic logging to Wait RPC
|
||||
- [ ] Try element_visible condition instead
|
||||
- [ ] Increase wait timeout to 10s
|
||||
|
||||
**Test Command**:
|
||||
```bash
|
||||
# Terminal 1: Start YAZE
|
||||
./build-grpc-test/bin/yaze.app/Contents/MacOS/yaze \
|
||||
--enable_test_harness \
|
||||
--test_harness_port=50052 \
|
||||
--rom_file=assets/zelda3.sfc &
|
||||
|
||||
# Terminal 2: Manual test sequence
|
||||
sleep 5 # Let YAZE fully initialize
|
||||
|
||||
# Click menu item
|
||||
grpcurl -plaintext -import-path src/app/core/proto -proto imgui_test_harness.proto \
|
||||
-d '{"target":"menuitem: Overworld Editor","type":"LEFT"}' \
|
||||
127.0.0.1:50052 yaze.test.ImGuiTestHarness/Click
|
||||
|
||||
# Wait a few frames
|
||||
sleep 2
|
||||
|
||||
# Try different window name variations
|
||||
grpcurl -plaintext -import-path src/app/core/proto -proto imgui_test_harness.proto \
|
||||
-d '{"condition":"window_visible:Overworld Editor","timeout_ms":10000}' \
|
||||
127.0.0.1:50052 yaze.test.ImGuiTestHarness/Wait
|
||||
|
||||
# Or with icon
|
||||
grpcurl -plaintext -import-path src/app/core/proto -proto imgui_test_harness.proto \
|
||||
-d '{"condition":"window_visible: Overworld Editor","timeout_ms":10000}' \
|
||||
127.0.0.1:50052 yaze.test.ImGuiTestHarness/Wait
|
||||
```
|
||||
|
||||
### 2. Fix Window Name Matching (1 hour)
|
||||
**Options**:
|
||||
|
||||
**Option A: Strip Icons from Target Names**
|
||||
```cpp
|
||||
// In Wait RPC handler
|
||||
std::string CleanWindowName(const std::string& name) {
|
||||
// Strip ICON_MD_ prefixes and leading spaces
|
||||
// " Overworld Editor" → "Overworld Editor"
|
||||
return absl::StripAsciiWhitespace(name);
|
||||
}
|
||||
```
|
||||
|
||||
**Option B: Use Partial Name Matching**
|
||||
```cpp
|
||||
// Check if window name contains target (case-insensitive)
|
||||
bool window_found = false;
|
||||
for (ImGuiWindow* window : ImGui::GetCurrentContext()->Windows) {
|
||||
if (absl::StrContains(absl::AsciiStrToLower(window->Name),
|
||||
absl::AsciiStrToLower(target))) {
|
||||
window_found = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Option C: Add Frame Yield**
|
||||
```cpp
|
||||
// In Click RPC, after successful click:
|
||||
// Yield control back to ImGui to process one frame
|
||||
ImGuiTestEngine_Yield(engine);
|
||||
// Or sleep briefly
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(500));
|
||||
```
|
||||
|
||||
### 3. Update E2E Test Script (15 min)
|
||||
Once window detection works, update test script:
|
||||
```bash
|
||||
# Use working window names
|
||||
run_test "Wait (Overworld Editor)" "Wait" \
|
||||
'{"condition":"window_visible:Overworld Editor","timeout_ms":10000,"poll_interval_ms":100}'
|
||||
|
||||
# Add delay between click and wait
|
||||
echo "Waiting for window to appear..."
|
||||
sleep 2
|
||||
```
|
||||
|
||||
### 4. Document Widget Naming Convention (30 min)
|
||||
Create guide for test writers:
|
||||
|
||||
**Widget Naming Patterns**:
|
||||
- Menu items: `menuitem:<Name>` (with or without icon prefix)
|
||||
- Buttons: `button:<Label>`
|
||||
- Windows: `window:<Title>` (may include icons)
|
||||
- Input fields: `input:<Label>`
|
||||
- Tabs: `tab:<Name>`
|
||||
|
||||
**Best Practices**:
|
||||
- Use partial matching for windows (handles icon prefixes)
|
||||
- Add 1-2s delay after menu clicks before checking windows
|
||||
- Use 10s+ timeouts for initial window creation
|
||||
- Use shorter timeouts (2-5s) for interactions within open windows
|
||||
|
||||
## Test Coverage Status
|
||||
|
||||
| RPC Method | Status | Notes |
|
||||
|------------|--------|-------|
|
||||
| Ping | ✅ Working | Health check operational |
|
||||
| Click | ⚠️ Partial | Menu items work, window detection needs fix |
|
||||
| Type | 📋 Pending | Depends on window detection fix |
|
||||
| Wait | ⚠️ Partial | Polling works, condition matching needs fix |
|
||||
| Assert | ⚠️ Partial | Same as Wait - condition matching issue |
|
||||
| Screenshot | 🔧 Blocked | Proto mismatch - non-critical |
|
||||
|
||||
## Time Investment Today
|
||||
|
||||
- Build system fixes: 1h
|
||||
- Type conversion debugging: 0.5h
|
||||
- Test script updates: 0.5h
|
||||
- Widget investigation: 1h
|
||||
- **Total**: 3 hours
|
||||
|
||||
## Estimated Remaining Work
|
||||
|
||||
- Debug window detection: 0.5h
|
||||
- Fix window matching logic: 1h
|
||||
- Update tests and validate: 0.5h
|
||||
- Document findings: 0.5h
|
||||
- **Total**: 2.5 hours
|
||||
|
||||
**Target Completion**: October 3, 2025 (tomorrow morning)
|
||||
|
||||
## Success Criteria for Validation Complete
|
||||
|
||||
- [ ] All 5 main RPCs working (Ping, Click, Wait, Assert, Type)
|
||||
- [ ] Can open editor windows via menu clicks
|
||||
- [ ] Can detect window visibility after opening
|
||||
- [ ] Can assert on window state
|
||||
- [ ] E2E test script passes all tests except Screenshot
|
||||
- [ ] Documentation updated with real examples
|
||||
- [ ] Widget naming conventions documented
|
||||
|
||||
## Next Phase After Validation
|
||||
|
||||
Once E2E validation complete:
|
||||
→ **Priority 3: Policy Evaluation Framework (AW-04)**
|
||||
- 6-8 hours estimated
|
||||
- Can work in parallel with validation improvements
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: October 2, 2025, 9:30 PM
|
||||
**Author**: GitHub Copilot (with @scawful)
|
||||
**Status**: In progress - window detection debugging needed
|
||||
Reference in New Issue
Block a user