feat: Add GUI automation client and test workflow generator
- Implemented GuiAutomationClient for gRPC communication with the test harness. - Added methods for various GUI actions: Click, Type, Wait, Assert, and Screenshot. - Created TestWorkflowGenerator to convert natural language prompts into structured test workflows. - Enhanced HandleTestCommand to support new command-line arguments for GUI automation. - Updated CMakeLists.txt to include new source files for GUI automation and workflow generation.
This commit is contained in:
scawful
344
docs/z3ed/AGENT_TEST_QUICKREF.md
Normal file
344
docs/z3ed/AGENT_TEST_QUICKREF.md
Normal file
@@ -0,0 +1,344 @@
|
||||
# z3ed Agent Test Command - Quick Reference
|
||||
|
||||
**Last Updated**: October 2, 2025
|
||||
**Feature**: IT-02 CLI Agent Test Command
|
||||
|
||||
---
|
||||
|
||||
## Command Syntax
|
||||
|
||||
```bash
|
||||
z3ed agent test --prompt "<natural_language_prompt>" \
|
||||
[--host <hostname>] \
|
||||
[--port <port>] \
|
||||
[--timeout <seconds>]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Supported Prompts
|
||||
|
||||
### 1. Open Editor
|
||||
**Pattern**: "Open <Editor> editor"
|
||||
**Example**: `"Open Overworld editor"`
|
||||
**Actions**:
|
||||
- Click button → Wait for window
|
||||
|
||||
```bash
|
||||
z3ed agent test --prompt "Open Overworld editor"
|
||||
z3ed agent test --prompt "Open Dungeon editor"
|
||||
z3ed agent test --prompt "Open Sprite editor"
|
||||
```
|
||||
|
||||
### 2. Open and Verify
|
||||
**Pattern**: "Open <Editor> and verify it loads"
|
||||
**Example**: `"Open Dungeon editor and verify it loads"`
|
||||
**Actions**:
|
||||
- Click button → Wait for window → Assert visible
|
||||
|
||||
```bash
|
||||
z3ed agent test --prompt "Open Overworld editor and verify it loads"
|
||||
z3ed agent test --prompt "Open Dungeon editor and verify it loads"
|
||||
```
|
||||
|
||||
### 3. Click Button
|
||||
**Pattern**: "Click <Button>"
|
||||
**Example**: `"Click Open ROM button"`
|
||||
**Actions**:
|
||||
- Single click action
|
||||
|
||||
```bash
|
||||
z3ed agent test --prompt "Click Open ROM button"
|
||||
z3ed agent test --prompt "Click Save button"
|
||||
z3ed agent test --prompt "Click Overworld"
|
||||
```
|
||||
|
||||
### 4. Type Input
|
||||
**Pattern**: "Type '<text>' in <input>"
|
||||
**Example**: `"Type 'zelda3.sfc' in filename input"`
|
||||
**Actions**:
|
||||
- Click input → Type text (with clear_first)
|
||||
|
||||
```bash
|
||||
z3ed agent test --prompt "Type 'zelda3.sfc' in filename input"
|
||||
z3ed agent test --prompt "Type 'test' in search"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### 1. Build with gRPC
|
||||
```bash
|
||||
cmake -B build-grpc-test -DYAZE_WITH_GRPC=ON
|
||||
cmake --build build-grpc-test --target yaze -j$(sysctl -n hw.ncpu)
|
||||
cmake --build build-grpc-test --target z3ed -j$(sysctl -n hw.ncpu)
|
||||
```
|
||||
|
||||
### 2. Start YAZE Test Harness
|
||||
```bash
|
||||
./build-grpc-test/bin/yaze.app/Contents/MacOS/yaze \
|
||||
--enable_test_harness \
|
||||
--test_harness_port=50052 \
|
||||
--rom_file=assets/zelda3.sfc &
|
||||
```
|
||||
|
||||
### 3. Verify Connection
|
||||
```bash
|
||||
# Check if server is running
|
||||
lsof -i :50052
|
||||
|
||||
# Quick health check
|
||||
grpcurl -plaintext -import-path src/app/core/proto \
|
||||
-proto imgui_test_harness.proto \
|
||||
-d '{"message":"test"}' \
|
||||
127.0.0.1:50052 yaze.test.ImGuiTestHarness/Ping
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example Workflows
|
||||
|
||||
### Full Overworld Editor Test
|
||||
```bash
|
||||
# 1. Start test harness (if not running)
|
||||
./build-grpc-test/bin/yaze.app/Contents/MacOS/yaze \
|
||||
--enable_test_harness \
|
||||
--test_harness_port=50052 \
|
||||
--rom_file=assets/zelda3.sfc &
|
||||
|
||||
# 2. Wait for startup
|
||||
sleep 3
|
||||
|
||||
# 3. Run test
|
||||
./build-grpc-test/bin/z3ed agent test \
|
||||
--prompt "Open Overworld editor and verify it loads"
|
||||
|
||||
# Expected output:
|
||||
# === GUI Automation Test ===
|
||||
# Prompt: Open Overworld editor and verify it loads
|
||||
# Server: localhost:50052
|
||||
#
|
||||
# Generated workflow:
|
||||
# Workflow: Open and verify Overworld Editor
|
||||
# 1. Click(button:Overworld)
|
||||
# 2. Wait(window_visible:Overworld Editor, 5000ms)
|
||||
# 3. Assert(visible:Overworld Editor)
|
||||
#
|
||||
# ✓ Connected to test harness
|
||||
#
|
||||
# [1/3] Click(button:Overworld) ... ✓ (125ms)
|
||||
# [2/3] Wait(window_visible:Overworld Editor, 5000ms) ... ✓ (1250ms)
|
||||
# [3/3] Assert(visible:Overworld Editor) ... ✓ (50ms)
|
||||
#
|
||||
# ✅ Test passed in 1425ms
|
||||
```
|
||||
|
||||
### Custom Server Configuration
|
||||
```bash
|
||||
# Connect to remote test harness
|
||||
./build-grpc-test/bin/z3ed agent test \
|
||||
--prompt "Open Dungeon editor" \
|
||||
--host 192.168.1.100 \
|
||||
--port 50053 \
|
||||
--timeout 60
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Messages
|
||||
|
||||
### Connection Error
|
||||
```
|
||||
Failed to connect to test harness at localhost:50052
|
||||
Make sure YAZE is running with:
|
||||
./yaze --enable_test_harness --test_harness_port=50052 --rom_file=<rom>
|
||||
|
||||
Error: Connection refused
|
||||
```
|
||||
|
||||
**Solution**: Start YAZE with test harness enabled
|
||||
|
||||
### Unsupported Prompt
|
||||
```
|
||||
Unable to parse prompt: "Do something complex"
|
||||
|
||||
Supported patterns:
|
||||
- Open <Editor> editor
|
||||
- Open <Editor> and verify it loads
|
||||
- Type '<text>' in <input>
|
||||
- Click <button>
|
||||
|
||||
Examples:
|
||||
- Open Overworld editor
|
||||
- Open Dungeon editor and verify it loads
|
||||
- Type 'zelda3.sfc' in filename input
|
||||
- Click Open ROM button
|
||||
```
|
||||
|
||||
**Solution**: Use one of the supported prompt patterns
|
||||
|
||||
### Widget Not Found
|
||||
```
|
||||
[1/2] Click(button:NonExistent) ... ✗ FAILED
|
||||
Error: Button 'NonExistent' not found
|
||||
|
||||
Step 1 failed: Button 'NonExistent' not found
|
||||
```
|
||||
|
||||
**Solution**:
|
||||
- Verify widget exists in YAZE
|
||||
- Check spelling (case-sensitive)
|
||||
- Use exact label from GUI
|
||||
|
||||
### Timeout Error
|
||||
```
|
||||
[2/2] Wait(window_visible:Slow Editor, 5000ms) ... ✗ FAILED
|
||||
Error: Condition not met after 5000 ms
|
||||
|
||||
Step 2 failed: Condition not met after 5000 ms
|
||||
```
|
||||
|
||||
**Solution**:
|
||||
- Increase timeout: `--timeout 10`
|
||||
- Verify window actually opens
|
||||
- Check for errors in YAZE
|
||||
|
||||
---
|
||||
|
||||
## Exit Codes
|
||||
|
||||
- `0` - Success (all steps passed)
|
||||
- `1` - Failure (connection, parsing, or execution error)
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Port Already in Use
|
||||
```bash
|
||||
# Kill existing instances
|
||||
killall yaze
|
||||
|
||||
# Wait for cleanup
|
||||
sleep 2
|
||||
|
||||
# Use different port
|
||||
./yaze --enable_test_harness --test_harness_port=50053 ...
|
||||
./z3ed agent test --port 50053 ...
|
||||
```
|
||||
|
||||
### gRPC Not Available
|
||||
```
|
||||
GUI automation requires YAZE_WITH_GRPC=ON at build time.
|
||||
Rebuild with: cmake -B build -DYAZE_WITH_GRPC=ON
|
||||
```
|
||||
|
||||
**Solution**: Rebuild with gRPC support enabled
|
||||
|
||||
### Widget Names Unknown
|
||||
```bash
|
||||
# Manual exploration with grpcurl
|
||||
grpcurl -plaintext -import-path src/app/core/proto \
|
||||
-proto imgui_test_harness.proto \
|
||||
-d '{"condition":"visible:Main Window"}' \
|
||||
127.0.0.1:50052 yaze.test.ImGuiTestHarness/Assert
|
||||
|
||||
# Try different widget names until you find the right one
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Shell Script Integration
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Start YAZE
|
||||
./yaze --enable_test_harness --rom_file=zelda3.sfc &
|
||||
YAZE_PID=$!
|
||||
sleep 3
|
||||
|
||||
# Run tests
|
||||
./z3ed agent test --prompt "Open Overworld editor" || exit 1
|
||||
./z3ed agent test --prompt "Open Dungeon editor" || exit 1
|
||||
|
||||
# Cleanup
|
||||
kill $YAZE_PID
|
||||
```
|
||||
|
||||
### CI/CD Pipeline
|
||||
```yaml
|
||||
# .github/workflows/gui-tests.yml
|
||||
- name: Start YAZE Test Harness
|
||||
run: |
|
||||
./yaze --enable_test_harness --rom_file=zelda3.sfc &
|
||||
sleep 5
|
||||
|
||||
- name: Run GUI Tests
|
||||
run: |
|
||||
./z3ed agent test --prompt "Open Overworld editor"
|
||||
./z3ed agent test --prompt "Open Dungeon editor"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Typical Timings
|
||||
- **Click**: 50-200ms
|
||||
- **Type**: 100-300ms
|
||||
- **Wait**: 100-5000ms (depends on condition)
|
||||
- **Assert**: 10-100ms
|
||||
|
||||
### Total Test Duration
|
||||
- Simple click: ~100ms
|
||||
- Open editor: ~1-2s
|
||||
- Open + verify: ~1.5-2.5s
|
||||
- Complex workflow: ~3-5s
|
||||
|
||||
---
|
||||
|
||||
## Extending Functionality
|
||||
|
||||
### Add New Pattern Type
|
||||
|
||||
1. **Add pattern matcher** (`test_workflow_generator.h`):
|
||||
```cpp
|
||||
bool MatchesYourPattern(const std::string& prompt, ...);
|
||||
```
|
||||
|
||||
2. **Add workflow builder** (`test_workflow_generator.cc`):
|
||||
```cpp
|
||||
TestWorkflow BuildYourPatternWorkflow(...);
|
||||
```
|
||||
|
||||
3. **Add to GenerateWorkflow()** (`test_workflow_generator.cc`):
|
||||
```cpp
|
||||
if (MatchesYourPattern(prompt, ¶ms)) {
|
||||
return BuildYourPatternWorkflow(params);
|
||||
}
|
||||
```
|
||||
|
||||
### Add New Widget Type
|
||||
|
||||
Currently supported: `button:`, `input:`, `window:`
|
||||
|
||||
To add more, extend the target format in RPC calls.
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- **Full Documentation**: [IT-01-QUICKSTART.md](IT-01-QUICKSTART.md)
|
||||
- **E2E Validation**: [E2E_VALIDATION_GUIDE.md](E2E_VALIDATION_GUIDE.md)
|
||||
- **Implementation Details**: [IMPLEMENTATION_PROGRESS_OCT2.md](IMPLEMENTATION_PROGRESS_OCT2.md)
|
||||
- **Architecture Overview**: [E6-z3ed-implementation-plan.md](E6-z3ed-implementation-plan.md)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: October 2, 2025
|
||||
**Version**: IT-02 Complete
|
||||
**Status**: Ready for validation
|
||||
Reference in New Issue
Block a user