yaze/docs/z3ed/AGENT_TEST_QUICKREF.md

z3ed Agent Test Command - Quick Reference

Last Updated: October 2, 2025
Feature: IT-02 CLI Agent Test Command

---

Command Syntax

z3ed agent test --prompt "<natural_language_prompt>" \
  [--host <hostname>] \
  [--port <port>] \
  [--timeout <seconds>]

---

Supported Prompts

1. Open Editor

Pattern: "Open editor"
Example: "Open Overworld editor"
Actions:

• Click button → Wait for window

z3ed agent test --prompt "Open Overworld editor"
z3ed agent test --prompt "Open Dungeon editor"
z3ed agent test --prompt "Open Sprite editor"

2. Open and Verify

Pattern: "Open and verify it loads"
Example: "Open Dungeon editor and verify it loads"
Actions:

• Click button → Wait for window → Assert visible

z3ed agent test --prompt "Open Overworld editor and verify it loads"
z3ed agent test --prompt "Open Dungeon editor and verify it loads"

3. Click Button

Pattern: "Click "
Example: "Click Open ROM button"
Actions:

• Single click action

z3ed agent test --prompt "Click Open ROM button"
z3ed agent test --prompt "Click Save button"
z3ed agent test --prompt "Click Overworld"

4. Type Input

Pattern: "Type '' in "
Example: "Type 'zelda3.sfc' in filename input"
Actions:

• Click input → Type text (with clear_first)

z3ed agent test --prompt "Type 'zelda3.sfc' in filename input"
z3ed agent test --prompt "Type 'test' in search"

---

Prerequisites

1. Build with gRPC

cmake -B build-grpc-test -DYAZE_WITH_GRPC=ON
cmake --build build-grpc-test --target yaze -j$(sysctl -n hw.ncpu)
cmake --build build-grpc-test --target z3ed -j$(sysctl -n hw.ncpu)

2. Start YAZE Test Harness

./build-grpc-test/bin/yaze.app/Contents/MacOS/yaze \
  --enable_test_harness \
  --test_harness_port=50052 \
  --rom_file=assets/zelda3.sfc &

3. Verify Connection

# Check if server is running
lsof -i :50052

# Quick health check
grpcurl -plaintext -import-path src/app/core/proto \
  -proto imgui_test_harness.proto \
  -d '{"message":"test"}' \
  127.0.0.1:50052 yaze.test.ImGuiTestHarness/Ping

---

Example Workflows

Full Overworld Editor Test

# 1. Start test harness (if not running)
./build-grpc-test/bin/yaze.app/Contents/MacOS/yaze \
  --enable_test_harness \
  --test_harness_port=50052 \
  --rom_file=assets/zelda3.sfc &

# 2. Wait for startup
sleep 3

# 3. Run test
./build-grpc-test/bin/z3ed agent test \
  --prompt "Open Overworld editor and verify it loads"

# Expected output:
# === GUI Automation Test ===
# Prompt: Open Overworld editor and verify it loads
# Server: localhost:50052
#
# Generated workflow:
# Workflow: Open and verify Overworld Editor
#   1. Click(button:Overworld)
#   2. Wait(window_visible:Overworld Editor, 5000ms)
#   3. Assert(visible:Overworld Editor)
#
# ✓ Connected to test harness
#
# [1/3] Click(button:Overworld) ... ✓ (125ms)
# [2/3] Wait(window_visible:Overworld Editor, 5000ms) ... ✓ (1250ms)
# [3/3] Assert(visible:Overworld Editor) ... ✓ (50ms)
#
# ✅ Test passed in 1425ms

Custom Server Configuration

# Connect to remote test harness
./build-grpc-test/bin/z3ed agent test \
  --prompt "Open Dungeon editor" \
  --host 192.168.1.100 \
  --port 50053 \
  --timeout 60

---

Error Messages

Connection Error

Failed to connect to test harness at localhost:50052
Make sure YAZE is running with:
  ./yaze --enable_test_harness --test_harness_port=50052 --rom_file=<rom>

Error: Connection refused

Solution: Start YAZE with test harness enabled

Unsupported Prompt

Unable to parse prompt: "Do something complex"

Supported patterns:
  - Open <Editor> editor
  - Open <Editor> and verify it loads
  - Type '<text>' in <input>
  - Click <button>

Examples:
  - Open Overworld editor
  - Open Dungeon editor and verify it loads
  - Type 'zelda3.sfc' in filename input
  - Click Open ROM button

Solution: Use one of the supported prompt patterns

Widget Not Found

[1/2] Click(button:NonExistent) ... ✗ FAILED
  Error: Button 'NonExistent' not found

Step 1 failed: Button 'NonExistent' not found

Solution:

• Verify widget exists in YAZE
• Check spelling (case-sensitive)
• Use exact label from GUI

Timeout Error

[2/2] Wait(window_visible:Slow Editor, 5000ms) ... ✗ FAILED
  Error: Condition not met after 5000 ms

Step 2 failed: Condition not met after 5000 ms

Solution:

• Increase timeout: --timeout 10
• Verify window actually opens
• Check for errors in YAZE

---

Exit Codes

• 0 - Success (all steps passed)
• 1 - Failure (connection, parsing, or execution error)

---

Troubleshooting

Port Already in Use

# Kill existing instances
killall yaze

# Wait for cleanup
sleep 2

# Use different port
./yaze --enable_test_harness --test_harness_port=50053 ...
./z3ed agent test --port 50053 ...

gRPC Not Available

GUI automation requires YAZE_WITH_GRPC=ON at build time.
Rebuild with: cmake -B build -DYAZE_WITH_GRPC=ON

Solution: Rebuild with gRPC support enabled

Widget Names Unknown

# Manual exploration with grpcurl
grpcurl -plaintext -import-path src/app/core/proto \
  -proto imgui_test_harness.proto \
  -d '{"condition":"visible:Main Window"}' \
  127.0.0.1:50052 yaze.test.ImGuiTestHarness/Assert

# Try different widget names until you find the right one

---

Advanced Usage

Shell Script Integration

#!/bin/bash
set -e

# Start YAZE
./yaze --enable_test_harness --rom_file=zelda3.sfc &
YAZE_PID=$!
sleep 3

# Run tests
./z3ed agent test --prompt "Open Overworld editor" || exit 1
./z3ed agent test --prompt "Open Dungeon editor" || exit 1

# Cleanup
kill $YAZE_PID

CI/CD Pipeline

# .github/workflows/gui-tests.yml
- name: Start YAZE Test Harness
  run: |
    ./yaze --enable_test_harness --rom_file=zelda3.sfc &
    sleep 5

- name: Run GUI Tests
  run: |
    ./z3ed agent test --prompt "Open Overworld editor"
    ./z3ed agent test --prompt "Open Dungeon editor"

---

Performance Characteristics

Typical Timings

• Click: 50-200ms
• Type: 100-300ms
• Wait: 100-5000ms (depends on condition)
• Assert: 10-100ms

Total Test Duration

• Simple click: ~100ms
• Open editor: ~1-2s
• Open + verify: ~1.5-2.5s
• Complex workflow: ~3-5s

---

Extending Functionality

Add New Pattern Type

1. Add pattern matcher (test_workflow_generator.h):

bool MatchesYourPattern(const std::string& prompt, ...);

2. Add workflow builder (test_workflow_generator.cc):

TestWorkflow BuildYourPatternWorkflow(...);

3. Add to GenerateWorkflow() (test_workflow_generator.cc):

if (MatchesYourPattern(prompt, &params)) {
  return BuildYourPatternWorkflow(params);
}

Add New Widget Type

Currently supported: button:, input:, window:

To add more, extend the target format in RPC calls.

---

See Also

• Full Documentation: IT-01-QUICKSTART.md
• E2E Validation: E2E_VALIDATION_GUIDE.md
• Implementation Details: IMPLEMENTATION_PROGRESS_OCT2.md
• Architecture Overview: E6-z3ed-implementation-plan.md

---

Last Updated: October 2, 2025
Version: IT-02 Complete
Status: Ready for validation