feat: Implement auto-capture of screenshots and diagnostics on test failures
- Added a new helper function `CaptureHarnessScreenshot` to encapsulate SDL screenshot logic. - Updated `ImGuiTestHarnessServiceImpl::Screenshot` to utilize the new screenshot helper. - Enhanced `TestManager::CaptureFailureContext` to automatically capture screenshots and widget state on test failures. - Introduced new fields in the `GetTestResultsResponse` proto for screenshot path, size, failure context, and widget state. - Updated CLI and gRPC client to expose new diagnostic fields in test results. - Ensured that screenshots are saved in a structured directory under the system's temp directory. - Improved logging for auto-capture events, including success and failure messages.
This commit is contained in:
scawful
@@ -107,27 +107,23 @@ The z3ed CLI and AI agent workflow system has completed major infrastructure mil
|
||||
- **Application Consistency**: z3ed, EditorManager, and core services emit heterogeneous error formats
|
||||
|
||||
#### IT-05: Test Introspection API (6-8 hours)
|
||||
**Status (Oct 2, 2025)**: 🟡 *Server-side RPCs implemented; CLI + E2E pending*
|
||||
**Status (Oct 2, 2025)**: ✅ Completed
|
||||
|
||||
**Progress**:
|
||||
- ✅ `imgui_test_harness.proto` expanded with GetTestStatus/ListTests/GetTestResults messages.
|
||||
- ✅ `TestManager` maintains execution history (queued→running→completed) with logs, metrics, and aggregates.
|
||||
- ✅ `ImGuiTestHarnessServiceImpl` exposes the three introspection RPCs with pagination, status conversion, and log/metric marshalling.
|
||||
- ⚠️ `agent` CLI commands (`test status`, `test list`, `test results`) still stubbed.
|
||||
- ⚠️ End-to-end introspection script (`scripts/test_introspection_e2e.sh`) not implemented; regression script `test_harness_e2e.sh` currently failing because it references the unfinished CLI.
|
||||
**Highlights**:
|
||||
- `imgui_test_harness.proto` now exposes `GetTestStatus`, `ListTests`, and
|
||||
`GetTestResults` RPCs backed by `TestManager`'s execution history.
|
||||
- CLI commands (`z3ed agent test status|list|results`) are fully wired with
|
||||
JSON/YAML formatting, follow-mode polling, and filtering options.
|
||||
- `GuiAutomationClient` provides typed wrappers for introspection APIs so agent
|
||||
workflows can poll status programmatically.
|
||||
- Regression coverage lives in `scripts/test_harness_e2e.sh`; a slimmer
|
||||
introspection smoke (`scripts/test_introspection_e2e.sh`) is queued for CI
|
||||
automation but manual verification paths are documented.
|
||||
|
||||
**Immediate Next Steps**:
|
||||
1. **Wire CLI Client Methods**
|
||||
- Implement gRPC client wrappers for the new RPCs in the automation client.
|
||||
- Add user-facing commands under `z3ed agent test ...` with JSON/YAML output options.
|
||||
2. **Author E2E Validation Script**
|
||||
- Spin up harness, run Click/Assert workflow, poll via `agent test status`, fetch results.
|
||||
- Update CI notes with the new script and expected output.
|
||||
3. **Documentation & Examples**
|
||||
- Extend `E6-z3ed-reference.md` with full usage examples and sample outputs.
|
||||
- Add troubleshooting section covering common errors (unknown test_id, timeout, etc.).
|
||||
4. **Stretch (Optional Before IT-06)**
|
||||
- Capture assertion metadata (expected/actual) for richer `AssertionResult` payloads.
|
||||
**Future Enhancements**:
|
||||
- Capture richer assertion metadata (expected/actual pairs) for improved
|
||||
failure messaging when the underlying harness exposes it.
|
||||
- Add pagination helpers to CLI once history volume grows (low priority).
|
||||
|
||||
**Example Usage**:
|
||||
```bash
|
||||
@@ -270,16 +266,16 @@ message DiscoverWidgetsResponse {
|
||||
**Implementation Tracks**:
|
||||
1. **Harness-Level Diagnostics**
|
||||
- ✅ IT-08a: Screenshot RPC implemented (SDL-based, BMP format, 1536x864)
|
||||
- ✅ IT-08b: Auto-capture screenshots and context on test failure
|
||||
- <20> IT-08c: Widget tree dumps and recent ImGui events on failure (NEXT)
|
||||
- Serialize results to both structured JSON (for automation) and human-friendly HTML bundles
|
||||
- Persist artifacts under `test-results/<test_id>/` with timestamped directories
|
||||
- ✅ IT-08b: Auto-capture screenshots and context on test failure using shared
|
||||
helper that writes to `${TMPDIR}/yaze/test-results/<test_id>/`
|
||||
- ✅ IT-08c: Widget tree JSON dumps emitted alongside failure context
|
||||
- ⏳ HTML bundle exporter (screenshots + widget tree) remains a stretch goal
|
||||
|
||||
2. **CLI Experience Improvements**
|
||||
- Surface artifact paths, failure context, and widget state in CLI output (DONE)
|
||||
- Standardize error envelopes in z3ed (`absl::Status` + structured payload)
|
||||
- Surface artifact paths, summarized failure reason, and next-step hints in CLI output
|
||||
- Add `--format html` / `--format json` flags to `z3ed agent test results` to emit richer context
|
||||
- Integrate with recording workflow: replay failures using captured state for fast reproduction
|
||||
- Add `--format html` flag to emit rich bundles (planned)
|
||||
- Integrate with recording workflow: replay failures using captured state (planned)
|
||||
|
||||
3. **EditorManager & Application Integration**
|
||||
- Introduce shared `ErrorAnnotatedResult` utility exposing `status`, `context`, `actionable_hint`
|
||||
@@ -299,7 +295,7 @@ message DiscoverWidgetsResponse {
|
||||
"assertion": "visible:Overworld",
|
||||
"expected": "visible",
|
||||
"actual": "hidden",
|
||||
"screenshot": "/tmp/yaze_test_12345678.png",
|
||||
"screenshot": "/tmp/yaze/test-results/grpc_assert_12345678/failure_1696357220000.bmp",
|
||||
"widget_state": {
|
||||
"active_window": "Main Window",
|
||||
"focused_widget": null,
|
||||
|
||||
@@ -58,182 +58,72 @@ absl::Status ImGuiTestHarnessServiceImpl::Screenshot(
|
||||
if (!backend_data || !backend_data->Renderer) {
|
||||
response->set_success(false);
|
||||
response->set_message("SDL renderer not available");
|
||||
return absl::FailedPreconditionError("No SDL renderer available");
|
||||
}
|
||||
|
||||
SDL_Renderer* renderer = backend_data->Renderer;
|
||||
|
||||
// 2. Get renderer output size
|
||||
int width, height;
|
||||
SDL_GetRendererOutputSize(renderer, &width, &height);
|
||||
|
||||
// 3. Create surface to hold screenshot
|
||||
SDL_Surface* surface = SDL_CreateRGBSurface(0, width, height, 32,
|
||||
0x00FF0000, 0x0000FF00,
|
||||
0x000000FF, 0xFF000000);
|
||||
|
||||
// 4. Read pixels from renderer (ARGB8888 format)
|
||||
SDL_RenderReadPixels(renderer, nullptr, SDL_PIXELFORMAT_ARGB8888,
|
||||
surface->pixels, surface->pitch);
|
||||
|
||||
// 5. Determine output path (custom or auto-generated)
|
||||
std::string output_path = request->output_path();
|
||||
if (output_path.empty()) {
|
||||
output_path = absl::StrFormat("/tmp/yaze_screenshot_%lld.bmp",
|
||||
absl::ToUnixMillis(absl::Now()));
|
||||
}
|
||||
|
||||
// 6. Save to BMP file
|
||||
SDL_SaveBMP(surface, output_path.c_str());
|
||||
|
||||
// 7. Get file size and clean up
|
||||
std::ifstream file(output_path, std::ios::binary | std::ios::ate);
|
||||
int64_t file_size = file.tellg();
|
||||
|
||||
SDL_FreeSurface(surface);
|
||||
|
||||
// 8. Return success response
|
||||
response->set_success(true);
|
||||
response->set_message(absl::StrFormat("Screenshot saved to %s (%dx%d)",
|
||||
output_path, width, height));
|
||||
response->set_file_path(output_path);
|
||||
response->set_file_size_bytes(file_size);
|
||||
|
||||
return absl::OkStatus();
|
||||
}
|
||||
```
|
||||
|
||||
### Testing Results
|
||||
|
||||
**Test Command**:
|
||||
```bash
|
||||
grpcurl -plaintext \
|
||||
-import-path /Users/scawful/Code/yaze/src/app/core/proto \
|
||||
-proto imgui_test_harness.proto \
|
||||
-d '{"output_path": "/tmp/test_screenshot.bmp"}' \
|
||||
localhost:50052 yaze.test.ImGuiTestHarness/Screenshot
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Screenshot saved to /tmp/test_screenshot.bmp (1536x864)",
|
||||
"filePath": "/tmp/test_screenshot.bmp",
|
||||
"fileSizeBytes": "5308538"
|
||||
}
|
||||
```
|
||||
|
||||
**File Verification**:
|
||||
```bash
|
||||
$ ls -lh /tmp/test_screenshot.bmp
|
||||
-rw-r--r-- 1 scawful wheel 5.1M Oct 2 20:16 /tmp/test_screenshot.bmp
|
||||
|
||||
$ file /tmp/test_screenshot.bmp
|
||||
/tmp/test_screenshot.bmp: PC bitmap, Windows 95/NT4 and newer format, 1536 x 864 x 32, cbSize 5308538, bits offset 122
|
||||
```
|
||||
|
||||
✅ **Result**: Screenshot successfully captured, saved, and validated!
|
||||
|
||||
---
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### Why BMP Format?
|
||||
|
||||
**Chosen**: SDL's built-in `SDL_SaveBMP` function
|
||||
**Rationale**:
|
||||
- ✅ Zero external dependencies (no need for libpng, stb_image_write, etc.)
|
||||
- ✅ Guaranteed to work on all platforms where SDL works
|
||||
- ✅ Simple, reliable, and fast
|
||||
- ✅ Adequate for debugging/error reporting (file size not critical)
|
||||
- ⚠️ Larger file sizes (5.3MB vs ~500KB for PNG), but acceptable for temporary debug files
|
||||
|
||||
**Future Consideration**: If disk space becomes an issue, can add PNG encoding using stb_image_write (single-header library, easy to integrate)
|
||||
|
||||
### SDL Backend Integration
|
||||
|
||||
**Challenge**: How to access the SDL_Renderer from ImGui?
|
||||
**Solution**:
|
||||
- ImGui's `BackendRendererUserData` points to an `ImGui_ImplSDLRenderer2_Data` struct
|
||||
- This struct contains the `Renderer` pointer as its first member
|
||||
- Cast `BackendRendererUserData` to access the renderer safely
|
||||
|
||||
**Why Not Store Renderer Globally?**
|
||||
- Multiple ImGui contexts could use different renderers
|
||||
- Backend data pattern follows ImGui's architecture conventions
|
||||
- More maintainable and future-proof
|
||||
|
||||
---
|
||||
|
||||
## Integration with Test System
|
||||
|
||||
### Current Usage (Manual RPC)
|
||||
|
||||
AI agents or CLI tools can manually capture screenshots:
|
||||
|
||||
```bash
|
||||
# Capture screenshot after opening editor
|
||||
z3ed agent test --prompt "Open Overworld Editor"
|
||||
grpcurl ... yaze.test.ImGuiTestHarness/Screenshot
|
||||
```
|
||||
|
||||
### Next Step: Auto-Capture on Failure
|
||||
|
||||
The screenshot RPC is now ready to be integrated with TestManager to automatically capture context when tests fail:
|
||||
|
||||
**Planned Implementation** (IT-08 Phase 2):
|
||||
```cpp
|
||||
// In TestManager::MarkHarnessTestCompleted()
|
||||
if (test_result == IMGUI_TEST_STATUS_FAILED ||
|
||||
test_result == IMGUI_TEST_STATUS_TIMEOUT) {
|
||||
|
||||
// Auto-capture screenshot
|
||||
ScreenshotRequest req;
|
||||
req.set_output_path(absl::StrFormat("/tmp/test_%s_failure.bmp", test_id));
|
||||
|
||||
ScreenshotResponse resp;
|
||||
harness_service_->Screenshot(&req, &resp);
|
||||
|
||||
test_history_[test_id].screenshot_path = resp.file_path();
|
||||
|
||||
// Also capture widget state (IT-08 Phase 3)
|
||||
test_history_[test_id].widget_state = CaptureWidgetState();
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## IT-08b: Auto-Capture on Test Failure ✅ COMPLETE
|
||||
|
||||
**Date Completed**: October 2, 2025
|
||||
**Time**: 1.5 hours
|
||||
## IT-08b: Auto-Capture on Test Failure ✅ COMPLETE
|
||||
|
||||
### Implementation Summary
|
||||
**Date Completed**: October 2, 2025
|
||||
**Artifacts**: `CaptureFailureContext`, `screenshot_utils.{h,cc}`, CLI introspection updates
|
||||
|
||||
Successfully implemented automatic screenshot and context capture when tests fail or timeout.
|
||||
### Highlights
|
||||
|
||||
### What Was Built
|
||||
- **Shared SDL helper**: New `CaptureHarnessScreenshot()` centralizes renderer
|
||||
capture and writes BMP files into `${TMPDIR}/yaze/test-results/<test_id>/`.
|
||||
- **TestManager integration**: Failure context now records ImGui window/nav
|
||||
state, widget hierarchy (`CaptureWidgetState`), and screenshot metadata while
|
||||
keeping `HarnessTestExecution` aggregates in sync.
|
||||
- **Graceful fallbacks**: When `YAZE_WITH_GRPC` is disabled we emit a harness
|
||||
log noting that screenshot capture is unavailable.
|
||||
- **End-user surfacing**: `GuiAutomationClient::GetTestResults` and
|
||||
`z3ed agent test results` expose `screenshot_path`, `screenshot_size_bytes`,
|
||||
`failure_context`, and `widget_state` in both YAML and JSON modes.
|
||||
|
||||
1. **TestManager Integration**:
|
||||
- Added failure diagnostic fields to `HarnessTestExecution` struct
|
||||
- Modified `MarkHarnessTestCompleted()` to auto-trigger capture on failure/timeout
|
||||
- Implemented `CaptureFailureContext()` method with execution context capture
|
||||
### Key Touch Points
|
||||
|
||||
2. **Failure Context Capture**:
|
||||
- Frame count at failure time
|
||||
- Active window name
|
||||
- Focused widget ID
|
||||
- Screenshot path placeholder for future RPC integration
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `src/app/core/service/screenshot_utils.{h,cc}` | SDL renderer capture reused by RPC + auto-capture |
|
||||
| `src/app/test/test_manager.cc` | Auto-capture pipeline with per-test artifact directories |
|
||||
| `src/app/core/service/imgui_test_harness_service.cc` | Screenshot RPC delegates to shared helper |
|
||||
| `src/cli/service/gui_automation_client.*` | Propagates new proto fields to CLI |
|
||||
| `src/cli/handlers/agent/test_commands.cc` | Presents diagnostics to users/agents |
|
||||
|
||||
3. **Proto Schema Updates**:
|
||||
- Added `screenshot_path`, `screenshot_size_bytes`, `failure_context`, `widget_state` to `GetTestResultsResponse`
|
||||
### Validation Checklist
|
||||
|
||||
4. **gRPC Service Integration**:
|
||||
- Updated `GetTestResults` RPC to include failure diagnostics in response
|
||||
```bash
|
||||
# Build (needs YAZE_WITH_GRPC=ON)
|
||||
cmake --build build-grpc-test --target yaze -j$(sysctl -n hw.ncpu)
|
||||
|
||||
# Start harness
|
||||
./build-grpc-test/bin/yaze.app/Contents/MacOS/yaze \
|
||||
--enable_test_harness --test_harness_port=50052 \
|
||||
--rom_file=assets/zelda3.sfc &
|
||||
|
||||
# Queue a failing automation step
|
||||
grpcurl -plaintext \
|
||||
-import-path src/app/core/proto \
|
||||
-proto imgui_test_harness.proto \
|
||||
-d '{"target":"button:DoesNotExist","type":"LEFT"}' \
|
||||
localhost:50052 yaze.test.ImGuiTestHarness/Click
|
||||
|
||||
# Fetch diagnostics
|
||||
z3ed agent test results --test-id <captured_id> --include-logs --format yaml
|
||||
|
||||
# Inspect artifact directory
|
||||
ls ${TMPDIR}/yaze/test-results/<captured_id>/
|
||||
```
|
||||
|
||||
You should see a `.bmp` failure screenshot, widget JSON in the CLI output, and
|
||||
logs noting the auto-capture event. When the helper fails (e.g., renderer not
|
||||
ready) the harness log and CLI output record the failure reason.
|
||||
|
||||
### Next Steps
|
||||
|
||||
- Wire the same helper into HTML bundle generation (IT-08c follow-up).
|
||||
- Add configurable artifact root (`--error-artifact-dir`) for CI separation.
|
||||
- Consider PNG encoding via `stb_image_write` if file size becomes an issue.
|
||||
|
||||
---
|
||||
### Technical Implementation
|
||||
|
||||
**Location**: `/Users/scawful/Code/yaze/src/app/test/test_manager.{h,cc}`
|
||||
@@ -323,7 +213,7 @@ grpcurl -plaintext \
|
||||
"category": "grpc",
|
||||
"executedAtMs": "1696357200000",
|
||||
"durationMs": 150,
|
||||
"screenshotPath": "/tmp/yaze_test_grpc_click_12345678_failure.bmp",
|
||||
"screenshotPath": "/tmp/yaze/test-results/grpc_click_12345678/failure_1696357200000.bmp",
|
||||
"failureContext": "Frame: 1234, Active Window: Main Window, Focused Widget: 0x00000000"
|
||||
}
|
||||
```
|
||||
@@ -336,12 +226,12 @@ grpcurl -plaintext \
|
||||
- ✅ No deadlocks (mutex released before calling CaptureFailureContext)
|
||||
- ✅ Proto schema updated with new fields
|
||||
|
||||
### Next Steps
|
||||
### Retro Notes
|
||||
|
||||
The screenshot path is currently a placeholder. Future integration will:
|
||||
1. Call the Screenshot RPC from within CaptureFailureContext
|
||||
2. Wait for screenshot completion and store the actual file size
|
||||
3. Integrate with IT-08c for widget state dumps
|
||||
- Placeholder screenshot paths have been replaced by the shared helper that
|
||||
writes into `${TMPDIR}/yaze/test-results/<test_id>/` and records byte sizes.
|
||||
- Widget state capture (IT-08c) is now invoked directly from
|
||||
`CaptureFailureContext`, removing the TODOs from the original plan.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -122,128 +122,117 @@ void TestManager::MarkHarnessTestCompleted(const std::string& test_id,
|
||||
history.execution_time_ms = absl::ToInt64Milliseconds(
|
||||
history.end_time - history.start_time);
|
||||
|
||||
// Auto-capture diagnostics on failure
|
||||
if (status == ImGuiTestStatus_Error || status == ImGuiTestStatus_Warning) {
|
||||
CaptureFailureContext(test_id);
|
||||
}
|
||||
|
||||
// Notify waiting threads
|
||||
cv_.notify_all();
|
||||
}
|
||||
```
|
||||
# IT-08b: Auto-Capture on Test Failure
|
||||
|
||||
### Step 4: Update GetTestResults RPC (30 minutes)
|
||||
**Status**: ✅ Complete
|
||||
**Completed**: October 2, 2025
|
||||
**Owner**: Harness Platform Team
|
||||
**Depends On**: IT-08a (Screenshot RPC), IT-05 (execution history store)
|
||||
|
||||
**File**: `src/app/core/proto/imgui_test_harness.proto`
|
||||
---
|
||||
|
||||
Add fields to response:
|
||||
## Summary
|
||||
|
||||
```proto
|
||||
message GetTestResultsResponse {
|
||||
string test_id = 1;
|
||||
TestStatus status = 2;
|
||||
int64 execution_time_ms = 3;
|
||||
repeated string logs = 4;
|
||||
map<string, string> metrics = 5;
|
||||
|
||||
// IT-08b: Failure diagnostics
|
||||
string screenshot_path = 6;
|
||||
int64 screenshot_size_bytes = 7;
|
||||
string failure_context = 8;
|
||||
|
||||
// IT-08c: Widget state (future)
|
||||
string widget_state = 9;
|
||||
}
|
||||
```
|
||||
Harness failures now emit rich diagnostics automatically. Whenever a GUI test
|
||||
transitions into `FAILED` or `TIMEOUT` we capture:
|
||||
|
||||
**File**: `src/app/core/service/imgui_test_harness_service.cc`
|
||||
- A full-frame SDL screenshot written to a stable per-test artifact folder
|
||||
- ImGui execution context (frame number, active/nav/hovered windows & IDs)
|
||||
- Serialized widget hierarchy snapshot (`CaptureWidgetState`) for IT-08c
|
||||
- Append-only log entries surfaced through `GetTestResults`
|
||||
|
||||
Update implementation:
|
||||
All artifacts are exposed through both the gRPC API and the `z3ed agent test
|
||||
results` command (JSON/YAML), enabling AI agents and humans to retrieve the same
|
||||
diagnostics without extra RPC calls.
|
||||
|
||||
```cpp
|
||||
absl::Status ImGuiTestHarnessServiceImpl::GetTestResults(
|
||||
const GetTestResultsRequest* request,
|
||||
GetTestResultsResponse* response) {
|
||||
|
||||
const std::string& test_id = request->test_id();
|
||||
auto history = test_manager_->GetTestHistory(test_id);
|
||||
|
||||
if (!history.has_value()) {
|
||||
return absl::NotFoundError(
|
||||
absl::StrFormat("Test not found: %s", test_id));
|
||||
}
|
||||
|
||||
const auto& h = history.value();
|
||||
|
||||
// Basic info
|
||||
response->set_test_id(h.test_id);
|
||||
response->set_status(ConvertImGuiTestStatusToProto(h.status));
|
||||
response->set_execution_time_ms(h.execution_time_ms);
|
||||
|
||||
// Logs and metrics
|
||||
for (const auto& log : h.logs) {
|
||||
response->add_logs(log);
|
||||
}
|
||||
for (const auto& [key, value] : h.metrics) {
|
||||
(*response->mutable_metrics())[key] = value;
|
||||
}
|
||||
|
||||
// IT-08b: Failure diagnostics
|
||||
if (!h.screenshot_path.empty()) {
|
||||
response->set_screenshot_path(h.screenshot_path);
|
||||
response->set_screenshot_size_bytes(h.screenshot_size_bytes);
|
||||
}
|
||||
if (!h.failure_context.empty()) {
|
||||
response->set_failure_context(h.failure_context);
|
||||
}
|
||||
|
||||
// IT-08c: Widget state (future)
|
||||
if (!h.widget_state.empty()) {
|
||||
response->set_widget_state(h.widget_state);
|
||||
}
|
||||
|
||||
return absl::OkStatus();
|
||||
}
|
||||
```
|
||||
---
|
||||
|
||||
---
|
||||
## What Shipped
|
||||
|
||||
## Testing
|
||||
### Shared Screenshot Helper
|
||||
- New helper (`screenshot_utils.{h,cc}`) centralizes SDL capture logic.
|
||||
- Generates deterministic default paths under
|
||||
`${TMPDIR}/yaze/test-results/<test_id>/failure_<timestamp>.bmp`.
|
||||
- Reused by the manual `Screenshot` RPC to avoid duplicate code.
|
||||
|
||||
### Build and Start Test Harness
|
||||
### TestManager Auto-Capture Pipeline
|
||||
- `CaptureFailureContext` now:
|
||||
- Computes ImGui context metadata even when the test finishes on a worker
|
||||
thread.
|
||||
- Allocates artifact folders per test ID and requests a screenshot via the
|
||||
shared helper (guarded when gRPC is disabled).
|
||||
- Persists screenshot path, byte size, failure context, and widget state back
|
||||
into `HarnessTestExecution` while keeping aggregate caches in sync.
|
||||
- Emits structured harness logs for success/failure of the auto-capture.
|
||||
|
||||
```bash
|
||||
# 1. Rebuild with changes
|
||||
cmake --build build-grpc-test --target yaze -j$(sysctl -n hw.ncpu)
|
||||
### CLI & Client Updates
|
||||
- `GuiAutomationClient::GetTestResults` propagates new proto fields:
|
||||
`screenshot_path`, `screenshot_size_bytes`, `failure_context`, `widget_state`.
|
||||
- `z3ed agent test results` shows diagnostics in both human (YAML) and machine
|
||||
(JSON) modes, including `null` markers when artifacts are unavailable.
|
||||
- JSON output is now agent-ready: screenshot path + size enable downstream
|
||||
fetchers, failure context aids chain-of-thought prompts, widget state allows
|
||||
LLMs to reason about UI layout when debugging.
|
||||
|
||||
# 2. Start test harness
|
||||
./build-grpc-test/bin/yaze.app/Contents/MacOS/yaze \
|
||||
--enable_test_harness \
|
||||
--test_harness_port=50052 \
|
||||
--rom_file=assets/zelda3.sfc &
|
||||
```
|
||||
### Build Integration
|
||||
- gRPC build stanza now compiles the new helper files so both harness server and
|
||||
in-process capture use the same implementation.
|
||||
|
||||
### Trigger Test Failure
|
||||
---
|
||||
|
||||
```bash
|
||||
# 3. Trigger a failing test (nonexistent widget)
|
||||
grpcurl -plaintext \
|
||||
-import-path src/app/core/proto \
|
||||
-proto imgui_test_harness.proto \
|
||||
-d '{"target":"nonexistent_widget","type":"LEFT"}' \
|
||||
127.0.0.1:50052 yaze.test.ImGuiTestHarness/Click
|
||||
## Developer Notes
|
||||
|
||||
# Response should indicate failure
|
||||
```
|
||||
| Concern | Resolution |
|
||||
|---------|------------|
|
||||
| Deadlocks while capturing | Screenshot helper runs outside `harness_history_mutex_`; mutex is reacquired only for bookkeeping. |
|
||||
| Non-gRPC builds | Auto-capture logs a descriptive "unavailable" message and skips the SDL call, keeping deterministic behaviour when harness is stubbed. |
|
||||
| Artifact collisions | Paths are timestamped and namespaced per test ID; directories are created idempotently with error-code handling. |
|
||||
| Large widget dumps | Stored as JSON strings; CLI wraps them with quoting so they can be piped to `jq`/`yq` safely. |
|
||||
|
||||
### Verify Screenshot Captured
|
||||
---
|
||||
|
||||
```bash
|
||||
# 4. Check for auto-captured screenshot
|
||||
ls -lh /tmp/yaze_test_*_failure.bmp
|
||||
## Usage
|
||||
|
||||
# Expected: BMP file created (5.3MB)
|
||||
```
|
||||
1. Trigger a harness failure (e.g. click a nonexistent widget):
|
||||
```bash
|
||||
z3ed agent test --prompt "Click widget:nonexistent"
|
||||
```
|
||||
2. Fetch diagnostics:
|
||||
```bash
|
||||
z3ed agent test results --test-id grpc_click_deadbeef --include-logs --format json
|
||||
```
|
||||
3. Inspect artifacts:
|
||||
```bash
|
||||
open "$(jq -r '.screenshot_path' results.json)"
|
||||
```
|
||||
|
||||
Example YAML excerpt:
|
||||
|
||||
```yaml
|
||||
screenshot_path: "/var/folders/.../yaze/test-results/grpc_click_deadbeef/failure_1727890045123.bmp"
|
||||
screenshot_size_bytes: 5308538
|
||||
failure_context: "frame=1287 current_window=MainWindow nav_window=Agent hovered_window=Agent active_id=0x00000000 hovered_id=0x00000000"
|
||||
widget_state: '{"active_window":"MainWindow","visible_windows":["MainWindow","Agent"],"focused_widget":null}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
- Manual harness failure emits screenshot + widget dump under `/tmp`.
|
||||
- `GetTestResults` returns the new fields (verified via `grpcurl`).
|
||||
- CLI JSON/YAML output includes diagnostics with correct escaping.
|
||||
- Non-gRPC build path compiles (guarded sections).
|
||||
|
||||
---
|
||||
|
||||
## Follow-Up
|
||||
|
||||
- IT-08c leverages the persisted widget JSON to produce HTML bundles.
|
||||
- IT-08d will standardize error envelopes across CLI/services using these
|
||||
diagnostics.
|
||||
- Investigate persisting artifacts under configurable directories
|
||||
(`--artifact-dir`) for CI separation.
|
||||
|
||||
### Query Test Results
|
||||
|
||||
|
||||
Reference in New Issue
Block a user