yaze/docs/z3ed/IT-08b-AUTO-CAPTURE.md

IT-08b: Auto-Capture on Test Failure - Implementation Guide

Status: 🔄 Ready to Implement
Priority: High (Next Phase of IT-08)
Time Estimate: 1-1.5 hours
Date: October 2, 2025

---

Overview

Automatically capture screenshots and execution context when tests fail, enabling better debugging and diagnostics for AI agents.

Goal: Every failed test produces:

• Screenshot of GUI state at failure
• Execution context (frame count, active windows, focused widgets)
• Foundation for IT-08c (widget state dumps)

---

Implementation Steps

Step 1: Update TestHistory Structure (15 minutes)

File: src/app/core/test_manager.h

Add failure diagnostics fields:

struct TestHistory {
  std::string test_id;
  std::string test_name;
  ImGuiTestStatus status;
  absl::Time start_time;
  absl::Time end_time;
  int64_t execution_time_ms;
  std::vector<std::string> logs;
  std::map<std::string, std::string> metrics;
  
  // IT-08b: Failure diagnostics
  std::string screenshot_path;
  int64_t screenshot_size_bytes = 0;
  std::string failure_context;
  
  // IT-08c: Widget state (future)
  std::string widget_state;
};

Step 2: Add CaptureFailureContext Method (30 minutes)

File: src/app/core/test_manager.cc

Add new method after MarkHarnessTestCompleted:

void TestManager::CaptureFailureContext(const std::string& test_id) {
  if (test_history_.find(test_id) == test_history_.end()) {
    return;
  }
  
  auto& history = test_history_[test_id];
  
  // 1. Capture screenshot via harness service
  if (harness_service_) {
    std::string screenshot_path = 
        absl::StrFormat("/tmp/yaze_test_%s_failure.bmp", test_id);
    
    ScreenshotRequest req;
    req.set_output_path(screenshot_path);
    
    ScreenshotResponse resp;
    auto status = harness_service_->Screenshot(&req, &resp);
    
    if (status.ok() && resp.success()) {
      history.screenshot_path = resp.file_path();
      history.screenshot_size_bytes = resp.file_size_bytes();
    } else {
      YAZE_LOG(ERROR) << "Failed to capture screenshot for " << test_id 
                      << ": " << status.message();
    }
  }
  
  // 2. Capture execution context
  ImGuiContext* ctx = ImGui::GetCurrentContext();
  if (ctx) {
    ImGuiWindow* current_window = ImGui::GetCurrentWindow();
    std::string window_name = current_window ? current_window->Name : "none";
    
    ImGuiID active_id = ImGui::GetActiveID();
    ImGuiID hovered_id = ImGui::GetHoveredID();
    
    history.failure_context = absl::StrFormat(
        "Frame: %d, Window: %s, Active: %u, Hovered: %u",
        ImGui::GetFrameCount(),
        window_name,
        active_id,
        hovered_id);
  }
  
  // 3. Widget state capture (IT-08c - placeholder)
  // history.widget_state = CaptureWidgetState();
}

Step 3: Integrate with MarkHarnessTestCompleted (15 minutes)

File: src/app/core/test_manager.cc

Modify existing method to call CaptureFailureContext:

void TestManager::MarkHarnessTestCompleted(const std::string& test_id,
                                           ImGuiTestStatus status) {
  if (test_history_.find(test_id) == test_history_.end()) {
    return;
  }
  
  auto& history = test_history_[test_id];
  history.status = status;
  history.end_time = absl::Now();
  history.execution_time_ms = absl::ToInt64Milliseconds(
      history.end_time - history.start_time);
  
  // Auto-capture diagnostics on failure
  if (status == ImGuiTestStatus_Error || status == ImGuiTestStatus_Warning) {
    CaptureFailureContext(test_id);
  }
  
  // Notify waiting threads
  cv_.notify_all();
}

Step 4: Update GetTestResults RPC (30 minutes)

File: src/app/core/proto/imgui_test_harness.proto

Add fields to response:

message GetTestResultsResponse {
  string test_id = 1;
  TestStatus status = 2;
  int64 execution_time_ms = 3;
  repeated string logs = 4;
  map<string, string> metrics = 5;
  
  // IT-08b: Failure diagnostics
  string screenshot_path = 6;
  int64 screenshot_size_bytes = 7;
  string failure_context = 8;
  
  // IT-08c: Widget state (future)
  string widget_state = 9;
}

File: src/app/core/service/imgui_test_harness_service.cc

Update implementation:

absl::Status ImGuiTestHarnessServiceImpl::GetTestResults(
    const GetTestResultsRequest* request,
    GetTestResultsResponse* response) {
  
  const std::string& test_id = request->test_id();
  auto history = test_manager_->GetTestHistory(test_id);
  
  if (!history.has_value()) {
    return absl::NotFoundError(
        absl::StrFormat("Test not found: %s", test_id));
  }
  
  const auto& h = history.value();
  
  // Basic info
  response->set_test_id(h.test_id);
  response->set_status(ConvertImGuiTestStatusToProto(h.status));
  response->set_execution_time_ms(h.execution_time_ms);
  
  // Logs and metrics
  for (const auto& log : h.logs) {
    response->add_logs(log);
  }
  for (const auto& [key, value] : h.metrics) {
    (*response->mutable_metrics())[key] = value;
  }
  
  // IT-08b: Failure diagnostics
  if (!h.screenshot_path.empty()) {
    response->set_screenshot_path(h.screenshot_path);
    response->set_screenshot_size_bytes(h.screenshot_size_bytes);
  }
  if (!h.failure_context.empty()) {
    response->set_failure_context(h.failure_context);
  }
  
  // IT-08c: Widget state (future)
  if (!h.widget_state.empty()) {
    response->set_widget_state(h.widget_state);
  }
  
  return absl::OkStatus();
}

---

Testing

Build and Start Test Harness

# 1. Rebuild with changes
cmake --build build-grpc-test --target yaze -j$(sysctl -n hw.ncpu)

# 2. Start test harness
./build-grpc-test/bin/yaze.app/Contents/MacOS/yaze \
  --enable_test_harness \
  --test_harness_port=50052 \
  --rom_file=assets/zelda3.sfc &

Trigger Test Failure

# 3. Trigger a failing test (nonexistent widget)
grpcurl -plaintext \
  -import-path src/app/core/proto \
  -proto imgui_test_harness.proto \
  -d '{"target":"nonexistent_widget","type":"LEFT"}' \
  127.0.0.1:50052 yaze.test.ImGuiTestHarness/Click

# Response should indicate failure

Verify Screenshot Captured

# 4. Check for auto-captured screenshot
ls -lh /tmp/yaze_test_*_failure.bmp

# Expected: BMP file created (5.3MB)

Query Test Results

# 5. Get test results (replace <test_id> with actual ID from Click response)
grpcurl -plaintext \
  -import-path src/app/core/proto \
  -proto imgui_test_harness.proto \
  -d '{"test_id":"<test_id>"}' \
  127.0.0.1:50052 yaze.test.ImGuiTestHarness/GetTestResults

# Expected output:
{
  "testId": "grpc_click_12345678",
  "status": "FAILED",
  "executionTimeMs": "1234",
  "logs": [...],
  "screenshotPath": "/tmp/yaze_test_grpc_click_12345678_failure.bmp",
  "screenshotSizeBytes": "5308538",
  "failureContext": "Frame: 1234, Window: Main Window, Active: 0, Hovered: 0"
}

End-to-End Test Script

Create scripts/test_auto_capture.sh:

#!/bin/bash
set -e

echo "=== IT-08b Auto-Capture Test ==="

# Clean up old screenshots
rm -f /tmp/yaze_test_*_failure.bmp

# Start YAZE with test harness
echo "Starting YAZE..."
./build-grpc-test/bin/yaze.app/Contents/MacOS/yaze \
  --enable_test_harness \
  --test_harness_port=50052 \
  --rom_file=assets/zelda3.sfc &
YAZE_PID=$!

# Wait for server to start
sleep 3

# Trigger failing test
echo "Triggering test failure..."
TEST_ID=$(grpcurl -plaintext \
  -import-path src/app/core/proto \
  -proto imgui_test_harness.proto \
  -d '{"target":"nonexistent_widget","type":"LEFT"}' \
  127.0.0.1:50052 yaze.test.ImGuiTestHarness/Click | \
  jq -r '.testId')

echo "Test ID: $TEST_ID"

# Wait for test to complete
sleep 2

# Check screenshot captured
if [ -f "/tmp/yaze_test_${TEST_ID}_failure.bmp" ]; then
  echo "✅ Screenshot captured: /tmp/yaze_test_${TEST_ID}_failure.bmp"
else
  echo "❌ Screenshot NOT captured"
  kill $YAZE_PID
  exit 1
fi

# Query test results
echo "Querying test results..."
RESULTS=$(grpcurl -plaintext \
  -import-path src/app/core/proto \
  -proto imgui_test_harness.proto \
  -d "{\"test_id\":\"$TEST_ID\"}" \
  127.0.0.1:50052 yaze.test.ImGuiTestHarness/GetTestResults)

echo "$RESULTS"

# Verify fields present
if echo "$RESULTS" | jq -e '.screenshotPath' > /dev/null; then
  echo "✅ Screenshot path in results"
else
  echo "❌ Screenshot path missing"
  kill $YAZE_PID
  exit 1
fi

if echo "$RESULTS" | jq -e '.failureContext' > /dev/null; then
  echo "✅ Failure context in results"
else
  echo "❌ Failure context missing"
  kill $YAZE_PID
  exit 1
fi

echo "=== All tests passed! ==="

# Cleanup
kill $YAZE_PID

---

Success Criteria

• ✅ Screenshots auto-captured on test failure (Error or Warning status)
• ✅ Screenshot path stored in TestHistory
• ✅ Failure context captured (frame, window, widgets)
• ✅ GetTestResults RPC returns screenshot_path and failure_context
• ✅ No performance impact on passing tests (capture only on failure)
• ✅ Clean error handling if screenshot capture fails

---

Files Modified

1. src/app/core/test_manager.h - TestHistory structure
2. src/app/core/test_manager.cc - CaptureFailureContext method
3. src/app/core/proto/imgui_test_harness.proto - GetTestResultsResponse fields
4. src/app/core/service/imgui_test_harness_service.cc - GetTestResults implementation

---

Next Steps

After IT-08b Complete:

1. IT-08c: Widget State Dumps (30-45 minutes)
2. IT-08d: Error Envelope Standardization (1-2 hours)
3. IT-08e: CLI Error Improvements (1 hour)

Documentation Updates:

1. Update IT-08-IMPLEMENTATION-GUIDE.md with IT-08b complete status
2. Update E6-z3ed-implementation-plan.md progress tracking
3. Update README.md with new capabilities

---

Last Updated: October 2, 2025
Status: Ready to implement
Estimated Completion: October 2-3, 2025 (1-1.5 hours)