Enhance ImGuiTestHarness with dynamic test integration and end-to-end validation

- Updated README.md to reflect the completion of IT-01 and the transition to end-to-end validation phase. - Introduced a new end-to-end test script (scripts/test_harness_e2e.sh) for validating all RPC methods of the ImGuiTestHarness gRPC service. - Implemented dynamic test functionality in ImGuiTestHarnessService for Type, Wait, and Assert methods, utilizing ImGuiTestEngine. - Enhanced error handling and response messages for better clarity during test execution. - Updated existing methods to support dynamic test registration and execution, ensuring robust interaction with the GUI elements.
2025-10-02 00:49:28 -04:00
parent 4320b67da1
commit 286efdec6a
19 changed files with 7325 additions and 222 deletions
--- a/docs/z3ed/archive/IT-01-getting-started-grpc.md
+++ b/docs/z3ed/archive/IT-01-getting-started-grpc.md
@@ -0,0 +1,624 @@
+# IT-01 Getting Started: gRPC Implementation
+
+**Goal**: Add gRPC-based ImGuiTestHarness to YAZE for automated GUI testing  
+**Timeline**: 10-14 hours over 2-3 days  
+**Current Phase**: Setup & Prototype
+
+## Quick Start Checklist
+
+- [ ] **Step 1**: Add gRPC to vcpkg.json (15 min)
+- [ ] **Step 2**: Update CMakeLists.txt (30 min)
+- [ ] **Step 3**: Create minimal .proto (15 min)
+- [ ] **Step 4**: Build and verify (30 min)
+- [ ] **Step 5**: Implement Ping service (1 hour)
+- [ ] **Step 6**: Test with grpcurl (15 min)
+- [ ] **Step 7**: Implement Click handler (2 hours)
+- [ ] **Step 8**: Add remaining operations (3-4 hours)
+- [ ] **Step 9**: CLI client integration (2 hours)
+- [ ] **Step 10**: Windows testing (2-3 hours)
+
+## Step-by-Step Implementation
+
+### Step 0: Read Dependency Management Guide (5 min)
+
+⚠️ **IMPORTANT**: Before adding gRPC, understand the existing infrastructure:
+- **[DEPENDENCY_MANAGEMENT.md](DEPENDENCY_MANAGEMENT.md)** - Cross-platform build strategy
+
+**Good News**: YAZE already has gRPC support via CMake FetchContent!
+- ✅ `cmake/grpc.cmake` exists with gRPC v1.70.1 + Protobuf v29.3
+- ✅ Builds from source (no vcpkg needed for gRPC)
+- ✅ Works identically on macOS, Linux, Windows
+- ✅ Statically linked (no DLL issues)
+
+**What We Need To Do**:
+1. Test that existing gRPC infrastructure works
+2. Add CMake option to enable gRPC (currently always-off)
+3. Create `.proto` schema for ImGuiTestHarness
+4. Implement gRPC service using existing `target_add_protobuf()` helper
+
+### Step 1: Verify Existing gRPC Infrastructure (30 min)
+
+**Goal**: Confirm `cmake/grpc.cmake` works on your system
+
+```bash
+cd /Users/scawful/Code/yaze
+
+# Check existing gRPC infrastructure
+cat cmake/grpc.cmake | head -20
+# Should show FetchContent_Declare for grpc v1.70.1
+
+# Create isolated test build
+cmake -B build-grpc-test -DYAZE_WITH_GRPC=ON
+
+# This will:
+# 1. Download gRPC v1.70.1 from GitHub (~100MB, 2-3 min)
+# 2. Download Protobuf v29.3 from GitHub (~50MB, 1-2 min)  
+# 3. Build both from source (~15-20 min first time)
+# 4. Cache everything in build-grpc-test/ for reuse
+
+# Watch for:
+#   -- Fetching grpc...
+#   -- Fetching protobuf...
+#   -- Building CXX object (lots of output)...
+
+# If this fails, gRPC infrastructure needs fixing first
+```
+
+**Expected Output**:
+```
+-- Fetching grpc...
+-- Populating grpc
+-- Fetching protobuf...
+-- Populating protobuf
+-- Building grpc (this will take 15-20 minutes)...
+[lots of compilation output]
+-- Configuring done
+-- Generating done
+```
+
+**Success Criteria**:
+- ✅ FetchContent downloads gRPC + Protobuf successfully
+- ✅ Both build without errors
+- ✅ No need to install anything (all in build directory)
+
+**If This Fails**: Stop here, investigate errors in `build-grpc-test/CMakeFiles/CMakeError.log`
+
+### Step 2: Add CMake Option for gRPC (15 min)
+
+**Goal**: Make gRPC opt-in via CMake flag
+
+The existing `cmake/grpc.cmake` is always included, but we need to make it optional.
+
+Add to `CMakeLists.txt` (after `project(yaze VERSION ...)`, around line 50):
+
+```cmake
+# Optional gRPC support for ImGuiTestHarness
+option(YAZE_WITH_GRPC "Enable gRPC-based ImGuiTestHarness (experimental)" OFF)
+
+if(YAZE_WITH_GRPC)
+  message(STATUS "✓ gRPC support enabled (FetchContent will download source)")
+  message(STATUS "  Note: First build takes 15-20 minutes to compile gRPC")
+  
+  # Include existing gRPC infrastructure
+  include(cmake/grpc.cmake)
+  
+  # Pass to source code
+  add_compile_definitions(YAZE_WITH_GRPC)
+  
+  set(YAZE_HAS_GRPC TRUE)
+else()
+  message(STATUS "○ gRPC support disabled (set YAZE_WITH_GRPC=ON to enable)")
+  set(YAZE_HAS_GRPC FALSE)
+endif()
+```
+
+**Why This Works**:
+- `cmake/grpc.cmake` already uses FetchContent to download + build gRPC
+- It provides `target_add_protobuf(target proto_file)` helper
+- We just need to make it conditional
+
+**Test the CMake change**:
+```bash
+# Reconfigure with gRPC enabled
+cmake -B build-grpc-test -DYAZE_WITH_GRPC=ON
+
+# Should show:
+#   -- ✓ gRPC support enabled (FetchContent will download source)
+#   -- Note: First build takes 15-20 minutes to compile gRPC
+#   -- Fetching grpc...
+
+# Without flag (default):
+cmake -B build-default
+
+# Should show:
+#   -- ○ gRPC support disabled (set YAZE_WITH_GRPC=ON to enable)
+```
+
+### Step 2: Update CMakeLists.txt (30 min)
+
+Add to root `CMakeLists.txt`:
+
+```cmake
+# Optional gRPC support for ImGuiTestHarness
+option(YAZE_WITH_GRPC "Enable gRPC-based ImGuiTestHarness" OFF)
+
+if(YAZE_WITH_GRPC)
+  find_package(gRPC CONFIG REQUIRED)
+  find_package(Protobuf CONFIG REQUIRED)
+  
+  message(STATUS "gRPC support enabled")
+  message(STATUS "  gRPC version: ${gRPC_VERSION}")
+  message(STATUS "  Protobuf version: ${Protobuf_VERSION}")
+  
+  # Function to generate C++ from .proto
+  function(yaze_add_grpc_proto target proto_file)
+    get_filename_component(proto_dir ${proto_file} DIRECTORY)
+    get_filename_component(proto_name ${proto_file} NAME_WE)
+    
+    set(proto_srcs "${CMAKE_CURRENT_BINARY_DIR}/${proto_name}.pb.cc")
+    set(proto_hdrs "${CMAKE_CURRENT_BINARY_DIR}/${proto_name}.pb.h")
+    set(grpc_srcs "${CMAKE_CURRENT_BINARY_DIR}/${proto_name}.grpc.pb.cc")
+    set(grpc_hdrs "${CMAKE_CURRENT_BINARY_DIR}/${proto_name}.grpc.pb.h")
+    
+    add_custom_command(
+      OUTPUT ${proto_srcs} ${proto_hdrs} ${grpc_srcs} ${grpc_hdrs}
+      COMMAND protobuf::protoc
+        --proto_path=${proto_dir}
+        --cpp_out=${CMAKE_CURRENT_BINARY_DIR}
+        --grpc_out=${CMAKE_CURRENT_BINARY_DIR}
+        --plugin=protoc-gen-grpc=$<TARGET_FILE:gRPC::grpc_cpp_plugin>
+        ${proto_file}
+      DEPENDS ${proto_file}
+      COMMENT "Generating C++ from ${proto_file}"
+    )
+    
+    target_sources(${target} PRIVATE ${proto_srcs} ${grpc_srcs})
+    target_include_directories(${target} PRIVATE ${CMAKE_CURRENT_BINARY_DIR})
+  endfunction()
+endif()
+```
+
+### Step 3: Create minimal .proto (15 min)
+
+```bash
+mkdir -p src/app/core/proto
+```
+
+Create `src/app/core/proto/imgui_test_harness.proto`:
+
+```protobuf
+syntax = "proto3";
+
+package yaze.test;
+
+// ImGuiTestHarness service for remote GUI testing
+service ImGuiTestHarness {
+  // Health check
+  rpc Ping(PingRequest) returns (PingResponse);
+  
+  // TODO: Add Click, Type, Wait, Assert
+}
+
+message PingRequest {
+  string message = 1;
+}
+
+message PingResponse {
+  string message = 1;
+  int64 timestamp_ms = 2;
+}
+```
+
+### Step 4: Build and verify (20 min)
+
+```bash
+# Build YAZE with gRPC enabled
+cd /Users/scawful/Code/yaze
+
+cmake --build build-grpc-test --target yaze -j8
+
+# First time: 15-20 minutes (compiling gRPC)
+# Subsequent: ~30 seconds (using cached gRPC)
+
+# Watch for:
+#   [1/500] Building CXX object (gRPC compilation)
+#   ...
+#   [500/500] Linking CXX executable yaze
+```
+
+**Expected Outcomes**:
+
+✅ **Success**:
+- Build completes without errors
+- Binary size: `build-grpc-test/bin/yaze.app` is ~10-15MB larger
+- `YAZE_WITH_GRPC` preprocessor flag defined in code
+- `target_add_protobuf()` function available
+
+⚠️ **Common Issues**:
+- **"error: 'absl::...' not found"**: gRPC needs abseil. Check `cmake/absl.cmake` is included.
+- **Long compile time**: Normal! gRPC is a large library (~500 source files)
+- **Out of disk space**: gRPC build artifacts ~2GB. Clean old builds: `rm -rf build/`
+
+**Verify gRPC is linked**:
+```bash
+# macOS: Check for gRPC symbols
+nm build-grpc-test/bin/yaze.app/Contents/MacOS/yaze | grep grpc | head -5
+# Should show symbols like: _grpc_completion_queue_create
+
+# Check binary size
+ls -lh build-grpc-test/bin/yaze.app/Contents/MacOS/yaze
+# Should be ~60-70MB (vs ~50MB without gRPC)
+```
+
+**Rollback**: If anything goes wrong:
+```bash
+# Delete test build, use original
+rm -rf build-grpc-test
+cmake --build build --target yaze -j8  # Still works!
+```
+
+### Step 5: Implement Ping service (1 hour)
+
+Create `src/app/core/imgui_test_harness_service.h`:
+
+```cpp
+#ifndef YAZE_APP_CORE_IMGUI_TEST_HARNESS_SERVICE_H_
+#define YAZE_APP_CORE_IMGUI_TEST_HARNESS_SERVICE_H_
+
+#ifdef YAZE_WITH_GRPC
+
+#include <memory>
+#include <grpcpp/grpcpp.h>
+#include "proto/imgui_test_harness.grpc.pb.h"
+#include "absl/status/status.h"
+
+namespace yaze {
+namespace test {
+
+// Implementation of ImGuiTestHarness gRPC service
+class ImGuiTestHarnessServiceImpl final 
+    : public ImGuiTestHarness::Service {
+ public:
+  grpc::Status Ping(
+      grpc::ServerContext* context,
+      const PingRequest* request,
+      PingResponse* response) override;
+};
+
+// Singleton server managing the gRPC service
+class ImGuiTestHarnessServer {
+ public:
+  static ImGuiTestHarnessServer& Instance();
+  
+  // Start server on specified port (default 50051)
+  absl::Status Start(int port = 50051);
+  
+  // Shutdown server gracefully
+  void Shutdown();
+  
+  // Check if server is running
+  bool IsRunning() const { return server_ != nullptr; }
+  
+  int Port() const { return port_; }
+  
+ private:
+  ImGuiTestHarnessServer() = default;
+  ~ImGuiTestHarnessServer() { Shutdown(); }
+  
+  std::unique_ptr<grpc::Server> server_;
+  ImGuiTestHarnessServiceImpl service_;
+  int port_ = 0;
+};
+
+}  // namespace test
+}  // namespace yaze
+
+#endif  // YAZE_WITH_GRPC
+#endif  // YAZE_APP_CORE_IMGUI_TEST_HARNESS_SERVICE_H_
+```
+
+Create `src/app/core/imgui_test_harness_service.cc`:
+
+```cpp
+#include "app/core/imgui_test_harness_service.h"
+
+#ifdef YAZE_WITH_GRPC
+
+#include <chrono>
+#include "absl/strings/str_format.h"
+
+namespace yaze {
+namespace test {
+
+// Implement Ping RPC
+grpc::Status ImGuiTestHarnessServiceImpl::Ping(
+    grpc::ServerContext* context,
+    const PingRequest* request,
+    PingResponse* response) {
+  
+  // Echo back the message
+  response->set_message(
+      absl::StrFormat("Pong: %s", request->message()));
+  
+  // Add timestamp
+  auto now = std::chrono::system_clock::now();
+  auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(
+      now.time_since_epoch());
+  response->set_timestamp_ms(ms.count());
+  
+  return grpc::Status::OK;
+}
+
+// Singleton instance
+ImGuiTestHarnessServer& ImGuiTestHarnessServer::Instance() {
+  static ImGuiTestHarnessServer* instance = new ImGuiTestHarnessServer();
+  return *instance;
+}
+
+absl::Status ImGuiTestHarnessServer::Start(int port) {
+  if (server_) {
+    return absl::FailedPreconditionError("Server already running");
+  }
+  
+  std::string server_address = absl::StrFormat("127.0.0.1:%d", port);
+  
+  grpc::ServerBuilder builder;
+  
+  // Listen on localhost only (security)
+  builder.AddListeningPort(server_address, 
+                          grpc::InsecureServerCredentials());
+  
+  // Register service
+  builder.RegisterService(&service_);
+  
+  // Build and start
+  server_ = builder.BuildAndStart();
+  
+  if (!server_) {
+    return absl::InternalError(
+        absl::StrFormat("Failed to start gRPC server on %s", 
+                       server_address));
+  }
+  
+  port_ = port;
+  
+  std::cout << "✓ ImGuiTestHarness gRPC server listening on " 
+            << server_address << "\n";
+  
+  return absl::OkStatus();
+}
+
+void ImGuiTestHarnessServer::Shutdown() {
+  if (server_) {
+    server_->Shutdown();
+    server_.reset();
+    port_ = 0;
+    std::cout << "✓ ImGuiTestHarness gRPC server stopped\n";
+  }
+}
+
+}  // namespace test
+}  // namespace yaze
+
+#endif  // YAZE_WITH_GRPC
+```
+
+Update `src/CMakeLists.txt` or `src/app/app.cmake`:
+
+```cmake
+if(YAZE_WITH_GRPC)
+  # Generate gRPC code from .proto
+  yaze_add_grpc_proto(yaze 
+    ${CMAKE_CURRENT_SOURCE_DIR}/app/core/proto/imgui_test_harness.proto)
+  
+  # Add service implementation
+  target_sources(yaze PRIVATE
+    app/core/imgui_test_harness_service.cc
+    app/core/imgui_test_harness_service.h)
+  
+  # Link gRPC libraries
+  target_link_libraries(yaze PRIVATE
+    gRPC::grpc++
+    gRPC::grpc++_reflection
+    protobuf::libprotobuf)
+  
+  # Add compile definition
+  target_compile_definitions(yaze PRIVATE YAZE_WITH_GRPC)
+endif()
+```
+
+### Step 6: Test with grpcurl (15 min)
+
+```bash
+# Rebuild with service implementation
+cmake --build build --target yaze -j8
+
+# Start YAZE with --enable-test-harness flag
+# (You'll need to add this flag handler in main.cc first)
+./build/bin/yaze.app/Contents/MacOS/yaze --enable-test-harness &
+
+# Install grpcurl if not already
+brew install grpcurl
+
+# Test the Ping RPC
+grpcurl -plaintext -d '{"message": "Hello from grpcurl"}' \
+  127.0.0.1:50051 yaze.test.ImGuiTestHarness/Ping
+
+# Expected output:
+# {
+#   "message": "Pong: Hello from grpcurl",
+#   "timestamp_ms": "1696204800000"
+# }
+```
+
+**Success Criteria**: You should see the Pong response with a timestamp!
+
+### Step 7: Add --enable-test-harness flag (30 min)
+
+Add to `src/app/main.cc`:
+
+```cpp
+#include "absl/flags/flag.h"
+
+#ifdef YAZE_WITH_GRPC
+#include "app/core/imgui_test_harness_service.h"
+
+ABSL_FLAG(bool, enable_test_harness, false,
+          "Start gRPC test harness server for automated testing");
+ABSL_FLAG(int, test_harness_port, 50051,
+          "Port for gRPC test harness (default 50051)");
+#endif
+
+// In main() after SDL/ImGui initialization:
+#ifdef YAZE_WITH_GRPC
+  if (absl::GetFlag(FLAGS_enable_test_harness)) {
+    auto& harness = yaze::test::ImGuiTestHarnessServer::Instance();
+    auto status = harness.Start(absl::GetFlag(FLAGS_test_harness_port));
+    if (!status.ok()) {
+      std::cerr << "Failed to start test harness: " 
+                << status.message() << "\n";
+      return 1;
+    }
+  }
+#endif
+```
+
+### Step 8: Implement Click handler (2 hours)
+
+Extend `.proto`:
+
+```protobuf
+service ImGuiTestHarness {
+  rpc Ping(PingRequest) returns (PingResponse);
+  rpc Click(ClickRequest) returns (ClickResponse);  // NEW
+}
+
+message ClickRequest {
+  string target = 1;  // e.g. "button:Open ROM"
+  ClickType type = 2;
+  
+  enum ClickType {
+    LEFT = 0;
+    RIGHT = 1;
+    DOUBLE = 2;
+  }
+}
+
+message ClickResponse {
+  bool success = 1;
+  string message = 2;
+  int32 execution_time_ms = 3;
+}
+```
+
+Implement in service:
+
+```cpp
+grpc::Status ImGuiTestHarnessServiceImpl::Click(
+    grpc::ServerContext* context,
+    const ClickRequest* request,
+    ClickResponse* response) {
+  
+  auto start = std::chrono::steady_clock::now();
+  
+  // Parse target: "button:Open ROM" -> type=button, label="Open ROM"
+  std::string target = request->target();
+  size_t colon_pos = target.find(':');
+  
+  if (colon_pos == std::string::npos) {
+    response->set_success(false);
+    response->set_message("Invalid target format. Use 'type:label'");
+    return grpc::Status::OK;
+  }
+  
+  std::string widget_type = target.substr(0, colon_pos);
+  std::string widget_label = target.substr(colon_pos + 1);
+  
+  // TODO: Integrate with ImGuiTestEngine
+  // For now, just simulate success
+  
+  auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(
+      std::chrono::steady_clock::now() - start);
+  
+  response->set_success(true);
+  response->set_message(
+      absl::StrFormat("Clicked %s '%s'", widget_type, widget_label));
+  response->set_execution_time_ms(elapsed.count());
+  
+  return grpc::Status::OK;
+}
+```
+
+## Next Steps After Prototype
+
+Once you have Ping + Click working:
+
+1. **Add remaining operations** (Type, Wait, Assert, Screenshot) - 3-4 hours
+2. **CLI integration** (`z3ed agent test`) - 2 hours
+3. **Windows testing** - 2-3 hours
+4. **Documentation** - 1 hour
+
+## Windows Testing Checklist
+
+For Windows contributors:
+
+```powershell
+# Install vcpkg
+git clone https://github.com/Microsoft/vcpkg.git C:\vcpkg
+C:\vcpkg\bootstrap-vcpkg.bat
+C:\vcpkg\vcpkg integrate install
+
+# Install dependencies
+C:\vcpkg\vcpkg install grpc:x64-windows protobuf:x64-windows
+
+# Build YAZE
+cmake -B build -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake ^
+  -DYAZE_WITH_GRPC=ON -A x64
+cmake --build build --config Release
+
+# Test
+.\build\bin\Release\yaze.exe --enable-test-harness
+```
+
+## Troubleshooting
+
+### "gRPC not found"
+```bash
+vcpkg install grpc:arm64-osx  # or x64-osx, x64-windows
+vcpkg integrate install
+```
+
+### "protoc not found"
+```bash
+vcpkg install protobuf:arm64-osx
+export PATH=$PATH:$(vcpkg list protobuf | grep 'tools' | cut -d: -f1)/tools/protobuf
+```
+
+### Build errors on Windows
+- Use Developer Command Prompt for Visual Studio
+- Ensure CMake 3.20+
+- Try clean build: `rmdir /s /q build`
+
+## Success Metrics
+
+✅ **Phase 1 Complete When**:
+- [ ] gRPC builds without errors
+- [ ] Ping RPC responds via grpcurl
+- [ ] YAZE starts with `--enable-test-harness` flag
+
+✅ **Phase 2 Complete When**:
+- [ ] Click RPC simulates button click
+- [ ] Type RPC sends text input
+- [ ] Wait RPC polls for conditions
+- [ ] Assert RPC validates state
+
+✅ **Phase 3 Complete When**:
+- [ ] Windows build succeeds
+- [ ] Windows contributor can test
+- [ ] CI job runs on Windows
+
+---
+
+**Estimated Total**: 10-14 hours  
+**Current Status**: Ready to start Step 1  
+**Next Session**: Add gRPC to vcpkg.json and build