yaze/docs/internal/agents/archive/wasm-planning-2025/wasm-widget-tracking-implementation.md

WASM Widget Tracking Implementation

Date: 2025-11-24 Author: Claude (AI Agent) Status: Implemented Related Files:

• /Users/scawful/Code/yaze/src/app/platform/wasm/wasm_control_api.cc
• /Users/scawful/Code/yaze/src/app/gui/automation/widget_id_registry.h
• /Users/scawful/Code/yaze/src/app/gui/automation/widget_measurement.h
• /Users/scawful/Code/yaze/src/app/controller.cc

Overview

This document describes the implementation of actual ImGui widget bounds tracking for GUI automation in the YAZE WASM build. The system replaces placeholder hardcoded bounds with real-time widget position data from the WidgetIdRegistry.

Problem Statement

The original WasmControlApi::GetUIElementTree() and GetUIElementBounds() implementations returned hardcoded placeholder bounds:

// OLD: Hardcoded placeholder
elem["bounds"] = {{"x", 0}, {"y", 0}, {"width", 100}, {"height", 30}};

This prevented accurate GUI automation, as agents and test frameworks couldn't reliably click on or query widget positions.

Solution Architecture

1. Existing Infrastructure (Already in Place)

YAZE already had a comprehensive widget tracking system:

• WidgetIdRegistry (src/app/gui/automation/widget_id_registry.h): Centralized registry that tracks all ImGui widgets with their bounds, visibility, and state
• WidgetMeasurement (src/app/gui/automation/widget_measurement.h): Measures widget dimensions using ImGui::GetItemRectMin() and ImGui::GetItemRectMax()
• Frame lifecycle hooks: BeginFrame() and EndFrame() calls already integrated in Controller::OnLoad() (lines 96-98)

2. Integration with WASM Control API

The implementation connects the WASM API to the existing widget registry:

Updated GetUIElementTree()

File: /Users/scawful/Code/yaze/src/app/platform/wasm/wasm_control_api.cc (lines 1386-1433)

std::string WasmControlApi::GetUIElementTree() {
  nlohmann::json result;

  if (!IsReady()) {
    result["error"] = "Control API not initialized";
    result["elements"] = nlohmann::json::array();
    return result.dump();
  }

  // Query the WidgetIdRegistry for all registered widgets
  auto& registry = gui::WidgetIdRegistry::Instance();
  const auto& all_widgets = registry.GetAllWidgets();

  nlohmann::json elements = nlohmann::json::array();

  // Convert WidgetInfo to JSON elements
  for (const auto& [path, info] : all_widgets) {
    nlohmann::json elem;
    elem["id"] = info.full_path;
    elem["type"] = info.type;
    elem["label"] = info.label;
    elem["enabled"] = info.enabled;
    elem["visible"] = info.visible;
    elem["window"] = info.window_name;

    // Add bounds if available
    if (info.bounds.valid) {
      elem["bounds"] = {
        {"x", info.bounds.min_x},
        {"y", info.bounds.min_y},
        {"width", info.bounds.max_x - info.bounds.min_x},
        {"height", info.bounds.max_y - info.bounds.min_y}
      };
    } else {
      elem["bounds"] = {
        {"x", 0}, {"y", 0}, {"width", 0}, {"height", 0}
      };
    }

    // Add metadata
    if (!info.description.empty()) {
      elem["description"] = info.description;
    }
    elem["imgui_id"] = static_cast<uint32_t>(info.imgui_id);
    elem["last_seen_frame"] = info.last_seen_frame;

    elements.push_back(elem);
  }

  result["elements"] = elements;
  result["count"] = elements.size();
  result["source"] = "WidgetIdRegistry";

  return result.dump();
}

Changes:

• Removed hardcoded editor-specific element generation
• Queries WidgetIdRegistry::GetAllWidgets() for real widget data
• Returns actual bounds from info.bounds if valid
• Includes metadata: imgui_id, last_seen_frame, description
• Adds source: "WidgetIdRegistry" to JSON for debugging

Updated GetUIElementBounds()

File: /Users/scawful/Code/yaze/src/app/platform/wasm/wasm_control_api.cc (lines ~1435+)

std::string WasmControlApi::GetUIElementBounds(const std::string& element_id) {
  nlohmann::json result;

  if (!IsReady()) {
    result["error"] = "Control API not initialized";
    return result.dump();
  }

  // Query the WidgetIdRegistry for the specific widget
  auto& registry = gui::WidgetIdRegistry::Instance();
  const auto* widget_info = registry.GetWidgetInfo(element_id);

  result["id"] = element_id;

  if (widget_info == nullptr) {
    result["found"] = false;
    result["error"] = "Element not found: " + element_id;
    return result.dump();
  }

  result["found"] = true;
  result["visible"] = widget_info->visible;
  result["enabled"] = widget_info->enabled;
  result["type"] = widget_info->type;
  result["label"] = widget_info->label;
  result["window"] = widget_info->window_name;

  // Add bounds if available
  if (widget_info->bounds.valid) {
    result["x"] = widget_info->bounds.min_x;
    result["y"] = widget_info->bounds.min_y;
    result["width"] = widget_info->bounds.max_x - widget_info->bounds.min_x;
    result["height"] = widget_info->bounds.max_y - widget_info->bounds.min_y;
    result["bounds_valid"] = true;
  } else {
    result["x"] = 0;
    result["y"] = 0;
    result["width"] = 0;
    result["height"] = 0;
    result["bounds_valid"] = false;
  }

  // Add metadata
  result["imgui_id"] = static_cast<uint32_t>(widget_info->imgui_id);
  result["last_seen_frame"] = widget_info->last_seen_frame;

  if (!widget_info->description.empty()) {
    result["description"] = widget_info->description;
  }

  return result.dump();
}

Changes:

• Removed hardcoded element ID pattern matching
• Queries WidgetIdRegistry::GetWidgetInfo(element_id) for specific widget
• Returns found: false if widget doesn't exist
• Returns actual bounds with bounds_valid flag
• Includes full widget metadata

3. Frame Lifecycle Integration

File: /Users/scawful/Code/yaze/src/app/controller.cc (lines 96-98)

The widget registry is already integrated into the main render loop:

absl::Status Controller::OnLoad() {
  // ... ImGui::NewFrame() setup ...

  gui::WidgetIdRegistry::Instance().BeginFrame();
  absl::Status update_status = editor_manager_.Update();
  gui::WidgetIdRegistry::Instance().EndFrame();

  RETURN_IF_ERROR(update_status);
  return absl::OkStatus();
}

Frame Lifecycle:

1. BeginFrame(): Resets seen_in_current_frame flag for all widgets
2. Widget rendering: Editors register widgets during editor_manager_.Update()
3. EndFrame(): Marks unseen widgets as invisible, prunes stale entries

4. Widget Registration (Future Work)

Current State: Widget registration infrastructure exists but editors are not yet registering widgets.

Registration Pattern (to be implemented in editors):

// Example: Dungeon Editor registering a card
{
  gui::WidgetIdScope scope("DungeonEditor");

  if (ImGui::Begin("Room Selector##dungeon")) {
    // Widget now has full path: "DungeonEditor/Room Selector"

    if (ImGui::Button("Load Room")) {
      // After rendering button, register it
      gui::WidgetIdRegistry::Instance().RegisterWidget(
        scope.GetWidgetPath("button", "Load Room"),
        "button",
        ImGui::GetItemID(),
        "Loads the selected room into the editor"
      );
    }
  }
  ImGui::End();
}

Macros Available:

• YAZE_WIDGET_SCOPE(name): RAII scope for hierarchical widget paths
• YAZE_REGISTER_WIDGET(type, name): Register widget after rendering
• YAZE_REGISTER_CURRENT_WIDGET(type): Auto-extract widget name from ImGui

API Usage

JavaScript API

Get All UI Elements:

const elements = window.yaze.control.getUIElementTree();
console.log(elements);
// Output:
// {
//   "elements": [
//     {
//       "id": "DungeonEditor/RoomSelector/button:LoadRoom",
//       "type": "button",
//       "label": "Load Room",
//       "visible": true,
//       "enabled": true,
//       "window": "DungeonEditor",
//       "bounds": {"x": 150, "y": 200, "width": 100, "height": 30},
//       "imgui_id": 12345,
//       "last_seen_frame": 4567
//     }
//   ],
//   "count": 1,
//   "source": "WidgetIdRegistry"
// }

Get Specific Widget Bounds:

const bounds = window.yaze.control.getUIElementBounds("DungeonEditor/RoomSelector/button:LoadRoom");
console.log(bounds);
// Output:
// {
//   "id": "DungeonEditor/RoomSelector/button:LoadRoom",
//   "found": true,
//   "visible": true,
//   "enabled": true,
//   "type": "button",
//   "label": "Load Room",
//   "window": "DungeonEditor",
//   "x": 150,
//   "y": 200,
//   "width": 100,
//   "height": 30,
//   "bounds_valid": true,
//   "imgui_id": 12345,
//   "last_seen_frame": 4567
// }

Performance Considerations

1. Memory: WidgetIdRegistry stores widget metadata in std::unordered_map, which grows with UI complexity. Stale widgets are pruned after 600 frames of inactivity.

2. CPU Overhead:

   • BeginFrame(): O(n) iteration to reset flags (n = number of widgets)
   • Widget registration: O(1) hash map lookup/insert
   • EndFrame(): O(n) iteration for pruning stale entries

3. Optimization: Widget measurement can be disabled globally:

   gui::WidgetMeasurement::Instance().SetEnabled(false);
   

Testing

Manual Test (WASM Build):

# Build WASM
./scripts/build-wasm.sh

# Serve locally
cd build-wasm
python3 -m http.server 8080

# Open browser console
window.yaze.control.getUIElementTree();

Expected Behavior:

• Initially, elements array will be empty (no widgets registered yet)
• After editors implement registration, widgets will appear with real bounds
• bounds_valid: false for widgets not yet rendered in current frame

Next Steps

1. Add widget registration to editors:

   • DungeonEditorV2: Register room tabs, cards, buttons
   • OverworldEditor: Register canvas, tile selectors, property panels
   • GraphicsEditor: Register graphics sheets, palette pickers

2. Add registration helpers:

   • Create AgentUI::RegisterButton(), AgentUI::RegisterCard() wrappers
   • Auto-register common widget patterns (cards with visibility flags)

3. Extend API:

   • FindWidgetsByPattern(pattern): Search widgets by regex
   • ClickWidget(element_id): Simulate click via automation API

References

• Widget ID Registry: /Users/scawful/Code/yaze/src/app/gui/automation/widget_id_registry.h
• Widget Measurement: /Users/scawful/Code/yaze/src/app/gui/automation/widget_measurement.h
• WASM Control API: /Users/scawful/Code/yaze/src/app/platform/wasm/wasm_control_api.h
• Controller Integration: /Users/scawful/Code/yaze/src/app/controller.cc (lines 96-98)

Revision History

Date	Author	Changes
2025-11-24	Claude	Initial implementation and documentation