feat: Implement widget discovery feature in GUI automation

- Added `DiscoverWidgets` RPC to the ImGuiTestHarness service for enumerating GUI widgets.
- Introduced `WidgetDiscoveryService` to handle widget collection and filtering based on various criteria.
- Updated `agent gui discover` command to support new options for filtering and output formats.
- Enhanced `GuiAutomationClient` to facilitate widget discovery requests and responses.
- Added necessary protobuf messages for widget discovery in `imgui_test_harness.proto`.
- Updated CLI command handling to include new GUI discovery functionality.
- Improved documentation for the `agent gui discover` command with examples and output formats.
This commit is contained in:
scawful
2025-10-02 16:56:15 -04:00
parent 3944861b38
commit 22f0e5006b
11 changed files with 947 additions and 31 deletions

View File

@@ -248,44 +248,54 @@ Examples:
##### `agent gui discover` - Enumerate available widgets
```bash
z3ed agent gui discover [--window <name>] [--type <widget_type>] [--format <json|yaml>]
z3ed agent gui discover \
[--host <name>] [--port <port>] \
[--window <name>] [--path-prefix <path>] \
[--type <widget_type>] [--include-invisible] [--include-disabled] \
[--format <table|json>] [--limit <n>]
Options:
--window <name> Filter by window name (e.g. "Overworld")
--type <type> Filter by widget type: button, input, menu, tab, checkbox
--format <format> Output format: json or yaml (default: yaml)
--host <name> Harness host (default: localhost)
--port <port> Harness port (default: 50052)
--window <name> Filter by window name (case-insensitive substring)
--path-prefix <path> Require widget path to start with prefix
--type <type> Filter widget type: button, input, menu, tab,
checkbox, slider, canvas, selectable, other
--include-invisible Include widgets whose parent window is hidden
--include-disabled Include widgets flagged as disabled
--format <mode> Output as `table` (default) or `json`
--limit <n> Maximum widgets to display (useful for large UIs)
Examples:
# Discover all widgets
# Discover all widgets currently registered
z3ed agent gui discover
# Find all buttons in Overworld editor
# Focus on buttons inside the Overworld editor window
z3ed agent gui discover --window "Overworld" --type button
# Get JSON for AI consumption
z3ed agent gui discover --format json > widgets.json
# Export a JSON snapshot for an automation agent (showing first 50 widgets)
z3ed agent gui discover --format json --limit 50 > widgets.json
```
**Output Example**:
```yaml
windows:
- name: Main Window
visible: true
widgets:
- id: menu_file
label: File
type: menu
enabled: true
suggested_action: "Click menuitem: File"
- name: Overworld
visible: true
widgets:
- id: btn_save
label: Save
type: button
enabled: true
position: "10,20,100,30"
suggested_action: "Click button:Save"
**Table Output Example**:
```
=== Widget Discovery ===
Server: localhost:50052
Window filter: Overworld
Type filter: button
Include invisible: no
Include disabled: no
Window: Overworld (visible)
• [button] Save
Path: Overworld/Toolbar/button:Save
Suggested: Click button:Save
State: visible, enabled
Bounds: (24.0, 64.0) → (112.0, 92.0)
Widget ID: 0x13fc41a2
Widgets shown: 3 of 18 (truncated)
Snapshot: 2025-01-16 19:42:05
```
**Use Cases**: