Enhance z3ed Agent Roadmap and CLI Reference with New Tool Commands and Chat Features

This commit is contained in:
scawful
2025-10-03 13:51:01 -04:00
parent ba9f6533a4
commit c7a7707d25
7 changed files with 268 additions and 49 deletions

View File

@@ -108,13 +108,12 @@ We have made significant progress in laying the foundation for the conversationa
- **GUI Chat Widget Stub**: An `AgentChatWidget` is integrated into the main GUI.
- **Initial Agent "Tools"**: `resource-list` and `dungeon-list-sprites` commands are implemented.
- **Tool Use Foundation**: The `ToolDispatcher` is implemented, and the AI services are aware of the new tool call format.
- **Tool Loop Improvements**: Conversational flow now handles multi-step tool calls with default JSON output, allowing results to feed back into the chat without recursion.
### ⚠️ Current Blocker: Build Configuration
We are currently facing a linker error when building the main `yaze` application with gRPC support. The `ToolDispatcher` is unable to find the definitions for the `HandleResourceListCommand` and `HandleDungeonListSpritesCommand` functions.
**Root Cause**: These handler functions are only compiled as part of the `z3ed` target, not the `yaze` target. The `ToolDispatcher`, which is now included in the `yaze` build, depends on them.
### Build Configuration Issue Resolved
The linker error is fixed. Both the CLI and GUI targets now link against `yaze_agent`, so the shared agent handlers (`HandleResourceListCommand`, `HandleDungeonListSpritesCommand`, etc.) compile once and are available to `ToolDispatcher` everywhere.
### 🚀 Next Steps
1. **Resolve the Build Issue**: The immediate next step is to fix the linker error. This will likely involve a thoughtful refactoring of our CMake configuration to better share sources between the `yaze` and `z3ed` targets.
2. **Simplify CMake Structure**: As discussed, the current structure of including `.cmake` files from various subdirectories is becoming difficult to manage. We should consider flattening this into a more centralized source list in the main `src/CMakeLists.txt`.
3. **Continue with Tool Integration**: Once the build is fixed, we can proceed with integrating the tool execution results back into the conversational loop.
1. **Share ROM Context with the Agent**: Inject the active GUI ROM into `ConversationalAgentService` so tool calls work even when `--rom` flags are unavailable.
2. **Surface Tool Output in the UI**: Present JSON/table responses in the chat widgets with formatting instead of raw text dumps.
3. **Expand Tool Coverage**: Add the next batch of read-only utilities (`dungeon get-info`, `overworld find-tile`) now that the tooling loop is stable.

View File

@@ -207,6 +207,52 @@ Examples:
- `dungeon` - Dungeon editing
- `agent` - Agent commands
#### `agent resource-list` - Enumerate labeled resources for the AI
```bash
z3ed agent resource-list --type <resource> [--format <table|json>]
Options:
--type <resource> Required label family (dungeon, overworld, sprite, palette, etc.)
--format <mode> Output format, defaults to `table`. Use `json` for LLM tooling.
Examples:
# Show dungeon labels in a table
z3ed agent resource-list --type dungeon
# Emit JSON for the conversation agent to consume
z3ed agent resource-list --type overworld --format json
```
**Notes**:
- When the conversation agent invokes this tool, JSON output is requested automatically.
- Labels are loaded from `ResourceContextBuilder`, so the command reflects project-specific metadata.
#### `agent dungeon-list-sprites` - Inspect sprites in a dungeon room
```bash
z3ed agent dungeon-list-sprites --room <hex_id> [--format <table|json>]
Options:
--room <hex_id> Dungeon room ID (hexadecimal). Accepts `0x` prefixes or decimal.
--format <mode> Output format, defaults to `table`.
Examples:
z3ed agent dungeon-list-sprites --room 0x012
z3ed agent dungeon-list-sprites --room 18 --format json
```
**Output**:
- Table view prints sprite id/x/y in hex+decimal for quick inspection.
- JSON view is tailored for the LLM toolchain and is returned automatically during tool calls.
#### `agent chat` - Interactive terminal chat (TUI prototype)
```bash
z3ed agent chat
```
- Opens an FTXUI-based interface with scrolling history and input box.
- Uses the shared `ConversationalAgentService`, so the same backend powers the GUI widget.
- Useful for manual testing of tool dispatching and new prompting strategies.
#### `agent test` - Automated GUI testing (IT-02)
```bash
z3ed agent test --prompt "<test_description>" [--host <hostname>] [--port <port>]

View File

@@ -40,11 +40,20 @@ z3ed agent plan --prompt "Place a tree at position 10, 10 on map 0"
# Execute in sandbox with auto-approval
z3ed agent run --prompt "Create a 3x3 water pond at 15, 20" --rom zelda3.sfc --sandbox
# Chat with the agent in the terminal (FTXUI prototype)
z3ed agent chat
# List all proposals
z3ed agent list
# View proposal details
z3ed agent diff --proposal <id>
# Inspect project metadata for the LLM toolchain
z3ed agent resource-list --type dungeon --format json
# Dump sprite placements for a dungeon room
z3ed agent dungeon-list-sprites --room 0x012
```
### GUI Testing Commands
@@ -220,6 +229,11 @@ AI agent features require:
- ✅ Updated README with clear dependency requirements
- ✅ Added Windows compatibility notes
### Conversational Loop
- ✅ Tool dispatcher defaults to JSON when invoked by the agent, keeping outputs machine-readable.
- ✅ Conversational service now replays tool results without recursion, improving chat stability.
- ✅ Mock AI service issues sample tool calls so the loop can be exercised without a live LLM.
## Troubleshooting
### "OpenSSL not found" warning