231 lines
8.9 KiB
Markdown
231 lines
8.9 KiB
Markdown
# AI Infrastructure Improvements Plan
|
|
|
|
**Branch:** `feature/ai-infra-improvements`
|
|
**Created:** 2025-11-21
|
|
**Status:** Planning
|
|
|
|
## Overview
|
|
|
|
This document outlines the gaps in yaze's AI infrastructure (gRPC services, MCP integration) and the planned improvements to make yaze-mcp fully functional for AI-assisted SNES ROM hacking.
|
|
|
|
## Current State Analysis
|
|
|
|
### yaze-mcp Tools (28 Emulator + 10 ROM = 38 total)
|
|
|
|
| Category | Tools | Status |
|
|
|----------|-------|--------|
|
|
| Emulator Lifecycle | pause, resume, reset | Working |
|
|
| Execution Control | step_instruction, run_to_breakpoint | Working |
|
|
| Execution Control | step_over, step_out | Partial (step_over falls back to single-step) |
|
|
| Memory | read_memory, write_memory, get_game_state | Working |
|
|
| Breakpoints | add/remove/list/toggle | Working (execute only during emulation) |
|
|
| Watchpoints | add/remove/list/history | NOT WIRED (only called from RPC handlers) |
|
|
| Disassembly | get_disassembly | Partial (returns OPCODE_XX placeholders) |
|
|
| Tracing | get_execution_trace | BROKEN (returns FAILED_PRECONDITION) |
|
|
| Symbols | load/resolve/get_symbol_at | DISABLED (ASAR integration disabled) |
|
|
| ROM Basic | get_rom_info, read_rom_bytes | Working |
|
|
| ROM Advanced | overworld/dungeon/sprite reads | NOT IMPLEMENTED (stubs return errors) |
|
|
| ROM Versioning | create/list snapshots | Working |
|
|
| Input | press_buttons | Working |
|
|
|
|
### Critical Gaps
|
|
|
|
1. **gRPC Server Not Started** - `AgentControlServer` exists but is never instantiated
|
|
2. **Watchpoints Bypass Emulation** - Only triggered by RPC read/write, not CPU bus activity
|
|
3. **Disassembly Uses Placeholders** - No proper 65816 disassembler integration
|
|
4. **Execution Trace Not Buffered** - No ring buffer for instruction history
|
|
5. **Symbols Disabled** - ASAR integration commented out
|
|
6. **ROM Domain RPCs Stubbed** - Overworld/dungeon/sprite return "not yet implemented"
|
|
|
|
## Implementation Plan
|
|
|
|
### Phase 1: gRPC Server Hosting (Priority: Critical)
|
|
|
|
**Goal:** Stand up a unified gRPC server that registers EmulatorService + RomService
|
|
|
|
**Files to Modify:**
|
|
- `src/app/editor/editor_manager.cc` - Start AgentControlServer when YAZE_ENABLE_REMOTE_AUTOMATION
|
|
- `src/cli/service/agent/agent_control_server.cc` - Register both EmulatorService and RomService
|
|
- `src/app/service/unified_grpc_server.cc` - Consider merging with AgentControlServer
|
|
|
|
**Tasks:**
|
|
- [ ] Add `StartAgentServer()` method to EditorManager
|
|
- [ ] Wire startup to `YAZE_ENABLE_REMOTE_AUTOMATION` flag or `--enable-grpc` CLI flag
|
|
- [ ] Register EmulatorService and RomService on same server
|
|
- [ ] Add configurable port (default 50051)
|
|
- [ ] Test with yaze-mcp `check_status`
|
|
|
|
### Phase 2: Emulator Debug RPCs (Priority: High)
|
|
|
|
**Goal:** Flesh out disassembly, execution trace, and stepping
|
|
|
|
**2a. Proper Disassembly**
|
|
- Use DisassemblyViewer's existing instruction recording
|
|
- Or integrate a standalone 65816 disassembler (bsnes style)
|
|
- File: `src/cli/service/agent/emulator_service_impl.cc` lines 661-705
|
|
|
|
**2b. Execution Trace Buffer**
|
|
- Add ring buffer (1000-10000 entries) to DisassemblyViewer
|
|
- Record: address, opcode, operands, cycle count, register snapshot
|
|
- File: `src/app/emu/debug/disassembly_viewer.h`
|
|
|
|
**2c. StepOver Implementation**
|
|
- Detect JSR (0x20) and JSL (0x22) opcodes
|
|
- Set temporary breakpoint at return address (PC + instruction length)
|
|
- Run until breakpoint hit, then remove temporary BP
|
|
- File: `src/cli/service/agent/emulator_service_impl.cc` lines 598-607
|
|
|
|
**Tasks:**
|
|
- [ ] Integrate real 65816 disassembly into GetDisassembly RPC
|
|
- [ ] Add ExecutionTraceBuffer class with ring buffer
|
|
- [ ] Implement GetExecutionTrace from buffer
|
|
- [ ] Implement proper StepOver with JSR/JSL detection
|
|
|
|
### Phase 3: Breakpoint/Watchpoint Memory Integration (Priority: High)
|
|
|
|
**Goal:** Wire memory breakpoints and watchpoints into emulator memory bus
|
|
|
|
**Current State:**
|
|
- `BreakpointManager::ShouldBreakOnExecute()` IS called via CPU callback
|
|
- `BreakpointManager::ShouldBreakOnMemoryAccess()` IS NOT called during emulation
|
|
- `WatchpointManager::OnMemoryAccess()` IS NOT called during emulation
|
|
|
|
**Files to Modify:**
|
|
- `src/app/emu/snes.h` - Add read/write callbacks
|
|
- `src/app/emu/snes.cc` - Invoke breakpoint/watchpoint managers in CpuRead/CpuWrite
|
|
- `src/app/emu/emulator.cc` - Wire managers to callbacks
|
|
|
|
**Implementation:**
|
|
```cpp
|
|
// In Snes::CpuRead() or via callback:
|
|
if (debugging_enabled_) {
|
|
if (breakpoint_manager_.ShouldBreakOnMemoryAccess(addr, BreakpointManager::AccessType::READ)) {
|
|
running_ = false;
|
|
}
|
|
watchpoint_manager_.OnMemoryAccess(addr, /*is_write=*/false, value);
|
|
}
|
|
|
|
// In Snes::CpuWrite() or via callback:
|
|
if (debugging_enabled_) {
|
|
if (breakpoint_manager_.ShouldBreakOnMemoryAccess(addr, BreakpointManager::AccessType::WRITE)) {
|
|
running_ = false;
|
|
}
|
|
watchpoint_manager_.OnMemoryAccess(addr, /*is_write=*/true, value);
|
|
}
|
|
```
|
|
|
|
**Tasks:**
|
|
- [ ] Add `on_memory_read_` and `on_memory_write_` callbacks to CPU
|
|
- [ ] Invoke BreakpointManager from callbacks
|
|
- [ ] Invoke WatchpointManager from callbacks
|
|
- [ ] Add MCP tools for watchpoints: `add_watchpoint`, `list_watchpoints`, `get_watchpoint_history`
|
|
- [ ] Test memory breakpoints and watchpoints with yaze-mcp
|
|
|
|
### Phase 4: Symbol Loading & Resolution (Priority: Medium)
|
|
|
|
**Goal:** Load ASAR/WLA-DX/CA65 symbol files and enable label resolution
|
|
|
|
**Current State:**
|
|
- EmulatorServiceImpl has stubbed symbol methods returning "not available"
|
|
- ASAR wrapper exists in `src/core/asar_wrapper.h`
|
|
|
|
**Implementation Approach:**
|
|
1. Create `SymbolTable` class to store symbols (name -> address map)
|
|
2. Implement parsers for each format:
|
|
- ASAR: `.sym` files with `label = $XXXXXX` format
|
|
- WLA-DX: `.sym` files with different format
|
|
- CA65: `.dbg` or `.map` files
|
|
- Mesen: `.mlb` label files
|
|
3. Wire LoadSymbols RPC to parse and populate SymbolTable
|
|
4. Wire ResolveSymbol/GetSymbolAt to query SymbolTable
|
|
|
|
**Files to Create:**
|
|
- `src/app/emu/debug/symbol_table.h`
|
|
- `src/app/emu/debug/symbol_table.cc`
|
|
- `src/app/emu/debug/symbol_parser.h` - Format parsers
|
|
|
|
**Tasks:**
|
|
- [ ] Design SymbolTable class (bidirectional lookup)
|
|
- [ ] Implement ASAR .sym parser
|
|
- [ ] Implement WLA-DX parser
|
|
- [ ] Wire to EmulatorServiceImpl
|
|
- [ ] Test with Oracle of Secrets symbols
|
|
|
|
### Phase 5: ROM Domain RPCs (Priority: Medium)
|
|
|
|
**Goal:** Implement overworld/dungeon/sprite read/write RPCs
|
|
|
|
**Current State:**
|
|
- All domain RPCs return "not yet implemented"
|
|
- ROM class has raw access, but not structured zelda3 data
|
|
|
|
**Implementation:**
|
|
- Leverage `zelda3::Overworld`, `zelda3::Dungeon`, `zelda3::Sprite` classes
|
|
- Need to instantiate these in RomServiceImpl or get from shared state
|
|
|
|
**Files to Modify:**
|
|
- `src/app/net/rom_service_impl.cc` - Implement ReadOverworldMap, ReadDungeonRoom, ReadSprite
|
|
- Proto messages already defined in `rom_service.proto`
|
|
|
|
**Tasks:**
|
|
- [ ] Add zelda3::Overworld access to RomServiceImpl
|
|
- [ ] Implement ReadOverworldMap (tile16 data for 160 maps)
|
|
- [ ] Implement WriteOverworldTile
|
|
- [ ] Add zelda3::Dungeon access
|
|
- [ ] Implement ReadDungeonRoom (tile16 data for 296 rooms)
|
|
- [ ] Implement WriteDungeonTile
|
|
- [ ] Implement ReadSprite
|
|
|
|
### Phase 6: yaze-mcp Enhancements (Priority: Low)
|
|
|
|
**Goal:** Improve MCP error handling and add missing tools
|
|
|
|
**Tasks:**
|
|
- [ ] Add timeout/retry logic based on gRPC status codes
|
|
- [ ] Add clearer error messages for unimplemented RPCs
|
|
- [ ] Add watchpoint tools to server.py
|
|
- [ ] Document required build preset and port
|
|
- [ ] Add connection health monitoring
|
|
|
|
## File Reference
|
|
|
|
### Emulator Service
|
|
- **Header:** `src/cli/service/agent/emulator_service_impl.h`
|
|
- **Implementation:** `src/cli/service/agent/emulator_service_impl.cc` (822 lines)
|
|
- **Server:** `src/cli/service/agent/agent_control_server.cc`
|
|
|
|
### ROM Service
|
|
- **Header:** `src/app/net/rom_service_impl.h`
|
|
- **Implementation:** `src/app/net/rom_service_impl.cc`
|
|
- **Version Manager:** `src/app/net/rom_version_manager.h`
|
|
|
|
### Debug Managers
|
|
- **Breakpoints:** `src/app/emu/debug/breakpoint_manager.h|cc`
|
|
- **Watchpoints:** `src/app/emu/debug/watchpoint_manager.h|cc`
|
|
- **Disassembly:** `src/app/emu/debug/disassembly_viewer.h`
|
|
|
|
### Emulator Core
|
|
- **Emulator:** `src/app/emu/emulator.h|cc`
|
|
- **SNES:** `src/app/emu/snes.h|cc`
|
|
- **CPU:** `src/app/emu/cpu/cpu.h`
|
|
|
|
### MCP Server
|
|
- **Location:** `/Users/scawful/Code/yaze-mcp/server.py`
|
|
- **Proto Stubs:** `/Users/scawful/Code/yaze-mcp/protos/`
|
|
|
|
## Success Criteria
|
|
|
|
1. **yaze-mcp `check_status`** connects and returns full emulator state
|
|
2. **Memory breakpoints** pause emulation on WRAM/SRAM access
|
|
3. **Watchpoints** track and log all memory accesses in specified ranges
|
|
4. **`get_disassembly`** returns proper 65816 mnemonics
|
|
5. **`get_execution_trace`** returns last N instructions executed
|
|
6. **Symbol loading** works with ASAR output from Oracle of Secrets
|
|
7. **ROM domain RPCs** return structured overworld/dungeon/sprite data
|
|
|
|
## Notes
|
|
|
|
- Consider performance impact of memory access callbacks (may need optimization)
|
|
- May want debug mode toggle to enable/disable expensive instrumentation
|
|
- Future: Canvas automation service for GUI automation via MCP
|