feat(docs): add comprehensive AI agent architecture and debugging guides
- Introduced a new document detailing the architecture of the z3ed AI agent system, covering features like learned knowledge, TODO management, and advanced routing. - Added a debugging guide for the AI agent, outlining the gRPC-based debugging service, available tools, and practical debugging workflows. - Updated existing documentation to reflect recent improvements in the emulator's audio system and overall debugging capabilities. Benefits: - Provides clear guidance for developers on the AI agent's architecture and debugging processes, enhancing usability and understanding of the system. - Facilitates faster onboarding and better collaboration by offering structured documentation and real-world examples.
This commit is contained in:
scawful
454
docs/C3-agent-architecture.md
Normal file
454
docs/C3-agent-architecture.md
Normal file
@@ -0,0 +1,454 @@
|
||||
# C3 - z3ed Agent Architecture Guide
|
||||
|
||||
**Date**: October 12, 2025
|
||||
**Version**: v0.2.2-alpha
|
||||
**Status**: Core Features Integrated ✅
|
||||
|
||||
## Overview
|
||||
|
||||
This guide documents the architecture of the z3ed AI agent system, including learned knowledge, TODO management, advanced routing, pretraining, and agent handoff capabilities.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────────────────────────────┐
|
||||
│ User / AI Agent │
|
||||
└────────────┬──────────────────────────────────────────────────┘
|
||||
│
|
||||
│ z3ed CLI commands
|
||||
│
|
||||
┌────────────▼──────────────────────────────────────────────────┐
|
||||
│ CLI Command Router (agent.cc) │
|
||||
│ │
|
||||
│ Routes to: │
|
||||
│ ├─ agent simple-chat → SimpleChatCommand │
|
||||
│ ├─ agent learn → HandleLearnCommand │
|
||||
│ ├─ agent todo → HandleTodoCommand │
|
||||
│ ├─ agent test → HandleTestCommand │
|
||||
│ ├─ agent plan/run/diff → Proposal system │
|
||||
│ └─ emulator-* → EmulatorCommandHandler │
|
||||
└───────────┬───────────────────────────────────────────────────┘
|
||||
│
|
||||
┌───────────▼───────────────────────────────────────────────────┐
|
||||
│ ConversationalAgentService │
|
||||
│ │
|
||||
│ Integrates: │
|
||||
│ ├─ LearnedKnowledgeService (preferences, patterns, memory) │
|
||||
│ ├─ TodoManager (task tracking, dependencies) │
|
||||
│ ├─ AdvancedRouter (response enhancement) │
|
||||
│ ├─ AgentPretraining (knowledge injection) │
|
||||
│ └─ ToolDispatcher (command execution) │
|
||||
└────────────┬──────────────────────────────────────────────────┘
|
||||
│
|
||||
┌────────────▼──────────────────────────────────────────────────┐
|
||||
│ Tool Dispatcher │
|
||||
│ │
|
||||
│ Routes tool calls to: │
|
||||
│ ├─ Resource Commands (dungeon, overworld, sprites) │
|
||||
│ ├─ Emulator Commands (breakpoints, memory, step) │
|
||||
│ ├─ GUI Commands (automation, screenshots) │
|
||||
│ └─ Custom Tools (extensible via CommandHandler) │
|
||||
└────────────┬──────────────────────────────────────────────────┘
|
||||
│
|
||||
┌────────────▼──────────────────────────────────────────────────┐
|
||||
│ Command Handlers (CommandHandler base class) │
|
||||
│ │
|
||||
│ Unified pattern: │
|
||||
│ 1. Parse arguments (ArgumentParser) │
|
||||
│ 2. Get ROM context (CommandContext) │
|
||||
│ 3. Execute business logic │
|
||||
│ 4. Format output (OutputFormatter) │
|
||||
└────────────┬──────────────────────────────────────────────────┘
|
||||
│
|
||||
┌────────────▼──────────────────────────────────────────────────┐
|
||||
│ Persistent Storage │
|
||||
│ │
|
||||
│ ~/.yaze/agent/ │
|
||||
│ ├─ preferences.json (user preferences) │
|
||||
│ ├─ patterns.json (learned ROM patterns) │
|
||||
│ ├─ projects.json (project contexts) │
|
||||
│ ├─ memories.json (conversation summaries) │
|
||||
│ ├─ todos.json (task management) │
|
||||
│ └─ sessions/ (collaborative chat history) │
|
||||
└────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Feature 1: Learned Knowledge Service
|
||||
|
||||
### What It Does
|
||||
|
||||
Persists information across agent sessions:
|
||||
- **Preferences**: User's default settings (palette, tool choices)
|
||||
- **ROM Patterns**: Learned behaviors (frequently accessed rooms, sprite patterns)
|
||||
- **Project Context**: ROM-specific goals and notes
|
||||
- **Conversation Memory**: Summaries of past discussions for continuity
|
||||
|
||||
### Integration Status: ✅ Complete
|
||||
|
||||
**Files**:
|
||||
- `cli/service/agent/learned_knowledge_service.{h,cc}` - Core service
|
||||
- `cli/handlers/agent/general_commands.cc` - CLI handlers
|
||||
- `cli/handlers/agent.cc` - Routing
|
||||
|
||||
### Usage Examples
|
||||
|
||||
```bash
|
||||
# Save preference
|
||||
z3ed agent learn --preference default_palette=2
|
||||
|
||||
# Get preference
|
||||
z3ed agent learn --get-preference default_palette
|
||||
|
||||
# Save project context
|
||||
z3ed agent learn --project "myrom" --context "Vanilla+ difficulty hack"
|
||||
|
||||
# Get project details
|
||||
z3ed agent learn --get-project "myrom"
|
||||
|
||||
# Search past conversations
|
||||
z3ed agent learn --search-memories "dungeon room 5"
|
||||
|
||||
# Export all learned data
|
||||
z3ed agent learn --export learned_data.json
|
||||
|
||||
# View statistics
|
||||
z3ed agent learn --stats
|
||||
```
|
||||
|
||||
### AI Agent Integration
|
||||
|
||||
The ConversationalAgentService now:
|
||||
1. Initializes `LearnedKnowledgeService` on startup
|
||||
2. Can inject learned context into prompts (when `inject_learned_context_=true`)
|
||||
3. Can access preferences/patterns/memories during tool execution
|
||||
|
||||
**API**:
|
||||
```cpp
|
||||
ConversationalAgentService service;
|
||||
service.learned_knowledge().SetPreference("palette", "2");
|
||||
auto pref = service.learned_knowledge().GetPreference("palette");
|
||||
```
|
||||
|
||||
### Data Persistence
|
||||
|
||||
**Location**: `~/.yaze/agent/`
|
||||
**Format**: JSON
|
||||
**Files**:
|
||||
- `preferences.json` - Key-value pairs
|
||||
- `patterns.json` - Timestamped ROM patterns with confidence scores
|
||||
- `projects.json` - Project metadata and context
|
||||
- `memories.json` - Conversation summaries (last 100)
|
||||
|
||||
## Feature 2: TODO Management System
|
||||
|
||||
### What It Does
|
||||
|
||||
Enables AI agents to break down complex tasks into executable steps with dependency tracking and prioritization.
|
||||
|
||||
### Integration Status: ✅ Complete
|
||||
|
||||
**Files**:
|
||||
- `cli/service/agent/todo_manager.{h,cc}` - Core service
|
||||
- `cli/handlers/agent/todo_commands.{h,cc}` - CLI handlers
|
||||
- `cli/handlers/agent.cc` - Routing
|
||||
|
||||
### Usage Examples
|
||||
|
||||
```bash
|
||||
# Create TODO
|
||||
z3ed agent todo create "Fix input handling" --category=emulator --priority=1
|
||||
|
||||
# List TODOs
|
||||
z3ed agent todo list
|
||||
|
||||
# Filter by status
|
||||
z3ed agent todo list --status=in_progress
|
||||
|
||||
# Update status
|
||||
z3ed agent todo update 1 --status=completed
|
||||
|
||||
# Get next actionable task
|
||||
z3ed agent todo next
|
||||
|
||||
# Generate dependency-aware execution plan
|
||||
z3ed agent todo plan
|
||||
|
||||
# Clear completed
|
||||
z3ed agent todo clear-completed
|
||||
```
|
||||
|
||||
### AI Agent Integration
|
||||
|
||||
```cpp
|
||||
ConversationalAgentService service;
|
||||
service.todo_manager().CreateTodo("Debug A button", "emulator", 1);
|
||||
auto next = service.todo_manager().GetNextActionableTodo();
|
||||
```
|
||||
|
||||
### Storage
|
||||
|
||||
**Location**: `~/.yaze/agent/todos.json`
|
||||
**Format**: JSON array with dependencies:
|
||||
```json
|
||||
{
|
||||
"todos": [
|
||||
{
|
||||
"id": "1",
|
||||
"description": "Debug input handling",
|
||||
"status": "in_progress",
|
||||
"category": "emulator",
|
||||
"priority": 1,
|
||||
"dependencies": [],
|
||||
"tools_needed": ["emulator-set-breakpoint", "emulator-read-memory"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Feature 3: Advanced Routing
|
||||
|
||||
### What It Does
|
||||
|
||||
Optimizes tool responses for AI consumption with:
|
||||
- **Data type inference** (sprite data vs tile data vs palette)
|
||||
- **Pattern extraction** (repeating values, structures)
|
||||
- **Structured summaries** (high-level + detailed + next steps)
|
||||
- **GUI action generation** (converts analysis → automation script)
|
||||
|
||||
### Integration Status: ⏳ Implemented, Not Integrated
|
||||
|
||||
**Files**:
|
||||
- `cli/service/agent/advanced_routing.{h,cc}` - Implementation ✅
|
||||
- `cli/agent.cmake` - Added to build ✅
|
||||
- `cli/service/agent/conversational_agent_service.cc` - **Needs integration** ⏳
|
||||
|
||||
### How to Integrate
|
||||
|
||||
**Option 1: In ToolDispatcher (Automatic)**
|
||||
```cpp
|
||||
// In tool_dispatcher.cc, after tool execution:
|
||||
auto result = handler->Run(args, rom_context_);
|
||||
if (result.ok()) {
|
||||
std::string output = output_buffer.str();
|
||||
|
||||
// Route through advanced router for enhanced response
|
||||
AdvancedRouter::RouteContext ctx;
|
||||
ctx.rom = rom_context_;
|
||||
ctx.tool_calls_made = {call.tool_name};
|
||||
|
||||
if (call.tool_name == "hex-read") {
|
||||
auto routed = AdvancedRouter::RouteHexAnalysis(data, address, ctx);
|
||||
return absl::StrCat(routed.summary, "\n\n", routed.detailed_data);
|
||||
}
|
||||
|
||||
return output;
|
||||
}
|
||||
```
|
||||
|
||||
**Option 2: In ConversationalAgentService (Selective)**
|
||||
```cpp
|
||||
// After getting tool results, enhance the response:
|
||||
ChatMessage ConversationalAgentService::EnhanceResponse(
|
||||
const ChatMessage& response,
|
||||
const std::string& user_message) {
|
||||
|
||||
AdvancedRouter::RouteContext ctx;
|
||||
ctx.rom = rom_context_;
|
||||
ctx.user_intent = user_message;
|
||||
|
||||
// Use advanced router to synthesize multi-tool responses
|
||||
auto routed = AdvancedRouter::SynthesizeMultiToolResponse(
|
||||
tool_results_, ctx);
|
||||
|
||||
ChatMessage enhanced = response;
|
||||
enhanced.message = routed.summary;
|
||||
// Attach routed.gui_actions as metadata
|
||||
|
||||
return enhanced;
|
||||
}
|
||||
```
|
||||
|
||||
## Feature 4: Agent Pretraining
|
||||
|
||||
### What It Does
|
||||
|
||||
Injects structured knowledge into the agent's first message to teach it about:
|
||||
- ROM structure (memory map, data formats)
|
||||
- Hex analysis patterns (how to recognize sprites, tiles, palettes)
|
||||
- Map editing workflows (tile placement, warp creation)
|
||||
- Tool usage best practices
|
||||
|
||||
### Integration Status: ⏳ Implemented, Not Integrated
|
||||
|
||||
**Files**:
|
||||
- `cli/service/agent/agent_pretraining.{h,cc}` - Implementation ✅
|
||||
- `cli/agent.cmake` - Added to build ✅
|
||||
- `cli/service/agent/conversational_agent_service.cc` - **Needs integration** ⏳
|
||||
|
||||
### How to Integrate
|
||||
|
||||
**In ConversationalAgentService::SendMessage()**:
|
||||
```cpp
|
||||
absl::StatusOr<ChatMessage> ConversationalAgentService::SendMessage(
|
||||
const std::string& message) {
|
||||
|
||||
// One-time pretraining injection on first message
|
||||
if (inject_pretraining_ && !pretraining_injected_ && rom_context_) {
|
||||
std::string pretraining = AgentPretraining::GeneratePretrainingPrompt(rom_context_);
|
||||
|
||||
ChatMessage pretraining_msg;
|
||||
pretraining_msg.sender = ChatMessage::Sender::kUser;
|
||||
pretraining_msg.message = pretraining;
|
||||
pretraining_msg.is_internal = true; // Don't show to user
|
||||
|
||||
history_.insert(history_.begin(), pretraining_msg);
|
||||
pretraining_injected_ = true;
|
||||
}
|
||||
|
||||
// Continue with normal message processing...
|
||||
}
|
||||
```
|
||||
|
||||
### Knowledge Modules
|
||||
|
||||
```cpp
|
||||
auto modules = AgentPretraining::GetModules();
|
||||
for (const auto& module : modules) {
|
||||
std::cout << "Module: " << module.name << std::endl;
|
||||
std::cout << "Required: " << (module.required ? "Yes" : "No") << std::endl;
|
||||
std::cout << module.content << std::endl;
|
||||
}
|
||||
```
|
||||
|
||||
Modules include:
|
||||
- `rom_structure` - Memory map, data formats
|
||||
- `hex_analysis` - Pattern recognition for sprites/tiles/palettes
|
||||
- `map_editing` - Overworld/dungeon editing workflows
|
||||
- `tool_usage` - Best practices for tool calling
|
||||
|
||||
## Feature 5: Agent Handoff
|
||||
|
||||
### Concept
|
||||
|
||||
**Handoff** allows transitioning control between:
|
||||
1. **CLI → GUI**: Start debugging in terminal, continue in editor
|
||||
2. **Agent → Agent**: Specialized agents for different tasks
|
||||
3. **Human → AI**: Let AI continue work autonomously
|
||||
|
||||
### Implementation Status: 🚧 Architecture Defined
|
||||
|
||||
### Handoff Data Structure
|
||||
|
||||
```cpp
|
||||
struct HandoffContext {
|
||||
std::string handoff_id;
|
||||
std::string source_agent;
|
||||
std::string target_agent;
|
||||
|
||||
// State preservation
|
||||
std::vector<ChatMessage> conversation_history;
|
||||
Rom* rom_snapshot; // ROM state at handoff
|
||||
std::vector<uint32_t> active_breakpoints;
|
||||
std::map<std::string, std::string> variables; // Key findings
|
||||
|
||||
// Task context
|
||||
std::vector<TodoItem> remaining_todos;
|
||||
std::string current_goal;
|
||||
std::string progress_summary;
|
||||
|
||||
// Tool state
|
||||
std::vector<std::string> tools_used;
|
||||
std::map<std::string, std::string> cached_results;
|
||||
};
|
||||
```
|
||||
|
||||
### Implementation Plan
|
||||
|
||||
**Phase 1: State Serialization**
|
||||
- [ ] Serialize ConversationalAgentService state to JSON
|
||||
- [ ] Include learned knowledge, TODOs, breakpoints
|
||||
- [ ] Generate handoff token (UUID + encrypted state)
|
||||
|
||||
**Phase 2: Cross-Surface Handoff**
|
||||
- [ ] CLI saves handoff to `~/.yaze/agent/handoffs/<token>.json`
|
||||
- [ ] GUI Agent Chat widget can import handoff
|
||||
- [ ] Restore full conversation + tool state
|
||||
|
||||
**Phase 3: Specialized Agents**
|
||||
- [ ] Define agent personas (EmulatorDebugAgent, ROMHackAgent, TestAgent)
|
||||
- [ ] Implement handoff protocol (request → accept → execute → return)
|
||||
- [ ] Add handoff commands to CLI
|
||||
|
||||
## Current Integration Status
|
||||
|
||||
### ✅ Fully Integrated
|
||||
|
||||
1. **LearnedKnowledgeService**
|
||||
- ✅ Implemented and integrated into ConversationalAgentService
|
||||
- ✅ CLI commands available
|
||||
- ✅ Persistent storage in `~/.yaze/agent/`
|
||||
|
||||
2. **TodoManager**
|
||||
- ✅ Implemented and integrated into ConversationalAgentService
|
||||
- ✅ CLI commands available
|
||||
- ✅ Persistent storage in `~/.yaze/agent/todos.json`
|
||||
|
||||
3. **Emulator Debugging Service**
|
||||
- ✅ gRPC service implemented
|
||||
- ✅ 20/24 methods implemented
|
||||
- ✅ Function schemas for AI tool calling
|
||||
- ✅ See E9-ai-agent-debugging-guide.md for details
|
||||
|
||||
### ⏳ Implemented But Not Integrated
|
||||
|
||||
4. **AdvancedRouter**
|
||||
- ✅ Implemented
|
||||
- ⏳ Needs integration into ToolDispatcher or ConversationalAgentService
|
||||
|
||||
5. **AgentPretraining**
|
||||
- ✅ Implemented
|
||||
- ⏳ Needs injection into first message of conversation
|
||||
|
||||
### 🚧 Architecture Defined
|
||||
|
||||
6. **Agent Handoff**
|
||||
- ⏳ Architecture designed
|
||||
- ⏳ Implementation pending
|
||||
|
||||
## Benefits Summary
|
||||
|
||||
### For AI Agents
|
||||
|
||||
| Feature | Without Integration | With Integration |
|
||||
|---------|---------------------|------------------|
|
||||
| **Learned Knowledge** | Forgets between sessions | Remembers preferences, patterns |
|
||||
| **TODO Management** | Ad-hoc task tracking | Structured dependency-aware plans |
|
||||
| **Advanced Routing** | Raw tool output | Synthesized insights + GUI actions |
|
||||
| **Pretraining** | Generic LLM knowledge | ROM-specific expertise |
|
||||
| **Handoff** | Restart from scratch | Seamless context preservation |
|
||||
|
||||
### For Users
|
||||
|
||||
- **Faster onboarding**: AI learns your preferences
|
||||
- **Better continuity**: Past conversations inform current session
|
||||
- **Complex tasks**: AI breaks down goals automatically
|
||||
- **Cross-surface**: Start in CLI, continue in GUI
|
||||
- **Reproducible**: TODO plans serve as executable scripts
|
||||
|
||||
## References
|
||||
|
||||
- **Main CLI Guide**: C1-z3ed-agent-guide.md
|
||||
- **Debugging Guide**: E9-ai-agent-debugging-guide.md
|
||||
- **Changelog**: H1-changelog.md (v0.2.2 section)
|
||||
- **Learned Knowledge**: `cli/service/agent/learned_knowledge_service.{h,cc}`
|
||||
- **TODO Manager**: `cli/service/agent/todo_manager.{h,cc}`
|
||||
- **Advanced Routing**: `cli/service/agent/advanced_routing.{h,cc}`
|
||||
- **Pretraining**: `cli/service/agent/agent_pretraining.{h,cc}`
|
||||
- **Agent Service**: `cli/service/agent/conversational_agent_service.{h,cc}`
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: October 12, 2025
|
||||
**Status**: Core Features Integrated ✅
|
||||
**Next**: Context injection, Advanced routing, Handoff protocol
|
||||
|
||||
Reference in New Issue
Block a user