# C3 - z3ed Agent Architecture Guide **Date**: October 12, 2025 **Version**: v0.2.2-alpha **Status**: Core Features Integrated ## Overview This guide documents the architecture of the z3ed AI agent system, including learned knowledge, TODO management, advanced routing, pretraining, and agent handoff capabilities. ## Architecture Overview ``` ┌───────────────────────────────────────────────────────────────┐ │ User / AI Agent │ └────────────┬──────────────────────────────────────────────────┘ │ │ z3ed CLI commands │ ┌────────────▼──────────────────────────────────────────────────┐ │ CLI Command Router (agent.cc) │ │ │ │ Routes to: │ │ ├─ agent simple-chat → SimpleChatCommand │ │ ├─ agent learn → HandleLearnCommand │ │ ├─ agent todo → HandleTodoCommand │ │ ├─ agent test → HandleTestCommand │ │ ├─ agent plan/run/diff → Proposal system │ │ └─ emulator-* → EmulatorCommandHandler │ └───────────┬───────────────────────────────────────────────────┘ │ ┌───────────▼───────────────────────────────────────────────────┐ │ ConversationalAgentService │ │ │ │ Integrates: │ │ ├─ LearnedKnowledgeService (preferences, patterns, memory) │ │ ├─ TodoManager (task tracking, dependencies) │ │ ├─ AdvancedRouter (response enhancement) │ │ ├─ AgentPretraining (knowledge injection) │ │ └─ ToolDispatcher (command execution) │ └────────────┬──────────────────────────────────────────────────┘ │ ┌────────────▼──────────────────────────────────────────────────┐ │ Tool Dispatcher │ │ │ │ Routes tool calls to: │ │ ├─ Resource Commands (dungeon, overworld, sprites) │ │ ├─ Emulator Commands (breakpoints, memory, step) │ │ ├─ GUI Commands (automation, screenshots) │ │ └─ Custom Tools (extensible via CommandHandler) │ └────────────┬──────────────────────────────────────────────────┘ │ ┌────────────▼──────────────────────────────────────────────────┐ │ Command Handlers (CommandHandler base class) │ │ │ │ Unified pattern: │ │ 1. Parse arguments (ArgumentParser) │ │ 2. Get ROM context (CommandContext) │ │ 3. Execute business logic │ │ 4. Format output (OutputFormatter) │ └────────────┬──────────────────────────────────────────────────┘ │ ┌────────────▼──────────────────────────────────────────────────┐ │ Persistent Storage │ │ │ │ ~/.yaze/agent/ │ │ ├─ preferences.json (user preferences) │ │ ├─ patterns.json (learned ROM patterns) │ │ ├─ projects.json (project contexts) │ │ ├─ memories.json (conversation summaries) │ │ ├─ todos.json (task management) │ │ └─ sessions/ (collaborative chat history) │ └────────────────────────────────────────────────────────────────┘ ``` ## Feature 1: Learned Knowledge Service ### What It Does Persists information across agent sessions: - **Preferences**: User's default settings (palette, tool choices) - **ROM Patterns**: Learned behaviors (frequently accessed rooms, sprite patterns) - **Project Context**: ROM-specific goals and notes - **Conversation Memory**: Summaries of past discussions for continuity ### Integration Status: Complete **Files**: - `cli/service/agent/learned_knowledge_service.{h,cc}` - Core service - `cli/handlers/agent/general_commands.cc` - CLI handlers - `cli/handlers/agent.cc` - Routing ### Usage Examples ```bash # Save preference z3ed agent learn --preference default_palette=2 # Get preference z3ed agent learn --get-preference default_palette # Save project context z3ed agent learn --project "myrom" --context "Vanilla+ difficulty hack" # Get project details z3ed agent learn --get-project "myrom" # Search past conversations z3ed agent learn --search-memories "dungeon room 5" # Export all learned data z3ed agent learn --export learned_data.json # View statistics z3ed agent learn --stats ``` ### AI Agent Integration The ConversationalAgentService now: 1. Initializes `LearnedKnowledgeService` on startup 2. Can inject learned context into prompts (when `inject_learned_context_=true`) 3. Can access preferences/patterns/memories during tool execution **API**: ```cpp ConversationalAgentService service; service.learned_knowledge().SetPreference("palette", "2"); auto pref = service.learned_knowledge().GetPreference("palette"); ``` ### Data Persistence **Location**: `~/.yaze/agent/` **Format**: JSON **Files**: - `preferences.json` - Key-value pairs - `patterns.json` - Timestamped ROM patterns with confidence scores - `projects.json` - Project metadata and context - `memories.json` - Conversation summaries (last 100) ### Current Integration - `cli/service/agent/learned_knowledge_service.{h,cc}` is constructed inside `ConversationalAgentService`. - CLI commands such as `z3ed agent learn …` and `agent recall …` exercise this API. - JSON artifacts persist under `~/.yaze/agent/`. ## Feature 2: TODO Management System ### What It Does Enables AI agents to break down complex tasks into executable steps with dependency tracking and prioritization. ### Current Integration - Core service in `cli/service/agent/todo_manager.{h,cc}`. - CLI routing in `cli/handlers/agent/todo_commands.{h,cc}` and `cli/handlers/agent.cc`. - JSON storage at `~/.yaze/agent/todos.json`. ### Usage Examples ```bash # Create TODO z3ed agent todo create "Fix input handling" --category=emulator --priority=1 # List TODOs z3ed agent todo list # Filter by status z3ed agent todo list --status=in_progress # Update status z3ed agent todo update 1 --status=completed # Get next actionable task z3ed agent todo next # Generate dependency-aware execution plan z3ed agent todo plan # Clear completed z3ed agent todo clear-completed ``` ### AI Agent Integration ```cpp ConversationalAgentService service; service.todo_manager().CreateTodo("Debug A button", "emulator", 1); auto next = service.todo_manager().GetNextActionableTodo(); ``` ### Storage **Location**: `~/.yaze/agent/todos.json` **Format**: JSON array with dependencies: ```json { "todos": [ { "id": "1", "description": "Debug input handling", "status": "in_progress", "category": "emulator", "priority": 1, "dependencies": [], "tools_needed": ["emulator-set-breakpoint", "emulator-read-memory"] } ] } ``` ## Feature 3: Advanced Routing ### What It Does Optimizes tool responses for AI consumption with: - **Data type inference** (sprite data vs tile data vs palette) - **Pattern extraction** (repeating values, structures) - **Structured summaries** (high-level + detailed + next steps) - **GUI action generation** (converts analysis → automation script) ### Status - Implementation lives in `cli/service/agent/advanced_routing.{h,cc}` and is compiled via `cli/agent.cmake`. - Hook-ups to `ToolDispatcher` / `ConversationalAgentService` remain on the backlog. ### How to Integrate **Option 1: In ToolDispatcher (Automatic)** ```cpp // In tool_dispatcher.cc, after tool execution: auto result = handler->Run(args, rom_context_); if (result.ok()) { std::string output = output_buffer.str(); // Route through advanced router for enhanced response AdvancedRouter::RouteContext ctx; ctx.rom = rom_context_; ctx.tool_calls_made = {call.tool_name}; if (call.tool_name == "hex-read") { auto routed = AdvancedRouter::RouteHexAnalysis(data, address, ctx); return absl::StrCat(routed.summary, "\n\n", routed.detailed_data); } return output; } ``` **Option 2: In ConversationalAgentService (Selective)** ```cpp // After getting tool results, enhance the response: ChatMessage ConversationalAgentService::EnhanceResponse( const ChatMessage& response, const std::string& user_message) { AdvancedRouter::RouteContext ctx; ctx.rom = rom_context_; ctx.user_intent = user_message; // Use advanced router to synthesize multi-tool responses auto routed = AdvancedRouter::SynthesizeMultiToolResponse( tool_results_, ctx); ChatMessage enhanced = response; enhanced.message = routed.summary; // Attach routed.gui_actions as metadata return enhanced; } ``` ## Feature 4: Agent Pretraining ### What It Does Injects structured knowledge into the agent's first message to teach it about: - ROM structure (memory map, data formats) - Hex analysis patterns (how to recognize sprites, tiles, palettes) - Map editing workflows (tile placement, warp creation) - Tool usage best practices ### Status - Pretraining scaffolding (`cli/service/agent/agent_pretraining.{h,cc}`) builds today. - The one-time injection step in `ConversationalAgentService` is still disabled. ### How to Integrate **In ConversationalAgentService::SendMessage()**: ```cpp absl::StatusOr ConversationalAgentService::SendMessage( const std::string& message) { // One-time pretraining injection on first message if (inject_pretraining_ && !pretraining_injected_ && rom_context_) { std::string pretraining = AgentPretraining::GeneratePretrainingPrompt(rom_context_); ChatMessage pretraining_msg; pretraining_msg.sender = ChatMessage::Sender::kUser; pretraining_msg.message = pretraining; pretraining_msg.is_internal = true; // Don't show to user history_.insert(history_.begin(), pretraining_msg); pretraining_injected_ = true; } // Continue with normal message processing... } ``` ### Knowledge Modules ```cpp auto modules = AgentPretraining::GetModules(); for (const auto& module : modules) { std::cout << "Module: " << module.name << std::endl; std::cout << "Required: " << (module.required ? "Yes" : "No") << std::endl; std::cout << module.content << std::endl; } ``` Modules include: - `rom_structure` - Memory map, data formats - `hex_analysis` - Pattern recognition for sprites/tiles/palettes - `map_editing` - Overworld/dungeon editing workflows - `tool_usage` - Best practices for tool calling ## Feature 5: Agent Handoff Handoff covers CLI ↔ GUI transfers, specialised agent delegation, and human/AI ownership changes. The proposed `HandoffContext` structure (see code listing earlier) captures conversation history, ROM state, TODOs, and transient tool data. Serialization, cross-surface loading, and persona-specific workflows remain unimplemented. ## Current Integration Snapshot Integrated components: - Learned knowledge service (`cli/service/agent/learned_knowledge_service.{h,cc}`) with CLI commands and JSON persistence under `~/.yaze/agent/`. - TODO manager (`cli/service/agent/todo_manager.{h,cc}` plus CLI handlers) with storage at `~/.yaze/agent/todos.json`. - Emulator debugging gRPC service; 20 of 24 methods are implemented (see `E9-ai-agent-debugging-guide.md`). Pending integration: - Advanced router (`cli/service/agent/advanced_routing.{h,cc}`) needs wiring into `ToolDispatcher` or `ConversationalAgentService`. - Agent pretraining (`cli/service/agent/agent_pretraining.{h,cc}`) needs the one-time injection path enabled. - Handoff serialization and import/export tooling are still design-only. ## References - **Main CLI Guide**: C1-z3ed-agent-guide.md - **Debugging Guide**: E9-ai-agent-debugging-guide.md - **Changelog**: H1-changelog.md (v0.2.2 section) - **Learned Knowledge**: `cli/service/agent/learned_knowledge_service.{h,cc}` - **TODO Manager**: `cli/service/agent/todo_manager.{h,cc}` - **Advanced Routing**: `cli/service/agent/advanced_routing.{h,cc}` - **Pretraining**: `cli/service/agent/agent_pretraining.{h,cc}` - **Agent Service**: `cli/service/agent/conversational_agent_service.{h,cc}` --- **Last Updated**: October 12, 2025 **In progress**: Context injection for pretraining, advanced routing integration, agent handoff implementation.