yaze/docs/C3-agent-architecture.md

C3 - z3ed Agent Architecture Guide

Date: October 12, 2025
Version: v0.2.2-alpha
Status: Core Features Integrated ✅

Overview

This guide documents the architecture of the z3ed AI agent system, including learned knowledge, TODO management, advanced routing, pretraining, and agent handoff capabilities.

Architecture Overview

┌───────────────────────────────────────────────────────────────┐
│              User / AI Agent                                   │
└────────────┬──────────────────────────────────────────────────┘
             │
             │ z3ed CLI commands
             │
┌────────────▼──────────────────────────────────────────────────┐
│         CLI Command Router (agent.cc)                          │
│                                                                │
│  Routes to:                                                    │
│  ├─ agent simple-chat    → SimpleChatCommand                  │
│  ├─ agent learn          → HandleLearnCommand                 │
│  ├─ agent todo           → HandleTodoCommand                  │
│  ├─ agent test           → HandleTestCommand                  │
│  ├─ agent plan/run/diff  → Proposal system                    │
│  └─ emulator-*           → EmulatorCommandHandler             │
└───────────┬───────────────────────────────────────────────────┘
            │
┌───────────▼───────────────────────────────────────────────────┐
│      ConversationalAgentService                                │
│                                                                │
│  Integrates:                                                   │
│  ├─ LearnedKnowledgeService  (preferences, patterns, memory)  │
│  ├─ TodoManager              (task tracking, dependencies)    │
│  ├─ AdvancedRouter           (response enhancement)           │
│  ├─ AgentPretraining         (knowledge injection)            │
│  └─ ToolDispatcher           (command execution)              │
└────────────┬──────────────────────────────────────────────────┘
             │
┌────────────▼──────────────────────────────────────────────────┐
│         Tool Dispatcher                                        │
│                                                                │
│  Routes tool calls to:                                         │
│  ├─ Resource Commands   (dungeon, overworld, sprites)         │
│  ├─ Emulator Commands   (breakpoints, memory, step)           │
│  ├─ GUI Commands        (automation, screenshots)             │
│  └─ Custom Tools        (extensible via CommandHandler)       │
└────────────┬──────────────────────────────────────────────────┘
             │
┌────────────▼──────────────────────────────────────────────────┐
│      Command Handlers (CommandHandler base class)              │
│                                                                │
│  Unified pattern:                                              │
│  1. Parse arguments (ArgumentParser)                           │
│  2. Get ROM context (CommandContext)                           │
│  3. Execute business logic                                     │
│  4. Format output (OutputFormatter)                            │
└────────────┬──────────────────────────────────────────────────┘
             │
┌────────────▼──────────────────────────────────────────────────┐
│      Persistent Storage                                        │
│                                                                │
│  ~/.yaze/agent/                                                │
│  ├─ preferences.json     (user preferences)                    │
│  ├─ patterns.json        (learned ROM patterns)                │
│  ├─ projects.json        (project contexts)                    │
│  ├─ memories.json        (conversation summaries)              │
│  ├─ todos.json           (task management)                     │
│  └─ sessions/            (collaborative chat history)          │
└────────────────────────────────────────────────────────────────┘

Feature 1: Learned Knowledge Service

What It Does

Persists information across agent sessions:

• Preferences: User's default settings (palette, tool choices)
• ROM Patterns: Learned behaviors (frequently accessed rooms, sprite patterns)
• Project Context: ROM-specific goals and notes
• Conversation Memory: Summaries of past discussions for continuity

Integration Status: ✅ Complete

Files:

• cli/service/agent/learned_knowledge_service.{h,cc} - Core service
• cli/handlers/agent/general_commands.cc - CLI handlers
• cli/handlers/agent.cc - Routing

Usage Examples

# Save preference
z3ed agent learn --preference default_palette=2

# Get preference
z3ed agent learn --get-preference default_palette

# Save project context
z3ed agent learn --project "myrom" --context "Vanilla+ difficulty hack"

# Get project details
z3ed agent learn --get-project "myrom"

# Search past conversations
z3ed agent learn --search-memories "dungeon room 5"

# Export all learned data
z3ed agent learn --export learned_data.json

# View statistics
z3ed agent learn --stats

AI Agent Integration

The ConversationalAgentService now:

1. Initializes LearnedKnowledgeService on startup
2. Can inject learned context into prompts (when inject_learned_context_=true)
3. Can access preferences/patterns/memories during tool execution

API:

ConversationalAgentService service;
service.learned_knowledge().SetPreference("palette", "2");
auto pref = service.learned_knowledge().GetPreference("palette");

Data Persistence

Location: ~/.yaze/agent/
Format: JSON
Files:

• preferences.json - Key-value pairs
• patterns.json - Timestamped ROM patterns with confidence scores
• projects.json - Project metadata and context
• memories.json - Conversation summaries (last 100)

Feature 2: TODO Management System

What It Does

Enables AI agents to break down complex tasks into executable steps with dependency tracking and prioritization.

Integration Status: ✅ Complete

Files:

• cli/service/agent/todo_manager.{h,cc} - Core service
• cli/handlers/agent/todo_commands.{h,cc} - CLI handlers
• cli/handlers/agent.cc - Routing

Usage Examples

# Create TODO
z3ed agent todo create "Fix input handling" --category=emulator --priority=1

# List TODOs
z3ed agent todo list

# Filter by status
z3ed agent todo list --status=in_progress

# Update status
z3ed agent todo update 1 --status=completed

# Get next actionable task
z3ed agent todo next

# Generate dependency-aware execution plan
z3ed agent todo plan

# Clear completed
z3ed agent todo clear-completed

AI Agent Integration

ConversationalAgentService service;
service.todo_manager().CreateTodo("Debug A button", "emulator", 1);
auto next = service.todo_manager().GetNextActionableTodo();

Storage

Location: ~/.yaze/agent/todos.json
Format: JSON array with dependencies:

{
  "todos": [
    {
      "id": "1",
      "description": "Debug input handling",
      "status": "in_progress",
      "category": "emulator",
      "priority": 1,
      "dependencies": [],
      "tools_needed": ["emulator-set-breakpoint", "emulator-read-memory"]
    }
  ]
}

Feature 3: Advanced Routing

What It Does

Optimizes tool responses for AI consumption with:

• Data type inference (sprite data vs tile data vs palette)
• Pattern extraction (repeating values, structures)
• Structured summaries (high-level + detailed + next steps)
• GUI action generation (converts analysis → automation script)

Integration Status: ⏳ Implemented, Not Integrated

Files:

• cli/service/agent/advanced_routing.{h,cc} - Implementation ✅
• cli/agent.cmake - Added to build ✅
• cli/service/agent/conversational_agent_service.cc - Needs integration ⏳

How to Integrate

Option 1: In ToolDispatcher (Automatic)

// In tool_dispatcher.cc, after tool execution:
auto result = handler->Run(args, rom_context_);
if (result.ok()) {
  std::string output = output_buffer.str();
  
  // Route through advanced router for enhanced response
  AdvancedRouter::RouteContext ctx;
  ctx.rom = rom_context_;
  ctx.tool_calls_made = {call.tool_name};
  
  if (call.tool_name == "hex-read") {
    auto routed = AdvancedRouter::RouteHexAnalysis(data, address, ctx);
    return absl::StrCat(routed.summary, "\n\n", routed.detailed_data);
  }
  
  return output;
}

Option 2: In ConversationalAgentService (Selective)

// After getting tool results, enhance the response:
ChatMessage ConversationalAgentService::EnhanceResponse(
    const ChatMessage& response, 
    const std::string& user_message) {
  
  AdvancedRouter::RouteContext ctx;
  ctx.rom = rom_context_;
  ctx.user_intent = user_message;
  
  // Use advanced router to synthesize multi-tool responses
  auto routed = AdvancedRouter::SynthesizeMultiToolResponse(
      tool_results_, ctx);
  
  ChatMessage enhanced = response;
  enhanced.message = routed.summary;
  // Attach routed.gui_actions as metadata
  
  return enhanced;
}

Feature 4: Agent Pretraining

What It Does

Injects structured knowledge into the agent's first message to teach it about:

• ROM structure (memory map, data formats)
• Hex analysis patterns (how to recognize sprites, tiles, palettes)
• Map editing workflows (tile placement, warp creation)
• Tool usage best practices

Integration Status: ⏳ Implemented, Not Integrated

Files:

• cli/service/agent/agent_pretraining.{h,cc} - Implementation ✅
• cli/agent.cmake - Added to build ✅
• cli/service/agent/conversational_agent_service.cc - Needs integration ⏳

How to Integrate

In ConversationalAgentService::SendMessage():

absl::StatusOr<ChatMessage> ConversationalAgentService::SendMessage(
    const std::string& message) {
  
  // One-time pretraining injection on first message
  if (inject_pretraining_ && !pretraining_injected_ && rom_context_) {
    std::string pretraining = AgentPretraining::GeneratePretrainingPrompt(rom_context_);
    
    ChatMessage pretraining_msg;
    pretraining_msg.sender = ChatMessage::Sender::kUser;
    pretraining_msg.message = pretraining;
    pretraining_msg.is_internal = true;  // Don't show to user
    
    history_.insert(history_.begin(), pretraining_msg);
    pretraining_injected_ = true;
  }
  
  // Continue with normal message processing...
}

Knowledge Modules

auto modules = AgentPretraining::GetModules();
for (const auto& module : modules) {
  std::cout << "Module: " << module.name << std::endl;
  std::cout << "Required: " << (module.required ? "Yes" : "No") << std::endl;
  std::cout << module.content << std::endl;
}

Modules include:

• rom_structure - Memory map, data formats
• hex_analysis - Pattern recognition for sprites/tiles/palettes
• map_editing - Overworld/dungeon editing workflows
• tool_usage - Best practices for tool calling

Feature 5: Agent Handoff

Concept

Handoff allows transitioning control between:

1. CLI → GUI: Start debugging in terminal, continue in editor
2. Agent → Agent: Specialized agents for different tasks
3. Human → AI: Let AI continue work autonomously

Implementation Status: 🚧 Architecture Defined

Handoff Data Structure

struct HandoffContext {
  std::string handoff_id;
  std::string source_agent;
  std::string target_agent;
  
  // State preservation
  std::vector<ChatMessage> conversation_history;
  Rom* rom_snapshot;  // ROM state at handoff
  std::vector<uint32_t> active_breakpoints;
  std::map<std::string, std::string> variables;  // Key findings
  
  // Task context
  std::vector<TodoItem> remaining_todos;
  std::string current_goal;
  std::string progress_summary;
  
  // Tool state
  std::vector<std::string> tools_used;
  std::map<std::string, std::string> cached_results;
};

Implementation Plan

Phase 1: State Serialization

• Serialize ConversationalAgentService state to JSON
• Include learned knowledge, TODOs, breakpoints
• Generate handoff token (UUID + encrypted state)

Phase 2: Cross-Surface Handoff

• CLI saves handoff to ~/.yaze/agent/handoffs/<token>.json
• GUI Agent Chat widget can import handoff
• Restore full conversation + tool state

Phase 3: Specialized Agents

• Define agent personas (EmulatorDebugAgent, ROMHackAgent, TestAgent)
• Implement handoff protocol (request → accept → execute → return)
• Add handoff commands to CLI

Current Integration Status

✅ Fully Integrated

1. LearnedKnowledgeService

   • ✅ Implemented and integrated into ConversationalAgentService
   • ✅ CLI commands available
   • ✅ Persistent storage in ~/.yaze/agent/

2. TodoManager

   • ✅ Implemented and integrated into ConversationalAgentService
   • ✅ CLI commands available
   • ✅ Persistent storage in ~/.yaze/agent/todos.json

3. Emulator Debugging Service

   • ✅ gRPC service implemented
   • ✅ 20/24 methods implemented
   • ✅ Function schemas for AI tool calling
   • ✅ See E9-ai-agent-debugging-guide.md for details

⏳ Implemented But Not Integrated

4. AdvancedRouter

   • ✅ Implemented
   • ⏳ Needs integration into ToolDispatcher or ConversationalAgentService

5. AgentPretraining

   • ✅ Implemented
   • ⏳ Needs injection into first message of conversation

🚧 Architecture Defined

6. Agent Handoff
   • ⏳ Architecture designed
   • ⏳ Implementation pending

Benefits Summary

For AI Agents

Feature	Without Integration	With Integration
Learned Knowledge	Forgets between sessions	Remembers preferences, patterns
TODO Management	Ad-hoc task tracking	Structured dependency-aware plans
Advanced Routing	Raw tool output	Synthesized insights + GUI actions
Pretraining	Generic LLM knowledge	ROM-specific expertise
Handoff	Restart from scratch	Seamless context preservation

For Users

• Faster onboarding: AI learns your preferences
• Better continuity: Past conversations inform current session
• Complex tasks: AI breaks down goals automatically
• Cross-surface: Start in CLI, continue in GUI
• Reproducible: TODO plans serve as executable scripts

References

• Main CLI Guide: C1-z3ed-agent-guide.md
• Debugging Guide: E9-ai-agent-debugging-guide.md
• Changelog: H1-changelog.md (v0.2.2 section)
• Learned Knowledge: cli/service/agent/learned_knowledge_service.{h,cc}
• TODO Manager: cli/service/agent/todo_manager.{h,cc}
• Advanced Routing: cli/service/agent/advanced_routing.{h,cc}
• Pretraining: cli/service/agent/agent_pretraining.{h,cc}
• Agent Service: cli/service/agent/conversational_agent_service.{h,cc}

---

Last Updated: October 12, 2025
Status: Core Features Integrated ✅
Next: Context injection, Advanced routing, Handoff protocol