yaze/docs/C3-agent-architecture.md

C3 - z3ed Agent Architecture Guide

Date: October 12, 2025
Version: v0.2.2-alpha
Status: Core Features Integrated

Overview

This guide documents the architecture of the z3ed AI agent system, including learned knowledge, TODO management, advanced routing, pretraining, and agent handoff capabilities.

Architecture Overview

┌───────────────────────────────────────────────────────────────┐
│              User / AI Agent                                   │
└────────────┬──────────────────────────────────────────────────┘
             │
             │ z3ed CLI commands
             │
┌────────────▼──────────────────────────────────────────────────┐
│         CLI Command Router (agent.cc)                          │
│                                                                │
│  Routes to:                                                    │
│  ├─ agent simple-chat    → SimpleChatCommand                  │
│  ├─ agent learn          → HandleLearnCommand                 │
│  ├─ agent todo           → HandleTodoCommand                  │
│  ├─ agent test           → HandleTestCommand                  │
│  ├─ agent plan/run/diff  → Proposal system                    │
│  └─ emulator-*           → EmulatorCommandHandler             │
└───────────┬───────────────────────────────────────────────────┘
            │
┌───────────▼───────────────────────────────────────────────────┐
│      ConversationalAgentService                                │
│                                                                │
│  Integrates:                                                   │
│  ├─ LearnedKnowledgeService  (preferences, patterns, memory)  │
│  ├─ TodoManager              (task tracking, dependencies)    │
│  ├─ AdvancedRouter           (response enhancement)           │
│  ├─ AgentPretraining         (knowledge injection)            │
│  └─ ToolDispatcher           (command execution)              │
└────────────┬──────────────────────────────────────────────────┘
             │
┌────────────▼──────────────────────────────────────────────────┐
│         Tool Dispatcher                                        │
│                                                                │
│  Routes tool calls to:                                         │
│  ├─ Resource Commands   (dungeon, overworld, sprites)         │
│  ├─ Emulator Commands   (breakpoints, memory, step)           │
│  ├─ GUI Commands        (automation, screenshots)             │
│  └─ Custom Tools        (extensible via CommandHandler)       │
└────────────┬──────────────────────────────────────────────────┘
             │
┌────────────▼──────────────────────────────────────────────────┐
│      Command Handlers (CommandHandler base class)              │
│                                                                │
│  Unified pattern:                                              │
│  1. Parse arguments (ArgumentParser)                           │
│  2. Get ROM context (CommandContext)                           │
│  3. Execute business logic                                     │
│  4. Format output (OutputFormatter)                            │
└────────────┬──────────────────────────────────────────────────┘
             │
┌────────────▼──────────────────────────────────────────────────┐
│      Persistent Storage                                        │
│                                                                │
│  ~/.yaze/agent/                                                │
│  ├─ preferences.json     (user preferences)                    │
│  ├─ patterns.json        (learned ROM patterns)                │
│  ├─ projects.json        (project contexts)                    │
│  ├─ memories.json        (conversation summaries)              │
│  ├─ todos.json           (task management)                     │
│  └─ sessions/            (collaborative chat history)          │
└────────────────────────────────────────────────────────────────┘

Feature 1: Learned Knowledge Service

What It Does

Persists information across agent sessions:

• Preferences: User's default settings (palette, tool choices)
• ROM Patterns: Learned behaviors (frequently accessed rooms, sprite patterns)
• Project Context: ROM-specific goals and notes
• Conversation Memory: Summaries of past discussions for continuity

Integration Status: Complete

Files:

• cli/service/agent/learned_knowledge_service.{h,cc} - Core service
• cli/handlers/agent/general_commands.cc - CLI handlers
• cli/handlers/agent.cc - Routing

Usage Examples

# Save preference
z3ed agent learn --preference default_palette=2

# Get preference
z3ed agent learn --get-preference default_palette

# Save project context
z3ed agent learn --project "myrom" --context "Vanilla+ difficulty hack"

# Get project details
z3ed agent learn --get-project "myrom"

# Search past conversations
z3ed agent learn --search-memories "dungeon room 5"

# Export all learned data
z3ed agent learn --export learned_data.json

# View statistics
z3ed agent learn --stats

AI Agent Integration

The ConversationalAgentService now:

1. Initializes LearnedKnowledgeService on startup
2. Can inject learned context into prompts (when inject_learned_context_=true)
3. Can access preferences/patterns/memories during tool execution

API:

ConversationalAgentService service;
service.learned_knowledge().SetPreference("palette", "2");
auto pref = service.learned_knowledge().GetPreference("palette");

Data Persistence

Location: ~/.yaze/agent/
Format: JSON
Files:

• preferences.json - Key-value pairs
• patterns.json - Timestamped ROM patterns with confidence scores
• projects.json - Project metadata and context
• memories.json - Conversation summaries (last 100)

Current Integration

• cli/service/agent/learned_knowledge_service.{h,cc} is constructed inside ConversationalAgentService.
• CLI commands such as z3ed agent learn … and agent recall … exercise this API.
• JSON artifacts persist under ~/.yaze/agent/.

Feature 2: TODO Management System

What It Does

Enables AI agents to break down complex tasks into executable steps with dependency tracking and prioritization.

Current Integration

• Core service in cli/service/agent/todo_manager.{h,cc}.
• CLI routing in cli/handlers/agent/todo_commands.{h,cc} and cli/handlers/agent.cc.
• JSON storage at ~/.yaze/agent/todos.json.

Usage Examples

# Create TODO
z3ed agent todo create "Fix input handling" --category=emulator --priority=1

# List TODOs
z3ed agent todo list

# Filter by status
z3ed agent todo list --status=in_progress

# Update status
z3ed agent todo update 1 --status=completed

# Get next actionable task
z3ed agent todo next

# Generate dependency-aware execution plan
z3ed agent todo plan

# Clear completed
z3ed agent todo clear-completed

AI Agent Integration

ConversationalAgentService service;
service.todo_manager().CreateTodo("Debug A button", "emulator", 1);
auto next = service.todo_manager().GetNextActionableTodo();

Storage

Location: ~/.yaze/agent/todos.json
Format: JSON array with dependencies:

{
  "todos": [
    {
      "id": "1",
      "description": "Debug input handling",
      "status": "in_progress",
      "category": "emulator",
      "priority": 1,
      "dependencies": [],
      "tools_needed": ["emulator-set-breakpoint", "emulator-read-memory"]
    }
  ]
}

Feature 3: Advanced Routing

What It Does

Optimizes tool responses for AI consumption with:

• Data type inference (sprite data vs tile data vs palette)
• Pattern extraction (repeating values, structures)
• Structured summaries (high-level + detailed + next steps)
• GUI action generation (converts analysis → automation script)

Status

• Implementation lives in cli/service/agent/advanced_routing.{h,cc} and is compiled via cli/agent.cmake.
• Hook-ups to ToolDispatcher / ConversationalAgentService remain on the backlog.

How to Integrate

Option 1: In ToolDispatcher (Automatic)

// In tool_dispatcher.cc, after tool execution:
auto result = handler->Run(args, rom_context_);
if (result.ok()) {
  std::string output = output_buffer.str();
  
  // Route through advanced router for enhanced response
  AdvancedRouter::RouteContext ctx;
  ctx.rom = rom_context_;
  ctx.tool_calls_made = {call.tool_name};
  
  if (call.tool_name == "hex-read") {
    auto routed = AdvancedRouter::RouteHexAnalysis(data, address, ctx);
    return absl::StrCat(routed.summary, "\n\n", routed.detailed_data);
  }
  
  return output;
}

Option 2: In ConversationalAgentService (Selective)

// After getting tool results, enhance the response:
ChatMessage ConversationalAgentService::EnhanceResponse(
    const ChatMessage& response, 
    const std::string& user_message) {
  
  AdvancedRouter::RouteContext ctx;
  ctx.rom = rom_context_;
  ctx.user_intent = user_message;
  
  // Use advanced router to synthesize multi-tool responses
  auto routed = AdvancedRouter::SynthesizeMultiToolResponse(
      tool_results_, ctx);
  
  ChatMessage enhanced = response;
  enhanced.message = routed.summary;
  // Attach routed.gui_actions as metadata
  
  return enhanced;
}

Feature 4: Agent Pretraining

What It Does

Injects structured knowledge into the agent's first message to teach it about:

• ROM structure (memory map, data formats)
• Hex analysis patterns (how to recognize sprites, tiles, palettes)
• Map editing workflows (tile placement, warp creation)
• Tool usage best practices

Status

• Pretraining scaffolding (cli/service/agent/agent_pretraining.{h,cc}) builds today.
• The one-time injection step in ConversationalAgentService is still disabled.

How to Integrate

In ConversationalAgentService::SendMessage():

absl::StatusOr<ChatMessage> ConversationalAgentService::SendMessage(
    const std::string& message) {
  
  // One-time pretraining injection on first message
  if (inject_pretraining_ && !pretraining_injected_ && rom_context_) {
    std::string pretraining = AgentPretraining::GeneratePretrainingPrompt(rom_context_);
    
    ChatMessage pretraining_msg;
    pretraining_msg.sender = ChatMessage::Sender::kUser;
    pretraining_msg.message = pretraining;
    pretraining_msg.is_internal = true;  // Don't show to user
    
    history_.insert(history_.begin(), pretraining_msg);
    pretraining_injected_ = true;
  }
  
  // Continue with normal message processing...
}

Knowledge Modules

auto modules = AgentPretraining::GetModules();
for (const auto& module : modules) {
  std::cout << "Module: " << module.name << std::endl;
  std::cout << "Required: " << (module.required ? "Yes" : "No") << std::endl;
  std::cout << module.content << std::endl;
}

Modules include:

• rom_structure - Memory map, data formats
• hex_analysis - Pattern recognition for sprites/tiles/palettes
• map_editing - Overworld/dungeon editing workflows
• tool_usage - Best practices for tool calling

Feature 5: Agent Handoff

Handoff covers CLI ↔ GUI transfers, specialised agent delegation, and human/AI ownership changes. The proposed HandoffContext structure (see code listing earlier) captures conversation history, ROM state, TODOs, and transient tool data. Serialization, cross-surface loading, and persona-specific workflows remain unimplemented.

Current Integration Snapshot

Integrated components:

• Learned knowledge service (cli/service/agent/learned_knowledge_service.{h,cc}) with CLI commands and JSON persistence under ~/.yaze/agent/.
• TODO manager (cli/service/agent/todo_manager.{h,cc} plus CLI handlers) with storage at ~/.yaze/agent/todos.json.
• Emulator debugging gRPC service; 20 of 24 methods are implemented (see E9-ai-agent-debugging-guide.md).

Pending integration:

• Advanced router (cli/service/agent/advanced_routing.{h,cc}) needs wiring into ToolDispatcher or ConversationalAgentService.
• Agent pretraining (cli/service/agent/agent_pretraining.{h,cc}) needs the one-time injection path enabled.
• Handoff serialization and import/export tooling are still design-only.

References

• Main CLI Guide: C1-z3ed-agent-guide.md
• Debugging Guide: E9-ai-agent-debugging-guide.md
• Changelog: H1-changelog.md (v0.2.2 section)
• Learned Knowledge: cli/service/agent/learned_knowledge_service.{h,cc}
• TODO Manager: cli/service/agent/todo_manager.{h,cc}
• Advanced Routing: cli/service/agent/advanced_routing.{h,cc}
• Pretraining: cli/service/agent/agent_pretraining.{h,cc}
• Agent Service: cli/service/agent/conversational_agent_service.{h,cc}

---

Last Updated: October 12, 2025
In progress: Context injection for pretraining, advanced routing integration, agent handoff implementation.