# Overworld Agent Guide - AI-Powered Overworld Editing

**Version**: 1.0  
**Last Updated**: October 6, 2025  
**Audience**: AI Agents, z3ed users, automation developers

---

## Overview

This guide explains how AI agents can interact with YAZE's overworld editor through the `z3ed` CLI and automation APIs. It covers:
- Available tools and commands
- Multimodal vision workflows
- Proposal-based editing
- Best practices for AI-generated edits

---

## Quick Start

### Prerequisites
```bash
# Build YAZE with AI and gRPC support
cmake -B build -DZ3ED_AI=ON -DYAZE_WITH_GRPC=ON
cmake --build build --target z3ed

# Set up AI provider (Gemini recommended for vision)
export GEMINI_API_KEY="your-key-here"
```

### First Agent Interaction
```bash
# Ask AI about a map
z3ed agent simple-chat "What tiles are at position 10,10 on map 0?" --rom zelda3.sfc

# AI agent generates edits
z3ed agent run --prompt "Place trees in a 3x3 grid at position 10,10 on map 0" \
  --rom zelda3.sfc --sandbox

# Review and accept
z3ed agent diff --proposal-id <id>
z3ed agent accept --proposal-id <id>
```

---

## Available Tools

### Read-Only Tools (Safe for AI)

#### overworld-get-tile
Query tile ID at coordinates.

**Purpose**: Analyze existing tile placement  
**Safety**: Read-only, no ROM modification  
**Rate Limit**: None

```json
{
  "tool": "overworld-get-tile",
  "parameters": {
    "map": 0,
    "x": 10,
    "y": 10
  }
}
```

**Response**:
```json
{
  "tile_id": 66,
  "tile_id_hex": "0x0042",
  "position": {"x": 10, "y": 10}
}
```

**Use Cases**:
- Check what tile currently exists before painting
- Analyze patterns in tile placement
- Verify expected tiles after edits

---

#### overworld-get-visible-region
Get tiles currently visible on canvas.

**Purpose**: Understand what the user is looking at  
**Safety**: Read-only  
**Rate Limit**: None

```json
{
  "tool": "overworld-get-visible-region",
  "parameters": {
    "map": 0
  }
}
```

**Response**:
```json
{
  "region": {
    "x_start": 0,
    "y_start": 0,
    "x_end": 31,
    "y_end": 31
  },
  "tiles": [
    {"x": 0, "y": 0, "tile_id": 40},
    {"x": 1, "y": 0, "tile_id": 40},
    ...
  ]
}
```

**Use Cases**:
- Analyze visible area before suggesting edits
- Generate context-aware suggestions
- Understand user's current focus

---

#### overworld-analyze-region
Get tile composition and patterns in a region.

**Purpose**: Deep analysis of tile distribution  
**Safety**: Read-only  
**Rate Limit**: Large regions (>1000 tiles) may be slow

```json
{
  "tool": "overworld-analyze-region",
  "parameters": {
    "map": 0,
    "x1": 0,
    "y1": 0,
    "x2": 31,
    "y2": 31
  }
}
```

**Response**:
```json
{
  "tile_counts": {
    "40": 512,   // Grass
    "66": 128,   // Tree
    "80": 64     // Water
  },
  "patterns": [
    {
      "type": "forest",
      "center": {"x": 15, "y": 15},
      "size": {"width": 10, "height": 10}
    }
  ],
  "statistics": {
    "total_tiles": 1024,
    "unique_tiles": 15,
    "most_common_tile": 40
  }
}
```

**Use Cases**:
- Understand map composition before edits
- Detect patterns (forests, water bodies, paths)
- Generate statistics for reports

---

### Write Tools (Sandboxed - Creates Proposals)

#### overworld-set-tile
Paint a single tile (creates proposal).

**Purpose**: Modify single tile  
**Safety**: Sandboxed, creates proposal  
**Rate Limit**: Reasonable (don't spam)

```json
{
  "tool": "overworld-set-tile",
  "parameters": {
    "map": 0,
    "x": 10,
    "y": 10,
    "tile_id": 66
  }
}
```

**Response**:
```json
{
  "proposal_id": "abc123",
  "success": true,
  "message": "Proposal created: Set tile at (10,10) to 0x0042"
}
```

**Use Cases**:
- Fix individual tiles
- Place objects at specific coordinates
- Correct tile placement errors

---

#### overworld-set-tiles-batch
Paint multiple tiles in one operation (creates proposal).

**Purpose**: Efficient multi-tile editing  
**Safety**: Sandboxed, creates proposal  
**Rate Limit**: Max 1000 tiles per batch

```json
{
  "tool": "overworld-set-tiles-batch",
  "parameters": {
    "map": 0,
    "tiles": [
      {"x": 10, "y": 10, "tile_id": 66},
      {"x": 11, "y": 10, "tile_id": 66},
      {"x": 12, "y": 10, "tile_id": 66}
    ]
  }
}
```

**Response**:
```json
{
  "proposal_id": "abc123",
  "tiles_painted": 3,
  "success": true
}
```

**Use Cases**:
- Create patterns (forests, paths, water bodies)
- Fill regions with specific tiles
- Generate complex map structures

---

## Multimodal Vision Workflow

### Step 1: Capture Canvas Screenshot
```bash
# From CLI
z3ed agent vision --capture-canvas "Overworld Canvas" \
  --prompt "Analyze this overworld map" \
  --rom zelda3.sfc

# From agent workflow
z3ed agent run --prompt "Analyze map 0 and suggest improvements" \
  --rom zelda3.sfc --sandbox
```

### Step 2: AI Analyzes Screenshot
Gemini Vision API receives:
- Screenshot of canvas (PNG/JPEG)
- User prompt
- Context (map index, visible region)

AI returns:
```json
{
  "analysis": {
    "observations": [
      "Grass tiles dominate the visible area",
      "Tree tiles are sparse and unnatural",
      "Water tiles at (15,15) have incorrect palette colors",
      "Path from (5,5) to (25,5) is broken"
    ],
    "composition_score": 6.5,
    "issues": [
      {
        "type": "sparse_trees",
        "severity": "medium",
        "location": {"x": 10, "y": 10},
        "suggestion": "Add more tree tiles for forest theme"
      }
    ]
  }
}
```

### Step 3: Generate Edit Plan
AI creates actionable plan:
```json
{
  "plan": [
    {
      "tool": "overworld-set-tiles-batch",
      "parameters": {
        "map": 0,
        "tiles": [
          {"x": 10, "y": 10, "tile_id": 66},
          {"x": 11, "y": 10, "tile_id": 66},
          {"x": 12, "y": 10, "tile_id": 66}
        ]
      },
      "reason": "Create denser forest area"
    }
  ]
}
```

### Step 4: Execute Plan (Sandbox)
```bash
# z3ed executes plan in sandbox
z3ed agent run --plan plan.json --rom zelda3.sfc --sandbox
```

### Step 5: Human Review
```bash
# View proposed changes
z3ed agent diff --proposal-id abc123

# Accept or reject
z3ed agent accept --proposal-id abc123
# or
z3ed agent reject --proposal-id abc123
```

---

## Example Workflows

### Workflow 1: Create Forest Area

**User Prompt**: "Create a forest clearing at position 15,15 with grass in the center"

**AI Plan**:
```json
{
  "steps": [
    {
      "step": 1,
      "description": "Check current tiles in region",
      "tool": "overworld-analyze-region",
      "parameters": {
        "map": 0,
        "x1": 10,
        "y1": 10,
        "x2": 20,
        "y2": 20
      }
    },
    {
      "step": 2,
      "description": "Paint grass clearing (center 5x5)",
      "tool": "overworld-set-tiles-batch",
      "parameters": {
        "map": 0,
        "tiles": [
          {"x": 13, "y": 13, "tile_id": 40},
          {"x": 14, "y": 13, "tile_id": 40},
          ...
        ]
      }
    },
    {
      "step": 3,
      "description": "Plant trees around edges",
      "tool": "overworld-set-tiles-batch",
      "parameters": {
        "map": 0,
        "tiles": [
          {"x": 10, "y": 10, "tile_id": 66},
          {"x": 11, "y": 10, "tile_id": 66},
          ...
        ]
      }
    }
  ]
}
```

**CLI Execution**:
```bash
z3ed agent run --prompt "Create a forest clearing at position 15,15 with grass in the center" \
  --rom zelda3.sfc --sandbox
```

**Result**: Proposal created with 50+ tile changes

---

### Workflow 2: Fix Tile Placement Errors

**User Prompt**: "Fix any water tiles that should be grass in the visible region"

**AI Plan**:
```json
{
  "steps": [
    {
      "step": 1,
      "tool": "overworld-get-visible-region",
      "parameters": {"map": 0}
    },
    {
      "step": 2,
      "tool": "overworld-analyze-region",
      "parameters": {
        "map": 0,
        "x1": 0,
        "y1": 0,
        "x2": 31,
        "y2": 31
      }
    },
    {
      "step": 3,
      "description": "Identify misplaced water tiles",
      "logic": "Find water tiles (80) surrounded by grass (40)"
    },
    {
      "step": 4,
      "tool": "overworld-set-tiles-batch",
      "parameters": {
        "map": 0,
        "tiles": [
          {"x": 5, "y": 5, "tile_id": 40},
          {"x": 12, "y": 8, "tile_id": 40}
        ]
      }
    }
  ]
}
```

---

### Workflow 3: Generate Path

**User Prompt**: "Create a path from (5,5) to (25,25) using path tiles"

**AI Plan**:
```json
{
  "steps": [
    {
      "step": 1,
      "description": "Calculate path coordinates",
      "logic": "Line from (5,5) to (25,25)"
    },
    {
      "step": 2,
      "tool": "overworld-set-tiles-batch",
      "parameters": {
        "map": 0,
        "tiles": [
          {"x": 5, "y": 5, "tile_id": 50},
          {"x": 6, "y": 6, "tile_id": 50},
          {"x": 7, "y": 7, "tile_id": 50},
          ...
        ]
      }
    }
  ]
}
```

---

## Common Tile IDs Reference

### Grass & Ground
- `0x0028` (40) - Grass
- `0x0029` (41) - Dark grass
- `0x002A` (42) - Dirt
- `0x002B` (43) - Sand

### Trees & Plants
- `0x0042` (66) - Tree
- `0x0043` (67) - Bush
- `0x0044` (68) - Flower

### Water
- `0x0050` (80) - Water
- `0x0051` (81) - Deep water
- `0x0052` (82) - Shore

### Paths & Roads
- `0x0032` (50) - Path
- `0x0033` (51) - Road
- `0x0034` (52) - Bridge

### Structures
- `0x0060` (96) - Wall
- `0x0061` (97) - Door
- `0x0062` (98) - Window

---

## Best Practices for AI Agents

### 1. Always Analyze Before Editing
```bash
# GOOD: Check current state first
z3ed agent run --prompt "Analyze map 0 then suggest improvements" --rom zelda3.sfc --sandbox

# BAD: Blindly paint without context
z3ed agent run --prompt "Paint trees everywhere" --rom zelda3.sfc --sandbox
```

### 2. Use Batch Operations
```bash
# GOOD: Single batch operation
overworld-set-tiles-batch (50 tiles)

# BAD: 50 individual operations
overworld-set-tile (×50)
```

### 3. Provide Clear Reasoning
```json
{
  "tool": "overworld-set-tile",
  "parameters": {"x": 10, "y": 10, "tile_id": 66},
  "reason": "Creating forest theme - tree tile at center"
}
```

### 4. Respect Tile Boundaries
Large maps (0x00-0x09, 0x80-0x89) are 512×512 pixels = 32×32 tiles.  
Don't paint beyond `(31, 31)` for these maps.

### 5. Check Visibility
```json
{
  "step": 1,
  "tool": "overworld-get-visible-region",
  "reason": "Ensure tiles are visible before analysis"
}
```

### 6. Create Reversible Edits
Always generate proposals that can be rejected:
```bash
z3ed agent run --prompt "..." --rom zelda3.sfc --sandbox  # Creates proposal
z3ed agent reject --proposal-id abc123                     # Can undo
```

---

## Error Handling

### "Tile ID out of range"
- **Cause**: Invalid tile ID (>4095 for Tile16)
- **Fix**: Validate tile IDs before `set-tile`

### "Coordinates out of bounds"
- **Cause**: Painting beyond map boundaries
- **Fix**: Check map dimensions (typically 32×32 tiles)

### "Proposal rejected"
- **Cause**: Human reviewer rejected changes
- **Fix**: Analyze feedback, adjust plan, try again

### "ROM file locked"
- **Cause**: ROM file open in another process
- **Fix**: Close other instances of YAZE

---

## Testing AI-Generated Edits

### Manual Testing
```bash
# Generate proposal
z3ed agent run --prompt "..." --rom zelda3.sfc --sandbox

# Review in YAZE GUI
yaze zelda3.sfc
# Open Debug → Agent Chat → Proposals
# Review proposal, accept/reject
```

### Automated Testing
```bash
# GUI automation test
z3ed agent test replay overworld_ai_edit.jsonl --rom zelda3.sfc --grpc localhost:50051

# Validate tile placement
z3ed agent test assert --tile-at 10,10 --expected-tile 66 --rom zelda3.sfc
```

---

## Advanced Techniques

### Technique 1: Pattern Recognition
Use multimodal vision to detect patterns:
```bash
z3ed agent vision --capture-canvas "Overworld Canvas" \
  --prompt "Identify repeated tile patterns in this map" \
  --rom zelda3.sfc
```

AI detects:
- Forest clusters
- Water bodies
- Paths and roads
- Building layouts

### Technique 2: Style Transfer
```bash
z3ed agent run --prompt "Make this map look like Kakariko Village from the dark world" \
  --rom zelda3.sfc --sandbox
```

AI:
1. Analyzes Kakariko Village (map 0x18)
2. Extracts tile palette and patterns
3. Applies similar patterns to target map

### Technique 3: Procedural Generation
```bash
z3ed agent run --prompt "Generate a random forest area at 10,10 with natural-looking tree placement" \
  --rom zelda3.sfc --sandbox
```

AI uses procedural algorithms:
- Perlin noise for natural randomness
- Clustering for realistic tree placement
- Edge smoothing for organic boundaries

---

## Integration with GUI Automation

### Record Human Edits
```bash
# Record editing session
z3ed agent test record --suite overworld_forest.jsonl --rom zelda3.sfc
```

### Replay for AI Training
```bash
# Replay recorded session
z3ed agent test replay overworld_forest.jsonl --rom zelda3.sfc

# AI learns from human edits
z3ed agent learn --from-recording overworld_forest.jsonl
```

### Validate AI Edits
```bash
# AI generates edits
z3ed agent run --prompt "..." --rom zelda3.sfc --sandbox

# GUI automation validates
z3ed agent test verify --proposal-id abc123 --suite validation_tests.jsonl
```

---

## Collaboration Features

### Network Collaboration
```bash
# Connect to yaze-server
z3ed net connect ws://localhost:8765

# Join session
z3ed net join ABC123 --username "ai-agent"

# AI agent edits, humans review in real-time
z3ed agent run --prompt "..." --rom zelda3.sfc --sandbox

# Proposal synced to all participants
```

### Proposal Voting
```bash
# Submit proposal to session
z3ed proposal submit --proposal-id abc123 --session ABC123

# Wait for votes
z3ed proposal wait --proposal-id abc123

# Check result
z3ed proposal status --proposal-id abc123
# Output: approved (3/3 votes)
```

---

## Troubleshooting

### Agent Not Responding
```bash
# Check AI provider
z3ed agent ping

# Test simple query
z3ed agent simple-chat "Hello" --rom zelda3.sfc
```

### Tools Not Available
```bash
# Verify z3ed build
z3ed agent describe --resource overworld

# Should show:
# - overworld-get-tile
# - overworld-set-tile
# - overworld-analyze-region
```

### gRPC Connection Failed
```bash
# Check YAZE is running with gRPC
z3ed agent test ping --grpc localhost:50051

# Start YAZE with gRPC enabled
yaze --enable-grpc zelda3.sfc
```

---

## See Also

- [Canvas Automation API](../canvas_automation_api.md) - C++ API reference
- [GUI Automation Scenarios](gui_automation_scenarios.md) - Test examples
- [z3ed README](README.md) - CLI documentation
- [Multimodal Vision](README.md#multimodal-vision-gemini) - Screenshot analysis