Files
yaze/docs/internal/agents/composite-layer-system.md

536 lines
19 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# SNES Dungeon Composite Layer System
**Document Status:** Implementation Reference
**Owner:** ai-systems-analyst
**Created:** 2025-12-05
**Last Reviewed:** 2025-12-05
**Next Review:** 2025-12-19
**Update (2025-12-05):** Per-tile priority support has been implemented. Section 5.1 updated.
---
## Overview
This document describes the dungeon room composite layer system used in yaze. It covers:
1. SNES hardware background layer architecture
2. ROM-based LayerMergeType settings
3. C++ implementation in yaze
4. Known issues and limitations
---
## 1. SNES Hardware Background Layer Architecture
### 1.1 Mode 1 Background Layers
The SNES uses **Mode 1** for dungeon rooms, which provides:
- **BG1**: Primary foreground layer (8x8 tiles, 16 colors per tile)
- **BG2**: Secondary background layer (8x8 tiles, 16 colors per tile)
- **BG3**: Text/overlay layer (4 colors per tile)
- **OBJ**: Sprite layer (Link, enemies, items)
**Critical: In SNES Mode 1, BG1 is ALWAYS rendered on top of BG2.** This is hardware behavior controlled by the PPU and cannot be changed by software.
### 1.2 Key PPU Registers
| Register | Address | Purpose |
|----------|---------|---------|
| `BGMODE` | $2105 | Background mode selection (Mode 1 = $09) |
| `TM` | $212C | Main screen layer enable (which BGs appear) |
| `TS` | $212D | Sub screen layer enable (for color math) |
| `CGWSEL` | $2130 | Color math window/clip control |
| `CGADSUB` | $2131 | Color math add/subtract control |
| `COLDATA` | $2132 | Fixed color for color math effects |
### 1.3 Color Math (Transparency Effects)
The SNES achieves transparency through **color math** between the main screen and sub screen:
```
CGWSEL ($2130):
Bits 7-6: Direct color / clip mode
Bits 5-4: Prevent color math (never=0, outside window=1, inside=2, always=3)
Bits 1-0: Sub screen BG/color (main=0, subscreen=1, fixed=2)
CGADSUB ($2131):
Bit 7: Subtract instead of add
Bit 6: Half color math result
Bits 5-0: Enable color math for OBJ, BG4, BG3, BG2, BG1, backdrop
```
**How Transparency Works:**
1. Main screen renders visible pixels (BG1, BG2 in priority order)
2. Sub screen provides "behind" pixels for blending
3. Color math combines main + sub pixels (add or average)
4. Result: Semi-transparent overlay effect
### 1.4 Tile Priority Bit
Each SNES tile has a **priority bit** in its tilemap entry:
```
Tilemap Word: YXPCCCTT TTTTTTTT
Y = Y-flip
X = X-flip
P = Priority (0=low, 1=high)
C = Palette (3 bits)
T = Tile number (10 bits)
```
**Priority Bit Behavior in Mode 1:**
- Priority 0 BG1 tiles appear BELOW priority 1 BG2 tiles
- Priority 1 BG1 tiles appear ABOVE all BG2 tiles
- This allows BG2 to "peek through" parts of BG1
**yaze implements per-tile priority** via the `BackgroundBuffer::priority_buffer_` and priority-aware compositing in `RoomLayerManager::CompositeToOutput()`.
---
## 2. ROM-Based LayerMergeType Settings
### 2.1 Room Header Structure
Each dungeon room has a header containing layer settings:
- **BG2 Mode**: Determines if BG2 is enabled and how it behaves
- **Layer Merging**: Index into LayerMergeType table (0-8)
- **Collision**: Which layers have collision data
### 2.2 LayerMergeType Table
```cpp
// From room.h
// LayerMergeType(id, name, Layer2Visible, Layer2OnTop, Layer2Translucent)
LayerMerge00{0x00, "Off", true, false, false}; // BG2 visible, no color math
LayerMerge01{0x01, "Parallax", true, false, false}; // Parallax scrolling effect
LayerMerge02{0x02, "Dark", true, true, true}; // BG2 color math + translucent
LayerMerge03{0x03, "On top", false, true, false}; // BG2 hidden but in subscreen
LayerMerge04{0x04, "Translucent", true, true, true}; // Translucent BG2
LayerMerge05{0x05, "Addition", true, true, true}; // Additive blending
LayerMerge06{0x06, "Normal", true, false, false}; // Standard dungeon
LayerMerge07{0x07, "Transparent", true, true, true}; // Water/fog effect
LayerMerge08{0x08, "Dark room", true, true, true}; // Unlit room (master brightness)
```
### 2.3 Flag Meanings
| Flag | ASM Effect | Purpose |
|------|------------|---------|
| `Layer2Visible` | Sets BG2 bit in TM ($212C) | Whether BG2 appears on main screen |
| `Layer2OnTop` | Sets BG2 bit in TS ($212D) | Whether BG2 participates in sub-screen color math |
| `Layer2Translucent` | Sets bit in CGADSUB ($2131) | Whether color math is enabled for blending |
**Important Clarification:**
- `Layer2OnTop` does **NOT** change Z-order (BG1 is always above BG2)
- It controls whether BG2 is on the **sub-screen** for color math
- When enabled, BG1 can blend with BG2 to create transparency effects
---
## 3. C++ Implementation in yaze
### 3.1 Architecture Overview
```
Room RoomLayerManager
├── bg1_buffer_ → ├── layer_visible_[4]
├── bg2_buffer_ ├── layer_blend_mode_[4]
├── object_bg1_buffer_ ├── bg2_on_top_
├── object_bg2_buffer_ └── CompositeToOutput()
└── composite_bitmap_
DungeonCanvasViewer
├── Separate Mode (draws each buffer individually)
└── Composite Mode (uses CompositeToOutput)
```
### 3.2 BackgroundBuffer Class
Located in `src/app/gfx/render/background_buffer.h`:
```cpp
class BackgroundBuffer {
std::vector<uint16_t> buffer_; // Tile ID buffer (64x64 tiles)
std::vector<uint8_t> priority_buffer_; // Per-pixel priority (0, 1, or 0xFF)
gfx::Bitmap bitmap_; // 512x512 8-bit indexed bitmap
void DrawFloor(...); // Sets up tile buffer from ROM floor data
void DrawBackground(...); // Renders tile buffer to bitmap pixels
void DrawTile(...); // Draws single 8x8 tile to bitmap + priority
// Priority buffer accessors
void ClearPriorityBuffer();
uint8_t GetPriorityAt(int x, int y) const;
void SetPriorityAt(int x, int y, uint8_t priority);
const std::vector<uint8_t>& priority_data() const;
};
```
**Key Points:**
- Each buffer is 512x512 pixels (64x64 tiles × 8 pixels)
- Uses 8-bit indexed color (palette indices 0-255)
- Transparent fill color is 255 (not 0!)
- Priority buffer tracks per-pixel priority bit (0, 1, or 0xFF for unset)
### 3.3 RoomLayerManager Class
Located in `src/zelda3/dungeon/room_layer_manager.h`:
**LayerType Enum:**
```cpp
enum class LayerType {
BG1_Layout, // Floor tiles on BG1
BG1_Objects, // Objects drawn to BG1 (layer 0, 2)
BG2_Layout, // Floor tiles on BG2
BG2_Objects // Objects drawn to BG2 (layer 1)
};
```
**LayerBlendMode Enum:**
```cpp
enum class LayerBlendMode {
Normal, // Full opacity (255 alpha)
Translucent, // 50% alpha (180)
Addition, // Additive blend (220)
Dark, // Darkened (120)
Off // Hidden (0)
};
```
### 3.4 CompositeToOutput Algorithm (Priority-Aware)
The compositing algorithm implements SNES Mode 1 per-tile priority:
**Effective Z-Order Table:**
| Layer | Priority | Effective Order |
|-------|----------|-----------------|
| BG1 | 0 | 0 (back) |
| BG2 | 0 | 1 |
| BG2 | 1 | 2 |
| BG1 | 1 | 3 (front) |
```cpp
void RoomLayerManager::CompositeToOutput(Room& room, gfx::Bitmap& output) {
// 1. Clear output to transparent (255)
output.Fill(255);
// 2. Create output priority buffer (tracks effective Z-order per pixel)
std::vector<uint8_t> output_priority(kPixelCount, 0xFF);
// 3. Helper to calculate effective Z-order
int GetEffectiveOrder(bool is_bg1, uint8_t priority) {
if (is_bg1) return priority ? 3 : 0; // BG1: 0 or 3
else return priority ? 2 : 1; // BG2: 1 or 2
}
// 4. For each layer, composite with priority comparison:
auto CompositeWithPriority = [&](BackgroundBuffer& buffer, bool is_bg1) {
for (int idx = 0; idx < kPixelCount; ++idx) {
uint8_t src_pixel = src_data[idx];
if (src_pixel == 255) continue; // Skip transparent
uint8_t src_prio = buffer.priority_data()[idx];
int src_order = GetEffectiveOrder(is_bg1, src_prio);
int dst_order = (output_priority[idx] == 0xFF) ? -1 : output_priority[idx];
// Source overwrites if higher or equal effective Z-order
if (dst_order == -1 || src_order >= dst_order) {
dst_data[idx] = src_pixel;
output_priority[idx] = src_order;
}
}
};
// 5. Process all layers (BG2 first, then BG1)
CompositeWithPriority(bg2_layout, false);
CompositeWithPriority(bg2_objects, false);
CompositeWithPriority(bg1_layout, true);
CompositeWithPriority(bg1_objects, true);
// 6. Apply palette and effects
ApplySDLPaletteToBitmap(src_surface, output);
// 7. Handle DarkRoom effect (merge type 0x08)
if (current_merge_type_id_ == 0x08) {
SDL_SetSurfaceColorMod(surface, 128, 128, 128); // 50% brightness
}
}
```
### 3.5 Priority Flow
1. **DrawTile()** in `BackgroundBuffer` writes `tile.over_` (priority bit) to `priority_buffer_`
2. **WriteTile8()** in `ObjectDrawer` also updates `priority_buffer_` for each tile drawn
3. **CompositeToOutput()** uses priority values to determine pixel ordering
**Note:** Blend modes still use simple pixel replacement. True color blending would require expensive RGB palette lookups. Visual effects are handled at SDL display time via alpha modulation.
---
## 4. Object Layer Assignment
### 4.1 Object Layer Field
Each room object has a `layer_` field (0, 1, or 2):
- **Layer 0**: Draws to BG1 buffer
- **Layer 1**: Draws to BG2 buffer
- **Layer 2**: Draws to BG1 buffer (priority variant)
### 4.2 Buffer Routing in ObjectDrawer
```cpp
// In ObjectDrawer::DrawObject()
BackgroundBuffer& target_bg =
(object.layer_ == 1) ? bg2_buffer : bg1_buffer;
// Some routines draw to BOTH buffers (walls, corners)
if (RoutineDrawsToBothBGs(routine_id)) {
DrawToBuffer(bg1_buffer, ...);
DrawToBuffer(bg2_buffer, ...);
}
```
### 4.3 kBothBGRoutines
These draw routines render to both BG1 and BG2:
```cpp
static constexpr int kBothBGRoutines[] = {
0, // DrawRightwards2x2_1to15or32 (ceiling 0x00)
1, // DrawRightwards2x4_1to15or26 (layout walls 0x001, 0x002)
8, // DrawDownwards4x2_1to15or26 (layout walls 0x061, 0x062)
19, // DrawCorner4x4 (layout corners 0x100-0x103)
3, // Rightwards2x4_1to16_BothBG (diagonal walls)
9, // Downwards4x2_1to16_BothBG (diagonal walls)
17, // DiagonalAcute_1to16_BothBG
18, // DiagonalGrave_1to16_BothBG
35, // 4x4Corner_BothBG (Type 2: 0x108-0x10F)
36, // WeirdCornerBottom_BothBG (Type 2: 0x110-0x113)
37, // WeirdCornerTop_BothBG (Type 2: 0x114-0x117)
97, // PrisonCell (dual-layer bars)
};
```
---
## 5. Known Issues and Limitations
### 5.1 Per-Tile Priority (IMPLEMENTED)
**Status:** Implemented as of December 2025.
**Implementation:**
- `BackgroundBuffer::priority_buffer_` stores per-pixel priority (0, 1, or 0xFF)
- `DrawTile()` and `WriteTile8()` write priority from `TileInfo.over_`
- `CompositeToOutput()` uses `GetEffectiveOrder()` for priority-aware compositing
**Effective Z-Order:**
- BG1 priority 0: Order 0 (back)
- BG2 priority 0: Order 1
- BG2 priority 1: Order 2
- BG1 priority 1: Order 3 (front)
**Known Discrepancy (Dec 2025):**
Some objects visible in "Separate Mode" (individual layer view) are hidden in "Composite Mode". This is expected SNES Mode 1 behavior where BG2 priority 1 tiles can appear above BG1 priority 0 tiles.
**Resolution:**
A "Priority Compositing" toggle (P checkbox) was added to the layer controls:
- **ON (default)**: Accurate SNES Mode 1 behavior - BG2-P1 can appear above BG1-P0
- **OFF**: Simple layer order - BG1 always appears above BG2
**Debugging tools:**
- "Show Priority Debug" in context menu shows per-layer priority statistics
- Pixels with "NO PRIORITY SET" indicate missing priority writes
- The Priority Debug window shows Z-order reference table
### 5.2 Simplified Color Blending
**Problem:** True color math requires RGB palette lookups, which is expensive.
**Current Workaround:**
- Blend modes use simple pixel replacement at indexed level
- SDL alpha modulation applied at display time
- Result is approximate, not pixel-accurate
### 5.3 DarkRoom Implementation
**Problem:** SNES DarkRoom uses master brightness register (INIDISP $2100).
**Current Implementation:** SDL color modulation to 50% (128, 128, 128).
### 5.4 Transparency Index
**Issue:** Both 0 and 255 have been treated as transparent at various points.
**Correct Behavior:**
- Index 0 is a VALID color in dungeon palettes (first actual color)
- Index 255 is the fill color for undrawn areas (should be transparent)
- CompositeLayer should only skip pixels with value 255
---
## 6. Related Files
| File | Purpose |
|------|---------|
| `src/zelda3/dungeon/room_layer_manager.h` | Layer visibility and compositing control |
| `src/zelda3/dungeon/room_layer_manager.cc` | CompositeToOutput implementation |
| `src/zelda3/dungeon/room.h` | LayerMergeType definitions |
| `src/zelda3/dungeon/room.cc` | Room rendering (RenderRoomGraphics) |
| `src/app/gfx/render/background_buffer.h` | BackgroundBuffer class |
| `src/app/gfx/render/background_buffer.cc` | Floor/tile drawing implementation |
| `src/zelda3/dungeon/object_drawer.cc` | Object rendering and buffer routing |
| `src/app/editor/dungeon/dungeon_canvas_viewer.cc` | Editor display (separate vs composite mode) |
---
## 7. ASM Reference: Color Math Registers
### CGWSEL ($2130) - Color Addition Select
```
7-6: Direct color mode / Prevent color math region
00 = Always perform color math
01 = Inside window only
10 = Outside window only
11 = Never perform color math
5-4: Clip colors to black region (same as 7-6)
3-2: Unused
1-0: Sub screen backdrop selection
00 = From palette (main screen)
01 = Sub screen
10 = Fixed color (COLDATA)
11 = Fixed color (COLDATA)
```
### CGADSUB ($2131) - Color Math Designation
```
7: Color subtract mode (0=add, 1=subtract)
6: Half color math (0=full, 1=half result)
5: Enable color math for OBJ/Sprites
4: Enable color math for BG4
3: Enable color math for BG3
2: Enable color math for BG2
1: Enable color math for BG1
0: Enable color math for backdrop
```
### COLDATA ($2132) - Fixed Color Data
```
7: Blue intensity enable
6: Green intensity enable
5: Red intensity enable
4-0: Intensity value (0-31)
```
---
## 8. Future Work
1. ~~**Per-Tile Priority**: Implement priority bit tracking for accurate Z-ordering~~ **DONE**
2. **True Color Blending**: Optional accurate blend mode with palette lookups
3. **HDMA Effects**: Support for scanline-based color math changes
4. ~~**Debug Visualization**: Show layer buffers with priority/blend annotations~~ **DONE**
- Added "Show Priority Debug" menu item in dungeon canvas context menu
- Priority Debug window shows per-layer statistics:
- Total non-transparent pixels
- Pixels with priority 0 vs priority 1
- Pixels with NO PRIORITY SET (indicates missing priority writes)
5. **Fix Missing Priority Writes**: Investigate objects that don't update priority buffer
---
## 9. Next Agent Steps: Fix Hidden Objects in Combo Rooms
**Priority:** HIGH - Objects hidden in composite mode regardless of priority toggle setting
### 9.1 Problem Description
Certain rooms have objects that are hidden in composite mode (both with and without priority compositing enabled). This occurs specifically in:
1. **BG Merge "Normal" (ID 0x06) rooms** - Standard dungeon layer merging
2. **BG2 Layer Behavior "Off" combo rooms** - Rooms where BG2 visibility is disabled by ROM
These are NOT priority-related issues since the objects remain hidden even when the "P" (priority) checkbox is unchecked.
### 9.2 Debugging Steps
1. **Identify affected rooms:**
- Open dungeon editor with a test ROM
- Navigate to rooms with layer_merging().ID == 0x06 ("Normal")
- Toggle between composite mode (M checkbox) and separate layer view
- Note which objects appear in separate mode but are hidden in composite mode
2. **Use Priority Debug window:**
- Right-click canvas → Debug → Show Priority Debug
- Check for "NO PRIORITY SET" pixels on BG1 Objects layer
- Check if the affected objects are in BG1 or BG2 buffers
3. **Check buffer contents:**
- In separate mode, verify each layer (BG1, O1, BG2, O2) individually
- Identify which buffer the "missing" objects are actually drawn to
### 9.3 Likely Root Causes
1. **BG2 visibility not respected:**
- `LayerMergeType.Layer2Visible` may not be correctly applied
- Check `ApplyLayerMerging()` in `room_layer_manager.cc`
- Verify BG2 layers are included when `Layer2Visible == true`
2. **Object layer assignment mismatch:**
- Objects may be drawn to wrong buffer (BG1 vs BG2)
- Check `RoomObject.layer_` field values
- Verify `ObjectDrawer::DrawObject()` buffer routing logic
3. **Transparency index conflict:**
- Pixel value 0 vs 255 confusion
- Check if objects are being skipped as "transparent" incorrectly
- Verify `IsTransparent()` only checks for 255
4. **BothBG routines priority handling:**
- Objects drawn to both BG1 and BG2 may have conflicting priorities
- Check routines in `kBothBGRoutines[]` list
- Verify both buffer draws update priority correctly
### 9.4 Files to Investigate
| File | What to Check |
|------|---------------|
| `room_layer_manager.cc` | `ApplyLayerMerging()`, `CompositeWithPriority()` lambda |
| `room_layer_manager.h` | `LayerMergeType` handling, visibility flags |
| `object_drawer.cc` | Buffer routing in `DrawObject()`, `RoutineDrawsToBothBGs()` |
| `room.h` | `LayerMergeType` definitions (Section 2.2 of this doc) |
| `background_buffer.cc` | `DrawTile()` priority writes, transparency handling |
### 9.5 Recommended Fixes to Try
1. **Add logging to composite loop:**
```cpp
// In CompositeWithPriority lambda, add:
static int debug_count = 0;
if (debug_count++ < 100 && !IsTransparent(src_pixel)) {
printf("[Composite] layer=%d idx=%d pixel=%d prio=%d\n",
static_cast<int>(layer_type), idx, src_pixel, src_prio);
}
```
2. **Verify BG2 visibility in composite:**
- Ensure `IsLayerVisible(LayerType::BG2_Layout)` returns true for Normal merge
- Check if `Layer2Visible` from ROM is being incorrectly overridden
3. **Check for early-exit conditions:**
- Search for `return` statements in `CompositeWithPriority` that might skip layers
- Verify `blend_mode == LayerBlendMode::Off` check isn't incorrectly triggered
### 9.6 Test Cases
After making fixes, verify with these room types:
- Room with layer_merging ID 0x06 (Normal) - objects should appear
- Room with layer_merging ID 0x00 (Off) - BG2 should still be visible
- Room with BothBG objects (walls, corners) - should render correctly
- Dark room (ID 0x08) - should have correct dimming
### 9.7 Success Criteria
- Objects visible in separate mode should also be visible in composite mode
- Priority toggle (P checkbox) should only affect Z-ordering, not visibility
- No regression in rooms that currently render correctly