Files

scawful 5c4cd57ff8 backend-infra-engineer: Post v0.3.9-hotfix7 snapshot (build cleanup)

2025-12-22 00:20:49 +00:00

19 KiB

Raw Blame History

SNES Dungeon Composite Layer System

Document Status: Implementation Reference
Owner: ai-systems-analyst
Created: 2025-12-05
Last Reviewed: 2025-12-05
Next Review: 2025-12-19

Update (2025-12-05): Per-tile priority support has been implemented. Section 5.1 updated.

Overview

This document describes the dungeon room composite layer system used in yaze. It covers:

SNES hardware background layer architecture
ROM-based LayerMergeType settings
C++ implementation in yaze
Known issues and limitations

1. SNES Hardware Background Layer Architecture

1.1 Mode 1 Background Layers

The SNES uses Mode 1 for dungeon rooms, which provides:

BG1: Primary foreground layer (8x8 tiles, 16 colors per tile)
BG2: Secondary background layer (8x8 tiles, 16 colors per tile)
BG3: Text/overlay layer (4 colors per tile)
OBJ: Sprite layer (Link, enemies, items)

Critical: In SNES Mode 1, BG1 is ALWAYS rendered on top of BG2. This is hardware behavior controlled by the PPU and cannot be changed by software.

1.2 Key PPU Registers

Register	Address	Purpose
`BGMODE`	$2105	Background mode selection (Mode 1 = $09)
`TM`	$212C	Main screen layer enable (which BGs appear)
`TS`	$212D	Sub screen layer enable (for color math)
`CGWSEL`	$2130	Color math window/clip control
`CGADSUB`	$2131	Color math add/subtract control
`COLDATA`	$2132	Fixed color for color math effects

1.3 Color Math (Transparency Effects)

The SNES achieves transparency through color math between the main screen and sub screen:

CGWSEL ($2130):
  Bits 7-6: Direct color / clip mode
  Bits 5-4: Prevent color math (never=0, outside window=1, inside=2, always=3)
  Bits 1-0: Sub screen BG/color (main=0, subscreen=1, fixed=2)

CGADSUB ($2131):
  Bit 7: Subtract instead of add
  Bit 6: Half color math result
  Bits 5-0: Enable color math for OBJ, BG4, BG3, BG2, BG1, backdrop

How Transparency Works:

Main screen renders visible pixels (BG1, BG2 in priority order)
Sub screen provides "behind" pixels for blending
Color math combines main + sub pixels (add or average)
Result: Semi-transparent overlay effect

1.4 Tile Priority Bit

Each SNES tile has a priority bit in its tilemap entry:

Tilemap Word: YXPCCCTT TTTTTTTT
  Y = Y-flip
  X = X-flip
  P = Priority (0=low, 1=high)
  C = Palette (3 bits)
  T = Tile number (10 bits)

Priority Bit Behavior in Mode 1:

Priority 0 BG1 tiles appear BELOW priority 1 BG2 tiles
Priority 1 BG1 tiles appear ABOVE all BG2 tiles
This allows BG2 to "peek through" parts of BG1

yaze implements per-tile priority via the BackgroundBuffer::priority_buffer_ and priority-aware compositing in RoomLayerManager::CompositeToOutput().

2. ROM-Based LayerMergeType Settings

2.1 Room Header Structure

Each dungeon room has a header containing layer settings:

BG2 Mode: Determines if BG2 is enabled and how it behaves
Layer Merging: Index into LayerMergeType table (0-8)
Collision: Which layers have collision data

2.2 LayerMergeType Table

// From room.h
// LayerMergeType(id, name, Layer2Visible, Layer2OnTop, Layer2Translucent)
LayerMerge00{0x00, "Off", true, false, false};        // BG2 visible, no color math
LayerMerge01{0x01, "Parallax", true, false, false};   // Parallax scrolling effect
LayerMerge02{0x02, "Dark", true, true, true};         // BG2 color math + translucent
LayerMerge03{0x03, "On top", false, true, false};     // BG2 hidden but in subscreen
LayerMerge04{0x04, "Translucent", true, true, true};  // Translucent BG2
LayerMerge05{0x05, "Addition", true, true, true};     // Additive blending
LayerMerge06{0x06, "Normal", true, false, false};     // Standard dungeon
LayerMerge07{0x07, "Transparent", true, true, true};  // Water/fog effect
LayerMerge08{0x08, "Dark room", true, true, true};    // Unlit room (master brightness)

2.3 Flag Meanings

Flag	ASM Effect	Purpose
`Layer2Visible`	Sets BG2 bit in TM ($212C)	Whether BG2 appears on main screen
`Layer2OnTop`	Sets BG2 bit in TS ($212D)	Whether BG2 participates in sub-screen color math
`Layer2Translucent`	Sets bit in CGADSUB ($2131)	Whether color math is enabled for blending

Important Clarification:

Layer2OnTop does NOT change Z-order (BG1 is always above BG2)
It controls whether BG2 is on the sub-screen for color math
When enabled, BG1 can blend with BG2 to create transparency effects

3. C++ Implementation in yaze

3.1 Architecture Overview

Room                          RoomLayerManager
├── bg1_buffer_         →     ├── layer_visible_[4]
├── bg2_buffer_               ├── layer_blend_mode_[4]
├── object_bg1_buffer_        ├── bg2_on_top_
├── object_bg2_buffer_        └── CompositeToOutput()
└── composite_bitmap_
         ↓
    DungeonCanvasViewer
    ├── Separate Mode (draws each buffer individually)
    └── Composite Mode (uses CompositeToOutput)

3.2 BackgroundBuffer Class

Located in src/app/gfx/render/background_buffer.h:

class BackgroundBuffer {
  std::vector<uint16_t> buffer_;      // Tile ID buffer (64x64 tiles)
  std::vector<uint8_t> priority_buffer_; // Per-pixel priority (0, 1, or 0xFF)
  gfx::Bitmap bitmap_;                // 512x512 8-bit indexed bitmap
  
  void DrawFloor(...);      // Sets up tile buffer from ROM floor data
  void DrawBackground(...); // Renders tile buffer to bitmap pixels
  void DrawTile(...);       // Draws single 8x8 tile to bitmap + priority
  
  // Priority buffer accessors
  void ClearPriorityBuffer();
  uint8_t GetPriorityAt(int x, int y) const;
  void SetPriorityAt(int x, int y, uint8_t priority);
  const std::vector<uint8_t>& priority_data() const;
};

Key Points:

Each buffer is 512x512 pixels (64x64 tiles × 8 pixels)
Uses 8-bit indexed color (palette indices 0-255)
Transparent fill color is 255 (not 0!)
Priority buffer tracks per-pixel priority bit (0, 1, or 0xFF for unset)

3.3 RoomLayerManager Class

Located in src/zelda3/dungeon/room_layer_manager.h:

LayerType Enum:

enum class LayerType {
  BG1_Layout,   // Floor tiles on BG1
  BG1_Objects,  // Objects drawn to BG1 (layer 0, 2)
  BG2_Layout,   // Floor tiles on BG2
  BG2_Objects   // Objects drawn to BG2 (layer 1)
};

LayerBlendMode Enum:

enum class LayerBlendMode {
  Normal,       // Full opacity (255 alpha)
  Translucent,  // 50% alpha (180)
  Addition,     // Additive blend (220)
  Dark,         // Darkened (120)
  Off           // Hidden (0)
};

3.4 CompositeToOutput Algorithm (Priority-Aware)

The compositing algorithm implements SNES Mode 1 per-tile priority:

Effective Z-Order Table:

Layer	Priority	Effective Order
BG1	0	0 (back)
BG2	0	1
BG2	1	2
BG1	1	3 (front)

void RoomLayerManager::CompositeToOutput(Room& room, gfx::Bitmap& output) {
  // 1. Clear output to transparent (255)
  output.Fill(255);
  
  // 2. Create output priority buffer (tracks effective Z-order per pixel)
  std::vector<uint8_t> output_priority(kPixelCount, 0xFF);
  
  // 3. Helper to calculate effective Z-order
  int GetEffectiveOrder(bool is_bg1, uint8_t priority) {
    if (is_bg1) return priority ? 3 : 0;  // BG1: 0 or 3
    else        return priority ? 2 : 1;  // BG2: 1 or 2
  }
  
  // 4. For each layer, composite with priority comparison:
  auto CompositeWithPriority = [&](BackgroundBuffer& buffer, bool is_bg1) {
    for (int idx = 0; idx < kPixelCount; ++idx) {
      uint8_t src_pixel = src_data[idx];
      if (src_pixel == 255) continue;  // Skip transparent
      
      uint8_t src_prio = buffer.priority_data()[idx];
      int src_order = GetEffectiveOrder(is_bg1, src_prio);
      int dst_order = (output_priority[idx] == 0xFF) ? -1 : output_priority[idx];
      
      // Source overwrites if higher or equal effective Z-order
      if (dst_order == -1 || src_order >= dst_order) {
        dst_data[idx] = src_pixel;
        output_priority[idx] = src_order;
      }
    }
  };
  
  // 5. Process all layers (BG2 first, then BG1)
  CompositeWithPriority(bg2_layout, false);
  CompositeWithPriority(bg2_objects, false);
  CompositeWithPriority(bg1_layout, true);
  CompositeWithPriority(bg1_objects, true);
  
  // 6. Apply palette and effects
  ApplySDLPaletteToBitmap(src_surface, output);
  
  // 7. Handle DarkRoom effect (merge type 0x08)
  if (current_merge_type_id_ == 0x08) {
    SDL_SetSurfaceColorMod(surface, 128, 128, 128);  // 50% brightness
  }
}

3.5 Priority Flow

DrawTile() in BackgroundBuffer writes tile.over_ (priority bit) to priority_buffer_
WriteTile8() in ObjectDrawer also updates priority_buffer_ for each tile drawn
CompositeToOutput() uses priority values to determine pixel ordering

Note: Blend modes still use simple pixel replacement. True color blending would require expensive RGB palette lookups. Visual effects are handled at SDL display time via alpha modulation.

4. Object Layer Assignment

4.1 Object Layer Field

Each room object has a layer_ field (0, 1, or 2):

Layer 0: Draws to BG1 buffer
Layer 1: Draws to BG2 buffer
Layer 2: Draws to BG1 buffer (priority variant)

4.2 Buffer Routing in ObjectDrawer

// In ObjectDrawer::DrawObject()
BackgroundBuffer& target_bg = 
    (object.layer_ == 1) ? bg2_buffer : bg1_buffer;

// Some routines draw to BOTH buffers (walls, corners)
if (RoutineDrawsToBothBGs(routine_id)) {
  DrawToBuffer(bg1_buffer, ...);
  DrawToBuffer(bg2_buffer, ...);
}

4.3 kBothBGRoutines

These draw routines render to both BG1 and BG2:

static constexpr int kBothBGRoutines[] = {
  0,   // DrawRightwards2x2_1to15or32 (ceiling 0x00)
  1,   // DrawRightwards2x4_1to15or26 (layout walls 0x001, 0x002)
  8,   // DrawDownwards4x2_1to15or26 (layout walls 0x061, 0x062)
  19,  // DrawCorner4x4 (layout corners 0x100-0x103)
  3,   // Rightwards2x4_1to16_BothBG (diagonal walls)
  9,   // Downwards4x2_1to16_BothBG (diagonal walls)
  17,  // DiagonalAcute_1to16_BothBG
  18,  // DiagonalGrave_1to16_BothBG
  35,  // 4x4Corner_BothBG (Type 2: 0x108-0x10F)
  36,  // WeirdCornerBottom_BothBG (Type 2: 0x110-0x113)
  37,  // WeirdCornerTop_BothBG (Type 2: 0x114-0x117)
  97,  // PrisonCell (dual-layer bars)
};

5. Known Issues and Limitations

5.1 Per-Tile Priority (IMPLEMENTED)

Status: Implemented as of December 2025.

Implementation:

BackgroundBuffer::priority_buffer_ stores per-pixel priority (0, 1, or 0xFF)
DrawTile() and WriteTile8() write priority from TileInfo.over_
CompositeToOutput() uses GetEffectiveOrder() for priority-aware compositing

Effective Z-Order:

BG1 priority 0: Order 0 (back)
BG2 priority 0: Order 1
BG2 priority 1: Order 2
BG1 priority 1: Order 3 (front)

Known Discrepancy (Dec 2025): Some objects visible in "Separate Mode" (individual layer view) are hidden in "Composite Mode". This is expected SNES Mode 1 behavior where BG2 priority 1 tiles can appear above BG1 priority 0 tiles.

Resolution: A "Priority Compositing" toggle (P checkbox) was added to the layer controls:

ON (default): Accurate SNES Mode 1 behavior - BG2-P1 can appear above BG1-P0
OFF: Simple layer order - BG1 always appears above BG2

Debugging tools:

"Show Priority Debug" in context menu shows per-layer priority statistics
Pixels with "NO PRIORITY SET" indicate missing priority writes
The Priority Debug window shows Z-order reference table

5.2 Simplified Color Blending

Problem: True color math requires RGB palette lookups, which is expensive.

Current Workaround:

Blend modes use simple pixel replacement at indexed level
SDL alpha modulation applied at display time
Result is approximate, not pixel-accurate

5.3 DarkRoom Implementation

Problem: SNES DarkRoom uses master brightness register (INIDISP $2100).

Current Implementation: SDL color modulation to 50% (128, 128, 128).

5.4 Transparency Index

Issue: Both 0 and 255 have been treated as transparent at various points.

Correct Behavior:

Index 0 is a VALID color in dungeon palettes (first actual color)
Index 255 is the fill color for undrawn areas (should be transparent)
CompositeLayer should only skip pixels with value 255

File	Purpose
`src/zelda3/dungeon/room_layer_manager.h`	Layer visibility and compositing control
`src/zelda3/dungeon/room_layer_manager.cc`	CompositeToOutput implementation
`src/zelda3/dungeon/room.h`	LayerMergeType definitions
`src/zelda3/dungeon/room.cc`	Room rendering (RenderRoomGraphics)
`src/app/gfx/render/background_buffer.h`	BackgroundBuffer class
`src/app/gfx/render/background_buffer.cc`	Floor/tile drawing implementation
`src/zelda3/dungeon/object_drawer.cc`	Object rendering and buffer routing
`src/app/editor/dungeon/dungeon_canvas_viewer.cc`	Editor display (separate vs composite mode)

7. ASM Reference: Color Math Registers

CGWSEL ($2130) - Color Addition Select

7-6: Direct color mode / Prevent color math region
       00 = Always perform color math
       01 = Inside window only
       10 = Outside window only
       11 = Never perform color math
5-4: Clip colors to black region (same as 7-6)
3-2: Unused
1-0: Sub screen backdrop selection
       00 = From palette (main screen)
       01 = Sub screen
       10 = Fixed color (COLDATA)
       11 = Fixed color (COLDATA)

CGADSUB ($2131) - Color Math Designation

7: Color subtract mode (0=add, 1=subtract)
6: Half color math (0=full, 1=half result)
5: Enable color math for OBJ/Sprites
4: Enable color math for BG4
3: Enable color math for BG3
2: Enable color math for BG2
1: Enable color math for BG1
0: Enable color math for backdrop

COLDATA ($2132) - Fixed Color Data

7: Blue intensity enable
6: Green intensity enable
5: Red intensity enable
4-0: Intensity value (0-31)

8. Future Work

Per-Tile Priority: Implement priority bit tracking for accurate Z-ordering DONE
True Color Blending: Optional accurate blend mode with palette lookups
HDMA Effects: Support for scanline-based color math changes
Debug Visualization: Show layer buffers with priority/blend annotations DONE
- Added "Show Priority Debug" menu item in dungeon canvas context menu
- Priority Debug window shows per-layer statistics:
  - Total non-transparent pixels
  - Pixels with priority 0 vs priority 1
  - Pixels with NO PRIORITY SET (indicates missing priority writes)
Fix Missing Priority Writes: Investigate objects that don't update priority buffer

9. Next Agent Steps: Fix Hidden Objects in Combo Rooms

Priority: HIGH - Objects hidden in composite mode regardless of priority toggle setting

9.1 Problem Description

Certain rooms have objects that are hidden in composite mode (both with and without priority compositing enabled). This occurs specifically in:

BG Merge "Normal" (ID 0x06) rooms - Standard dungeon layer merging
BG2 Layer Behavior "Off" combo rooms - Rooms where BG2 visibility is disabled by ROM

These are NOT priority-related issues since the objects remain hidden even when the "P" (priority) checkbox is unchecked.

9.2 Debugging Steps

Identify affected rooms:
- Open dungeon editor with a test ROM
- Navigate to rooms with layer_merging().ID == 0x06 ("Normal")
- Toggle between composite mode (M checkbox) and separate layer view
- Note which objects appear in separate mode but are hidden in composite mode
Use Priority Debug window:
- Right-click canvas → Debug → Show Priority Debug
- Check for "NO PRIORITY SET" pixels on BG1 Objects layer
- Check if the affected objects are in BG1 or BG2 buffers
Check buffer contents:
- In separate mode, verify each layer (BG1, O1, BG2, O2) individually
- Identify which buffer the "missing" objects are actually drawn to

9.3 Likely Root Causes

BG2 visibility not respected:
- LayerMergeType.Layer2Visible may not be correctly applied
- Check ApplyLayerMerging() in room_layer_manager.cc
- Verify BG2 layers are included when Layer2Visible == true
Object layer assignment mismatch:
- Objects may be drawn to wrong buffer (BG1 vs BG2)
- Check RoomObject.layer_ field values
- Verify ObjectDrawer::DrawObject() buffer routing logic
Transparency index conflict:
- Pixel value 0 vs 255 confusion
- Check if objects are being skipped as "transparent" incorrectly
- Verify IsTransparent() only checks for 255
BothBG routines priority handling:
- Objects drawn to both BG1 and BG2 may have conflicting priorities
- Check routines in kBothBGRoutines[] list
- Verify both buffer draws update priority correctly

9.4 Files to Investigate

File	What to Check
`room_layer_manager.cc`	`ApplyLayerMerging()`, `CompositeWithPriority()` lambda
`room_layer_manager.h`	`LayerMergeType` handling, visibility flags
`object_drawer.cc`	Buffer routing in `DrawObject()`, `RoutineDrawsToBothBGs()`
`room.h`	`LayerMergeType` definitions (Section 2.2 of this doc)
`background_buffer.cc`	`DrawTile()` priority writes, transparency handling

9.5 Recommended Fixes to Try

Add logging to composite loop:

// In CompositeWithPriority lambda, add:
static int debug_count = 0;
if (debug_count++ < 100 && !IsTransparent(src_pixel)) {
  printf("[Composite] layer=%d idx=%d pixel=%d prio=%d\n", 
         static_cast<int>(layer_type), idx, src_pixel, src_prio);
}

Verify BG2 visibility in composite:
- Ensure IsLayerVisible(LayerType::BG2_Layout) returns true for Normal merge
- Check if Layer2Visible from ROM is being incorrectly overridden
Check for early-exit conditions:
- Search for return statements in CompositeWithPriority that might skip layers
- Verify blend_mode == LayerBlendMode::Off check isn't incorrectly triggered

9.6 Test Cases

After making fixes, verify with these room types:

Room with layer_merging ID 0x06 (Normal) - objects should appear
Room with layer_merging ID 0x00 (Off) - BG2 should still be visible
Room with BothBG objects (walls, corners) - should render correctly
Dark room (ID 0x08) - should have correct dimming

9.7 Success Criteria

Objects visible in separate mode should also be visible in composite mode
Priority toggle (P checkbox) should only affect Z-ordering, not visibility
No regression in rooms that currently render correctly

19 KiB Raw Blame History Unescape Escape