# Dungeon Layer Compositing Research & Fix Plan

**Status:** PHASE 1 COMPLETE  
**Owner:** Requires multi-phase approach  
**Created:** 2025-12-07  
**Updated:** 2025-12-07  
**Problem:** BG2 content not visible through BG1 in dungeon editor

## Executive Summary

The current dungeon rendering has fundamental issues where BG2 content (Layer 1 objects like platforms, statues, overlays) is hidden under BG1 floor tiles. Multiple quick-fix attempts have failed because the underlying architecture doesn't match SNES behavior.

## Part 1: SNES Dungeon Rendering Architecture

### 1.1 Tilemap RAM Buffers

From `bank_01.asm` (lines 930-962):
```asm
RoomData_TilemapPointers:
.upper_layer          ; BG1 tilemap at $7E2000
  dl $7E2000+$000
  dl $7E2000+$002
  ...
.lower_layer          ; BG2 tilemap at $7E4000
  dl $7E4000+$000
  dl $7E4000+$002
  ...
```

**Key insight:** SNES has exactly TWO tilemap buffers:
- `$7E2000` = BG1 (upper/foreground layer)
- `$7E4000` = BG2 (lower/background layer)

All drawing operations (floor, layout, objects) write TILE IDs to these buffers.
The PPU then renders these tilemaps to screen.

### 1.2 Room Build Order (`LoadAndBuildRoom` at $01873A)

From `bank_01.asm` (lines 965-1157):

```
1. LoadRoomHeader
2. STZ $BA (reset stream index)
3. RoomDraw_DrawFloors      <- Fills BOTH tilemaps with floor tiles
4. Layout objects           <- Overwrites tilemap entries
5. Main room objects        <- Overwrites tilemap entries  
6. INC $BA twice            <- Skip 0xFFFF sentinel
7. Load lower_layer pointers to $C0
8. RoomDraw_DrawAllObjects  <- BG2 overlay (writes to $7E4000)
9. INC $BA twice            <- Skip 0xFFFF sentinel
10. Load upper_layer pointers to $C0
11. RoomDraw_DrawAllObjects <- BG1 overlay (writes to $7E2000)
12. Pushable blocks and torches
```

**Critical finding:** Objects OVERWRITE tilemap entries, they don't draw pixels.
Later objects can replace earlier tiles, including setting entries to 0 (transparent).

### 1.3 Floor Drawing (`RoomDraw_DrawFloors` at $0189DC)

From `bank_01.asm` (lines 1458-1548):

```asm
RoomDraw_DrawFloors:
  ; First pass: Load lower_layer pointers (BG2)
  ; Draw floor tiles to BG2 tilemap ($7E4000)
  JSR RoomDraw_FloorChunks
  
  ; Second pass: Load upper_layer pointers (BG1)  
  ; Draw floor tiles to BG1 tilemap ($7E2000)
  JMP RoomDraw_FloorChunks
```

The floor drawing:
1. Uses floor graphics from room data (high nibble = BG2, low nibble = BG1)
2. Fills the ENTIRE tilemap with floor tile patterns
3. Uses `RoomDraw_FloorChunks` to stamp 4x4 "super squares"

### 1.4 SNES PPU Layer Rendering

SNES Mode 1 layer priority (from `registers.asm`):
- TM ($212C): Main screen layer enable
- TS ($212D): Sub screen layer enable
- CGWSEL ($2130): Color math control
- CGADSUB ($2131): Color addition/subtraction select

Default priority order: BG1 > BG2 > BG3 (BG1 is always on top)

**BUT:** Each tile has a priority bit. High-priority BG2 tiles can appear
above low-priority BG1 tiles. The PPU sorts per-scanline based on:
1. Layer (BG1/BG2)
2. Tile priority bit
3. BG priority setting in BGMODE

Transparency: Tile color 0 is ALWAYS transparent in SNES graphics.
If a BG1 tile pixel is color 0, BG2 shows through at that pixel.

### 1.5 Layer Merge Types

From `room.h` (lines 100-112):
```cpp
LayerMerge00{0x00, "Off", true, false, false};      // BG2 visible, normal
LayerMerge01{0x01, "Parallax", true, false, false}; // BG2 visible, parallax scroll
LayerMerge02{0x02, "Dark", true, true, true};       // Dark room effect
LayerMerge03{0x03, "On top", false, true, false};   // BG2 hidden?
LayerMerge04{0x04, "Translucent", true, true, true}; // Color math blend
LayerMerge05{0x05, "Addition", true, true, true};   // Additive blend
LayerMerge06{0x06, "Normal", true, false, false};   // Standard
LayerMerge07{0x07, "Transparent", true, true, true}; // Transparency
LayerMerge08{0x08, "Dark room", true, true, true};  // Unlit room
```

These control PPU register settings (TM/TS/CGWSEL/CGADSUB), NOT draw order.

## Part 2: Current Editor Architecture

### 2.1 Four-Buffer Design

```cpp
bg1_buffer_        // Floor + Layout for BG1
bg2_buffer_        // Floor + Layout for BG2
object_bg1_buffer_ // Room objects for BG1
object_bg2_buffer_ // Room objects for BG2
```

### 2.2 Current Rendering Flow

```
1. DrawFloor() fills tile buffer with floor patterns
2. DrawBackground() renders tile buffer to BITMAP pixels
3. LoadLayoutTilesToBuffer() draws layout objects to BITMAP
4. RenderObjectsToBackground() draws room objects to OBJECT BITMAP
5. CompositeToOutput() layers: BG2_Layout → BG2_Objects → BG1_Layout → BG1_Objects
```

### 2.3 The Fundamental Problem

**SNES:** Objects write to TILEMAP BUFFER → PPU renders tilemaps with transparency
**Editor:** Objects write to BITMAP → Compositing layers opaque pixels

In SNES: An object can SET a tilemap entry to tile 0 (transparent), creating a "hole"
In Editor: Floor renders to bitmap first, objects can only draw ON TOP

**Result:** BG1 floor pixels are solid, completely covering BG2 content beneath.

## Part 3: Specific Issues to Fix

### Issue 1: BG1 Floor Covers BG2 Content
- **Symptom:** Center platform/statues in room 001 invisible with BG1 ON
- **Cause:** BG1 floor bitmap has solid pixels everywhere
- **SNES behavior:** BG1 tilemap would have transparent tiles in overlay areas

### Issue 2: Object Drawing Order
- **Symptom:** Layout objects may appear wrong relative to room objects
- **Cause:** Current code doesn't match SNES four-pass rendering
- **SNES behavior:** Strict order - layout → main → BG2 overlay → BG1 overlay

### Issue 3: Both-BG Routines
- **Symptom:** Diagonal walls/corners may not render correctly
- **Cause:** _BothBG routines need to write to both buffers simultaneously
- **Current:** Handled in DrawObject but may not interact correctly with layers

### Issue 4: Layer Merge Effects
- **Symptom:** Translucent/dark room effects not working
- **Cause:** LayerMergeType flags not properly applied
- **SNES behavior:** Controls PPU color math registers

## Part 4: Proposed Solution Architecture

### Option A: Tilemap-First Rendering (SNES-Accurate)

Change architecture to match SNES:
1. All drawing writes to TILE BUFFER (not bitmap)
2. Objects overwrite tile buffer entries
3. Single `RenderTilemapToBitmap()` call at end
4. Transparency via tile 0 or special tile entries

**Pros:** Most accurate, handles all edge cases
**Cons:** Major refactor, breaks existing layer visibility feature

### Option B: Mask Buffer System

Add a mask buffer to track "transparent" areas:
1. DrawFloor renders to bitmap
2. BG2 objects mark mask buffer in their area
3. When drawing BG1 floor, check mask and skip those pixels
4. Or: Apply mask after all rendering to "punch holes"

**Pros:** Moderate changes, keeps 4-buffer design
**Cons:** Additional buffer, may not handle all cases

### Option C: Deferred Floor Rendering

Change order to allow object interaction:
1. Objects draw first (marking occupied areas)
2. Floor draws AFTER, skipping marked areas
3. Compositing remains same

**Pros:** Simpler change
**Cons:** Doesn't match SNES order, may have edge cases

### Option D: Hybrid Tile/Pixel System

Best of both:
1. Keep tile buffer for SNES-accurate object placement
2. Objects can SET tile buffer entries to 0xFFFF (skip)
3. DrawBackground checks for skip markers
4. Then render layout/objects to bitmap

**Pros:** Can match SNES behavior while keeping visibility feature
**Cons:** Requires careful coordination

## Part 5: Recommended Phased Approach

### Phase 1: Research & Validation (COMPLETE)
- [x] Document exact SNES behavior for room 001
- [x] Trace through ASM for specific object types (platforms, overlays)
- [x] Identify which objects should create "holes" in BG1
- [x] Create test case matrix (Room 001 as primary case)

### Phase 2: Prototype Solution
- [ ] Implement Option D (Hybrid) in isolated branch
- [ ] Test with room 001 and other known problem rooms
- [ ] Validate layer visibility feature still works
- [ ] Document any regressions

### Phase 3: Object Classification
- [ ] Identify all objects that need BG1 masking
- [ ] Add metadata to draw routines for mask behavior
- [ ] Implement proper handling per object type

### Phase 4: Layer Merge Effects  
- [ ] Implement proper color math simulation
- [ ] Handle dark rooms, translucency, addition
- [ ] Test against real game screenshots

### Phase 5: Integration & Testing
- [ ] Merge to main branch
- [ ] Full regression testing
- [ ] Performance validation
- [ ] Documentation update

## Part 6: Next Immediate Steps

1. **Examine room 001 object data:**
   - What objects are on Layer 0 (BG1)?
   - What objects are on Layer 1 (BG2)?
   - Are there any "mask" or "pit" objects?

2. **Trace SNES rendering for room 001:**
   - What tilemap entries end up in BG1 for center area?
   - Are they tile 0 or floor tiles?

3. **Test hypothesis:**
   - If BG1 center has tile 0, our issue is in tile buffer management
   - If BG1 center has floor tiles with transparent pixels, issue is in graphics

## Part 7: Room 001 Case Study

### 7.1 Header Data

From `bank_04.asm` line 6123:
```asm
RoomHeader_Room0001:
  db $C0, $00, $00, $04, $00, $00, $00, $00, $00, $00, $72, $00, $50, $52
```

Decoded:
- **BG2PROP:** 0xC0
  - Layer2Mode = (0xC0 >> 5) = 6
  - LayerMerging = kLayerMergeTypeList[(0xC0 & 0x0C) >> 2] = LayerMerge00 "Off"
- **PALETTE:** 0x00
- **BLKSET:** 0x00 
- **SPRSET:** 0x04
- **Effects:** None

### 7.2 Layer Merge "Off" Behavior

LayerMerge00 "Off" = {Layer2Visible=true, Layer2OnTop=false, Layer2Translucent=false}

This means:
- BG2 IS visible on main screen
- BG2 does NOT use color math effects
- Standard Mode 1 priority: BG1 above BG2

### 7.3 What This Tells Us

Room 001 uses standard SNES Mode 1 rendering with no special layer effects.
BG2 content SHOULD be visible where BG1 has transparent pixels (color 0) or
where BG1 tilemap entries are tile 0 (empty).

The issue must be in HOW the tiles are written or rendered, not in the layer settings.

### 7.4 Next Step: Examine Object Data

Need to parse `bin/rooms/room0001.bin` to see:
1. What objects are on Layer 0 (BG1)?
2. What objects are on Layer 1 (BG2)?
3. Are there any "mask" or "pit" objects?
4. What happens at the 0xFFFF sentinels?

## Part 8: Phase 1 Research Findings (2025-12-07)

### 8.1 Room 001 Object Stream Analysis

Parsed room 001 object data from `alttp_vanilla.sfc`:

**Room Metadata:**
- Floor byte: 0x66 → Floor1=6, Floor2=6 (same floor graphic for both layers)
- Layout: 4

**Layer 0 (BG1 Main): 23 objects**
- Walls: 0x001 (2x), 0x002 (2x) - horizontal walls
- Corners: 0x100-0x103 (concave), 0x108-0x10B (4x4 corners)
- Diagonals: 0x003 (2x), 0x004 (2x), 0x063, 0x064
- Ceiling: 0x000 (4x)
- Other: 0x03A (decor)

**Layer 1 (BG2 Overlay): 11 objects**
- Walls at edges: 0x001 (2x), 0x002 (2x) at positions (1,13), (1,17), (59,13), (59,17)
- **Center platform objects:**
  - 0x13B @ (30,10) - Inter-room staircase
  - 0x033 @ (22,13) size=4 - `RoomDraw_Rightwards4x4_1to16` (4x4 platform)
  - 0x034 @ (23,16) size=14 - `RoomDraw_Rightwards1x1Solid_1to16_plus3` (solid tiles)
  - 0x071 @ (22,13), (41,13) - `RoomDraw_Downwards1x1Solid_1to16_plus3` (vertical solid)
  - 0x038 @ (24,12), (34,12) - `RoomDraw_RightwardsStatue2x3spaced2_1to16` (statues)

**Layer 2 (BG1 Priority): 8 objects**
- All torches: 0x0C6 (8x) at various positions

### 8.2 SNES 4-Pass Rendering (Confirmed)

From `bank_01.asm` analysis:

1. **Pass 1 (line 1104):** Layout objects drawn with default pointers → BG1
2. **Pass 2 (line 1120):** Main room objects (Layer 0) → BG1
3. **Pass 3 (lines 1127-1138):** Load `lower_layer` pointers ($7E4000), draw Layer 1 → BG2
4. **Pass 4 (lines 1145-1156):** Load `upper_layer` pointers ($7E2000), draw Layer 2 → BG1

**Key Insight:** After floor drawing, the tilemap pointers remain set to upper_layer (BG1).
Layout and Layer 0 objects write to BG1. Only Layer 1 writes to BG2.

### 8.3 The Transparency Mechanism

**How BG2 shows through BG1 in SNES:**
1. Floor tiles are drawn to BOTH BG1 and BG2 tilemaps (same graphic)
2. Layer 1 objects OVERWRITE BG2 tilemap entries with platform graphics
3. BG1 tilemap retains floor tiles in the platform area
4. **CRITICAL:** Floor tiles have color 0 (transparent) pixels
5. PPU composites: where BG1 has color 0 pixels, BG2 shows through

**Current Editor Code (from `background_buffer.cc` line 161):**
```cpp
if (pixel != 0) {
  // Pixel 0 is transparent. Pixel 1 maps to palette index 0.
  uint8_t final_color = (pixel - 1) + palette_offset;
  canvas[dest_index] = final_color;
}
```
The editor correctly skips pixel 0 as transparent! The bitmap is initialized to 255 (transparent).

### 8.4 Root Cause Identified

The issue is NOT in pixel-level transparency handling - that works correctly.

**The actual problem:** Floor graphic 6 tiles may be completely solid (no color 0 pixels),
OR the compositing order doesn't match SNES behavior.

**Verification needed:**
1. Check if floor graphic 6 tiles have any color 0 pixels
2. If they do, verify compositing respects those transparent pixels
3. If they don't, need a different mechanism (mask objects)

### 8.5 BG2MaskFull Object

Found `RoomDraw_BG2MaskFull` at routine 0x273 (line 6325):
```asm
RoomDraw_BG2MaskFull:
  STZ.b $0C
  LDX.w #obj00E0-RoomDrawObjectData
  JMP.w RoomDraw_FloorChunks
```

This is an explicit mask object that draws floor tiles to create "holes" in a layer.
However, room 001 does NOT use this object type.

### 8.6 Recommended Fix: Option E (Pixel-Perfect Compositing)

Based on research, the fix should ensure:

1. **Floor tile transparency is preserved:** Already working in `DrawTile()`
2. **Compositing respects transparency:** `CompositeToOutput()` skips transparent pixels
3. **Layer order is correct:** BG2 drawn first, BG1 drawn on top

**Specific changes needed:**

1. Verify floor graphic 6 tiles in the graphics data have color 0 pixels
2. If floor tiles are solid, implement BG2MaskFull-style approach:
   - When Layer 1 objects are drawn, also clear corresponding BG1 pixels to transparent (255)
   - OR: Track Layer 1 object positions and skip those areas when drawing BG1 floor

3. Alternative: For rooms using LayerMerge00 "Off" with Layer 1 objects:
   - Draw BG2 floor + Layer 1 objects first
   - Draw BG1 floor with per-pixel transparency check
   - Where BG2 has non-transparent content from Layer 1, make BG1 transparent

### 8.7 Implementation Plan

**Phase 2 Tasks:**
1. Create debug tool to visualize floor tile pixels (check for color 0)
2. If floor tiles have transparency, fix compositing order/logic
3. If floor tiles are solid, implement mask propagation from Layer 1 to BG1
4. Test with room 001 and other known overlay rooms

**Files to modify:**
- `src/app/gfx/render/background_buffer.cc` - Floor tile handling
- `src/zelda3/dungeon/room_layer_manager.cc` - Compositing logic
- `src/zelda3/dungeon/room.cc` - Layer 1 object tracking

## Answers to Open Questions

1. **How does SNES handle overlay areas?** Pixel-level transparency via color 0 in floor tiles.
   No explicit mask objects needed for standard overlays.

2. **What determines if BG2 shows through?** Color 0 pixels in BG1 tiles.
   Layer 1 objects only write to BG2, not BG1.

3. **Room-level data?** Floor graphics determine which tiles are used.
   Different floor graphics may have different transparency patterns.

4. **Per-tile priority bits?** Control BG1 vs BG2 priority per-tile.
   Not directly related to transparency (handled separately).

5. **Room 001 platform objects:** 0x033, 0x034, 0x071, 0x038 on Layer 1.

6. **Floor tile transparency:** Needs verification - check floor graphic 6 in graphics data.

---

**Status:** PHASE 1 COMPLETE - Ready for Phase 2 implementation

*Analysis script: `scripts/analyze_room.py`*

Usage examples:
```bash
# Analyze specific room
python scripts/analyze_room.py 1

# Analyze with layer compositing details
python scripts/analyze_room.py 1 --compositing

# List all rooms with BG2 overlay objects (94 total)
python scripts/analyze_room.py --list-bg2

# Analyze range of rooms
python scripts/analyze_room.py --range 0 20 --summary

# Output as JSON for programmatic use
python scripts/analyze_room.py 1 --json
```