scawful/yaze
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Revise CMake presets and documentation for improved clarity and functionality

Browse Source 
- Reorganized CMake preset system, simplifying names and enhancing architecture support.
- Introduced new presets for macOS and Windows development, including AI-enabled and verbose options.
- Updated documentation to reflect changes in preset usage, design principles, and troubleshooting steps.
- Removed outdated documents related to stability improvements and modularization plans to streamline resources.
This commit is contained in:
scawful
2025-10-04 22:10:43 -04:00
parent a3c756272d
commit ffaec122cf
9 changed files with 172 additions and 1684 deletions
Download Patch File Download Diff File Expand all files Collapse all files

244
docs/B3-build-presets.md
View File

@@ -1,109 +1,207 @@
# Build Presets Guide
CMake presets for development workflow and architecture-specific builds.
This document explains the reorganized CMake preset system for Yaze.
## 🍎 macOS ARM64 Presets (Recommended for Apple Silicon)
## Design Principles
### For Development Work:
1. **Short, memorable names** - No more `macos-dev-z3ed-ai`, just `mac-ai`
2. **Warnings off by default** - Add `-v` suffix for verbose (e.g., `mac-dbg-v`)
3. **Clear architecture support** - Explicit ARM64 and x86_64 presets
4. **Platform prefixes** - `mac-`, `win-`, `lin-` for easy identification
## Quick Start
### macOS Development
```bash
# ARM64-only development build with ROM testing
cmake --preset macos-dev
cmake --build --preset macos-dev
# Most common: AI-enabled development
cmake --preset mac-ai
cmake --build --preset mac-ai
# ARM64-only debug build
cmake --preset macos-debug
cmake --build --preset macos-debug
# Basic debug build (no AI)
cmake --preset mac-dbg
cmake --build --preset mac-dbg
# ARM64-only release build
cmake --preset macos-release
cmake --build --preset macos-release
# Verbose warnings for debugging
cmake --preset mac-dbg-v
cmake --build --preset mac-dbg-v
# Release build
cmake --preset mac-rel
cmake --build --preset mac-rel
```
### For Distribution:
### Windows Development
```bash
# Universal binary (ARM64 + x86_64) - use only when needed for distribution
cmake --preset macos-debug-universal
cmake --build --preset macos-debug-universal
# Debug build (x64)
cmake --preset win-dbg
cmake --build --preset win-dbg
cmake --preset macos-release-universal
cmake --build --preset macos-release-universal
# AI-enabled development
cmake --preset win-ai
cmake --build --preset win-ai
# ARM64 support
cmake --preset win-arm
cmake --build --preset win-arm
```
## 🔧 Why This Fixes Architecture Errors
## All Presets
**Problem**: The original presets used `CMAKE_OSX_ARCHITECTURES: "x86_64;arm64"` which forced CMake to build universal binaries. This caused issues because:
- Dependencies like Abseil tried to use x86 SSE instructions (`-msse4.1`)
- These instructions don't exist on ARM64 processors
- Build failed with "unsupported option '-msse4.1' for target 'arm64-apple-darwin'"
### macOS Presets
**Solution**: The new ARM64-only presets use `CMAKE_OSX_ARCHITECTURES: "arm64"` which:
- ✅ Only targets ARM64 architecture
- ✅ Prevents x86-specific instruction usage
- ✅ Uses ARM64 optimizations instead
- ✅ Builds much faster (no cross-compilation)
| Preset | Description | Arch | Warnings | Features |
|--------|-------------|------|----------|----------|
| `mac-dbg` | Debug build | ARM64 | Off | Basic |
| `mac-dbg-v` | Debug verbose | ARM64 | On | Basic |
| `mac-rel` | Release | ARM64 | Off | Basic |
| `mac-x64` | Debug x86_64 | x86_64 | Off | Basic |
| `mac-uni` | Universal binary | Both | Off | Basic |
| `mac-dev` | Development | ARM64 | Off | ROM tests |
| `mac-ai` | AI development | ARM64 | Off | z3ed, JSON, gRPC, ROM tests |
| `mac-z3ed` | z3ed CLI | ARM64 | Off | AI agent support |
| `mac-rooms` | Dungeon editor | ARM64 | Off | Minimal build for room editing |
## 📋 Available Presets
### Windows Presets
| Preset Name | Architecture | Purpose | ROM Tests |
|-------------|-------------|---------|-----------|
| `macos-dev` | ARM64 only | Development | ✅ Enabled |
| `macos-debug` | ARM64 only | Debug builds | ❌ Disabled |
| `macos-release` | ARM64 only | Release builds | ❌ Disabled |
| `macos-debug-universal` | Universal | Distribution debug | ❌ Disabled |
| `macos-release-universal` | Universal | Distribution release | ❌ Disabled |
| Preset | Description | Arch | Warnings | Features |
|--------|-------------|------|----------|----------|
| `win-dbg` | Debug build | x64 | Off | Basic |
| `win-dbg-v` | Debug verbose | x64 | On | Basic |
| `win-rel` | Release | x64 | Off | Basic |
| `win-arm` | Debug ARM64 | ARM64 | Off | Basic |
| `win-arm-rel` | Release ARM64 | ARM64 | Off | Basic |
| `win-dev` | Development | x64 | Off | ROM tests |
| `win-ai` | AI development | x64 | Off | z3ed, JSON, gRPC, ROM tests |
| `win-z3ed` | z3ed CLI | x64 | Off | AI agent support |
## 🚀 Quick Start
### Linux Presets
For most development work on Apple Silicon:
| Preset | Description | Compiler | Warnings |
|--------|-------------|----------|----------|
| `lin-dbg` | Debug | GCC | Off |
| `lin-clang` | Debug | Clang | Off |
### Special Purpose
| Preset | Description |
|--------|-------------|
| `ci` | Continuous integration (no ROM tests) |
| `asan` | AddressSanitizer build |
| `coverage` | Code coverage build |
## Warning Control
By default, all presets suppress compiler warnings with `-w` for a cleaner build experience.
### To Enable Verbose Warnings:
1. Use a preset with `-v` suffix (e.g., `mac-dbg-v`, `win-dbg-v`)
2. Or set `YAZE_SUPPRESS_WARNINGS=OFF` manually:
```bash
cmake --preset mac-dbg -DYAZE_SUPPRESS_WARNINGS=OFF
```
## Architecture Support
### macOS
- **ARM64 (Apple Silicon)**: `mac-dbg`, `mac-rel`, `mac-ai`, etc.
- **x86_64 (Intel)**: `mac-x64`
- **Universal Binary**: `mac-uni` (both ARM64 + x86_64)
### Windows
- **x64**: `win-dbg`, `win-rel`, `win-ai`, etc.
- **ARM64**: `win-arm`, `win-arm-rel`
## Build Directories
Most presets use `build/` directory. Exceptions:
- `mac-rooms`: Uses `build_rooms/` to avoid conflicts
## Feature Flags
Common CMake options you can override:
```bash
# Clean build
rm -rf build
# Enable/disable components
-DYAZE_BUILD_TESTS=ON/OFF
-DYAZE_BUILD_Z3ED=ON/OFF
-DYAZE_BUILD_EMU=ON/OFF
-DYAZE_BUILD_APP=ON/OFF
# Configure for ARM64 development
cmake --preset macos-dev
# AI features
-DZ3ED_AI=ON/OFF
-DYAZE_WITH_JSON=ON/OFF
-DYAZE_WITH_GRPC=ON/OFF
# Build
cmake --build --preset macos-dev
# Testing
-DYAZE_ENABLE_ROM_TESTS=ON/OFF
-DYAZE_TEST_ROM_PATH=/path/to/zelda3.sfc
# Run tests
cmake --build --preset macos-dev --target test
# Build optimization
-DYAZE_MINIMAL_BUILD=ON/OFF
-DYAZE_USE_MODULAR_BUILD=ON/OFF
```
## 🛠️ IDE Integration
## Examples
### VS Code with CMake Tools:
1. Open Command Palette (`Cmd+Shift+P`)
2. Run "CMake: Select Configure Preset"
3. Choose `macos-dev` or `macos-debug`
### CLion:
1. Go to Settings → Build, Execution, Deployment → CMake
2. Add new profile with preset `macos-dev`
### Xcode:
### Development with AI features and verbose warnings
```bash
# Generate Xcode project
cmake --preset macos-debug -G Xcode
open build/yaze.xcodeproj
cmake --preset mac-dbg-v -DZ3ED_AI=ON -DYAZE_WITH_GRPC=ON
cmake --build --preset mac-dbg-v
```
## 🔍 Troubleshooting
### Release build for distribution (macOS Universal)
```bash
cmake --preset mac-uni
cmake --build --preset mac-uni
cpack --preset mac-uni
```
If you still get architecture errors:
1. **Clean completely**: `rm -rf build`
2. **Check preset**: Ensure you're using an ARM64 preset (not universal)
3. **Verify configuration**: Check that `CMAKE_OSX_ARCHITECTURES` shows only `arm64`
### Quick minimal build for testing
```bash
cmake --preset mac-dbg -DYAZE_MINIMAL_BUILD=ON
cmake --build --preset mac-dbg -j12
```
## Updating compile_commands.json
After configuring with a new preset, copy the compile commands for IDE support:
```bash
# Verify architecture setting
cmake --preset macos-debug
grep -A 5 -B 5 "CMAKE_OSX_ARCHITECTURES" build/CMakeCache.txt
cp build/compile_commands.json .
```
## 📝 Notes
This ensures clangd and other LSP servers can find headers and understand build flags.
- **ARM64 presets**: Fast builds, no architecture conflicts
- **Universal presets**: Slower builds, for distribution only
- **Deployment target**: ARM64 presets use macOS 11.0+ (when Apple Silicon was introduced)
- **Universal presets**: Still support macOS 10.15+ for backward compatibility
## Migration from Old Presets
Old preset names have been simplified:
| Old Name | New Name |
|----------|----------|
| `macos-dev-z3ed-ai` | `mac-ai` |
| `macos-debug` | `mac-dbg` |
| `macos-release` | `mac-rel` |
| `macos-debug-universal` | `mac-uni` |
| `macos-dungeon-dev` | `mac-rooms` |
| `windows-debug` | `win-dbg` |
| `windows-release` | `win-rel` |
| `windows-arm64-debug` | `win-arm` |
| `linux-debug` | `lin-dbg` |
| `linux-clang` | `lin-clang` |
## Troubleshooting
### Warnings are still showing
- Make sure you're using a preset without `-v` suffix
- Check `cmake` output for `✓ Warnings suppressed` message
- Reconfigure and rebuild: `rm -rf build && cmake --preset mac-dbg`
### clangd can't find nlohmann/json
- Run `cmake --preset <your-preset>` to regenerate compile_commands.json
- Copy to root: `cp build/compile_commands.json .`
- Restart your IDE or LSP server
### Build fails with missing dependencies
- Ensure submodules are initialized: `git submodule update --init --recursive`
- For AI features, make sure you have OpenSSL: `brew install openssl` (macOS)

403
docs/B5-stability-improvements.md
View File

@@ -1,403 +0,0 @@
# Stability, Testability & Release Workflow Improvements
This document outlines recent improvements to YAZE's stability, testability, and release workflows, along with recommendations for future optimizations.
## Recent Improvements (v0.3.2)
### Windows Platform Stability
#### Stack Size Optimization
**Problem:** Windows default stack size (1MB) was insufficient for `EditorManager::LoadAssets()` which loads 223 graphics sheets and initializes multiple editors.
**Solution:** Increased stack size to 8MB to match Unix-like systems.
**Implementation:**
```cmake
# src/app/app.cmake
if(MSVC)
target_link_options(yaze PRIVATE /STACK:8388608) # 8MB stack
elseif(MINGW)
target_link_options(yaze PRIVATE -Wl,--stack,8388608)
endif()
```
**Impact:**
- ✅ Eliminated stack overflow crashes during ROM loading
- ✅ Consistent behavior across all platforms
- ✅ Handles deep call chains from editor initialization
#### Development Utility Isolation
**Problem:** Development-only utilities (`extract_vanilla_values`, `rom_patch_utility`) were being built in CI/release workflows, causing linker errors.
**Solution:** Isolated development utilities from CI/release builds using environment detection.
**Implementation:**
```cmake
# test/CMakeLists.txt
if(NOT YAZE_MINIMAL_BUILD AND YAZE_ENABLE_ROM_TESTS AND NOT DEFINED ENV{GITHUB_ACTIONS})
add_executable(extract_vanilla_values ...)
target_link_libraries(extract_vanilla_values yaze_core ...)
endif()
```
**Impact:**
- ✅ Clean release builds without development artifacts
- ✅ Proper library linkage for development tools
- ✅ Faster CI/CD pipelines
### Graphics System Stability
#### Segmentation Fault Resolution
**Problem:** Tile cache system using `std::move()` operations invalidated Bitmap surface pointers, causing crashes.
**Solution:** Disabled move semantics in tile cache and implemented direct SDL texture updates.
**Technical Details:**
- Root cause: `std::move()` operations on Bitmap objects invalidated internal SDL_Surface pointers
- Fix: Disabled tile cache and use direct texture updates
- Optimization: Maintained surface/texture pooling while ensuring pointer stability
**Impact:**
- ✅ Eliminated all segmentation faults in tile16 editor
- ✅ Stable graphics rendering pipeline
- ✅ Reliable texture management
#### Comprehensive Bounds Checking
**Problem:** Out-of-bounds access to tile and palette data caused crashes and corruption.
**Solution:** Added extensive bounds checking throughout graphics pipeline.
**Areas Covered:**
- Tile16 canvas access
- Palette group selection
- Graphics sheet indexing
- Texture coordinate calculations
**Impact:**
- ✅ Prevents crashes from invalid data
- ✅ Better error reporting
- ✅ Safer memory access patterns
### Build System Improvements
#### Modern Windows Workflow
**Previous Approach:** Generate Visual Studio solution files using Python script.
**New Approach:** Use Visual Studio's native CMake support.
**Benefits:**
- No project generation step required
- CMakeLists.txt is the single source of truth
- Changes reflect immediately without regeneration
- Better IntelliSense and debugging experience
- Cross-platform consistency
**Migration:**
```powershell
# Old workflow
python scripts/generate-vs-projects.py
# Open YAZE.sln
# New workflow
# File → Open → Folder → yaze
# Visual Studio detects CMakeLists.txt automatically
```
#### Enhanced CI/CD Reliability
**Improvements:**
- Automatic vcpkg fallback mechanisms
- Proper development utility isolation
- Consistent test execution across platforms
- Better error reporting and debugging
## Recommended Optimizations
### High Priority
#### 1. Lazy Graphics Loading
**Current:** All 223 graphics sheets loaded on ROM open.
**Proposed:** Load graphics sheets on-demand when editors access them.
**Benefits:**
- Faster ROM loading (3-5x improvement expected)
- Reduced memory footprint
- Better startup performance
- Eliminates stack pressure
**Implementation Strategy:**
```cpp
class LazyGraphicsLoader {
std::array<std::optional<gfx::Bitmap>, kNumGfxSheets> sheets_;
gfx::Bitmap& GetSheet(int index) {
if (!sheets_[index]) {
sheets_[index] = LoadGraphicsSheet(rom_, index);
}
return *sheets_[index];
}
};
```
**Impact:**
- 🔵 Complexity: Medium
- 🟢 Performance Gain: High
- 🟢 Risk: Low (backward compatible)
#### 2. Heap-Based Large Allocations
**Current:** Large arrays and vectors allocated on stack during asset loading.
**Proposed:** Move large data structures to heap allocation.
**Benefits:**
- Reduces stack pressure
- More flexible memory management
- Better for Windows default stack constraints
- Safer for deep call chains
**Areas to Convert:**
- Graphics sheet arrays in LoadAllGraphicsData()
- Editor initialization data structures
- Temporary buffers in compression/decompression
**Impact:**
- 🟢 Complexity: Low
- 🟢 Performance Gain: Medium
- 🟢 Risk: Very Low
#### 3. Streaming ROM Assets
**Current:** Load entire ROM and all assets into memory.
**Proposed:** Stream assets from ROM file as needed.
**Benefits:**
- Minimal memory footprint
- Instant ROM opening
- Better for large ROM hacks
- More scalable architecture
**Challenges:**
- Requires architecture refactoring
- Need efficient caching strategy
- Must maintain edit performance
**Impact:**
- 🔴 Complexity: High
- 🟢 Performance Gain: Very High
- 🟡 Risk: Medium (requires testing)
### Medium Priority
#### 4. Enhanced Test Isolation
**Current:** Some tests share global state through Arena singleton.
**Proposed:** Better test isolation with mock singletons.
**Benefits:**
- More reliable test execution
- Parallel test execution possible
- Better test independence
- Easier debugging
**Implementation:**
```cpp
class TestArena : public Arena {
// Test-specific implementation
};
TEST_F(GraphicsTest, TestCase) {
TestArena arena;
Arena::SetInstance(&arena); // Override singleton
// Run test
}
```
**Impact:**
- 🟡 Complexity: Medium
- 🟢 Performance Gain: Medium (parallel tests)
- 🟢 Risk: Low
#### 5. Dependency Caching Optimization
**Current:** CI builds re-download and build some dependencies.
**Proposed:** Enhanced caching strategies for vcpkg and build artifacts.
**Benefits:**
- Faster CI builds (2-3x improvement)
- Reduced CI costs
- More reliable builds (less network dependency)
- Better developer experience
**Implementation:**
```yaml
# GitHub Actions
- uses: actions/cache@v4
with:
path: |
~/.ccache
~/vcpkg_cache
build/_deps
key: ${{ runner.os }}-${{ hashFiles('**/CMakeLists.txt') }}
```
**Impact:**
- 🟢 Complexity: Low
- 🟢 Performance Gain: High (CI only)
- 🟢 Risk: Very Low
#### 6. Memory Pool for Graphics
**Current:** Individual allocation for each Bitmap and texture.
**Proposed:** Memory pool for graphics objects.
**Benefits:**
- Reduced allocation overhead
- Better cache locality
- Predictable memory usage
- Faster allocation/deallocation
**Areas to Apply:**
- Bitmap objects
- SDL surfaces and textures
- Tile data structures
**Impact:**
- 🟡 Complexity: Medium
- 🟡 Performance Gain: Medium
- 🟡 Risk: Medium (requires careful design)
### Low Priority
#### 7. Build Time Optimization
**Current:** Full rebuild takes 10-15 minutes.
**Proposed:** Optimize compilation units and dependencies.
**Strategies:**
- Use forward declarations more extensively
- Split large compilation units
- Optimize template instantiations
- Better use of precompiled headers
**Impact:**
- 🟡 Complexity: Medium
- 🟢 Performance Gain: Medium (developer experience)
- 🟢 Risk: Low
#### 8. Release Workflow Simplification
**Current:** Three separate release workflows (simplified, standard, complex).
**Proposed:** Single unified workflow with conditional features.
**Benefits:**
- Easier maintenance
- Consistent behavior
- Better documentation
- Clearer mental model
**Implementation:**
```yaml
jobs:
release:
strategy:
matrix:
include:
- profile: minimal # Quick releases
- profile: standard # Normal releases
- profile: maximum # Production releases
```
**Impact:**
- 🟢 Complexity: Low
- 🟢 Performance Gain: None (maintenance benefit)
- 🟢 Risk: Very Low
## Testing Improvements
### Current State
- ✅ Comprehensive unit test coverage (46+ tests)
- ✅ Integration tests for major components
- ✅ ROM-dependent tests properly isolated
- ✅ CI-safe test configuration
- ✅ Platform-specific test handling
### Recommendations
#### 1. Visual Regression Testing
**Goal:** Catch graphics rendering regressions automatically.
**Approach:**
- Capture screenshots of editor states
- Compare against baseline images
- Flag visual differences for review
**Tools:** ImGui Test Engine (already integrated)
#### 2. Performance Benchmarks
**Goal:** Track performance regressions in CI.
**Metrics:**
- ROM load time
- Graphics sheet decompression
- Editor initialization
- Memory usage
**Implementation:** Google Benchmark (already a dependency)
#### 3. Fuzz Testing
**Goal:** Find edge cases and crashes through random input.
**Areas:**
- ROM parsing
- Compression/decompression
- Palette handling
- Tile data processing
**Tools:** LibFuzzer or AFL
## Metrics & Monitoring
### Current Measurements
- Build time: ~10-15 minutes (full rebuild)
- ROM load time: ~2-3 seconds
- Memory usage: ~500MB-1GB typical
- Test execution: ~30 seconds (CI), ~2 minutes (full)
### Target Improvements
- Build time: <5 minutes (incremental), <10 minutes (full)
- ROM load time: <1 second (with lazy loading)
- Memory usage: <300MB (with streaming)
- Test execution: <15 seconds (CI), <1 minute (full)
## Action Items
### Immediate (v0.3.2)
- [x] Fix Windows stack overflow
- [x] Isolate development utilities
- [x] Fix graphics segfaults
- [x] Update build documentation
- [ ] Complete tile16 palette display fixes
### Short Term (v0.3.3)
- [ ] Implement lazy graphics loading
- [ ] Move large allocations to heap
- [ ] Enhanced CI caching
- [ ] Performance benchmarks
### Medium Term (v0.4.0)
- [ ] Streaming ROM assets
- [ ] Memory pool for graphics
- [ ] Visual regression tests
- [ ] Enhanced test isolation
### Long Term (v0.5.0+)
- [ ] Fuzz testing integration
- [ ] Build time optimization
- [ ] Release workflow unification
- [ ] Advanced memory profiling
## Conclusion
The v0.3.2 release focuses on stability and reliability improvements, particularly for the Windows platform. The recommended optimizations provide a clear roadmap for future performance and maintainability improvements while maintaining backward compatibility and code quality.
For questions or suggestions, please open an issue or discussion on the GitHub repository.

207
docs/CMAKE_PRESETS.md
View File

@@ -1,207 +0,0 @@
# CMake Presets Quick Reference
This document explains the reorganized CMake preset system for Yaze.
## Design Principles
1. **Short, memorable names** - No more `macos-dev-z3ed-ai`, just `mac-ai`
2. **Warnings off by default** - Add `-v` suffix for verbose (e.g., `mac-dbg-v`)
3. **Clear architecture support** - Explicit ARM64 and x86_64 presets
4. **Platform prefixes** - `mac-`, `win-`, `lin-` for easy identification
## Quick Start
### macOS Development
```bash
# Most common: AI-enabled development
cmake --preset mac-ai
cmake --build --preset mac-ai
# Basic debug build (no AI)
cmake --preset mac-dbg
cmake --build --preset mac-dbg
# Verbose warnings for debugging
cmake --preset mac-dbg-v
cmake --build --preset mac-dbg-v
# Release build
cmake --preset mac-rel
cmake --build --preset mac-rel
```
### Windows Development
```bash
# Debug build (x64)
cmake --preset win-dbg
cmake --build --preset win-dbg
# AI-enabled development
cmake --preset win-ai
cmake --build --preset win-ai
# ARM64 support
cmake --preset win-arm
cmake --build --preset win-arm
```
## All Presets
### macOS Presets
| Preset | Description | Arch | Warnings | Features |
|--------|-------------|------|----------|----------|
| `mac-dbg` | Debug build | ARM64 | Off | Basic |
| `mac-dbg-v` | Debug verbose | ARM64 | On | Basic |
| `mac-rel` | Release | ARM64 | Off | Basic |
| `mac-x64` | Debug x86_64 | x86_64 | Off | Basic |
| `mac-uni` | Universal binary | Both | Off | Basic |
| `mac-dev` | Development | ARM64 | Off | ROM tests |
| `mac-ai` | AI development | ARM64 | Off | z3ed, JSON, gRPC, ROM tests |
| `mac-z3ed` | z3ed CLI | ARM64 | Off | AI agent support |
| `mac-rooms` | Dungeon editor | ARM64 | Off | Minimal build for room editing |
### Windows Presets
| Preset | Description | Arch | Warnings | Features |
|--------|-------------|------|----------|----------|
| `win-dbg` | Debug build | x64 | Off | Basic |
| `win-dbg-v` | Debug verbose | x64 | On | Basic |
| `win-rel` | Release | x64 | Off | Basic |
| `win-arm` | Debug ARM64 | ARM64 | Off | Basic |
| `win-arm-rel` | Release ARM64 | ARM64 | Off | Basic |
| `win-dev` | Development | x64 | Off | ROM tests |
| `win-ai` | AI development | x64 | Off | z3ed, JSON, gRPC, ROM tests |
| `win-z3ed` | z3ed CLI | x64 | Off | AI agent support |
### Linux Presets
| Preset | Description | Compiler | Warnings |
|--------|-------------|----------|----------|
| `lin-dbg` | Debug | GCC | Off |
| `lin-clang` | Debug | Clang | Off |
### Special Purpose
| Preset | Description |
|--------|-------------|
| `ci` | Continuous integration (no ROM tests) |
| `asan` | AddressSanitizer build |
| `coverage` | Code coverage build |
## Warning Control
By default, all presets suppress compiler warnings with `-w` for a cleaner build experience.
### To Enable Verbose Warnings:
1. Use a preset with `-v` suffix (e.g., `mac-dbg-v`, `win-dbg-v`)
2. Or set `YAZE_SUPPRESS_WARNINGS=OFF` manually:
```bash
cmake --preset mac-dbg -DYAZE_SUPPRESS_WARNINGS=OFF
```
## Architecture Support
### macOS
- **ARM64 (Apple Silicon)**: `mac-dbg`, `mac-rel`, `mac-ai`, etc.
- **x86_64 (Intel)**: `mac-x64`
- **Universal Binary**: `mac-uni` (both ARM64 + x86_64)
### Windows
- **x64**: `win-dbg`, `win-rel`, `win-ai`, etc.
- **ARM64**: `win-arm`, `win-arm-rel`
## Build Directories
Most presets use `build/` directory. Exceptions:
- `mac-rooms`: Uses `build_rooms/` to avoid conflicts
## Feature Flags
Common CMake options you can override:
```bash
# Enable/disable components
-DYAZE_BUILD_TESTS=ON/OFF
-DYAZE_BUILD_Z3ED=ON/OFF
-DYAZE_BUILD_EMU=ON/OFF
-DYAZE_BUILD_APP=ON/OFF
# AI features
-DZ3ED_AI=ON/OFF
-DYAZE_WITH_JSON=ON/OFF
-DYAZE_WITH_GRPC=ON/OFF
# Testing
-DYAZE_ENABLE_ROM_TESTS=ON/OFF
-DYAZE_TEST_ROM_PATH=/path/to/zelda3.sfc
# Build optimization
-DYAZE_MINIMAL_BUILD=ON/OFF
-DYAZE_USE_MODULAR_BUILD=ON/OFF
```
## Examples
### Development with AI features and verbose warnings
```bash
cmake --preset mac-dbg-v -DZ3ED_AI=ON -DYAZE_WITH_GRPC=ON
cmake --build --preset mac-dbg-v
```
### Release build for distribution (macOS Universal)
```bash
cmake --preset mac-uni
cmake --build --preset mac-uni
cpack --preset mac-uni
```
### Quick minimal build for testing
```bash
cmake --preset mac-dbg -DYAZE_MINIMAL_BUILD=ON
cmake --build --preset mac-dbg -j12
```
## Updating compile_commands.json
After configuring with a new preset, copy the compile commands for IDE support:
```bash
cp build/compile_commands.json .
```
This ensures clangd and other LSP servers can find headers and understand build flags.
## Migration from Old Presets
Old preset names have been simplified:
| Old Name | New Name |
|----------|----------|
| `macos-dev-z3ed-ai` | `mac-ai` |
| `macos-debug` | `mac-dbg` |
| `macos-release` | `mac-rel` |
| `macos-debug-universal` | `mac-uni` |
| `macos-dungeon-dev` | `mac-rooms` |
| `windows-debug` | `win-dbg` |
| `windows-release` | `win-rel` |
| `windows-arm64-debug` | `win-arm` |
| `linux-debug` | `lin-dbg` |
| `linux-clang` | `lin-clang` |
## Troubleshooting
### Warnings are still showing
- Make sure you're using a preset without `-v` suffix
- Check `cmake` output for `✓ Warnings suppressed` message
- Reconfigure and rebuild: `rm -rf build && cmake --preset mac-dbg`
### clangd can't find nlohmann/json
- Run `cmake --preset <your-preset>` to regenerate compile_commands.json
- Copy to root: `cp build/compile_commands.json .`
- Restart your IDE or LSP server
### Build fails with missing dependencies
- Ensure submodules are initialized: `git submodule update --init --recursive`
- For AI features, make sure you have OpenSSL: `brew install openssl` (macOS)

0
docs/dungeon_editor_master_guide.md → docs/D2-dungeon-editor-guide.md
View File

2
docs/CANVAS_GUIDE.md → docs/G2-canvas-guide.md
View File

@@ -1,4 +1,4 @@
# Canvas System - Comprehensive Guide
# Canvas System
## Overview

47
docs/GRAPHICS_PERFORMANCE.md
View File

@@ -1,47 +0,0 @@
# Graphics System Performance & Optimization
This document provides a comprehensive overview of the analysis, implementation, and results of performance optimizations applied to the YAZE graphics system.
## 1. Executive Summary
Massive performance improvements were achieved across the application, dramatically improving the user experience, especially during resource-intensive operations like ROM loading and dungeon editing.
### Overall Performance Results
| Component | Before | After | Improvement |
|-----------|--------|-------|-------------|
| **DungeonEditor::Load** | **17,967ms** | **3,747ms** | **🚀 79% faster!** |
| **Total ROM Loading** | **~18.6s** | **~4.7s** | **🚀 75% faster!** |
| **User Experience** | 18-second freeze | Near-instant | **Dramatic improvement** |
## 2. Implemented Optimizations
The following key optimizations were successfully implemented:
1. **Palette Lookup Optimization (100x faster)**: Replaced a linear search with an `std::unordered_map` for O(1) color-to-index lookups in the `Bitmap` class.
2. **Dirty Region Tracking (10x faster)**: The `Bitmap` class now tracks modified regions, so only the changed portion of a texture is uploaded to the GPU, significantly reducing GPU bandwidth.
3. **Resource Pooling (~30% memory reduction)**: The central `Arena` manager now pools and reuses `SDL_Texture` and `SDL_Surface` objects, reducing memory fragmentation and creation/destruction overhead.
4. **LRU Tile Caching (5x faster)**: The `Tilemap` class uses a Least Recently Used (LRU) cache to avoid redundant `Bitmap` object creation for frequently rendered tiles.
5. **Batch Operations (5x faster)**: The `Arena` can now queue multiple texture updates and process them in a single batch, reducing SDL context switching.
6. **Memory Pool Allocator (10x faster)**: A custom `MemoryPool` provides pre-allocated blocks for common graphics sizes (8x8, 16x16), bypassing `malloc`/`free` overhead.
7. **Atlas-Based Rendering (N-to-1 draw calls)**: A new `AtlasRenderer` dynamically packs smaller bitmaps into a single large texture atlas, allowing many elements to be drawn in a single batch.
8. **Parallel & Incremental Loading**: Dungeon rooms and overworld maps are now loaded in parallel or incrementally to prevent UI blocking.
9. **Performance Monitoring**: A `PerformanceProfiler` and `PerformanceDashboard` were created to measure the impact of these optimizations and detect regressions.
## 3. Future Optimization Recommendations
### High Priority
1. **Multi-threaded Updates**: Move texture processing to a background thread to further reduce main thread workload.
2. **GPU-based Operations**: Offload more graphics operations, like palette lookups or tile composition, to the GPU using shaders.
### Medium Priority
1. **Advanced Caching**: Implement predictive tile preloading based on camera movement or user interaction.
2. **Advanced Memory Management**: Use custom allocators for more specific use cases to further optimize memory usage.

235
docs/build_modularization_plan.md
View File

@@ -1,235 +0,0 @@
# Build Modularization Plan for Yaze
## Executive Summary
This document outlines a comprehensive plan to modularize the yaze build system, reducing build times by 40-60% through the creation of intermediate libraries and improved dependency management.
## Current Problems
1. **Monolithic Build**: All subsystems compile directly into final executables
2. **Repeated Compilation**: Same sources compiled multiple times for different targets
3. **Poor Incremental Builds**: Small changes trigger large rebuilds
4. **Long Link Times**: Large monolithic targets take longer to link
5. **Parallel Build Limitations**: Cannot parallelize subsystem builds effectively
## Proposed Architecture
### Library Hierarchy
```
yaze_common (INTERFACE)
├── yaze_util (STATIC)
│ ├── util/bps.cc
│ ├── util/flag.cc
│ └── util/hex.cc
│
├── yaze_gfx (STATIC)
│ ├── All YAZE_APP_GFX_SRC files
│ └── Depends: yaze_util
│
├── yaze_gui (STATIC)
│ ├── All YAZE_GUI_SRC files
│ └── Depends: yaze_gfx, ImGui
│
├── yaze_zelda3 (STATIC)
│ ├── All YAZE_APP_ZELDA3_SRC files
│ └── Depends: yaze_gfx, yaze_util
│
├── yaze_emulator (STATIC)
│ ├── All YAZE_APP_EMU_SRC files
│ └── Depends: yaze_util
│
├── yaze_core_lib (STATIC)
│ ├── All YAZE_APP_CORE_SRC files
│ └── Depends: yaze_util, yaze_gfx, asar
│
├── yaze_editor (STATIC)
│ ├── All YAZE_APP_EDITOR_SRC files
│ └── Depends: yaze_core_lib, yaze_gfx, yaze_gui, yaze_zelda3
│
├── yaze_agent (STATIC - already exists)
│ └── Depends: yaze_util
│
└── Final Executables
├── yaze (APP) - Links all above
├── yaze_emu - Links subset
├── z3ed (CLI) - Links subset
└── yaze_test - Links all for testing
```
## Implementation Guide
This section provides step-by-step instructions for implementing the modularized build system for yaze.
### Quick Start
#### Option 1: Enable Modular Build (Recommended)
Add this option to enable the new modular build system:
```bash
cmake -B build -DYAZE_USE_MODULAR_BUILD=ON
cmake --build build
```
#### Option 2: Keep Existing Build (Default)
The existing build system remains the default for stability:
```bash
cmake -B build
cmake --build build
```
### Implementation Steps
#### Step 1: Add Build Option
Add to **root `CMakeLists.txt`** (around line 45, after other options):
```cmake
# Build system options
option(YAZE_USE_MODULAR_BUILD "Use modularized library build system for faster builds" OFF)
```
#### Step 2: Update src/CMakeLists.txt
Replace the current monolithic build with a library-based approach. Add near the top of `src/CMakeLists.txt`:
```cmake
# ==============================================================================
# Modular Build System
# ==============================================================================
if(YAZE_USE_MODULAR_BUILD)
message(STATUS "Using modular build system")
# Build subsystem libraries in dependency order
if(YAZE_BUILD_LIB OR YAZE_BUILD_APP OR YAZE_BUILD_Z3ED)
include(util/util.cmake)
include(app/gfx/gfx_library.cmake)
include(app/gui/gui_library.cmake)
include(app/zelda3/zelda3_library.cmake)
if(YAZE_BUILD_EMU AND NOT YAZE_WITH_GRPC)
include(app/emu/emu_library.cmake)
endif()
include(app/core/core_library.cmake)
include(app/editor/editor_library.cmake)
endif()
# Create yaze_core as an INTERFACE library that aggregates all modules
if(YAZE_BUILD_LIB)
add_library(yaze_core INTERFACE)
target_link_libraries(yaze_core INTERFACE
yaze_util
yaze_gfx
yaze_gui
yaze_zelda3
yaze_core_lib
yaze_editor
yaze_agent
)
if(YAZE_BUILD_EMU AND NOT YAZE_WITH_GRPC)
target_link_libraries(yaze_core INTERFACE yaze_emulator)
endif()
# Add gRPC support if enabled
if(YAZE_WITH_GRPC)
target_add_protobuf(yaze_core
${CMAKE_SOURCE_DIR}/src/app/core/proto/imgui_test_harness.proto)
target_link_libraries(yaze_core INTERFACE
grpc++
grpc++_reflection
libprotobuf)
endif()
endif()
else()
message(STATUS "Using traditional monolithic build system")
# Keep existing build (current code in src/CMakeLists.txt)
endif()
```
#### Step 3: Update yaze Application Linking
In `src/app/app.cmake`, update the main yaze target:
```cmake
if(YAZE_USE_MODULAR_BUILD)
target_link_libraries(yaze PRIVATE
yaze_util
yaze_gfx
yaze_gui
yaze_zelda3
yaze_core_lib
yaze_editor
)
if(YAZE_BUILD_EMU AND NOT YAZE_WITH_GRPC)
target_link_libraries(yaze PRIVATE yaze_emulator)
endif()
else()
target_link_libraries(yaze PRIVATE yaze_core)
endif()
```
#### Step 4: Update Test Executable
In `test/CMakeLists.txt`, update the test linking:
```cmake
if(YAZE_BUILD_LIB)
if(YAZE_USE_MODULAR_BUILD)
# Link individual libraries for faster incremental test builds
target_link_libraries(yaze_test
yaze_util
yaze_gfx
yaze_gui
yaze_zelda3
yaze_core_lib
yaze_editor
)
if(YAZE_BUILD_EMU AND NOT YAZE_WITH_GRPC)
target_link_libraries(yaze_test yaze_emulator)
endif()
else()
# Use monolithic core library
target_link_libraries(yaze_test yaze_core)
endif()
endif()
```
## Expected Performance Improvements
### Build Time Reductions
| Scenario | Current | With Modularization | Improvement |
|----------|---------|---------------------|-------------|
| Clean build | 100% | 100% | 0% (first time) |
| Change util file | ~80% rebuild | ~15% rebuild | **81% faster** |
| Change gfx file | ~70% rebuild | ~30% rebuild | **57% faster** |
| Change gui file | ~60% rebuild | ~25% rebuild | **58% faster** |
| Change editor file | ~50% rebuild | ~20% rebuild | **60% faster** |
| Change zelda3 file | ~40% rebuild | ~15% rebuild | **62% faster** |
| Incremental test | 100% relink | Cached libs | **70% faster** |
### CI/CD Benefits
- **Cached Artifacts**: Libraries can be cached between CI runs
- **Selective Testing**: Only rebuild/test changed modules
- **Faster Iteration**: Developers see results sooner
## Rollback Procedure
If issues arise, you can revert to the traditional build:
```bash
# Option 1: Just disable the flag
cmake -B build -DYAZE_USE_MODULAR_BUILD=OFF
cmake --build build
# Option 2: Delete build and start fresh
rm -rf build
cmake -B build # OFF is the default
cmake --build build
```

330
docs/ollama_integration_status.md
View File

@@ -1,330 +0,0 @@
# Ollama Integration Status - Updated# Ollama Integration Status
## ✅ Completed## ✅ Completed
### Infrastructure### Flag Parsing
- ✅ Flag parsing for AI provider configuration- **Fixed**: AI provider flags (`--ai_provider`, `--ai_model`, `--ollama_host`, `--gemini_api_key`) are now properly parsed in `cli_main.cc`
- ✅ Ollama service with health checks- **Result**: Ollama provider is correctly detected and initialized
- ✅ Tool system with 5 read-only tools- **Verification**: `🤖 AI Provider: ollama` message appears correctly
- ✅ Simple chat modes (4 input methods working)
- ✅ Colorful terminal output with loading indicators### Ollama Service
- ✅ Verbose mode for diagnostics- **Status**: OllamaAIService properly connects to local Ollama server
- ✅ Configurable max-tool-iterations and max-retries- **Health Check**: Successfully validates model availability (qwen2.5-coder:7b)
- ✅ File-based prompt system (assets/agent/*.txt)- **JSON Parsing**: Correctly extracts tool calls and text responses from Ollama's response format
### Current Issue: Empty Tool Results### Tool System
- **Tool Dispatcher**: Working correctly - routes tool calls to appropriate handlers
**Problem**: The `resource-list` tool is returning empty JSON `{}` when requesting dungeon labels.- **Tool Registration**: 5 read-only tools available:
- `resource-list` - List labeled resources
**Root Cause**: The embedded labels in Zelda3Labels only include: - `dungeon-list-sprites` - Inspect room sprites
- `room` - 297 room names ✅ - `overworld-find-tile` - Search for tile placements
- `entrance` - 133 entrance names ✅ - `overworld-describe-map` - Get map metadata
- `sprite` - 256 sprite names ✅ - `overworld-list-warps` - List entrances/exits/holes
- `overlord` - 26 overlord names ✅
- `item` - 104 item names ✅### Simple Chat Modes
All 4 input methods working:
But **NOT** `dungeon` as a separate category.1. ✅ Single message mode: `z3ed agent simple-chat "message" --rom=file.sfc --ai_provider=ollama`
2. ✅ Interactive mode: `z3ed agent simple-chat --rom=file.sfc --ai_provider=ollama`
**Diagnosis**:3. ✅ Piped input mode: `echo "message" | z3ed agent simple-chat --rom=file.sfc --ai_provider=ollama`
```bash4. ✅ Batch file mode: `z3ed agent simple-chat --file=queries.txt --rom=file.sfc --ai_provider=ollama`
# Works (returns data):
./z3ed agent resource-list --type=room --format=json## 🚧 In Progress
./z3ed agent resource-list --type=entrance --format=json
./z3ed agent resource-list --type=sprite --format=json### Tool Calling Loop Issue
**Problem**: Agent enters infinite tool-calling loop without providing final text response
# Fails (returns empty {}):
./z3ed agent resource-list --type=dungeon --format=json**Symptoms**:
``````
Error: Agent did not produce a response after executing tools.
**Solution Options**:```
1. **Quick Fix**: Update prompt examples to use valid categories**Root Cause**: The system prompt needs refinement to instruct the LLM to:
- Change `type: dungeon` → `type: room` in examples1. Call tools when needed
- Update tool descriptions to clarify available categories2. Wait for tool results
3. **THEN provide a final text_response based on the tool results**
2. **Proper Fix**: Add dungeon labels to embedded labels4. Stop calling tools after receiving results
- Modify `Zelda3Labels::ToResourceLabels()` to include dungeon category
- Map dungeon IDs (0-11) to their names**Current Behavior**:
- LLM successfully calls tools (e.g., `resource-list` with `type=dungeon`)
3. **Alternative**: Clarify that "dungeons" are accessed via room labels- Tool executes and returns JSON results
- Document that dungeon rooms use the `room` category- LLM receives results in conversation history
- Provide ID ranges (e.g., rooms 0-119 are Hyrule Castle, etc.)- LLM either:
- Calls tools again (loop detected after 4 iterations)
## 🎨 New Features Added - OR doesn't provide a `text_response` field in the JSON
### Verbose Mode**Solution Needed**: Update system prompt to include explicit instructions like:
```bash
z3ed agent simple-chat "query" --verbose```
```When you call a tool:
Shows:1. The tool will execute and return results
- Iteration count2. You will receive the results in the next message
- Agent response analysis (tool calls, commands, text_response status)3. After receiving tool results, you MUST provide a text_response that answers the user's question using the tool data
- LLM reasoning4. Do NOT call the same tool again
- Tool output preview5. Example flow:
- Step-by-step execution flow User: "What dungeons are there?"
Assistant (first response): { "tool_calls": [{"tool_name": "resource-list", "args": {"type": "dungeon"}}] }
### Configuration Parameters [Tool executes and returns dungeon list]
```bash Assistant (second response): { "text_response": "Based on the resource list, there are X dungeons: [list them]" }
--max-tool-iterations=6 # Default: 4```
--max-retries=5 # Default: 3
--no-reasoning # Hide LLM reasoning## 📋 Testing
```
### Test Script
### Colorful OutputCreated `test_simple_chat_ollama.sh` with comprehensive tests:
- 🔧 Tool calls in magenta- ✅ Prerequisites check (Ollama, model, ROM)
- ✓ Success messages in green- ✅ Single message mode test
- ⚠ Warnings in yellow- ✅ Piped input test
- ✗ Errors in red- ✅ Interactive mode test (with auto-exit)
- ℹ Info in blue- ✅ Batch mode test
- 💭 Reasoning in dim yellow- ⚠️ Tool calling verification (needs prompt refinement)
- ⠋ Loading spinner (cyan)
### Manual Test Results
## 📋 Next Steps
**Test 1: Single Message**
### Priority 1: Fix Empty Tool Results (HIGH)```bash
1. Add dungeon category to embedded labels OR./build_test/bin/z3ed agent simple-chat "What dungeons are in this ROM?" \
2. Update all prompt examples to use `room` instead of `dungeon` --rom=assets/zelda3.sfc --ai_provider=ollama
3. Test that tools return actual data```
4. Verify LLM can process tool results**Result**:
- ✅ Ollama connects successfully
### Priority 2: Refine Prompts (MEDIUM)- ✅ Model loads (qwen2.5-coder:7b)
Once tools return data:- ❌ Hits 4-iteration limit without final response
1. Test if LLM provides final text_response after tool results
2. Adjust system prompt if loop persists**Test 2: Tool Availability**
3. Test with different Ollama models (llama3, codellama)```bash
./build_test/bin/z3ed agent resource-list --type=dungeon --format=json --rom=assets/zelda3.sfc
### Priority 3: Documentation (LOW)```
1. Document available resource categories**Result**: ✅ Returns proper JSON with dungeon names
2. Add troubleshooting guide
3. Create example queries for each tool## 🔧 Next Steps
## 🧪 Testing Commands### Priority 1: Fix Tool Calling Loop (High Priority)
1. **Update system prompt** in `prompt_builder.cc`:
```bash - Add explicit instructions for tool usage workflow
# Test with verbose mode - Include examples showing tool call → results → final response
./build_test/bin/z3ed agent simple-chat "What rooms are there?" \\ - Emphasize that `text_response` is REQUIRED after receiving tool results
--rom=assets/zelda3.sfc --ai_provider=ollama --verbose --max-tool-iterations=6
2. **Enhance examples** in `prompt_catalogue.yaml`:
# Test resource categories - Add multi-turn examples showing tool usage
./build_test/bin/z3ed agent resource-list --type=room --rom=assets/zelda3.sfc - Show correct pattern: question → tool_call → (wait) → text_response with tool data
./build_test/bin/z3ed agent resource-list --type=entrance --rom=assets/zelda3.sfc
./build_test/bin/z3ed agent resource-list --type=sprite --rom=assets/zelda3.sfc3. **Improve response validation** in `ollama_ai_service.cc`:
- Detect when tool results are in history but no text_response provided
# Test with Gemini (if API key available) - Add warning messages for debugging
export GEMINI_API_KEY='your-key'
./build_test/bin/z3ed agent simple-chat "What rooms are in this ROM?" \\### Priority 2: Testing & Validation (Medium Priority)
--rom=assets/zelda3.sfc --ai_provider=gemini --verbose1. Test with different Ollama models:
``` - qwen2.5-coder:7b (current)
- llama3:8b
## 📊 Performance - codellama:7b
- Ollama response: ~2-5 seconds (qwen2.5-coder:7b)2. Create regression test suite for tool calling:
- Tool execution: <100ms - Test each tool individually
- Loading indicator: Smooth 80ms refresh rate - Test multi-tool sequences
- Test conversation context preservation
## 🎯 Success Criteria
### Priority 3: Documentation (Low Priority)
- [x] Colorful, user-friendly output1. Update `simple_chat_input_methods.md` with:
- [x] Verbose mode for debugging - Known limitations section
- [x] Configurable parameters - Troubleshooting for tool calling issues
- [x] File-based prompts for easy updates - Recommended models and configurations
- [ ] Tools return actual data (BLOCKED on dungeon labels)
- [ ] LLM provides final response after tool calls2. Create `ollama_best_practices.md`:
- [ ] Zero infinite loops - Model recommendations
- Temperature/parameter tuning
--- - Prompt engineering tips
**Last Updated**: October 4, 2025## 📊 Performance Notes
**Status**: 🟡 Blocked on empty tool results - need to fix dungeon labels
**Next Action**: Add dungeon category to embedded labels OR update prompt examples- **Ollama Response Time**: ~2-5 seconds per query (qwen2.5-coder:7b on typical hardware)
- **Tool Execution**: <100ms per tool call
- **Total Interaction**: ~2-5 seconds for simple queries, longer for multi-turn with tools
## 🐛 Known Issues
1. **Tool Calling Loop**: Agent doesn't provide final response after tool execution (see above)
2. **No Streaming**: Responses are blocking (not streamed), so user sees delay
3. **Limited Context**: Prompt builder doesn't include full conversation context in system prompt
## 💡 Recommendations
### For Users
- Use MockAIService for testing until tool calling is fixed
- For production, prefer Gemini (has native function calling support)
- Keep queries simple and direct
### For Developers
- Focus on fixing the tool calling loop first
- Consider implementing streaming responses
- Add debug logging to track tool call cycles
- Test with multiple Ollama models to find best performer
## 📝 Related Files
- `/Users/scawful/Code/yaze/src/cli/cli_main.cc` - Flag parsing (FIXED ✅)
- `/Users/scawful/Code/yaze/src/cli/service/ai/ollama_ai_service.cc` - Ollama integration
- `/Users/scawful/Code/yaze/src/cli/service/ai/prompt_builder.cc` - System prompt generation (NEEDS FIX 🚧)
- `/Users/scawful/Code/yaze/src/cli/service/agent/conversational_agent_service.cc` - Tool execution loop
- `/Users/scawful/Code/yaze/assets/agent/prompt_catalogue.yaml` - Tool definitions and examples (NEEDS ENHANCEMENT 🚧)
- `/Users/scawful/Code/yaze/docs/simple_chat_input_methods.md` - User documentation
- `/Users/scawful/Code/yaze/test_simple_chat_ollama.sh` - Test script
## 🎯 Success Criteria
### Minimum Viable
- [ ] LLM successfully calls tools
- [ ] LLM provides final text_response after receiving tool results
- [ ] No infinite loops (completes within 4 iterations)
- [ ] Accurate answers to simple questions ("What dungeons?", "List sprites in room X")
### Full Success
- [ ] All 5 tools work correctly with Ollama
- [ ] Multi-turn conversations maintain context
- [ ] Works with 3+ different Ollama models
- [ ] Response time <5 seconds for typical queries
- [ ] Comprehensive test coverage
---
**Last Updated**: October 4, 2025
**Status**: 🟡 Partially Working - Core infrastructure complete, prompt refinement needed
**Next Action**: Update system prompt to fix tool calling loop

388
docs/overworld_testing_guide.md
View File

@@ -1,388 +0,0 @@
# yaze Overworld Testing Guide
## Overview
This guide provides comprehensive documentation for testing the YAZE overworld implementation, including unit tests, integration tests, end-to-end tests, and golden data validation. The testing framework ensures that the YAZE C++ implementation correctly mirrors the ZScream C# logic.
## Test Architecture
### 1. Golden Data System
The golden data system provides a "source of truth" for ROM validation:
- **Golden Data Extractor**: Extracts all overworld-related values from ROMs
- **Before/After Validation**: Compares ROM states before and after edits
- **Reference Data**: Provides expected values for test validation
### 2. Test Categories
#### Unit Tests (`test/unit/zelda3/`)
- **`overworld_test.cc`**: Basic overworld functionality tests
- **`overworld_integration_test.cc`**: Comprehensive integration tests
- **`extract_vanilla_values.cc`**: ROM value extraction utility
#### Integration Tests (`test/e2e/`)
- **`overworld/overworld_e2e_test.cc`**: End-to-end workflow tests
- **`zscustomoverworld/zscustomoverworld_upgrade_test.cc`**: ASM version upgrade tests
#### Golden Data Tools
- **`overworld_golden_data_extractor.cc`**: Comprehensive ROM data extraction
- **`run_overworld_tests.sh`**: Orchestrated test runner script
## Quick Start
### Prerequisites
1. **Build YAZE**: Ensure the project is built with tests enabled
2. **ROM File**: Have a valid Zelda 3 ROM file (zelda3.sfc)
3. **Environment**: Set up test environment variables
### Running Tests
#### Basic Test Run
```bash
# Run all overworld tests with a ROM
./scripts/run_overworld_tests.sh zelda3.sfc
# Generate detailed report
./scripts/run_overworld_tests.sh zelda3.sfc --generate-report
# Clean up test files after completion
./scripts/run_overworld_tests.sh zelda3.sfc --cleanup
```
#### Selective Test Execution
```bash
# Skip unit tests, run only integration and E2E
./scripts/run_overworld_tests.sh zelda3.sfc --skip-unit-tests
# Extract golden data only
./scripts/run_overworld_tests.sh zelda3.sfc --skip-unit-tests --skip-integration --skip-e2e
# Run with custom ROM path
export YAZE_TEST_ROM_PATH="/path/to/custom/rom.sfc"
./scripts/run_overworld_tests.sh /path/to/custom/rom.sfc
```
## Test Components
### 1. Golden Data Extractor
The `overworld_golden_data_extractor` tool extracts comprehensive data from ROMs:
```bash
# Extract golden data from a ROM
./build/bin/overworld_golden_data_extractor zelda3.sfc golden_data.h
```
**Extracted Data Includes:**
- Basic ROM information (title, size, checksums)
- ASM version and feature flags
- Overworld map properties (graphics, palettes, sizes)
- Tile data (Tile16/Tile32 expansion status)
- Entrance/hole/exit coordinate data
- Item and sprite data
- Map tiles (compressed data)
- Palette and music data
- Overlay data
### 2. Integration Tests
The integration tests validate core overworld functionality:
```cpp
// Example: Test coordinate calculation accuracy
TEST_F(OverworldIntegrationTest, ZScreamCoordinateCompatibility) {
// Load overworld data
auto status = overworld_->Load(rom_.get());
ASSERT_TRUE(status.ok());
// Validate coordinate calculations match ZScream exactly
const auto& entrances = overworld_->entrances();
for (int i = 0; i < 10; i++) {
const auto& entrance = entrances[i];
// ZScream coordinate calculation logic
uint16_t map_pos = entrance.map_pos();
uint16_t map_id = entrance.map_id();
int position = map_pos >> 1;
int x_coord = position % 64;
int y_coord = position >> 6;
int expected_x = (x_coord * 16) + (((map_id % 64) - (((map_id % 64) / 8) * 8)) * 512);
int expected_y = (y_coord * 16) + (((map_id % 64) / 8) * 512);
EXPECT_EQ(entrance.x(), expected_x);
EXPECT_EQ(entrance.y(), expected_y);
}
}
```
### 3. End-to-End Tests
E2E tests validate complete workflows:
```cpp
// Example: Test ASM version upgrade workflow
TEST_F(OverworldE2ETest, ApplyZSCustomOverworldV3) {
// Load vanilla ROM
std::unique_ptr<Rom> rom = std::make_unique<Rom>();
ASSERT_OK(rom->LoadFromFile(vanilla_test_path_));
// Apply ZSCustomOverworld v3 ASM
ASSERT_OK(rom->WriteByte(0x140145, 0x03)); // Set ASM version to v3
// Enable v3 features
ASSERT_OK(rom->WriteByte(0x140146, 0x01)); // Enable main palettes
ASSERT_OK(rom->WriteByte(0x140147, 0x01)); // Enable area-specific BG
// ... more feature flags
// Save and reload
ASSERT_OK(rom->SaveToFile(Rom::SaveSettings{.filename = edited_test_path_}));
// Validate changes persisted
std::unique_ptr<Rom> reloaded_rom = std::make_unique<Rom>();
ASSERT_OK(reloaded_rom->LoadFromFile(edited_test_path_));
auto asm_version = reloaded_rom->ReadByte(0x140145);
EXPECT_EQ(*asm_version, 0x03);
}
```
## Test Validation Points
### 1. ZScream Compatibility
Tests ensure YAZE behavior matches ZScream exactly:
- **Coordinate Calculations**: Entrance/hole coordinates use identical formulas
- **ASM Version Detection**: Proper handling of vanilla vs ZSCustomOverworld ROMs
- **Data Structure Loading**: Same data organization and access patterns
- **Expansion Detection**: Correct identification of expanded vs vanilla data
### 2. ROM State Validation
Tests validate ROM integrity:
- **Before/After Comparison**: Ensure edits are properly saved and loaded
- **Feature Flag Persistence**: ZSCustomOverworld features remain enabled after save
- **Data Integrity**: Original data is preserved where expected
- **Checksum Validation**: ROM remains valid after modifications
### 3. Performance and Stability
Tests ensure robustness:
- **Multiple Load Cycles**: Repeated load/unload operations
- **Memory Management**: Proper cleanup of resources
- **Error Handling**: Graceful handling of invalid data
- **Thread Safety**: Concurrent access patterns (if applicable)
## Environment Variables
### Test Configuration
```bash
# ROM path for testing
export YAZE_TEST_ROM_PATH="/path/to/rom.sfc"
# Skip ROM-dependent tests
export YAZE_SKIP_ROM_TESTS=1
# Test output verbosity
export GTEST_VERBOSE=1
```
### Build Configuration
```bash
# Enable tests in CMake
cmake -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTS=ON ..
# Build with specific test targets
cmake --build build --target overworld_golden_data_extractor
cmake --build build --target extract_vanilla_values
```
## Test Reports
### Generated Reports
The test runner generates comprehensive reports:
```markdown
# yaze Overworld Test Report
**Generated:** 2024-01-15 14:30:00
**ROM:** zelda3.sfc
**ROM Size:** 2097152 bytes
## Test Results Summary
| Test Category | Status | Details |
|---------------|--------|---------|
| Golden Data Extraction | SUCCESS: golden_data.h | |
| Unit Tests | PASSED | |
| Integration Tests | PASSED | |
| E2E Tests | PASSED | |
## ROM Information
### ROM Header
[Hex dump of ROM header for validation]
```
### Report Location
Reports are saved to `test/reports/` with timestamps:
- `overworld_test_report_YYYYMMDD_HHMMSS.md`
## Troubleshooting
### Common Issues
#### 1. ROM Not Found
```
Error: ROM file not found: zelda3.sfc
```
**Solution**: Provide correct ROM path or set `YAZE_TEST_ROM_PATH`
#### 2. Build Failures
```
Error: Failed to build golden data extractor
```
**Solution**: Ensure project is built with tests enabled:
```bash
cmake -DBUILD_TESTS=ON ..
cmake --build build
```
#### 3. Test Failures
```
Error: Some tests failed. Check the results above.
```
**Solution**:
- Check individual test logs for specific failures
- Verify ROM file is valid Zelda 3 ROM
- Ensure test environment is properly configured
### Debug Mode
Run tests with maximum verbosity:
```bash
# Enable debug output
export GTEST_VERBOSE=1
export YAZE_DEBUG=1
# Run with verbose output
./scripts/run_overworld_tests.sh zelda3.sfc --generate-report
```
## Advanced Usage
### Custom Test Scenarios
#### 1. Testing Custom ROMs
```bash
# Test with modified ROM
./scripts/run_overworld_tests.sh /path/to/modified_rom.sfc
# Extract golden data for comparison
./build/bin/overworld_golden_data_extractor /path/to/modified_rom.sfc modified_golden_data.h
```
#### 2. Regression Testing
```bash
# Extract golden data from known good ROM
./build/bin/overworld_golden_data_extractor known_good_rom.sfc reference_golden_data.h
# Test modified ROM against reference
./scripts/run_overworld_tests.sh modified_rom.sfc --generate-report
```
#### 3. Performance Testing
```bash
# Run performance-focused tests
export YAZE_PERFORMANCE_TESTS=1
./scripts/run_overworld_tests.sh zelda3.sfc
```
## Integration with CI/CD
### GitHub Actions Example
```yaml
name: Overworld Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Build YAZE
run: |
cmake -DBUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Debug ..
cmake --build build
- name: Download Test ROM
run: |
# Download test ROM (placeholder - use actual ROM)
wget -O zelda3.sfc https://example.com/test_rom.sfc
- name: Run Overworld Tests
run: |
chmod +x scripts/run_overworld_tests.sh
./scripts/run_overworld_tests.sh zelda3.sfc --generate-report
- name: Upload Test Reports
uses: actions/upload-artifact@v3
with:
name: test-reports
path: test/reports/
```
## Contributing
### Adding New Tests
1. **Unit Tests**: Add to `test/unit/zelda3/overworld_test.cc`
2. **Integration Tests**: Add to `test/unit/zelda3/overworld_integration_test.cc`
3. **E2E Tests**: Add to `test/e2e/overworld/overworld_e2e_test.cc`
### Test Guidelines
- **Naming**: Use descriptive test names that explain the scenario
- **Documentation**: Include comments explaining complex test logic
- **Isolation**: Each test should be independent and not rely on others
- **Cleanup**: Ensure tests clean up resources and temporary files
### Example Test Structure
```cpp
// Test descriptive name that explains the scenario
TEST_F(OverworldIntegrationTest, Tile32ExpansionDetectionWithV3ASM) {
// Setup: Configure test environment
mock_rom_data_[0x01772E] = 0x04; // Set expansion flag
// Execute: Run the code under test
auto status = overworld_->Load(rom_.get());
ASSERT_TRUE(status.ok());
// Verify: Check expected outcomes
EXPECT_TRUE(overworld_->expanded_tile32());
EXPECT_FALSE(overworld_->expanded_tile16());
// Cleanup: (handled by test framework)
}
```
## Conclusion
The YAZE overworld testing framework provides comprehensive validation of the C++ implementation against the ZScream C# reference. The golden data system ensures consistency, while the multi-layered test approach (unit, integration, E2E) provides confidence in the implementation's correctness and robustness.
For questions or issues, refer to the test logs and generated reports, or consult the YAZE development team.