feat: Revise CMake presets and documentation for improved clarity and functionality
- Reorganized CMake preset system, simplifying names and enhancing architecture support. - Introduced new presets for macOS and Windows development, including AI-enabled and verbose options. - Updated documentation to reflect changes in preset usage, design principles, and troubleshooting steps. - Removed outdated documents related to stability improvements and modularization plans to streamline resources.
This commit is contained in:
scawful
@@ -1,109 +1,207 @@
|
|||||||
# Build Presets Guide
|
# Build Presets Guide
|
||||||
|
|
||||||
CMake presets for development workflow and architecture-specific builds.
|
This document explains the reorganized CMake preset system for Yaze.
|
||||||
|
|
||||||
## 🍎 macOS ARM64 Presets (Recommended for Apple Silicon)
|
## Design Principles
|
||||||
|
|
||||||
### For Development Work:
|
1. **Short, memorable names** - No more `macos-dev-z3ed-ai`, just `mac-ai`
|
||||||
|
2. **Warnings off by default** - Add `-v` suffix for verbose (e.g., `mac-dbg-v`)
|
||||||
|
3. **Clear architecture support** - Explicit ARM64 and x86_64 presets
|
||||||
|
4. **Platform prefixes** - `mac-`, `win-`, `lin-` for easy identification
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### macOS Development
|
||||||
```bash
|
```bash
|
||||||
# ARM64-only development build with ROM testing
|
# Most common: AI-enabled development
|
||||||
cmake --preset macos-dev
|
cmake --preset mac-ai
|
||||||
cmake --build --preset macos-dev
|
cmake --build --preset mac-ai
|
||||||
|
|
||||||
# ARM64-only debug build
|
# Basic debug build (no AI)
|
||||||
cmake --preset macos-debug
|
cmake --preset mac-dbg
|
||||||
cmake --build --preset macos-debug
|
cmake --build --preset mac-dbg
|
||||||
|
|
||||||
# ARM64-only release build
|
# Verbose warnings for debugging
|
||||||
cmake --preset macos-release
|
cmake --preset mac-dbg-v
|
||||||
cmake --build --preset macos-release
|
cmake --build --preset mac-dbg-v
|
||||||
|
|
||||||
|
# Release build
|
||||||
|
cmake --preset mac-rel
|
||||||
|
cmake --build --preset mac-rel
|
||||||
```
|
```
|
||||||
|
|
||||||
### For Distribution:
|
### Windows Development
|
||||||
```bash
|
```bash
|
||||||
# Universal binary (ARM64 + x86_64) - use only when needed for distribution
|
# Debug build (x64)
|
||||||
cmake --preset macos-debug-universal
|
cmake --preset win-dbg
|
||||||
cmake --build --preset macos-debug-universal
|
cmake --build --preset win-dbg
|
||||||
|
|
||||||
cmake --preset macos-release-universal
|
# AI-enabled development
|
||||||
cmake --build --preset macos-release-universal
|
cmake --preset win-ai
|
||||||
|
cmake --build --preset win-ai
|
||||||
|
|
||||||
|
# ARM64 support
|
||||||
|
cmake --preset win-arm
|
||||||
|
cmake --build --preset win-arm
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🔧 Why This Fixes Architecture Errors
|
## All Presets
|
||||||
|
|
||||||
**Problem**: The original presets used `CMAKE_OSX_ARCHITECTURES: "x86_64;arm64"` which forced CMake to build universal binaries. This caused issues because:
|
### macOS Presets
|
||||||
- Dependencies like Abseil tried to use x86 SSE instructions (`-msse4.1`)
|
|
||||||
- These instructions don't exist on ARM64 processors
|
|
||||||
- Build failed with "unsupported option '-msse4.1' for target 'arm64-apple-darwin'"
|
|
||||||
|
|
||||||
**Solution**: The new ARM64-only presets use `CMAKE_OSX_ARCHITECTURES: "arm64"` which:
|
| Preset | Description | Arch | Warnings | Features |
|
||||||
- ✅ Only targets ARM64 architecture
|
|--------|-------------|------|----------|----------|
|
||||||
- ✅ Prevents x86-specific instruction usage
|
| `mac-dbg` | Debug build | ARM64 | Off | Basic |
|
||||||
- ✅ Uses ARM64 optimizations instead
|
| `mac-dbg-v` | Debug verbose | ARM64 | On | Basic |
|
||||||
- ✅ Builds much faster (no cross-compilation)
|
| `mac-rel` | Release | ARM64 | Off | Basic |
|
||||||
|
| `mac-x64` | Debug x86_64 | x86_64 | Off | Basic |
|
||||||
|
| `mac-uni` | Universal binary | Both | Off | Basic |
|
||||||
|
| `mac-dev` | Development | ARM64 | Off | ROM tests |
|
||||||
|
| `mac-ai` | AI development | ARM64 | Off | z3ed, JSON, gRPC, ROM tests |
|
||||||
|
| `mac-z3ed` | z3ed CLI | ARM64 | Off | AI agent support |
|
||||||
|
| `mac-rooms` | Dungeon editor | ARM64 | Off | Minimal build for room editing |
|
||||||
|
|
||||||
## 📋 Available Presets
|
### Windows Presets
|
||||||
|
|
||||||
| Preset Name | Architecture | Purpose | ROM Tests |
|
| Preset | Description | Arch | Warnings | Features |
|
||||||
|-------------|-------------|---------|-----------|
|
|--------|-------------|------|----------|----------|
|
||||||
| `macos-dev` | ARM64 only | Development | ✅ Enabled |
|
| `win-dbg` | Debug build | x64 | Off | Basic |
|
||||||
| `macos-debug` | ARM64 only | Debug builds | ❌ Disabled |
|
| `win-dbg-v` | Debug verbose | x64 | On | Basic |
|
||||||
| `macos-release` | ARM64 only | Release builds | ❌ Disabled |
|
| `win-rel` | Release | x64 | Off | Basic |
|
||||||
| `macos-debug-universal` | Universal | Distribution debug | ❌ Disabled |
|
| `win-arm` | Debug ARM64 | ARM64 | Off | Basic |
|
||||||
| `macos-release-universal` | Universal | Distribution release | ❌ Disabled |
|
| `win-arm-rel` | Release ARM64 | ARM64 | Off | Basic |
|
||||||
|
| `win-dev` | Development | x64 | Off | ROM tests |
|
||||||
|
| `win-ai` | AI development | x64 | Off | z3ed, JSON, gRPC, ROM tests |
|
||||||
|
| `win-z3ed` | z3ed CLI | x64 | Off | AI agent support |
|
||||||
|
|
||||||
## 🚀 Quick Start
|
### Linux Presets
|
||||||
|
|
||||||
For most development work on Apple Silicon:
|
| Preset | Description | Compiler | Warnings |
|
||||||
|
|--------|-------------|----------|----------|
|
||||||
|
| `lin-dbg` | Debug | GCC | Off |
|
||||||
|
| `lin-clang` | Debug | Clang | Off |
|
||||||
|
|
||||||
|
### Special Purpose
|
||||||
|
|
||||||
|
| Preset | Description |
|
||||||
|
|--------|-------------|
|
||||||
|
| `ci` | Continuous integration (no ROM tests) |
|
||||||
|
| `asan` | AddressSanitizer build |
|
||||||
|
| `coverage` | Code coverage build |
|
||||||
|
|
||||||
|
## Warning Control
|
||||||
|
|
||||||
|
By default, all presets suppress compiler warnings with `-w` for a cleaner build experience.
|
||||||
|
|
||||||
|
### To Enable Verbose Warnings:
|
||||||
|
|
||||||
|
1. Use a preset with `-v` suffix (e.g., `mac-dbg-v`, `win-dbg-v`)
|
||||||
|
2. Or set `YAZE_SUPPRESS_WARNINGS=OFF` manually:
|
||||||
|
```bash
|
||||||
|
cmake --preset mac-dbg -DYAZE_SUPPRESS_WARNINGS=OFF
|
||||||
|
```
|
||||||
|
|
||||||
|
## Architecture Support
|
||||||
|
|
||||||
|
### macOS
|
||||||
|
- **ARM64 (Apple Silicon)**: `mac-dbg`, `mac-rel`, `mac-ai`, etc.
|
||||||
|
- **x86_64 (Intel)**: `mac-x64`
|
||||||
|
- **Universal Binary**: `mac-uni` (both ARM64 + x86_64)
|
||||||
|
|
||||||
|
### Windows
|
||||||
|
- **x64**: `win-dbg`, `win-rel`, `win-ai`, etc.
|
||||||
|
- **ARM64**: `win-arm`, `win-arm-rel`
|
||||||
|
|
||||||
|
## Build Directories
|
||||||
|
|
||||||
|
Most presets use `build/` directory. Exceptions:
|
||||||
|
- `mac-rooms`: Uses `build_rooms/` to avoid conflicts
|
||||||
|
|
||||||
|
## Feature Flags
|
||||||
|
|
||||||
|
Common CMake options you can override:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Clean build
|
# Enable/disable components
|
||||||
rm -rf build
|
-DYAZE_BUILD_TESTS=ON/OFF
|
||||||
|
-DYAZE_BUILD_Z3ED=ON/OFF
|
||||||
|
-DYAZE_BUILD_EMU=ON/OFF
|
||||||
|
-DYAZE_BUILD_APP=ON/OFF
|
||||||
|
|
||||||
# Configure for ARM64 development
|
# AI features
|
||||||
cmake --preset macos-dev
|
-DZ3ED_AI=ON/OFF
|
||||||
|
-DYAZE_WITH_JSON=ON/OFF
|
||||||
|
-DYAZE_WITH_GRPC=ON/OFF
|
||||||
|
|
||||||
# Build
|
# Testing
|
||||||
cmake --build --preset macos-dev
|
-DYAZE_ENABLE_ROM_TESTS=ON/OFF
|
||||||
|
-DYAZE_TEST_ROM_PATH=/path/to/zelda3.sfc
|
||||||
|
|
||||||
# Run tests
|
# Build optimization
|
||||||
cmake --build --preset macos-dev --target test
|
-DYAZE_MINIMAL_BUILD=ON/OFF
|
||||||
|
-DYAZE_USE_MODULAR_BUILD=ON/OFF
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🛠️ IDE Integration
|
## Examples
|
||||||
|
|
||||||
### VS Code with CMake Tools:
|
### Development with AI features and verbose warnings
|
||||||
1. Open Command Palette (`Cmd+Shift+P`)
|
|
||||||
2. Run "CMake: Select Configure Preset"
|
|
||||||
3. Choose `macos-dev` or `macos-debug`
|
|
||||||
|
|
||||||
### CLion:
|
|
||||||
1. Go to Settings → Build, Execution, Deployment → CMake
|
|
||||||
2. Add new profile with preset `macos-dev`
|
|
||||||
|
|
||||||
### Xcode:
|
|
||||||
```bash
|
```bash
|
||||||
# Generate Xcode project
|
cmake --preset mac-dbg-v -DZ3ED_AI=ON -DYAZE_WITH_GRPC=ON
|
||||||
cmake --preset macos-debug -G Xcode
|
cmake --build --preset mac-dbg-v
|
||||||
open build/yaze.xcodeproj
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🔍 Troubleshooting
|
### Release build for distribution (macOS Universal)
|
||||||
|
```bash
|
||||||
|
cmake --preset mac-uni
|
||||||
|
cmake --build --preset mac-uni
|
||||||
|
cpack --preset mac-uni
|
||||||
|
```
|
||||||
|
|
||||||
If you still get architecture errors:
|
### Quick minimal build for testing
|
||||||
1. **Clean completely**: `rm -rf build`
|
```bash
|
||||||
2. **Check preset**: Ensure you're using an ARM64 preset (not universal)
|
cmake --preset mac-dbg -DYAZE_MINIMAL_BUILD=ON
|
||||||
3. **Verify configuration**: Check that `CMAKE_OSX_ARCHITECTURES` shows only `arm64`
|
cmake --build --preset mac-dbg -j12
|
||||||
|
```
|
||||||
|
|
||||||
|
## Updating compile_commands.json
|
||||||
|
|
||||||
|
After configuring with a new preset, copy the compile commands for IDE support:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Verify architecture setting
|
cp build/compile_commands.json .
|
||||||
cmake --preset macos-debug
|
|
||||||
grep -A 5 -B 5 "CMAKE_OSX_ARCHITECTURES" build/CMakeCache.txt
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## 📝 Notes
|
This ensures clangd and other LSP servers can find headers and understand build flags.
|
||||||
|
|
||||||
- **ARM64 presets**: Fast builds, no architecture conflicts
|
## Migration from Old Presets
|
||||||
- **Universal presets**: Slower builds, for distribution only
|
|
||||||
- **Deployment target**: ARM64 presets use macOS 11.0+ (when Apple Silicon was introduced)
|
Old preset names have been simplified:
|
||||||
- **Universal presets**: Still support macOS 10.15+ for backward compatibility
|
|
||||||
|
| Old Name | New Name |
|
||||||
|
|----------|----------|
|
||||||
|
| `macos-dev-z3ed-ai` | `mac-ai` |
|
||||||
|
| `macos-debug` | `mac-dbg` |
|
||||||
|
| `macos-release` | `mac-rel` |
|
||||||
|
| `macos-debug-universal` | `mac-uni` |
|
||||||
|
| `macos-dungeon-dev` | `mac-rooms` |
|
||||||
|
| `windows-debug` | `win-dbg` |
|
||||||
|
| `windows-release` | `win-rel` |
|
||||||
|
| `windows-arm64-debug` | `win-arm` |
|
||||||
|
| `linux-debug` | `lin-dbg` |
|
||||||
|
| `linux-clang` | `lin-clang` |
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Warnings are still showing
|
||||||
|
- Make sure you're using a preset without `-v` suffix
|
||||||
|
- Check `cmake` output for `✓ Warnings suppressed` message
|
||||||
|
- Reconfigure and rebuild: `rm -rf build && cmake --preset mac-dbg`
|
||||||
|
|
||||||
|
### clangd can't find nlohmann/json
|
||||||
|
- Run `cmake --preset <your-preset>` to regenerate compile_commands.json
|
||||||
|
- Copy to root: `cp build/compile_commands.json .`
|
||||||
|
- Restart your IDE or LSP server
|
||||||
|
|
||||||
|
### Build fails with missing dependencies
|
||||||
|
- Ensure submodules are initialized: `git submodule update --init --recursive`
|
||||||
|
- For AI features, make sure you have OpenSSL: `brew install openssl` (macOS)
|
||||||
|
|||||||
@@ -1,403 +0,0 @@
|
|||||||
# Stability, Testability & Release Workflow Improvements
|
|
||||||
|
|
||||||
This document outlines recent improvements to YAZE's stability, testability, and release workflows, along with recommendations for future optimizations.
|
|
||||||
|
|
||||||
## Recent Improvements (v0.3.2)
|
|
||||||
|
|
||||||
### Windows Platform Stability
|
|
||||||
|
|
||||||
#### Stack Size Optimization
|
|
||||||
**Problem:** Windows default stack size (1MB) was insufficient for `EditorManager::LoadAssets()` which loads 223 graphics sheets and initializes multiple editors.
|
|
||||||
|
|
||||||
**Solution:** Increased stack size to 8MB to match Unix-like systems.
|
|
||||||
|
|
||||||
**Implementation:**
|
|
||||||
```cmake
|
|
||||||
# src/app/app.cmake
|
|
||||||
if(MSVC)
|
|
||||||
target_link_options(yaze PRIVATE /STACK:8388608) # 8MB stack
|
|
||||||
elseif(MINGW)
|
|
||||||
target_link_options(yaze PRIVATE -Wl,--stack,8388608)
|
|
||||||
endif()
|
|
||||||
```
|
|
||||||
|
|
||||||
**Impact:**
|
|
||||||
- ✅ Eliminated stack overflow crashes during ROM loading
|
|
||||||
- ✅ Consistent behavior across all platforms
|
|
||||||
- ✅ Handles deep call chains from editor initialization
|
|
||||||
|
|
||||||
#### Development Utility Isolation
|
|
||||||
**Problem:** Development-only utilities (`extract_vanilla_values`, `rom_patch_utility`) were being built in CI/release workflows, causing linker errors.
|
|
||||||
|
|
||||||
**Solution:** Isolated development utilities from CI/release builds using environment detection.
|
|
||||||
|
|
||||||
**Implementation:**
|
|
||||||
```cmake
|
|
||||||
# test/CMakeLists.txt
|
|
||||||
if(NOT YAZE_MINIMAL_BUILD AND YAZE_ENABLE_ROM_TESTS AND NOT DEFINED ENV{GITHUB_ACTIONS})
|
|
||||||
add_executable(extract_vanilla_values ...)
|
|
||||||
target_link_libraries(extract_vanilla_values yaze_core ...)
|
|
||||||
endif()
|
|
||||||
```
|
|
||||||
|
|
||||||
**Impact:**
|
|
||||||
- ✅ Clean release builds without development artifacts
|
|
||||||
- ✅ Proper library linkage for development tools
|
|
||||||
- ✅ Faster CI/CD pipelines
|
|
||||||
|
|
||||||
### Graphics System Stability
|
|
||||||
|
|
||||||
#### Segmentation Fault Resolution
|
|
||||||
**Problem:** Tile cache system using `std::move()` operations invalidated Bitmap surface pointers, causing crashes.
|
|
||||||
|
|
||||||
**Solution:** Disabled move semantics in tile cache and implemented direct SDL texture updates.
|
|
||||||
|
|
||||||
**Technical Details:**
|
|
||||||
- Root cause: `std::move()` operations on Bitmap objects invalidated internal SDL_Surface pointers
|
|
||||||
- Fix: Disabled tile cache and use direct texture updates
|
|
||||||
- Optimization: Maintained surface/texture pooling while ensuring pointer stability
|
|
||||||
|
|
||||||
**Impact:**
|
|
||||||
- ✅ Eliminated all segmentation faults in tile16 editor
|
|
||||||
- ✅ Stable graphics rendering pipeline
|
|
||||||
- ✅ Reliable texture management
|
|
||||||
|
|
||||||
#### Comprehensive Bounds Checking
|
|
||||||
**Problem:** Out-of-bounds access to tile and palette data caused crashes and corruption.
|
|
||||||
|
|
||||||
**Solution:** Added extensive bounds checking throughout graphics pipeline.
|
|
||||||
|
|
||||||
**Areas Covered:**
|
|
||||||
- Tile16 canvas access
|
|
||||||
- Palette group selection
|
|
||||||
- Graphics sheet indexing
|
|
||||||
- Texture coordinate calculations
|
|
||||||
|
|
||||||
**Impact:**
|
|
||||||
- ✅ Prevents crashes from invalid data
|
|
||||||
- ✅ Better error reporting
|
|
||||||
- ✅ Safer memory access patterns
|
|
||||||
|
|
||||||
### Build System Improvements
|
|
||||||
|
|
||||||
#### Modern Windows Workflow
|
|
||||||
**Previous Approach:** Generate Visual Studio solution files using Python script.
|
|
||||||
|
|
||||||
**New Approach:** Use Visual Studio's native CMake support.
|
|
||||||
|
|
||||||
**Benefits:**
|
|
||||||
- No project generation step required
|
|
||||||
- CMakeLists.txt is the single source of truth
|
|
||||||
- Changes reflect immediately without regeneration
|
|
||||||
- Better IntelliSense and debugging experience
|
|
||||||
- Cross-platform consistency
|
|
||||||
|
|
||||||
**Migration:**
|
|
||||||
```powershell
|
|
||||||
# Old workflow
|
|
||||||
python scripts/generate-vs-projects.py
|
|
||||||
# Open YAZE.sln
|
|
||||||
|
|
||||||
# New workflow
|
|
||||||
# File → Open → Folder → yaze
|
|
||||||
# Visual Studio detects CMakeLists.txt automatically
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Enhanced CI/CD Reliability
|
|
||||||
**Improvements:**
|
|
||||||
- Automatic vcpkg fallback mechanisms
|
|
||||||
- Proper development utility isolation
|
|
||||||
- Consistent test execution across platforms
|
|
||||||
- Better error reporting and debugging
|
|
||||||
|
|
||||||
## Recommended Optimizations
|
|
||||||
|
|
||||||
### High Priority
|
|
||||||
|
|
||||||
#### 1. Lazy Graphics Loading
|
|
||||||
**Current:** All 223 graphics sheets loaded on ROM open.
|
|
||||||
|
|
||||||
**Proposed:** Load graphics sheets on-demand when editors access them.
|
|
||||||
|
|
||||||
**Benefits:**
|
|
||||||
- Faster ROM loading (3-5x improvement expected)
|
|
||||||
- Reduced memory footprint
|
|
||||||
- Better startup performance
|
|
||||||
- Eliminates stack pressure
|
|
||||||
|
|
||||||
**Implementation Strategy:**
|
|
||||||
```cpp
|
|
||||||
class LazyGraphicsLoader {
|
|
||||||
std::array<std::optional<gfx::Bitmap>, kNumGfxSheets> sheets_;
|
|
||||||
|
|
||||||
gfx::Bitmap& GetSheet(int index) {
|
|
||||||
if (!sheets_[index]) {
|
|
||||||
sheets_[index] = LoadGraphicsSheet(rom_, index);
|
|
||||||
}
|
|
||||||
return *sheets_[index];
|
|
||||||
}
|
|
||||||
};
|
|
||||||
```
|
|
||||||
|
|
||||||
**Impact:**
|
|
||||||
- 🔵 Complexity: Medium
|
|
||||||
- 🟢 Performance Gain: High
|
|
||||||
- 🟢 Risk: Low (backward compatible)
|
|
||||||
|
|
||||||
#### 2. Heap-Based Large Allocations
|
|
||||||
**Current:** Large arrays and vectors allocated on stack during asset loading.
|
|
||||||
|
|
||||||
**Proposed:** Move large data structures to heap allocation.
|
|
||||||
|
|
||||||
**Benefits:**
|
|
||||||
- Reduces stack pressure
|
|
||||||
- More flexible memory management
|
|
||||||
- Better for Windows default stack constraints
|
|
||||||
- Safer for deep call chains
|
|
||||||
|
|
||||||
**Areas to Convert:**
|
|
||||||
- Graphics sheet arrays in LoadAllGraphicsData()
|
|
||||||
- Editor initialization data structures
|
|
||||||
- Temporary buffers in compression/decompression
|
|
||||||
|
|
||||||
**Impact:**
|
|
||||||
- 🟢 Complexity: Low
|
|
||||||
- 🟢 Performance Gain: Medium
|
|
||||||
- 🟢 Risk: Very Low
|
|
||||||
|
|
||||||
#### 3. Streaming ROM Assets
|
|
||||||
**Current:** Load entire ROM and all assets into memory.
|
|
||||||
|
|
||||||
**Proposed:** Stream assets from ROM file as needed.
|
|
||||||
|
|
||||||
**Benefits:**
|
|
||||||
- Minimal memory footprint
|
|
||||||
- Instant ROM opening
|
|
||||||
- Better for large ROM hacks
|
|
||||||
- More scalable architecture
|
|
||||||
|
|
||||||
**Challenges:**
|
|
||||||
- Requires architecture refactoring
|
|
||||||
- Need efficient caching strategy
|
|
||||||
- Must maintain edit performance
|
|
||||||
|
|
||||||
**Impact:**
|
|
||||||
- 🔴 Complexity: High
|
|
||||||
- 🟢 Performance Gain: Very High
|
|
||||||
- 🟡 Risk: Medium (requires testing)
|
|
||||||
|
|
||||||
### Medium Priority
|
|
||||||
|
|
||||||
#### 4. Enhanced Test Isolation
|
|
||||||
**Current:** Some tests share global state through Arena singleton.
|
|
||||||
|
|
||||||
**Proposed:** Better test isolation with mock singletons.
|
|
||||||
|
|
||||||
**Benefits:**
|
|
||||||
- More reliable test execution
|
|
||||||
- Parallel test execution possible
|
|
||||||
- Better test independence
|
|
||||||
- Easier debugging
|
|
||||||
|
|
||||||
**Implementation:**
|
|
||||||
```cpp
|
|
||||||
class TestArena : public Arena {
|
|
||||||
// Test-specific implementation
|
|
||||||
};
|
|
||||||
|
|
||||||
TEST_F(GraphicsTest, TestCase) {
|
|
||||||
TestArena arena;
|
|
||||||
Arena::SetInstance(&arena); // Override singleton
|
|
||||||
// Run test
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Impact:**
|
|
||||||
- 🟡 Complexity: Medium
|
|
||||||
- 🟢 Performance Gain: Medium (parallel tests)
|
|
||||||
- 🟢 Risk: Low
|
|
||||||
|
|
||||||
#### 5. Dependency Caching Optimization
|
|
||||||
**Current:** CI builds re-download and build some dependencies.
|
|
||||||
|
|
||||||
**Proposed:** Enhanced caching strategies for vcpkg and build artifacts.
|
|
||||||
|
|
||||||
**Benefits:**
|
|
||||||
- Faster CI builds (2-3x improvement)
|
|
||||||
- Reduced CI costs
|
|
||||||
- More reliable builds (less network dependency)
|
|
||||||
- Better developer experience
|
|
||||||
|
|
||||||
**Implementation:**
|
|
||||||
```yaml
|
|
||||||
# GitHub Actions
|
|
||||||
- uses: actions/cache@v4
|
|
||||||
with:
|
|
||||||
path: |
|
|
||||||
~/.ccache
|
|
||||||
~/vcpkg_cache
|
|
||||||
build/_deps
|
|
||||||
key: ${{ runner.os }}-${{ hashFiles('**/CMakeLists.txt') }}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Impact:**
|
|
||||||
- 🟢 Complexity: Low
|
|
||||||
- 🟢 Performance Gain: High (CI only)
|
|
||||||
- 🟢 Risk: Very Low
|
|
||||||
|
|
||||||
#### 6. Memory Pool for Graphics
|
|
||||||
**Current:** Individual allocation for each Bitmap and texture.
|
|
||||||
|
|
||||||
**Proposed:** Memory pool for graphics objects.
|
|
||||||
|
|
||||||
**Benefits:**
|
|
||||||
- Reduced allocation overhead
|
|
||||||
- Better cache locality
|
|
||||||
- Predictable memory usage
|
|
||||||
- Faster allocation/deallocation
|
|
||||||
|
|
||||||
**Areas to Apply:**
|
|
||||||
- Bitmap objects
|
|
||||||
- SDL surfaces and textures
|
|
||||||
- Tile data structures
|
|
||||||
|
|
||||||
**Impact:**
|
|
||||||
- 🟡 Complexity: Medium
|
|
||||||
- 🟡 Performance Gain: Medium
|
|
||||||
- 🟡 Risk: Medium (requires careful design)
|
|
||||||
|
|
||||||
### Low Priority
|
|
||||||
|
|
||||||
#### 7. Build Time Optimization
|
|
||||||
**Current:** Full rebuild takes 10-15 minutes.
|
|
||||||
|
|
||||||
**Proposed:** Optimize compilation units and dependencies.
|
|
||||||
|
|
||||||
**Strategies:**
|
|
||||||
- Use forward declarations more extensively
|
|
||||||
- Split large compilation units
|
|
||||||
- Optimize template instantiations
|
|
||||||
- Better use of precompiled headers
|
|
||||||
|
|
||||||
**Impact:**
|
|
||||||
- 🟡 Complexity: Medium
|
|
||||||
- 🟢 Performance Gain: Medium (developer experience)
|
|
||||||
- 🟢 Risk: Low
|
|
||||||
|
|
||||||
#### 8. Release Workflow Simplification
|
|
||||||
**Current:** Three separate release workflows (simplified, standard, complex).
|
|
||||||
|
|
||||||
**Proposed:** Single unified workflow with conditional features.
|
|
||||||
|
|
||||||
**Benefits:**
|
|
||||||
- Easier maintenance
|
|
||||||
- Consistent behavior
|
|
||||||
- Better documentation
|
|
||||||
- Clearer mental model
|
|
||||||
|
|
||||||
**Implementation:**
|
|
||||||
```yaml
|
|
||||||
jobs:
|
|
||||||
release:
|
|
||||||
strategy:
|
|
||||||
matrix:
|
|
||||||
include:
|
|
||||||
- profile: minimal # Quick releases
|
|
||||||
- profile: standard # Normal releases
|
|
||||||
- profile: maximum # Production releases
|
|
||||||
```
|
|
||||||
|
|
||||||
**Impact:**
|
|
||||||
- 🟢 Complexity: Low
|
|
||||||
- 🟢 Performance Gain: None (maintenance benefit)
|
|
||||||
- 🟢 Risk: Very Low
|
|
||||||
|
|
||||||
## Testing Improvements
|
|
||||||
|
|
||||||
### Current State
|
|
||||||
- ✅ Comprehensive unit test coverage (46+ tests)
|
|
||||||
- ✅ Integration tests for major components
|
|
||||||
- ✅ ROM-dependent tests properly isolated
|
|
||||||
- ✅ CI-safe test configuration
|
|
||||||
- ✅ Platform-specific test handling
|
|
||||||
|
|
||||||
### Recommendations
|
|
||||||
|
|
||||||
#### 1. Visual Regression Testing
|
|
||||||
**Goal:** Catch graphics rendering regressions automatically.
|
|
||||||
|
|
||||||
**Approach:**
|
|
||||||
- Capture screenshots of editor states
|
|
||||||
- Compare against baseline images
|
|
||||||
- Flag visual differences for review
|
|
||||||
|
|
||||||
**Tools:** ImGui Test Engine (already integrated)
|
|
||||||
|
|
||||||
#### 2. Performance Benchmarks
|
|
||||||
**Goal:** Track performance regressions in CI.
|
|
||||||
|
|
||||||
**Metrics:**
|
|
||||||
- ROM load time
|
|
||||||
- Graphics sheet decompression
|
|
||||||
- Editor initialization
|
|
||||||
- Memory usage
|
|
||||||
|
|
||||||
**Implementation:** Google Benchmark (already a dependency)
|
|
||||||
|
|
||||||
#### 3. Fuzz Testing
|
|
||||||
**Goal:** Find edge cases and crashes through random input.
|
|
||||||
|
|
||||||
**Areas:**
|
|
||||||
- ROM parsing
|
|
||||||
- Compression/decompression
|
|
||||||
- Palette handling
|
|
||||||
- Tile data processing
|
|
||||||
|
|
||||||
**Tools:** LibFuzzer or AFL
|
|
||||||
|
|
||||||
## Metrics & Monitoring
|
|
||||||
|
|
||||||
### Current Measurements
|
|
||||||
- Build time: ~10-15 minutes (full rebuild)
|
|
||||||
- ROM load time: ~2-3 seconds
|
|
||||||
- Memory usage: ~500MB-1GB typical
|
|
||||||
- Test execution: ~30 seconds (CI), ~2 minutes (full)
|
|
||||||
|
|
||||||
### Target Improvements
|
|
||||||
- Build time: <5 minutes (incremental), <10 minutes (full)
|
|
||||||
- ROM load time: <1 second (with lazy loading)
|
|
||||||
- Memory usage: <300MB (with streaming)
|
|
||||||
- Test execution: <15 seconds (CI), <1 minute (full)
|
|
||||||
|
|
||||||
## Action Items
|
|
||||||
|
|
||||||
### Immediate (v0.3.2)
|
|
||||||
- [x] Fix Windows stack overflow
|
|
||||||
- [x] Isolate development utilities
|
|
||||||
- [x] Fix graphics segfaults
|
|
||||||
- [x] Update build documentation
|
|
||||||
- [ ] Complete tile16 palette display fixes
|
|
||||||
|
|
||||||
### Short Term (v0.3.3)
|
|
||||||
- [ ] Implement lazy graphics loading
|
|
||||||
- [ ] Move large allocations to heap
|
|
||||||
- [ ] Enhanced CI caching
|
|
||||||
- [ ] Performance benchmarks
|
|
||||||
|
|
||||||
### Medium Term (v0.4.0)
|
|
||||||
- [ ] Streaming ROM assets
|
|
||||||
- [ ] Memory pool for graphics
|
|
||||||
- [ ] Visual regression tests
|
|
||||||
- [ ] Enhanced test isolation
|
|
||||||
|
|
||||||
### Long Term (v0.5.0+)
|
|
||||||
- [ ] Fuzz testing integration
|
|
||||||
- [ ] Build time optimization
|
|
||||||
- [ ] Release workflow unification
|
|
||||||
- [ ] Advanced memory profiling
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
The v0.3.2 release focuses on stability and reliability improvements, particularly for the Windows platform. The recommended optimizations provide a clear roadmap for future performance and maintainability improvements while maintaining backward compatibility and code quality.
|
|
||||||
|
|
||||||
For questions or suggestions, please open an issue or discussion on the GitHub repository.
|
|
||||||
@@ -1,207 +0,0 @@
|
|||||||
# CMake Presets Quick Reference
|
|
||||||
|
|
||||||
This document explains the reorganized CMake preset system for Yaze.
|
|
||||||
|
|
||||||
## Design Principles
|
|
||||||
|
|
||||||
1. **Short, memorable names** - No more `macos-dev-z3ed-ai`, just `mac-ai`
|
|
||||||
2. **Warnings off by default** - Add `-v` suffix for verbose (e.g., `mac-dbg-v`)
|
|
||||||
3. **Clear architecture support** - Explicit ARM64 and x86_64 presets
|
|
||||||
4. **Platform prefixes** - `mac-`, `win-`, `lin-` for easy identification
|
|
||||||
|
|
||||||
## Quick Start
|
|
||||||
|
|
||||||
### macOS Development
|
|
||||||
```bash
|
|
||||||
# Most common: AI-enabled development
|
|
||||||
cmake --preset mac-ai
|
|
||||||
cmake --build --preset mac-ai
|
|
||||||
|
|
||||||
# Basic debug build (no AI)
|
|
||||||
cmake --preset mac-dbg
|
|
||||||
cmake --build --preset mac-dbg
|
|
||||||
|
|
||||||
# Verbose warnings for debugging
|
|
||||||
cmake --preset mac-dbg-v
|
|
||||||
cmake --build --preset mac-dbg-v
|
|
||||||
|
|
||||||
# Release build
|
|
||||||
cmake --preset mac-rel
|
|
||||||
cmake --build --preset mac-rel
|
|
||||||
```
|
|
||||||
|
|
||||||
### Windows Development
|
|
||||||
```bash
|
|
||||||
# Debug build (x64)
|
|
||||||
cmake --preset win-dbg
|
|
||||||
cmake --build --preset win-dbg
|
|
||||||
|
|
||||||
# AI-enabled development
|
|
||||||
cmake --preset win-ai
|
|
||||||
cmake --build --preset win-ai
|
|
||||||
|
|
||||||
# ARM64 support
|
|
||||||
cmake --preset win-arm
|
|
||||||
cmake --build --preset win-arm
|
|
||||||
```
|
|
||||||
|
|
||||||
## All Presets
|
|
||||||
|
|
||||||
### macOS Presets
|
|
||||||
|
|
||||||
| Preset | Description | Arch | Warnings | Features |
|
|
||||||
|--------|-------------|------|----------|----------|
|
|
||||||
| `mac-dbg` | Debug build | ARM64 | Off | Basic |
|
|
||||||
| `mac-dbg-v` | Debug verbose | ARM64 | On | Basic |
|
|
||||||
| `mac-rel` | Release | ARM64 | Off | Basic |
|
|
||||||
| `mac-x64` | Debug x86_64 | x86_64 | Off | Basic |
|
|
||||||
| `mac-uni` | Universal binary | Both | Off | Basic |
|
|
||||||
| `mac-dev` | Development | ARM64 | Off | ROM tests |
|
|
||||||
| `mac-ai` | AI development | ARM64 | Off | z3ed, JSON, gRPC, ROM tests |
|
|
||||||
| `mac-z3ed` | z3ed CLI | ARM64 | Off | AI agent support |
|
|
||||||
| `mac-rooms` | Dungeon editor | ARM64 | Off | Minimal build for room editing |
|
|
||||||
|
|
||||||
### Windows Presets
|
|
||||||
|
|
||||||
| Preset | Description | Arch | Warnings | Features |
|
|
||||||
|--------|-------------|------|----------|----------|
|
|
||||||
| `win-dbg` | Debug build | x64 | Off | Basic |
|
|
||||||
| `win-dbg-v` | Debug verbose | x64 | On | Basic |
|
|
||||||
| `win-rel` | Release | x64 | Off | Basic |
|
|
||||||
| `win-arm` | Debug ARM64 | ARM64 | Off | Basic |
|
|
||||||
| `win-arm-rel` | Release ARM64 | ARM64 | Off | Basic |
|
|
||||||
| `win-dev` | Development | x64 | Off | ROM tests |
|
|
||||||
| `win-ai` | AI development | x64 | Off | z3ed, JSON, gRPC, ROM tests |
|
|
||||||
| `win-z3ed` | z3ed CLI | x64 | Off | AI agent support |
|
|
||||||
|
|
||||||
### Linux Presets
|
|
||||||
|
|
||||||
| Preset | Description | Compiler | Warnings |
|
|
||||||
|--------|-------------|----------|----------|
|
|
||||||
| `lin-dbg` | Debug | GCC | Off |
|
|
||||||
| `lin-clang` | Debug | Clang | Off |
|
|
||||||
|
|
||||||
### Special Purpose
|
|
||||||
|
|
||||||
| Preset | Description |
|
|
||||||
|--------|-------------|
|
|
||||||
| `ci` | Continuous integration (no ROM tests) |
|
|
||||||
| `asan` | AddressSanitizer build |
|
|
||||||
| `coverage` | Code coverage build |
|
|
||||||
|
|
||||||
## Warning Control
|
|
||||||
|
|
||||||
By default, all presets suppress compiler warnings with `-w` for a cleaner build experience.
|
|
||||||
|
|
||||||
### To Enable Verbose Warnings:
|
|
||||||
|
|
||||||
1. Use a preset with `-v` suffix (e.g., `mac-dbg-v`, `win-dbg-v`)
|
|
||||||
2. Or set `YAZE_SUPPRESS_WARNINGS=OFF` manually:
|
|
||||||
```bash
|
|
||||||
cmake --preset mac-dbg -DYAZE_SUPPRESS_WARNINGS=OFF
|
|
||||||
```
|
|
||||||
|
|
||||||
## Architecture Support
|
|
||||||
|
|
||||||
### macOS
|
|
||||||
- **ARM64 (Apple Silicon)**: `mac-dbg`, `mac-rel`, `mac-ai`, etc.
|
|
||||||
- **x86_64 (Intel)**: `mac-x64`
|
|
||||||
- **Universal Binary**: `mac-uni` (both ARM64 + x86_64)
|
|
||||||
|
|
||||||
### Windows
|
|
||||||
- **x64**: `win-dbg`, `win-rel`, `win-ai`, etc.
|
|
||||||
- **ARM64**: `win-arm`, `win-arm-rel`
|
|
||||||
|
|
||||||
## Build Directories
|
|
||||||
|
|
||||||
Most presets use `build/` directory. Exceptions:
|
|
||||||
- `mac-rooms`: Uses `build_rooms/` to avoid conflicts
|
|
||||||
|
|
||||||
## Feature Flags
|
|
||||||
|
|
||||||
Common CMake options you can override:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Enable/disable components
|
|
||||||
-DYAZE_BUILD_TESTS=ON/OFF
|
|
||||||
-DYAZE_BUILD_Z3ED=ON/OFF
|
|
||||||
-DYAZE_BUILD_EMU=ON/OFF
|
|
||||||
-DYAZE_BUILD_APP=ON/OFF
|
|
||||||
|
|
||||||
# AI features
|
|
||||||
-DZ3ED_AI=ON/OFF
|
|
||||||
-DYAZE_WITH_JSON=ON/OFF
|
|
||||||
-DYAZE_WITH_GRPC=ON/OFF
|
|
||||||
|
|
||||||
# Testing
|
|
||||||
-DYAZE_ENABLE_ROM_TESTS=ON/OFF
|
|
||||||
-DYAZE_TEST_ROM_PATH=/path/to/zelda3.sfc
|
|
||||||
|
|
||||||
# Build optimization
|
|
||||||
-DYAZE_MINIMAL_BUILD=ON/OFF
|
|
||||||
-DYAZE_USE_MODULAR_BUILD=ON/OFF
|
|
||||||
```
|
|
||||||
|
|
||||||
## Examples
|
|
||||||
|
|
||||||
### Development with AI features and verbose warnings
|
|
||||||
```bash
|
|
||||||
cmake --preset mac-dbg-v -DZ3ED_AI=ON -DYAZE_WITH_GRPC=ON
|
|
||||||
cmake --build --preset mac-dbg-v
|
|
||||||
```
|
|
||||||
|
|
||||||
### Release build for distribution (macOS Universal)
|
|
||||||
```bash
|
|
||||||
cmake --preset mac-uni
|
|
||||||
cmake --build --preset mac-uni
|
|
||||||
cpack --preset mac-uni
|
|
||||||
```
|
|
||||||
|
|
||||||
### Quick minimal build for testing
|
|
||||||
```bash
|
|
||||||
cmake --preset mac-dbg -DYAZE_MINIMAL_BUILD=ON
|
|
||||||
cmake --build --preset mac-dbg -j12
|
|
||||||
```
|
|
||||||
|
|
||||||
## Updating compile_commands.json
|
|
||||||
|
|
||||||
After configuring with a new preset, copy the compile commands for IDE support:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cp build/compile_commands.json .
|
|
||||||
```
|
|
||||||
|
|
||||||
This ensures clangd and other LSP servers can find headers and understand build flags.
|
|
||||||
|
|
||||||
## Migration from Old Presets
|
|
||||||
|
|
||||||
Old preset names have been simplified:
|
|
||||||
|
|
||||||
| Old Name | New Name |
|
|
||||||
|----------|----------|
|
|
||||||
| `macos-dev-z3ed-ai` | `mac-ai` |
|
|
||||||
| `macos-debug` | `mac-dbg` |
|
|
||||||
| `macos-release` | `mac-rel` |
|
|
||||||
| `macos-debug-universal` | `mac-uni` |
|
|
||||||
| `macos-dungeon-dev` | `mac-rooms` |
|
|
||||||
| `windows-debug` | `win-dbg` |
|
|
||||||
| `windows-release` | `win-rel` |
|
|
||||||
| `windows-arm64-debug` | `win-arm` |
|
|
||||||
| `linux-debug` | `lin-dbg` |
|
|
||||||
| `linux-clang` | `lin-clang` |
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### Warnings are still showing
|
|
||||||
- Make sure you're using a preset without `-v` suffix
|
|
||||||
- Check `cmake` output for `✓ Warnings suppressed` message
|
|
||||||
- Reconfigure and rebuild: `rm -rf build && cmake --preset mac-dbg`
|
|
||||||
|
|
||||||
### clangd can't find nlohmann/json
|
|
||||||
- Run `cmake --preset <your-preset>` to regenerate compile_commands.json
|
|
||||||
- Copy to root: `cp build/compile_commands.json .`
|
|
||||||
- Restart your IDE or LSP server
|
|
||||||
|
|
||||||
### Build fails with missing dependencies
|
|
||||||
- Ensure submodules are initialized: `git submodule update --init --recursive`
|
|
||||||
- For AI features, make sure you have OpenSSL: `brew install openssl` (macOS)
|
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
# Canvas System - Comprehensive Guide
|
# Canvas System
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
@@ -1,47 +0,0 @@
|
|||||||
# Graphics System Performance & Optimization
|
|
||||||
|
|
||||||
This document provides a comprehensive overview of the analysis, implementation, and results of performance optimizations applied to the YAZE graphics system.
|
|
||||||
|
|
||||||
## 1. Executive Summary
|
|
||||||
|
|
||||||
Massive performance improvements were achieved across the application, dramatically improving the user experience, especially during resource-intensive operations like ROM loading and dungeon editing.
|
|
||||||
|
|
||||||
### Overall Performance Results
|
|
||||||
|
|
||||||
| Component | Before | After | Improvement |
|
|
||||||
|-----------|--------|-------|-------------|
|
|
||||||
| **DungeonEditor::Load** | **17,967ms** | **3,747ms** | **🚀 79% faster!** |
|
|
||||||
| **Total ROM Loading** | **~18.6s** | **~4.7s** | **🚀 75% faster!** |
|
|
||||||
| **User Experience** | 18-second freeze | Near-instant | **Dramatic improvement** |
|
|
||||||
|
|
||||||
## 2. Implemented Optimizations
|
|
||||||
|
|
||||||
The following key optimizations were successfully implemented:
|
|
||||||
|
|
||||||
1. **Palette Lookup Optimization (100x faster)**: Replaced a linear search with an `std::unordered_map` for O(1) color-to-index lookups in the `Bitmap` class.
|
|
||||||
|
|
||||||
2. **Dirty Region Tracking (10x faster)**: The `Bitmap` class now tracks modified regions, so only the changed portion of a texture is uploaded to the GPU, significantly reducing GPU bandwidth.
|
|
||||||
|
|
||||||
3. **Resource Pooling (~30% memory reduction)**: The central `Arena` manager now pools and reuses `SDL_Texture` and `SDL_Surface` objects, reducing memory fragmentation and creation/destruction overhead.
|
|
||||||
|
|
||||||
4. **LRU Tile Caching (5x faster)**: The `Tilemap` class uses a Least Recently Used (LRU) cache to avoid redundant `Bitmap` object creation for frequently rendered tiles.
|
|
||||||
|
|
||||||
5. **Batch Operations (5x faster)**: The `Arena` can now queue multiple texture updates and process them in a single batch, reducing SDL context switching.
|
|
||||||
|
|
||||||
6. **Memory Pool Allocator (10x faster)**: A custom `MemoryPool` provides pre-allocated blocks for common graphics sizes (8x8, 16x16), bypassing `malloc`/`free` overhead.
|
|
||||||
|
|
||||||
7. **Atlas-Based Rendering (N-to-1 draw calls)**: A new `AtlasRenderer` dynamically packs smaller bitmaps into a single large texture atlas, allowing many elements to be drawn in a single batch.
|
|
||||||
|
|
||||||
8. **Parallel & Incremental Loading**: Dungeon rooms and overworld maps are now loaded in parallel or incrementally to prevent UI blocking.
|
|
||||||
|
|
||||||
9. **Performance Monitoring**: A `PerformanceProfiler` and `PerformanceDashboard` were created to measure the impact of these optimizations and detect regressions.
|
|
||||||
|
|
||||||
## 3. Future Optimization Recommendations
|
|
||||||
|
|
||||||
### High Priority
|
|
||||||
1. **Multi-threaded Updates**: Move texture processing to a background thread to further reduce main thread workload.
|
|
||||||
2. **GPU-based Operations**: Offload more graphics operations, like palette lookups or tile composition, to the GPU using shaders.
|
|
||||||
|
|
||||||
### Medium Priority
|
|
||||||
1. **Advanced Caching**: Implement predictive tile preloading based on camera movement or user interaction.
|
|
||||||
2. **Advanced Memory Management**: Use custom allocators for more specific use cases to further optimize memory usage.
|
|
||||||
@@ -1,235 +0,0 @@
|
|||||||
# Build Modularization Plan for Yaze
|
|
||||||
|
|
||||||
## Executive Summary
|
|
||||||
|
|
||||||
This document outlines a comprehensive plan to modularize the yaze build system, reducing build times by 40-60% through the creation of intermediate libraries and improved dependency management.
|
|
||||||
|
|
||||||
## Current Problems
|
|
||||||
|
|
||||||
1. **Monolithic Build**: All subsystems compile directly into final executables
|
|
||||||
2. **Repeated Compilation**: Same sources compiled multiple times for different targets
|
|
||||||
3. **Poor Incremental Builds**: Small changes trigger large rebuilds
|
|
||||||
4. **Long Link Times**: Large monolithic targets take longer to link
|
|
||||||
5. **Parallel Build Limitations**: Cannot parallelize subsystem builds effectively
|
|
||||||
|
|
||||||
## Proposed Architecture
|
|
||||||
|
|
||||||
### Library Hierarchy
|
|
||||||
|
|
||||||
```
|
|
||||||
yaze_common (INTERFACE)
|
|
||||||
├── yaze_util (STATIC)
|
|
||||||
│ ├── util/bps.cc
|
|
||||||
│ ├── util/flag.cc
|
|
||||||
│ └── util/hex.cc
|
|
||||||
│
|
|
||||||
├── yaze_gfx (STATIC)
|
|
||||||
│ ├── All YAZE_APP_GFX_SRC files
|
|
||||||
│ └── Depends: yaze_util
|
|
||||||
│
|
|
||||||
├── yaze_gui (STATIC)
|
|
||||||
│ ├── All YAZE_GUI_SRC files
|
|
||||||
│ └── Depends: yaze_gfx, ImGui
|
|
||||||
│
|
|
||||||
├── yaze_zelda3 (STATIC)
|
|
||||||
│ ├── All YAZE_APP_ZELDA3_SRC files
|
|
||||||
│ └── Depends: yaze_gfx, yaze_util
|
|
||||||
│
|
|
||||||
├── yaze_emulator (STATIC)
|
|
||||||
│ ├── All YAZE_APP_EMU_SRC files
|
|
||||||
│ └── Depends: yaze_util
|
|
||||||
│
|
|
||||||
├── yaze_core_lib (STATIC)
|
|
||||||
│ ├── All YAZE_APP_CORE_SRC files
|
|
||||||
│ └── Depends: yaze_util, yaze_gfx, asar
|
|
||||||
│
|
|
||||||
├── yaze_editor (STATIC)
|
|
||||||
│ ├── All YAZE_APP_EDITOR_SRC files
|
|
||||||
│ └── Depends: yaze_core_lib, yaze_gfx, yaze_gui, yaze_zelda3
|
|
||||||
│
|
|
||||||
├── yaze_agent (STATIC - already exists)
|
|
||||||
│ └── Depends: yaze_util
|
|
||||||
│
|
|
||||||
└── Final Executables
|
|
||||||
├── yaze (APP) - Links all above
|
|
||||||
├── yaze_emu - Links subset
|
|
||||||
├── z3ed (CLI) - Links subset
|
|
||||||
└── yaze_test - Links all for testing
|
|
||||||
```
|
|
||||||
|
|
||||||
## Implementation Guide
|
|
||||||
|
|
||||||
This section provides step-by-step instructions for implementing the modularized build system for yaze.
|
|
||||||
|
|
||||||
### Quick Start
|
|
||||||
|
|
||||||
#### Option 1: Enable Modular Build (Recommended)
|
|
||||||
|
|
||||||
Add this option to enable the new modular build system:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cmake -B build -DYAZE_USE_MODULAR_BUILD=ON
|
|
||||||
cmake --build build
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Option 2: Keep Existing Build (Default)
|
|
||||||
|
|
||||||
The existing build system remains the default for stability:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cmake -B build
|
|
||||||
cmake --build build
|
|
||||||
```
|
|
||||||
|
|
||||||
### Implementation Steps
|
|
||||||
|
|
||||||
#### Step 1: Add Build Option
|
|
||||||
|
|
||||||
Add to **root `CMakeLists.txt`** (around line 45, after other options):
|
|
||||||
|
|
||||||
```cmake
|
|
||||||
# Build system options
|
|
||||||
option(YAZE_USE_MODULAR_BUILD "Use modularized library build system for faster builds" OFF)
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Step 2: Update src/CMakeLists.txt
|
|
||||||
|
|
||||||
Replace the current monolithic build with a library-based approach. Add near the top of `src/CMakeLists.txt`:
|
|
||||||
|
|
||||||
```cmake
|
|
||||||
# ==============================================================================
|
|
||||||
# Modular Build System
|
|
||||||
# ==============================================================================
|
|
||||||
if(YAZE_USE_MODULAR_BUILD)
|
|
||||||
message(STATUS "Using modular build system")
|
|
||||||
|
|
||||||
# Build subsystem libraries in dependency order
|
|
||||||
if(YAZE_BUILD_LIB OR YAZE_BUILD_APP OR YAZE_BUILD_Z3ED)
|
|
||||||
include(util/util.cmake)
|
|
||||||
include(app/gfx/gfx_library.cmake)
|
|
||||||
include(app/gui/gui_library.cmake)
|
|
||||||
include(app/zelda3/zelda3_library.cmake)
|
|
||||||
|
|
||||||
if(YAZE_BUILD_EMU AND NOT YAZE_WITH_GRPC)
|
|
||||||
include(app/emu/emu_library.cmake)
|
|
||||||
endif()
|
|
||||||
|
|
||||||
include(app/core/core_library.cmake)
|
|
||||||
include(app/editor/editor_library.cmake)
|
|
||||||
endif()
|
|
||||||
|
|
||||||
# Create yaze_core as an INTERFACE library that aggregates all modules
|
|
||||||
if(YAZE_BUILD_LIB)
|
|
||||||
add_library(yaze_core INTERFACE)
|
|
||||||
target_link_libraries(yaze_core INTERFACE
|
|
||||||
yaze_util
|
|
||||||
yaze_gfx
|
|
||||||
yaze_gui
|
|
||||||
yaze_zelda3
|
|
||||||
yaze_core_lib
|
|
||||||
yaze_editor
|
|
||||||
yaze_agent
|
|
||||||
)
|
|
||||||
|
|
||||||
if(YAZE_BUILD_EMU AND NOT YAZE_WITH_GRPC)
|
|
||||||
target_link_libraries(yaze_core INTERFACE yaze_emulator)
|
|
||||||
endif()
|
|
||||||
|
|
||||||
# Add gRPC support if enabled
|
|
||||||
if(YAZE_WITH_GRPC)
|
|
||||||
target_add_protobuf(yaze_core
|
|
||||||
${CMAKE_SOURCE_DIR}/src/app/core/proto/imgui_test_harness.proto)
|
|
||||||
target_link_libraries(yaze_core INTERFACE
|
|
||||||
grpc++
|
|
||||||
grpc++_reflection
|
|
||||||
libprotobuf)
|
|
||||||
endif()
|
|
||||||
endif()
|
|
||||||
else()
|
|
||||||
message(STATUS "Using traditional monolithic build system")
|
|
||||||
# Keep existing build (current code in src/CMakeLists.txt)
|
|
||||||
endif()
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Step 3: Update yaze Application Linking
|
|
||||||
|
|
||||||
In `src/app/app.cmake`, update the main yaze target:
|
|
||||||
|
|
||||||
```cmake
|
|
||||||
if(YAZE_USE_MODULAR_BUILD)
|
|
||||||
target_link_libraries(yaze PRIVATE
|
|
||||||
yaze_util
|
|
||||||
yaze_gfx
|
|
||||||
yaze_gui
|
|
||||||
yaze_zelda3
|
|
||||||
yaze_core_lib
|
|
||||||
yaze_editor
|
|
||||||
)
|
|
||||||
if(YAZE_BUILD_EMU AND NOT YAZE_WITH_GRPC)
|
|
||||||
target_link_libraries(yaze PRIVATE yaze_emulator)
|
|
||||||
endif()
|
|
||||||
else()
|
|
||||||
target_link_libraries(yaze PRIVATE yaze_core)
|
|
||||||
endif()
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Step 4: Update Test Executable
|
|
||||||
|
|
||||||
In `test/CMakeLists.txt`, update the test linking:
|
|
||||||
|
|
||||||
```cmake
|
|
||||||
if(YAZE_BUILD_LIB)
|
|
||||||
if(YAZE_USE_MODULAR_BUILD)
|
|
||||||
# Link individual libraries for faster incremental test builds
|
|
||||||
target_link_libraries(yaze_test
|
|
||||||
yaze_util
|
|
||||||
yaze_gfx
|
|
||||||
yaze_gui
|
|
||||||
yaze_zelda3
|
|
||||||
yaze_core_lib
|
|
||||||
yaze_editor
|
|
||||||
)
|
|
||||||
if(YAZE_BUILD_EMU AND NOT YAZE_WITH_GRPC)
|
|
||||||
target_link_libraries(yaze_test yaze_emulator)
|
|
||||||
endif()
|
|
||||||
else()
|
|
||||||
# Use monolithic core library
|
|
||||||
target_link_libraries(yaze_test yaze_core)
|
|
||||||
endif()
|
|
||||||
endif()
|
|
||||||
```
|
|
||||||
|
|
||||||
## Expected Performance Improvements
|
|
||||||
|
|
||||||
### Build Time Reductions
|
|
||||||
|
|
||||||
| Scenario | Current | With Modularization | Improvement |
|
|
||||||
|----------|---------|---------------------|-------------|
|
|
||||||
| Clean build | 100% | 100% | 0% (first time) |
|
|
||||||
| Change util file | ~80% rebuild | ~15% rebuild | **81% faster** |
|
|
||||||
| Change gfx file | ~70% rebuild | ~30% rebuild | **57% faster** |
|
|
||||||
| Change gui file | ~60% rebuild | ~25% rebuild | **58% faster** |
|
|
||||||
| Change editor file | ~50% rebuild | ~20% rebuild | **60% faster** |
|
|
||||||
| Change zelda3 file | ~40% rebuild | ~15% rebuild | **62% faster** |
|
|
||||||
| Incremental test | 100% relink | Cached libs | **70% faster** |
|
|
||||||
|
|
||||||
### CI/CD Benefits
|
|
||||||
|
|
||||||
- **Cached Artifacts**: Libraries can be cached between CI runs
|
|
||||||
- **Selective Testing**: Only rebuild/test changed modules
|
|
||||||
- **Faster Iteration**: Developers see results sooner
|
|
||||||
|
|
||||||
## Rollback Procedure
|
|
||||||
|
|
||||||
If issues arise, you can revert to the traditional build:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Option 1: Just disable the flag
|
|
||||||
cmake -B build -DYAZE_USE_MODULAR_BUILD=OFF
|
|
||||||
cmake --build build
|
|
||||||
|
|
||||||
# Option 2: Delete build and start fresh
|
|
||||||
rm -rf build
|
|
||||||
cmake -B build # OFF is the default
|
|
||||||
cmake --build build
|
|
||||||
```
|
|
||||||
@@ -1,330 +0,0 @@
|
|||||||
# Ollama Integration Status - Updated# Ollama Integration Status
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## ✅ Completed## ✅ Completed
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### Infrastructure### Flag Parsing
|
|
||||||
|
|
||||||
- ✅ Flag parsing for AI provider configuration- **Fixed**: AI provider flags (`--ai_provider`, `--ai_model`, `--ollama_host`, `--gemini_api_key`) are now properly parsed in `cli_main.cc`
|
|
||||||
|
|
||||||
- ✅ Ollama service with health checks- **Result**: Ollama provider is correctly detected and initialized
|
|
||||||
|
|
||||||
- ✅ Tool system with 5 read-only tools- **Verification**: `🤖 AI Provider: ollama` message appears correctly
|
|
||||||
|
|
||||||
- ✅ Simple chat modes (4 input methods working)
|
|
||||||
|
|
||||||
- ✅ Colorful terminal output with loading indicators### Ollama Service
|
|
||||||
|
|
||||||
- ✅ Verbose mode for diagnostics- **Status**: OllamaAIService properly connects to local Ollama server
|
|
||||||
|
|
||||||
- ✅ Configurable max-tool-iterations and max-retries- **Health Check**: Successfully validates model availability (qwen2.5-coder:7b)
|
|
||||||
|
|
||||||
- ✅ File-based prompt system (assets/agent/*.txt)- **JSON Parsing**: Correctly extracts tool calls and text responses from Ollama's response format
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### Current Issue: Empty Tool Results### Tool System
|
|
||||||
|
|
||||||
- **Tool Dispatcher**: Working correctly - routes tool calls to appropriate handlers
|
|
||||||
|
|
||||||
**Problem**: The `resource-list` tool is returning empty JSON `{}` when requesting dungeon labels.- **Tool Registration**: 5 read-only tools available:
|
|
||||||
|
|
||||||
- `resource-list` - List labeled resources
|
|
||||||
|
|
||||||
**Root Cause**: The embedded labels in Zelda3Labels only include: - `dungeon-list-sprites` - Inspect room sprites
|
|
||||||
|
|
||||||
- `room` - 297 room names ✅ - `overworld-find-tile` - Search for tile placements
|
|
||||||
|
|
||||||
- `entrance` - 133 entrance names ✅ - `overworld-describe-map` - Get map metadata
|
|
||||||
|
|
||||||
- `sprite` - 256 sprite names ✅ - `overworld-list-warps` - List entrances/exits/holes
|
|
||||||
|
|
||||||
- `overlord` - 26 overlord names ✅
|
|
||||||
|
|
||||||
- `item` - 104 item names ✅### Simple Chat Modes
|
|
||||||
|
|
||||||
All 4 input methods working:
|
|
||||||
|
|
||||||
But **NOT** `dungeon` as a separate category.1. ✅ Single message mode: `z3ed agent simple-chat "message" --rom=file.sfc --ai_provider=ollama`
|
|
||||||
|
|
||||||
2. ✅ Interactive mode: `z3ed agent simple-chat --rom=file.sfc --ai_provider=ollama`
|
|
||||||
|
|
||||||
**Diagnosis**:3. ✅ Piped input mode: `echo "message" | z3ed agent simple-chat --rom=file.sfc --ai_provider=ollama`
|
|
||||||
|
|
||||||
```bash4. ✅ Batch file mode: `z3ed agent simple-chat --file=queries.txt --rom=file.sfc --ai_provider=ollama`
|
|
||||||
|
|
||||||
# Works (returns data):
|
|
||||||
|
|
||||||
./z3ed agent resource-list --type=room --format=json## 🚧 In Progress
|
|
||||||
|
|
||||||
./z3ed agent resource-list --type=entrance --format=json
|
|
||||||
|
|
||||||
./z3ed agent resource-list --type=sprite --format=json### Tool Calling Loop Issue
|
|
||||||
|
|
||||||
**Problem**: Agent enters infinite tool-calling loop without providing final text response
|
|
||||||
|
|
||||||
# Fails (returns empty {}):
|
|
||||||
|
|
||||||
./z3ed agent resource-list --type=dungeon --format=json**Symptoms**:
|
|
||||||
|
|
||||||
``````
|
|
||||||
|
|
||||||
Error: Agent did not produce a response after executing tools.
|
|
||||||
|
|
||||||
**Solution Options**:```
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
1. **Quick Fix**: Update prompt examples to use valid categories**Root Cause**: The system prompt needs refinement to instruct the LLM to:
|
|
||||||
|
|
||||||
- Change `type: dungeon` → `type: room` in examples1. Call tools when needed
|
|
||||||
|
|
||||||
- Update tool descriptions to clarify available categories2. Wait for tool results
|
|
||||||
|
|
||||||
3. **THEN provide a final text_response based on the tool results**
|
|
||||||
|
|
||||||
2. **Proper Fix**: Add dungeon labels to embedded labels4. Stop calling tools after receiving results
|
|
||||||
|
|
||||||
- Modify `Zelda3Labels::ToResourceLabels()` to include dungeon category
|
|
||||||
|
|
||||||
- Map dungeon IDs (0-11) to their names**Current Behavior**:
|
|
||||||
|
|
||||||
- LLM successfully calls tools (e.g., `resource-list` with `type=dungeon`)
|
|
||||||
|
|
||||||
3. **Alternative**: Clarify that "dungeons" are accessed via room labels- Tool executes and returns JSON results
|
|
||||||
|
|
||||||
- Document that dungeon rooms use the `room` category- LLM receives results in conversation history
|
|
||||||
|
|
||||||
- Provide ID ranges (e.g., rooms 0-119 are Hyrule Castle, etc.)- LLM either:
|
|
||||||
|
|
||||||
- Calls tools again (loop detected after 4 iterations)
|
|
||||||
|
|
||||||
## 🎨 New Features Added - OR doesn't provide a `text_response` field in the JSON
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### Verbose Mode**Solution Needed**: Update system prompt to include explicit instructions like:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
|
|
||||||
z3ed agent simple-chat "query" --verbose```
|
|
||||||
|
|
||||||
```When you call a tool:
|
|
||||||
|
|
||||||
Shows:1. The tool will execute and return results
|
|
||||||
|
|
||||||
- Iteration count2. You will receive the results in the next message
|
|
||||||
|
|
||||||
- Agent response analysis (tool calls, commands, text_response status)3. After receiving tool results, you MUST provide a text_response that answers the user's question using the tool data
|
|
||||||
|
|
||||||
- LLM reasoning4. Do NOT call the same tool again
|
|
||||||
|
|
||||||
- Tool output preview5. Example flow:
|
|
||||||
|
|
||||||
- Step-by-step execution flow User: "What dungeons are there?"
|
|
||||||
|
|
||||||
Assistant (first response): { "tool_calls": [{"tool_name": "resource-list", "args": {"type": "dungeon"}}] }
|
|
||||||
|
|
||||||
### Configuration Parameters [Tool executes and returns dungeon list]
|
|
||||||
|
|
||||||
```bash Assistant (second response): { "text_response": "Based on the resource list, there are X dungeons: [list them]" }
|
|
||||||
|
|
||||||
--max-tool-iterations=6 # Default: 4```
|
|
||||||
|
|
||||||
--max-retries=5 # Default: 3
|
|
||||||
|
|
||||||
--no-reasoning # Hide LLM reasoning## 📋 Testing
|
|
||||||
|
|
||||||
```
|
|
||||||
|
|
||||||
### Test Script
|
|
||||||
|
|
||||||
### Colorful OutputCreated `test_simple_chat_ollama.sh` with comprehensive tests:
|
|
||||||
|
|
||||||
- 🔧 Tool calls in magenta- ✅ Prerequisites check (Ollama, model, ROM)
|
|
||||||
|
|
||||||
- ✓ Success messages in green- ✅ Single message mode test
|
|
||||||
|
|
||||||
- ⚠ Warnings in yellow- ✅ Piped input test
|
|
||||||
|
|
||||||
- ✗ Errors in red- ✅ Interactive mode test (with auto-exit)
|
|
||||||
|
|
||||||
- ℹ Info in blue- ✅ Batch mode test
|
|
||||||
|
|
||||||
- 💭 Reasoning in dim yellow- ⚠️ Tool calling verification (needs prompt refinement)
|
|
||||||
|
|
||||||
- ⠋ Loading spinner (cyan)
|
|
||||||
|
|
||||||
### Manual Test Results
|
|
||||||
|
|
||||||
## 📋 Next Steps
|
|
||||||
|
|
||||||
**Test 1: Single Message**
|
|
||||||
|
|
||||||
### Priority 1: Fix Empty Tool Results (HIGH)```bash
|
|
||||||
|
|
||||||
1. Add dungeon category to embedded labels OR./build_test/bin/z3ed agent simple-chat "What dungeons are in this ROM?" \
|
|
||||||
|
|
||||||
2. Update all prompt examples to use `room` instead of `dungeon` --rom=assets/zelda3.sfc --ai_provider=ollama
|
|
||||||
|
|
||||||
3. Test that tools return actual data```
|
|
||||||
|
|
||||||
4. Verify LLM can process tool results**Result**:
|
|
||||||
|
|
||||||
- ✅ Ollama connects successfully
|
|
||||||
|
|
||||||
### Priority 2: Refine Prompts (MEDIUM)- ✅ Model loads (qwen2.5-coder:7b)
|
|
||||||
|
|
||||||
Once tools return data:- ❌ Hits 4-iteration limit without final response
|
|
||||||
|
|
||||||
1. Test if LLM provides final text_response after tool results
|
|
||||||
|
|
||||||
2. Adjust system prompt if loop persists**Test 2: Tool Availability**
|
|
||||||
|
|
||||||
3. Test with different Ollama models (llama3, codellama)```bash
|
|
||||||
|
|
||||||
./build_test/bin/z3ed agent resource-list --type=dungeon --format=json --rom=assets/zelda3.sfc
|
|
||||||
|
|
||||||
### Priority 3: Documentation (LOW)```
|
|
||||||
|
|
||||||
1. Document available resource categories**Result**: ✅ Returns proper JSON with dungeon names
|
|
||||||
|
|
||||||
2. Add troubleshooting guide
|
|
||||||
|
|
||||||
3. Create example queries for each tool## 🔧 Next Steps
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## 🧪 Testing Commands### Priority 1: Fix Tool Calling Loop (High Priority)
|
|
||||||
|
|
||||||
1. **Update system prompt** in `prompt_builder.cc`:
|
|
||||||
|
|
||||||
```bash - Add explicit instructions for tool usage workflow
|
|
||||||
|
|
||||||
# Test with verbose mode - Include examples showing tool call → results → final response
|
|
||||||
|
|
||||||
./build_test/bin/z3ed agent simple-chat "What rooms are there?" \\ - Emphasize that `text_response` is REQUIRED after receiving tool results
|
|
||||||
|
|
||||||
--rom=assets/zelda3.sfc --ai_provider=ollama --verbose --max-tool-iterations=6
|
|
||||||
|
|
||||||
2. **Enhance examples** in `prompt_catalogue.yaml`:
|
|
||||||
|
|
||||||
# Test resource categories - Add multi-turn examples showing tool usage
|
|
||||||
|
|
||||||
./build_test/bin/z3ed agent resource-list --type=room --rom=assets/zelda3.sfc - Show correct pattern: question → tool_call → (wait) → text_response with tool data
|
|
||||||
|
|
||||||
./build_test/bin/z3ed agent resource-list --type=entrance --rom=assets/zelda3.sfc
|
|
||||||
|
|
||||||
./build_test/bin/z3ed agent resource-list --type=sprite --rom=assets/zelda3.sfc3. **Improve response validation** in `ollama_ai_service.cc`:
|
|
||||||
|
|
||||||
- Detect when tool results are in history but no text_response provided
|
|
||||||
|
|
||||||
# Test with Gemini (if API key available) - Add warning messages for debugging
|
|
||||||
|
|
||||||
export GEMINI_API_KEY='your-key'
|
|
||||||
|
|
||||||
./build_test/bin/z3ed agent simple-chat "What rooms are in this ROM?" \\### Priority 2: Testing & Validation (Medium Priority)
|
|
||||||
|
|
||||||
--rom=assets/zelda3.sfc --ai_provider=gemini --verbose1. Test with different Ollama models:
|
|
||||||
|
|
||||||
``` - qwen2.5-coder:7b (current)
|
|
||||||
|
|
||||||
- llama3:8b
|
|
||||||
|
|
||||||
## 📊 Performance - codellama:7b
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
- Ollama response: ~2-5 seconds (qwen2.5-coder:7b)2. Create regression test suite for tool calling:
|
|
||||||
|
|
||||||
- Tool execution: <100ms - Test each tool individually
|
|
||||||
|
|
||||||
- Loading indicator: Smooth 80ms refresh rate - Test multi-tool sequences
|
|
||||||
|
|
||||||
- Test conversation context preservation
|
|
||||||
|
|
||||||
## 🎯 Success Criteria
|
|
||||||
|
|
||||||
### Priority 3: Documentation (Low Priority)
|
|
||||||
|
|
||||||
- [x] Colorful, user-friendly output1. Update `simple_chat_input_methods.md` with:
|
|
||||||
|
|
||||||
- [x] Verbose mode for debugging - Known limitations section
|
|
||||||
|
|
||||||
- [x] Configurable parameters - Troubleshooting for tool calling issues
|
|
||||||
|
|
||||||
- [x] File-based prompts for easy updates - Recommended models and configurations
|
|
||||||
|
|
||||||
- [ ] Tools return actual data (BLOCKED on dungeon labels)
|
|
||||||
|
|
||||||
- [ ] LLM provides final response after tool calls2. Create `ollama_best_practices.md`:
|
|
||||||
|
|
||||||
- [ ] Zero infinite loops - Model recommendations
|
|
||||||
|
|
||||||
- Temperature/parameter tuning
|
|
||||||
|
|
||||||
--- - Prompt engineering tips
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
**Last Updated**: October 4, 2025## 📊 Performance Notes
|
|
||||||
|
|
||||||
**Status**: 🟡 Blocked on empty tool results - need to fix dungeon labels
|
|
||||||
|
|
||||||
**Next Action**: Add dungeon category to embedded labels OR update prompt examples- **Ollama Response Time**: ~2-5 seconds per query (qwen2.5-coder:7b on typical hardware)
|
|
||||||
|
|
||||||
- **Tool Execution**: <100ms per tool call
|
|
||||||
- **Total Interaction**: ~2-5 seconds for simple queries, longer for multi-turn with tools
|
|
||||||
|
|
||||||
## 🐛 Known Issues
|
|
||||||
|
|
||||||
1. **Tool Calling Loop**: Agent doesn't provide final response after tool execution (see above)
|
|
||||||
2. **No Streaming**: Responses are blocking (not streamed), so user sees delay
|
|
||||||
3. **Limited Context**: Prompt builder doesn't include full conversation context in system prompt
|
|
||||||
|
|
||||||
## 💡 Recommendations
|
|
||||||
|
|
||||||
### For Users
|
|
||||||
- Use MockAIService for testing until tool calling is fixed
|
|
||||||
- For production, prefer Gemini (has native function calling support)
|
|
||||||
- Keep queries simple and direct
|
|
||||||
|
|
||||||
### For Developers
|
|
||||||
- Focus on fixing the tool calling loop first
|
|
||||||
- Consider implementing streaming responses
|
|
||||||
- Add debug logging to track tool call cycles
|
|
||||||
- Test with multiple Ollama models to find best performer
|
|
||||||
|
|
||||||
## 📝 Related Files
|
|
||||||
|
|
||||||
- `/Users/scawful/Code/yaze/src/cli/cli_main.cc` - Flag parsing (FIXED ✅)
|
|
||||||
- `/Users/scawful/Code/yaze/src/cli/service/ai/ollama_ai_service.cc` - Ollama integration
|
|
||||||
- `/Users/scawful/Code/yaze/src/cli/service/ai/prompt_builder.cc` - System prompt generation (NEEDS FIX 🚧)
|
|
||||||
- `/Users/scawful/Code/yaze/src/cli/service/agent/conversational_agent_service.cc` - Tool execution loop
|
|
||||||
- `/Users/scawful/Code/yaze/assets/agent/prompt_catalogue.yaml` - Tool definitions and examples (NEEDS ENHANCEMENT 🚧)
|
|
||||||
- `/Users/scawful/Code/yaze/docs/simple_chat_input_methods.md` - User documentation
|
|
||||||
- `/Users/scawful/Code/yaze/test_simple_chat_ollama.sh` - Test script
|
|
||||||
|
|
||||||
## 🎯 Success Criteria
|
|
||||||
|
|
||||||
### Minimum Viable
|
|
||||||
- [ ] LLM successfully calls tools
|
|
||||||
- [ ] LLM provides final text_response after receiving tool results
|
|
||||||
- [ ] No infinite loops (completes within 4 iterations)
|
|
||||||
- [ ] Accurate answers to simple questions ("What dungeons?", "List sprites in room X")
|
|
||||||
|
|
||||||
### Full Success
|
|
||||||
- [ ] All 5 tools work correctly with Ollama
|
|
||||||
- [ ] Multi-turn conversations maintain context
|
|
||||||
- [ ] Works with 3+ different Ollama models
|
|
||||||
- [ ] Response time <5 seconds for typical queries
|
|
||||||
- [ ] Comprehensive test coverage
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Last Updated**: October 4, 2025
|
|
||||||
**Status**: 🟡 Partially Working - Core infrastructure complete, prompt refinement needed
|
|
||||||
**Next Action**: Update system prompt to fix tool calling loop
|
|
||||||
@@ -1,388 +0,0 @@
|
|||||||
# yaze Overworld Testing Guide
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
This guide provides comprehensive documentation for testing the YAZE overworld implementation, including unit tests, integration tests, end-to-end tests, and golden data validation. The testing framework ensures that the YAZE C++ implementation correctly mirrors the ZScream C# logic.
|
|
||||||
|
|
||||||
## Test Architecture
|
|
||||||
|
|
||||||
### 1. Golden Data System
|
|
||||||
|
|
||||||
The golden data system provides a "source of truth" for ROM validation:
|
|
||||||
|
|
||||||
- **Golden Data Extractor**: Extracts all overworld-related values from ROMs
|
|
||||||
- **Before/After Validation**: Compares ROM states before and after edits
|
|
||||||
- **Reference Data**: Provides expected values for test validation
|
|
||||||
|
|
||||||
### 2. Test Categories
|
|
||||||
|
|
||||||
#### Unit Tests (`test/unit/zelda3/`)
|
|
||||||
- **`overworld_test.cc`**: Basic overworld functionality tests
|
|
||||||
- **`overworld_integration_test.cc`**: Comprehensive integration tests
|
|
||||||
- **`extract_vanilla_values.cc`**: ROM value extraction utility
|
|
||||||
|
|
||||||
#### Integration Tests (`test/e2e/`)
|
|
||||||
- **`overworld/overworld_e2e_test.cc`**: End-to-end workflow tests
|
|
||||||
- **`zscustomoverworld/zscustomoverworld_upgrade_test.cc`**: ASM version upgrade tests
|
|
||||||
|
|
||||||
#### Golden Data Tools
|
|
||||||
- **`overworld_golden_data_extractor.cc`**: Comprehensive ROM data extraction
|
|
||||||
- **`run_overworld_tests.sh`**: Orchestrated test runner script
|
|
||||||
|
|
||||||
## Quick Start
|
|
||||||
|
|
||||||
### Prerequisites
|
|
||||||
|
|
||||||
1. **Build YAZE**: Ensure the project is built with tests enabled
|
|
||||||
2. **ROM File**: Have a valid Zelda 3 ROM file (zelda3.sfc)
|
|
||||||
3. **Environment**: Set up test environment variables
|
|
||||||
|
|
||||||
### Running Tests
|
|
||||||
|
|
||||||
#### Basic Test Run
|
|
||||||
```bash
|
|
||||||
# Run all overworld tests with a ROM
|
|
||||||
./scripts/run_overworld_tests.sh zelda3.sfc
|
|
||||||
|
|
||||||
# Generate detailed report
|
|
||||||
./scripts/run_overworld_tests.sh zelda3.sfc --generate-report
|
|
||||||
|
|
||||||
# Clean up test files after completion
|
|
||||||
./scripts/run_overworld_tests.sh zelda3.sfc --cleanup
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Selective Test Execution
|
|
||||||
```bash
|
|
||||||
# Skip unit tests, run only integration and E2E
|
|
||||||
./scripts/run_overworld_tests.sh zelda3.sfc --skip-unit-tests
|
|
||||||
|
|
||||||
# Extract golden data only
|
|
||||||
./scripts/run_overworld_tests.sh zelda3.sfc --skip-unit-tests --skip-integration --skip-e2e
|
|
||||||
|
|
||||||
# Run with custom ROM path
|
|
||||||
export YAZE_TEST_ROM_PATH="/path/to/custom/rom.sfc"
|
|
||||||
./scripts/run_overworld_tests.sh /path/to/custom/rom.sfc
|
|
||||||
```
|
|
||||||
|
|
||||||
## Test Components
|
|
||||||
|
|
||||||
### 1. Golden Data Extractor
|
|
||||||
|
|
||||||
The `overworld_golden_data_extractor` tool extracts comprehensive data from ROMs:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Extract golden data from a ROM
|
|
||||||
./build/bin/overworld_golden_data_extractor zelda3.sfc golden_data.h
|
|
||||||
```
|
|
||||||
|
|
||||||
**Extracted Data Includes:**
|
|
||||||
- Basic ROM information (title, size, checksums)
|
|
||||||
- ASM version and feature flags
|
|
||||||
- Overworld map properties (graphics, palettes, sizes)
|
|
||||||
- Tile data (Tile16/Tile32 expansion status)
|
|
||||||
- Entrance/hole/exit coordinate data
|
|
||||||
- Item and sprite data
|
|
||||||
- Map tiles (compressed data)
|
|
||||||
- Palette and music data
|
|
||||||
- Overlay data
|
|
||||||
|
|
||||||
### 2. Integration Tests
|
|
||||||
|
|
||||||
The integration tests validate core overworld functionality:
|
|
||||||
|
|
||||||
```cpp
|
|
||||||
// Example: Test coordinate calculation accuracy
|
|
||||||
TEST_F(OverworldIntegrationTest, ZScreamCoordinateCompatibility) {
|
|
||||||
// Load overworld data
|
|
||||||
auto status = overworld_->Load(rom_.get());
|
|
||||||
ASSERT_TRUE(status.ok());
|
|
||||||
|
|
||||||
// Validate coordinate calculations match ZScream exactly
|
|
||||||
const auto& entrances = overworld_->entrances();
|
|
||||||
for (int i = 0; i < 10; i++) {
|
|
||||||
const auto& entrance = entrances[i];
|
|
||||||
|
|
||||||
// ZScream coordinate calculation logic
|
|
||||||
uint16_t map_pos = entrance.map_pos();
|
|
||||||
uint16_t map_id = entrance.map_id();
|
|
||||||
|
|
||||||
int position = map_pos >> 1;
|
|
||||||
int x_coord = position % 64;
|
|
||||||
int y_coord = position >> 6;
|
|
||||||
int expected_x = (x_coord * 16) + (((map_id % 64) - (((map_id % 64) / 8) * 8)) * 512);
|
|
||||||
int expected_y = (y_coord * 16) + (((map_id % 64) / 8) * 512);
|
|
||||||
|
|
||||||
EXPECT_EQ(entrance.x(), expected_x);
|
|
||||||
EXPECT_EQ(entrance.y(), expected_y);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. End-to-End Tests
|
|
||||||
|
|
||||||
E2E tests validate complete workflows:
|
|
||||||
|
|
||||||
```cpp
|
|
||||||
// Example: Test ASM version upgrade workflow
|
|
||||||
TEST_F(OverworldE2ETest, ApplyZSCustomOverworldV3) {
|
|
||||||
// Load vanilla ROM
|
|
||||||
std::unique_ptr<Rom> rom = std::make_unique<Rom>();
|
|
||||||
ASSERT_OK(rom->LoadFromFile(vanilla_test_path_));
|
|
||||||
|
|
||||||
// Apply ZSCustomOverworld v3 ASM
|
|
||||||
ASSERT_OK(rom->WriteByte(0x140145, 0x03)); // Set ASM version to v3
|
|
||||||
|
|
||||||
// Enable v3 features
|
|
||||||
ASSERT_OK(rom->WriteByte(0x140146, 0x01)); // Enable main palettes
|
|
||||||
ASSERT_OK(rom->WriteByte(0x140147, 0x01)); // Enable area-specific BG
|
|
||||||
// ... more feature flags
|
|
||||||
|
|
||||||
// Save and reload
|
|
||||||
ASSERT_OK(rom->SaveToFile(Rom::SaveSettings{.filename = edited_test_path_}));
|
|
||||||
|
|
||||||
// Validate changes persisted
|
|
||||||
std::unique_ptr<Rom> reloaded_rom = std::make_unique<Rom>();
|
|
||||||
ASSERT_OK(reloaded_rom->LoadFromFile(edited_test_path_));
|
|
||||||
|
|
||||||
auto asm_version = reloaded_rom->ReadByte(0x140145);
|
|
||||||
EXPECT_EQ(*asm_version, 0x03);
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Test Validation Points
|
|
||||||
|
|
||||||
### 1. ZScream Compatibility
|
|
||||||
|
|
||||||
Tests ensure YAZE behavior matches ZScream exactly:
|
|
||||||
|
|
||||||
- **Coordinate Calculations**: Entrance/hole coordinates use identical formulas
|
|
||||||
- **ASM Version Detection**: Proper handling of vanilla vs ZSCustomOverworld ROMs
|
|
||||||
- **Data Structure Loading**: Same data organization and access patterns
|
|
||||||
- **Expansion Detection**: Correct identification of expanded vs vanilla data
|
|
||||||
|
|
||||||
### 2. ROM State Validation
|
|
||||||
|
|
||||||
Tests validate ROM integrity:
|
|
||||||
|
|
||||||
- **Before/After Comparison**: Ensure edits are properly saved and loaded
|
|
||||||
- **Feature Flag Persistence**: ZSCustomOverworld features remain enabled after save
|
|
||||||
- **Data Integrity**: Original data is preserved where expected
|
|
||||||
- **Checksum Validation**: ROM remains valid after modifications
|
|
||||||
|
|
||||||
### 3. Performance and Stability
|
|
||||||
|
|
||||||
Tests ensure robustness:
|
|
||||||
|
|
||||||
- **Multiple Load Cycles**: Repeated load/unload operations
|
|
||||||
- **Memory Management**: Proper cleanup of resources
|
|
||||||
- **Error Handling**: Graceful handling of invalid data
|
|
||||||
- **Thread Safety**: Concurrent access patterns (if applicable)
|
|
||||||
|
|
||||||
## Environment Variables
|
|
||||||
|
|
||||||
### Test Configuration
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# ROM path for testing
|
|
||||||
export YAZE_TEST_ROM_PATH="/path/to/rom.sfc"
|
|
||||||
|
|
||||||
# Skip ROM-dependent tests
|
|
||||||
export YAZE_SKIP_ROM_TESTS=1
|
|
||||||
|
|
||||||
# Test output verbosity
|
|
||||||
export GTEST_VERBOSE=1
|
|
||||||
```
|
|
||||||
|
|
||||||
### Build Configuration
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Enable tests in CMake
|
|
||||||
cmake -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTS=ON ..
|
|
||||||
|
|
||||||
# Build with specific test targets
|
|
||||||
cmake --build build --target overworld_golden_data_extractor
|
|
||||||
cmake --build build --target extract_vanilla_values
|
|
||||||
```
|
|
||||||
|
|
||||||
## Test Reports
|
|
||||||
|
|
||||||
### Generated Reports
|
|
||||||
|
|
||||||
The test runner generates comprehensive reports:
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
# yaze Overworld Test Report
|
|
||||||
|
|
||||||
**Generated:** 2024-01-15 14:30:00
|
|
||||||
**ROM:** zelda3.sfc
|
|
||||||
**ROM Size:** 2097152 bytes
|
|
||||||
|
|
||||||
## Test Results Summary
|
|
||||||
|
|
||||||
| Test Category | Status | Details |
|
|
||||||
|---------------|--------|---------|
|
|
||||||
| Golden Data Extraction | SUCCESS: golden_data.h | |
|
|
||||||
| Unit Tests | PASSED | |
|
|
||||||
| Integration Tests | PASSED | |
|
|
||||||
| E2E Tests | PASSED | |
|
|
||||||
|
|
||||||
## ROM Information
|
|
||||||
|
|
||||||
### ROM Header
|
|
||||||
[Hex dump of ROM header for validation]
|
|
||||||
```
|
|
||||||
|
|
||||||
### Report Location
|
|
||||||
|
|
||||||
Reports are saved to `test/reports/` with timestamps:
|
|
||||||
- `overworld_test_report_YYYYMMDD_HHMMSS.md`
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### Common Issues
|
|
||||||
|
|
||||||
#### 1. ROM Not Found
|
|
||||||
```
|
|
||||||
Error: ROM file not found: zelda3.sfc
|
|
||||||
```
|
|
||||||
**Solution**: Provide correct ROM path or set `YAZE_TEST_ROM_PATH`
|
|
||||||
|
|
||||||
#### 2. Build Failures
|
|
||||||
```
|
|
||||||
Error: Failed to build golden data extractor
|
|
||||||
```
|
|
||||||
**Solution**: Ensure project is built with tests enabled:
|
|
||||||
```bash
|
|
||||||
cmake -DBUILD_TESTS=ON ..
|
|
||||||
cmake --build build
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 3. Test Failures
|
|
||||||
```
|
|
||||||
Error: Some tests failed. Check the results above.
|
|
||||||
```
|
|
||||||
**Solution**:
|
|
||||||
- Check individual test logs for specific failures
|
|
||||||
- Verify ROM file is valid Zelda 3 ROM
|
|
||||||
- Ensure test environment is properly configured
|
|
||||||
|
|
||||||
### Debug Mode
|
|
||||||
|
|
||||||
Run tests with maximum verbosity:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Enable debug output
|
|
||||||
export GTEST_VERBOSE=1
|
|
||||||
export YAZE_DEBUG=1
|
|
||||||
|
|
||||||
# Run with verbose output
|
|
||||||
./scripts/run_overworld_tests.sh zelda3.sfc --generate-report
|
|
||||||
```
|
|
||||||
|
|
||||||
## Advanced Usage
|
|
||||||
|
|
||||||
### Custom Test Scenarios
|
|
||||||
|
|
||||||
#### 1. Testing Custom ROMs
|
|
||||||
```bash
|
|
||||||
# Test with modified ROM
|
|
||||||
./scripts/run_overworld_tests.sh /path/to/modified_rom.sfc
|
|
||||||
|
|
||||||
# Extract golden data for comparison
|
|
||||||
./build/bin/overworld_golden_data_extractor /path/to/modified_rom.sfc modified_golden_data.h
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 2. Regression Testing
|
|
||||||
```bash
|
|
||||||
# Extract golden data from known good ROM
|
|
||||||
./build/bin/overworld_golden_data_extractor known_good_rom.sfc reference_golden_data.h
|
|
||||||
|
|
||||||
# Test modified ROM against reference
|
|
||||||
./scripts/run_overworld_tests.sh modified_rom.sfc --generate-report
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 3. Performance Testing
|
|
||||||
```bash
|
|
||||||
# Run performance-focused tests
|
|
||||||
export YAZE_PERFORMANCE_TESTS=1
|
|
||||||
./scripts/run_overworld_tests.sh zelda3.sfc
|
|
||||||
```
|
|
||||||
|
|
||||||
## Integration with CI/CD
|
|
||||||
|
|
||||||
### GitHub Actions Example
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
name: Overworld Tests
|
|
||||||
|
|
||||||
on: [push, pull_request]
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
test:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@v3
|
|
||||||
|
|
||||||
- name: Build YAZE
|
|
||||||
run: |
|
|
||||||
cmake -DBUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Debug ..
|
|
||||||
cmake --build build
|
|
||||||
|
|
||||||
- name: Download Test ROM
|
|
||||||
run: |
|
|
||||||
# Download test ROM (placeholder - use actual ROM)
|
|
||||||
wget -O zelda3.sfc https://example.com/test_rom.sfc
|
|
||||||
|
|
||||||
- name: Run Overworld Tests
|
|
||||||
run: |
|
|
||||||
chmod +x scripts/run_overworld_tests.sh
|
|
||||||
./scripts/run_overworld_tests.sh zelda3.sfc --generate-report
|
|
||||||
|
|
||||||
- name: Upload Test Reports
|
|
||||||
uses: actions/upload-artifact@v3
|
|
||||||
with:
|
|
||||||
name: test-reports
|
|
||||||
path: test/reports/
|
|
||||||
```
|
|
||||||
|
|
||||||
## Contributing
|
|
||||||
|
|
||||||
### Adding New Tests
|
|
||||||
|
|
||||||
1. **Unit Tests**: Add to `test/unit/zelda3/overworld_test.cc`
|
|
||||||
2. **Integration Tests**: Add to `test/unit/zelda3/overworld_integration_test.cc`
|
|
||||||
3. **E2E Tests**: Add to `test/e2e/overworld/overworld_e2e_test.cc`
|
|
||||||
|
|
||||||
### Test Guidelines
|
|
||||||
|
|
||||||
- **Naming**: Use descriptive test names that explain the scenario
|
|
||||||
- **Documentation**: Include comments explaining complex test logic
|
|
||||||
- **Isolation**: Each test should be independent and not rely on others
|
|
||||||
- **Cleanup**: Ensure tests clean up resources and temporary files
|
|
||||||
|
|
||||||
### Example Test Structure
|
|
||||||
|
|
||||||
```cpp
|
|
||||||
// Test descriptive name that explains the scenario
|
|
||||||
TEST_F(OverworldIntegrationTest, Tile32ExpansionDetectionWithV3ASM) {
|
|
||||||
// Setup: Configure test environment
|
|
||||||
mock_rom_data_[0x01772E] = 0x04; // Set expansion flag
|
|
||||||
|
|
||||||
// Execute: Run the code under test
|
|
||||||
auto status = overworld_->Load(rom_.get());
|
|
||||||
ASSERT_TRUE(status.ok());
|
|
||||||
|
|
||||||
// Verify: Check expected outcomes
|
|
||||||
EXPECT_TRUE(overworld_->expanded_tile32());
|
|
||||||
EXPECT_FALSE(overworld_->expanded_tile16());
|
|
||||||
|
|
||||||
// Cleanup: (handled by test framework)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
The YAZE overworld testing framework provides comprehensive validation of the C++ implementation against the ZScream C# reference. The golden data system ensures consistency, while the multi-layered test approach (unit, integration, E2E) provides confidence in the implementation's correctness and robustness.
|
|
||||||
|
|
||||||
For questions or issues, refer to the test logs and generated reports, or consult the YAZE development team.
|
|
||||||
Reference in New Issue
Block a user