diff --git a/scripts/README.md b/scripts/README.md index bc542f70..d52910a6 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,134 +1,117 @@ -# yaze Build Scripts +# YAZE Build Scripts -This directory contains build and setup scripts for YAZE development on different platforms. +This directory contains build automation and maintenance scripts for the YAZE project. -## Windows Scripts +## build_cleaner.py -### vcpkg Setup (Optional) -- **`setup-vcpkg-windows.ps1`** - Setup vcpkg for SDL2 dependency (PowerShell) -- **`setup-vcpkg-windows.bat`** - Setup vcpkg for SDL2 dependency (Batch) +Automates CMake source list maintenance and header include management with IWYU-style analysis. -**Note**: vcpkg is optional. YAZE uses bundled dependencies by default. +### Features -## Windows Build Workflow - -### Recommended: Visual Studio CMake Mode - -**YAZE uses Visual Studio's native CMake support - no project generation needed.** - -1. **Install Visual Studio 2022** with "Desktop development with C++" workload -2. **Open Visual Studio 2022** -3. **File → Open → Folder** -4. **Navigate to yaze directory** -5. **Select configuration** (Debug/Release) from toolbar -6. **Press F5** to build and run - -### Command Line Build - -```powershell -# Standard CMake workflow -cmake -B build -DCMAKE_BUILD_TYPE=Debug -cmake --build build --config Debug - -# Run the application -.\build\bin\Debug\yaze.exe -``` - -### Compiler Notes - -The CMake configuration is designed to work with **MSVC** (Visual Studio's compiler). The build system includes automatic workarounds for C++23 compatibility: -- Abseil int128 is automatically excluded on Windows -- C++23 deprecation warnings are properly silenced -- Platform-specific definitions are handled automatically - -## Quick Start (Windows) - -### Option 1: Visual Studio (Recommended) -1. Open Visual Studio 2022 -2. File → Open → Folder -3. Navigate to yaze directory -4. Select configuration (Debug/Release) -5. Press F5 to build and run - -### Option 2: Command Line -```powershell -# Standard CMake build -cmake -B build -DCMAKE_BUILD_TYPE=Debug -cmake --build build --config Debug -``` - -### Option 3: With vcpkg (Optional) -```powershell -# Setup vcpkg for SDL2 -.\scripts\setup-vcpkg-windows.ps1 - -# Then build normally in Visual Studio or command line -``` - -## Troubleshooting - -### Common Issues - -1. **PowerShell Execution Policy** - ```powershell - Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser - ``` - -2. **Visual Studio Not Detecting CMakeLists.txt** - - Ensure you opened the folder (not a solution file) - - Check that CMakeLists.txt exists in the root directory - - Try: File → Close Folder, then File → Open → Folder - -3. **vcpkg Issues (if using vcpkg)** - - Run `.\scripts\setup-vcpkg-windows.ps1` to reinstall - - Check internet connection for dependency downloads - - Note: vcpkg is optional; bundled dependencies work by default - -4. **CMake Configuration Errors** - - Delete the `build/` directory - - Close and reopen the folder in Visual Studio - - Or run: `cmake -B build -DCMAKE_BUILD_TYPE=Debug` - -5. **Missing Git Submodules** - ```powershell - git submodule update --init --recursive - ``` - -### Getting Help - -1. Check the [Build Instructions](../docs/02-build-instructions.md) -2. Review CMake output for specific error messages -3. Ensure all prerequisites are installed (Visual Studio 2022 with C++ workload) - -## Other Scripts - -- **`create_release.sh`** - Create GitHub releases (Linux/macOS) -- **`extract_changelog.py`** - Extract changelog for releases -- **`quality_check.sh`** - Code quality checks (Linux/macOS) -- **`create-macos-bundle.sh`** - Create macOS application bundle for releases - -## Build Environment Verification - -This directory also contains build environment verification scripts. - -### `verify-build-environment.ps1` / `.sh` - -A comprehensive script that checks: - -- ✅ **CMake Installation** - Version 3.16+ required -- ✅ **Git Installation** - With submodule support -- ✅ **C++ Compiler** - GCC 13+, Clang 16+, or MSVC 2019+ -- ✅ **Platform Tools** - Xcode (macOS), Visual Studio (Windows), build-essential (Linux) -- ✅ **Git Submodules** - All dependencies synchronized +1. **CMake Source List Maintenance**: Automatically updates source file lists in CMake files +2. **Self-Header Includes**: Ensures source files include their corresponding headers +3. **IWYU-Style Analysis**: Suggests missing headers based on symbol usage +4. **.gitignore Support**: Respects .gitignore patterns when scanning files +5. **Auto-Discovery**: Can discover CMake libraries that opt-in to auto-maintenance ### Usage -**Windows (PowerShell):** -```powershell -.\scripts\verify-build-environment.ps1 +```bash +# Dry-run to see what would change (recommended first step) +python3 scripts/build_cleaner.py --dry-run + +# Update CMake source lists and header includes +python3 scripts/build_cleaner.py + +# Run IWYU-style header analysis +python3 scripts/build_cleaner.py --iwyu + +# Auto-discover CMake libraries marked for auto-maintenance +python3 scripts/build_cleaner.py --auto-discover + +# Update only CMake source lists +python3 scripts/build_cleaner.py --cmake-only + +# Update only header includes +python3 scripts/build_cleaner.py --includes-only ``` -**macOS/Linux:** -```bash -./scripts/verify-build-environment.sh +### Opting-In to Auto-Maintenance + +By default, the script only auto-maintains source lists that are explicitly marked. To mark a CMake variable for auto-maintenance, add a comment above the `set()` statement: + +```cmake +# This list is auto-maintained by scripts/build_cleaner.py +set( + YAZE_APP_EMU_SRC + app/emu/audio/apu.cc + app/emu/cpu/cpu.cc + # ... more files +) ``` + +The script looks for comments containing "auto-maintain" (case-insensitive) within 3 lines above the `set()` statement. + +### Excluding Files from Processing + +To exclude a specific file from all processing (CMake lists, header includes, IWYU), add this token near the top of the file: + +```cpp +// build_cleaner:ignore +``` + +### .gitignore Support + +The script automatically respects `.gitignore` patterns. To enable this feature, install the `pathspec` dependency: + +```bash +pip3 install -r scripts/requirements.txt +# or +pip3 install pathspec +``` + +### IWYU Configuration + +The script includes basic IWYU-style analysis that suggests headers based on symbol prefixes. To customize which headers are suggested, edit the `COMMON_HEADERS` dictionary in the script: + +```python +COMMON_HEADERS = { + 'std::': ['', '', '', ...], + 'absl::': ['', ...], + 'ImGui::': [''], + 'SDL_': [''], +} +``` + +**Note**: The IWYU analysis is conservative and may suggest headers that are already transitively included. Use with care and review suggestions before applying. + +### Integration with CMake + +The script is integrated into the CMake build system: + +```bash +# Run as a CMake target +cmake --build build --target build_cleaner +``` + +### Dependencies + +- Python 3.7+ +- `pathspec` (optional, for .gitignore support): `pip3 install pathspec` + +### How It Works + +1. **CMake Maintenance**: Scans directories specified in the configuration and updates `set(VAR_NAME ...)` blocks with the current list of source files +2. **Self-Headers**: For each `.cc`/`.cpp` file, ensures it includes its corresponding `.h` file +3. **IWYU Analysis**: Scans source files for symbols and suggests appropriate headers based on prefix matching + +### Current Auto-Maintained Variables + +**All 20 library source lists are now auto-maintained by default:** + +- Core: `YAZE_APP_EMU_SRC`, `YAZE_APP_CORE_SRC`, `YAZE_APP_EDITOR_SRC`, `YAZE_APP_ZELDA3_SRC`, `YAZE_NET_SRC`, `YAZE_UTIL_SRC` +- GFX: `GFX_TYPES_SRC`, `GFX_BACKEND_SRC`, `GFX_RESOURCE_SRC`, `GFX_CORE_SRC`, `GFX_UTIL_SRC`, `GFX_RENDER_SRC`, `GFX_DEBUG_SRC` +- GUI: `GUI_CORE_SRC`, `CANVAS_SRC`, `GUI_WIDGETS_SRC`, `GUI_AUTOMATION_SRC`, `GUI_APP_SRC` +- Other: `YAZE_AGENT_SOURCES`, `YAZE_TEST_SOURCES` + +The script intelligently preserves conditional blocks (if/endif) and excludes conditional files from the main source list. diff --git a/scripts/build_cleaner.py b/scripts/build_cleaner.py index 2609cbf1..ba9c6fe8 100644 --- a/scripts/build_cleaner.py +++ b/scripts/build_cleaner.py @@ -7,7 +7,15 @@ import argparse from dataclasses import dataclass, field import re from pathlib import Path -from typing import Dict, Iterable, List, Optional, Sequence, Set +from typing import Any, Iterable, List, Optional, Sequence, Set + +try: + import pathspec + HAS_PATHSPEC = True +except ImportError: + HAS_PATHSPEC = False + print("Warning: 'pathspec' module not found. Install with: pip3 install pathspec") + print(" .gitignore support will be disabled.") PROJECT_ROOT = Path(__file__).resolve().parent.parent SOURCE_ROOT = PROJECT_ROOT / "src" @@ -16,6 +24,14 @@ SUPPORTED_EXTENSIONS = (".cc", ".c", ".cpp", ".cxx", ".mm") HEADER_EXTENSIONS = (".h", ".hh", ".hpp", ".hxx") BUILD_CLEANER_IGNORE_TOKEN = "build_cleaner:ignore" +# Common SNES/ROM header patterns to include +COMMON_HEADERS = { + 'std::': ['', '', '', '', '', '', ''], + 'absl::': ['', '', ''], + 'ImGui::': [''], + 'SDL_': [''], +} + @dataclass(frozen=True) class DirectorySpec: @@ -43,7 +59,96 @@ class CMakeSourceBlock: exclude: Set[Path] = field(default_factory=set) -CONFIG: Sequence[CMakeSourceBlock] = ( +def load_gitignore(): + """Load .gitignore patterns into a pathspec matcher.""" + if not HAS_PATHSPEC: + return None + + gitignore_path = PROJECT_ROOT / ".gitignore" + if not gitignore_path.exists(): + return None + + try: + with gitignore_path.open('r', encoding='utf-8') as f: + patterns = [line.strip() for line in f if line.strip() and not line.startswith('#')] + return pathspec.PathSpec.from_lines('gitwildmatch', patterns) + except Exception as e: + print(f"Warning: Could not load .gitignore: {e}") + return None + + +def is_ignored(path: Path, gitignore_spec) -> bool: + """Check if a path should be ignored based on .gitignore patterns.""" + if gitignore_spec is None or not HAS_PATHSPEC: + return False + + try: + rel_path = path.relative_to(PROJECT_ROOT) + return gitignore_spec.match_file(str(rel_path)) + except ValueError: + return False + + +def discover_cmake_libraries() -> List[CMakeSourceBlock]: + """ + Auto-discover CMake library files that explicitly opt-in to auto-maintenance. + + Looks for comments like "# This list is auto-maintained by scripts/build_cleaner.py" + or "# AUTO-MAINTAINED" to identify variables that should be auto-updated. + """ + blocks = [] + seen_vars: Set[str] = set() + + # Scan for cmake library files + for cmake_file in SOURCE_ROOT.rglob("*.cmake"): + if 'lib/' in str(cmake_file) or 'third_party/' in str(cmake_file): + continue + + try: + content = cmake_file.read_text(encoding='utf-8') + lines = content.splitlines() + + # Look for source variable definitions with auto-maintain markers + for i, line in enumerate(lines): + # Check if previous lines indicate auto-maintenance + auto_maintained = False + for j in range(max(0, i-3), i): + if 'auto-maintain' in lines[j].lower() or 'auto_maintain' in lines[j].lower(): + auto_maintained = True + break + + if not auto_maintained: + continue + + # Extract variable name from set() statement + match = re.search(r'set\s*\(\s*(\w+(?:_SRC|_SOURCES|_SOURCE))\s', line) + if match: + var_name = match.group(1) + + # Skip if we've already seen this variable + if var_name in seen_vars: + continue + + seen_vars.add(var_name) + cmake_dir = cmake_file.parent + + # Determine if recursive based on cmake file location or content + is_recursive = cmake_dir != SOURCE_ROOT / "app/core" + + blocks.append(CMakeSourceBlock( + variable=var_name, + cmake_path=cmake_file, + directories=(DirectorySpec(cmake_dir, recursive=is_recursive),), + )) + except Exception as e: + print(f"Warning: Could not process {cmake_file}: {e}") + + return blocks + + +# Static configuration for all library source lists +# The script now auto-maintains all libraries while preserving conditional sections +STATIC_CONFIG: Sequence[CMakeSourceBlock] = ( CMakeSourceBlock( variable="YAZE_APP_EMU_SRC", cmake_path=SOURCE_ROOT / "CMakeLists.txt", @@ -54,16 +159,6 @@ CONFIG: Sequence[CMakeSourceBlock] = ( cmake_path=SOURCE_ROOT / "app/core/core_library.cmake", directories=(DirectorySpec(SOURCE_ROOT / "app/core", recursive=False),), ), - CMakeSourceBlock( - variable="YAZE_APP_GFX_SRC", - cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake", - directories=(DirectorySpec(SOURCE_ROOT / "app/gfx"),), - ), - CMakeSourceBlock( - variable="YAZE_GUI_SRC", - cmake_path=SOURCE_ROOT / "app/gui/gui_library.cmake", - directories=(DirectorySpec(SOURCE_ROOT / "app/gui"),), - ), CMakeSourceBlock( variable="YAZE_APP_EDITOR_SRC", cmake_path=SOURCE_ROOT / "app/editor/editor_library.cmake", @@ -85,6 +180,85 @@ CONFIG: Sequence[CMakeSourceBlock] = ( cmake_path=SOURCE_ROOT / "util/util.cmake", directories=(DirectorySpec(SOURCE_ROOT / "util"),), ), + CMakeSourceBlock( + variable="GFX_TYPES_SRC", + cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake", + directories=(DirectorySpec(SOURCE_ROOT / "app/gfx/types"),), + ), + CMakeSourceBlock( + variable="GFX_BACKEND_SRC", + cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake", + directories=(DirectorySpec(SOURCE_ROOT / "app/gfx/backend"),), + ), + CMakeSourceBlock( + variable="GFX_RESOURCE_SRC", + cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake", + directories=(DirectorySpec(SOURCE_ROOT / "app/gfx/resource"),), + exclude={Path("app/gfx/render/background_buffer.cc")}, # This is in resource but used by render + ), + CMakeSourceBlock( + variable="GFX_CORE_SRC", + cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake", + directories=(DirectorySpec(SOURCE_ROOT / "app/gfx/core"),), + ), + CMakeSourceBlock( + variable="GFX_UTIL_SRC", + cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake", + directories=(DirectorySpec(SOURCE_ROOT / "app/gfx/util"),), + ), + CMakeSourceBlock( + variable="GFX_RENDER_SRC", + cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake", + directories=(DirectorySpec(SOURCE_ROOT / "app/gfx/render"),), + ), + CMakeSourceBlock( + variable="GFX_DEBUG_SRC", + cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake", + directories=(DirectorySpec(SOURCE_ROOT / "app/gfx/debug"),), + ), + CMakeSourceBlock( + variable="GUI_CORE_SRC", + cmake_path=SOURCE_ROOT / "app/gui/gui_library.cmake", + directories=(DirectorySpec(SOURCE_ROOT / "app/gui/core"),), + ), + CMakeSourceBlock( + variable="CANVAS_SRC", + cmake_path=SOURCE_ROOT / "app/gui/gui_library.cmake", + directories=(DirectorySpec(SOURCE_ROOT / "app/gui/canvas"),), + ), + CMakeSourceBlock( + variable="GUI_WIDGETS_SRC", + cmake_path=SOURCE_ROOT / "app/gui/gui_library.cmake", + directories=(DirectorySpec(SOURCE_ROOT / "app/gui/widgets"),), + ), + CMakeSourceBlock( + variable="GUI_AUTOMATION_SRC", + cmake_path=SOURCE_ROOT / "app/gui/gui_library.cmake", + directories=(DirectorySpec(SOURCE_ROOT / "app/gui/automation"),), + ), + CMakeSourceBlock( + variable="GUI_APP_SRC", + cmake_path=SOURCE_ROOT / "app/gui/gui_library.cmake", + directories=(DirectorySpec(SOURCE_ROOT / "app/gui/app"),), + ), + CMakeSourceBlock( + variable="YAZE_AGENT_SOURCES", + cmake_path=SOURCE_ROOT / "cli/agent.cmake", + directories=( + DirectorySpec(SOURCE_ROOT / "cli", recursive=False), # For flags.cc + DirectorySpec(SOURCE_ROOT / "cli/service"), + DirectorySpec(SOURCE_ROOT / "cli/handlers"), + ), + exclude={ + Path("cli/cli.cc"), # Part of z3ed executable + Path("cli/cli_main.cc"), # Part of z3ed executable + }, + ), + CMakeSourceBlock( + variable="YAZE_TEST_SOURCES", + cmake_path=SOURCE_ROOT / "app/test/test.cmake", + directories=(DirectorySpec(SOURCE_ROOT / "app/test"),), + ), ) @@ -113,16 +287,79 @@ def parse_entry(line: str) -> Optional[str]: return stripped -def gather_expected_sources(block: CMakeSourceBlock) -> List[str]: +def extract_conditional_files(cmake_path: Path, variable: str) -> Set[str]: + """Extract files that are added to the variable via conditional blocks (if/endif).""" + conditional_files: Set[str] = set() + + try: + lines = cmake_path.read_text(encoding='utf-8').splitlines() + except Exception: + return conditional_files + + in_conditional = False + conditional_depth = 0 + + for i, line in enumerate(lines): + stripped = line.strip() + + # Track if/endif blocks + if stripped.startswith('if(') or stripped.startswith('if '): + if in_conditional: + conditional_depth += 1 + else: + in_conditional = True + conditional_depth = 0 + elif stripped.startswith('endif(') or stripped == 'endif()': + if conditional_depth > 0: + conditional_depth -= 1 + else: + in_conditional = False + + # Check if this line appends to our variable + if in_conditional and f'APPEND {variable}' in line: + # Handle single-line list(APPEND VAR file.cc) + if ')' in line: + # Extract file from same line + match = re.search(rf'APPEND\s+{re.escape(variable)}\s+(.+?)\)', line) + if match: + file_str = match.group(1).strip() + # Can be multiple files separated by space + for f in file_str.split(): + f = f.strip() + if f and not f.startswith('$') and '/' in f and f.endswith('.cc'): + conditional_files.add(f) + else: + # Multi-line list(APPEND) - extract files from following lines + j = i + 1 + while j < len(lines) and not lines[j].strip().startswith(')'): + entry = parse_entry(lines[j]) + if entry: + conditional_files.add(entry) + j += 1 + + return conditional_files + + +def gather_expected_sources(block: CMakeSourceBlock, gitignore_spec: Any = None) -> List[str]: + # First, find files that are in conditional blocks + conditional_files = extract_conditional_files(block.cmake_path, block.variable) + entries: Set[str] = set() for directory in block.directories: for source_file in directory.iter_files(): if should_ignore_path(source_file): continue + if is_ignored(source_file, gitignore_spec): + continue rel_path = relative_to_source(source_file) if rel_path in block.exclude: continue - entries.add(str(rel_path).replace("\\", "/")) + + # Exclude files that are in conditional blocks + rel_path_str = str(rel_path).replace("\\", "/") + if rel_path_str not in conditional_files: + entries.add(rel_path_str) + return sorted(entries) @@ -135,7 +372,118 @@ def should_ignore_path(path: Path) -> bool: return BUILD_CLEANER_IGNORE_TOKEN in head -def update_cmake_block(block: CMakeSourceBlock, dry_run: bool) -> bool: +def extract_includes(file_path: Path) -> Set[str]: + """Extract all #include statements from a source file.""" + includes = set() + try: + with file_path.open('r', encoding='utf-8') as f: + for line in f: + # Match #include "..." or #include <...> + match = re.match(r'^\s*#include\s+[<"]([^>"]+)[>"]', line) + if match: + includes.add(match.group(1)) + except (OSError, UnicodeDecodeError): + pass + return includes + + +def extract_symbols(file_path: Path) -> Set[str]: + """Extract potential symbols/identifiers that might need headers.""" + symbols = set() + try: + with file_path.open('r', encoding='utf-8') as f: + content = f.read() + + # Find namespace-qualified symbols (e.g., std::, absl::, ImGui::) + namespace_symbols = re.findall(r'\b([a-zA-Z_]\w*::)', content) + symbols.update(namespace_symbols) + + # Find common function calls that might need headers + func_calls = re.findall(r'\b([A-Z][a-zA-Z0-9_]*)\s*\(', content) + symbols.update(func_calls) + + except (OSError, UnicodeDecodeError): + pass + return symbols + + +def find_missing_headers(source: Path) -> List[str]: + """Analyze a source file and suggest missing headers based on symbol usage.""" + if should_ignore_path(source): + return [] + + current_includes = extract_includes(source) + symbols = extract_symbols(source) + missing = [] + + # Check for common headers based on symbol prefixes + for symbol_prefix, headers in COMMON_HEADERS.items(): + if any(symbol_prefix in sym for sym in symbols): + for header in headers: + # Extract just the header name from angle brackets + header_name = header.strip('<>') + if header_name not in ' '.join(current_includes): + missing.append(header) + + return missing + + +def find_conditional_blocks_after(cmake_lines: List[str], end_idx: int, variable: str) -> List[str]: + """ + Find conditional blocks (if/endif) that append to the variable after the main set() block. + Returns lines that should be preserved. + """ + conditional_lines = [] + idx = end_idx + 1 + + while idx < len(cmake_lines): + line = cmake_lines[idx] + stripped = line.strip() + + # Stop at next major block or empty lines + if not stripped: + idx += 1 + continue + + # Check if this is a conditional that appends to our variable + if stripped.startswith('if(') or stripped.startswith('if ('): + # Look ahead to see if this block modifies our variable + block_start = idx + block_depth = 1 + modifies_var = False + + temp_idx = idx + 1 + while temp_idx < len(cmake_lines) and block_depth > 0: + temp_line = cmake_lines[temp_idx].strip() + if temp_line.startswith('if(') or temp_line.startswith('if '): + block_depth += 1 + elif temp_line.startswith('endif(') or temp_line == 'endif()': + block_depth -= 1 + + # Check if this block modifies our variable + if f'APPEND {variable}' in temp_line or f'APPEND\n {variable}' in cmake_lines[temp_idx]: + modifies_var = True + + temp_idx += 1 + + if modifies_var: + # Include the entire conditional block + conditional_lines.extend(cmake_lines[block_start:temp_idx]) + idx = temp_idx + continue + else: + # This conditional doesn't touch our variable, stop scanning + break + else: + # Hit something else, stop scanning + break + + idx += 1 + + return conditional_lines + + +def update_cmake_block(block: CMakeSourceBlock, dry_run: bool, gitignore_spec: Any = None) -> bool: cmake_lines = (block.cmake_path.read_text(encoding="utf-8")).splitlines() pattern = re.compile(rf"\s*set\(\s*{re.escape(block.variable)}\b") @@ -195,7 +543,7 @@ def update_cmake_block(block: CMakeSourceBlock, dry_run: bool) -> bool: else: postlude.append(line) - expected_entries = gather_expected_sources(block) + expected_entries = gather_expected_sources(block, gitignore_spec) expected_set = set(expected_entries) if set(existing_entries) == expected_set: @@ -209,12 +557,12 @@ def update_cmake_block(block: CMakeSourceBlock, dry_run: bool) -> bool: rebuilt_block = prelude + [f"{indent}{entry}" for entry in expected_entries] + postlude if dry_run: - print(f"[DRY-RUN] Would update {block.cmake_path.relative_to(PROJECT_ROOT)}") + print(f"[DRY-RUN] Would update {block.cmake_path.relative_to(PROJECT_ROOT)} :: {block.variable}") return True cmake_lines[start_idx + 1 : end_idx] = rebuilt_block block.cmake_path.write_text("\n".join(cmake_lines) + "\n", encoding="utf-8") - print(f"Updated {block.cmake_path.relative_to(PROJECT_ROOT)}") + print(f"Updated {block.cmake_path.relative_to(PROJECT_ROOT)} :: {block.variable}") missing = sorted(expected_set - set(existing_entries)) removed = sorted(set(existing_entries) - expected_set) if missing: @@ -307,9 +655,47 @@ def ensure_self_header_include(source: Path, dry_run: bool) -> bool: return True -def collect_source_files() -> Set[Path]: +def add_missing_headers(source: Path, dry_run: bool, iwyu_mode: bool) -> bool: + """Add missing headers based on IWYU-style analysis.""" + if not iwyu_mode or should_ignore_path(source): + return False + + missing_headers = find_missing_headers(source) + if not missing_headers: + return False + + try: + lines = source.read_text(encoding="utf-8").splitlines() + except UnicodeDecodeError: + return False + + # Find where to insert the headers + insert_idx = find_insert_index(lines) + + # Move past any existing includes to add new ones after them + while insert_idx < len(lines) and lines[insert_idx].strip().startswith('#include'): + insert_idx += 1 + + # Insert missing headers + for header in missing_headers: + lines.insert(insert_idx, f'#include {header}') + insert_idx += 1 + + if dry_run: + rel = source.relative_to(PROJECT_ROOT) + print(f"[DRY-RUN] Would add missing headers to {rel}: {', '.join(missing_headers)}") + return True + + source.write_text("\n".join(lines) + "\n", encoding="utf-8") + print(f"Added missing headers to {source.relative_to(PROJECT_ROOT)}: {', '.join(missing_headers)}") + return True + + +def collect_source_files(config: List[CMakeSourceBlock], gitignore_spec: Any = None) -> Set[Path]: + """Collect all source files from the given configuration, respecting .gitignore patterns.""" managed_dirs: Set[Path] = set() - for block in CONFIG: + + for block in config: for directory in block.directories: managed_dirs.add(directory.path) @@ -319,42 +705,119 @@ def collect_source_files() -> Set[Path]: continue for file_path in directory.rglob("*"): if file_path.is_file() and file_path.suffix in SUPPORTED_EXTENSIONS: - result.add(file_path) + if not is_ignored(file_path, gitignore_spec): + result.add(file_path) return result -def run(dry_run: bool, cmake_only: bool, includes_only: bool) -> int: +def get_config(auto_discover: bool = False) -> List[CMakeSourceBlock]: + """Get the full configuration, optionally including auto-discovered libraries.""" + # Always start with static config (all known libraries) + config = list(STATIC_CONFIG) + + # Optionally add auto-discovered libraries, avoiding duplicates + if auto_discover: + discovered = discover_cmake_libraries() + static_vars = {block.variable for block in STATIC_CONFIG} + + for block in discovered: + if block.variable not in static_vars: + config.append(block) + print(f" Auto-discovered: {block.variable} in {block.cmake_path.name}") + + return config + + +def run(dry_run: bool, cmake_only: bool, includes_only: bool, iwyu_mode: bool, auto_discover: bool) -> int: if cmake_only and includes_only: raise ValueError("Cannot use --cmake-only and --includes-only together") + # Load .gitignore patterns + gitignore_spec = load_gitignore() + if gitignore_spec: + print("✓ Loaded .gitignore patterns") + changed = False + + # Get configuration (all libraries by default, with optional auto-discovery) + config = get_config(auto_discover) + + if auto_discover: + print(f"✓ Using {len(config)} library configurations (with auto-discovery)") + else: + print(f"✓ Using {len(config)} library configurations") if not includes_only: - for block in CONFIG: - changed |= update_cmake_block(block, dry_run) + print("\n📋 Updating CMake source lists...") + for block in config: + changed |= update_cmake_block(block, dry_run, gitignore_spec) if not cmake_only: - for source in collect_source_files(): + print("\n📝 Checking self-header includes...") + source_files = collect_source_files(config, gitignore_spec) + print(f" Scanning {len(source_files)} source files") + + for source in source_files: changed |= ensure_self_header_include(source, dry_run) + + if iwyu_mode: + print("\n🔍 Running IWYU-style header analysis...") + for source in source_files: + changed |= add_missing_headers(source, dry_run, iwyu_mode) if dry_run and not changed: - print("No changes required (dry-run)") - if not dry_run and not changed: - print("No changes required") + print("\n✅ No changes required (dry-run)") + elif not dry_run and not changed: + print("\n✅ No changes required") + elif dry_run: + print("\n✅ Dry-run complete - use without --dry-run to apply changes") + else: + print("\n✅ All changes applied successfully") + return 0 def main() -> int: - parser = argparse.ArgumentParser(description="Maintain CMake source lists and self-header includes.") - parser.add_argument("--dry-run", action="store_true", help="Report prospective changes without editing files") - parser.add_argument("--cmake-only", action="store_true", help="Only update CMake source lists") - parser.add_argument("--includes-only", action="store_true", help="Only ensure self-header includes") + parser = argparse.ArgumentParser( + description="Maintain CMake source lists and ensure proper header includes (IWYU-style).", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" +Examples: + # Dry-run to see what would change: + %(prog)s --dry-run + + # Auto-discover libraries and update CMake files: + %(prog)s --auto-discover + + # Run IWYU-style header analysis: + %(prog)s --iwyu + + # Update only CMake source lists: + %(prog)s --cmake-only + + # Update only header includes: + %(prog)s --includes-only + """ + ) + parser.add_argument("--dry-run", action="store_true", + help="Report prospective changes without editing files") + parser.add_argument("--cmake-only", action="store_true", + help="Only update CMake source lists") + parser.add_argument("--includes-only", action="store_true", + help="Only ensure self-header includes") + parser.add_argument("--iwyu", action="store_true", + help="Run IWYU-style analysis to add missing headers") + parser.add_argument("--auto-discover", action="store_true", + help="Auto-discover CMake library files (*.cmake, *_library.cmake)") args = parser.parse_args() try: - return run(args.dry_run, args.cmake_only, args.includes_only) + return run(args.dry_run, args.cmake_only, args.includes_only, args.iwyu, args.auto_discover) except Exception as exc: # pylint: disable=broad-except - print(f"build_cleaner failed: {exc}") + import traceback + print(f"❌ build_cleaner failed: {exc}") + if args.dry_run: # Show traceback in dry-run mode for debugging + traceback.print_exc() return 1 diff --git a/scripts/requirements.txt b/scripts/requirements.txt new file mode 100644 index 00000000..00b26e3c --- /dev/null +++ b/scripts/requirements.txt @@ -0,0 +1,6 @@ +# Python dependencies for YAZE build scripts +# Install with: pip3 install -r scripts/requirements.txt + +# For .gitignore pattern matching in build_cleaner.py +pathspec>=0.11.0 + diff --git a/src/app/core/controller.cc b/src/app/core/controller.cc index 5a5c39e4..a903dfdb 100644 --- a/src/app/core/controller.cc +++ b/src/app/core/controller.cc @@ -8,7 +8,7 @@ #include "app/core/timing.h" #include "app/core/window.h" #include "app/editor/editor_manager.h" -#include "app/gui/app/background_renderer.h" +#include "app/gui/core/background_renderer.h" #include "app/gfx/resource/arena.h" // Add include for Arena #include "app/gfx/backend/sdl2_renderer.h" // Add include for new renderer #include "app/gui/core/theme_manager.h" diff --git a/src/app/core/core_library.cmake b/src/app/core/core_library.cmake index 1be51b74..e7c38e47 100644 --- a/src/app/core/core_library.cmake +++ b/src/app/core/core_library.cmake @@ -111,10 +111,6 @@ if(WIN32 OR (UNIX AND NOT APPLE)) target_include_directories(yaze_core_lib PUBLIC ${CMAKE_SOURCE_DIR}/src/lib/nativefiledialog-extended/src/include) endif() -target_sources(yaze_core_lib PRIVATE - ${CMAKE_SOURCE_DIR}/src/cli/service/testing/test_workflow_generator.cc -) - if(YAZE_WITH_GRPC) target_include_directories(yaze_core_lib PRIVATE ${CMAKE_SOURCE_DIR}/third_party/json/include) diff --git a/src/app/editor/editor_library.cmake b/src/app/editor/editor_library.cmake index 1f8bf0f2..7813fed4 100644 --- a/src/app/editor/editor_library.cmake +++ b/src/app/editor/editor_library.cmake @@ -1,13 +1,6 @@ set( YAZE_APP_EDITOR_SRC app/editor/agent/agent_chat_history_codec.cc - app/editor/agent/agent_chat_history_popup.cc - app/editor/agent/agent_chat_widget.cc - app/editor/agent/agent_collaboration_coordinator.cc - app/editor/agent/agent_editor.cc - app/editor/agent/agent_ui_theme.cc - app/editor/agent/automation_bridge.cc - app/editor/agent/network_collaboration_coordinator.cc app/editor/code/assembly_editor.cc app/editor/code/memory_editor.cc app/editor/code/project_file_editor.cc @@ -24,8 +17,6 @@ set( app/editor/graphics/gfx_group_editor.cc app/editor/graphics/graphics_editor.cc app/editor/graphics/screen_editor.cc - app/editor/palette/palette_editor.cc - app/editor/palette/palette_group_card.cc app/editor/message/message_data.cc app/editor/message/message_editor.cc app/editor/message/message_preview.cc @@ -36,6 +27,9 @@ set( app/editor/overworld/overworld_entity_renderer.cc app/editor/overworld/scratch_space.cc app/editor/overworld/tile16_editor.cc + app/editor/palette/palette_editor.cc + app/editor/palette/palette_group_card.cc + app/editor/palette/palette_utility.cc app/editor/sprite/sprite_editor.cc app/editor/system/command_manager.cc app/editor/system/command_palette.cc diff --git a/src/app/editor/editor_manager.cc b/src/app/editor/editor_manager.cc index 86be55a2..8c498370 100644 --- a/src/app/editor/editor_manager.cc +++ b/src/app/editor/editor_manager.cc @@ -30,7 +30,7 @@ #include "app/emu/emulator.h" #include "app/gfx/resource/arena.h" #include "app/gfx/debug/performance/performance_profiler.h" -#include "app/gui/app/background_renderer.h" +#include "app/gui/core/background_renderer.h" #include "app/gui/core/icons.h" #include "app/gui/core/input.h" #include "app/gui/core/style.h" diff --git a/src/app/gfx/gfx_library.cmake b/src/app/gfx/gfx_library.cmake index 1ac80236..0e6c00da 100644 --- a/src/app/gfx/gfx_library.cmake +++ b/src/app/gfx/gfx_library.cmake @@ -75,7 +75,6 @@ message(STATUS " - GFX Tier: gfx_backend configured") set(GFX_RESOURCE_SRC app/gfx/resource/arena.cc app/gfx/resource/memory_pool.cc - app/gfx/render/background_buffer.cc ) add_library(yaze_gfx_resource STATIC ${GFX_RESOURCE_SRC}) configure_gfx_library(yaze_gfx_resource) @@ -121,6 +120,7 @@ message(STATUS " - GFX Tier: gfx_util configured") # ============================================================================== set(GFX_RENDER_SRC app/gfx/render/atlas_renderer.cc + app/gfx/render/background_buffer.cc app/gfx/render/texture_atlas.cc app/gfx/render/tilemap.cc ) @@ -174,4 +174,4 @@ if(PNG_FOUND) target_link_libraries(yaze_gfx INTERFACE ${PNG_LIBRARIES}) endif() -message(STATUS "✓ yaze_gfx library configured with tiered architecture") \ No newline at end of file +message(STATUS "✓ yaze_gfx library configured with tiered architecture") diff --git a/src/app/gui/app/background_renderer.cc b/src/app/gui/core/background_renderer.cc similarity index 99% rename from src/app/gui/app/background_renderer.cc rename to src/app/gui/core/background_renderer.cc index c2793c1a..c13640b8 100644 --- a/src/app/gui/app/background_renderer.cc +++ b/src/app/gui/core/background_renderer.cc @@ -1,4 +1,4 @@ -#include "app/gui/app/background_renderer.h" +#include "app/gui/core/background_renderer.h" #include #include diff --git a/src/app/gui/app/background_renderer.h b/src/app/gui/core/background_renderer.h similarity index 100% rename from src/app/gui/app/background_renderer.h rename to src/app/gui/core/background_renderer.h diff --git a/src/app/gui/core/style.cc b/src/app/gui/core/style.cc index c5fcd45a..32af6d28 100644 --- a/src/app/gui/core/style.cc +++ b/src/app/gui/core/style.cc @@ -4,7 +4,7 @@ #include "util/file_util.h" #include "app/gui/core/theme_manager.h" -#include "app/gui/app/background_renderer.h" +#include "app/gui/core/background_renderer.h" #include "app/platform/font_loader.h" #include "app/gui/core/color.h" #include "app/gui/core/icons.h" diff --git a/src/app/gui/gui_library.cmake b/src/app/gui/gui_library.cmake index 1eb7337a..1bba3951 100644 --- a/src/app/gui/gui_library.cmake +++ b/src/app/gui/gui_library.cmake @@ -9,6 +9,7 @@ # 1. Define Source Groups for each sub-library set(GUI_CORE_SRC + app/gui/core/background_renderer.cc app/gui/core/color.cc app/gui/core/input.cc app/gui/core/layout_helpers.cc @@ -47,7 +48,6 @@ set(GUI_AUTOMATION_SRC set(GUI_APP_SRC app/gui/app/agent_chat_widget.cc - app/gui/app/background_renderer.cc app/gui/app/collaboration_panel.cc app/gui/app/editor_card_manager.cc app/gui/app/editor_layout.cc diff --git a/src/cli/agent.cmake b/src/cli/agent.cmake index 665667c4..ebc35bae 100644 --- a/src/cli/agent.cmake +++ b/src/cli/agent.cmake @@ -1,63 +1,73 @@ set(YAZE_AGENT_SOURCES # Core infrastructure - cli/service/command_registry.cc - cli/service/agent/proposal_executor.cc + cli/flags.cc + cli/handlers/agent.cc + cli/handlers/agent/common.cc + cli/handlers/agent/conversation_test.cc + cli/handlers/agent/general_commands.cc + cli/handlers/agent/simple_chat_command.cc + cli/handlers/agent/test_commands.cc + cli/handlers/agent/test_common.cc cli/handlers/agent/todo_commands.cc - cli/service/agent/conversational_agent_service.cc - cli/service/agent/simple_chat_session.cc - cli/service/agent/enhanced_tui.cc - cli/service/agent/tool_dispatcher.cc - - # Advanced features - cli/service/agent/learned_knowledge_service.cc - cli/service/agent/todo_manager.cc + cli/handlers/command_handlers.cc + cli/handlers/game/dialogue_commands.cc + cli/handlers/game/dungeon.cc + cli/handlers/game/dungeon_commands.cc + cli/handlers/game/message.cc + cli/handlers/game/message_commands.cc + cli/handlers/game/music_commands.cc + cli/handlers/game/overworld.cc + cli/handlers/game/overworld_commands.cc + cli/handlers/game/overworld_inspect.cc + cli/handlers/graphics/gfx.cc + cli/handlers/graphics/hex_commands.cc + cli/handlers/graphics/palette.cc + cli/handlers/graphics/palette_commands.cc + cli/handlers/graphics/sprite_commands.cc + cli/handlers/net/net_commands.cc + cli/handlers/rom/mock_rom.cc + cli/handlers/rom/project_commands.cc + cli/handlers/rom/rom_commands.cc + cli/handlers/tools/gui_commands.cc + cli/handlers/tools/resource_commands.cc cli/service/agent/advanced_routing.cc cli/service/agent/agent_pretraining.cc + cli/service/agent/conversational_agent_service.cc + cli/service/agent/enhanced_tui.cc + cli/service/agent/learned_knowledge_service.cc + cli/service/agent/prompt_manager.cc + cli/service/agent/proposal_executor.cc + cli/service/agent/simple_chat_session.cc + cli/service/agent/todo_manager.cc + cli/service/agent/tool_dispatcher.cc cli/service/agent/vim_mode.cc - cli/service/ai/ai_service.cc cli/service/ai/ai_action_parser.cc - cli/service/ai/vision_action_refiner.cc cli/service/ai/ai_gui_controller.cc + cli/service/ai/ai_service.cc cli/service/ai/ollama_ai_service.cc cli/service/ai/prompt_builder.cc cli/service/ai/service_factory.cc + cli/service/ai/vision_action_refiner.cc + cli/service/command_registry.cc cli/service/gui/gui_action_generator.cc cli/service/gui/gui_automation_client.cc cli/service/net/z3ed_network_client.cc - cli/handlers/net/net_commands.cc cli/service/planning/policy_evaluator.cc cli/service/planning/proposal_registry.cc cli/service/planning/tile16_proposal_generator.cc - cli/service/resources/resource_catalog.cc - cli/service/resources/resource_context_builder.cc cli/service/resources/command_context.cc cli/service/resources/command_handler.cc - cli/handlers/agent.cc - cli/handlers/command_handlers.cc - cli/handlers/agent/simple_chat_command.cc - cli/handlers/agent/general_commands.cc - cli/handlers/agent/test_commands.cc - cli/handlers/agent/conversation_test.cc - cli/handlers/agent/common.cc - cli/handlers/game/overworld_inspect.cc - cli/handlers/game/message.cc - cli/handlers/rom/mock_rom.cc - # CommandHandler-based implementations - cli/handlers/tools/resource_commands.cc - cli/handlers/game/dungeon_commands.cc - cli/handlers/game/overworld_commands.cc - cli/handlers/tools/gui_commands.cc - cli/handlers/graphics/hex_commands.cc - cli/handlers/game/dialogue_commands.cc - cli/handlers/game/music_commands.cc - cli/handlers/graphics/palette_commands.cc - cli/handlers/game/message_commands.cc - cli/handlers/graphics/sprite_commands.cc - # ROM commands - cli/handlers/rom/rom_commands.cc - cli/handlers/rom/project_commands.cc - cli/flags.cc + cli/service/resources/resource_catalog.cc + cli/service/resources/resource_context_builder.cc cli/service/rom/rom_sandbox_manager.cc + cli/service/testing/test_suite_loader.cc + cli/service/testing/test_suite_reporter.cc + cli/service/testing/test_suite_writer.cc + cli/service/testing/test_workflow_generator.cc + + # Advanced features + # CommandHandler-based implementations + # ROM commands ) # gRPC-dependent sources (only added when gRPC is enabled) diff --git a/src/zelda3/zelda3_library.cmake b/src/zelda3/zelda3_library.cmake index e7a130ce..7db307cc 100644 --- a/src/zelda3/zelda3_library.cmake +++ b/src/zelda3/zelda3_library.cmake @@ -10,6 +10,7 @@ set( zelda3/music/tracker.cc zelda3/overworld/overworld.cc zelda3/overworld/overworld_map.cc + zelda3/palette_constants.cc zelda3/screen/dungeon_map.cc zelda3/screen/inventory.cc zelda3/screen/title_screen.cc