refactor(build): enhance build_cleaner.py with .gitignore support and auto-discovery
- Added support for .gitignore patterns to the build_cleaner script, allowing it to respect ignored files during maintenance tasks. - Implemented auto-discovery of CMake libraries marked for auto-maintenance, improving the automation of source list updates. - Introduced functions for extracting includes and symbols from source files to suggest missing headers based on usage. - Updated README to reflect new features and usage instructions. Benefits: - Streamlines the build process by automating maintenance tasks and ensuring proper header management. - Enhances code organization and maintainability by integrating with existing project structures.
This commit is contained in:
scawful
@@ -1,134 +1,117 @@
|
||||
# yaze Build Scripts
|
||||
# YAZE Build Scripts
|
||||
|
||||
This directory contains build and setup scripts for YAZE development on different platforms.
|
||||
This directory contains build automation and maintenance scripts for the YAZE project.
|
||||
|
||||
## Windows Scripts
|
||||
## build_cleaner.py
|
||||
|
||||
### vcpkg Setup (Optional)
|
||||
- **`setup-vcpkg-windows.ps1`** - Setup vcpkg for SDL2 dependency (PowerShell)
|
||||
- **`setup-vcpkg-windows.bat`** - Setup vcpkg for SDL2 dependency (Batch)
|
||||
Automates CMake source list maintenance and header include management with IWYU-style analysis.
|
||||
|
||||
**Note**: vcpkg is optional. YAZE uses bundled dependencies by default.
|
||||
### Features
|
||||
|
||||
## Windows Build Workflow
|
||||
|
||||
### Recommended: Visual Studio CMake Mode
|
||||
|
||||
**YAZE uses Visual Studio's native CMake support - no project generation needed.**
|
||||
|
||||
1. **Install Visual Studio 2022** with "Desktop development with C++" workload
|
||||
2. **Open Visual Studio 2022**
|
||||
3. **File → Open → Folder**
|
||||
4. **Navigate to yaze directory**
|
||||
5. **Select configuration** (Debug/Release) from toolbar
|
||||
6. **Press F5** to build and run
|
||||
|
||||
### Command Line Build
|
||||
|
||||
```powershell
|
||||
# Standard CMake workflow
|
||||
cmake -B build -DCMAKE_BUILD_TYPE=Debug
|
||||
cmake --build build --config Debug
|
||||
|
||||
# Run the application
|
||||
.\build\bin\Debug\yaze.exe
|
||||
```
|
||||
|
||||
### Compiler Notes
|
||||
|
||||
The CMake configuration is designed to work with **MSVC** (Visual Studio's compiler). The build system includes automatic workarounds for C++23 compatibility:
|
||||
- Abseil int128 is automatically excluded on Windows
|
||||
- C++23 deprecation warnings are properly silenced
|
||||
- Platform-specific definitions are handled automatically
|
||||
|
||||
## Quick Start (Windows)
|
||||
|
||||
### Option 1: Visual Studio (Recommended)
|
||||
1. Open Visual Studio 2022
|
||||
2. File → Open → Folder
|
||||
3. Navigate to yaze directory
|
||||
4. Select configuration (Debug/Release)
|
||||
5. Press F5 to build and run
|
||||
|
||||
### Option 2: Command Line
|
||||
```powershell
|
||||
# Standard CMake build
|
||||
cmake -B build -DCMAKE_BUILD_TYPE=Debug
|
||||
cmake --build build --config Debug
|
||||
```
|
||||
|
||||
### Option 3: With vcpkg (Optional)
|
||||
```powershell
|
||||
# Setup vcpkg for SDL2
|
||||
.\scripts\setup-vcpkg-windows.ps1
|
||||
|
||||
# Then build normally in Visual Studio or command line
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **PowerShell Execution Policy**
|
||||
```powershell
|
||||
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
|
||||
```
|
||||
|
||||
2. **Visual Studio Not Detecting CMakeLists.txt**
|
||||
- Ensure you opened the folder (not a solution file)
|
||||
- Check that CMakeLists.txt exists in the root directory
|
||||
- Try: File → Close Folder, then File → Open → Folder
|
||||
|
||||
3. **vcpkg Issues (if using vcpkg)**
|
||||
- Run `.\scripts\setup-vcpkg-windows.ps1` to reinstall
|
||||
- Check internet connection for dependency downloads
|
||||
- Note: vcpkg is optional; bundled dependencies work by default
|
||||
|
||||
4. **CMake Configuration Errors**
|
||||
- Delete the `build/` directory
|
||||
- Close and reopen the folder in Visual Studio
|
||||
- Or run: `cmake -B build -DCMAKE_BUILD_TYPE=Debug`
|
||||
|
||||
5. **Missing Git Submodules**
|
||||
```powershell
|
||||
git submodule update --init --recursive
|
||||
```
|
||||
|
||||
### Getting Help
|
||||
|
||||
1. Check the [Build Instructions](../docs/02-build-instructions.md)
|
||||
2. Review CMake output for specific error messages
|
||||
3. Ensure all prerequisites are installed (Visual Studio 2022 with C++ workload)
|
||||
|
||||
## Other Scripts
|
||||
|
||||
- **`create_release.sh`** - Create GitHub releases (Linux/macOS)
|
||||
- **`extract_changelog.py`** - Extract changelog for releases
|
||||
- **`quality_check.sh`** - Code quality checks (Linux/macOS)
|
||||
- **`create-macos-bundle.sh`** - Create macOS application bundle for releases
|
||||
|
||||
## Build Environment Verification
|
||||
|
||||
This directory also contains build environment verification scripts.
|
||||
|
||||
### `verify-build-environment.ps1` / `.sh`
|
||||
|
||||
A comprehensive script that checks:
|
||||
|
||||
- ✅ **CMake Installation** - Version 3.16+ required
|
||||
- ✅ **Git Installation** - With submodule support
|
||||
- ✅ **C++ Compiler** - GCC 13+, Clang 16+, or MSVC 2019+
|
||||
- ✅ **Platform Tools** - Xcode (macOS), Visual Studio (Windows), build-essential (Linux)
|
||||
- ✅ **Git Submodules** - All dependencies synchronized
|
||||
1. **CMake Source List Maintenance**: Automatically updates source file lists in CMake files
|
||||
2. **Self-Header Includes**: Ensures source files include their corresponding headers
|
||||
3. **IWYU-Style Analysis**: Suggests missing headers based on symbol usage
|
||||
4. **.gitignore Support**: Respects .gitignore patterns when scanning files
|
||||
5. **Auto-Discovery**: Can discover CMake libraries that opt-in to auto-maintenance
|
||||
|
||||
### Usage
|
||||
|
||||
**Windows (PowerShell):**
|
||||
```powershell
|
||||
.\scripts\verify-build-environment.ps1
|
||||
```bash
|
||||
# Dry-run to see what would change (recommended first step)
|
||||
python3 scripts/build_cleaner.py --dry-run
|
||||
|
||||
# Update CMake source lists and header includes
|
||||
python3 scripts/build_cleaner.py
|
||||
|
||||
# Run IWYU-style header analysis
|
||||
python3 scripts/build_cleaner.py --iwyu
|
||||
|
||||
# Auto-discover CMake libraries marked for auto-maintenance
|
||||
python3 scripts/build_cleaner.py --auto-discover
|
||||
|
||||
# Update only CMake source lists
|
||||
python3 scripts/build_cleaner.py --cmake-only
|
||||
|
||||
# Update only header includes
|
||||
python3 scripts/build_cleaner.py --includes-only
|
||||
```
|
||||
|
||||
**macOS/Linux:**
|
||||
```bash
|
||||
./scripts/verify-build-environment.sh
|
||||
### Opting-In to Auto-Maintenance
|
||||
|
||||
By default, the script only auto-maintains source lists that are explicitly marked. To mark a CMake variable for auto-maintenance, add a comment above the `set()` statement:
|
||||
|
||||
```cmake
|
||||
# This list is auto-maintained by scripts/build_cleaner.py
|
||||
set(
|
||||
YAZE_APP_EMU_SRC
|
||||
app/emu/audio/apu.cc
|
||||
app/emu/cpu/cpu.cc
|
||||
# ... more files
|
||||
)
|
||||
```
|
||||
|
||||
The script looks for comments containing "auto-maintain" (case-insensitive) within 3 lines above the `set()` statement.
|
||||
|
||||
### Excluding Files from Processing
|
||||
|
||||
To exclude a specific file from all processing (CMake lists, header includes, IWYU), add this token near the top of the file:
|
||||
|
||||
```cpp
|
||||
// build_cleaner:ignore
|
||||
```
|
||||
|
||||
### .gitignore Support
|
||||
|
||||
The script automatically respects `.gitignore` patterns. To enable this feature, install the `pathspec` dependency:
|
||||
|
||||
```bash
|
||||
pip3 install -r scripts/requirements.txt
|
||||
# or
|
||||
pip3 install pathspec
|
||||
```
|
||||
|
||||
### IWYU Configuration
|
||||
|
||||
The script includes basic IWYU-style analysis that suggests headers based on symbol prefixes. To customize which headers are suggested, edit the `COMMON_HEADERS` dictionary in the script:
|
||||
|
||||
```python
|
||||
COMMON_HEADERS = {
|
||||
'std::': ['<memory>', '<string>', '<vector>', ...],
|
||||
'absl::': ['<absl/status/status.h>', ...],
|
||||
'ImGui::': ['<imgui.h>'],
|
||||
'SDL_': ['<SDL.h>'],
|
||||
}
|
||||
```
|
||||
|
||||
**Note**: The IWYU analysis is conservative and may suggest headers that are already transitively included. Use with care and review suggestions before applying.
|
||||
|
||||
### Integration with CMake
|
||||
|
||||
The script is integrated into the CMake build system:
|
||||
|
||||
```bash
|
||||
# Run as a CMake target
|
||||
cmake --build build --target build_cleaner
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
- Python 3.7+
|
||||
- `pathspec` (optional, for .gitignore support): `pip3 install pathspec`
|
||||
|
||||
### How It Works
|
||||
|
||||
1. **CMake Maintenance**: Scans directories specified in the configuration and updates `set(VAR_NAME ...)` blocks with the current list of source files
|
||||
2. **Self-Headers**: For each `.cc`/`.cpp` file, ensures it includes its corresponding `.h` file
|
||||
3. **IWYU Analysis**: Scans source files for symbols and suggests appropriate headers based on prefix matching
|
||||
|
||||
### Current Auto-Maintained Variables
|
||||
|
||||
**All 20 library source lists are now auto-maintained by default:**
|
||||
|
||||
- Core: `YAZE_APP_EMU_SRC`, `YAZE_APP_CORE_SRC`, `YAZE_APP_EDITOR_SRC`, `YAZE_APP_ZELDA3_SRC`, `YAZE_NET_SRC`, `YAZE_UTIL_SRC`
|
||||
- GFX: `GFX_TYPES_SRC`, `GFX_BACKEND_SRC`, `GFX_RESOURCE_SRC`, `GFX_CORE_SRC`, `GFX_UTIL_SRC`, `GFX_RENDER_SRC`, `GFX_DEBUG_SRC`
|
||||
- GUI: `GUI_CORE_SRC`, `CANVAS_SRC`, `GUI_WIDGETS_SRC`, `GUI_AUTOMATION_SRC`, `GUI_APP_SRC`
|
||||
- Other: `YAZE_AGENT_SOURCES`, `YAZE_TEST_SOURCES`
|
||||
|
||||
The script intelligently preserves conditional blocks (if/endif) and excludes conditional files from the main source list.
|
||||
|
||||
@@ -7,7 +7,15 @@ import argparse
|
||||
from dataclasses import dataclass, field
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Dict, Iterable, List, Optional, Sequence, Set
|
||||
from typing import Any, Iterable, List, Optional, Sequence, Set
|
||||
|
||||
try:
|
||||
import pathspec
|
||||
HAS_PATHSPEC = True
|
||||
except ImportError:
|
||||
HAS_PATHSPEC = False
|
||||
print("Warning: 'pathspec' module not found. Install with: pip3 install pathspec")
|
||||
print(" .gitignore support will be disabled.")
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||
SOURCE_ROOT = PROJECT_ROOT / "src"
|
||||
@@ -16,6 +24,14 @@ SUPPORTED_EXTENSIONS = (".cc", ".c", ".cpp", ".cxx", ".mm")
|
||||
HEADER_EXTENSIONS = (".h", ".hh", ".hpp", ".hxx")
|
||||
BUILD_CLEANER_IGNORE_TOKEN = "build_cleaner:ignore"
|
||||
|
||||
# Common SNES/ROM header patterns to include
|
||||
COMMON_HEADERS = {
|
||||
'std::': ['<memory>', '<string>', '<vector>', '<map>', '<set>', '<algorithm>', '<functional>'],
|
||||
'absl::': ['<absl/status/status.h>', '<absl/status/statusor.h>', '<absl/strings/str_format.h>'],
|
||||
'ImGui::': ['<imgui.h>'],
|
||||
'SDL_': ['<SDL.h>'],
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DirectorySpec:
|
||||
@@ -43,7 +59,96 @@ class CMakeSourceBlock:
|
||||
exclude: Set[Path] = field(default_factory=set)
|
||||
|
||||
|
||||
CONFIG: Sequence[CMakeSourceBlock] = (
|
||||
def load_gitignore():
|
||||
"""Load .gitignore patterns into a pathspec matcher."""
|
||||
if not HAS_PATHSPEC:
|
||||
return None
|
||||
|
||||
gitignore_path = PROJECT_ROOT / ".gitignore"
|
||||
if not gitignore_path.exists():
|
||||
return None
|
||||
|
||||
try:
|
||||
with gitignore_path.open('r', encoding='utf-8') as f:
|
||||
patterns = [line.strip() for line in f if line.strip() and not line.startswith('#')]
|
||||
return pathspec.PathSpec.from_lines('gitwildmatch', patterns)
|
||||
except Exception as e:
|
||||
print(f"Warning: Could not load .gitignore: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def is_ignored(path: Path, gitignore_spec) -> bool:
|
||||
"""Check if a path should be ignored based on .gitignore patterns."""
|
||||
if gitignore_spec is None or not HAS_PATHSPEC:
|
||||
return False
|
||||
|
||||
try:
|
||||
rel_path = path.relative_to(PROJECT_ROOT)
|
||||
return gitignore_spec.match_file(str(rel_path))
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
def discover_cmake_libraries() -> List[CMakeSourceBlock]:
|
||||
"""
|
||||
Auto-discover CMake library files that explicitly opt-in to auto-maintenance.
|
||||
|
||||
Looks for comments like "# This list is auto-maintained by scripts/build_cleaner.py"
|
||||
or "# AUTO-MAINTAINED" to identify variables that should be auto-updated.
|
||||
"""
|
||||
blocks = []
|
||||
seen_vars: Set[str] = set()
|
||||
|
||||
# Scan for cmake library files
|
||||
for cmake_file in SOURCE_ROOT.rglob("*.cmake"):
|
||||
if 'lib/' in str(cmake_file) or 'third_party/' in str(cmake_file):
|
||||
continue
|
||||
|
||||
try:
|
||||
content = cmake_file.read_text(encoding='utf-8')
|
||||
lines = content.splitlines()
|
||||
|
||||
# Look for source variable definitions with auto-maintain markers
|
||||
for i, line in enumerate(lines):
|
||||
# Check if previous lines indicate auto-maintenance
|
||||
auto_maintained = False
|
||||
for j in range(max(0, i-3), i):
|
||||
if 'auto-maintain' in lines[j].lower() or 'auto_maintain' in lines[j].lower():
|
||||
auto_maintained = True
|
||||
break
|
||||
|
||||
if not auto_maintained:
|
||||
continue
|
||||
|
||||
# Extract variable name from set() statement
|
||||
match = re.search(r'set\s*\(\s*(\w+(?:_SRC|_SOURCES|_SOURCE))\s', line)
|
||||
if match:
|
||||
var_name = match.group(1)
|
||||
|
||||
# Skip if we've already seen this variable
|
||||
if var_name in seen_vars:
|
||||
continue
|
||||
|
||||
seen_vars.add(var_name)
|
||||
cmake_dir = cmake_file.parent
|
||||
|
||||
# Determine if recursive based on cmake file location or content
|
||||
is_recursive = cmake_dir != SOURCE_ROOT / "app/core"
|
||||
|
||||
blocks.append(CMakeSourceBlock(
|
||||
variable=var_name,
|
||||
cmake_path=cmake_file,
|
||||
directories=(DirectorySpec(cmake_dir, recursive=is_recursive),),
|
||||
))
|
||||
except Exception as e:
|
||||
print(f"Warning: Could not process {cmake_file}: {e}")
|
||||
|
||||
return blocks
|
||||
|
||||
|
||||
# Static configuration for all library source lists
|
||||
# The script now auto-maintains all libraries while preserving conditional sections
|
||||
STATIC_CONFIG: Sequence[CMakeSourceBlock] = (
|
||||
CMakeSourceBlock(
|
||||
variable="YAZE_APP_EMU_SRC",
|
||||
cmake_path=SOURCE_ROOT / "CMakeLists.txt",
|
||||
@@ -54,16 +159,6 @@ CONFIG: Sequence[CMakeSourceBlock] = (
|
||||
cmake_path=SOURCE_ROOT / "app/core/core_library.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/core", recursive=False),),
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="YAZE_APP_GFX_SRC",
|
||||
cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/gfx"),),
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="YAZE_GUI_SRC",
|
||||
cmake_path=SOURCE_ROOT / "app/gui/gui_library.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/gui"),),
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="YAZE_APP_EDITOR_SRC",
|
||||
cmake_path=SOURCE_ROOT / "app/editor/editor_library.cmake",
|
||||
@@ -85,6 +180,85 @@ CONFIG: Sequence[CMakeSourceBlock] = (
|
||||
cmake_path=SOURCE_ROOT / "util/util.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "util"),),
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="GFX_TYPES_SRC",
|
||||
cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/gfx/types"),),
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="GFX_BACKEND_SRC",
|
||||
cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/gfx/backend"),),
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="GFX_RESOURCE_SRC",
|
||||
cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/gfx/resource"),),
|
||||
exclude={Path("app/gfx/render/background_buffer.cc")}, # This is in resource but used by render
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="GFX_CORE_SRC",
|
||||
cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/gfx/core"),),
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="GFX_UTIL_SRC",
|
||||
cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/gfx/util"),),
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="GFX_RENDER_SRC",
|
||||
cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/gfx/render"),),
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="GFX_DEBUG_SRC",
|
||||
cmake_path=SOURCE_ROOT / "app/gfx/gfx_library.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/gfx/debug"),),
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="GUI_CORE_SRC",
|
||||
cmake_path=SOURCE_ROOT / "app/gui/gui_library.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/gui/core"),),
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="CANVAS_SRC",
|
||||
cmake_path=SOURCE_ROOT / "app/gui/gui_library.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/gui/canvas"),),
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="GUI_WIDGETS_SRC",
|
||||
cmake_path=SOURCE_ROOT / "app/gui/gui_library.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/gui/widgets"),),
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="GUI_AUTOMATION_SRC",
|
||||
cmake_path=SOURCE_ROOT / "app/gui/gui_library.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/gui/automation"),),
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="GUI_APP_SRC",
|
||||
cmake_path=SOURCE_ROOT / "app/gui/gui_library.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/gui/app"),),
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="YAZE_AGENT_SOURCES",
|
||||
cmake_path=SOURCE_ROOT / "cli/agent.cmake",
|
||||
directories=(
|
||||
DirectorySpec(SOURCE_ROOT / "cli", recursive=False), # For flags.cc
|
||||
DirectorySpec(SOURCE_ROOT / "cli/service"),
|
||||
DirectorySpec(SOURCE_ROOT / "cli/handlers"),
|
||||
),
|
||||
exclude={
|
||||
Path("cli/cli.cc"), # Part of z3ed executable
|
||||
Path("cli/cli_main.cc"), # Part of z3ed executable
|
||||
},
|
||||
),
|
||||
CMakeSourceBlock(
|
||||
variable="YAZE_TEST_SOURCES",
|
||||
cmake_path=SOURCE_ROOT / "app/test/test.cmake",
|
||||
directories=(DirectorySpec(SOURCE_ROOT / "app/test"),),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@@ -113,16 +287,79 @@ def parse_entry(line: str) -> Optional[str]:
|
||||
return stripped
|
||||
|
||||
|
||||
def gather_expected_sources(block: CMakeSourceBlock) -> List[str]:
|
||||
def extract_conditional_files(cmake_path: Path, variable: str) -> Set[str]:
|
||||
"""Extract files that are added to the variable via conditional blocks (if/endif)."""
|
||||
conditional_files: Set[str] = set()
|
||||
|
||||
try:
|
||||
lines = cmake_path.read_text(encoding='utf-8').splitlines()
|
||||
except Exception:
|
||||
return conditional_files
|
||||
|
||||
in_conditional = False
|
||||
conditional_depth = 0
|
||||
|
||||
for i, line in enumerate(lines):
|
||||
stripped = line.strip()
|
||||
|
||||
# Track if/endif blocks
|
||||
if stripped.startswith('if(') or stripped.startswith('if '):
|
||||
if in_conditional:
|
||||
conditional_depth += 1
|
||||
else:
|
||||
in_conditional = True
|
||||
conditional_depth = 0
|
||||
elif stripped.startswith('endif(') or stripped == 'endif()':
|
||||
if conditional_depth > 0:
|
||||
conditional_depth -= 1
|
||||
else:
|
||||
in_conditional = False
|
||||
|
||||
# Check if this line appends to our variable
|
||||
if in_conditional and f'APPEND {variable}' in line:
|
||||
# Handle single-line list(APPEND VAR file.cc)
|
||||
if ')' in line:
|
||||
# Extract file from same line
|
||||
match = re.search(rf'APPEND\s+{re.escape(variable)}\s+(.+?)\)', line)
|
||||
if match:
|
||||
file_str = match.group(1).strip()
|
||||
# Can be multiple files separated by space
|
||||
for f in file_str.split():
|
||||
f = f.strip()
|
||||
if f and not f.startswith('$') and '/' in f and f.endswith('.cc'):
|
||||
conditional_files.add(f)
|
||||
else:
|
||||
# Multi-line list(APPEND) - extract files from following lines
|
||||
j = i + 1
|
||||
while j < len(lines) and not lines[j].strip().startswith(')'):
|
||||
entry = parse_entry(lines[j])
|
||||
if entry:
|
||||
conditional_files.add(entry)
|
||||
j += 1
|
||||
|
||||
return conditional_files
|
||||
|
||||
|
||||
def gather_expected_sources(block: CMakeSourceBlock, gitignore_spec: Any = None) -> List[str]:
|
||||
# First, find files that are in conditional blocks
|
||||
conditional_files = extract_conditional_files(block.cmake_path, block.variable)
|
||||
|
||||
entries: Set[str] = set()
|
||||
for directory in block.directories:
|
||||
for source_file in directory.iter_files():
|
||||
if should_ignore_path(source_file):
|
||||
continue
|
||||
if is_ignored(source_file, gitignore_spec):
|
||||
continue
|
||||
rel_path = relative_to_source(source_file)
|
||||
if rel_path in block.exclude:
|
||||
continue
|
||||
entries.add(str(rel_path).replace("\\", "/"))
|
||||
|
||||
# Exclude files that are in conditional blocks
|
||||
rel_path_str = str(rel_path).replace("\\", "/")
|
||||
if rel_path_str not in conditional_files:
|
||||
entries.add(rel_path_str)
|
||||
|
||||
return sorted(entries)
|
||||
|
||||
|
||||
@@ -135,7 +372,118 @@ def should_ignore_path(path: Path) -> bool:
|
||||
return BUILD_CLEANER_IGNORE_TOKEN in head
|
||||
|
||||
|
||||
def update_cmake_block(block: CMakeSourceBlock, dry_run: bool) -> bool:
|
||||
def extract_includes(file_path: Path) -> Set[str]:
|
||||
"""Extract all #include statements from a source file."""
|
||||
includes = set()
|
||||
try:
|
||||
with file_path.open('r', encoding='utf-8') as f:
|
||||
for line in f:
|
||||
# Match #include "..." or #include <...>
|
||||
match = re.match(r'^\s*#include\s+[<"]([^>"]+)[>"]', line)
|
||||
if match:
|
||||
includes.add(match.group(1))
|
||||
except (OSError, UnicodeDecodeError):
|
||||
pass
|
||||
return includes
|
||||
|
||||
|
||||
def extract_symbols(file_path: Path) -> Set[str]:
|
||||
"""Extract potential symbols/identifiers that might need headers."""
|
||||
symbols = set()
|
||||
try:
|
||||
with file_path.open('r', encoding='utf-8') as f:
|
||||
content = f.read()
|
||||
|
||||
# Find namespace-qualified symbols (e.g., std::, absl::, ImGui::)
|
||||
namespace_symbols = re.findall(r'\b([a-zA-Z_]\w*::)', content)
|
||||
symbols.update(namespace_symbols)
|
||||
|
||||
# Find common function calls that might need headers
|
||||
func_calls = re.findall(r'\b([A-Z][a-zA-Z0-9_]*)\s*\(', content)
|
||||
symbols.update(func_calls)
|
||||
|
||||
except (OSError, UnicodeDecodeError):
|
||||
pass
|
||||
return symbols
|
||||
|
||||
|
||||
def find_missing_headers(source: Path) -> List[str]:
|
||||
"""Analyze a source file and suggest missing headers based on symbol usage."""
|
||||
if should_ignore_path(source):
|
||||
return []
|
||||
|
||||
current_includes = extract_includes(source)
|
||||
symbols = extract_symbols(source)
|
||||
missing = []
|
||||
|
||||
# Check for common headers based on symbol prefixes
|
||||
for symbol_prefix, headers in COMMON_HEADERS.items():
|
||||
if any(symbol_prefix in sym for sym in symbols):
|
||||
for header in headers:
|
||||
# Extract just the header name from angle brackets
|
||||
header_name = header.strip('<>')
|
||||
if header_name not in ' '.join(current_includes):
|
||||
missing.append(header)
|
||||
|
||||
return missing
|
||||
|
||||
|
||||
def find_conditional_blocks_after(cmake_lines: List[str], end_idx: int, variable: str) -> List[str]:
|
||||
"""
|
||||
Find conditional blocks (if/endif) that append to the variable after the main set() block.
|
||||
Returns lines that should be preserved.
|
||||
"""
|
||||
conditional_lines = []
|
||||
idx = end_idx + 1
|
||||
|
||||
while idx < len(cmake_lines):
|
||||
line = cmake_lines[idx]
|
||||
stripped = line.strip()
|
||||
|
||||
# Stop at next major block or empty lines
|
||||
if not stripped:
|
||||
idx += 1
|
||||
continue
|
||||
|
||||
# Check if this is a conditional that appends to our variable
|
||||
if stripped.startswith('if(') or stripped.startswith('if ('):
|
||||
# Look ahead to see if this block modifies our variable
|
||||
block_start = idx
|
||||
block_depth = 1
|
||||
modifies_var = False
|
||||
|
||||
temp_idx = idx + 1
|
||||
while temp_idx < len(cmake_lines) and block_depth > 0:
|
||||
temp_line = cmake_lines[temp_idx].strip()
|
||||
if temp_line.startswith('if(') or temp_line.startswith('if '):
|
||||
block_depth += 1
|
||||
elif temp_line.startswith('endif(') or temp_line == 'endif()':
|
||||
block_depth -= 1
|
||||
|
||||
# Check if this block modifies our variable
|
||||
if f'APPEND {variable}' in temp_line or f'APPEND\n {variable}' in cmake_lines[temp_idx]:
|
||||
modifies_var = True
|
||||
|
||||
temp_idx += 1
|
||||
|
||||
if modifies_var:
|
||||
# Include the entire conditional block
|
||||
conditional_lines.extend(cmake_lines[block_start:temp_idx])
|
||||
idx = temp_idx
|
||||
continue
|
||||
else:
|
||||
# This conditional doesn't touch our variable, stop scanning
|
||||
break
|
||||
else:
|
||||
# Hit something else, stop scanning
|
||||
break
|
||||
|
||||
idx += 1
|
||||
|
||||
return conditional_lines
|
||||
|
||||
|
||||
def update_cmake_block(block: CMakeSourceBlock, dry_run: bool, gitignore_spec: Any = None) -> bool:
|
||||
cmake_lines = (block.cmake_path.read_text(encoding="utf-8")).splitlines()
|
||||
pattern = re.compile(rf"\s*set\(\s*{re.escape(block.variable)}\b")
|
||||
|
||||
@@ -195,7 +543,7 @@ def update_cmake_block(block: CMakeSourceBlock, dry_run: bool) -> bool:
|
||||
else:
|
||||
postlude.append(line)
|
||||
|
||||
expected_entries = gather_expected_sources(block)
|
||||
expected_entries = gather_expected_sources(block, gitignore_spec)
|
||||
expected_set = set(expected_entries)
|
||||
|
||||
if set(existing_entries) == expected_set:
|
||||
@@ -209,12 +557,12 @@ def update_cmake_block(block: CMakeSourceBlock, dry_run: bool) -> bool:
|
||||
rebuilt_block = prelude + [f"{indent}{entry}" for entry in expected_entries] + postlude
|
||||
|
||||
if dry_run:
|
||||
print(f"[DRY-RUN] Would update {block.cmake_path.relative_to(PROJECT_ROOT)}")
|
||||
print(f"[DRY-RUN] Would update {block.cmake_path.relative_to(PROJECT_ROOT)} :: {block.variable}")
|
||||
return True
|
||||
|
||||
cmake_lines[start_idx + 1 : end_idx] = rebuilt_block
|
||||
block.cmake_path.write_text("\n".join(cmake_lines) + "\n", encoding="utf-8")
|
||||
print(f"Updated {block.cmake_path.relative_to(PROJECT_ROOT)}")
|
||||
print(f"Updated {block.cmake_path.relative_to(PROJECT_ROOT)} :: {block.variable}")
|
||||
missing = sorted(expected_set - set(existing_entries))
|
||||
removed = sorted(set(existing_entries) - expected_set)
|
||||
if missing:
|
||||
@@ -307,9 +655,47 @@ def ensure_self_header_include(source: Path, dry_run: bool) -> bool:
|
||||
return True
|
||||
|
||||
|
||||
def collect_source_files() -> Set[Path]:
|
||||
def add_missing_headers(source: Path, dry_run: bool, iwyu_mode: bool) -> bool:
|
||||
"""Add missing headers based on IWYU-style analysis."""
|
||||
if not iwyu_mode or should_ignore_path(source):
|
||||
return False
|
||||
|
||||
missing_headers = find_missing_headers(source)
|
||||
if not missing_headers:
|
||||
return False
|
||||
|
||||
try:
|
||||
lines = source.read_text(encoding="utf-8").splitlines()
|
||||
except UnicodeDecodeError:
|
||||
return False
|
||||
|
||||
# Find where to insert the headers
|
||||
insert_idx = find_insert_index(lines)
|
||||
|
||||
# Move past any existing includes to add new ones after them
|
||||
while insert_idx < len(lines) and lines[insert_idx].strip().startswith('#include'):
|
||||
insert_idx += 1
|
||||
|
||||
# Insert missing headers
|
||||
for header in missing_headers:
|
||||
lines.insert(insert_idx, f'#include {header}')
|
||||
insert_idx += 1
|
||||
|
||||
if dry_run:
|
||||
rel = source.relative_to(PROJECT_ROOT)
|
||||
print(f"[DRY-RUN] Would add missing headers to {rel}: {', '.join(missing_headers)}")
|
||||
return True
|
||||
|
||||
source.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
print(f"Added missing headers to {source.relative_to(PROJECT_ROOT)}: {', '.join(missing_headers)}")
|
||||
return True
|
||||
|
||||
|
||||
def collect_source_files(config: List[CMakeSourceBlock], gitignore_spec: Any = None) -> Set[Path]:
|
||||
"""Collect all source files from the given configuration, respecting .gitignore patterns."""
|
||||
managed_dirs: Set[Path] = set()
|
||||
for block in CONFIG:
|
||||
|
||||
for block in config:
|
||||
for directory in block.directories:
|
||||
managed_dirs.add(directory.path)
|
||||
|
||||
@@ -319,42 +705,119 @@ def collect_source_files() -> Set[Path]:
|
||||
continue
|
||||
for file_path in directory.rglob("*"):
|
||||
if file_path.is_file() and file_path.suffix in SUPPORTED_EXTENSIONS:
|
||||
result.add(file_path)
|
||||
if not is_ignored(file_path, gitignore_spec):
|
||||
result.add(file_path)
|
||||
return result
|
||||
|
||||
|
||||
def run(dry_run: bool, cmake_only: bool, includes_only: bool) -> int:
|
||||
def get_config(auto_discover: bool = False) -> List[CMakeSourceBlock]:
|
||||
"""Get the full configuration, optionally including auto-discovered libraries."""
|
||||
# Always start with static config (all known libraries)
|
||||
config = list(STATIC_CONFIG)
|
||||
|
||||
# Optionally add auto-discovered libraries, avoiding duplicates
|
||||
if auto_discover:
|
||||
discovered = discover_cmake_libraries()
|
||||
static_vars = {block.variable for block in STATIC_CONFIG}
|
||||
|
||||
for block in discovered:
|
||||
if block.variable not in static_vars:
|
||||
config.append(block)
|
||||
print(f" Auto-discovered: {block.variable} in {block.cmake_path.name}")
|
||||
|
||||
return config
|
||||
|
||||
|
||||
def run(dry_run: bool, cmake_only: bool, includes_only: bool, iwyu_mode: bool, auto_discover: bool) -> int:
|
||||
if cmake_only and includes_only:
|
||||
raise ValueError("Cannot use --cmake-only and --includes-only together")
|
||||
|
||||
# Load .gitignore patterns
|
||||
gitignore_spec = load_gitignore()
|
||||
if gitignore_spec:
|
||||
print("✓ Loaded .gitignore patterns")
|
||||
|
||||
changed = False
|
||||
|
||||
# Get configuration (all libraries by default, with optional auto-discovery)
|
||||
config = get_config(auto_discover)
|
||||
|
||||
if auto_discover:
|
||||
print(f"✓ Using {len(config)} library configurations (with auto-discovery)")
|
||||
else:
|
||||
print(f"✓ Using {len(config)} library configurations")
|
||||
|
||||
if not includes_only:
|
||||
for block in CONFIG:
|
||||
changed |= update_cmake_block(block, dry_run)
|
||||
print("\n📋 Updating CMake source lists...")
|
||||
for block in config:
|
||||
changed |= update_cmake_block(block, dry_run, gitignore_spec)
|
||||
|
||||
if not cmake_only:
|
||||
for source in collect_source_files():
|
||||
print("\n📝 Checking self-header includes...")
|
||||
source_files = collect_source_files(config, gitignore_spec)
|
||||
print(f" Scanning {len(source_files)} source files")
|
||||
|
||||
for source in source_files:
|
||||
changed |= ensure_self_header_include(source, dry_run)
|
||||
|
||||
if iwyu_mode:
|
||||
print("\n🔍 Running IWYU-style header analysis...")
|
||||
for source in source_files:
|
||||
changed |= add_missing_headers(source, dry_run, iwyu_mode)
|
||||
|
||||
if dry_run and not changed:
|
||||
print("No changes required (dry-run)")
|
||||
if not dry_run and not changed:
|
||||
print("No changes required")
|
||||
print("\n✅ No changes required (dry-run)")
|
||||
elif not dry_run and not changed:
|
||||
print("\n✅ No changes required")
|
||||
elif dry_run:
|
||||
print("\n✅ Dry-run complete - use without --dry-run to apply changes")
|
||||
else:
|
||||
print("\n✅ All changes applied successfully")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Maintain CMake source lists and self-header includes.")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Report prospective changes without editing files")
|
||||
parser.add_argument("--cmake-only", action="store_true", help="Only update CMake source lists")
|
||||
parser.add_argument("--includes-only", action="store_true", help="Only ensure self-header includes")
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Maintain CMake source lists and ensure proper header includes (IWYU-style).",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
# Dry-run to see what would change:
|
||||
%(prog)s --dry-run
|
||||
|
||||
# Auto-discover libraries and update CMake files:
|
||||
%(prog)s --auto-discover
|
||||
|
||||
# Run IWYU-style header analysis:
|
||||
%(prog)s --iwyu
|
||||
|
||||
# Update only CMake source lists:
|
||||
%(prog)s --cmake-only
|
||||
|
||||
# Update only header includes:
|
||||
%(prog)s --includes-only
|
||||
"""
|
||||
)
|
||||
parser.add_argument("--dry-run", action="store_true",
|
||||
help="Report prospective changes without editing files")
|
||||
parser.add_argument("--cmake-only", action="store_true",
|
||||
help="Only update CMake source lists")
|
||||
parser.add_argument("--includes-only", action="store_true",
|
||||
help="Only ensure self-header includes")
|
||||
parser.add_argument("--iwyu", action="store_true",
|
||||
help="Run IWYU-style analysis to add missing headers")
|
||||
parser.add_argument("--auto-discover", action="store_true",
|
||||
help="Auto-discover CMake library files (*.cmake, *_library.cmake)")
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
return run(args.dry_run, args.cmake_only, args.includes_only)
|
||||
return run(args.dry_run, args.cmake_only, args.includes_only, args.iwyu, args.auto_discover)
|
||||
except Exception as exc: # pylint: disable=broad-except
|
||||
print(f"build_cleaner failed: {exc}")
|
||||
import traceback
|
||||
print(f"❌ build_cleaner failed: {exc}")
|
||||
if args.dry_run: # Show traceback in dry-run mode for debugging
|
||||
traceback.print_exc()
|
||||
return 1
|
||||
|
||||
|
||||
|
||||
6
scripts/requirements.txt
Normal file
6
scripts/requirements.txt
Normal file
@@ -0,0 +1,6 @@
|
||||
# Python dependencies for YAZE build scripts
|
||||
# Install with: pip3 install -r scripts/requirements.txt
|
||||
|
||||
# For .gitignore pattern matching in build_cleaner.py
|
||||
pathspec>=0.11.0
|
||||
|
||||
Reference in New Issue
Block a user