yaze/scripts/README.md

# YAZE Build Scripts

This directory contains build automation and maintenance scripts for the YAZE project.

## build_cleaner.py

Automates CMake source list maintenance and header include management with IWYU-style analysis.

### Features

1. **CMake Source List Maintenance**: Automatically updates source file lists in CMake files
2. **Self-Header Includes**: Ensures source files include their corresponding headers
3. **IWYU-Style Analysis**: Suggests missing headers based on symbol usage
4. **.gitignore Support**: Respects .gitignore patterns when scanning files
5. **Auto-Discovery**: Can discover CMake libraries that opt-in to auto-maintenance

### Usage

```bash
# Dry-run to see what would change (recommended first step)
python3 scripts/build_cleaner.py --dry-run

# Update CMake source lists and header includes
python3 scripts/build_cleaner.py

# Run IWYU-style header analysis
python3 scripts/build_cleaner.py --iwyu

# Auto-discover CMake libraries marked for auto-maintenance
python3 scripts/build_cleaner.py --auto-discover

# Update only CMake source lists
python3 scripts/build_cleaner.py --cmake-only

# Update only header includes
python3 scripts/build_cleaner.py --includes-only
```

### Opting-In to Auto-Maintenance

By default, the script only auto-maintains source lists that are explicitly marked. To mark a CMake variable for auto-maintenance, add a comment above the `set()` statement:

```cmake
# This list is auto-maintained by scripts/build_cleaner.py
set(
  YAZE_APP_EMU_SRC
  app/emu/audio/apu.cc
  app/emu/cpu/cpu.cc
  # ... more files
)
```

The script looks for comments containing "auto-maintain" (case-insensitive) within 3 lines above the `set()` statement.

### Excluding Files from Processing

To exclude a specific file from all processing (CMake lists, header includes, IWYU), add this token near the top of the file:

```cpp
// build_cleaner:ignore
```

### .gitignore Support

The script automatically respects `.gitignore` patterns. To enable this feature, install the `pathspec` dependency:

```bash
pip3 install -r scripts/requirements.txt
# or
pip3 install pathspec
```

### IWYU Configuration

The script includes basic IWYU-style analysis that suggests headers based on symbol prefixes. To customize which headers are suggested, edit the `COMMON_HEADERS` dictionary in the script:

```python
COMMON_HEADERS = {
    'std::': ['<memory>', '<string>', '<vector>', ...],
    'absl::': ['<absl/status/status.h>', ...],
    'ImGui::': ['<imgui.h>'],
    'SDL_': ['<SDL.h>'],
}
```

**Note**: The IWYU analysis is conservative and may suggest headers that are already transitively included. Use with care and review suggestions before applying.

### Integration with CMake

The script is integrated into the CMake build system:

```bash
# Run as a CMake target
cmake --build build --target build_cleaner
```

### Dependencies

- Python 3.7+
- `pathspec` (optional, for .gitignore support): `pip3 install pathspec`

### How It Works

1. **CMake Maintenance**: Scans directories specified in the configuration and updates `set(VAR_NAME ...)` blocks with the current list of source files
2. **Self-Headers**: For each `.cc`/`.cpp` file, ensures it includes its corresponding `.h` file
3. **IWYU Analysis**: Scans source files for symbols and suggests appropriate headers based on prefix matching

### Current Auto-Maintained Variables

**All 20 library source lists are now auto-maintained by default:**

- Core: `YAZE_APP_EMU_SRC`, `YAZE_APP_CORE_SRC`, `YAZE_APP_EDITOR_SRC`, `YAZE_APP_ZELDA3_SRC`, `YAZE_NET_SRC`, `YAZE_UTIL_SRC`
- GFX: `GFX_TYPES_SRC`, `GFX_BACKEND_SRC`, `GFX_RESOURCE_SRC`, `GFX_CORE_SRC`, `GFX_UTIL_SRC`, `GFX_RENDER_SRC`, `GFX_DEBUG_SRC`
- GUI: `GUI_CORE_SRC`, `CANVAS_SRC`, `GUI_WIDGETS_SRC`, `GUI_AUTOMATION_SRC`, `GUI_APP_SRC`
- Other: `YAZE_AGENT_SOURCES`, `YAZE_TEST_SOURCES`

The script intelligently preserves conditional blocks (if/endif) and excludes conditional files from the main source list.

## verify-build-environment.\*

`verify-build-environment.ps1` (Windows) and `verify-build-environment.sh` (macOS/Linux) are the primary diagnostics for contributors. They now:

- Check for `clang-cl`, Ninja, NASM, Visual Studio workloads, and VS Code (optional).
- Validate vcpkg bootstrap status plus `vcpkg/installed` cache contents.
- Warn about missing ROM assets (`zelda3.sfc`, `assets/zelda3.sfc`, etc.).
- Offer `-FixIssues` and `-CleanCache` switches to repair Git config, resync submodules, and wipe stale build directories.

Run the script once per machine (and rerun after major toolchain updates) to ensure presets such as `win-dbg`, `win-ai`, `mac-ai`, and `ci-windows-ai` have everything they need.

## setup-vcpkg-windows.ps1

Automates the vcpkg bootstrap flow on Windows:

1. Clones and bootstraps vcpkg (if not already present).
2. Verifies that `git`, `clang-cl`, and Ninja are available, printing friendly instructions when they are missing.
3. Installs the default triplet (`x64-windows` or `arm64-windows` when detected) and confirms that `vcpkg/installed/<triplet>` is populated.
4. Reminds you to rerun `.\scripts\verify-build-environment.ps1 -FixIssues` to double-check the environment.

Use it immediately after cloning the repository or whenever you need to refresh your local dependency cache before running `win-ai` or `ci-windows-ai` presets.

## CMake Validation Tools

A comprehensive toolkit for validating CMake configuration and catching dependency issues early. These tools help prevent build failures by detecting configuration problems before compilation.

### validate-cmake-config.cmake

Validates CMake configuration by checking targets, flags, and platform-specific settings.

```bash
# Validate default build directory
cmake -P scripts/validate-cmake-config.cmake

# Validate specific build directory
cmake -P scripts/validate-cmake-config.cmake build_ai
```

**What it checks:**
- Required targets exist
- Feature flag consistency (AI requires gRPC, etc.)
- Compiler settings (C++23, MSVC runtime on Windows)
- Abseil configuration on Windows (prevents missing include issues)
- Output directories
- Common configuration mistakes

### check-include-paths.sh

Validates include paths in compile_commands.json to catch missing includes before build.

```bash
# Check default build directory
./scripts/check-include-paths.sh

# Check specific build
./scripts/check-include-paths.sh build_ai

# Verbose mode (show all include dirs)
VERBOSE=1 ./scripts/check-include-paths.sh build
```

**Requires:** `jq` for better parsing (optional but recommended): `brew install jq`

**What it checks:**
- Common dependencies (SDL2, ImGui, yaml-cpp)
- Platform-specific includes
- Abseil includes from gRPC build (critical on Windows)
- Suspicious configurations (missing -I flags, relative paths)

### visualize-deps.py

Generates dependency graphs and detects circular dependencies.

```bash
# Generate GraphViz diagram
python3 scripts/visualize-deps.py build --format graphviz -o deps.dot
dot -Tpng deps.dot -o deps.png

# Generate Mermaid diagram
python3 scripts/visualize-deps.py build --format mermaid -o deps.mmd

# Show statistics
python3 scripts/visualize-deps.py build --stats
```

**Formats:**
- **graphviz**: DOT format for rendering with `dot` command
- **mermaid**: For embedding in Markdown/documentation
- **text**: Simple text tree for quick overview

**Features:**
- Detects circular dependencies (highlighted in red)
- Shows dependency statistics
- Color-coded targets (executables, libraries, etc.)

### test-cmake-presets.sh

Tests that all CMake presets can configure successfully.

```bash
# Test all presets for current platform
./scripts/test-cmake-presets.sh

# Test specific preset
./scripts/test-cmake-presets.sh --preset mac-ai

# Test in parallel (faster)
./scripts/test-cmake-presets.sh --platform mac --parallel 4

# Quick mode (don't clean between tests)
./scripts/test-cmake-presets.sh --quick
```

**Options:**
- `--parallel N`: Test N presets in parallel (default: 4)
- `--preset NAME`: Test only specific preset
- `--platform PLATFORM`: Test only mac/win/lin presets
- `--quick`: Skip cleaning between tests
- `--verbose`: Show full CMake output

### Usage in Development Workflow

**After configuring CMake:**
```bash
cmake --preset mac-ai
cmake -P scripts/validate-cmake-config.cmake build
./scripts/check-include-paths.sh build
```

**Before committing:**
```bash
# Test all platform presets configure successfully
./scripts/test-cmake-presets.sh --platform mac
```

**When adding new targets:**
```bash
# Check for circular dependencies
python3 scripts/visualize-deps.py build --stats
```

**For full details**, see [docs/internal/testing/cmake-validation.md](../docs/internal/testing/cmake-validation.md)

## Symbol Conflict Detection

Tools to detect One Definition Rule (ODR) violations and duplicate symbol definitions **before linking fails**.

### Quick Start

```bash
# Extract symbols from object files
./scripts/extract-symbols.sh

# Check for conflicts
./scripts/check-duplicate-symbols.sh

# Run tests
./scripts/test-symbol-detection.sh
```

### Scripts

#### extract-symbols.sh

Scans compiled object files and creates a JSON database of all symbols and their locations.

**Features:**
- Cross-platform: macOS/Linux (nm), Windows (dumpbin)
- Fast: ~2-3 seconds for typical builds
- Identifies duplicate definitions across object files
- Tracks symbol type (text, data, read-only, etc.)

**Usage:**
```bash
# Extract from build directory
./scripts/extract-symbols.sh build

# Custom output file
./scripts/extract-symbols.sh build symbols.json
```

**Output:** `build/symbol_database.json` - JSON with symbol conflicts listed

#### check-duplicate-symbols.sh

Analyzes symbol database and reports conflicts in developer-friendly format.

**Usage:**
```bash
# Check default database
./scripts/check-duplicate-symbols.sh

# Verbose output
./scripts/check-duplicate-symbols.sh --verbose

# Include fix suggestions
./scripts/check-duplicate-symbols.sh --fix-suggestions
```

**Exit codes:**
- `0` = No conflicts found
- `1` = Conflicts detected (fails in CI/pre-commit)

#### test-symbol-detection.sh

Integration test suite for symbol detection system.

**Usage:**
```bash
./scripts/test-symbol-detection.sh
```

**Validates:**
- Scripts are executable
- Build directory and object files exist
- Symbol extraction works correctly
- JSON database is valid
- Duplicate checker runs successfully
- Pre-commit hook is configured

### Git Hook Integration

**First-time setup:**
```bash
git config core.hooksPath .githooks
chmod +x .githooks/pre-commit
```

The pre-commit hook automatically runs symbol checks on changed files:
- Fast: ~1-2 seconds
- Only checks affected objects
- Warns about conflicts
- Can skip with `--no-verify` if needed

### CI/CD Integration

The `symbol-detection.yml` GitHub Actions workflow runs on:
- All pushes to `master` and `develop`
- All pull requests affecting C++ files
- Workflows can be triggered manually

**What it does:**
1. Builds project
2. Extracts symbols from all object files
3. Checks for conflicts
4. Uploads symbol database as artifact
5. Fails job if conflicts found

### Common Fixes

**Duplicate global variable:**
```cpp
// Bad - defined in both files
ABSL_FLAG(std::string, rom, "", "ROM path");

// Fix 1: Use static (internal linkage)
static ABSL_FLAG(std::string, rom, "", "ROM path");

// Fix 2: Use anonymous namespace
namespace {
  ABSL_FLAG(std::string, rom, "", "ROM path");
}
```

**Duplicate function:**
```cpp
// Bad - defined in both files
void ProcessData() { /* ... */ }

// Fix: Make inline or use static
inline void ProcessData() { /* ... */ }
```

### Performance

| Operation | Time |
|-----------|------|
| Extract (4000 objects, macOS) | ~3s |
| Extract (4000 objects, Windows) | ~5-7s |
| Check duplicates | <100ms |
| Pre-commit hook | ~1-2s |

### Documentation

Full documentation available in:
- [docs/internal/testing/symbol-conflict-detection.md](../docs/internal/testing/symbol-conflict-detection.md)
- [docs/internal/testing/sample-symbol-database.json](../docs/internal/testing/sample-symbol-database.json)