yaze/docs/internal/test-suite-configuration.md

Test Suite Configuration Guide

Overview

The yaze test suite has been reorganized to improve CI performance and developer experience. Optional test suites (ROM-dependent, AI experimental, benchmarks) are now gated OFF by default and only run in nightly CI or when explicitly enabled.

CMake Options for Test Suites

Option	Default	Description	Required For
YAZE_BUILD_TESTS	ON	Build test executables	All tests
YAZE_ENABLE_ROM_TESTS	OFF	Enable tests requiring ROM files	ROM-dependent tests
YAZE_ENABLE_AI_RUNTIME	OFF	Enable AI runtime integration tests	Experimental AI tests
YAZE_ENABLE_BENCHMARK_TESTS	OFF	Enable performance benchmarks	Benchmark suite
YAZE_TEST_ROM_PATH	build/bin/zelda3.sfc	Path to test ROM file	ROM-dependent tests

Test Categories and Labels

Default Test Suites (Always Enabled)

• stable: Core functionality tests that should always pass

  • Unit tests for core components
  • Integration tests without external dependencies
  • Basic smoke tests
  • Label: stable

• gui: GUI framework tests using ImGuiTestEngine

  • Canvas automation tests
  • Editor smoke tests
  • Can run headlessly with -nogui flag
  • Labels: gui, experimental

Optional Test Suites (Off by Default)

• rom_dependent: Tests requiring Zelda3 ROM file

  • ROM loading and manipulation tests
  • Version upgrade tests
  • Full editor workflow tests
  • Label: rom_dependent
  • Enable with: -DYAZE_ENABLE_ROM_TESTS=ON

• experimental: AI and experimental feature tests

  • Gemini vision API tests
  • AI-powered test generation
  • Agent automation tests
  • Label: experimental
  • Enable with: -DYAZE_ENABLE_AI_RUNTIME=ON

• benchmark: Performance benchmarks

  • Graphics optimization benchmarks
  • Memory pool performance tests
  • Label: benchmark
  • Enable with: -DYAZE_ENABLE_BENCHMARK_TESTS=ON

Running Tests

Quick Start (Stable Tests Only)

# Configure with default settings (optional suites OFF)
cmake --preset mac-dbg
cmake --build build --target yaze_test
ctest --test-dir build -L stable

With ROM-Dependent Tests

# Configure with ROM tests enabled
cmake --preset mac-dbg -DYAZE_ENABLE_ROM_TESTS=ON -DYAZE_TEST_ROM_PATH=~/zelda3.sfc
cmake --build build
ctest --test-dir build -L rom_dependent

With AI/Experimental Tests

# Configure with AI runtime enabled
cmake --preset mac-ai  # or lin-ai, win-ai
cmake --build build
ctest --test-dir build -L experimental

# Note: AI tests require:
# - GEMINI_API_KEY environment variable for Gemini tests
# - Ollama running locally for Ollama tests

With Benchmark Tests

# Configure with benchmarks enabled
cmake --preset mac-dbg -DYAZE_ENABLE_BENCHMARK_TESTS=ON
cmake --build build
ctest --test-dir build -L benchmark

Run All Tests (Nightly Configuration)

# Enable all optional suites
cmake --preset mac-dbg \
  -DYAZE_ENABLE_ROM_TESTS=ON \
  -DYAZE_ENABLE_AI_RUNTIME=ON \
  -DYAZE_ENABLE_BENCHMARK_TESTS=ON \
  -DYAZE_TEST_ROM_PATH=~/zelda3.sfc
cmake --build build
ctest --test-dir build

CI/CD Configuration

PR/Push Workflow (Fast Feedback)

• Runs on: push to master/develop, pull requests
• Test suites: stable only
• Approximate runtime: 5-10 minutes
• Purpose: Quick regression detection

Nightly Workflow (Comprehensive Coverage)

• Runs on: Schedule (3 AM UTC daily)
• Test suites: All (stable, rom_dependent, experimental, benchmark)
• Approximate runtime: 30-45 minutes
• Purpose: Deep validation, performance tracking

Graceful Test Skipping

Tests automatically skip when prerequisites are missing:

AI Tests

• Check for GEMINI_API_KEY environment variable
• Skip with GTEST_SKIP() if not present
• Example:

void SetUp() override {
  const char* api_key = std::getenv("GEMINI_API_KEY");
  if (!api_key || std::string(api_key).empty()) {
    GTEST_SKIP() << "GEMINI_API_KEY not set. Skipping multimodal tests.";
  }
}

ROM-Dependent Tests

• Check for ROM file at configured path
• Skip if file doesn't exist
• Controlled by YAZE_ENABLE_ROM_TESTS CMake option

Backward Compatibility

The changes maintain backward compatibility:

• Existing developer workflows continue to work
• Default cmake --build build --target yaze_test still builds core tests
• Optional suites only built when explicitly enabled
• CI presets unchanged for existing workflows

Preset Configurations

Development Presets (Optional Tests OFF)

• mac-dbg, lin-dbg, win-dbg: Debug builds, core tests only
• mac-rel, lin-rel, win-rel: Release builds, core tests only

AI Development Presets (AI Tests ON, Others OFF)

• mac-ai, lin-ai, win-ai: AI runtime enabled
• Includes YAZE_ENABLE_AI_RUNTIME=ON
• For AI/agent development and testing

CI Presets

• ci-linux, ci-macos, ci-windows: Minimal CI builds
• ci-windows-ai: Windows with AI runtime for agent testing

Migration Guide for Developers

If you were running all tests:

Before: ./build/bin/yaze_test After: Same command still works, but only runs stable tests by default

To run the same comprehensive suite as before:

cmake --preset your-preset \
  -DYAZE_ENABLE_ROM_TESTS=ON \
  -DYAZE_ENABLE_BENCHMARK_TESTS=ON
./build/bin/yaze_test

For AI developers:

Use the AI-specific presets: mac-ai, lin-ai, or win-ai

Troubleshooting

Tests Not Found

If expected tests are missing, check:

1. CMake option is enabled (e.g., -DYAZE_ENABLE_ROM_TESTS=ON)
2. Dependencies are available (ROM file, API keys)
3. Correct test label used with ctest (e.g., -L rom_dependent)

AI Tests Failing

Ensure:

• GEMINI_API_KEY is set in environment
• Ollama is running (for Ollama tests): ollama serve
• Network connectivity is available

ROM Tests Failing

Verify:

• ROM file exists at YAZE_TEST_ROM_PATH
• ROM is valid Zelda3 US version
• Path is absolute, not relative