e930789f4b
- Updated CMakeLists.txt to introduce options for enabling experimental tests and minimal CI builds. - Modified CMakePresets.json to refine test presets, including stable and experimental test configurations. - Enhanced CI workflows in ci.yml to streamline testing processes, ensuring stable tests are prioritized and experimental tests are run separately. - Added new documentation files outlining the CI/CD testing strategy and testing categories for better clarity on testing practices. - Improved test discovery in CMakeLists.txt to simplify labeling and enhance CI efficiency.
203 lines
6.0 KiB
Markdown
203 lines
6.0 KiB
Markdown
# Testing Strategy for Yaze v0.3.0
|
|
|
|
## Test Categories
|
|
|
|
Yaze uses a comprehensive testing strategy with different categories of tests to ensure quality while maintaining efficient CI/CD pipelines.
|
|
|
|
### Stable Tests (STABLE)
|
|
**Always run in CI/CD - Required for releases**
|
|
|
|
- **AsarWrapperTest**: Core Asar functionality tests
|
|
- **SnesTileTest**: SNES tile format handling
|
|
- **CompressionTest**: Data compression/decompression
|
|
- **SnesPaletteTest**: SNES palette operations
|
|
- **HexTest**: Hexadecimal utilities
|
|
- **AsarIntegrationTest**: Asar integration without ROM dependencies
|
|
|
|
**Characteristics:**
|
|
- Fast execution (< 30 seconds total)
|
|
- No external dependencies (ROMs, complex setup)
|
|
- High reliability and deterministic results
|
|
- Core functionality testing
|
|
|
|
### ROM-Dependent Tests (ROM_DEPENDENT)
|
|
**Only run in development with available ROM files**
|
|
|
|
- **AsarRomIntegrationTest**: Real ROM patching and symbol extraction
|
|
- **ROM-based integration tests**: Tests requiring actual game ROM files
|
|
|
|
**Characteristics:**
|
|
- Require specific ROM files to be present
|
|
- Test real-world functionality
|
|
- May be slower due to large file operations
|
|
- Automatically skipped in CI if ROM files unavailable
|
|
|
|
### Experimental Tests (EXPERIMENTAL)
|
|
**Run separately, allowed to fail**
|
|
|
|
- **CpuTest**: 65816 CPU emulation tests (complex, may have timing issues)
|
|
- **Spc700Test**: SPC700 audio processor tests
|
|
- **ApuTest**: Audio Processing Unit tests
|
|
- **PpuTest**: Picture Processing Unit tests
|
|
- **Complex Integration Tests**: Multi-component integration tests
|
|
|
|
**Characteristics:**
|
|
- May be unstable due to emulation complexity
|
|
- Test advanced/experimental features
|
|
- Allowed to fail without blocking releases
|
|
- Run in separate CI job with `continue-on-error: true`
|
|
|
|
### GUI Tests (GUI_TEST)
|
|
**Tests requiring graphical components**
|
|
|
|
- **DungeonEditorIntegrationTest**: GUI-based dungeon editing
|
|
- **Editor Integration Tests**: Tests requiring ImGui components
|
|
|
|
**Characteristics:**
|
|
- Require display/graphics context
|
|
- May not work in headless CI environments
|
|
- Focus on user interface functionality
|
|
|
|
## CI/CD Strategy
|
|
|
|
### Main CI Pipeline
|
|
```yaml
|
|
# Always run - required for merge
|
|
- Run Stable Tests: --label-regex "STABLE"
|
|
|
|
# Optional - allowed to fail
|
|
- Run Experimental Tests: --label-regex "EXPERIMENTAL" (continue-on-error: true)
|
|
```
|
|
|
|
### Development Testing
|
|
```bash
|
|
# Quick development testing
|
|
ctest --preset stable
|
|
|
|
# Full development testing with ROM
|
|
ctest --preset dev
|
|
|
|
# Test specific functionality
|
|
ctest --preset asar-only
|
|
```
|
|
|
|
### Release Testing
|
|
```bash
|
|
# Release candidate testing
|
|
ctest --preset stable --parallel
|
|
ctest --preset ci
|
|
```
|
|
|
|
## Test Execution Examples
|
|
|
|
### Command Line Usage
|
|
|
|
```bash
|
|
# Run only stable tests (release-ready)
|
|
ctest --test-dir build --label-regex "STABLE"
|
|
|
|
# Run experimental tests (allowed to fail)
|
|
ctest --test-dir build --label-regex "EXPERIMENTAL"
|
|
|
|
# Run Asar-specific tests
|
|
ctest --test-dir build -R "*Asar*"
|
|
|
|
# Run tests excluding ROM-dependent ones
|
|
ctest --test-dir build --label-exclude "ROM_DEPENDENT"
|
|
|
|
# Run with specific preset
|
|
ctest --preset stable
|
|
ctest --preset experimental
|
|
```
|
|
|
|
### CMake Preset Usage
|
|
|
|
```bash
|
|
# Development workflow
|
|
cmake --preset dev
|
|
cmake --build --preset dev
|
|
ctest --preset dev
|
|
|
|
# CI workflow
|
|
cmake --preset ci
|
|
cmake --build --preset ci
|
|
ctest --preset ci
|
|
|
|
# Release workflow
|
|
cmake --preset release
|
|
cmake --build --preset release
|
|
ctest --preset stable
|
|
```
|
|
|
|
## Test Development Guidelines
|
|
|
|
### Writing Stable Tests
|
|
- **Fast execution**: Aim for < 1 second per test
|
|
- **No external dependencies**: Self-contained test data
|
|
- **Deterministic**: Same results every run
|
|
- **Core functionality**: Test essential features only
|
|
|
|
### Writing Experimental Tests
|
|
- **Complex scenarios**: Multi-component integration
|
|
- **Advanced features**: Emulation, complex algorithms
|
|
- **Performance tests**: May vary by system
|
|
- **GUI components**: May require display context
|
|
|
|
### Writing ROM-Dependent Tests
|
|
- **Use TestRomManager**: Proper ROM file handling
|
|
- **Graceful skipping**: Skip if ROM not available
|
|
- **Real-world scenarios**: Test with actual game data
|
|
- **Label appropriately**: Always include ROM_DEPENDENT label
|
|
|
|
## CI/CD Efficiency
|
|
|
|
### Fast Feedback Loop
|
|
1. **Stable tests run first** - Quick feedback for developers
|
|
2. **Experimental tests run in parallel** - Don't block on unstable tests
|
|
3. **ROM tests skipped** - No dependency on external files
|
|
4. **Selective test execution** - Only run relevant tests for changes
|
|
|
|
### Release Quality Gates
|
|
1. **All stable tests must pass** - No exceptions
|
|
2. **Experimental tests informational only** - Don't block releases
|
|
3. **ROM tests run manually** - When ROM files available
|
|
4. **Performance benchmarks** - Track regression trends
|
|
|
|
## Maintenance Strategy
|
|
|
|
### Regular Review
|
|
- **Monthly review** of experimental test failures
|
|
- **Promote stable experimental tests** to stable category
|
|
- **Deprecate obsolete tests** that no longer provide value
|
|
- **Update test categorization** as features mature
|
|
|
|
### Performance Monitoring
|
|
- **Track test execution times** for CI efficiency
|
|
- **Identify slow tests** for optimization or recategorization
|
|
- **Monitor CI resource usage** and adjust parallelism
|
|
- **Benchmark critical path tests** for performance regression
|
|
|
|
## Test Categories by Feature
|
|
|
|
### Asar Integration
|
|
- **Stable**: AsarWrapperTest, AsarIntegrationTest
|
|
- **ROM-Dependent**: AsarRomIntegrationTest
|
|
- **Focus**: Core assembly patching and symbol extraction
|
|
|
|
### Graphics System
|
|
- **Stable**: SnesTileTest, SnesPaletteTest, CompressionTest
|
|
- **Experimental**: Complex rendering tests
|
|
- **Focus**: SNES graphics format handling
|
|
|
|
### Emulation
|
|
- **Experimental**: CpuTest, Spc700Test, ApuTest, PpuTest
|
|
- **Focus**: Hardware emulation accuracy
|
|
- **Note**: Complex timing-sensitive tests
|
|
|
|
### Editor Components
|
|
- **GUI**: DungeonEditorIntegrationTest, Editor integration tests
|
|
- **Experimental**: Complex editor workflow tests
|
|
- **Focus**: User interface functionality
|
|
|
|
This strategy ensures efficient CI/CD while maintaining comprehensive test coverage for quality assurance.
|