yaze/docs/internal/testing/gap-analysis.md

Testing Infrastructure Gap Analysis

Executive Summary

Recent CI failures revealed critical gaps in our testing infrastructure that allowed platform-specific build failures to reach CI. This document analyzes what we currently test, what we missed, and what infrastructure is needed to catch issues earlier.

Date: 2025-11-20 Triggered By: Multiple CI failures in commits 43a0e5e314, c2bb90a3f1, and related fixes

---

1. Issues We Didn't Catch Locally

1.1 Windows Abseil Include Path Issues (c2bb90a3f1)

Problem: Abseil headers not found during Windows/clang-cl compilation Why it wasn't caught:

• No local pre-push compilation check
• CMake configuration validates successfully, but compilation fails later
• Include path propagation from gRPC/Abseil not validated until full compile

What would have caught it:

• ✅ Smoke compilation test (compile subset of files to catch header issues)
• ✅ CMake configuration validator (check include path propagation)
• ✅ Header dependency checker

1.2 Linux FLAGS Symbol Conflicts (43a0e5e314, eb77bbeaff)

Problem: ODR (One Definition Rule) violation - multiple FLAGS symbols across libraries Why it wasn't caught:

• Symbol conflicts only appear at link time
• No cross-library symbol conflict detection
• Static analysis doesn't catch ODR violations
• Unit tests don't link full dependency graph

What would have caught it:

• ✅ Symbol conflict scanner (nm/objdump analysis)
• ✅ ODR violation detector
• ✅ Full integration build test (link all libraries together)

1.3 Platform-Specific Configuration Issues

Problem: Preprocessor flags, compiler detection, and platform-specific code paths Why it wasn't caught:

• No local cross-platform validation
• CMake configuration differences between platforms not tested
• Compiler detection logic (clang-cl vs MSVC) not validated

What would have caught it:

• ✅ CMake configuration dry-run on multiple platforms
• ✅ Preprocessor flag validation
• ✅ Compiler detection smoke test

---

2. Current Testing Coverage

2.1 What We Test Well

Unit Tests (test/unit/)

• Coverage: Core algorithms, data structures, parsers
• Speed: Fast (<1s for most tests)
• Isolation: Mocked dependencies, no ROM required
• CI: ✅ Runs on every PR
• Example: hex_test.cc, asar_wrapper_test.cc, snes_palette_test.cc

Strengths:

• Catches logic errors quickly
• Good for TDD
• Platform-independent

Gaps:

• Doesn't catch build system issues
• Doesn't catch linking problems
• Doesn't validate dependencies

Integration Tests (test/integration/)

• Coverage: Multi-component interactions, ROM operations
• Speed: Slower (1-10s per test)
• Dependencies: May require ROM files
• CI: ✅ Runs on develop/master
• Example: asar_integration_test.cc, dungeon_editor_v2_test.cc

Strengths:

• Tests component interactions
• Validates ROM operations

Gaps:

• Still doesn't catch platform-specific issues
• Doesn't validate symbol conflicts
• Doesn't test cross-library linking

E2E Tests (test/e2e/)

• Coverage: Full UI workflows, user interactions
• Speed: Very slow (10-60s per test)
• Dependencies: GUI, ImGuiTestEngine
• CI: ⚠️ Limited (only on macOS z3ed-agent-test)
• Example: dungeon_editor_smoke_test.cc, canvas_selection_test.cc

Strengths:

• Validates real user workflows
• Tests UI responsiveness

Gaps:

• Not run consistently across platforms
• Slow feedback loop
• Requires display/window system

2.2 What We DON'T Test

Build System Validation

• ❌ CMake configuration correctness per preset
• ❌ Include path propagation from dependencies
• ❌ Compiler flag compatibility
• ❌ Linker flag validation
• ❌ Cross-preset compatibility

Symbol-Level Issues

• ❌ ODR (One Definition Rule) violations
• ❌ Duplicate symbol detection across libraries
• ❌ Symbol visibility (public/private)
• ❌ ABI compatibility between libraries

Platform-Specific Compilation

• ❌ Header-only compilation checks
• ❌ Preprocessor branch coverage
• ❌ Platform macro validation
• ❌ Compiler-specific feature detection

Dependency Health

• ❌ Include path conflicts
• ❌ Library version mismatches
• ❌ Transitive dependency validation
• ❌ Static vs shared library conflicts

---

3. CI/CD Coverage Analysis

3.1 Current CI Matrix (.github/workflows/ci.yml)

Platform	Build	Test (stable)	Test (unit)	Test (integration)	Test (AI)
Ubuntu 22.04 (GCC-12)	✅	✅	✅	❌	❌
macOS 14 (Clang)	✅	✅	✅	❌	✅
Windows 2022 (Core)	✅	✅	✅	❌	❌
Windows 2022 (AI)	✅	✅	✅	❌	❌

CI Job Flow:

1. build: Configure + compile full project
2. test: Run stable + unit tests
3. windows-agent: Full AI stack (gRPC + AI runtime)
4. code-quality: clang-format, cppcheck, clang-tidy
5. memory-sanitizer: AddressSanitizer (Linux only)
6. z3ed-agent-test: Full agent test suite (macOS only)

3.2 CI Gaps

Missing Early Feedback

• ❌ No compilation-only job (fails after 15-20 min build)
• ❌ No CMake configuration validation job (would catch in <1 min)
• ❌ No symbol conflict checking job

Limited Platform Coverage

• ⚠️ Only Linux gets AddressSanitizer
• ⚠️ Only macOS gets full z3ed agent tests
• ⚠️ Windows AI stack not tested on PRs (only post-merge)

Incomplete Testing

• ❌ Integration tests not run in CI
• ❌ E2E tests not run on Linux/Windows
• ❌ No ROM-dependent testing
• ❌ No performance regression detection

---

4. Developer Workflow Gaps

4.1 Pre-Commit Hooks

Current State: None Gap: No automatic checks before local commits

Should Include:

• clang-format check
• Build system sanity check
• Copyright header validation

4.2 Pre-Push Validation

Current State: Manual testing only Gap: Easy to push broken code to CI

Should Include:

• Smoke build test (quick compilation check)
• Unit test run
• Symbol conflict detection

4.3 Local Cross-Platform Testing

Current State: Developer-dependent Gap: No easy way to test across platforms locally

Should Include:

• Docker-based Linux testing
• VM-based Windows testing (for macOS/Linux devs)
• Preset validation tool

---

5. Root Cause Analysis by Issue Type

5.1 Windows Abseil Include Paths

Timeline:

• ✅ Local macOS build succeeds
• ✅ CMake configuration succeeds on all platforms
• ❌ Windows compilation fails 15 minutes into CI
• ❌ Fix attempt 1 fails (14d1f5de4c)
• ❌ Fix attempt 2 fails (c2bb90a3f1)
• ✅ Final fix succeeds

Why Multiple Attempts:

1. No local Windows testing environment
2. CMake configuration doesn't validate actual compilation
3. No header-only compilation check
4. 15-20 minute feedback cycle from CI

Prevention:

• Header compilation smoke test
• CMake include path validator
• Local Windows testing (Docker/VM)

5.2 Linux FLAGS Symbol Conflicts

Timeline:

• ✅ Local macOS build succeeds
• ✅ Unit tests pass
• ❌ Linux full build fails at link time
• ❌ ODR violation: multiple FLAGS definitions
• ✅ Fix: move FLAGS definition, rename conflicts

Why It Happened:

1. gflags creates FLAGS_* symbols in headers
2. Multiple translation units define same symbols
3. macOS linker more permissive than Linux ld
4. No symbol conflict detection

Prevention:

• Symbol conflict scanner
• ODR violation checker
• Cross-platform link test

---

6. Recommended Testing Levels

We propose a 5-level testing pyramid:

Level 0: Static Analysis (< 1s)

• clang-format
• clang-tidy on changed files
• Copyright headers
• CMakeLists.txt syntax

Level 1: Configuration Validation (< 10s)

• CMake configure dry-run
• Include path validation
• Compiler detection check
• Preprocessor flag validation

Level 2: Smoke Compilation (< 2 min)

• Compile subset of files (1 file per library)
• Header-only compilation
• Template instantiation check
• Platform-specific branch validation

Level 3: Symbol Validation (< 5 min)

• Full project compilation
• Symbol conflict detection (nm/dumpbin)
• ODR violation check
• Library dependency graph

Level 4: Test Execution (5-30 min)

• Unit tests (fast)
• Integration tests (medium)
• E2E tests (slow)
• ROM-dependent tests (optional)

---

7. Actionable Recommendations

7.1 Immediate Actions (This Initiative)

1. Create pre-push scripts (scripts/pre-push-test.sh, scripts/pre-push-test.ps1)

   • Run Level 0-2 checks locally
   • Estimated time: <2 minutes
   • Blocks 90% of CI failures

2. Create symbol conflict detector (scripts/verify-symbols.sh)

   • Scan built libraries for duplicate symbols
   • Run as part of pre-push
   • Catches ODR violations

3. Document testing strategy (docs/internal/testing/testing-strategy.md)

   • Clear explanation of each test level
   • When to run which tests
   • CI vs local testing

4. Create pre-push checklist (docs/internal/testing/pre-push-checklist.md)

   • Interactive checklist for developers
   • Links to tools and scripts

7.2 Short-Term Improvements (Next Sprint)

1. Add CI compile-only job

   • Runs in <5 minutes
   • Catches compilation issues before full build
   • Fails fast

2. Add CI symbol checking job

   • Runs after compile-only
   • Detects ODR violations
   • Platform-specific

3. Add CMake configuration validation job

   • Tests all presets
   • Validates include paths
   • <2 minutes

4. Enable integration tests in CI

   • Run on develop/master only (not PRs)
   • Requires ROM file handling

7.3 Long-Term Improvements (Future)

1. Docker-based local testing

   • Linux environment for macOS/Windows devs
   • Matches CI exactly
   • Fast feedback

2. Cross-platform test matrix locally

   • Run tests across multiple platforms
   • Automated VM/container management

3. Performance regression detection

   • Benchmark suite
   • Historical tracking
   • Automatic alerts

4. Coverage tracking

   • Line coverage per PR
   • Coverage trends over time
   • Uncovered code reports

---

8. Success Metrics

8.1 Developer Experience

• Target: <2 minutes pre-push validation time
• Target: 90% reduction in CI build failures
• Target: <3 attempts to fix CI issues (down from 5-10)

8.2 CI Efficiency

• Target: <5 minutes to first failure signal
• Target: 50% reduction in wasted CI time
• Target: 95% PR pass rate (up from ~70%)

8.3 Code Quality

• Target: Zero ODR violations
• Target: Zero platform-specific include issues
• Target: 100% symbol conflict detection

---

9. Reference

Similar Issues in Recent History

• Windows std::filesystem support (19196ca87c, b556b155a5)
• Linux circular dependency (0812a84a22, e36d81f357)
• macOS z3ed linker error (9c562df277)
• Windows clang-cl detection (84cdb09a5b, cbdc6670a1)

Related Documentation

• docs/public/build/quick-reference.md - Build commands
• docs/public/build/troubleshooting.md - Platform-specific fixes
• CLAUDE.md - Build system guidelines
• .github/workflows/ci.yml - CI configuration

Tools Used

• nm (Unix) / dumpbin (Windows) - Symbol inspection
• clang-tidy - Static analysis
• cppcheck - Code quality
• cmake --preset <name> --list-presets - Preset validation