backend-infra-engineer: Release v0.3.3 snapshot
This commit is contained in:
scawful
339
docs/internal/testing/MATRIX_TESTING_README.md
Normal file
339
docs/internal/testing/MATRIX_TESTING_README.md
Normal file
@@ -0,0 +1,339 @@
|
||||
# Matrix Testing System for yaze
|
||||
|
||||
## What's This?
|
||||
|
||||
A comprehensive **platform/configuration matrix testing system** that validates CMake flag combinations across all platforms.
|
||||
|
||||
**Before**: Only tested default configurations, missed dangerous flag interactions.
|
||||
**After**: 7 local configurations + 14 nightly configurations = catch issues before they reach users.
|
||||
|
||||
## Quick Start (30 seconds)
|
||||
|
||||
### For Developers
|
||||
|
||||
Before pushing your code:
|
||||
|
||||
```bash
|
||||
./scripts/test-config-matrix.sh
|
||||
```
|
||||
|
||||
If all tests pass (green checkmarks), you're good to push.
|
||||
|
||||
### For CI/CD
|
||||
|
||||
Tests run automatically:
|
||||
- Every night at 2 AM UTC (comprehensive matrix)
|
||||
- On-demand with `[matrix]` in commit message
|
||||
- Results in GitHub Actions
|
||||
|
||||
## What Gets Tested?
|
||||
|
||||
### Tier 1: Core Configurations (Every Commit)
|
||||
3 standard presets everyone knows about:
|
||||
- Linux (gRPC + Agent CLI)
|
||||
- macOS (gRPC + Agent UI + Agent CLI)
|
||||
- Windows (gRPC core features)
|
||||
|
||||
### Tier 2: Feature Combinations (Nightly)
|
||||
Strategic testing of dangerous interactions:
|
||||
|
||||
**Linux**:
|
||||
- `minimal` - No AI, no gRPC
|
||||
- `grpc-only` - gRPC without automation
|
||||
- `full-ai` - All features enabled
|
||||
- `cli-no-grpc` - CLI without networking
|
||||
- `http-api` - REST endpoints
|
||||
- `no-json` - Ollama mode (no JSON parsing)
|
||||
|
||||
**macOS**:
|
||||
- `minimal` - GUI, no AI
|
||||
- `full-ai` - All features
|
||||
- `agent-ui` - Agent UI panels only
|
||||
- `universal` - ARM64 + x86_64 binary
|
||||
|
||||
**Windows**:
|
||||
- `minimal` - No AI
|
||||
- `full-ai` - All features
|
||||
- `grpc-remote` - gRPC + automation
|
||||
- `z3ed-cli` - CLI executable
|
||||
|
||||
### Tier 3: Platform-Specific (As Needed)
|
||||
Architecture-specific tests when issues arise.
|
||||
|
||||
## The Problem It Solves
|
||||
|
||||
Matrix testing catches **cross-configuration issues** that single preset testing misses:
|
||||
|
||||
### Example 1: gRPC Without Automation
|
||||
```bash
|
||||
cmake -B build -DYAZE_ENABLE_GRPC=ON -DYAZE_ENABLE_REMOTE_AUTOMATION=OFF
|
||||
# Before: Silent link error (gRPC headers but no server code)
|
||||
# After: CMake auto-enforces constraint, matrix tests validate
|
||||
```
|
||||
|
||||
### Example 2: HTTP API Without CLI Stack
|
||||
```bash
|
||||
cmake -B build -DYAZE_ENABLE_HTTP_API=ON -DYAZE_ENABLE_AGENT_CLI=OFF
|
||||
# Before: Runtime error (endpoints defined but no dispatcher)
|
||||
# After: CMake auto-enforces, matrix tests validate
|
||||
```
|
||||
|
||||
### Example 3: AI Runtime Without JSON
|
||||
```bash
|
||||
cmake -B build -DYAZE_ENABLE_AI_RUNTIME=ON -DYAZE_ENABLE_JSON=OFF
|
||||
# Before: Compile error (Gemini needs JSON)
|
||||
# After: Matrix test `no-json` catches this edge case
|
||||
```
|
||||
|
||||
**All 6 known problematic patterns are now documented and tested.**
|
||||
|
||||
## Files & Usage
|
||||
|
||||
### For Getting Started (5 min)
|
||||
📄 **`/docs/internal/testing/QUICKSTART.md`**
|
||||
- One-page quick reference
|
||||
- Common commands
|
||||
- Error troubleshooting
|
||||
|
||||
### For Understanding Strategy (20 min)
|
||||
📄 **`/docs/internal/testing/matrix-testing-strategy.md`**
|
||||
- Why we test this way
|
||||
- Real bug examples
|
||||
- Philosophy behind smart matrix testing
|
||||
- Monitoring and maintenance
|
||||
|
||||
### For Complete Reference (30 min)
|
||||
📄 **`/docs/internal/configuration-matrix.md`**
|
||||
- All 18 CMake flags documented
|
||||
- Dependency graph
|
||||
- Complete tested matrix
|
||||
- Problematic combinations and fixes
|
||||
|
||||
### For Hands-On Use
|
||||
🔧 **`/scripts/test-config-matrix.sh`**
|
||||
```bash
|
||||
./scripts/test-config-matrix.sh # Test all
|
||||
./scripts/test-config-matrix.sh --config minimal # Specific
|
||||
./scripts/test-config-matrix.sh --smoke # Quick 30s test
|
||||
./scripts/test-config-matrix.sh --verbose # Detailed output
|
||||
./scripts/test-config-matrix.sh --help # All options
|
||||
```
|
||||
|
||||
🔧 **`/scripts/validate-cmake-config.sh`**
|
||||
```bash
|
||||
./scripts/validate-cmake-config.sh \
|
||||
-DYAZE_ENABLE_GRPC=ON \
|
||||
-DYAZE_ENABLE_HTTP_API=ON
|
||||
# Warns about problematic combinations before build
|
||||
```
|
||||
|
||||
## Integration with Your Workflow
|
||||
|
||||
### Before Pushing (Recommended)
|
||||
```bash
|
||||
# Make your changes
|
||||
git add src/...
|
||||
|
||||
# Test locally
|
||||
./scripts/test-config-matrix.sh
|
||||
|
||||
# If green, commit and push
|
||||
git commit -m "feature: your change"
|
||||
git push
|
||||
```
|
||||
|
||||
### In CI/CD (Automatic)
|
||||
- Standard tests run on every push (Tier 1)
|
||||
- Comprehensive tests run nightly (Tier 2)
|
||||
- Can trigger with `[matrix]` in commit message
|
||||
|
||||
### When Adding New Features
|
||||
1. Update `cmake/options.cmake` (define flag + constraints)
|
||||
2. Document in `/docs/internal/configuration-matrix.md`
|
||||
3. Add test config to `/scripts/test-config-matrix.sh`
|
||||
4. Add CI job to `/.github/workflows/matrix-test.yml`
|
||||
|
||||
## Real Examples
|
||||
|
||||
### Example: Testing a Configuration Change
|
||||
|
||||
```bash
|
||||
# I want to test what happens with no JSON support
|
||||
./scripts/test-config-matrix.sh --config no-json
|
||||
|
||||
# Output:
|
||||
# [INFO] Testing: no-json
|
||||
# [✓] Configuration successful
|
||||
# [✓] Build successful
|
||||
# [✓] Unit tests passed
|
||||
# ✓ no-json: PASSED
|
||||
```
|
||||
|
||||
### Example: Validating Flag Combination
|
||||
|
||||
```bash
|
||||
# Is this combination valid?
|
||||
./scripts/validate-cmake-config.sh \
|
||||
-DYAZE_ENABLE_HTTP_API=ON \
|
||||
-DYAZE_ENABLE_AGENT_CLI=OFF
|
||||
|
||||
# Output:
|
||||
# ✗ ERROR: YAZE_ENABLE_HTTP_API=ON requires YAZE_ENABLE_AGENT_CLI=ON
|
||||
```
|
||||
|
||||
### Example: Smoke Test Before Push
|
||||
|
||||
```bash
|
||||
# Quick 30-second validation
|
||||
./scripts/test-config-matrix.sh --smoke
|
||||
|
||||
# Output:
|
||||
# [INFO] Testing: minimal
|
||||
# [INFO] Running smoke test (configure only)
|
||||
# [✓] Configuration successful
|
||||
# Results: 7/7 passed
|
||||
```
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
### 1. Smart Matrix, Not Exhaustive
|
||||
- Testing all 2^18 combinations = 262,144 tests (impossible)
|
||||
- Instead: 7 local configs + 14 nightly configs (practical)
|
||||
- Covers: baselines, extremes, interactions, platforms
|
||||
|
||||
### 2. Automatic Constraint Enforcement
|
||||
CMake automatically prevents invalid combinations:
|
||||
```cmake
|
||||
if(YAZE_ENABLE_REMOTE_AUTOMATION AND NOT YAZE_ENABLE_GRPC)
|
||||
set(YAZE_ENABLE_GRPC ON ... FORCE)
|
||||
endif()
|
||||
```
|
||||
|
||||
### 3. Tiered Execution
|
||||
- **Tier 1** (3 configs): Every commit, ~15 min
|
||||
- **Tier 2** (14 configs): Nightly, ~45 min
|
||||
- **Tier 3** (architecture-specific): On-demand
|
||||
|
||||
### 4. Developer-Friendly
|
||||
- Local testing before push (fast feedback)
|
||||
- Clear pass/fail reporting
|
||||
- Smoke mode for quick validation
|
||||
- Helpful error messages
|
||||
|
||||
## Performance Impact
|
||||
|
||||
### Local Testing
|
||||
```
|
||||
Full test: ~2-3 minutes (all 7 configs)
|
||||
Smoke test: ~30 seconds (configure only)
|
||||
Specific: ~20-30 seconds (one config)
|
||||
```
|
||||
|
||||
### CI/CD
|
||||
- Tier 1 (standard CI): No change (~15 min as before)
|
||||
- Tier 2 (nightly): New, but off the critical path (~45 min)
|
||||
- No impact on PR merge latency
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Test fails locally
|
||||
```bash
|
||||
# See detailed output
|
||||
./scripts/test-config-matrix.sh --config <name> --verbose
|
||||
|
||||
# Check build log
|
||||
tail -50 build_matrix/<name>/build.log
|
||||
|
||||
# Check cmake log
|
||||
tail -50 build_matrix/<name>/config.log
|
||||
```
|
||||
|
||||
### Don't have dependencies
|
||||
```bash
|
||||
# Install dependencies per platform
|
||||
macOS: brew install [dep]
|
||||
Linux: apt-get install [dep]
|
||||
Windows: choco install [dep] or build with vcpkg
|
||||
```
|
||||
|
||||
### Windows gRPC issues
|
||||
```bash
|
||||
# ci-windows preset uses stable gRPC 1.67.1
|
||||
# If you use different version, you'll get ABI errors
|
||||
# Solution: Use preset or update validation rules
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Daily
|
||||
Check nightly matrix test results in GitHub Actions
|
||||
|
||||
### Weekly
|
||||
Review failure patterns and fix root causes
|
||||
|
||||
### Monthly
|
||||
Audit matrix configuration and documentation
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- Binary size tracking per configuration
|
||||
- Compile time benchmarks
|
||||
- Performance regression detection
|
||||
- Configuration recommendation tool
|
||||
- Web dashboard of results
|
||||
|
||||
## Questions?
|
||||
|
||||
| Question | Answer |
|
||||
|----------|--------|
|
||||
| How do I use this? | Read `QUICKSTART.md` |
|
||||
| What's tested? | See `configuration-matrix.md` Section 3 |
|
||||
| Why this approach? | Read `matrix-testing-strategy.md` |
|
||||
| How do I add a config? | Check `MATRIX_TESTING_IMPLEMENTATION.md` |
|
||||
|
||||
## Files Overview
|
||||
|
||||
```
|
||||
Documentation:
|
||||
✓ docs/internal/configuration-matrix.md
|
||||
→ All flags, dependencies, tested matrix
|
||||
|
||||
✓ docs/internal/testing/matrix-testing-strategy.md
|
||||
→ Philosophy, examples, integration guide
|
||||
|
||||
✓ docs/internal/testing/QUICKSTART.md
|
||||
→ One-page reference for developers
|
||||
|
||||
✓ MATRIX_TESTING_IMPLEMENTATION.md
|
||||
→ Complete implementation guide
|
||||
|
||||
✓ MATRIX_TESTING_CHECKLIST.md
|
||||
→ Status, next steps, responsibilities
|
||||
|
||||
Automation:
|
||||
✓ .github/workflows/matrix-test.yml
|
||||
→ Nightly/on-demand CI testing
|
||||
|
||||
✓ scripts/test-config-matrix.sh
|
||||
→ Local pre-push validation
|
||||
|
||||
✓ scripts/validate-cmake-config.sh
|
||||
→ Flag combination validation
|
||||
```
|
||||
|
||||
## Getting Started Now
|
||||
|
||||
1. **Read**: `docs/internal/testing/QUICKSTART.md` (5 min)
|
||||
2. **Run**: `./scripts/test-config-matrix.sh` (2 min)
|
||||
3. **Add to workflow**: Use before pushing (optional)
|
||||
4. **Monitor**: Check nightly results in GitHub Actions
|
||||
|
||||
---
|
||||
|
||||
**Status**: Ready to use
|
||||
**Local Testing**: `./scripts/test-config-matrix.sh`
|
||||
**CI Testing**: Automatic nightly + on-demand
|
||||
**Questions**: See the QUICKSTART guide
|
||||
|
||||
Last Updated: 2025-11-20
|
||||
Owner: CLAUDE_MATRIX_TEST
|
||||
Reference in New Issue
Block a user