diff --git a/docs/C2-testing-without-roms.md b/docs/C2-testing-without-roms.md new file mode 100644 index 00000000..98eb31d5 --- /dev/null +++ b/docs/C2-testing-without-roms.md @@ -0,0 +1,285 @@ +# Testing z3ed Without ROM Files + +**Last Updated:** October 10, 2025 +**Status:** Active + +## Overview + +The `z3ed` AI agent now supports **mock ROM mode** for testing without requiring actual ROM files. This is essential for: + +- **CI/CD pipelines** - No ROM files can be committed to GitHub +- **Development testing** - Quick iterations without ROM dependencies +- **Contributors** - Test the agent without needing to provide ROMs +- **Automated testing** - Consistent, reproducible test environments + +## How Mock ROM Mode Works + +Mock ROM mode creates a minimal but valid ROM structure with: +- ✅ Proper SNES header (LoROM mapping, 1MB size) +- ✅ All Zelda3 embedded labels (rooms, sprites, entrances, items, music, etc.) +- ✅ Resource label manager fully initialized +- ❌ No actual ROM data (tiles, graphics, maps are empty) + +This allows the AI agent to: +- Answer questions about room names, sprite IDs, entrance numbers +- Lookup labels and constants +- Test function calling and tool dispatch +- Validate agent logic without game data + +## Usage + +### Command Line Flag + +Add `--mock-rom` to any `z3ed agent` command: + +```bash +# Simple chat with mock ROM +z3ed agent simple-chat "What is room 5?" --mock-rom + +# Test conversation with mock ROM +z3ed agent test-conversation --mock-rom + +# AI provider testing +z3ed agent simple-chat "List all dungeons" --mock-rom --ai_provider=ollama +``` + +### Test Suite + +The `agent_test_suite.sh` script now defaults to mock ROM mode: + +```bash +# Run tests with mock ROM (default) +./scripts/agent_test_suite.sh ollama + +# Or with Gemini +./scripts/agent_test_suite.sh gemini +``` + +To use a real ROM instead, edit the script: + +```bash +USE_MOCK_ROM=false # At the top of agent_test_suite.sh +``` + +## What Works with Mock ROM + +### ✅ Fully Supported + +**Label Queries:** +- "What is room 5?" → "Tower of Hera - Moldorm Boss" +- "What sprites are in the game?" → Lists all 256 sprite names +- "What is entrance 0?" → "Link's House Main" +- "List all items" → Bow, Boomerang, Hookshot, etc. + +**Resource Lookups:** +- Room names (296 rooms) +- Entrance names (133 entrances) +- Sprite names (256 sprites) +- Overlord names (14 overlords) +- Overworld map names (160 maps) +- Item names +- Music track names +- Graphics sheet names + +**AI Testing:** +- Function calling / tool dispatch +- Natural language understanding +- Error handling +- Tool output parsing +- Multi-turn conversations + +### ⚠️ Limited Support + +**Queries Requiring Data:** +- "What tiles are used in room 5?" → No tile data in mock ROM +- "Show me the palette for map 0" → No palette data +- "What's at coordinate X,Y?" → No map data +- "Export graphics from dungeon 1" → No graphics data + +These queries will either return empty results or errors indicating no ROM data is available. + +### ❌ Not Supported + +**Operations That Modify ROM:** +- Editing tiles +- Changing palettes +- Modifying sprites +- Patching ROM data + +## Testing Strategy + +### For Agent Logic +Use **mock ROM** for testing: +- Function calling mechanisms +- Tool dispatch and routing +- Natural language understanding +- Error handling +- Label resolution +- Resource lookups + +### For ROM Operations +Use **real ROM** for testing: +- Tile editing +- Graphics manipulation +- Palette modifications +- Data extraction +- ROM patching + +## CI/CD Integration + +### GitHub Actions Example + +```yaml +name: Test z3ed Agent + +on: [push, pull_request] + +jobs: + test-agent: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + + - name: Install Dependencies + run: | + # Install ollama if testing local models + curl -fsSL https://ollama.ai/install.sh | sh + ollama pull qwen2.5-coder + + - name: Build z3ed + run: | + cmake -B build_test + cmake --build build_test --parallel + + - name: Run Agent Tests (Mock ROM) + run: | + ./scripts/agent_test_suite.sh ollama + env: + # Or use Gemini with API key + GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }} +``` + +## Embedded Labels Reference + +Mock ROM includes all these labels from `zelda3::Zelda3Labels`: + +| Resource Type | Count | Example | +|--------------|-------|---------| +| Rooms | 296 | "Sewer - Throne Room" | +| Entrances | 133 | "Link's House Main" | +| Sprites | 256 | "Moldorm (Boss)" | +| Overlords | 14 | "Overlord - Agahnim's Barrier" | +| Overworld Maps | 160 | "Light World - Hyrule Castle" | +| Items | 64+ | "Bow", "Boomerang", "Hookshot" | +| Music Tracks | 64+ | "Title Theme", "Overworld", "Dark World" | +| Graphics Sheets | 128+ | "Link Sprites", "Enemy Pack 1" | + +See `src/app/zelda3/zelda3_labels.h` for the complete list. + +## Troubleshooting + +### "No ROM loaded" error + +Make sure you're using the `--mock-rom` flag: + +```bash +# Wrong +z3ed agent simple-chat "test" + +# Correct +z3ed agent simple-chat "test" --mock-rom +``` + +### Mock ROM fails to initialize + +Check the error message. Common issues: +- Build system didn't include `mock_rom.cc` +- Missing `zelda3_labels.cc` in build +- Linker errors with resource labels + +### Agent returns empty/wrong results + +Remember: Mock ROM has **labels only**, no actual game data. + +Queries like "What tiles are in room 5?" won't work because there's no tile data. + +Use queries about labels and IDs instead: "What is the name of room 5?" + +## Development + +### Adding New Labels + +To add new label types to mock ROM: + +1. **Add to `zelda3_labels.h`:** +```cpp +static const std::vector& GetNewResourceNames(); +``` + +2. **Implement in `zelda3_labels.cc`:** +```cpp +const std::vector& Zelda3Labels::GetNewResourceNames() { + static std::vector names = {"Item1", "Item2", ...}; + return names; +} +``` + +3. **Add to `ToResourceLabels()`:** +```cpp +const auto& new_resources = GetNewResourceNames(); +for (size_t i = 0; i < new_resources.size(); ++i) { + labels["new_resource"][std::to_string(i)] = new_resources[i]; +} +``` + +4. **Rebuild:** +```bash +cmake --build build --parallel +``` + +### Testing Mock ROM Directly + +```cpp +#include "cli/handlers/mock_rom.h" + +Rom rom; +auto status = InitializeMockRom(rom); +if (status.ok()) { + // ROM is ready with all labels + auto* label_mgr = rom.resource_label(); + std::string room_name = label_mgr->GetLabel("room", "5"); + // room_name == "Tower of Hera - Moldorm Boss" +} +``` + +## Best Practices + +### DO ✅ +- Use mock ROM for CI/CD and automated tests +- Use mock ROM for agent logic development +- Use mock ROM when contributing (no ROM files needed) +- Test with real ROM before releasing features +- Document which features require real ROM data + +### DON'T ❌ +- Commit ROM files to Git (legal issues) +- Assume mock ROM has actual game data +- Use mock ROM for testing data extraction +- Skip real ROM testing entirely + +## Related Documentation + +- [C1: z3ed Agent Guide](C1-z3ed-agent-guide.md) - Main agent documentation +- [A1: Testing Guide](A1-testing-guide.md) - General testing strategy +- [E3: API Reference](E3-api-reference.md) - ROM API documentation + +--- + +**Implementation Status:** ✅ Complete +**Since Version:** v0.3.3 +**Files:** +- `src/cli/handlers/mock_rom.h` +- `src/cli/handlers/mock_rom.cc` +- `src/cli/flags.cc` (--mock-rom flag) +- `scripts/agent_test_suite.sh` (updated) + diff --git a/docs/index.md b/docs/index.md index 60cc5c75..05d983eb 100644 --- a/docs/index.md +++ b/docs/index.md @@ -15,6 +15,7 @@ Welcome to the official documentation for yaze, a comprehensive ROM editor for T ## C: `z3ed` CLI - [C1: `z3ed` Agent Guide](C1-z3ed-agent-guide.md) - The AI-powered command-line interface. +- [C2: Testing Without ROMs](C2-testing-without-roms.md) - Using mock ROM mode for testing and CI/CD. ## E: Development & API - [E1: Assembly Style Guide](E1-asm-style-guide.md) - 65816 assembly coding standards. diff --git a/scripts/agent_test_suite.sh b/scripts/agent_test_suite.sh index 1963b8f4..f4de6259 100755 --- a/scripts/agent_test_suite.sh +++ b/scripts/agent_test_suite.sh @@ -10,8 +10,8 @@ BLUE='\033[0;34m' NC='\033[0m' # No Color Z3ED="./build_test/bin/z3ed" -ROM="assets/zelda3.sfc" RESULTS_FILE="/tmp/z3ed_ai_test_results.txt" +USE_MOCK_ROM=true # Set to false if you want to test with a real ROM echo "==========================================" echo " Z3ED AI Provider Test Suite" @@ -22,7 +22,6 @@ echo "" > "$RESULTS_FILE" # --- Pre-flight Checks --- -print_header "Performing Pre-flight Checks" if [ -z "$1" ]; then echo "❌ Error: No AI provider specified." @@ -32,28 +31,34 @@ fi PROVIDER=$1 echo "✅ Provider: $PROVIDER" -# Check binaries and files -for f in "$Z3ED_BIN" "$ROM_PATH" "$TEST_DIR/../prompt_catalogue.yaml" "$TEST_DIR/function_schemas.json"; do - if [ ! -f "$f" ]; then - echo -e "${RED}✗ Prerequisite file not found: $f${NC}" - exit 1 - fi -done -echo "✅ Core binaries and files found." - -# Verify schemas -if python3 -m json.tool "$TEST_DIR/function_schemas.json" > /dev/null 2>&1; then - echo "✅ Function schemas JSON is valid." -else - echo "${RED}✗ Invalid JSON in function_schemas.json${NC}" +# Check binary exists +if [ ! -f "$Z3ED" ]; then + echo -e "${RED}✗ z3ed binary not found at: $Z3ED${NC}" + echo "Run: cmake --build build_test" exit 1 fi +echo "✅ z3ed binary found" -# Verify manual tool execution -if "$Z3ED_BIN" agent overworld-find-tile --tile 0x02E --format json --rom "$ROM_PATH" > /dev/null 2>&1; then - echo "✅ Manual tool execution successful." +# Set ROM flags based on mode +if [ "$USE_MOCK_ROM" = true ]; then + ROM_FLAGS="--mock-rom" + echo "✅ Using mock ROM mode (no ROM file required)" else - echo "${RED}✗ Manual tool execution failed.${NC}" + ROM="assets/zelda3.sfc" + if [ ! -f "$ROM" ]; then + echo -e "${RED}✗ ROM file not found: $ROM${NC}" + echo "Tip: Use mock ROM mode by setting USE_MOCK_ROM=true" + exit 1 + fi + ROM_FLAGS="--rom=\"$ROM\"" + echo "✅ Real ROM found: $ROM" +fi + +# Verify z3ed can execute +if "$Z3ED" --help > /dev/null 2>&1; then + echo "✅ z3ed executable works" +else + echo "${RED}✗ z3ed failed to execute${NC}" exit 1 fi @@ -107,7 +112,7 @@ run_test() { echo "Query: $query" echo "" - local cmd="$Z3ED agent simple-chat \"$query\" --rom=\"$ROM\" --ai_provider=$provider $extra_args" + local cmd="$Z3ED agent simple-chat \"$query\" $ROM_FLAGS --ai_provider=$provider $extra_args" echo "Running: $cmd" echo "" diff --git a/src/cli/agent.cmake b/src/cli/agent.cmake index fcc97057..9f16a243 100644 --- a/src/cli/agent.cmake +++ b/src/cli/agent.cmake @@ -97,6 +97,7 @@ set(YAZE_AGENT_SOURCES cli/service/resources/resource_context_builder.cc cli/handlers/overworld_inspect.cc cli/handlers/message.cc + cli/handlers/mock_rom.cc cli/flags.cc cli/service/rom/rom_sandbox_manager.cc ) diff --git a/src/cli/flags.cc b/src/cli/flags.cc index ceb3b52c..f713ff42 100644 --- a/src/cli/flags.cc +++ b/src/cli/flags.cc @@ -3,6 +3,9 @@ #include "absl/flags/flag.h" ABSL_FLAG(std::string, rom, "", "Path to the ROM file"); +ABSL_FLAG(bool, mock_rom, false, + "Use mock ROM mode for testing without requiring an actual ROM file. " + "Loads all Zelda3 embedded labels but no actual ROM data."); // AI Service Configuration Flags ABSL_FLAG(std::string, ai_provider, "auto", diff --git a/src/cli/handlers/agent/conversation_test.cc b/src/cli/handlers/agent/conversation_test.cc index 8fd80fee..473bdabd 100644 --- a/src/cli/handlers/agent/conversation_test.cc +++ b/src/cli/handlers/agent/conversation_test.cc @@ -1,6 +1,7 @@ #include "cli/handlers/agent/commands.h" #include "app/rom.h" #include "app/core/project.h" +#include "cli/handlers/mock_rom.h" #include "absl/flags/declare.h" #include "absl/flags/flag.h" @@ -16,6 +17,7 @@ #include "nlohmann/json.hpp" ABSL_DECLARE_FLAG(std::string, rom); +ABSL_DECLARE_FLAG(bool, mock_rom); namespace yaze { namespace cli { @@ -28,10 +30,23 @@ absl::Status LoadRomForAgent(Rom& rom) { return ::absl::OkStatus(); } + // Check if mock ROM mode is enabled + bool use_mock = ::absl::GetFlag(FLAGS_mock_rom); + if (use_mock) { + // Initialize mock ROM with embedded labels + auto status = InitializeMockRom(rom); + if (!status.ok()) { + return status; + } + std::cout << "✅ Mock ROM initialized with embedded Zelda3 labels\n"; + return ::absl::OkStatus(); + } + + // Otherwise load from file std::string rom_path = ::absl::GetFlag(FLAGS_rom); if (rom_path.empty()) { return ::absl::InvalidArgumentError( - "No ROM loaded. Pass --rom= to z3ed agent test-conversation."); + "No ROM loaded. Pass --rom= or use --mock-rom for testing."); } auto status = rom.LoadFromFile(rom_path); diff --git a/src/cli/handlers/agent/tool_commands.cc b/src/cli/handlers/agent/tool_commands.cc index 0ee62045..a2845c35 100644 --- a/src/cli/handlers/agent/tool_commands.cc +++ b/src/cli/handlers/agent/tool_commands.cc @@ -23,12 +23,14 @@ #include "app/zelda3/dungeon/room.h" #include "app/zelda3/overworld/overworld.h" #include "cli/handlers/message.h" +#include "cli/handlers/mock_rom.h" #include "cli/handlers/overworld_inspect.h" #include "cli/service/resources/resource_context_builder.h" #include "nlohmann/json.hpp" #include "util/macro.h" ABSL_DECLARE_FLAG(std::string, rom); +ABSL_DECLARE_FLAG(bool, mock_rom); namespace yaze { namespace cli { @@ -37,10 +39,22 @@ namespace agent { namespace { absl::StatusOr LoadRomFromFlag() { + // Check if mock ROM mode is enabled + bool use_mock = absl::GetFlag(FLAGS_mock_rom); + if (use_mock) { + Rom rom; + auto status = InitializeMockRom(rom); + if (!status.ok()) { + return status; + } + return rom; + } + + // Otherwise load from file std::string rom_path = absl::GetFlag(FLAGS_rom); if (rom_path.empty()) { return absl::FailedPreconditionError( - "No ROM loaded. Use --rom= to specify ROM file."); + "No ROM loaded. Use --rom= or --mock-rom for testing."); } Rom rom; diff --git a/src/cli/handlers/mock_rom.cc b/src/cli/handlers/mock_rom.cc new file mode 100644 index 00000000..82116c7d --- /dev/null +++ b/src/cli/handlers/mock_rom.cc @@ -0,0 +1,87 @@ +#include "cli/handlers/mock_rom.h" + +#include + +#include "absl/flags/declare.h" +#include "absl/flags/flag.h" +#include "absl/strings/str_format.h" +#include "app/core/project.h" +#include "app/zelda3/zelda3_labels.h" + +ABSL_DECLARE_FLAG(bool, mock_rom); + +namespace yaze { +namespace cli { + +absl::Status InitializeMockRom(Rom& rom) { + // Create a minimal but valid SNES ROM header + // Zelda3 is a 1MB ROM (0x100000 bytes) in LoROM mapping + constexpr size_t kMockRomSize = 0x100000; // 1MB + std::vector mock_data(kMockRomSize, 0x00); + + // SNES header is at 0x7FC0 for LoROM + constexpr size_t kHeaderOffset = 0x7FC0; + + // Set ROM title (21 bytes at 0x7FC0) + const char* title = "YAZE MOCK ROM TEST "; // 21 chars including spaces + for (size_t i = 0; i < 21; ++i) { + mock_data[kHeaderOffset + i] = title[i]; + } + + // ROM makeup byte (0x7FD5): $20 = LoROM, no special chips + mock_data[kHeaderOffset + 0x15] = 0x20; + + // ROM type (0x7FD6): $00 = ROM only + mock_data[kHeaderOffset + 0x16] = 0x00; + + // ROM size (0x7FD7): $09 = 1MB (2^9 KB = 512 KB = 1MB with header) + mock_data[kHeaderOffset + 0x17] = 0x09; + + // SRAM size (0x7FD8): $03 = 8KB (Zelda3 standard) + mock_data[kHeaderOffset + 0x18] = 0x03; + + // Country code (0x7FD9): $01 = USA + mock_data[kHeaderOffset + 0x19] = 0x01; + + // Developer ID (0x7FDA): $33 = Extended header (Zelda3) + mock_data[kHeaderOffset + 0x1A] = 0x33; + + // Version number (0x7FDB): $00 = 1.0 + mock_data[kHeaderOffset + 0x1B] = 0x00; + + // Checksum complement (0x7FDC-0x7FDD): We'll leave as 0x0000 for mock + // Checksum (0x7FDE-0x7FDF): We'll leave as 0x0000 for mock + + // Load the mock data into the ROM + auto load_status = rom.LoadFromData(mock_data); + if (!load_status.ok()) { + return absl::InternalError( + absl::StrFormat("Failed to initialize mock ROM: %s", + load_status.message())); + } + + // Initialize embedded labels so queries work without actual ROM data + core::YazeProject project; + auto labels_status = project.InitializeEmbeddedLabels(); + if (!labels_status.ok()) { + return absl::InternalError( + absl::StrFormat("Failed to initialize embedded labels: %s", + labels_status.message())); + } + + // Attach labels to ROM's resource label manager + if (rom.resource_label()) { + rom.resource_label()->labels_ = project.resource_labels; + rom.resource_label()->labels_loaded_ = true; + } + + return absl::OkStatus(); +} + +bool ShouldUseMockRom() { + return absl::GetFlag(FLAGS_mock_rom); +} + +} // namespace cli +} // namespace yaze + diff --git a/src/cli/handlers/mock_rom.h b/src/cli/handlers/mock_rom.h new file mode 100644 index 00000000..c9d2d7d8 --- /dev/null +++ b/src/cli/handlers/mock_rom.h @@ -0,0 +1,35 @@ +#ifndef YAZE_CLI_HANDLERS_MOCK_ROM_H +#define YAZE_CLI_HANDLERS_MOCK_ROM_H + +#include "absl/status/status.h" +#include "app/rom.h" + +namespace yaze { +namespace cli { + +/** + * @brief Initialize a mock ROM for testing without requiring an actual ROM file + * + * This creates a minimal but valid ROM structure populated with: + * - All Zelda3 embedded labels (rooms, sprites, entrances, items, etc.) + * - Minimal header data to satisfy ROM validation + * - Empty but properly sized data sections + * + * Purpose: Allow AI agent testing and CI/CD without committing ROM files + * + * @param rom ROM object to initialize as mock + * @return absl::OkStatus() on success, error status on failure + */ +absl::Status InitializeMockRom(Rom& rom); + +/** + * @brief Check if mock ROM mode should be used based on flags + * @return true if --mock-rom flag is set + */ +bool ShouldUseMockRom(); + +} // namespace cli +} // namespace yaze + +#endif // YAZE_CLI_HANDLERS_MOCK_ROM_H +