yaze/docs/internal/plans/message_editor_implementation_roadmap.md

Message Editor Implementation Roadmap

Status: Active Development Last Updated: 2025-11-21 Owner: Frontend/UI Team Related Docs:

• docs/internal/architecture/message_system.md (Gemini's architecture vision)
• docs/internal/plans/message_system_improvement_plan.md (Gemini's feature proposals)

---

Executive Summary

This roadmap bridges Gemini's architectural vision with practical implementation steps for completing the Message Editor. The current implementation has the core foundation in place (message parsing, dictionary system, preview rendering) but lacks several key features proposed in Gemini's plan, particularly around JSON import/export, translation workflows, and theme integration.

---

Current State Analysis

What's Working (Completed Features)

Core Data Layer ✅

• MessageData: Full implementation with raw/parsed representations
• DictionaryEntry: Compression system with dictionary optimization
• TextElement: Command and special character parsing
• Character Encoding: Complete CharEncoder table (0x00-0x66)
• ROM Reading: ReadAllTextData() successfully loads all 396 messages
• ROM Writing: Save() handles two-bank text data with overflow detection

Message Preview System ✅

• MessagePreview: Live rendering of messages as they appear in-game
• Font Graphics: 2BPP font tiles loaded and displayed at 0x70000
• Character Widths: Proportional font support via width table at 0x74ADF
• Preview Bitmap: Real-time message rendering with proper palette support

Editor UI ✅

• Card System: Four dockable cards (Message List, Editor, Font Atlas, Dictionary)
• Message List: Table view with ID, contents, and address columns
• Text Editor: Multiline input with live preview updates
• Command Insertion: Buttons to insert text commands and special characters
• Dictionary Display: Read-only view of all 97 dictionary entries
• Expanded Messages: Basic support for loading external message bins

Testing Coverage ✅

• Unit Tests: 20+ tests covering parsing, encoding, dictionary optimization
• Integration Tests: ROM-dependent tests verify actual game data
• Command Parsing: Regression tests for argument handling bugs

CLI Integration ✅

• Message List: z3ed message list --format json --range 0-100
• Message Read: z3ed message read --id 5 --format json
• Message Search: z3ed message search --query "Link"
• Message Stats: z3ed message stats --format json

What's Missing (Gaps vs. Gemini's Vision)

1. JSON Import/Export ❌ (HIGH PRIORITY)

Status: Not implemented Gemini's Vision:

[
  {
    "id": 0,
    "address": 917504,
    "text": "[W:00][SPD:00]Welcome to [D:05]...",
    "context": "Uncle dying in sewers"
  }
]

Current Gap:

• No SerializeMessages() or DeserializeMessages() in MessageData
• No UI for export/import operations
• No context field for translator notes
• CLI has JSON output but not JSON input

Impact: Cannot version control text, cannot use external editors, cannot collaborate with translators

---

2. Translation Workspace ❌ (MEDIUM PRIORITY)

Status: Not implemented Gemini's Vision: Side-by-side view with reference ROM/JSON and editable translation

Current Gap:

• No reference text display
• No side-by-side layout
• No translation progress tracking
• No language-specific dictionary optimization

Impact: Manual translation workflows are tedious and error-prone

---

3. Search & Replace ⚠️ (PARTIAL)

Status: Stub implementation exists Gemini's Vision: Regex support, batch replace across all messages

Current Implementation:

• Find() method exists in MessageEditor (lines 574-600)
• Basic UI skeleton present (search input, case sensitivity toggle)
• Missing: Replace functionality, regex support, "Find All", multi-message operations

Impact: Global text edits require manual per-message changes

---

4. Theme Integration ❌ (LOW PRIORITY - UI POLISH)

Status: Not implemented Current Issues:

• No hardcoded ImVec4 colors found (GOOD!)
• Not using AgentUITheme system for consistency
• Missing semantic color names for message editor components

Impact: Message Editor UI may not match rest of application theme

---

5. Expanded ROM Support ⚠️ (PARTIAL)

Status: Basic implementation exists Gemini's Vision: Repointing text blocks to expanded ROM space (Banks 10+), automatic bank switching

Current Implementation:

• Can load expanded message bins (lines 322-334)
• Can save expanded messages (lines 497-508)
• Missing: Repointing logic, bank management, automatic overflow handling

Impact: Cannot support large translation projects that exceed vanilla space

---

6. Scripting Integration ❌ (FUTURE)

Status: Not planned Gemini's Vision: Lua/Python API for procedural text generation

Current Gap: No scripting hooks in message system

Impact: Low - nice-to-have for advanced users

---

Architectural Decisions Required

Decision 1: JSON Schema Design

Question: What fields should the JSON export include?

Proposal:

{
  "version": "1.0",
  "rom_name": "zelda3.sfc",
  "messages": [
    {
      "id": 0,
      "address": 917504,
      "address_hex": "0xE0000",
      "text": "[W:00][SPD:00]Welcome...",
      "context": "Optional translator note",
      "dictionary_optimized": true,
      "expanded": false
    }
  ],
  "dictionary": [
    {
      "id": 0,
      "token": "[D:00]",
      "contents": "the"
    }
  ]
}

Trade-offs:

• Verbose but human-readable
• Includes metadata for validation
• Context field for translator workflow

Status: ✅ RECOMMENDED

---

Decision 2: Translation Workspace Layout

Question: How should reference vs. translation be displayed?

Option A: Side-by-side split pane

┌────────────────┬────────────────┐
│ Reference      │ Translation    │
│ (English)      │ (Spanish)      │
│ [Read-only]    │ [Editable]     │
│                │                │
│ Message 0:     │ Message 0:     │
│ "Welcome to    │ "Bienvenido a  │
│  Hyrule"       │  Hyrule"       │
└────────────────┴────────────────┘

Option B: Top-bottom with context panel

┌────────────────────────────────┐
│ Reference: "Welcome to Hyrule" │
│ Context: Uncle's dying words   │
├────────────────────────────────┤
│ Translation:                   │
│ [Editable text box]            │
└────────────────────────────────┘

Recommendation: Option A for large screens, Option B for narrow windows

Status: ⚠️ NEEDS USER FEEDBACK

---

Decision 3: Dictionary Auto-Optimization

Question: Should we auto-generate optimal dictionary for new languages?

Challenges:

• Dictionary optimization is NP-hard (longest common substring problem)
• Need to preserve ROM space constraints (97 entries max)
• Different languages have different common phrases

Proposal:

1. Provide "Analyze Translation" button that suggests optimal dictionary
2. Let user accept/reject suggestions
3. Preserve manual dictionary entries

Status: ⚠️ NEEDS RESEARCH

---

Implementation Priority Matrix

Phase 1: Foundation (Sprint 1-2 weeks)

Goal: JSON import/export with UI integration

Task 1.1: Implement JSON Serialization

Location: src/app/editor/message/message_data.h, message_data.cc Priority: P0 (Blocker for translation workflow) Estimated Effort: 3 days

Implementation:

// In MessageData
nlohmann::json SerializeToJson() const;
static absl::StatusOr<MessageData> DeserializeFromJson(const nlohmann::json& j);

// Free functions
absl::Status ExportMessagesToJson(
    const std::vector<MessageData>& messages,
    const std::vector<DictionaryEntry>& dictionary,
    const std::string& output_path);

absl::StatusOr<std::vector<MessageData>> ImportMessagesFromJson(
    const std::string& input_path);

Dependencies: nlohmann/json (already in project via CPM)

Acceptance Criteria:

• Export all 396 messages to valid JSON
• Import JSON and verify byte-for-byte ROM equivalence
• Handle malformed JSON with clear error messages
• Preserve dictionary optimization
• Include context field in schema

---

Task 1.2: Add Export/Import UI

Location: src/app/editor/message/message_editor.cc Priority: P0 Estimated Effort: 2 days

UI Additions:

void MessageEditor::DrawExportImportPanel() {
  if (ImGui::Button("Export to JSON")) {
    std::string path = util::FileDialogWrapper::ShowSaveFileDialog("json");
    PRINT_IF_ERROR(ExportMessagesToJson(list_of_texts_,
                                        message_preview_.all_dictionaries_,
                                        path));
  }

  if (ImGui::Button("Import from JSON")) {
    std::string path = util::FileDialogWrapper::ShowOpenFileDialog();
    auto result = ImportMessagesFromJson(path);
    if (result.ok()) {
      list_of_texts_ = result.value();
      RefreshMessageList();
    }
  }
}

Acceptance Criteria:

• File dialogs open with correct filters
• Progress indicator for large exports
• Confirmation dialog on import (warns about overwriting)
• Error popup on import failure with details

---

Task 1.3: CLI JSON Import Support

Location: src/cli/handlers/game/message.cc Priority: P1 Estimated Effort: 1 day

Implementation:

z3ed message import --json messages.json --rom zelda3.sfc --output zelda3_translated.sfc

Acceptance Criteria:

• Import JSON and write to ROM
• Validate JSON schema before import
• Verify ROM size constraints
• Dry-run mode (validate without writing)

---

Phase 2: Translation Workflow (Sprint 2-3 weeks)

Goal: Side-by-side translation UI

Task 2.1: Add Translation Mode Card

Location: src/app/editor/message/message_editor.h, message_editor.cc Priority: P1 Estimated Effort: 5 days

New Components:

class TranslationWorkspace {
 public:
  void Initialize(Rom* reference_rom, Rom* translation_rom);
  void DrawUI();
  void LoadReferenceFromJson(const std::string& path);

 private:
  void DrawSideBySideView();
  void DrawProgressTracker();
  void UpdateTranslationProgress();

  std::vector<MessageData> reference_messages_;
  std::vector<MessageData> translation_messages_;
  std::map<int, bool> translation_complete_flags_;
  Rom* reference_rom_ = nullptr;
  Rom* translation_rom_ = nullptr;
};

UI Mockup:

┌─────────────────────────────────────────────────┐
│ Translation Progress: 42/396 (10.6%)           │
├──────────────────────┬──────────────────────────┤
│ Reference (EN)       │ Translation (ES)         │
├──────────────────────┼──────────────────────────┤
│ Message 0:           │ Message 0:               │
│ "Welcome to Hyrule"  │ [Editable input box]     │
│                      │                          │
│ Dictionary: [D:05]   │ Dictionary: [D:05]       │
├──────────────────────┴──────────────────────────┤
│ [Previous] [Next] [Mark Complete] [Skip]       │
└─────────────────────────────────────────────────┘

Acceptance Criteria:

• Load reference ROM or JSON
• Display messages side-by-side
• Track translation progress (per-message completion)
• Keyboard shortcuts for navigation (Ctrl+N, Ctrl+P)
• Auto-save translated ROM on completion

---

Task 2.2: Context/Notes System

Location: src/app/editor/message/message_data.h Priority: P2 Estimated Effort: 2 days

Schema Addition:

struct MessageData {
  // ... existing fields ...
  std::string context;  // Translator notes, scene context
  std::string screenshot_path;  // Optional screenshot reference

  nlohmann::json SerializeToJson() const {
    return {
      {"id", ID},
      {"address", Address},
      {"text", RawString},
      {"context", context},
      {"screenshot", screenshot_path}
    };
  }
};

UI Addition:

void MessageEditor::DrawContextPanel() {
  ImGui::InputTextMultiline("Context Notes", &current_message_.context);
  if (!current_message_.screenshot_path.empty()) {
    ImGui::Image(LoadScreenshot(current_message_.screenshot_path));
  }
}

Acceptance Criteria:

• Context field persists in JSON export/import
• Context displayed in translation workspace
• Optional screenshot attachment (stored as relative path)

---

Phase 3: Search & Replace (Sprint 3-1 week)

Goal: Complete Find/Replace implementation

Task 3.1: Implement Replace Functionality

Location: src/app/editor/message/message_editor.cc Priority: P2 Estimated Effort: 2 days

Implementation:

absl::Status MessageEditor::Replace(const std::string& find_text,
                                    const std::string& replace_text,
                                    bool case_sensitive,
                                    bool whole_word,
                                    bool all_messages) {
  int replaced_count = 0;

  if (all_messages) {
    for (auto& message : list_of_texts_) {
      replaced_count += ReplaceInMessage(message, find_text, replace_text,
                                        case_sensitive, whole_word);
    }
  } else {
    replaced_count += ReplaceInMessage(current_message_, find_text,
                                       replace_text, case_sensitive, whole_word);
  }

  return absl::OkStatus();
}

UI Updates:

void MessageEditor::DrawFindReplacePanel() {
  static char find_text[256] = "";
  static char replace_text[256] = "";

  ImGui::InputText("Find", find_text, IM_ARRAYSIZE(find_text));
  ImGui::InputText("Replace", replace_text, IM_ARRAYSIZE(replace_text));

  ImGui::Checkbox("Case Sensitive", &case_sensitive_);
  ImGui::Checkbox("Whole Word", &match_whole_word_);
  ImGui::Checkbox("All Messages", &replace_all_messages_);

  if (ImGui::Button("Replace")) {
    PRINT_IF_ERROR(Replace(find_text, replace_text, case_sensitive_,
                          match_whole_word_, false));
  }

  if (ImGui::Button("Replace All")) {
    PRINT_IF_ERROR(Replace(find_text, replace_text, case_sensitive_,
                          match_whole_word_, true));
  }
}

Acceptance Criteria:

• Replace in current message
• Replace in all messages
• Case-sensitive/insensitive options
• Whole word matching
• Undo support (requires history stack)

---

Task 3.2: Add Regex Support

Location: src/app/editor/message/message_editor.cc Priority: P3 (Nice-to-have) Estimated Effort: 2 days

Implementation:

absl::Status MessageEditor::ReplaceRegex(const std::string& pattern,
                                         const std::string& replacement,
                                         bool all_messages) {
  std::regex regex_pattern;
  try {
    regex_pattern = std::regex(pattern);
  } catch (const std::regex_error& e) {
    return absl::InvalidArgumentError(
        absl::StrFormat("Invalid regex: %s", e.what()));
  }

  // Perform replacement...
}

Acceptance Criteria:

• Regex validation with error messages
• Capture group support ($1, $2, etc.)
• Preview matches before replacement

---

Phase 4: UI Polish (Sprint 4-1 week)

Goal: Theme integration and UX improvements

Task 4.1: Integrate AgentUITheme

Location: src/app/editor/message/message_editor.cc Priority: P3 Estimated Effort: 1 day

Implementation:

void MessageEditor::DrawMessageList() {
  const auto& theme = AgentUI::GetTheme();

  AgentUI::PushPanelStyle();
  ImGui::PushStyleColor(ImGuiCol_Header, theme.panel_bg_darker);
  ImGui::PushStyleColor(ImGuiCol_HeaderHovered, theme.accent_color);

  // ... table rendering ...

  ImGui::PopStyleColor(2);
  AgentUI::PopPanelStyle();
}

Acceptance Criteria:

• All panels use AgentUI::PushPanelStyle()
• Section headers use AgentUI::RenderSectionHeader()
• Buttons use AgentUI::StyledButton() where appropriate
• Color scheme matches rest of editor

---

Task 4.2: Add Keyboard Shortcuts

Location: src/app/editor/message/message_editor.cc Priority: P3 Estimated Effort: 1 day

Shortcuts:

• Ctrl+F: Open Find/Replace
• Ctrl+E: Export to JSON
• Ctrl+I: Import from JSON
• Ctrl+S: Save ROM
• Ctrl+N: Next message (in translation mode)
• Ctrl+P: Previous message (in translation mode)

Implementation:

void MessageEditor::HandleKeyboardShortcuts() {
  if (ImGui::IsKeyPressed(ImGuiKey_F) &&
      ImGui::GetIO().KeyCtrl) {
    show_find_replace_ = true;
  }

  // ... other shortcuts ...
}

Acceptance Criteria:

• Shortcuts don't conflict with global shortcuts
• Shortcuts displayed in tooltips
• Configurable shortcuts (future enhancement)

---

Test Strategy

Unit Tests

Location: test/unit/message_data_test.cc (new file)

TEST(MessageDataTest, SerializeToJson_BasicMessage) {
  MessageData msg;
  msg.ID = 0;
  msg.Address = 0xE0000;
  msg.RawString = "Hello World";
  msg.context = "Test message";

  auto json = msg.SerializeToJson();

  EXPECT_EQ(json["id"], 0);
  EXPECT_EQ(json["text"], "Hello World");
  EXPECT_EQ(json["context"], "Test message");
}

TEST(MessageDataTest, DeserializeFromJson_RoundTrip) {
  MessageData original;
  original.ID = 5;
  original.RawString = "[W:00][K]Test";

  auto json = original.SerializeToJson();
  auto result = MessageData::DeserializeFromJson(json);

  ASSERT_TRUE(result.ok());
  EXPECT_EQ(result.value().ID, 5);
  EXPECT_EQ(result.value().RawString, "[W:00][K]Test");
}

Integration Tests

Location: test/integration/message_export_test.cc (new file)

TEST_F(MessageRomTest, ExportImport_RoundTrip) {
  // Export all messages to JSON
  std::string json_path = "/tmp/messages.json";
  EXPECT_OK(ExportMessagesToJson(list_of_texts_, dictionary_, json_path));

  // Import back
  auto imported = ImportMessagesFromJson(json_path);
  ASSERT_TRUE(imported.ok());

  // Verify identical
  EXPECT_EQ(imported.value().size(), list_of_texts_.size());
  for (size_t i = 0; i < list_of_texts_.size(); ++i) {
    EXPECT_EQ(imported.value()[i].RawString, list_of_texts_[i].RawString);
  }
}

E2E Tests

Location: test/e2e/message_editor_workflow_test.cc (new file)

TEST_F(MessageEditorE2ETest, TranslationWorkflow) {
  // Open translation workspace
  EXPECT_OK(ClickButton("Translation Mode"));

  // Load reference ROM
  EXPECT_OK(OpenFileDialog("reference_rom.sfc"));

  // Navigate to message 0
  EXPECT_EQ(GetCurrentMessageID(), 0);

  // Edit translation
  EXPECT_OK(SetTextBoxValue("Bienvenido a Hyrule"));

  // Mark complete
  EXPECT_OK(ClickButton("Mark Complete"));

  // Verify progress
  EXPECT_EQ(GetTranslationProgress(), "1/396");
}

---

Dependencies & Risks

External Dependencies

1. nlohmann/json: Already integrated via CPM ✅
2. ImGui Test Engine: Available for E2E tests ✅
3. File Dialog: util::FileDialogWrapper already exists ✅

Technical Risks

Risk 1: JSON Schema Evolution

Impact: Breaking changes to JSON format Mitigation:

• Include version number in schema
• Implement forward/backward compatibility
• Provide migration tool for old exports

Risk 2: Dictionary Auto-Optimization Complexity

Impact: Algorithm may be too slow for real-time use Mitigation:

• Run optimization in background thread
• Provide progress indicator
• Allow cancellation

Risk 3: Large ROM Size with Expanded Messages

Impact: May exceed bank boundaries Mitigation:

• Implement repointing logic early (Phase 5)
• Warn user when approaching limits
• Suggest dictionary optimization

---

Success Metrics

Quantitative Metrics

• 100% message export/import success rate (no data loss)
• JSON schema supports all 396 vanilla messages
• Translation workspace reduces edit time by 50% vs. current workflow
• Search/Replace handles batch operations in <1 second
• 90%+ test coverage for new code

Qualitative Metrics

• Translator feedback: "Translation workflow is intuitive"
• No hardcoded colors in Message Editor
• UI matches yaze style guide
• Documentation complete for all new features

---

Timeline Estimate

Phase	Duration	Effort (Dev Days)
Phase 1: JSON Export/Import	2 weeks	6 days
Phase 2: Translation Workspace	3 weeks	9 days
Phase 3: Search & Replace	1 week	4 days
Phase 4: UI Polish	1 week	2 days
Total	7 weeks	21 dev days

Note: Timeline assumes single developer working full-time. Adjust for part-time work or team collaboration.

---

Future Enhancements (Post-MVP)

1. Scripting API (Gemini's vision)

   • Expose MessageData to Lua/Python
   • Allow procedural text generation
   • Useful for randomizers

2. Cloud Translation Integration

   • Google Translate API for quick drafts
   • DeepL API for quality translations
   • Requires API key management

3. Collaborative Editing

   • Real-time multi-user translation
   • Conflict resolution for concurrent edits
   • Requires network infrastructure

4. ROM Patch Generation

   • Export as .ips or .bps patch files
   • Useful for distribution without ROM sharing
   • Requires patch generation library

5. Message Validation

   • Check for overlong messages (exceeds textbox width)
   • Verify all messages have terminators
   • Flag unused dictionary entries

---

Open Questions

1. Q: Should we support multiple translation languages simultaneously? A: TBD - May require multi-ROM workspace UI

2. Q: How should we handle custom dictionary entries for expanded ROMs? A: TBD - Need research into ROM space allocation

3. Q: Should translation progress be persisted? A: TBD - Could store in .yaze project file

4. Q: Do we need undo/redo for message editing? A: TBD - ImGui InputTextMultiline has built-in undo, may be sufficient

---

Conclusion

The Message Editor has a solid foundation with core parsing, preview, and UI systems in place. The main gaps are JSON export/import (P0), translation workspace (P1), and search/replace (P2).

Recommended Next Steps:

1. Start with Phase 1 (JSON export/import) - this unblocks all translation workflows
2. Get user feedback on translation workspace mockups before Phase 2
3. Defer theme integration to Phase 4 - not blocking functionality

Estimated Effort: ~7 weeks to MVP, ~21 dev days total.

Success Criteria: Translator can export messages to JSON, edit in external tool, and re-import without data loss. Side-by-side translation workspace reduces manual comparison time by 50%.