backend-infra-engineer: Post v0.3.9-hotfix7 snapshot (build cleanup)
This commit is contained in:
scawful
135
docs/internal/agents/archive/wasm-planning-2025/README.md
Normal file
135
docs/internal/agents/archive/wasm-planning-2025/README.md
Normal file
@@ -0,0 +1,135 @@
|
||||
# WASM Planning Documentation Archive
|
||||
|
||||
**Date Archived:** November 24, 2025
|
||||
**Archived By:** Documentation Janitor
|
||||
|
||||
This directory contains WASM development planning documents that represent historical design decisions and feature roadmaps. Most content has been superseded by implementation and integration into `docs/internal/wasm_dev_status.md`.
|
||||
|
||||
## Archived Documents
|
||||
|
||||
### 1. wasm-network-support-plan.md
|
||||
- **Original Purpose:** Detailed plan for implementing browser-compatible networking (Phases 1-5)
|
||||
- **Status:** Superseded by implementation
|
||||
- **Why Archived:** Network abstraction and WASM implementations have been completed; this planning document is no longer needed
|
||||
- **Current Reference:** See `wasm_dev_status.md` Section 1.1 (ROM Loading & Initialization)
|
||||
|
||||
### 2. wasm-web-features-roadmap.md
|
||||
- **Original Purpose:** Comprehensive feature roadmap (Phases 1-14)
|
||||
- **Status:** Mostly completed or planning-stage
|
||||
- **Why Archived:** Long-term planning document that predates actual implementation; many features are now in wasm_dev_status.md
|
||||
- **Current Reference:** See `wasm_dev_status.md` Sections 1-4 for completed features
|
||||
|
||||
### 3. wasm-web-app-enhancements-plan.md
|
||||
- **Original Purpose:** Detailed Phase 1-8 implementation plan
|
||||
- **Status:** Most phases completed
|
||||
- **Why Archived:** Highly structured planning document; actual implementations supersede these plans
|
||||
- **Current Reference:** See `wasm_dev_status.md` for current status of all phases
|
||||
|
||||
### 4. wasm-ai-integration-summary.md
|
||||
- **Original Purpose:** Summary of Phase 5 AI Service Integration implementation
|
||||
- **Status:** Consolidated into main status document
|
||||
- **Why Archived:** Content merged into `wasm_dev_status.md` AI Agent Integration section
|
||||
- **Current Reference:** See `wasm_dev_status.md` Section 1 (Completed Features → AI Agent Integration)
|
||||
|
||||
### 5. wasm-widget-tracking-implementation.md
|
||||
- **Original Purpose:** Detailed implementation notes for widget bounds tracking
|
||||
- **Status:** Consolidated into main status document
|
||||
- **Why Archived:** Implementation details merged into `wasm_dev_status.md` Control APIs section
|
||||
- **Current Reference:** See `wasm_dev_status.md` Section 1.4 (WASM Control APIs → Widget Tracking Infrastructure)
|
||||
|
||||
## Content Consolidation
|
||||
|
||||
The following information from archived documents has been consolidated into active documentation:
|
||||
|
||||
| Original Document | Content Moved To | Location |
|
||||
|---|---|---|
|
||||
| wasm-network-support-plan.md | wasm_dev_status.md | Section 1.1, Section 4 (Key Files) |
|
||||
| wasm-web-features-roadmap.md | wasm_dev_status.md | Section 1 (Completed Features) |
|
||||
| wasm-web-app-enhancements-plan.md | wasm_dev_status.md | Section 1 (Completed Features) |
|
||||
| wasm-ai-integration-summary.md | wasm_dev_status.md | Section 1 (AI Agent Integration) |
|
||||
| wasm-widget-tracking-implementation.md | wasm_dev_status.md | Section 1.4 (Widget Tracking Infrastructure) |
|
||||
|
||||
## Active WASM Documentation
|
||||
|
||||
The following documents remain in `docs/internal/` and are actively maintained:
|
||||
|
||||
1. **wasm_dev_status.md** - CANONICAL STATUS DOCUMENT
|
||||
- Current implementation status (updated Nov 24, 2025)
|
||||
- All completed features with file references
|
||||
- Technical debt and known issues
|
||||
- Roadmap for next steps
|
||||
|
||||
2. **wasm-debug-infrastructure.md** - HIGH-LEVEL DEBUGGING OVERVIEW
|
||||
- Debugging architecture and philosophy
|
||||
- File system fixes with explanations
|
||||
- Known limitations
|
||||
- Cross-references to detailed API docs
|
||||
|
||||
3. **wasm-yazeDebug-api-reference.md** - DETAILED API REFERENCE
|
||||
- Complete JavaScript API reference for `window.yazeDebug`
|
||||
- Authoritative source for all debug functions
|
||||
- Usage examples for each API section
|
||||
- Palette, ROM, overworld, arena, emulator debugging
|
||||
|
||||
4. **wasm_dungeon_debugging.md** - QUICK REFERENCE GUIDE
|
||||
- Short, practical debugging tips
|
||||
- God mode console inspector usage
|
||||
- Command line bridge reference
|
||||
- Feature parity notes between WASM and native builds
|
||||
|
||||
5. **debugging-wasm-memory-errors.md** - TECHNICAL REFERENCE
|
||||
- Memory debugging techniques
|
||||
- SAFE_HEAP usage
|
||||
- Common pitfalls and fixes
|
||||
- Function mapping methods
|
||||
|
||||
## How to Use This Archive
|
||||
|
||||
If you need historical context about WASM development decisions:
|
||||
1. Start with the relevant archived document
|
||||
2. Check the "Current Reference" section for where the content moved
|
||||
3. Consult the active documentation for implementation details
|
||||
|
||||
When searching for WASM documentation, use this hierarchy:
|
||||
1. **wasm_dev_status.md** - Status and overview (start here)
|
||||
2. **wasm-debug-infrastructure.md** - Debugging overview
|
||||
3. **wasm-yazeDebug-api-reference.md** - Detailed debug API
|
||||
4. **wasm_dungeon_debugging.md** - Quick reference for dungeon editor
|
||||
5. **debugging-wasm-memory-errors.md** - Memory debugging specifics
|
||||
|
||||
## Rationale for Archival
|
||||
|
||||
The WASM codebase evolved rapidly from November 2024 to November 2025:
|
||||
|
||||
- **Planning Phase** (early 2024): Detailed roadmaps and enhancement plans created
|
||||
- **Implementation Phase** (mid 2024 - Nov 2025): Features implemented incrementally
|
||||
- **Integration Phase** (current): All systems working, focus on maintenance and refinement
|
||||
|
||||
Archived documents represented the planning phase. As implementation completed, their value shifted from prescriptive (what to build) to historical (how we decided to build it). Consolidating information into `wasm_dev_status.md` provides:
|
||||
|
||||
- Single source of truth for current status
|
||||
- Easier maintenance (updates in one place)
|
||||
- Clearer navigation for developers
|
||||
- Better signal-to-noise ratio
|
||||
|
||||
## Future Archival Guidance
|
||||
|
||||
When new WASM documentation is created:
|
||||
- Keep planning/roadmap docs current or archive them promptly
|
||||
- Consolidate implementation summaries into main status document
|
||||
- Use high-level docs (like wasm-debug-infrastructure.md) for architecture overview
|
||||
- Use detailed reference docs (like wasm-yazeDebug-api-reference.md) for API details
|
||||
- Maintain clear cross-references between related docs
|
||||
|
||||
---
|
||||
|
||||
**Archive Directory Structure:**
|
||||
```
|
||||
docs/internal/agents/archive/wasm-planning-2025/
|
||||
├── README.md (this file)
|
||||
├── wasm-network-support-plan.md
|
||||
├── wasm-web-features-roadmap.md
|
||||
├── wasm-web-app-enhancements-plan.md
|
||||
├── wasm-ai-integration-summary.md
|
||||
└── wasm-widget-tracking-implementation.md
|
||||
```
|
||||
@@ -0,0 +1,199 @@
|
||||
# WASM AI Service Integration Summary
|
||||
|
||||
## Overview
|
||||
|
||||
This document summarizes the implementation of Phase 5: AI Service Integration for WASM web build, as specified in the wasm-web-app-enhancements-plan.md.
|
||||
|
||||
## Files Created
|
||||
|
||||
### 1. Browser AI Service (`src/cli/service/ai/`)
|
||||
|
||||
#### `browser_ai_service.h`
|
||||
- **Purpose**: Browser-based AI service interface for WASM builds
|
||||
- **Key Features**:
|
||||
- Implements `AIService` interface for consistency with native builds
|
||||
- Uses `IHttpClient` from network abstraction layer
|
||||
- Supports Gemini API for text generation
|
||||
- Provides vision model support for image analysis
|
||||
- Manages API keys securely via sessionStorage
|
||||
- CORS-compliant HTTP requests
|
||||
- Proper error handling with `absl::Status`
|
||||
- **Compilation**: Only compiled when `__EMSCRIPTEN__` is defined
|
||||
|
||||
#### `browser_ai_service.cc`
|
||||
- **Purpose**: Implementation of browser AI service
|
||||
- **Key Features**:
|
||||
- `GenerateResponse()` for single prompts and conversation history
|
||||
- `AnalyzeImage()` for vision model support
|
||||
- JSON request/response handling with nlohmann/json
|
||||
- Comprehensive error handling and status code mapping
|
||||
- Debug logging to browser console
|
||||
- Support for multiple Gemini models (2.0 Flash, 1.5 Pro, etc.)
|
||||
- Proper handling of API rate limits and quotas
|
||||
|
||||
### 2. Browser Storage (`src/app/platform/wasm/`)
|
||||
|
||||
#### `wasm_browser_storage.h`
|
||||
- **Purpose**: Browser storage wrapper for API keys and settings
|
||||
- **Note**: This is NOT actually secure storage - uses standard localStorage/sessionStorage
|
||||
- **Key Features**:
|
||||
- Dual storage modes: sessionStorage (default) and localStorage
|
||||
- API key management: Store, Retrieve, Clear, Check existence
|
||||
- Generic secret storage for other sensitive data
|
||||
- Storage quota tracking
|
||||
- Bulk operations (list all keys, clear all)
|
||||
- Browser storage availability checking
|
||||
|
||||
#### `wasm_browser_storage.cc`
|
||||
- **Purpose**: Implementation using Emscripten JavaScript interop
|
||||
- **Key Features**:
|
||||
- JavaScript bridge functions using `EM_JS` macros
|
||||
- SessionStorage access (cleared on tab close)
|
||||
- LocalStorage access (persistent)
|
||||
- Prefix-based key namespacing (`yaze_secure_api_`, `yaze_secure_secret_`)
|
||||
- Error handling for storage exceptions
|
||||
- Memory management for JS string conversions
|
||||
|
||||
## Build System Updates
|
||||
|
||||
### 1. CMake Configuration Updates
|
||||
|
||||
#### `src/cli/agent.cmake`
|
||||
- Modified to create a minimal `yaze_agent` library for WASM builds
|
||||
- Includes browser AI service sources
|
||||
- Links with network abstraction layer (`yaze_net`)
|
||||
- Enables JSON support for API communication
|
||||
|
||||
#### `src/app/app_core.cmake`
|
||||
- Added `wasm_browser_storage.cc` to WASM platform sources
|
||||
- Integrated with existing WASM file system and loading manager
|
||||
|
||||
#### `src/CMakeLists.txt`
|
||||
- Updated to include `net_library.cmake` for all builds (including WASM)
|
||||
- Network library now provides WASM-compatible HTTP client
|
||||
|
||||
#### `CMakePresets.json`
|
||||
- Added new `wasm-ai` preset for testing AI features in WASM
|
||||
- Configured with AI runtime enabled and Fetch API flags
|
||||
|
||||
## Integration with Existing Systems
|
||||
|
||||
### Network Abstraction Layer
|
||||
- Leverages existing `IHttpClient` interface
|
||||
- Uses `EmscriptenHttpClient` for browser-based HTTP requests
|
||||
- Supports CORS-compliant requests to Gemini API
|
||||
|
||||
### AI Service Interface
|
||||
- Implements standard `AIService` interface
|
||||
- Compatible with existing agent response structures
|
||||
- Supports tool calls and structured responses
|
||||
|
||||
### WASM Platform Support
|
||||
- Integrates with existing WASM error handler
|
||||
- Works alongside WASM storage and file dialog systems
|
||||
- Compatible with progressive loading manager
|
||||
|
||||
## API Key Security
|
||||
|
||||
### Storage Security Model
|
||||
1. **SessionStorage (Default)**:
|
||||
- Keys stored in browser memory
|
||||
- Automatically cleared when tab closes
|
||||
- No persistence across sessions
|
||||
- Recommended for security
|
||||
|
||||
2. **LocalStorage (Optional)**:
|
||||
- Persistent storage
|
||||
- Survives browser restarts
|
||||
- Less secure but more convenient
|
||||
- User choice based on preference
|
||||
|
||||
### Security Considerations
|
||||
- Keys never hardcoded in binary
|
||||
- Keys prefixed to avoid conflicts
|
||||
- No encryption currently (future enhancement)
|
||||
- Browser same-origin policy provides isolation
|
||||
|
||||
## Usage Example
|
||||
|
||||
```cpp
|
||||
#ifdef __EMSCRIPTEN__
|
||||
#include "cli/service/ai/browser_ai_service.h"
|
||||
#include "app/net/wasm/emscripten_http_client.h"
|
||||
#include "app/platform/wasm/wasm_browser_storage.h"
|
||||
|
||||
// Store API key from user input
|
||||
WasmBrowserStorage::StoreApiKey("gemini", user_api_key);
|
||||
}
|
||||
|
||||
// Create AI service
|
||||
BrowserAIConfig config;
|
||||
config.api_key = WasmBrowserStorage::RetrieveApiKey("gemini").value();
|
||||
config.model = "gemini-2.5-flash";
|
||||
|
||||
auto http_client = std::make_unique<EmscriptenHttpClient>();
|
||||
BrowserAIService ai_service(config, std::move(http_client));
|
||||
|
||||
// Generate response
|
||||
auto response = ai_service.GenerateResponse("Explain the Zelda 3 ROM format");
|
||||
#endif
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### Test File: `test/browser_ai_test.cc`
|
||||
- Verifies secure storage operations
|
||||
- Tests AI service creation
|
||||
- Validates model listing
|
||||
- Checks error handling
|
||||
|
||||
### Build and Test Commands
|
||||
```bash
|
||||
# Configure with AI support
|
||||
cmake --preset wasm-ai
|
||||
|
||||
# Build
|
||||
cmake --build build_wasm_ai
|
||||
|
||||
# Run in browser
|
||||
emrun build_wasm_ai/yaze.html
|
||||
```
|
||||
|
||||
## CORS Considerations
|
||||
|
||||
### Gemini API
|
||||
- ✅ Works with browser fetch (Google APIs support CORS)
|
||||
- ✅ No proxy required
|
||||
- ✅ Direct browser-to-API communication
|
||||
|
||||
### Ollama (Future)
|
||||
- ⚠️ Requires `--cors` flag on Ollama server
|
||||
- ⚠️ May need proxy for local instances
|
||||
- ⚠️ Security implications of CORS relaxation
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
1. **Encryption**: Add client-side encryption for stored API keys
|
||||
2. **Multiple Providers**: Support for OpenAI, Anthropic APIs
|
||||
3. **Streaming Responses**: Implement streaming for better UX
|
||||
4. **Offline Caching**: Cache AI responses for offline use
|
||||
5. **Web Worker Integration**: Move AI calls to background thread
|
||||
|
||||
## Limitations
|
||||
|
||||
1. **Browser Security**: Subject to browser security policies
|
||||
2. **CORS Restrictions**: Limited to CORS-enabled APIs
|
||||
3. **Storage Limits**: ~5-10MB for sessionStorage/localStorage
|
||||
4. **No File System**: Cannot access local models
|
||||
5. **Network Required**: No offline AI capabilities
|
||||
|
||||
## Conclusion
|
||||
|
||||
The WASM AI service integration successfully brings browser-based AI capabilities to yaze. The implementation:
|
||||
- ✅ Provides secure API key management
|
||||
- ✅ Integrates cleanly with existing architecture
|
||||
- ✅ Supports both text and vision models
|
||||
- ✅ Handles errors gracefully
|
||||
- ✅ Works within browser security constraints
|
||||
|
||||
This enables users to leverage AI assistance for ROM hacking directly in their browser without needing to install local AI models or tools.
|
||||
@@ -0,0 +1,415 @@
|
||||
# WASM Network Support Plan for yaze
|
||||
|
||||
## Executive Summary
|
||||
|
||||
This document outlines the architectural changes required to enable AI services (Gemini/Ollama) and WebSocket collaboration features in the browser-based WASM build of yaze. The main challenge is replacing native networking libraries (cpp-httplib, OpenSSL, curl) with browser-compatible APIs provided by Emscripten.
|
||||
|
||||
## Current Architecture Analysis
|
||||
|
||||
### 1. Gemini AI Service (`src/cli/service/ai/gemini_ai_service.cc`)
|
||||
|
||||
**Current Implementation:**
|
||||
- Uses `httplib` (cpp-httplib) for HTTPS requests with OpenSSL support
|
||||
- Falls back to `curl` command via `popen()` for API calls
|
||||
- Depends on OpenSSL for SSL/TLS encryption
|
||||
- Uses base64 encoding for image uploads
|
||||
|
||||
**Key Dependencies:**
|
||||
- `httplib.h` - C++ HTTP library
|
||||
- OpenSSL - SSL/TLS support
|
||||
- `popen()`/`pclose()` - Process execution for curl
|
||||
- File system access for temporary JSON files
|
||||
|
||||
### 2. Ollama AI Service (`src/cli/service/ai/ollama_ai_service.cc`)
|
||||
|
||||
**Current Implementation:**
|
||||
- Uses `httplib::Client` for HTTP requests to local Ollama server
|
||||
- Communicates over HTTP (not HTTPS) on localhost:11434
|
||||
- JSON parsing with nlohmann/json
|
||||
|
||||
**Key Dependencies:**
|
||||
- `httplib.h` - C++ HTTP library
|
||||
- No SSL/TLS requirement (local HTTP only)
|
||||
|
||||
### 3. WebSocket Client (`src/app/net/websocket_client.cc`)
|
||||
|
||||
**Current Implementation:**
|
||||
- Uses `httplib::Client` as a placeholder (not true WebSocket)
|
||||
- Currently implements HTTP POST fallback instead of WebSocket
|
||||
- Conditional OpenSSL support for secure connections
|
||||
- Thread-based receive loop
|
||||
|
||||
**Key Dependencies:**
|
||||
- `httplib.h` - C++ HTTP library
|
||||
- OpenSSL (optional) - For WSS support
|
||||
- `std::thread` - For receive loop
|
||||
- Platform-specific socket libraries (ws2_32 on Windows)
|
||||
|
||||
### 4. HTTP Server (`src/cli/service/api/http_server.cc`)
|
||||
|
||||
**Current Implementation:**
|
||||
- Uses `httplib::Server` for REST API endpoints
|
||||
- Runs in separate thread
|
||||
- Provides health check and model listing endpoints
|
||||
|
||||
**Key Dependencies:**
|
||||
- `httplib.h` - C++ HTTP server
|
||||
- `std::thread` - For server thread
|
||||
|
||||
## Emscripten Capabilities
|
||||
|
||||
### Available APIs:
|
||||
|
||||
1. **Fetch API** (`emscripten_fetch()`)
|
||||
- Asynchronous HTTP/HTTPS requests
|
||||
- Supports GET, POST, PUT, DELETE
|
||||
- CORS-aware
|
||||
- Can handle binary data and streams
|
||||
|
||||
2. **WebSocket API**
|
||||
- Native browser WebSocket support
|
||||
- Accessible via Emscripten's WebSocket wrapper
|
||||
- Full duplex communication
|
||||
- Binary and text message support
|
||||
|
||||
3. **Web Workers** (via pthread emulation)
|
||||
- Background processing
|
||||
- Shared memory support with SharedArrayBuffer
|
||||
- Can handle async operations
|
||||
|
||||
## Required Changes
|
||||
|
||||
### Phase 1: Abstract Network Layer
|
||||
|
||||
Create platform-agnostic network interfaces:
|
||||
|
||||
```cpp
|
||||
// src/app/net/http_client.h
|
||||
class IHttpClient {
|
||||
public:
|
||||
virtual ~IHttpClient() = default;
|
||||
virtual absl::StatusOr<HttpResponse> Get(const std::string& url,
|
||||
const Headers& headers) = 0;
|
||||
virtual absl::StatusOr<HttpResponse> Post(const std::string& url,
|
||||
const std::string& body,
|
||||
const Headers& headers) = 0;
|
||||
};
|
||||
|
||||
// src/app/net/websocket.h
|
||||
class IWebSocket {
|
||||
public:
|
||||
virtual ~IWebSocket() = default;
|
||||
virtual absl::Status Connect(const std::string& url) = 0;
|
||||
virtual absl::Status Send(const std::string& message) = 0;
|
||||
virtual void OnMessage(std::function<void(const std::string&)> callback) = 0;
|
||||
};
|
||||
```
|
||||
|
||||
### Phase 2: Native Implementation
|
||||
|
||||
Keep existing implementations for native builds:
|
||||
|
||||
```cpp
|
||||
// src/app/net/native/httplib_client.cc
|
||||
class HttpLibClient : public IHttpClient {
|
||||
// Current httplib implementation
|
||||
};
|
||||
|
||||
// src/app/net/native/httplib_websocket.cc
|
||||
class HttpLibWebSocket : public IWebSocket {
|
||||
// Current httplib-based implementation
|
||||
};
|
||||
```
|
||||
|
||||
### Phase 3: Emscripten Implementation
|
||||
|
||||
Create browser-compatible implementations:
|
||||
|
||||
```cpp
|
||||
// src/app/net/wasm/emscripten_http_client.cc
|
||||
#ifdef __EMSCRIPTEN__
|
||||
class EmscriptenHttpClient : public IHttpClient {
|
||||
absl::StatusOr<HttpResponse> Get(const std::string& url,
|
||||
const Headers& headers) override {
|
||||
emscripten_fetch_attr_t attr;
|
||||
emscripten_fetch_attr_init(&attr);
|
||||
strcpy(attr.requestMethod, "GET");
|
||||
attr.attributes = EMSCRIPTEN_FETCH_LOAD_TO_MEMORY;
|
||||
|
||||
// Set headers
|
||||
std::vector<const char*> header_strings;
|
||||
for (const auto& [key, value] : headers) {
|
||||
header_strings.push_back(key.c_str());
|
||||
header_strings.push_back(value.c_str());
|
||||
}
|
||||
header_strings.push_back(nullptr);
|
||||
attr.requestHeaders = header_strings.data();
|
||||
|
||||
// Synchronous fetch (blocks until complete)
|
||||
emscripten_fetch_t* fetch = emscripten_fetch(&attr, url.c_str());
|
||||
|
||||
HttpResponse response;
|
||||
response.status = fetch->status;
|
||||
response.body = std::string(fetch->data, fetch->numBytes);
|
||||
|
||||
emscripten_fetch_close(fetch);
|
||||
return response;
|
||||
}
|
||||
|
||||
absl::StatusOr<HttpResponse> Post(const std::string& url,
|
||||
const std::string& body,
|
||||
const Headers& headers) override {
|
||||
emscripten_fetch_attr_t attr;
|
||||
emscripten_fetch_attr_init(&attr);
|
||||
strcpy(attr.requestMethod, "POST");
|
||||
attr.attributes = EMSCRIPTEN_FETCH_LOAD_TO_MEMORY;
|
||||
attr.requestData = body.c_str();
|
||||
attr.requestDataSize = body.length();
|
||||
|
||||
// Similar implementation as Get()
|
||||
// ...
|
||||
}
|
||||
};
|
||||
#endif
|
||||
|
||||
// src/app/net/wasm/emscripten_websocket.cc
|
||||
#ifdef __EMSCRIPTEN__
|
||||
#include <emscripten/websocket.h>
|
||||
|
||||
class EmscriptenWebSocket : public IWebSocket {
|
||||
EMSCRIPTEN_WEBSOCKET_T socket_;
|
||||
|
||||
absl::Status Connect(const std::string& url) override {
|
||||
EmscriptenWebSocketCreateAttributes attrs = {
|
||||
url.c_str(),
|
||||
nullptr, // protocols
|
||||
EM_TRUE // createOnMainThread
|
||||
};
|
||||
|
||||
socket_ = emscripten_websocket_new(&attrs);
|
||||
if (socket_ <= 0) {
|
||||
return absl::InternalError("Failed to create WebSocket");
|
||||
}
|
||||
|
||||
// Set callbacks
|
||||
emscripten_websocket_set_onopen_callback(socket_, this, OnOpenCallback);
|
||||
emscripten_websocket_set_onmessage_callback(socket_, this, OnMessageCallback);
|
||||
emscripten_websocket_set_onerror_callback(socket_, this, OnErrorCallback);
|
||||
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
absl::Status Send(const std::string& message) override {
|
||||
EMSCRIPTEN_RESULT result = emscripten_websocket_send_text(
|
||||
socket_, message.c_str(), message.length());
|
||||
if (result != EMSCRIPTEN_RESULT_SUCCESS) {
|
||||
return absl::InternalError("Failed to send WebSocket message");
|
||||
}
|
||||
return absl::OkStatus();
|
||||
}
|
||||
};
|
||||
#endif
|
||||
```
|
||||
|
||||
### Phase 4: Factory Pattern
|
||||
|
||||
Create factories to instantiate the correct implementation:
|
||||
|
||||
```cpp
|
||||
// src/app/net/network_factory.cc
|
||||
std::unique_ptr<IHttpClient> CreateHttpClient() {
|
||||
#ifdef __EMSCRIPTEN__
|
||||
return std::make_unique<EmscriptenHttpClient>();
|
||||
#else
|
||||
return std::make_unique<HttpLibClient>();
|
||||
#endif
|
||||
}
|
||||
|
||||
std::unique_ptr<IWebSocket> CreateWebSocket() {
|
||||
#ifdef __EMSCRIPTEN__
|
||||
return std::make_unique<EmscriptenWebSocket>();
|
||||
#else
|
||||
return std::make_unique<HttpLibWebSocket>();
|
||||
#endif
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 5: Service Modifications
|
||||
|
||||
Update AI services to use the abstraction:
|
||||
|
||||
```cpp
|
||||
// src/cli/service/ai/gemini_ai_service.cc
|
||||
class GeminiAIService {
|
||||
std::unique_ptr<IHttpClient> http_client_;
|
||||
|
||||
GeminiAIService(const GeminiConfig& config)
|
||||
: config_(config),
|
||||
http_client_(CreateHttpClient()) {
|
||||
// Initialize
|
||||
}
|
||||
|
||||
absl::Status CheckAvailability() {
|
||||
std::string url = "https://generativelanguage.googleapis.com/v1beta/models/"
|
||||
+ config_.model;
|
||||
Headers headers = {{"x-goog-api-key", config_.api_key}};
|
||||
|
||||
auto response_or = http_client_->Get(url, headers);
|
||||
if (!response_or.ok()) {
|
||||
return response_or.status();
|
||||
}
|
||||
|
||||
auto& response = response_or.value();
|
||||
if (response.status != 200) {
|
||||
return absl::UnavailableError("API not available");
|
||||
}
|
||||
|
||||
return absl::OkStatus();
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
## CMake Configuration
|
||||
|
||||
Update CMake to handle WASM builds:
|
||||
|
||||
```cmake
|
||||
# src/app/net/net_library.cmake
|
||||
if(EMSCRIPTEN)
|
||||
set(YAZE_NET_SRC
|
||||
app/net/wasm/emscripten_http_client.cc
|
||||
app/net/wasm/emscripten_websocket.cc
|
||||
app/net/network_factory.cc
|
||||
)
|
||||
|
||||
# Add Emscripten fetch and WebSocket flags
|
||||
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -s FETCH=1")
|
||||
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -s WEBSOCKET=1")
|
||||
else()
|
||||
set(YAZE_NET_SRC
|
||||
app/net/native/httplib_client.cc
|
||||
app/net/native/httplib_websocket.cc
|
||||
app/net/network_factory.cc
|
||||
)
|
||||
|
||||
# Link native dependencies
|
||||
target_link_libraries(yaze_net PUBLIC httplib OpenSSL::SSL)
|
||||
endif()
|
||||
```
|
||||
|
||||
## CORS Considerations
|
||||
|
||||
For browser builds, the following CORS requirements apply:
|
||||
|
||||
1. **Gemini API**: Google's API servers must include appropriate CORS headers
|
||||
2. **Ollama**: Local Ollama server needs `--cors` flag or proxy setup
|
||||
3. **WebSocket Server**: Must handle WebSocket upgrade correctly
|
||||
|
||||
### Proxy Server Option
|
||||
|
||||
For services without CORS support, implement a proxy:
|
||||
|
||||
```javascript
|
||||
// proxy-server.js (Node.js)
|
||||
const express = require('express');
|
||||
const { createProxyMiddleware } = require('http-proxy-middleware');
|
||||
|
||||
const app = express();
|
||||
|
||||
// Proxy for Ollama
|
||||
app.use('/ollama', createProxyMiddleware({
|
||||
target: 'http://localhost:11434',
|
||||
changeOrigin: true,
|
||||
pathRewrite: { '^/ollama': '' }
|
||||
}));
|
||||
|
||||
// Proxy for Gemini
|
||||
app.use('/gemini', createProxyMiddleware({
|
||||
target: 'https://generativelanguage.googleapis.com',
|
||||
changeOrigin: true,
|
||||
pathRewrite: { '^/gemini': '' }
|
||||
}));
|
||||
|
||||
app.listen(3000);
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
1. **Unit Tests**: Mock network interfaces for both native and WASM
|
||||
2. **Integration Tests**: Test actual API calls in native builds
|
||||
3. **Browser Tests**: Manual testing of WASM build in browser
|
||||
4. **E2E Tests**: Selenium/Playwright for automated browser testing
|
||||
|
||||
## Implementation Timeline
|
||||
|
||||
### Week 1: Foundation
|
||||
- Create abstract interfaces (IHttpClient, IWebSocket)
|
||||
- Implement factory pattern
|
||||
- Update CMake for conditional compilation
|
||||
|
||||
### Week 2: Native Refactoring
|
||||
- Refactor existing code to use interfaces
|
||||
- Create native implementations
|
||||
- Ensure no regression in current functionality
|
||||
|
||||
### Week 3: WASM Implementation
|
||||
- Implement EmscriptenHttpClient
|
||||
- Implement EmscriptenWebSocket
|
||||
- Test basic functionality
|
||||
|
||||
### Week 4: Service Integration
|
||||
- Update GeminiAIService
|
||||
- Update OllamaAIService
|
||||
- Update WebSocketClient
|
||||
|
||||
### Week 5: Testing & Refinement
|
||||
- Comprehensive testing
|
||||
- CORS handling
|
||||
- Performance optimization
|
||||
|
||||
## Risk Mitigation
|
||||
|
||||
### Risk 1: CORS Blocking
|
||||
**Mitigation**: Implement proxy server as fallback, document CORS requirements
|
||||
|
||||
### Risk 2: API Key Security
|
||||
**Mitigation**:
|
||||
- Never embed API keys in WASM binary
|
||||
- Require user to input API key via UI
|
||||
- Store in browser's localStorage with encryption
|
||||
|
||||
### Risk 3: Performance Issues
|
||||
**Mitigation**:
|
||||
- Use Web Workers for background processing
|
||||
- Implement request caching
|
||||
- Add loading indicators for long operations
|
||||
|
||||
### Risk 4: Browser Compatibility
|
||||
**Mitigation**:
|
||||
- Test on Chrome, Firefox, Safari, Edge
|
||||
- Use feature detection
|
||||
- Provide fallbacks for unsupported features
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **API Keys**: Must be user-provided, never hardcoded
|
||||
2. **HTTPS Only**: All API calls must use HTTPS in production
|
||||
3. **Input Validation**: Sanitize all user inputs before API calls
|
||||
4. **Rate Limiting**: Implement client-side rate limiting
|
||||
5. **Content Security Policy**: Configure CSP headers properly
|
||||
|
||||
## Conclusion
|
||||
|
||||
The transition to WASM-compatible networking is achievable through careful abstraction and platform-specific implementations. The key is maintaining a clean separation between platform-agnostic interfaces and platform-specific implementations. This approach allows the codebase to support both native and browser environments without sacrificing functionality or performance.
|
||||
|
||||
The proposed architecture provides:
|
||||
- Clean abstraction layers
|
||||
- Minimal changes to existing service code
|
||||
- Easy testing and mocking
|
||||
- Future extensibility for other platforms
|
||||
|
||||
Next steps:
|
||||
1. Review and approve this plan
|
||||
2. Create feature branch for implementation
|
||||
3. Begin with abstract interface definitions
|
||||
4. Implement incrementally with continuous testing
|
||||
@@ -0,0 +1,554 @@
|
||||
# WASM Web App Enhancements Plan
|
||||
|
||||
## Executive Summary
|
||||
|
||||
This document outlines the comprehensive plan to make yaze's WASM web build fully featured, robust, and user-friendly. The goal is to achieve feature parity with the native desktop application where technically feasible, while leveraging browser-specific capabilities where appropriate.
|
||||
|
||||
## Current Status
|
||||
|
||||
### Completed
|
||||
- [x] Basic WASM build configuration
|
||||
- [x] pthread support for Emscripten
|
||||
- [x] Network abstraction layer (IHttpClient, IWebSocket)
|
||||
- [x] Emscripten HTTP client using `emscripten_fetch()`
|
||||
- [x] Emscripten WebSocket using browser API
|
||||
- [x] **Phase 1**: File System Layer (WasmStorage, WasmFileDialog)
|
||||
- [x] **Phase 2**: Error Handling Infrastructure (WasmErrorHandler)
|
||||
- [x] **Phase 3**: Progressive Loading UI (WasmLoadingManager)
|
||||
- [x] **Phase 4**: Offline Support (Service Workers, PWA manifest)
|
||||
- [x] **Phase 5**: AI Service Integration (BrowserAIService, WasmSecureStorage)
|
||||
- [x] **Phase 6**: Local Storage Persistence (WasmSettings, AutoSaveManager)
|
||||
|
||||
- [x] **Phase 7**: Web Workers for heavy processing (WasmWorkerPool)
|
||||
- [x] **Phase 8**: Emulator Audio (WebAudio, WasmAudioBackend)
|
||||
|
||||
### In Progress
|
||||
- [ ] WASM CI build verification
|
||||
- [ ] Integration of loading manager with gfx::Arena
|
||||
- [ ] Integration testing across all phases
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: File System Layer
|
||||
|
||||
### Overview
|
||||
WASM builds cannot access the local filesystem directly. We need a virtualized file system layer that uses browser storage APIs.
|
||||
|
||||
### Implementation
|
||||
|
||||
#### 1.1 IndexedDB Storage Backend
|
||||
```cpp
|
||||
// src/app/platform/wasm/wasm_storage.h
|
||||
class WasmStorage {
|
||||
public:
|
||||
// Store ROM data
|
||||
static absl::Status SaveRom(const std::string& name, const std::vector<uint8_t>& data);
|
||||
static absl::StatusOr<std::vector<uint8_t>> LoadRom(const std::string& name);
|
||||
static absl::Status DeleteRom(const std::string& name);
|
||||
static std::vector<std::string> ListRoms();
|
||||
|
||||
// Store project files (JSON, palettes, patches)
|
||||
static absl::Status SaveProject(const std::string& name, const std::string& json);
|
||||
static absl::StatusOr<std::string> LoadProject(const std::string& name);
|
||||
|
||||
// Store user preferences
|
||||
static absl::Status SavePreferences(const nlohmann::json& prefs);
|
||||
static absl::StatusOr<nlohmann::json> LoadPreferences();
|
||||
};
|
||||
```
|
||||
|
||||
#### 1.2 File Upload Handler
|
||||
```cpp
|
||||
// JavaScript interop for file input
|
||||
EM_JS(void, openFileDialog, (const char* accept, int callback_id), {
|
||||
const input = document.createElement('input');
|
||||
input.type = 'file';
|
||||
input.accept = UTF8ToString(accept);
|
||||
input.onchange = (e) => {
|
||||
const file = e.target.files[0];
|
||||
const reader = new FileReader();
|
||||
reader.onload = () => {
|
||||
const data = new Uint8Array(reader.result);
|
||||
Module._handleFileLoaded(callback_id, data);
|
||||
};
|
||||
reader.readAsArrayBuffer(file);
|
||||
};
|
||||
input.click();
|
||||
});
|
||||
```
|
||||
|
||||
#### 1.3 File Download Handler
|
||||
```cpp
|
||||
// Download ROM or project file
|
||||
EM_JS(void, downloadFile, (const char* filename, const uint8_t* data, size_t size), {
|
||||
const blob = new Blob([HEAPU8.subarray(data, data + size)], {type: 'application/octet-stream'});
|
||||
const url = URL.createObjectURL(blob);
|
||||
const a = document.createElement('a');
|
||||
a.href = url;
|
||||
a.download = UTF8ToString(filename);
|
||||
a.click();
|
||||
URL.revokeObjectURL(url);
|
||||
});
|
||||
```
|
||||
|
||||
### Files to Create
|
||||
- `src/app/platform/wasm/wasm_storage.h`
|
||||
- `src/app/platform/wasm/wasm_storage.cc`
|
||||
- `src/app/platform/wasm/wasm_file_dialog.h`
|
||||
- `src/app/platform/wasm/wasm_file_dialog.cc`
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Error Handling Infrastructure
|
||||
|
||||
### Overview
|
||||
Browser-based applications need specialized error handling that integrates with the web UI and provides user-friendly feedback.
|
||||
|
||||
### Implementation
|
||||
|
||||
#### 2.1 Browser Error Handler
|
||||
```cpp
|
||||
// src/app/platform/wasm/wasm_error_handler.h
|
||||
class WasmErrorHandler {
|
||||
public:
|
||||
// Display error in browser UI
|
||||
static void ShowError(const std::string& title, const std::string& message);
|
||||
static void ShowWarning(const std::string& title, const std::string& message);
|
||||
static void ShowInfo(const std::string& title, const std::string& message);
|
||||
|
||||
// Toast notifications (non-blocking)
|
||||
static void Toast(const std::string& message, ToastType type, int duration_ms = 3000);
|
||||
|
||||
// Progress indicators
|
||||
static void ShowProgress(const std::string& task, float progress);
|
||||
static void HideProgress();
|
||||
|
||||
// Confirmation dialogs
|
||||
static void Confirm(const std::string& message, std::function<void(bool)> callback);
|
||||
};
|
||||
```
|
||||
|
||||
#### 2.2 JavaScript Integration
|
||||
```javascript
|
||||
// src/web/error_handler.js
|
||||
window.showYazeError = function(title, message) {
|
||||
// Create styled error modal
|
||||
const modal = document.createElement('div');
|
||||
modal.className = 'yaze-error-modal';
|
||||
modal.innerHTML = `
|
||||
<div class="yaze-error-content">
|
||||
<h2>${escapeHtml(title)}</h2>
|
||||
<p>${escapeHtml(message)}</p>
|
||||
<button onclick="this.parentElement.parentElement.remove()">OK</button>
|
||||
</div>
|
||||
`;
|
||||
document.body.appendChild(modal);
|
||||
};
|
||||
|
||||
window.showYazeToast = function(message, type, duration) {
|
||||
const toast = document.createElement('div');
|
||||
toast.className = `yaze-toast yaze-toast-${type}`;
|
||||
toast.textContent = message;
|
||||
document.body.appendChild(toast);
|
||||
setTimeout(() => toast.remove(), duration);
|
||||
};
|
||||
```
|
||||
|
||||
### Files to Create
|
||||
- `src/app/platform/wasm/wasm_error_handler.h`
|
||||
- `src/app/platform/wasm/wasm_error_handler.cc`
|
||||
- `src/web/error_handler.js`
|
||||
- `src/web/error_handler.css`
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Progressive Loading UI
|
||||
|
||||
### Overview
|
||||
ROM loading and graphics processing can take significant time. Users need visual feedback and the ability to cancel long operations.
|
||||
|
||||
### Implementation
|
||||
|
||||
#### 3.1 Loading Manager
|
||||
```cpp
|
||||
// src/app/platform/wasm/wasm_loading_manager.h
|
||||
class WasmLoadingManager {
|
||||
public:
|
||||
// Start a loading operation with progress tracking
|
||||
static LoadingHandle BeginLoading(const std::string& task_name);
|
||||
|
||||
// Update progress (0.0 to 1.0)
|
||||
static void UpdateProgress(LoadingHandle handle, float progress);
|
||||
static void UpdateMessage(LoadingHandle handle, const std::string& message);
|
||||
|
||||
// Check if user requested cancel
|
||||
static bool IsCancelled(LoadingHandle handle);
|
||||
|
||||
// Complete the loading operation
|
||||
static void EndLoading(LoadingHandle handle);
|
||||
};
|
||||
```
|
||||
|
||||
#### 3.2 Integration with gfx::Arena
|
||||
```cpp
|
||||
// Modified graphics loading to report progress
|
||||
void Arena::LoadGraphicsWithProgress(Rom* rom) {
|
||||
auto handle = WasmLoadingManager::BeginLoading("Loading Graphics");
|
||||
|
||||
for (int i = 0; i < kNumGraphicsSheets; i++) {
|
||||
if (WasmLoadingManager::IsCancelled(handle)) {
|
||||
WasmLoadingManager::EndLoading(handle);
|
||||
return; // User cancelled
|
||||
}
|
||||
|
||||
LoadGraphicsSheet(rom, i);
|
||||
WasmLoadingManager::UpdateProgress(handle, static_cast<float>(i) / kNumGraphicsSheets);
|
||||
WasmLoadingManager::UpdateMessage(handle, absl::StrFormat("Sheet %d/%d", i, kNumGraphicsSheets));
|
||||
|
||||
// Yield to browser event loop periodically
|
||||
emscripten_sleep(0);
|
||||
}
|
||||
|
||||
WasmLoadingManager::EndLoading(handle);
|
||||
}
|
||||
```
|
||||
|
||||
### Files to Create
|
||||
- `src/app/platform/wasm/wasm_loading_manager.h`
|
||||
- `src/app/platform/wasm/wasm_loading_manager.cc`
|
||||
- `src/web/loading_indicator.js`
|
||||
- `src/web/loading_indicator.css`
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Offline Support (Service Workers)
|
||||
|
||||
### Overview
|
||||
Cache the WASM binary and assets for offline use, enabling users to work without an internet connection.
|
||||
|
||||
### Implementation
|
||||
|
||||
#### 4.1 Service Worker
|
||||
```javascript
|
||||
// src/web/service-worker.js
|
||||
const CACHE_NAME = 'yaze-cache-v1';
|
||||
const ASSETS = [
|
||||
'/yaze.wasm',
|
||||
'/yaze.js',
|
||||
'/yaze.data',
|
||||
'/index.html',
|
||||
'/style.css',
|
||||
'/fonts/',
|
||||
];
|
||||
|
||||
self.addEventListener('install', (event) => {
|
||||
event.waitUntil(
|
||||
caches.open(CACHE_NAME).then((cache) => cache.addAll(ASSETS))
|
||||
);
|
||||
});
|
||||
|
||||
self.addEventListener('fetch', (event) => {
|
||||
event.respondWith(
|
||||
caches.match(event.request).then((response) => {
|
||||
return response || fetch(event.request);
|
||||
})
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
#### 4.2 Registration
|
||||
```javascript
|
||||
// src/web/index.html
|
||||
if ('serviceWorker' in navigator) {
|
||||
navigator.serviceWorker.register('/service-worker.js')
|
||||
.then((registration) => {
|
||||
console.log('Service Worker registered:', registration.scope);
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### Files to Create
|
||||
- `src/web/service-worker.js`
|
||||
- `src/web/manifest.json` (PWA manifest)
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: AI Service Integration
|
||||
|
||||
### Overview
|
||||
Integrate the network abstraction layer with AI services (Gemini, Ollama) for browser-based AI assistance.
|
||||
|
||||
### Implementation
|
||||
|
||||
#### 5.1 Browser AI Client
|
||||
```cpp
|
||||
// src/cli/service/ai/browser_ai_service.h
|
||||
#ifdef __EMSCRIPTEN__
|
||||
class BrowserAIService : public IAIService {
|
||||
public:
|
||||
explicit BrowserAIService(const AIConfig& config);
|
||||
|
||||
absl::StatusOr<std::string> GenerateResponse(const std::string& prompt) override;
|
||||
absl::StatusOr<std::string> AnalyzeImage(const gfx::Bitmap& image,
|
||||
const std::string& prompt) override;
|
||||
|
||||
private:
|
||||
std::unique_ptr<IHttpClient> http_client_;
|
||||
std::string api_key_; // User-provided, never stored in binary
|
||||
std::string model_;
|
||||
};
|
||||
#endif
|
||||
```
|
||||
|
||||
#### 5.2 API Key Management
|
||||
```cpp
|
||||
// Secure API key storage in browser
|
||||
EM_JS(void, storeApiKey, (const char* service, const char* key), {
|
||||
// Use sessionStorage for temporary storage (cleared on tab close)
|
||||
// or encrypted localStorage for persistent storage
|
||||
sessionStorage.setItem('yaze_' + UTF8ToString(service) + '_key', UTF8ToString(key));
|
||||
});
|
||||
|
||||
EM_JS(char*, retrieveApiKey, (const char* service), {
|
||||
const key = sessionStorage.getItem('yaze_' + UTF8ToString(service) + '_key');
|
||||
if (!key) return null;
|
||||
const len = lengthBytesUTF8(key) + 1;
|
||||
const ptr = _malloc(len);
|
||||
stringToUTF8(key, ptr, len);
|
||||
return ptr;
|
||||
});
|
||||
```
|
||||
|
||||
### CORS Considerations
|
||||
- Gemini API: Should work with browser fetch (Google APIs support CORS)
|
||||
- Ollama: Requires `--cors` flag or proxy server for local instances
|
||||
|
||||
### Files to Create
|
||||
- `src/cli/service/ai/browser_ai_service.h`
|
||||
- `src/cli/service/ai/browser_ai_service.cc`
|
||||
- `src/app/platform/wasm/wasm_browser_storage.h`
|
||||
- `src/app/platform/wasm/wasm_browser_storage.cc`
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Local Storage Persistence
|
||||
|
||||
### Overview
|
||||
Persist user settings, recent files, undo history, and workspace layouts in browser storage.
|
||||
|
||||
### Implementation
|
||||
|
||||
#### 6.1 Settings Persistence
|
||||
```cpp
|
||||
// src/app/platform/wasm/wasm_settings.h
|
||||
class WasmSettings {
|
||||
public:
|
||||
// User preferences
|
||||
static void SaveTheme(const std::string& theme);
|
||||
static std::string LoadTheme();
|
||||
|
||||
// Recent files (stored as IndexedDB references)
|
||||
static void AddRecentFile(const std::string& name);
|
||||
static std::vector<std::string> GetRecentFiles();
|
||||
|
||||
// Workspace layouts
|
||||
static void SaveWorkspace(const std::string& name, const std::string& layout_json);
|
||||
static std::string LoadWorkspace(const std::string& name);
|
||||
|
||||
// Undo history (for crash recovery)
|
||||
static void SaveUndoHistory(const std::string& editor, const std::vector<uint8_t>& history);
|
||||
static std::vector<uint8_t> LoadUndoHistory(const std::string& editor);
|
||||
};
|
||||
```
|
||||
|
||||
#### 6.2 Auto-Save & Recovery
|
||||
```cpp
|
||||
// Periodic auto-save
|
||||
class AutoSaveManager {
|
||||
public:
|
||||
void Start(int interval_seconds = 60);
|
||||
void Stop();
|
||||
|
||||
// Called on page unload
|
||||
void EmergencySave();
|
||||
|
||||
// Called on startup
|
||||
bool HasRecoveryData();
|
||||
void RecoverLastSession();
|
||||
void ClearRecoveryData();
|
||||
};
|
||||
```
|
||||
|
||||
### Files to Create
|
||||
- `src/app/platform/wasm/wasm_settings.h`
|
||||
- `src/app/platform/wasm/wasm_settings.cc`
|
||||
- `src/app/platform/wasm/wasm_autosave.h`
|
||||
- `src/app/platform/wasm/wasm_autosave.cc`
|
||||
|
||||
---
|
||||
|
||||
## Phase 7: Web Workers for Heavy Processing
|
||||
|
||||
### Overview
|
||||
Offload CPU-intensive operations to Web Workers to prevent UI freezing.
|
||||
|
||||
### Implementation
|
||||
|
||||
#### 7.1 Background Processing Worker
|
||||
```cpp
|
||||
// Operations to run in Web Worker:
|
||||
// - ROM decompression (LC-LZ2)
|
||||
// - Graphics sheet decoding
|
||||
// - Palette calculations
|
||||
// - Asar assembly compilation
|
||||
|
||||
class WasmWorkerPool {
|
||||
public:
|
||||
using TaskCallback = std::function<void(const std::vector<uint8_t>&)>;
|
||||
|
||||
// Submit work to background thread
|
||||
void SubmitTask(const std::string& type,
|
||||
const std::vector<uint8_t>& input,
|
||||
TaskCallback callback);
|
||||
|
||||
// Wait for all tasks
|
||||
void WaitAll();
|
||||
};
|
||||
```
|
||||
|
||||
### Files to Create
|
||||
- `src/app/platform/wasm/wasm_worker_pool.h`
|
||||
- `src/app/platform/wasm/wasm_worker_pool.cc`
|
||||
- `src/web/worker.js`
|
||||
|
||||
---
|
||||
|
||||
## Phase 8: Emulator Integration
|
||||
|
||||
### Overview
|
||||
The SNES emulator can run in WASM with WebAudio for sound output.
|
||||
|
||||
### Implementation
|
||||
|
||||
#### 8.1 WebAudio Backend
|
||||
```cpp
|
||||
// src/app/emu/platform/wasm/wasm_audio.h
|
||||
class WasmAudioBackend : public IAudioBackend {
|
||||
public:
|
||||
void Initialize(int sample_rate, int buffer_size) override;
|
||||
void QueueSamples(const int16_t* samples, size_t count) override;
|
||||
void Shutdown() override;
|
||||
};
|
||||
```
|
||||
|
||||
#### 8.2 Canvas Rendering
|
||||
```cpp
|
||||
// Use EM_ASM for direct canvas manipulation if needed
|
||||
// Or use existing ImGui/SDL2 rendering (already WASM compatible)
|
||||
```
|
||||
|
||||
### Files to Create
|
||||
- `src/app/emu/platform/wasm/wasm_audio.h`
|
||||
- `src/app/emu/platform/wasm/wasm_audio.cc`
|
||||
|
||||
---
|
||||
|
||||
## Feature Availability Matrix
|
||||
|
||||
| Feature | Native | WASM | Notes |
|
||||
|---------|--------|------|-------|
|
||||
| ROM Loading | File dialog | File input + IndexedDB | Full support |
|
||||
| ROM Saving | Direct write | Blob download | Full support |
|
||||
| Overworld Editor | Full | Full | No limitations |
|
||||
| Dungeon Editor | Full | Full | No limitations |
|
||||
| Graphics Editor | Full | Full | No limitations |
|
||||
| Palette Editor | Full | Full | No limitations |
|
||||
| Emulator | Full | Full | WebAudio for sound |
|
||||
| AI (Gemini) | HTTP | Browser Fetch | Requires API key |
|
||||
| AI (Ollama) | HTTP | Requires proxy | CORS limitation |
|
||||
| Asar Assembly | Full | In-memory | No file I/O |
|
||||
| Collaboration | gRPC | WebSocket | Different protocol |
|
||||
| Crash Recovery | File-based | IndexedDB | Equivalent |
|
||||
| Offline Mode | N/A | Service Worker | WASM-only feature |
|
||||
|
||||
---
|
||||
|
||||
## CMake Configuration
|
||||
|
||||
### Recommended Preset Updates
|
||||
```json
|
||||
{
|
||||
"name": "wasm-release",
|
||||
"cacheVariables": {
|
||||
"YAZE_BUILD_WASM_PLATFORM": "ON",
|
||||
"YAZE_WASM_ENABLE_WORKERS": "ON",
|
||||
"YAZE_WASM_ENABLE_OFFLINE": "ON",
|
||||
"YAZE_WASM_ENABLE_AI": "ON"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Source Organization
|
||||
```
|
||||
src/app/platform/
|
||||
wasm/
|
||||
wasm_storage.h/.cc
|
||||
wasm_file_dialog.h/.cc
|
||||
wasm_error_handler.h/.cc
|
||||
wasm_loading_manager.h/.cc
|
||||
wasm_settings.h/.cc
|
||||
wasm_autosave.h/.cc
|
||||
wasm_worker_pool.h/.cc
|
||||
wasm_browser_storage.h/.cc
|
||||
src/web/
|
||||
index.html
|
||||
shell.html
|
||||
style.css
|
||||
error_handler.js
|
||||
loading_indicator.js
|
||||
service-worker.js
|
||||
worker.js
|
||||
manifest.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Priority
|
||||
|
||||
### High Priority (Required for MVP)
|
||||
1. File System Layer (Phase 1)
|
||||
2. Error Handling (Phase 2)
|
||||
3. Progressive Loading (Phase 3)
|
||||
|
||||
### Medium Priority (Enhanced Experience)
|
||||
4. Local Storage Persistence (Phase 6)
|
||||
5. Offline Support (Phase 4)
|
||||
|
||||
### Lower Priority (Advanced Features)
|
||||
6. AI Integration (Phase 5)
|
||||
7. Web Workers (Phase 7)
|
||||
8. Emulator Audio (Phase 8)
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] User can load ROM from local file system
|
||||
- [ ] User can save modified ROM to downloads
|
||||
- [ ] Loading progress is visible with cancel option
|
||||
- [ ] Errors display user-friendly messages
|
||||
- [ ] Settings persist across sessions
|
||||
- [ ] App works offline after first load
|
||||
- [ ] AI assistance available via Gemini API
|
||||
- [ ] No UI freezes during heavy operations
|
||||
- [ ] Emulator runs with sound
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [Emscripten Documentation](https://emscripten.org/docs/)
|
||||
- [IndexedDB API](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API)
|
||||
- [Service Workers](https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API)
|
||||
- [Web Workers](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API)
|
||||
- [WebAudio API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API)
|
||||
@@ -0,0 +1,224 @@
|
||||
# WASM Web Features Roadmap
|
||||
|
||||
This document captures planned features for the browser-based yaze editor.
|
||||
|
||||
## Foundation (Completed)
|
||||
|
||||
The following infrastructure is in place:
|
||||
|
||||
- **Phase 1**: File System Layer (WasmStorage, WasmFileDialog) - IndexedDB storage
|
||||
- **Phase 2**: Error Handling (WasmErrorHandler) - Browser UI integration
|
||||
- **Phase 3**: Progressive Loading (WasmLoadingManager) - Cancellable loading with progress
|
||||
- **Phase 4**: Offline Support (service-worker.js, manifest.json) - PWA capabilities
|
||||
- **Phase 5**: AI Integration (BrowserAIService, WasmSecureStorage) - WebLLM ready
|
||||
- **Phase 6**: Local Storage (WasmSettings, WasmAutosave) - Preferences persistence
|
||||
- **Phase 7**: Web Workers (WasmWorkerPool) - Background task processing
|
||||
- **Phase 8**: WebAudio (WasmAudio) - SPC700 audio playback
|
||||
|
||||
---
|
||||
|
||||
## Phase 9: Enhanced File Handling
|
||||
|
||||
### 9.1 Drag & Drop ROM Loading
|
||||
- **Status**: Planned
|
||||
- **Priority**: High
|
||||
- **Description**: Enhanced drag-and-drop interface for ROM files
|
||||
- **Features**:
|
||||
- Visual drop zone with hover effects
|
||||
- File type validation before processing
|
||||
- Preview panel showing ROM metadata
|
||||
- Multiple file support (load ROM + patch together)
|
||||
- **Files**: `src/app/platform/wasm/wasm_drop_handler.{h,cc}`, `src/web/drop_zone.{js,css}`
|
||||
|
||||
### 9.2 Export Options
|
||||
- **Status**: Planned
|
||||
- **Priority**: High
|
||||
- **Description**: Export modifications as patches instead of full ROMs
|
||||
- **Features**:
|
||||
- BPS patch generation (standard format)
|
||||
- IPS patch generation (legacy support)
|
||||
- UPS patch generation (alternative)
|
||||
- Patch preview showing changed bytes
|
||||
- Direct download or save to IndexedDB
|
||||
- **Files**: `src/app/platform/wasm/wasm_patch_export.{h,cc}`
|
||||
|
||||
---
|
||||
|
||||
## Phase 10: Collaboration
|
||||
|
||||
### 10.1 Real-time Collaboration
|
||||
- **Status**: Planned
|
||||
- **Priority**: High
|
||||
- **Description**: Multi-user editing via WebSocket
|
||||
- **Features**:
|
||||
- Session creation/joining with room codes
|
||||
- User presence indicators (cursors, selections)
|
||||
- Change synchronization via operational transforms
|
||||
- Chat/comments sidebar
|
||||
- Permission levels (owner, editor, viewer)
|
||||
- **Files**: `src/app/platform/wasm/wasm_collaboration.{h,cc}`, `src/web/collaboration_ui.{js,css}`
|
||||
- **Dependencies**: EmscriptenWebSocket (completed)
|
||||
|
||||
### 10.2 ShareLink Generation
|
||||
- **Status**: Future
|
||||
- **Priority**: Medium
|
||||
- **Description**: Create shareable URLs with embedded patches
|
||||
- **Features**:
|
||||
- Base64-encoded diff in URL hash
|
||||
- Short URL generation via service
|
||||
- QR code generation for mobile sharing
|
||||
|
||||
### 10.3 Comment Annotations
|
||||
- **Status**: Future
|
||||
- **Priority**: Low
|
||||
- **Description**: Add notes to map locations/rooms
|
||||
- **Features**:
|
||||
- Pin comments to coordinates
|
||||
- Threaded discussions
|
||||
- Export annotations as JSON
|
||||
|
||||
---
|
||||
|
||||
## Phase 11: Browser-Specific Enhancements
|
||||
|
||||
### 11.1 Keyboard Shortcut Overlay
|
||||
- **Status**: Future
|
||||
- **Priority**: Medium
|
||||
- **Description**: Help panel showing all keyboard shortcuts
|
||||
- **Features**:
|
||||
- Toggle with `?` key
|
||||
- Context-aware (shows relevant shortcuts for current editor)
|
||||
- Searchable
|
||||
|
||||
### 11.2 Touch Support
|
||||
- **Status**: Future
|
||||
- **Priority**: Medium
|
||||
- **Description**: Touch gestures for tablet/mobile browsers
|
||||
- **Features**:
|
||||
- Pinch to zoom
|
||||
- Two-finger pan
|
||||
- Long press for context menu
|
||||
- Touch-friendly toolbar
|
||||
|
||||
### 11.3 Fullscreen Mode
|
||||
- **Status**: Future
|
||||
- **Priority**: Low
|
||||
- **Description**: Dedicated fullscreen API integration
|
||||
- **Features**:
|
||||
- F11 toggle
|
||||
- Auto-hide toolbar in fullscreen
|
||||
- Escape to exit
|
||||
|
||||
### 11.4 Browser Notifications
|
||||
- **Status**: Future
|
||||
- **Priority**: Low
|
||||
- **Description**: Alert when long operations complete
|
||||
- **Features**:
|
||||
- Permission request flow
|
||||
- Build/export completion notifications
|
||||
- Background tab awareness
|
||||
|
||||
---
|
||||
|
||||
## Phase 12: AI Integration Enhancements
|
||||
|
||||
### 12.1 Browser-Local LLM
|
||||
- **Status**: Future
|
||||
- **Priority**: Medium
|
||||
- **Description**: In-browser AI using WebLLM
|
||||
- **Features**:
|
||||
- Model download/caching
|
||||
- Chat interface for ROM hacking questions
|
||||
- Code generation for ASM patches
|
||||
- **Dependencies**: BrowserAIService (completed)
|
||||
|
||||
### 12.2 ROM Analysis Reports
|
||||
- **Status**: Future
|
||||
- **Priority**: Low
|
||||
- **Description**: Generate sharable HTML reports
|
||||
- **Features**:
|
||||
- Modification summary
|
||||
- Changed areas visualization
|
||||
- Exportable as standalone HTML
|
||||
|
||||
---
|
||||
|
||||
## Phase 13: Performance Optimizations
|
||||
|
||||
### 13.1 WebGPU Rendering
|
||||
- **Status**: Future
|
||||
- **Priority**: Medium
|
||||
- **Description**: Modern GPU acceleration
|
||||
- **Features**:
|
||||
- Feature detection with fallback to WebGL
|
||||
- Hardware-accelerated tile rendering
|
||||
- Shader-based effects
|
||||
|
||||
### 13.2 Lazy Tile Loading
|
||||
- **Status**: Future
|
||||
- **Priority**: Medium
|
||||
- **Description**: Load only visible map sections
|
||||
- **Features**:
|
||||
- Virtual scrolling for large maps
|
||||
- Tile cache management
|
||||
- Preload adjacent areas
|
||||
|
||||
---
|
||||
|
||||
## Phase 14: Cloud Features
|
||||
|
||||
### 14.1 Cloud ROM Storage
|
||||
- **Status**: Future
|
||||
- **Priority**: Low
|
||||
- **Description**: Optional cloud sync for projects
|
||||
- **Features**:
|
||||
- User accounts
|
||||
- Project backup/restore
|
||||
- Cross-device sync
|
||||
|
||||
### 14.2 Screenshot Gallery
|
||||
- **Status**: Future
|
||||
- **Priority**: Low
|
||||
- **Description**: Save emulator screenshots
|
||||
- **Features**:
|
||||
- Auto-capture on emulator test
|
||||
- Gallery view in IndexedDB
|
||||
- Share to social media
|
||||
|
||||
---
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### Technology Stack
|
||||
- **Storage**: IndexedDB via WasmStorage
|
||||
- **Networking**: EmscriptenWebSocket for real-time features
|
||||
- **Background Processing**: WasmWorkerPool (pthread-based)
|
||||
- **Audio**: WebAudio API via WasmAudio
|
||||
- **UI**: ImGui with HTML overlays for browser-specific elements
|
||||
|
||||
### File Organization
|
||||
```
|
||||
src/app/platform/wasm/
|
||||
├── wasm_storage.{h,cc} # Phase 1 ✓
|
||||
├── wasm_file_dialog.{h,cc} # Phase 1 ✓
|
||||
├── wasm_error_handler.{h,cc} # Phase 2 ✓
|
||||
├── wasm_loading_manager.{h,cc} # Phase 3 ✓
|
||||
├── wasm_settings.{h,cc} # Phase 6 ✓
|
||||
├── wasm_autosave.{h,cc} # Phase 6 ✓
|
||||
├── wasm_browser_storage.{h,cc} # Phase 5 ✓
|
||||
├── wasm_worker_pool.{h,cc} # Phase 7 ✓
|
||||
├── wasm_drop_handler.{h,cc} # Phase 9 (planned)
|
||||
├── wasm_patch_export.{h,cc} # Phase 9 (planned)
|
||||
└── wasm_collaboration.{h,cc} # Phase 10 (planned)
|
||||
|
||||
src/app/emu/platform/wasm/
|
||||
└── wasm_audio.{h,cc} # Phase 8 ✓
|
||||
|
||||
src/web/
|
||||
├── error_handler.{js,css} # Phase 2 ✓
|
||||
├── loading_indicator.{js,css} # Phase 3 ✓
|
||||
├── service-worker.js # Phase 4 ✓
|
||||
├── manifest.json # Phase 4 ✓
|
||||
├── drop_zone.{js,css} # Phase 9 (planned)
|
||||
└── collaboration_ui.{js,css} # Phase 10 (planned)
|
||||
```
|
||||
@@ -0,0 +1,344 @@
|
||||
# WASM Widget Tracking Implementation
|
||||
|
||||
**Date**: 2025-11-24
|
||||
**Author**: Claude (AI Agent)
|
||||
**Status**: Implemented
|
||||
**Related Files**:
|
||||
- `/Users/scawful/Code/yaze/src/app/platform/wasm/wasm_control_api.cc`
|
||||
- `/Users/scawful/Code/yaze/src/app/gui/automation/widget_id_registry.h`
|
||||
- `/Users/scawful/Code/yaze/src/app/gui/automation/widget_measurement.h`
|
||||
- `/Users/scawful/Code/yaze/src/app/controller.cc`
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the implementation of actual ImGui widget bounds tracking for GUI automation in the YAZE WASM build. The system replaces placeholder hardcoded bounds with real-time widget position data from the `WidgetIdRegistry`.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The original `WasmControlApi::GetUIElementTree()` and `GetUIElementBounds()` implementations returned hardcoded placeholder bounds:
|
||||
|
||||
```cpp
|
||||
// OLD: Hardcoded placeholder
|
||||
elem["bounds"] = {{"x", 0}, {"y", 0}, {"width", 100}, {"height", 30}};
|
||||
```
|
||||
|
||||
This prevented accurate GUI automation, as agents and test frameworks couldn't reliably click on or query widget positions.
|
||||
|
||||
## Solution Architecture
|
||||
|
||||
### 1. Existing Infrastructure (Already in Place)
|
||||
|
||||
YAZE already had a comprehensive widget tracking system:
|
||||
|
||||
- **`WidgetIdRegistry`** (`src/app/gui/automation/widget_id_registry.h`): Centralized registry that tracks all ImGui widgets with their bounds, visibility, and state
|
||||
- **`WidgetMeasurement`** (`src/app/gui/automation/widget_measurement.h`): Measures widget dimensions using `ImGui::GetItemRectMin()` and `ImGui::GetItemRectMax()`
|
||||
- **Frame lifecycle hooks**: `BeginFrame()` and `EndFrame()` calls already integrated in `Controller::OnLoad()` (lines 96-98)
|
||||
|
||||
### 2. Integration with WASM Control API
|
||||
|
||||
The implementation connects the WASM API to the existing widget registry:
|
||||
|
||||
#### Updated `GetUIElementTree()`
|
||||
|
||||
**File**: `/Users/scawful/Code/yaze/src/app/platform/wasm/wasm_control_api.cc` (lines 1386-1433)
|
||||
|
||||
```cpp
|
||||
std::string WasmControlApi::GetUIElementTree() {
|
||||
nlohmann::json result;
|
||||
|
||||
if (!IsReady()) {
|
||||
result["error"] = "Control API not initialized";
|
||||
result["elements"] = nlohmann::json::array();
|
||||
return result.dump();
|
||||
}
|
||||
|
||||
// Query the WidgetIdRegistry for all registered widgets
|
||||
auto& registry = gui::WidgetIdRegistry::Instance();
|
||||
const auto& all_widgets = registry.GetAllWidgets();
|
||||
|
||||
nlohmann::json elements = nlohmann::json::array();
|
||||
|
||||
// Convert WidgetInfo to JSON elements
|
||||
for (const auto& [path, info] : all_widgets) {
|
||||
nlohmann::json elem;
|
||||
elem["id"] = info.full_path;
|
||||
elem["type"] = info.type;
|
||||
elem["label"] = info.label;
|
||||
elem["enabled"] = info.enabled;
|
||||
elem["visible"] = info.visible;
|
||||
elem["window"] = info.window_name;
|
||||
|
||||
// Add bounds if available
|
||||
if (info.bounds.valid) {
|
||||
elem["bounds"] = {
|
||||
{"x", info.bounds.min_x},
|
||||
{"y", info.bounds.min_y},
|
||||
{"width", info.bounds.max_x - info.bounds.min_x},
|
||||
{"height", info.bounds.max_y - info.bounds.min_y}
|
||||
};
|
||||
} else {
|
||||
elem["bounds"] = {
|
||||
{"x", 0}, {"y", 0}, {"width", 0}, {"height", 0}
|
||||
};
|
||||
}
|
||||
|
||||
// Add metadata
|
||||
if (!info.description.empty()) {
|
||||
elem["description"] = info.description;
|
||||
}
|
||||
elem["imgui_id"] = static_cast<uint32_t>(info.imgui_id);
|
||||
elem["last_seen_frame"] = info.last_seen_frame;
|
||||
|
||||
elements.push_back(elem);
|
||||
}
|
||||
|
||||
result["elements"] = elements;
|
||||
result["count"] = elements.size();
|
||||
result["source"] = "WidgetIdRegistry";
|
||||
|
||||
return result.dump();
|
||||
}
|
||||
```
|
||||
|
||||
**Changes**:
|
||||
- Removed hardcoded editor-specific element generation
|
||||
- Queries `WidgetIdRegistry::GetAllWidgets()` for real widget data
|
||||
- Returns actual bounds from `info.bounds` if valid
|
||||
- Includes metadata: `imgui_id`, `last_seen_frame`, `description`
|
||||
- Adds `source: "WidgetIdRegistry"` to JSON for debugging
|
||||
|
||||
#### Updated `GetUIElementBounds()`
|
||||
|
||||
**File**: `/Users/scawful/Code/yaze/src/app/platform/wasm/wasm_control_api.cc` (lines ~1435+)
|
||||
|
||||
```cpp
|
||||
std::string WasmControlApi::GetUIElementBounds(const std::string& element_id) {
|
||||
nlohmann::json result;
|
||||
|
||||
if (!IsReady()) {
|
||||
result["error"] = "Control API not initialized";
|
||||
return result.dump();
|
||||
}
|
||||
|
||||
// Query the WidgetIdRegistry for the specific widget
|
||||
auto& registry = gui::WidgetIdRegistry::Instance();
|
||||
const auto* widget_info = registry.GetWidgetInfo(element_id);
|
||||
|
||||
result["id"] = element_id;
|
||||
|
||||
if (widget_info == nullptr) {
|
||||
result["found"] = false;
|
||||
result["error"] = "Element not found: " + element_id;
|
||||
return result.dump();
|
||||
}
|
||||
|
||||
result["found"] = true;
|
||||
result["visible"] = widget_info->visible;
|
||||
result["enabled"] = widget_info->enabled;
|
||||
result["type"] = widget_info->type;
|
||||
result["label"] = widget_info->label;
|
||||
result["window"] = widget_info->window_name;
|
||||
|
||||
// Add bounds if available
|
||||
if (widget_info->bounds.valid) {
|
||||
result["x"] = widget_info->bounds.min_x;
|
||||
result["y"] = widget_info->bounds.min_y;
|
||||
result["width"] = widget_info->bounds.max_x - widget_info->bounds.min_x;
|
||||
result["height"] = widget_info->bounds.max_y - widget_info->bounds.min_y;
|
||||
result["bounds_valid"] = true;
|
||||
} else {
|
||||
result["x"] = 0;
|
||||
result["y"] = 0;
|
||||
result["width"] = 0;
|
||||
result["height"] = 0;
|
||||
result["bounds_valid"] = false;
|
||||
}
|
||||
|
||||
// Add metadata
|
||||
result["imgui_id"] = static_cast<uint32_t>(widget_info->imgui_id);
|
||||
result["last_seen_frame"] = widget_info->last_seen_frame;
|
||||
|
||||
if (!widget_info->description.empty()) {
|
||||
result["description"] = widget_info->description;
|
||||
}
|
||||
|
||||
return result.dump();
|
||||
}
|
||||
```
|
||||
|
||||
**Changes**:
|
||||
- Removed hardcoded element ID pattern matching
|
||||
- Queries `WidgetIdRegistry::GetWidgetInfo(element_id)` for specific widget
|
||||
- Returns `found: false` if widget doesn't exist
|
||||
- Returns actual bounds with `bounds_valid` flag
|
||||
- Includes full widget metadata
|
||||
|
||||
### 3. Frame Lifecycle Integration
|
||||
|
||||
**File**: `/Users/scawful/Code/yaze/src/app/controller.cc` (lines 96-98)
|
||||
|
||||
The widget registry is already integrated into the main render loop:
|
||||
|
||||
```cpp
|
||||
absl::Status Controller::OnLoad() {
|
||||
// ... ImGui::NewFrame() setup ...
|
||||
|
||||
gui::WidgetIdRegistry::Instance().BeginFrame();
|
||||
absl::Status update_status = editor_manager_.Update();
|
||||
gui::WidgetIdRegistry::Instance().EndFrame();
|
||||
|
||||
RETURN_IF_ERROR(update_status);
|
||||
return absl::OkStatus();
|
||||
}
|
||||
```
|
||||
|
||||
**Frame Lifecycle**:
|
||||
1. `BeginFrame()`: Resets `seen_in_current_frame` flag for all widgets
|
||||
2. Widget rendering: Editors register widgets during `editor_manager_.Update()`
|
||||
3. `EndFrame()`: Marks unseen widgets as invisible, prunes stale entries
|
||||
|
||||
### 4. Widget Registration (Future Work)
|
||||
|
||||
**Current State**: Widget registration infrastructure exists but **editors are not yet registering widgets**.
|
||||
|
||||
**Registration Pattern** (to be implemented in editors):
|
||||
|
||||
```cpp
|
||||
// Example: Dungeon Editor registering a card
|
||||
{
|
||||
gui::WidgetIdScope scope("DungeonEditor");
|
||||
|
||||
if (ImGui::Begin("Room Selector##dungeon")) {
|
||||
// Widget now has full path: "DungeonEditor/Room Selector"
|
||||
|
||||
if (ImGui::Button("Load Room")) {
|
||||
// After rendering button, register it
|
||||
gui::WidgetIdRegistry::Instance().RegisterWidget(
|
||||
scope.GetWidgetPath("button", "Load Room"),
|
||||
"button",
|
||||
ImGui::GetItemID(),
|
||||
"Loads the selected room into the editor"
|
||||
);
|
||||
}
|
||||
}
|
||||
ImGui::End();
|
||||
}
|
||||
```
|
||||
|
||||
**Macros Available**:
|
||||
- `YAZE_WIDGET_SCOPE(name)`: RAII scope for hierarchical widget paths
|
||||
- `YAZE_REGISTER_WIDGET(type, name)`: Register widget after rendering
|
||||
- `YAZE_REGISTER_CURRENT_WIDGET(type)`: Auto-extract widget name from ImGui
|
||||
|
||||
## API Usage
|
||||
|
||||
### JavaScript API
|
||||
|
||||
**Get All UI Elements**:
|
||||
```javascript
|
||||
const elements = window.yaze.control.getUIElementTree();
|
||||
console.log(elements);
|
||||
// Output:
|
||||
// {
|
||||
// "elements": [
|
||||
// {
|
||||
// "id": "DungeonEditor/RoomSelector/button:LoadRoom",
|
||||
// "type": "button",
|
||||
// "label": "Load Room",
|
||||
// "visible": true,
|
||||
// "enabled": true,
|
||||
// "window": "DungeonEditor",
|
||||
// "bounds": {"x": 150, "y": 200, "width": 100, "height": 30},
|
||||
// "imgui_id": 12345,
|
||||
// "last_seen_frame": 4567
|
||||
// }
|
||||
// ],
|
||||
// "count": 1,
|
||||
// "source": "WidgetIdRegistry"
|
||||
// }
|
||||
```
|
||||
|
||||
**Get Specific Widget Bounds**:
|
||||
```javascript
|
||||
const bounds = window.yaze.control.getUIElementBounds("DungeonEditor/RoomSelector/button:LoadRoom");
|
||||
console.log(bounds);
|
||||
// Output:
|
||||
// {
|
||||
// "id": "DungeonEditor/RoomSelector/button:LoadRoom",
|
||||
// "found": true,
|
||||
// "visible": true,
|
||||
// "enabled": true,
|
||||
// "type": "button",
|
||||
// "label": "Load Room",
|
||||
// "window": "DungeonEditor",
|
||||
// "x": 150,
|
||||
// "y": 200,
|
||||
// "width": 100,
|
||||
// "height": 30,
|
||||
// "bounds_valid": true,
|
||||
// "imgui_id": 12345,
|
||||
// "last_seen_frame": 4567
|
||||
// }
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
1. **Memory**: `WidgetIdRegistry` stores widget metadata in `std::unordered_map`, which grows with UI complexity. Stale widgets are pruned after 600 frames of inactivity.
|
||||
|
||||
2. **CPU Overhead**:
|
||||
- `BeginFrame()`: O(n) iteration to reset flags (n = number of widgets)
|
||||
- Widget registration: O(1) hash map lookup/insert
|
||||
- `EndFrame()`: O(n) iteration for pruning stale entries
|
||||
|
||||
3. **Optimization**: Widget measurement can be disabled globally:
|
||||
```cpp
|
||||
gui::WidgetMeasurement::Instance().SetEnabled(false);
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
**Manual Test (WASM Build)**:
|
||||
```bash
|
||||
# Build WASM
|
||||
./scripts/build-wasm.sh
|
||||
|
||||
# Serve locally
|
||||
cd build-wasm
|
||||
python3 -m http.server 8080
|
||||
|
||||
# Open browser console
|
||||
window.yaze.control.getUIElementTree();
|
||||
```
|
||||
|
||||
**Expected Behavior**:
|
||||
- Initially, `elements` array will be empty (no widgets registered yet)
|
||||
- After editors implement registration, widgets will appear with real bounds
|
||||
- `bounds_valid: false` for widgets not yet rendered in current frame
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Add widget registration to editors**:
|
||||
- `DungeonEditorV2`: Register room tabs, cards, buttons
|
||||
- `OverworldEditor`: Register canvas, tile selectors, property panels
|
||||
- `GraphicsEditor`: Register graphics sheets, palette pickers
|
||||
|
||||
2. **Add registration helpers**:
|
||||
- Create `AgentUI::RegisterButton()`, `AgentUI::RegisterCard()` wrappers
|
||||
- Auto-register common widget patterns (cards with visibility flags)
|
||||
|
||||
3. **Extend API**:
|
||||
- `FindWidgetsByPattern(pattern)`: Search widgets by regex
|
||||
- `ClickWidget(element_id)`: Simulate click via automation API
|
||||
|
||||
## References
|
||||
|
||||
- Widget ID Registry: `/Users/scawful/Code/yaze/src/app/gui/automation/widget_id_registry.h`
|
||||
- Widget Measurement: `/Users/scawful/Code/yaze/src/app/gui/automation/widget_measurement.h`
|
||||
- WASM Control API: `/Users/scawful/Code/yaze/src/app/platform/wasm/wasm_control_api.h`
|
||||
- Controller Integration: `/Users/scawful/Code/yaze/src/app/controller.cc` (lines 96-98)
|
||||
|
||||
## Revision History
|
||||
|
||||
| Date | Author | Changes |
|
||||
|------------|--------|--------------------------------------------|
|
||||
| 2025-11-24 | Claude | Initial implementation and documentation |
|
||||
Reference in New Issue
Block a user