Add Overworld tile search command and enhance related documentation

- Implemented `overworld-find-tile` command in the agent for searching tiles by ID.
- Updated `README.md` and `AGENT-ROADMAP.md` to reflect new command and usage.
- Enhanced `overworld_inspect` module with tile matching functionality.
This commit is contained in:
scawful
2025-10-03 15:57:42 -04:00
parent 21ef67c1ca
commit 467b0926e5
9 changed files with 325 additions and 89 deletions

View File

@@ -59,34 +59,30 @@ This vision will be realized through a shared interface available in both the `z
### Immediate Priorities
1. **Expand Overworld Tool Coverage**:
- Add read-only commands for tile searches, area summaries, and teleport destinations.
- Guarantee each tool returns both JSON and human-readable summaries for the chat renderers.
2. **Document & Test the New Tooling**:
- Update the main `README.md` and relevant docs to cover the new chat formatting.
- Add regression tests (unit or golden JSON fixtures) for the new Overworld tools.
3. **Polish the TUI Chat Experience**:
- ✅ Ship read-only tile searches (`overworld find-tile`) with shared formatting for CLI and agent calls.
- Next: add area summaries, teleport destination lookups, and keep JSON/Text parity for all new tools.
2. **Polish the TUI Chat Experience**:
- Tighten keyboard shortcuts, scrolling, and copy-to-clipboard behaviour.
- Align log file output with on-screen formatting for easier debugging.
4. **Integrate Tool Use with LLM**:
3. **Integrate Tool Use with LLM**:
- Modify the `AIService` to support function calling/tool use.
- Teach the agent to call the new read-only commands to answer questions.
5. **Land Overworld Tooling**:
- Ship at least two Overworld inspection commands with comprehensive tests.
6. **Build GUI Chat Widget**:
4. **Document & Test the New Tooling**:
- Update the main `README.md` and relevant docs to cover the new chat formatting.
- Add regression tests (unit or golden JSON fixtures) for the new Overworld tools.
5. **Build GUI Chat Widget**:
- Create the ImGui component.
- Ensure it shares the same backend service as the TUI.
7. **Full Integration with Proposal System**:
6. **Full Integration with Proposal System**:
- Implement the logic for the agent to transition from conversation to proposal generation.
8. **Expand Tool Arsenal**:
7. **Expand Tool Arsenal**:
- Continuously add new read-only commands to give the agent more capabilities to inspect the ROM.
9. **Multi-Modal Agent**:
8. **Multi-Modal Agent**:
- Explore the possibility of the agent generating and displaying images (e.g., a map of a dungeon room) in the chat.
10. **Advanced Configuration**:
9. **Advanced Configuration**:
- Implement environment variables for selecting AI providers and models (e.g., `YAZE_AI_PROVIDER`, `OLLAMA_MODEL`).
- Add CLI flags for overriding the provider and model on a per-command basis.
11. **Performance and Cost-Saving**:
10. **Performance and Cost-Saving**:
- Implement a response cache to reduce latency and API costs.
- Add token usage tracking and reporting.
@@ -103,10 +99,13 @@ We have made significant progress in laying the foundation for the conversationa
- **Tool Loop Improvements**: Conversational flow now handles multi-step tool calls with default JSON output, allowing results to feed back into the chat without recursion.
- **Structured Tool Output Rendering**: Both the TUI and GUI chat widgets now display tables and JSON payloads with friendly formatting, drastically improving readability.
- **Overworld Inspection Suite**: Added `overworld describe-map` and `overworld list-warps` commands producing text/JSON summaries for map metadata and warp points, with agent tooling hooks.
### ✅ Build Configuration Issue Resolved
The linker error is fixed. Both the CLI and GUI targets now link against `yaze_agent`, so the shared agent handlers (`HandleResourceListCommand`, `HandleDungeonListSpritesCommand`, etc.) compile once and are available to `ToolDispatcher` everywhere.
- **Overworld Tile Search Tool**: Added `overworld find-tile` across CLI and agent tooling with shared ROM context handling and regression tests.
### 🚀 Next Steps
1. **Share ROM Context with the Agent**: Inject the active GUI ROM into `ConversationalAgentService` so tool calls work even when `--rom` flags are unavailable. Analyze the `src/app/rom.cc` and `src/app/rom.h` and `src/app/editor/editor_manager.cc` files for guidance on accessing the current project/ROM.
2. **Expand Tool Coverage**: Target Overworld navigation helpers (`overworld find-tile`, `overworld list-warps`, region summaries) and dialogue inspectors. Prioritize commands that unblock common level-design questions and emit concise table/JSON payloads.
1. **Integrate Tool Use with LLM**:
- Modify the `AIService` to support function calling/tool use.
- Teach the agent to call the new read-only commands to answer questions.
2. **Polish the TUI Chat Experience**:
- Tighten keyboard shortcuts, scrolling, and copy-to-clipboard behaviour.
- Align log file output with on-screen formatting for easier debugging.
2. **Expand Tool Coverage**: Target additional Overworld navigation helpers (region summaries, teleport lookups) and dialogue inspectors. Prioritize commands that unblock common level-design questions and emit concise table/JSON payloads.

View File

@@ -54,6 +54,9 @@ z3ed agent resource-list --type dungeon --format json
# Dump sprite placements for a dungeon room
z3ed agent dungeon-list-sprites --room 0x012
# Search overworld maps for a tile ID using shared agent tooling
z3ed agent overworld-find-tile --tile 0x02E --map 0x05
```
### GUI Testing Commands
@@ -201,6 +204,9 @@ z3ed overworld find-tile --tile 0x02E --format json
# Narrow search to Light World map 0x05
z3ed overworld find-tile --tile 0x02E --map 0x05
# Ask the agent to perform the same lookup (returns JSON by default)
z3ed agent overworld-find-tile --tile 0x02E --map 0x05
```
### Label-Aware Dungeon Edit

View File

@@ -12,7 +12,7 @@ namespace agent {
namespace {
constexpr absl::string_view kUsage =
"Usage: agent <run|plan|diff|accept|test|gui|learn|list|commit|revert|describe|resource-list|dungeon-list-sprites|chat> "
"Usage: agent <run|plan|diff|accept|test|gui|learn|list|commit|revert|describe|resource-list|dungeon-list-sprites|overworld-find-tile|overworld-describe-map|overworld-list-warps|chat> "
"[options]";
} // namespace
@@ -65,6 +65,15 @@ absl::Status Agent::Run(const std::vector<std::string>& arg_vec) {
if (subcommand == "dungeon-list-sprites") {
return agent::HandleDungeonListSpritesCommand(subcommand_args);
}
if (subcommand == "overworld-find-tile") {
return agent::HandleOverworldFindTileCommand(subcommand_args);
}
if (subcommand == "overworld-describe-map") {
return agent::HandleOverworldDescribeMapCommand(subcommand_args);
}
if (subcommand == "overworld-list-warps") {
return agent::HandleOverworldListWarpsCommand(subcommand_args);
}
if (subcommand == "chat") {
return agent::HandleChatCommand(rom_);
}

View File

@@ -31,6 +31,9 @@ absl::Status HandleResourceListCommand(
absl::Status HandleDungeonListSpritesCommand(
const std::vector<std::string>& arg_vec,
Rom* rom_context = nullptr);
absl::Status HandleOverworldFindTileCommand(
const std::vector<std::string>& arg_vec,
Rom* rom_context = nullptr);
absl::Status HandleOverworldDescribeMapCommand(
const std::vector<std::string>& arg_vec,
Rom* rom_context = nullptr);

View File

@@ -209,6 +209,174 @@ absl::Status HandleDungeonListSpritesCommand(
return absl::OkStatus();
}
absl::Status HandleOverworldFindTileCommand(
const std::vector<std::string>& arg_vec, Rom* rom_context) {
std::optional<std::string> tile_value;
std::optional<std::string> map_value;
std::optional<std::string> world_value;
std::string format = "json";
std::optional<std::string> rom_override;
for (size_t i = 0; i < arg_vec.size(); ++i) {
const std::string& token = arg_vec[i];
if (token == "--tile") {
if (i + 1 >= arg_vec.size()) {
return absl::InvalidArgumentError("--tile requires a value.");
}
tile_value = arg_vec[++i];
} else if (absl::StartsWith(token, "--tile=")) {
tile_value = token.substr(7);
} else if (token == "--map") {
if (i + 1 >= arg_vec.size()) {
return absl::InvalidArgumentError("--map requires a value.");
}
map_value = arg_vec[++i];
} else if (absl::StartsWith(token, "--map=")) {
map_value = token.substr(6);
} else if (token == "--world") {
if (i + 1 >= arg_vec.size()) {
return absl::InvalidArgumentError("--world requires a value.");
}
world_value = arg_vec[++i];
} else if (absl::StartsWith(token, "--world=")) {
world_value = token.substr(8);
} else if (token == "--format") {
if (i + 1 >= arg_vec.size()) {
return absl::InvalidArgumentError("--format requires a value.");
}
format = absl::AsciiStrToLower(arg_vec[++i]);
} else if (absl::StartsWith(token, "--format=")) {
format = absl::AsciiStrToLower(token.substr(9));
} else if (token == "--rom") {
if (i + 1 >= arg_vec.size()) {
return absl::InvalidArgumentError("--rom requires a value.");
}
rom_override = arg_vec[++i];
} else if (absl::StartsWith(token, "--rom=")) {
rom_override = token.substr(6);
}
}
if (!tile_value.has_value()) {
return absl::InvalidArgumentError(
"Usage: agent overworld-find-tile --tile <id> [--map <id>] [--world <light|dark|special>] [--format <json|text>]");
}
ASSIGN_OR_RETURN(int tile_numeric,
overworld::ParseNumeric(*tile_value));
if (tile_numeric < 0 || tile_numeric > 0xFFFF) {
return absl::InvalidArgumentError(
absl::StrCat("Tile ID must be between 0x0000 and 0xFFFF (got ",
*tile_value, ")"));
}
std::optional<int> map_filter;
if (map_value.has_value()) {
ASSIGN_OR_RETURN(int parsed_map,
overworld::ParseNumeric(*map_value));
if (parsed_map < 0 || parsed_map >= zelda3::kNumOverworldMaps) {
return absl::InvalidArgumentError(
absl::StrCat("Map ID out of range: ", *map_value));
}
map_filter = parsed_map;
}
std::optional<int> world_filter;
if (world_value.has_value()) {
ASSIGN_OR_RETURN(int parsed_world,
overworld::ParseWorldSpecifier(*world_value));
world_filter = parsed_world;
}
if (map_filter.has_value()) {
ASSIGN_OR_RETURN(int inferred_world,
overworld::InferWorldFromMapId(*map_filter));
if (world_filter.has_value() && inferred_world != *world_filter) {
return absl::InvalidArgumentError(
absl::StrCat("Map 0x",
absl::StrFormat("%02X", *map_filter),
" belongs to the ",
overworld::WorldName(inferred_world),
" World but --world requested ",
overworld::WorldName(*world_filter)));
}
if (!world_filter.has_value()) {
world_filter = inferred_world;
}
}
if (format != "json" && format != "text") {
return absl::InvalidArgumentError(
absl::StrCat("Unsupported format: ", format));
}
Rom rom_storage;
Rom* rom = nullptr;
if (rom_context != nullptr && rom_context->is_loaded() && !rom_override.has_value()) {
rom = rom_context;
} else {
ASSIGN_OR_RETURN(auto rom_or,
LoadRomFromPathOrFlag(rom_override));
rom_storage = std::move(rom_or);
rom = &rom_storage;
}
zelda3::Overworld overworld_data(rom);
auto load_status = overworld_data.Load(rom);
if (!load_status.ok()) {
return load_status;
}
overworld::TileSearchOptions search_options;
search_options.map_id = map_filter;
search_options.world = world_filter;
ASSIGN_OR_RETURN(auto matches,
overworld::FindTileMatches(overworld_data,
static_cast<uint16_t>(tile_numeric),
search_options));
if (format == "json") {
std::cout << "{\n";
std::cout << absl::StrFormat(
" \"tile\": \"0x%04X\",\n", tile_numeric);
std::cout << absl::StrFormat(
" \"match_count\": %zu,\n", matches.size());
std::cout << " \"matches\": [\n";
for (size_t i = 0; i < matches.size(); ++i) {
const auto& match = matches[i];
std::cout << absl::StrFormat(
" {\"map\": \"0x%02X\", \"world\": \"%s\", "
"\"local\": {\"x\": %d, \"y\": %d}, "
"\"global\": {\"x\": %d, \"y\": %d}}%s\n",
match.map_id, overworld::WorldName(match.world), match.local_x,
match.local_y,
match.global_x, match.global_y,
(i + 1 == matches.size()) ? "" : ",");
}
std::cout << " ]\n";
std::cout << "}\n";
} else {
std::cout << absl::StrFormat(
"🔎 Tile 0x%04X → %zu match(es)\n",
tile_numeric, matches.size());
if (matches.empty()) {
std::cout << " No matches found." << std::endl;
return absl::OkStatus();
}
for (const auto& match : matches) {
std::cout << absl::StrFormat(
" • Map 0x%02X (%s World) local(%2d,%2d) global(%3d,%3d)\n",
match.map_id, overworld::WorldName(match.world), match.local_x,
match.local_y,
match.global_x, match.global_y);
}
}
return absl::OkStatus();
}
absl::Status HandleOverworldDescribeMapCommand(
const std::vector<std::string>& arg_vec, Rom* rom_context) {
std::optional<std::string> map_value;

View File

@@ -244,73 +244,13 @@ absl::Status OverworldFindTile::Run(const std::vector<std::string>& arg_vec) {
return ow_status;
}
struct TileMatch {
int map_id;
int world;
int local_x;
int local_y;
int global_x;
int global_y;
};
overworld::TileSearchOptions search_options;
search_options.map_id = map_filter;
search_options.world = world_filter;
std::vector<int> worlds_to_search;
if (world_filter.has_value()) {
worlds_to_search.push_back(*world_filter);
} else {
worlds_to_search = {0, 1, 2};
}
std::vector<TileMatch> matches;
for (int world : worlds_to_search) {
int world_start = 0;
int world_maps = 0;
switch (world) {
case 0:
world_start = 0x00;
world_maps = 0x40;
break;
case 1:
world_start = 0x40;
world_maps = 0x40;
break;
case 2:
world_start = 0x80;
world_maps = 0x20;
break;
default:
return absl::InvalidArgumentError(
absl::StrCat("Unknown world index: ", world));
}
overworld.set_current_world(world);
for (int local_map = 0; local_map < world_maps; ++local_map) {
int map_id = world_start + local_map;
if (map_filter.has_value() && map_id != *map_filter) {
continue;
}
int map_x_index = local_map % 8;
int map_y_index = local_map / 8;
int global_x_start = map_x_index * 32;
int global_y_start = map_y_index * 32;
for (int local_y = 0; local_y < 32; ++local_y) {
for (int local_x = 0; local_x < 32; ++local_x) {
int global_x = global_x_start + local_x;
int global_y = global_y_start + local_y;
uint16_t tile = overworld.GetTile(global_x, global_y);
if (tile == tile_id) {
matches.push_back({map_id, world, local_x, local_y, global_x,
global_y});
}
}
}
}
}
ASSIGN_OR_RETURN(auto matches,
overworld::FindTileMatches(overworld, tile_id,
search_options));
if (format == "json") {
std::cout << "{\n";

View File

@@ -295,6 +295,97 @@ absl::StatusOr<std::vector<WarpEntry>> CollectWarpEntries(
return entries;
}
absl::StatusOr<std::vector<TileMatch>> FindTileMatches(
zelda3::Overworld& overworld, uint16_t tile_id,
const TileSearchOptions& options) {
if (options.map_id.has_value()) {
RETURN_IF_ERROR(ValidateMapId(*options.map_id));
}
if (options.world.has_value()) {
if (*options.world < 0 || *options.world > 2) {
return absl::InvalidArgumentError(
absl::StrFormat("Unknown world index: %d", *options.world));
}
}
if (options.map_id.has_value() && options.world.has_value()) {
ASSIGN_OR_RETURN(int inferred_world,
InferWorldFromMapId(*options.map_id));
if (inferred_world != *options.world) {
return absl::InvalidArgumentError(
absl::StrFormat(
"Map 0x%02X belongs to the %s World but --world requested %s",
*options.map_id, WorldName(inferred_world),
WorldName(*options.world)));
}
}
std::vector<int> worlds;
if (options.world.has_value()) {
worlds.push_back(*options.world);
} else if (options.map_id.has_value()) {
ASSIGN_OR_RETURN(int inferred_world,
InferWorldFromMapId(*options.map_id));
worlds.push_back(inferred_world);
} else {
worlds = {0, 1, 2};
}
std::vector<TileMatch> matches;
for (int world : worlds) {
int world_start = 0;
int world_maps = 0;
switch (world) {
case 0:
world_start = 0x00;
world_maps = 0x40;
break;
case 1:
world_start = 0x40;
world_maps = 0x40;
break;
case 2:
world_start = 0x80;
world_maps = 0x20;
break;
default:
return absl::InvalidArgumentError(
absl::StrFormat("Unknown world index: %d", world));
}
overworld.set_current_world(world);
for (int local_map = 0; local_map < world_maps; ++local_map) {
int map_id = world_start + local_map;
if (options.map_id.has_value() && map_id != *options.map_id) {
continue;
}
int map_x_index = local_map % 8;
int map_y_index = local_map / 8;
int global_x_start = map_x_index * 32;
int global_y_start = map_y_index * 32;
for (int local_y = 0; local_y < 32; ++local_y) {
for (int local_x = 0; local_x < 32; ++local_x) {
int global_x = global_x_start + local_x;
int global_y = global_y_start + local_y;
uint16_t current_tile = overworld.GetTile(global_x, global_y);
if (current_tile == tile_id) {
matches.push_back({map_id, world, local_x, local_y, global_x,
global_y});
}
}
}
}
}
return matches;
}
} // namespace overworld
} // namespace cli
} // namespace yaze

View File

@@ -81,6 +81,20 @@ struct WarpQuery {
std::optional<WarpType> type;
};
struct TileMatch {
int map_id;
int world;
int local_x;
int local_y;
int global_x;
int global_y;
};
struct TileSearchOptions {
std::optional<int> map_id;
std::optional<int> world;
};
absl::StatusOr<int> ParseNumeric(std::string_view value, int base = 0);
absl::StatusOr<int> ParseWorldSpecifier(std::string_view value);
absl::StatusOr<int> InferWorldFromMapId(int map_id);
@@ -93,6 +107,10 @@ absl::StatusOr<MapSummary> BuildMapSummary(zelda3::Overworld& overworld,
absl::StatusOr<std::vector<WarpEntry>> CollectWarpEntries(
const zelda3::Overworld& overworld, const WarpQuery& query);
absl::StatusOr<std::vector<TileMatch>> FindTileMatches(
zelda3::Overworld& overworld, uint16_t tile_id,
const TileSearchOptions& options = {});
} // namespace overworld
} // namespace cli
} // namespace yaze

View File

@@ -38,6 +38,8 @@ absl::StatusOr<std::string> ToolDispatcher::Dispatch(
status = HandleResourceListCommand(args, rom_context_);
} else if (tool_call.tool_name == "dungeon-list-sprites") {
status = HandleDungeonListSpritesCommand(args, rom_context_);
} else if (tool_call.tool_name == "overworld-find-tile") {
status = HandleOverworldFindTileCommand(args, rom_context_);
} else if (tool_call.tool_name == "overworld-describe-map") {
status = HandleOverworldDescribeMapCommand(args, rom_context_);
} else if (tool_call.tool_name == "overworld-list-warps") {