diff --git a/docs/architecture/generative-ui-architecture.md b/docs/architecture/generative-ui-architecture.md index 3a2e67dda4..63ec95bb44 100644 --- a/docs/architecture/generative-ui-architecture.md +++ b/docs/architecture/generative-ui-architecture.md @@ -4,7 +4,7 @@ 新回复可以保存为 `openbitfun://current-session/artifacts/visualizations/v2/.html`。FlowChat 按原始内容引用中的 URI 找到同一轮、同一子 Agent 的成功 Write,使用调用中保存的字节创建隔离的 iframe。同轮 Edit 在回复发出前生效。执行端在成功的 Write 和 Edit 结果中记录会话 URI;通过该会话本地存储目录的绝对路径修改图时,也会带上同一 URI。新的 Edit 还保存该次修改的完整源码。执行端完成对话轮次时,将最后一次通过检查的 Write 或 Edit 中的原始源码和 UTF-8 字节数保存在该轮记录中。旧版写入记录沿用完成时的文件快照。通过脚本生成的图也能发布,前提是该轮使用了可视化技能,且被引用的文件在该轮内生成或修改。FlowChat 收到已完成的回复和重开历史消息时都使用这份源码。读取旧回复时,执行端从已记录的 Write 或完整 Edit 开始,按顺序精确回放截至该条回复的修改。后续对话可以只修改已有图,新的回复使用当轮 Edit 返回的完整源码,上一条回复保持修改前的样子。生成成功后,图直接出现在回复里。若模型在最终文本里漏写内容引用,FlowChat 会在已完成的该轮回复中补上有成功记录的图,已引用的图只显示一次。原版图表的 `mountChart`、四个专用组件、地图、插图、变体轮播和设计控件都能使用同一入口。参考图仅用于测试。模型按用户的题目自行编排内容。 FlowChat 将当前聊天区域的颜色和字体通过展示通道传给已打开的图。切换主题或皮肤时,图内使用原版 CSS 变量的标记和控件会随之更新,交互状态留在原来的文档中。 -常规构建随源码提供 OpenBitFun 的技能入口、隔离渲染外壳、Apache ECharts 和离线地图。Vite 打包地图投影 worker 及其数据,Rust 嵌入技能。构建入口会通过 SHA-256 检查随仓库分发的依赖和地理数据,开发者也可以运行 `pnpm run visualize:verify` 单独检查。Write 和 Edit 在桌面后台实际渲染并返回本次产物的引用;无图形宿主时检查延至阅读端。Codex Visualize 1.0.103 的本地兼容套件是可选项。已有原包的开发者可以运行 `pnpm run visualize:import-reference` 导入扩展指南及兼容资源。 +常规构建随源码提供 OpenBitFun 的技能入口、隔离渲染外壳、Apache ECharts 和离线地图。Vite 打包地图投影 worker 及其数据,Rust 嵌入技能。构建入口会通过 SHA-256 检查随仓库分发的依赖和地理数据,开发者也可以运行 `pnpm run visualize:verify` 单独检查。Write 和 Edit 在桌面后台实际渲染并返回本次产物的引用;无图形宿主时检查延至阅读端,使用产生图表的那条用户请求。若街道图只保存了声明,读者可以选择联网加载底图;成功后,同一修订可在本机离线回放。Codex Visualize 1.0.109 的本地兼容套件是可选项。原包标注为专有许可,Git 仓库记录文件校验值。Rust 的 debug 和 release 构建均排除本机原包和扩展指南。只有桌面开发服务器在本地确认套件可用时才向模型提供扩展指南。按时间规划出行时可使用日历;询问位置、地图或空间关系时可使用地理底图。已有原包的开发者可以运行 `pnpm run visualize:import-reference` 导入扩展指南及兼容资源。 普通城市标注图可以在技能交付的 HTML 中声明地点及可选范围。FlowChat 使用随产品分发的 Natural Earth 底图与地名,将声明投影成 SVG,检查城市标注和底图并保存该次检查的 SVG。断网重开历史图时,地理图层已经包含在回复中。皮肤控制画布和字体,水域、陆地、标记各自使用独立的语义颜色。 @@ -32,7 +32,7 @@ FlowChat 按会话、轮次和子 Agent 身份查找该段文字之前最后一 ## 组件验收 -对照 Codex visualize 1.0.103 的 charts 1.6.5 与 widgets 1.0.59 文档,OpenBitFun 使用独立的声明式输入和本地渲染器。可信代码随产品提供。macOS WKWebView 也验证了图表筛选、参数滑块、请求自身耗时、非连续 CPU 调用与刷新恢复。以下用例通过实际组件数据验算以及 Chromium 中的 FlowChat 回复回放检查。可运行的验收位于 [图表断言](../../src/web-ui/src/tools/visualize/components.test.ts) 和 [浏览器用例](../../scripts/test-visualize-browser.mjs)。 +当前兼容测试覆盖 Visualize 1.0.109 的 charts 1.8.1、历史图使用的 charts 1.6.5,以及 widgets 1.0.59。OpenBitFun 使用独立的声明式输入和本地渲染器。可信代码随产品提供。macOS WKWebView 也验证了图表筛选、参数滑块、请求自身耗时、非连续 CPU 调用与刷新恢复。以下用例通过实际组件数据验算以及 Chromium 中的 FlowChat 回复回放检查。可运行的验收位于 [图表断言](../../src/web-ui/src/tools/visualize/components.test.ts) 和 [浏览器用例](../../scripts/test-visualize-browser.mjs)。 | 组件 | 核对内容 | | --- | --- | diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index dd0794ec99..fcbfb2e621 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -396,6 +396,9 @@ importers: src/web-ui: dependencies: + '@mapbox/vector-tile': + specifier: 3.0.0 + version: 3.0.0 '@monaco-editor/react': specifier: ^4.6.0 version: 4.7.0(monaco-editor@0.52.2)(react-dom@18.3.1(react@18.3.1))(react@18.3.1) @@ -546,6 +549,9 @@ importers: path-browserify: specifier: ^1.0.1 version: 1.0.1 + pbf: + specifier: 5.1.2 + version: 5.1.2 pdf-lib: specifier: 1.17.1 version: 1.17.1 diff --git a/scripts/dev.cjs b/scripts/dev.cjs index 37e1fee41c..292be30f07 100644 --- a/scripts/dev.cjs +++ b/scripts/dev.cjs @@ -7,6 +7,7 @@ const fs = require('fs'); const net = require('net'); +const { visualizeDevGuide } = require('./visualize-dev-kit.cjs'); const { execSync, spawn } = require('child_process'); const path = require('path'); const { pathToFileURL } = require('url'); @@ -817,6 +818,7 @@ async function main() { // Point Remote Connect at the live workspace dist so a newly generated // QR code never uploads a stale mobile-web bundle. OPENBITFUN_MOBILE_WEB_DIR: path.join(ROOT_DIR, 'src/mobile-web/dist'), + OPENBITFUN_VISUALIZE_DEV_GUIDE: visualizeDevGuide(ROOT_DIR), }; try { const args = ['dev', '--config', tauriConfig, '--config', JSON.stringify({ build: { devUrl: `http://localhost:${DEV_SERVER_PORT}` } })]; @@ -828,10 +830,7 @@ async function main() { await runWindowsCommandArgs(tauriBin, args, desktopDir, tauriDevEnv); } else { const tauriBin = path.join(ROOT_DIR, 'node_modules', '.bin', 'tauri'); - await spawnCommand(tauriBin, args, desktopDir, { - CARGO_PROFILE_DEV_CODEGEN_UNITS: tauriDevEnv.CARGO_PROFILE_DEV_CODEGEN_UNITS, - OPENBITFUN_MOBILE_WEB_DIR: tauriDevEnv.OPENBITFUN_MOBILE_WEB_DIR, - }); + await spawnCommand(tauriBin, args, desktopDir, tauriDevEnv); } } finally { // Option B: prune only when the desktop:dev session ends, not on each rebuild. diff --git a/scripts/import-visualize.mjs b/scripts/import-visualize.mjs index 3b58b91165..7284ac25cd 100644 --- a/scripts/import-visualize.mjs +++ b/scripts/import-visualize.mjs @@ -1,6 +1,6 @@ import assert from 'node:assert/strict'; import { createHash } from 'node:crypto'; -import { cp, lstat, mkdir, mkdtemp, readFile, readdir, rename, rm, writeFile } from 'node:fs/promises'; +import { cp, mkdir, mkdtemp, readFile, readdir, rename, rm, writeFile } from 'node:fs/promises'; import os from 'node:os'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; @@ -9,6 +9,7 @@ const root = fileURLToPath(new URL('../', import.meta.url)); const manifest = JSON.parse(await readFile(new URL('./visualize-upstream.json', import.meta.url), 'utf8')); const owner = path.join(root, 'src/crates/assembly/core/builtin_skills/visualize'); const target = path.join(owner, 'upstream'); +const cache = path.join(root, '.visualize-runtime'); const args = process.argv.slice(2); const check = args.includes('--check'); const paths = args.filter((arg) => arg !== '--check'); @@ -43,28 +44,62 @@ async function verify(directory) { assert.deepEqual(found.sort(), Object.keys(manifest.files).sort(), 'Missing upstream files'); } -if (!check && paths.length) await verify(source); -try { - await lstat(target); -} catch (e) { - if (check || e.code !== 'ENOENT') throw e; - if (!paths.length) await verify(source); - // Stage alongside the containing skill, outside the embedded skill tree. - const stage = await mkdtemp(path.join(root, '.visualize-import-')); +const installed = await readFile(path.join(target, '.codex-plugin/plugin.json'), 'utf8') + .then(JSON.parse) + .catch(error => { + if (error.code !== 'ENOENT') throw error; + }); +if (!check && (paths.length || installed?.version !== manifest.version)) await verify(source); +if (!check && installed?.version !== manifest.version) { + // Keep the previous local install outside the embedded skill tree during upgrades. + await mkdir(cache, { recursive: true }); + const stage = await mkdtemp(path.join(cache, 'import-')); + const backup = path.join(stage, 'previous'); + let preserveBackup = false; try { await cp(source, path.join(stage, 'package'), { recursive: true, preserveTimestamps: true }); await verify(path.join(stage, 'package')); - await rename(path.join(stage, 'package'), target); + if (installed) await rename(target, backup); + try { + await rename(path.join(stage, 'package'), target); + } catch (error) { + if (installed) { + preserveBackup = true; + await rename(backup, target); + preserveBackup = false; + } + throw error; + } } finally { - await rm(stage, { recursive: true, force: true }); + if (!preserveBackup) await rm(stage, { recursive: true, force: true }); } } await verify(target); +// Before this importer cleaned its staging directory, a successful upgrade +// left exactly one "previous" install. Leave interrupted swaps and unrelated +// cache entries alone. Only prune after verifying the current install. +if (!check) { + const stages = await readdir(cache, { withFileTypes: true }).catch(error => { + if (error.code === 'ENOENT') return []; + throw error; + }); + for (const stage of stages.filter(entry => entry.isDirectory() && /^import-[A-Za-z0-9]{6}$/.test(entry.name))) { + const dir = path.join(cache, stage.name); + const entries = await readdir(dir, { withFileTypes: true }).catch(() => []); + if (entries.length !== 1 || entries[0].name !== 'previous' || !entries[0].isDirectory()) continue; + const plugin = await readFile(path.join(dir, 'previous/.codex-plugin/plugin.json'), 'utf8').then(JSON.parse).catch(() => null); + if (plugin?.name === 'visualize' && typeof plugin.version === 'string' && plugin.version !== manifest.version) + await rm(dir, { recursive: true, force: true }); + } +} + // Keep the packaged guides and validation workflow intact. Only the resource // location and the final reference differ between Codex and FlowChat. const guidePrefix = 'skills/visualize/'; const uri = 'openbitfun://current-session/artifacts/visualizations/v2/.html'; +const streetMaps = 'For a geographic map write to map-<slug>.html. For a map of sightseeing, walking or transit stops within one city, use detail:"street", city, optional ISO country and the ordered public stop names in places. Submit the names directly. The host resolves stops, fetches the visible tiles, and saves the verified map for offline replay. Use a short local query and a matching osmTag (railway:station or highway:pedestrian for those stop types), or a reported osmId to identify the intended place. Ask about unresolved ambiguity. Sourced WGS84 coordinates are also accepted. Do not invent roads, shorelines, route geometry or scales. For a missing-map follow-up write the atlas declaration again in the current turn using the earlier stops. Repair it with the verified candidate coordinates in the current receipt. If name lookup fails for a public stop, get sourced WGS84 coordinates with WebSearch or WebFetch and resubmit the same map with a source credit. If coordinates or tiles remain unavailable, report the current error in text. An itinerary request may still show its schedule as an <ol data-visual-itinerary>. For explicitly offline or private locations use sourced local geometry. For figures with bound data supply stop coordinates or local geometry.'; + function flowChatEntry(upstream, resourceDirectory) { const adaptedMap = resourceDirectory.startsWith('..') ? './maps.md' : './authoring/maps.md'; return upstream @@ -78,9 +113,9 @@ function flowChatEntry(upstream, resourceDirectory) { .replace(/### File[\s\S]*?(?=### Fragment)/, '### Publish in FlowChat\n\n' + '- An explicitly requested webpage (网页), website, or project page is a normal project file. Deliver a working independently openable page and link its entry file.\n' + - '- For a conversational figure, write the literal fragment directly with Write. The first line is `+++ ' + uri + '` (replace <title> with an ASCII lowercase slug). The next line starts the markup. This is your only HTML destination. Use Edit on that URI for corrections.\n' + + '- For an inline HTML figure, write the literal fragment directly with Write. The first line is `+++ ' + uri + '` (replace <title> with an ASCII lowercase slug). The next line starts the markup. This is your only HTML destination. Use Edit on that URI for corrections.\n' + '- FlowChat opens this exact source privately with the conversation renderer during publication. Write/Edit returns the render result. Repair reported issues on the same URI, then use the returned reference. All preview instructions in the guides, including “open it before replying” for maps, are fulfilled by this host check. Do not create scratch HTML, launch a server, open a browser, request screenshots, or produce tool-by-tool progress messages for verification.\n' + - '- For a city or regional marker map, publish the bundled offline atlas inline first. Write a heading and `<script type="application/json" data-visual-atlas="1">{"places":[{"name":"京都","country":"JP"},{"name":"大阪","country":"JP"}]}</script>` using the requested city names. Support for worldwide land, coasts, lakes, rivers, province boundaries, city coordinates, labels and projection is built in. It produces a complete standalone SVG in the saved reply. No downloads, npm, scratch scripts or local previews are necessary. Follow the adapted maps guide for other map types. For a moving simulation put data-visual-motion on the moving element or canvas.\n\n') + '- Choose a day calendar or timeline for a time-based itinerary. For today or tonight, verify the destination time and offer start times still available. An adjustable arrival time can shift its events and return time together. Choose a map for spatial questions. When both matter, combine them. For a city or regional marker map, publish the bundled offline atlas inline first. Write a heading and `<script type="application/json" data-visual-atlas="1">{"places":[{"name":"京都","country":"JP"},{"name":"大阪","country":"JP"}]}</script>` using the requested city names. Support for worldwide land, coasts, lakes, rivers, province boundaries, city coordinates, labels and projection is built in. It produces a complete standalone SVG in the saved reply. No downloads, npm, scratch scripts or local previews are necessary. ' + streetMaps + ' Follow the adapted maps guide for other map types. For a moving simulation put data-visual-motion on the moving element or canvas.\n\n') .replace(/- Check that JavaScript has no undefined identifiers[^\n]+/, '- Check selectors, JavaScript identifiers and the primary interaction. Use the FlowChat publication result for the render check.') .replace( @@ -115,7 +150,7 @@ for (const entry of Object.keys(manifest.files)) { ? flowChatEntry(await readFile(path.join(packagedGuides, name), 'utf8'), '../upstream/skills/visualize') : name === 'maps.md' ? (await readFile(path.join(target, entry), 'utf8')) - .replace('# Maps', '# Maps\n\nFor city, regional and multi-city marker maps use the packaged regional atlas. Write the diagram once to the FlowChat URI with a heading and `<script type="application/json" data-visual-atlas="1">{"places":[{"name":"京都","country":"JP"},{"name":"大阪","country":"JP"}]}</script>`. A place takes its local name and optional ISO-2 country, first-level region, or sourced coordinates [longitude, latitude]. Optional bounds are [west,south,east,north]. Use this same path for any covered region. It already supplies real coastlines, rivers, lakes and boundaries. Use custom geometry only for more local detail or a layer this atlas does not contain.') + .replace('# Maps', '# Maps\n\nFor city, regional and multi-city marker maps use the packaged regional atlas. Write the diagram once to the FlowChat URI with a heading and `<script type="application/json" data-visual-atlas="1">{"places":[{"name":"京都","country":"JP"},{"name":"大阪","country":"JP"}]}</script>`. A place takes its local name and optional ISO-2 country, first-level region, or sourced coordinates [longitude, latitude]. Optional bounds are [west,south,east,north]. Use this same path for any covered region. It already supplies real coastlines, rivers, lakes and boundaries. ' + streetMaps + ' For a custom projected map mark the SVG with data-visual-map and data-map-source (source URL), and its real basemap geometry with data-visual-region.') .replace('Open it before replying and fix', 'The background Write/Edit check opens the final fragment before the reply. Use its diagnostics to fix') : await readFile(path.join(target, entry), 'utf8'); await syncGuide(path.join(owner, 'authoring', name), expected); diff --git a/scripts/test-visualize-host.mjs b/scripts/test-visualize-host.mjs index 04e1c6fc29..4782902622 100644 --- a/scripts/test-visualize-host.mjs +++ b/scripts/test-visualize-host.mjs @@ -124,8 +124,8 @@ async function testHost() { const csv = 'city,count\nShanghai,3'; const request = { sessionId: 'owner', turnId: 'turn', toolCallId: 'plot', uri: root + 'map.html', source: '<!-- vis-bindings\n{"rows":{"type":"file","path":"./cities.csv"}}\n-->\n' + - '<main data-visual-map><svg viewBox="0 0 120 80"></svg></main><script>openai.visualization.onData("rows", async f => {' + - `if ((await f.blob.text()).includes("Shanghai,3")) document.querySelector("svg").innerHTML='<path d="M2 3L90 4L60 60Z"/>'; });</script>`, + '<main data-visual-map data-map-source="https://example.test/regions"><svg viewBox="0 0 120 80"></svg></main><script>openai.visualization.onData("rows", async f => {' + + `if ((await f.blob.text()).includes("Shanghai,3")) document.querySelector("svg").innerHTML='<path data-visual-region d="M2 3L90 4L60 60Z"/>'; });</script>`, }; const file = { type:'tool', toolName:'Write', status:'completed', toolCall: { id:'csv', input:{payload:'+++ '+root+'cities.csv\n'+csv} }, toolResult:{success:true,result:{file_path:root+'cities.csv',content_bytes:csv.length}} }; @@ -171,7 +171,122 @@ async function testHost() { assert.match(background.checks[2], /stationary/); assert.match(background.checks[3], /fixture-missing-input|Script error/); assert.equal(background.extraFrames, 0); + const selection = await page.evaluate(async () => { + const { inspectInBackground } = await import('/src/tools/visualize/backgroundInspection.ts'); + const fragment = '<select><option value="0">19:30</option><option value="30">20:00</option></select><output>19:30</output>' + + '<script>const picker = document.querySelector("select"); picker.onchange = () => document.querySelector("output").textContent = ["19:30","20:00"][+picker.value].trim()</script>'; + return Promise.all([ + inspectInBackground(fragment, {}, { mapExpected: false }), + inspectInBackground(fragment.replace('+picker.value', 'picker.selectedIndex'), {}, { mapExpected: false }), + inspectInBackground(fragment), + ]); + }); + assert.match(selection[0], /Selecting "20:00" failed/); + assert.deepEqual(selection.slice(1), [null, null]); console.log('Passed background checks, actionable errors and frame cleanup.'); + const mapPublication = await page.evaluate(async () => { + const { inspectWrittenFigure } = await import('/src/tools/visualize/backgroundInspection.ts'); + const { materializeRegionalMaps } = await import('/src/tools/visualize/regionalAtlas.ts'); + const route = '<svg aria-label="Walking route map" viewBox="0 0 300 100"><path d="M5 10L220 45"/><text y="80">1 km</text></svg>'; + const write = async (name, source, prompt = '') => { + try { await inspectWrittenFigure({ sessionId: 'session', turnId: 'turn', uri: 'openbitfun://current-session/artifacts/visualizations/v2/' + name + '.html', source, userRequest: prompt }, async () => { throw Error('Unexpected history lookup'); }); } + catch (error) { return error.message; } + return null; + }; + return Promise.all([ + write('outings', route), + write('map-trip', '<svg><path d="M5 10L220 45"/></svg>'), + write('map-trip', '<p>Street data is unavailable.</p><ol data-visual-itinerary><li>First stop</li></ol>'), + write('clock', '<svg aria-label="Pendulum"><path d="M5 10L50 45"/></svg>'), + materializeRegionalMaps('<script type="application/json" data-visual-atlas="1">{"detail":"street","places":[]}</script>') + .then(() => null, error => error.message), + write('outing', '<h2>Hangzhou</h2><p>West Lake</p>', '画一张杭州地图,标出西湖。'), + write('concept', '<h2>Architecture</h2>', 'Show me a map of the codebase.'), + write('hangzhou-route-map', '<ol><li>First stop</li></ol>', '地图给我看看'), + write('map-itinerary', '<h2>杭州一日出行规划</h2><script type="application/json" data-visual-atlas="1">{"places":["杭州"]}</script>'), + write('map-private', '<!-- vis-bindings\n{"rows":{"type":"json","value":[]}}\n-->\n<script type="application/json" data-visual-atlas="1">{"detail":"street","city":"Paris","places":["Private stop"]}</script>'), + ...['地图呢?', 'Where’s the map?'].map(prompt => + write('evening', '<ol data-visual-itinerary><li>First stop</li></ol>', prompt)), + write('schedule', '<viz-calendar></viz-calendar><h2>行程表</h2>', '画一张杭州地图,并给出行程表。'), + write('map-hangzhou', '<h2>杭州</h2>', '画一张杭州地图,并给出行程表。'), + write('map-services', '<h2>Services</h2>', 'Show me a map of our microservices.'), + write('journey', '<h2>User journey</h2>', 'Show me a map of the user journey.'), + write('map-plan', '<ol data-visual-itinerary><li><svg aria-hidden="true" width="16" height="16"><path d="M0 0L9 9"/></svg>First stop</li></ol>'), + write('outing', '<h2>Paris</h2>', 'Show me a map of Paris.'), + write('country', '<h2>France</h2>', 'Show me a map of France.'), + write('pendulum', '<h2>Pendulum</h2>', 'Draw a map of Paris and animate a pendulum.'), + write('clock', '<h2>摆钟</h2>', '画一张杭州地图,再给我做一个摆钟。'), + ]); + }); + assert.match(mapPublication[0], /Missing geographic basemap/); + assert.match(mapPublication[1], /Missing geographic basemap/); + assert.deepEqual(mapPublication.slice(2, 4), [null, null]); + assert.match(mapPublication[4], /no saved basemap/); + assert.match(mapPublication[5], /Missing geographic basemap/); + assert.equal(mapPublication[6], null); + assert.match(mapPublication[7], /Missing geographic basemap/); + assert.match(mapPublication[8], /shows only a city center/); + assert.match(mapPublication[9], /cannot send stop names/); + for (const issue of mapPublication.slice(10, 12)) assert.match(issue, /reader asked for a map/); + assert.equal(mapPublication[12], null); + assert.match(mapPublication[13], /Missing geographic basemap/); + assert.deepEqual(mapPublication.slice(14, 17), [null, null, null]); + assert.match(mapPublication[17], /Missing geographic basemap/); + assert.match(mapPublication[18], /Missing geographic basemap/); + assert.deepEqual(mapPublication.slice(19), [null, null]); + // Exercise a headless producer's record through the ordinary FlowChat reader. + const deferredUrl = new URL(url); + deferredUrl.searchParams.set('deferred-request', 'Show me a map of Paris.'); + await page.evaluate(() => sessionStorage.setItem('visualize-test-fragment', '<h2>Paris</h2>')); + await page.goto(deferredUrl); + await page.waitForFunction(() => document.body.textContent.includes('Missing geographic basemap')); + assert.equal(await page.$('[data-figure-check=passed]'), null); + deferredUrl.searchParams.set('deferred-request', '画一张杭州地图,并给出行程表。'); + await page.goto(deferredUrl); + const calendar = await open('<h2>行程表</h2><select><option>A</option><option>B</option></select>'); + assert.equal(await calendar.$eval('select', el => el.value), 'A'); + // Replace only the public tile service, leaving the projection worker and + // publication reader intact. One vector tile containing a diagonal street. + const hook = await page.evaluateOnNewDocument(() => { + const NativeWorker = window.Worker; + window.mapProjectionCount = 0; + window.Worker = class extends NativeWorker { + constructor(url, options) { + if (!String(url).includes('regionalProjection')) { super(url, options); return; } + window.mapProjectionCount++; + const bootstrap = ` + const fetchAsset = self.fetch.bind(self); + self.fetch = (url, opts) => new URL(url).hostname === 'tiles.versatiles.org' + ? Promise.resolve(new Response(Uint8Array.from(atob('Ghl4AgoHc3RyZWV0cxIMGAIiCAkAAAqAQIBA'), c => c.charCodeAt(0)))) + : fetchAsset(url, opts); + const pending = []; + self.onmessage = event => pending.push(event); + await import(${JSON.stringify(String(url))}); + for (const event of pending) self.onmessage(event); + `; + super(URL.createObjectURL(new Blob([bootstrap], { type: 'text/javascript' })), options); + } + }; + }); + const street = '<script type="application/json" data-visual-atlas="1">{"detail":"street","places":[{"name":"A","coordinates":[120.15,30.26]},{"name":"B","coordinates":[120.16,30.27]}]}</script>'; + await page.evaluate(source => sessionStorage.setItem('visualize-test-fragment', source), street); + await page.reload(); + const consentButton = '[data-codex-figure] [role=status] button'; + await page.waitForSelector(consentButton); + assert.equal(await page.evaluate(() => window.mapProjectionCount), 0); + await page.click(consentButton); + await page.waitForSelector('[data-figure-check=passed]'); + assert.equal(await page.evaluate(() => window.mapProjectionCount), 1); + const restoredMap = await open(street); + assert.ok(await restoredMap.$('path[data-street-layer="streets"]')); + assert.equal(await page.evaluate(() => window.mapProjectionCount), 0); + await page.evaluate(source => sessionStorage.setItem('visualize-test-fragment', source.replace('120.15,', '120.14,')), street); + await page.reload(); + await page.waitForSelector(consentButton); + assert.equal(await page.evaluate(() => window.mapProjectionCount), 0); + await page.removeScriptToEvaluateOnNewDocument(hook.identifier); + await page.goto(url); + console.log('Passed deferred map intent, reader consent, offline street replay and revision isolation.'); // A collapsed lake ring once covered the entire viewport, even though the // figure had both valid SVG paths and the right city labels. const deltaMap = await open('<script type="application/json" data-visual-atlas="1">{"places":["上海","杭州","苏州"],"language":"zh-CN"}</script>'); @@ -639,7 +754,7 @@ async function testHost() { const chart = await open(` <button id="sample">Change dataset</button><main></main> <script type="module"> - import { mountChart } from "https://persistent.oaistatic.com/plugins/visualize/charts/1.6.5/element.js"; + import { mountChart } from "https://persistent.oaistatic.com/plugins/visualize/charts/1.8.1/element.js"; const rows = [["A","C",1,5],["B","C",2,3],["E","D",3,2]]; const alternateRows = [["Z","Q",13,39],["Y","Q",19,22],["W","P",25,8],["V","P",61,1]]; const charts = ${JSON.stringify(specs)}.map(spec => { diff --git a/scripts/visualize-dev-kit.cjs b/scripts/visualize-dev-kit.cjs new file mode 100644 index 0000000000..2d39c81c72 --- /dev/null +++ b/scripts/visualize-dev-kit.cjs @@ -0,0 +1,13 @@ +const { existsSync } = require('node:fs'); +const path = require('node:path'); + +// Both halves of a desktop dev session must advertise the same optional kit. +function visualizeDevGuide(repoRoot) { + const kit = path.join(repoRoot, 'src/crates/assembly/core/builtin_skills/visualize'); + const guide = path.join(kit, 'authoring/SKILL.md'); + const assets = path.join(kit, 'upstream/skills/visualize/assets'); + return [guide, path.join(assets, 'visualize.html'), path.join(assets, 'standalone-host-bridge.js')] + .every(file => existsSync(file)) ? guide : undefined; +} + +module.exports = { visualizeDevGuide }; diff --git a/scripts/visualize-runtime.json b/scripts/visualize-runtime.json index c5ada4e4d3..c4945b1d7d 100644 --- a/scripts/visualize-runtime.json +++ b/scripts/visualize-runtime.json @@ -1,4 +1,16 @@ { + "chart-v1.8.1.js": { + "url": "https://persistent.oaistatic.com/plugins/visualize/charts/1.8.1/element.js", + "sha256": "d187ad9ddf2eb7e459ae9b25281378382241716a352f9ce74a20ebc022d56062" + }, + "chart-element-v1.8.1.js": { + "url": "https://persistent.oaistatic.com/plugins/visualize/charts/4f0917bb6b04d2a66330a984e928d18fa5e5fc65d8312efc14b5f50d99cce179/element.js", + "sha256": "5c2f1b79b4302ca7b0fd9675a0ab9db5dcc5c8297b0a81579aebb83c6dc2a3c4" + }, + "chart-runtime-v1.8.1.js": { + "url": "https://persistent.oaistatic.com/plugins/visualize/charts/4f0917bb6b04d2a66330a984e928d18fa5e5fc65d8312efc14b5f50d99cce179/runtime.js", + "sha256": "41e2dcff10997e8f9fbde499ba6dbef634765a690f7723a5b3f8251228efafd5" + }, "chart.js": { "url": "https://persistent.oaistatic.com/plugins/visualize/charts/1.6.5/element.js", "sha256": "362ce1cc20f78f2658b405dbd95b2d99d0fd1995970c8b7a16833206357e8d99" diff --git a/scripts/visualize-upstream.json b/scripts/visualize-upstream.json index c8d7a7740d..7f38b722a2 100644 --- a/scripts/visualize-upstream.json +++ b/scripts/visualize-upstream.json @@ -1,14 +1,14 @@ { - "version": "1.0.103", + "version": "1.0.109", "files": { ".app.json": "2698d1bae590beac93c6a9cbe98f01f5ceb4e3abb7de37540a0d3bbeea3c58d9", - ".codex-plugin/plugin.json": "010810cff87422d78bfe7e311c58f3874db561dab843363b3a8794e693b62f3d", + ".codex-plugin/plugin.json": "8ce750109613e3dd8a8e105d308d1b33fa50f31df0b3276ef6b29c91fd0e2023", "assets/dashboard-annotations.js": "537178dcfc57b7e171786aa7ea025d86be645de7f317b5fe6659d30827c21c91", "assets/dashboard-layout.js": "bffee39b4875ba6b97a3a057c87368901e7dc1d8ad0e3d8ca1c40e69a638842b", "assets/dashboard-source-actions.js": "1ed025a13e8c06bc84603c0c388ca6e01cac5110e05bd9dfd5c6b3d0a52a2a82", "assets/tweak.js": "570efb7eb49748a4515f3fc89abdc4b6f920fcdc2f1c379c21c6c60fb3327eeb", - "assets/visualize-dark.svg": "185208ef21fb7cb5c9a1ce6c09a5acae2d0965996e570b56652a53a85c91040c", - "assets/visualize.svg": "632df773209864f73213a88ce22d85df05da38078e47556759e9370761b2933a", + "assets/visualize-dark.svg": "c81646731389d8711191a3f21ef60be2dfaf3b18f2c8b9b6aa55ccb54058bc39", + "assets/visualize.svg": "16e4deca048d10205858f50b4d33a346ed4f888d877215498e368a3beb4e8fe0", "skills/visualize/SKILL.md": "4a24dd004e9a255b96ef03bfc854e0f0f74a0d8d9e7166b0f880658b47d8e7bb", "skills/visualize/agents/openai.yaml": "47abbdea0198c9efc6c11fd99ff82f0762e9da4f61538dba77a851de6f3691a8", "skills/visualize/assets/calendar.js": "19f40e0a15ae7a9b9a4e17bc53c5bb47c5b21e6d32317d80cf030f9e012c221b", @@ -30,7 +30,7 @@ "skills/visualize/charts/funnel.md": "399e03acf66d033401ad55907b43ad2c079aac5ee26f847aff0ff4d07d735d5f", "skills/visualize/charts/heatmap.md": "61bea8ad6ce2eb594ecd2d6ebf0cab317dcbcb2b93a358c54474c20ee43bf7f7", "skills/visualize/charts/histogram.md": "8a915627e5b9bb39b0c0beb0126d1b3d4128bdc259405e6e394dcd4866ee1d06", - "skills/visualize/charts/instructions.md": "468460f503b01f15db75bfd58e79b1e62f05708f208ae4345aa8dc6172613545", + "skills/visualize/charts/instructions.md": "2f21b6cbe13e4f2ff59ad16983a873d79fb21275bc2b73ba286d0a199dc775a6", "skills/visualize/charts/line.md": "e694ae4807e3d05194cd3f236c7eca9003273d51dd606c3819b7f56316f9112c", "skills/visualize/charts/pie.md": "b1bdf7549e657884111c0c7874d2771a75b6c11cbc714cf4e42542398237d4c0", "skills/visualize/charts/public-api.d.ts": "b99333086d0a09431a6a5e9cd995e1491867c4d20fd9ad8d35200dd8e46ac72b", @@ -42,10 +42,11 @@ "skills/visualize/maps.md": "721f2723d09d6dc31b2c9283e8faf9aacc9d76c6db0bce7a19a996b8b9168dbc", "skills/visualize/mockups.md": "07988a559e7915830a741fe6d95474a8ea52b0e6aaf0ddbe0e8e132e436aeea5", "skills/visualize/scripts/chart_check.NOTICES.txt": "49af0edc450916ef3e61ebbb874e81474bcde9c6c255eefe1ed658fe4b77ee9f", - "skills/visualize/scripts/chart_check.mjs": "6d94c0a2cfea2c3e2118d0e9555ddbe2ecadbe36324624f7dfe78154e0bba8fb", + "skills/visualize/scripts/chart_check.mjs": "d20f20c137f14eb7bfc49ae7f28e4db5a489b330aca949586fa72d0931a8cd38", "skills/visualize/scripts/chart_structure.py": "22246223dce06856fe1fcac2e0db2f77febe992a97b34a8ab384ade8ddc92e96", "skills/visualize/scripts/connector_mocks.py": "e9b56e9606682ff40a5ae3ab069cf66b0d12ab75b8e54cc389dc78329ffb2c6f", - "skills/visualize/scripts/render.py": "ba4e8ac84390a18b009ca7517152e65707c72bd3038f11b13774bd3139a83045", + "skills/visualize/scripts/dashboard_check.py": "81c8f491f18ab0d0381d2792fc45e843c64a54ec1d02e39847fa18e2ff679eba", + "skills/visualize/scripts/render.py": "9a735762742beccf9f85c1f48664c2cfbe9305d411b9a87d33bc124028c8341a", "skills/visualize/technical-illustrations.md": "4a55ba7e2e4ed9dd1138f62dfce119d8e9d01a4bff79c0d6cab99d51d475491c", "skills/visualize/tweak.md": "7d914a63d761f0f7663cc74d2b91e53fcdce57c5f5a840b87e2f162917f22237", "skills/visualize/widgets/calendar.md": "f4404c9f6fdc27971fe162d6bd97753a1a5472ad7b1da5821a2d7e210e5a0866", diff --git a/src/apps/cli/src/ui/tool_cards.rs b/src/apps/cli/src/ui/tool_cards.rs index af9e27e395..8e5b1b49d0 100644 --- a/src/apps/cli/src/ui/tool_cards.rs +++ b/src/apps/cli/src/ui/tool_cards.rs @@ -2143,7 +2143,12 @@ fn param_str(params: &serde_json::Value, keys: &[&str]) -> String { /// Extract the Skill tool's command, retaining legacy parameter aliases. fn skill_display_name(params: &serde_json::Value) -> String { - param_str(params, &["command", "name", "skill_name"]) + ["command", "skill", "name", "skill_name"] + .iter() + .filter_map(|key| params.get(*key)?.as_str().map(str::trim)) + .find(|name| !name.is_empty()) + .unwrap_or("unknown") + .to_string() } /// Extract an optional string parameter @@ -2323,6 +2328,20 @@ mod tests { fn skill_name_is_read_from_command_parameter() { let params = serde_json::json!({"command": "arkts-error-fixes"}); + for command in [ + serde_json::Value::Null, + serde_json::json!(" "), + serde_json::json!(3), + ] { + assert_eq!( + skill_display_name(&serde_json::json!({"command":command,"skill":" visualize "})), + "visualize" + ); + } assert_eq!(skill_display_name(¶ms), "arkts-error-fixes"); + assert_eq!( + skill_display_name(&serde_json::json!({"skill": "visualize"})), + "visualize" + ); } } diff --git a/src/apps/desktop/src/figure_verifier.rs b/src/apps/desktop/src/figure_verifier.rs index 57f3b84c79..d194d6a65d 100644 --- a/src/apps/desktop/src/figure_verifier.rs +++ b/src/apps/desktop/src/figure_verifier.rs @@ -20,6 +20,10 @@ pub(crate) fn install(app: AppHandle) { .and_then(|result| result); let _ = send.send(result); }); + let (progress, stages) = tokio::sync::watch::channel(()); + let stage_handle = app.listen(format!("figure-check-stage-{id}"), move |_| { + let _ = progress.send(()); + }); // Also unsubscribe if the tool future is cancelled or the shell reloads. struct Subscription(AppHandle, tauri::EventId); impl Drop for Subscription { @@ -28,18 +32,80 @@ pub(crate) fn install(app: AppHandle) { } } let _subscription = Subscription(app.clone(), handle); + let _stages = Subscription(app.clone(), stage_handle); + // Each projection and the final frame inspection have their own + // deadline. Watch for a stalled stage, not the sum of healthy stages. + let timeout_secs = if request.source.contains("data-visual-atlas") { + 75 + } else { + 30 + }; app.emit_to( EventTarget::webview("main"), "figure-check", serde_json::json!({"id": id, "request": request}), ) .map_err(|e| e.to_string())?; - tokio::time::timeout(Duration::from_secs(30), receive) - .await - .map_err(|_| { - "FlowChat renderer did not respond on the producing device".to_owned() - })? - .map_err(|_| "FlowChat renderer disconnected".to_owned())? + wait_for_renderer(receive, stages, Duration::from_secs(timeout_secs)).await }) })); } + +async fn wait_for_renderer( + mut receive: tokio::sync::oneshot::Receiver<Result<FigureCheckResult, String>>, + mut stages: tokio::sync::watch::Receiver<()>, + idle_timeout: Duration, +) -> Result<FigureCheckResult, String> { + loop { + tokio::select! { + result = &mut receive => { + return result.map_err(|_| "FlowChat renderer disconnected".to_owned())?; + } + change = stages.changed() => { + change.map_err(|_| "FlowChat renderer disconnected".to_owned())?; + } + _ = tokio::time::sleep(idle_timeout) => { + return Err("FlowChat renderer did not respond on the producing device".to_owned()); + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test(start_paused = true)] + async fn forwards_the_final_diagnostic_after_two_slow_map_stages() { + let (result, receive) = tokio::sync::oneshot::channel(); + let (progress, stages) = tokio::sync::watch::channel(()); + let renderer = tokio::spawn(async move { + for _ in 0..2 { + tokio::time::sleep(Duration::from_secs(59)).await; + progress.send(()).unwrap(); + } + tokio::time::sleep(Duration::from_secs(12)).await; + let _ = result.send(Err("Missing stop coordinates".into())); + progress + }); + assert_eq!( + wait_for_renderer(receive, stages, Duration::from_secs(75)) + .await + .err() + .as_deref(), + Some("Missing stop coordinates") + ); + renderer.await.unwrap(); + } + + #[tokio::test(start_paused = true)] + async fn still_times_out_when_the_renderer_stalls() { + let (_result, receive) = tokio::sync::oneshot::channel(); + let (_progress, stages) = tokio::sync::watch::channel(()); + assert!(wait_for_renderer(receive, stages, Duration::from_secs(75)) + .await + .err() + .unwrap() + .contains("did not respond")); + } +} diff --git a/src/crates/assembly/core/build.rs b/src/crates/assembly/core/build.rs index 4087b3a7f3..611629f718 100644 --- a/src/crates/assembly/core/build.rs +++ b/src/crates/assembly/core/build.rs @@ -1,9 +1,8 @@ fn main() { if std::env::var_os("CARGO_FEATURE_AGENT_RUNTIME").is_some() { emit_rerun_if_changed(std::path::Path::new("builtin_skills")); - if let Err(e) = build_embedded_builtin_skills_metadata() { - eprintln!("Warning: Failed to embed built-in skills metadata: {}", e); - } + build_embedded_builtin_skills_metadata() + .expect("could not prepare embedded built-in skills"); } } @@ -39,6 +38,25 @@ fn build_embedded_builtin_skills_metadata() -> Result<(), Box<dyn std::error::Er } files.sort(); + let out_dir = PathBuf::from(std::env::var("OUT_DIR")?); + // Profile does not identify the reader: tauri build --debug also ships a + // production frontend. Never embed workstation-only reference assets. + files.retain(|path| { + !path.starts_with("visualize/upstream") && !path.starts_with("visualize/authoring") + }); + let embed_root = out_dir.join("release_builtin_skills"); + if embed_root.exists() { + fs::remove_dir_all(&embed_root)?; + } + for relative in &files { + let dest = embed_root.join(relative); + fs::create_dir_all(dest.parent().expect("skill file has a parent"))?; + fs::copy(builtin_root.join(relative), dest)?; + } + println!( + "cargo:rustc-env=OPENBITFUN_EMBEDDED_SKILLS_DIR={}", + embed_root.display() + ); let mut hasher = Sha256::new(); for relative_path in files { let normalized = relative_path.to_string_lossy().replace('\\', "/"); @@ -49,8 +67,7 @@ fn build_embedded_builtin_skills_metadata() -> Result<(), Box<dyn std::error::Er } let bundle_hash = format!("{:x}", hasher.finalize()); - let out_dir = std::env::var("OUT_DIR")?; - let dest_path = Path::new(&out_dir).join("embedded_builtin_skills.rs"); + let dest_path = out_dir.join("embedded_builtin_skills.rs"); let mut file = fs::File::create(&dest_path)?; writeln!( diff --git a/src/crates/assembly/core/builtin_skills/visualize/SKILL.md b/src/crates/assembly/core/builtin_skills/visualize/SKILL.md index 550fa3044e..f0baa207c3 100644 --- a/src/crates/assembly/core/builtin_skills/visualize/SKILL.md +++ b/src/crates/assembly/core/builtin_skills/visualize/SKILL.md @@ -5,20 +5,35 @@ description: "Show a visual or an interactive tool in the conversation. Use for # Visual answers in FlowChat -For a visual explanation, publish the figure in this turn. For an independently openable webpage, implement it in the workspace and link the entry file. A request for a working object, such as a pendulum, is enough to publish an inline simulation. Use a simple Markdown table or Mermaid diagram when the requested information fits those formats. +Choose the format from what the reader needs to see or do. -Use Write with a single patch section. The first line names the session artifact. Put the literal HTML fragment below it. Replace the slug with a short ASCII name. +- Use a Markdown table for a comparison the reader can understand by scanning rows and columns. +- When named nodes and labeled links tell the whole story, put a fenced `mermaid` diagram directly in the reply. This works for dependencies, message sequences, processes and state transitions. Explain what the arrows mean. Split distinct views when combining them would make the diagram hard to read. The Markdown reply is the complete deliverable for these diagrams. +- Use an inline HTML figure when spatial placement matters, something moves, or the reader needs to change an input and see the result. For example, a diagram that lets the reader switch between execution paths can use a selector to update the highlighted path. A request for a working object such as a pendulum calls for an inline simulation. +- For an explicitly requested standalone webpage, implement it in the workspace and link the entry file. - +++ openbitfun://current-session/artifacts/visualizations/v2/city-map.html - <h2>Great Lakes cities</h2> - <script type="application/json" data-visual-atlas="1">{"places":[{"name":"Chicago","country":"US"},{"name":"Toronto","country":"CA"}]}</script> +For a layered overview, draw the endpoints of the actual edges. If the dependencies are between layers, use one node per layer, for example `Clients["Clients<br/>Desktop · Web"] --> API["Application API"]`. List member packages in the accompanying text. Expand a layer into a subgraph when showing how its members connect. Use the host Mermaid theme so labels and arrows follow the reader's skin. -For city and regional maps use that declaration directly with the requested places. The built-in atlas provides real coasts, lakes, rivers, administrative boundaries, projections and place labels offline. Each place accepts a name, optional ISO country code and region, or sourced coordinates in [longitude, latitude] order. Optional bounds use [west, south, east, north]. Set language for translated labels if needed. Report ambiguous names or ask for detail when the supplied names do not uniquely identify places. +## Inline HTML figures -For architectural diagrams and mockups, use readable HTML and SVG. Label responsibilities, relationships and direction. Provide a working selection or input when it explains something. Use native buttons and inputs, explicit axis labels and units, responsive dimensions and theme variables such as --foreground, --background, --muted-foreground and --viz-series-1. Tabs with role="tab" select panels by aria-controls. For charts the frame supplies Apache ECharts when the fragment uses echarts.init(). Give the chart a nonzero height and resize it with its container. For a simulation implement the moving object and mark it with data-visual-motion so the host can check it. Respect reduced motion and keep essential labels visible. +Choose the view around the reader's decision. For today or tonight, verify the current local time at the destination before choosing events. Offer start times the reader can still use today. For a dated day trip or a question about today's plans, show a day schedule with times and durations when that helps. If the arrival or departure time is flexible, let the reader change it and update the affected stops and return time together. Use a map for questions about locations, relative positions or a route on a map. Combine a schedule and map when the reader needs both. On a follow-up, preserve the places and choices already made. + +Use Write with a single patch section. The first line names the session artifact, for example `+++ openbitfun://current-session/artifacts/visualizations/v2/evening-plan.html`. Put the literal HTML fragment below it. For a day schedule use `<viz-calendar date="YYYY-MM-DD" start="17:00" end="20:00">` and assign its `events` array with local `start`, `end` and `title` values. Use a labeled native select for a time choice. Calculate times from the selected start and durations, or key each schedule by the actual option value. A simple HTML timeline also works. + +For geographic maps use a map-<slug>.html destination so the background check expects a basemap. For cities across a region use `<script type="application/json" data-visual-atlas="1">{"places":["Chicago","Toronto"]}</script>` with the requested places. The built-in atlas provides real coasts, lakes, rivers, administrative boundaries, projections and place labels offline. Each place accepts a name, optional ISO country code and region, or sourced coordinates in [longitude, latitude] order. Optional bounds use [west, south, east, north]. Set language for translated labels if needed. Report ambiguous names or ask for detail when the supplied names do not uniquely identify places. + +When presenting a geographic map of sights or transit stops within one city, put the requested stops on a street basemap. Use one atlas declaration with `detail: "street"`, `city`, optional ISO country code and the public names of the stops in visit order. For example: + + <script type="application/json" data-visual-atlas="1">{"detail":"street","city":"Paris","country":"FR","places":[{"name":"Tour Eiffel","osmTag":"man_made:tower"},{"name":"Arc de Triomphe","osmTag":"building:triumphal_arch"}]}</script> + +FlowChat looks up named public stops in the city and saves their resolved coordinates and the verified street basemap for offline replay. Submit each public name directly. Keep descriptions such as “evening walk” or “historic section” in the display label; use the short local place name as the query. Only supply WGS84 coordinates [longitude, latitude] when they come from an identified source. For disambiguation a stop accepts a public search name in `query`, an OSM classification in `osmTag`, or an `osmId` from the reported candidates. For a rail station use the local station name with `railway:station`. For a specifically requested pedestrian street use its street name with `highway:pedestrian`. Select an ID only if both its classification and its location match the itinerary. Use a verified entrance or street segment of the requested place and identify it in the label. If suggesting a nearby substitute, label it explicitly and explain which stop could not be located. On a failed lookup, use the reported candidates to repair the same map. A mismatched tag can be dropped. If the intended place is still unclear, ask for clarification. Keep each requested stop on the map, including when revising an earlier itinerary. A stop's marker has `data-map-stop` equal to its zero-based index for itinerary controls. Markers indicate stops; draw a navigation path only with sourced route geometry. Never invent a shoreline, a road, or a distance scale. For an explicitly offline or private location, use locally supplied, sourced geometry. Figures with bound data must supply coordinates or local geometry for their stops. Mark a custom projected SVG with `data-visual-map`, `data-map-source="https://…"`, and `data-visual-region` on its projected basemap geometry. If street data is unavailable while planning an itinerary, the schedule can still be shown as an ordered list (`<ol data-visual-itinerary>`). + +When the reader asks for a missing map, use the stops from the earlier itinerary and submit a fresh atlas declaration with Write in this turn, even if an earlier attempt failed or showed only a list. Read the new receipt. Use the returned WGS84 coordinates of confirmed places and the intended candidates to repair the map. If name lookup fails for a public stop, look up its WGS84 coordinates with WebSearch or WebFetch from a reliable public map or place record. Resubmit the same figure with those coordinates and credit the source. If coordinates or the basemap remain unavailable, report the current failure in ordinary text. A list or navigation links alone do not fulfill a map request. + +For an interactive diagram or mockup, use readable HTML and SVG. Keep labels close to the elements they describe and make each control visibly update the relevant view. Use native buttons and inputs, explicit axis labels and units, responsive dimensions and theme variables such as --foreground, --background, --muted-foreground and --viz-series-1. Tabs with role="tab" select panels by aria-controls. For charts the frame supplies Apache ECharts when the fragment uses echarts.init(). Give the chart a nonzero height and resize it with its container. For a simulation implement the moving object and mark it with data-visual-motion so the host can check it. Respect reduced motion and keep essential labels visible. Write and Edit check the fragment in the background. For a map the host resolves the atlas declaration and checks the resulting geometry. Read the tool result and repair reported errors with Edit on the same URI. Skip temporary scripts, local servers and browser previews. Finish with a short reader-facing reply. FlowChat attaches the checked figure automatically. To position multiple figures in a particular order, copy each exact reference onto its own line. - visualize{"path":"openbitfun://current-session/artifacts/visualizations/v2/city-map.html"} + visualize{"path":"openbitfun://current-session/artifacts/visualizations/v2/map-cities.html"} -Add "mode":"wide" to the reference for a wide figure. Use window.openai.setWidgetState({modelContent, privateContent}) for selections that survive replay. Keep reader-only selections in privateContent. To expose a design adjustment in the figure menu, create new Tweak({container, onChange}) and register a slider, toggle, color or selection with addSlider, addToggle, addColorPicker or addSelect. Live binding updates are available through window.openai.visualization.onData(name, callback). For a locally imported reference kit, authoring/SKILL.md contains its extended component API. +Add "mode":"wide" to the reference for a wide figure. Use window.openai.setWidgetState({modelContent, privateContent}) for selections that survive replay. Keep reader-only selections in privateContent. To expose a design adjustment in the figure menu, create new Tweak({container, onChange}) and register a slider, toggle, color or selection with addSlider, addToggle, addColorPicker or addSelect. Live binding updates are available through window.openai.visualization.onData(name, callback). diff --git a/src/crates/assembly/core/src/agentic/inline_figure.rs b/src/crates/assembly/core/src/agentic/inline_figure.rs index 00f08c4607..1441ff1c2b 100644 --- a/src/crates/assembly/core/src/agentic/inline_figure.rs +++ b/src/crates/assembly/core/src/agentic/inline_figure.rs @@ -57,6 +57,7 @@ pub(super) fn reference_hint(uri: &str) -> String { #[serde(rename_all = "camelCase")] pub struct FigureCheckRequest { pub source: String, + pub user_request: Option<String>, pub uri: String, pub session_id: Option<String>, pub turn_id: Option<String>, @@ -116,6 +117,11 @@ pub(super) async fn verify_written_figure( }; let request = FigureCheckRequest { source: source.to_owned(), + user_request: context + .custom_data + .get(super::tools::tool_context_runtime::TURN_USER_REQUEST) + .and_then(serde_json::Value::as_str) + .map(str::to_owned), uri: uri.into(), session_id: context.session_id.clone(), turn_id: context.dialog_turn_id.clone(), diff --git a/src/crates/assembly/core/src/agentic/persistence/manager/inline_figures.rs b/src/crates/assembly/core/src/agentic/persistence/manager/inline_figures.rs index 9e94178726..8743c0818e 100644 --- a/src/crates/assembly/core/src/agentic/persistence/manager/inline_figures.rs +++ b/src/crates/assembly/core/src/agentic/persistence/manager/inline_figures.rs @@ -268,7 +268,8 @@ impl PersistenceManager { let bindings = receipt.result["visualization_bindings"] .as_object() .cloned(); - checked.insert(uri.clone(), (exact.filter(|_| passed), bindings)); + let deferred = receipt.result["visualization_check_deferred"] == true; + checked.insert(uri.clone(), (exact.filter(|_| passed), bindings, deferred)); } registered.insert(uri); } @@ -308,7 +309,7 @@ impl PersistenceManager { let since = (!attested).then_some(turn.start_time); let filename = uri.rsplit('/').next().unwrap().to_owned(); let session = root.clone(); - let (source, bindings) = if let Some(exact) = checked.remove(uri) { + let (source, bindings, inspection_deferred) = if let Some(exact) = checked.remove(uri) { exact } else { let source = tokio::task::spawn_blocking(move || { @@ -317,11 +318,12 @@ impl PersistenceManager { .await .ok() .flatten(); - (source, None) + (source, None, false) }; turn.published_figures.insert( uri.clone(), source.map(|source| PublishedFigure { + inspection_deferred, bindings, content_bytes: source.len(), source, diff --git a/src/crates/assembly/core/src/agentic/tools/implementations/skill_tool.rs b/src/crates/assembly/core/src/agentic/tools/implementations/skill_tool.rs index 56ce7d64f6..96b7ca8ed5 100644 --- a/src/crates/assembly/core/src/agentic/tools/implementations/skill_tool.rs +++ b/src/crates/assembly/core/src/agentic/tools/implementations/skill_tool.rs @@ -23,6 +23,18 @@ impl SkillTool { Self } + // Keep command as the advertised field. Some providers emit skill instead. + // Permission checks, status text and execution must use the same identity. + pub(crate) fn requested_skill(input: &Value) -> Option<&str> { + ["command", "skill"].into_iter().find_map(|key| { + input + .get(key)? + .as_str() + .map(str::trim) + .filter(|name| !name.is_empty()) + }) + } + fn render_description(&self) -> String { r#"Execute a skill within the main conversation @@ -185,11 +197,7 @@ impl Tool for SkillTool { input: &Value, _context: &ToolUseContext, ) -> OpenBitFunResult<Vec<PermissionIntent>> { - let skill_name = input - .get("command") - .and_then(Value::as_str) - .map(str::trim) - .filter(|skill_name| !skill_name.is_empty()) + let skill_name = Self::requested_skill(input) .ok_or_else(|| OpenBitFunError::validation("command is required".to_string()))?; Ok(vec![PermissionIntent::new( "skill", @@ -202,11 +210,7 @@ impl Tool for SkillTool { input: &Value, _context: Option<&ToolUseContext>, ) -> ValidationResult { - if input - .get("command") - .and_then(|v| v.as_str()) - .is_none_or(|s| s.is_empty()) - { + if Self::requested_skill(input).is_none() { return ValidationResult { result: false, message: Some("command is required and cannot be empty".to_string()), @@ -235,7 +239,7 @@ impl Tool for SkillTool { } fn render_tool_use_message(&self, input: &Value, _options: &ToolRenderOptions) -> String { - if let Some(command) = input.get("command").and_then(|v| v.as_str()) { + if let Some(command) = Self::requested_skill(input) { format!("The \"{}\" skill is loaded.", command) } else { "Loading skill...".to_string() @@ -247,9 +251,7 @@ impl Tool for SkillTool { input: &Value, context: &ToolUseContext, ) -> OpenBitFunResult<Vec<ToolResult>> { - let skill_name = input - .get("command") - .and_then(|v| v.as_str()) + let skill_name = Self::requested_skill(input) .ok_or_else(|| OpenBitFunError::tool("command is required".to_string()))?; debug!("Skill tool executing skill: {}", skill_name); @@ -332,12 +334,21 @@ impl Tool for SkillTool { && skill_data.argument_names.is_empty(); let user_request = context .custom_data - .get(crate::agentic::tools::tool_context_runtime::SKILL_USER_REQUEST) + .get(crate::agentic::tools::tool_context_runtime::TURN_USER_REQUEST) .and_then(Value::as_str) .filter(|s| !s.trim().is_empty()); let file_delivery = context.custom_data .get(openbitfun_agent_runtime::remote_file_delivery::TOOL_CONTEXT_REMOTE_FILE_DELIVERY_KEY) .and_then(Value::as_bool) == Some(true); + #[cfg(debug_assertions)] + if !file_delivery + && !context.is_remote() + && skill_data.key == "user::openbitfun-system::visualize" + { + if let Ok(guide) = std::env::var("OPENBITFUN_VISUALIZE_DEV_GUIDE") { + skill_data.content.push_str(&format!("\n\nThe local development renderer supports the extended component API documented in {guide}.")); + } + } if file_delivery && skill_data.key == "user::openbitfun-system::visualize" { skill_data.content = openbitfun_agent_runtime::remote_file_delivery::remote_file_delivery_reminder() @@ -627,6 +638,45 @@ Use the remote project skill. assert_eq!(schema["required"], json!(["command"])); } + #[tokio::test] + async fn skill_alias_loads_the_same_skill_it_validates_and_authorizes() { + let temp = tempfile::tempdir().unwrap(); + let dir = temp.path().join(".openbitfun/skills/alias-test"); + fs::create_dir_all(&dir).unwrap(); + fs::write( + dir.join("SKILL.md"), + "---\nname: alias-test\ndescription: Alias fixture.\n---\nAlias body.", + ) + .unwrap(); + let context = local_context(temp.path().to_path_buf()); + let tool = SkillTool::new(); + for input in [ + json!({"skill":"alias-test"}), + json!({"command":"alias-test","skill":"other"}), + json!({"command":null,"skill":"alias-test"}), + json!({"command":" ","skill":"alias-test"}), + json!({"command":3,"skill":"alias-test"}), + ] { + assert!(tool.validate_input(&input, Some(&context)).await.result); + assert_eq!( + tool.permission_intents(&input, &context).unwrap()[0].resources, + ["alias-test"] + ); + let result = tool.call_impl(&input, &context).await.unwrap(); + assert!( + matches!(&result[0], ToolResult::Result { data, .. } if data["skill_name"] == "alias-test") + ); + } + for input in [ + json!({"skill":" "}), + json!({"skill":42}), + json!({"command":"","skill":" "}), + ] { + assert!(!tool.validate_input(&input, Some(&context)).await.result); + assert!(tool.permission_intents(&input, &context).is_err()); + } + } + #[tokio::test] async fn stable_key_loads_source_without_changing_original_name_resolution() { let temp = tempfile::tempdir().unwrap(); diff --git a/src/crates/assembly/core/src/agentic/tools/implementations/skills/builtin.rs b/src/crates/assembly/core/src/agentic/tools/implementations/skills/builtin.rs index 7fab008eb6..7916c22eb2 100644 --- a/src/crates/assembly/core/src/agentic/tools/implementations/skills/builtin.rs +++ b/src/crates/assembly/core/src/agentic/tools/implementations/skills/builtin.rs @@ -17,7 +17,7 @@ use std::time::{SystemTime, UNIX_EPOCH}; use tokio::fs; use tokio::task; -static BUILTIN_SKILLS_DIR: Dir = include_dir!("$CARGO_MANIFEST_DIR/builtin_skills"); +static BUILTIN_SKILLS_DIR: Dir = include_dir!("$OPENBITFUN_EMBEDDED_SKILLS_DIR"); static BUILTIN_SKILL_DIR_NAMES: OnceLock<HashSet<String>> = OnceLock::new(); include!(concat!(env!("OUT_DIR"), "/embedded_builtin_skills.rs")); diff --git a/src/crates/assembly/core/src/agentic/tools/pipeline/tool_pipeline.rs b/src/crates/assembly/core/src/agentic/tools/pipeline/tool_pipeline.rs index 62891a3d9d..7bc0c28d84 100644 --- a/src/crates/assembly/core/src/agentic/tools/pipeline/tool_pipeline.rs +++ b/src/crates/assembly/core/src/agentic/tools/pipeline/tool_pipeline.rs @@ -282,7 +282,7 @@ fn resolve_pipeline_invocation( tool_call: &ToolCall, context: &ToolExecutionContext, ) -> (ResolvedToolInvocation, Option<String>) { - let invocation = match ResolvedToolInvocation::from_wire_call( + let mut invocation = match ResolvedToolInvocation::from_wire_call( tool_call.tool_name.clone(), tool_call.arguments.clone(), ) { @@ -313,6 +313,19 @@ fn resolve_pipeline_invocation( ); } + // Hooks and permission rules operate on canonical fields, while history + // retains the model's wire input for replay and diagnostics. + if invocation.effective_tool_name == "Skill" { + use crate::agentic::tools::implementations::SkillTool; + let name = SkillTool::requested_skill(&invocation.effective_arguments).map(str::to_owned); + if let Some(args) = invocation.effective_arguments.as_object_mut() { + args.remove("skill"); + if let Some(name) = name { + args.insert("command".into(), serde_json::Value::String(name)); + } + } + } + (invocation, None) } @@ -6474,6 +6487,30 @@ mod tests { .any(|record| matches!(record.event, PermissionAuditEvent::Cancelled { .. }))); } + #[test] + fn skill_alias_reaches_hooks_as_command_without_changing_wire_history() { + let mut context = test_tool_execution_context(); + context.deferred_tools.push("Skill".into()); + let args = json!({"command":null, "skill": "visualize", "arguments": "monthly sales"}); + for (name, wire) in [ + ("Skill", args.clone()), + ( + CALL_DEFERRED_TOOL_NAME, + json!({"tool_name": "Skill", "args": args}), + ), + ] { + let mut call = test_tool_call("skill-alias", name); + call.arguments = wire.clone(); + let (invocation, error) = resolve_pipeline_invocation(&call, &context); + assert!(error.is_none()); + assert_eq!(invocation.wire_arguments, wire); + assert_eq!( + invocation.effective_arguments, + json!({"command": "visualize", "arguments": "monthly sales"}) + ); + } + } + #[tokio::test] async fn deferred_gateway_normalizes_arguments_and_executes_effective_target() { let pipeline = test_tool_pipeline(); diff --git a/src/crates/assembly/core/src/agentic/tools/tool_context_runtime.rs b/src/crates/assembly/core/src/agentic/tools/tool_context_runtime.rs index 0f92115f63..a6a8749cc4 100644 --- a/src/crates/assembly/core/src/agentic/tools/tool_context_runtime.rs +++ b/src/crates/assembly/core/src/agentic/tools/tool_context_runtime.rs @@ -287,7 +287,7 @@ pub(crate) async fn call_tool_with_runtime_hooks<T: Tool + ?Sized>( call_with_tool_runtime_hooks(tool.name(), input, context, tool.call_impl(input, context)).await } -pub(crate) const SKILL_USER_REQUEST: &str = "__openbitfun_skill_user_request"; +pub(crate) const TURN_USER_REQUEST: &str = "__openbitfun_skill_user_request"; pub(crate) fn build_tool_use_context_for_task( task: &ToolTask, @@ -301,11 +301,14 @@ pub(crate) fn build_tool_use_context_for_task( cancellation_token, task.round_injection_preemption_token.clone(), ); - if task.invocation.effective_tool_name == "Skill" { + if matches!( + task.invocation.effective_tool_name.as_str(), + "Skill" | "Write" | "Edit" + ) { if let Some(request) = &task.options.current_user_message { context .custom_data - .insert(SKILL_USER_REQUEST.into(), Value::String(request.clone())); + .insert(TURN_USER_REQUEST.into(), Value::String(request.clone())); } } context diff --git a/src/crates/services/services-core/src/session/types.rs b/src/crates/services/services-core/src/session/types.rs index 7bd8df59e7..400b0b8dcf 100644 --- a/src/crates/services/services-core/src/session/types.rs +++ b/src/crates/services/services-core/src/session/types.rs @@ -631,6 +631,8 @@ pub struct DialogTurnData { #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "camelCase")] pub struct PublishedFigure { + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + pub inspection_deferred: bool, #[serde(default, skip_serializing_if = "Option::is_none")] pub bindings: Option<serde_json::Map<String, serde_json::Value>>, pub source: String, @@ -1438,6 +1440,20 @@ mod tests { }; use openbitfun_core_types::{SessionContinuationPolicy, SessionKind}; + #[test] + fn published_figure_preserves_legacy_and_deferred_snapshots() { + let old = serde_json::json!({"source": "<p>map</p>", "contentBytes": 10}); + let figure: super::PublishedFigure = serde_json::from_value(old.clone()).unwrap(); + assert!(!figure.inspection_deferred); + assert_eq!(serde_json::to_value(figure).unwrap(), old); + + let mut deferred = old; + deferred["inspectionDeferred"] = true.into(); + let figure: super::PublishedFigure = serde_json::from_value(deferred.clone()).unwrap(); + assert!(figure.inspection_deferred); + assert_eq!(serde_json::to_value(figure).unwrap(), deferred); + } + #[test] fn dialog_turn_kind_defaults_to_user_dialog_for_legacy_payloads() { let payload = serde_json::json!({ diff --git a/src/shared/visualize/regional-map.css b/src/shared/visualize/regional-map.css index 00bd2af275..32bb5d8df5 100644 --- a/src/shared/visualize/regional-map.css +++ b/src/shared/visualize/regional-map.css @@ -17,3 +17,13 @@ svg[data-visual-atlas-map] [data-map-context], svg[data-visual-atlas-map] [data- svg[data-visual-atlas-map] [data-map-context="provinces"], svg[data-visual-atlas-map] [data-map-scale] { fill: var(--muted-foreground); } svg[data-visual-atlas-map] [data-visual-marker] circle { fill: var(--red); stroke: var(--background); } svg[data-visual-atlas-map] [data-visual-marker] text { fill: var(--foreground); stroke: var(--background); } +svg[data-visual-atlas-map] [data-street-ground] { fill: var(--background); } +svg[data-visual-atlas-map] [data-street-layer="land"] { fill: var(--atlas-land); } +svg[data-visual-atlas-map] [data-street-layer="water_polygons"], +svg[data-visual-atlas-map] [data-street-layer="ocean"] { fill: var(--atlas-water); } +svg[data-visual-atlas-map] [data-street-layer="water_lines"] { fill: none; stroke: var(--atlas-water); stroke-width: 2; } +svg[data-visual-atlas-map] [data-street-layer="streets"], +svg[data-visual-atlas-map] [data-street-layer="main-roads"] { fill: none; stroke: var(--muted-foreground); opacity: .36; stroke-width: .85; } +svg[data-visual-atlas-map] [data-street-layer="main-roads"] { opacity: .62; stroke-width: 1.4; } +svg[data-visual-atlas-map] [data-map-context="street_labels_points"], +svg[data-visual-atlas-map] [data-map-context="place_labels"] { fill: var(--muted-foreground); } diff --git a/src/web-ui/package.json b/src/web-ui/package.json index a802692ffb..1926dbe90c 100644 --- a/src/web-ui/package.json +++ b/src/web-ui/package.json @@ -31,6 +31,7 @@ "test:run": "node scripts/run-vitest.mjs run --maxWorkers=50%" }, "dependencies": { + "@mapbox/vector-tile": "3.0.0", "@monaco-editor/react": "^4.6.0", "@openbitfun/design-tokens": "workspace:^", "@openbitfun/flow-chat-presentation": "workspace:*", @@ -81,6 +82,7 @@ "parse-entities": "^4.0.2", "partial-json": "^0.1.7", "path-browserify": "^1.0.1", + "pbf": "5.1.2", "pdf-lib": "1.17.1", "pdfjs-dist": "5.4.624", "prismjs": "^1.30.0", diff --git a/src/web-ui/src/flow_chat/grouping/activityGrouping.test.ts b/src/web-ui/src/flow_chat/grouping/activityGrouping.test.ts index d8d338cb9c..65809616d3 100644 --- a/src/web-ui/src/flow_chat/grouping/activityGrouping.test.ts +++ b/src/web-ui/src/flow_chat/grouping/activityGrouping.test.ts @@ -39,6 +39,12 @@ describe('execution activity grouping', () => { expect(folded.map(item => item.id)).toEqual(['scratch']); expect(prepare(model('later', [tool('terminal', 'ExecCommand')]).data)).toBeUndefined(); }); + it('does not fold exploration after an unsuccessful skill alias call', () => { + const prepare = figurePreparation('/work/project'); + const failed = tool('failed', 'Skill', { skill: 'visualize' }, 'error'); + expect(prepare(model('failed-round', [failed, tool('shell', 'ExecCommand')]).data)).toBeUndefined(); + }); + it('keeps inline group identities independent of their position in a projected list', () => { const first = tool('read', 'Read'); const second = tool('run', 'ExecCommand'); diff --git a/src/web-ui/src/flow_chat/grouping/figurePreparation.ts b/src/web-ui/src/flow_chat/grouping/figurePreparation.ts index 7ee6587458..5dddff680a 100644 --- a/src/web-ui/src/flow_chat/grouping/figurePreparation.ts +++ b/src/web-ui/src/flow_chat/grouping/figurePreparation.ts @@ -1,5 +1,5 @@ import type { FlowToolItem, ModelRound } from '../types/flow-chat'; -import { projectEffectiveToolItem } from '../utils/toolInvocationIdentity'; +import { projectEffectiveToolItem, skillInvocationName } from '../utils/toolInvocationIdentity'; import { awaitsApproval, isShellActivity } from './activityClassification'; import { fileEditTarget } from './fileEdits'; import { classifyFlowGroupItem } from './policies'; @@ -26,16 +26,13 @@ export function figurePreparation(workspace?: string) { if (item.type !== 'tool' || item.subagentSessionId) continue; const tool = projectEffectiveToolItem(item as FlowToolItem); if (tool.importedSource) continue; - if (tool.toolName === 'Skill') { - active = (tool.toolResult?.result?.skill_name ?? tool.toolCall.input?.command) === 'visualize'; + if (awaitsApproval(tool) || ['rejected', 'cancelled', 'error'].includes(tool.status) + || tool.toolResult?.success === false) { + if (tool.toolName === 'Skill') active = false; + continue; } + if (tool.toolName === 'Skill') active = skillInvocationName(tool) === 'visualize'; if (!active) continue; - if ( - awaitsApproval(tool) || - ['rejected', 'cancelled', 'error'].includes(tool.status) || - tool.toolResult?.success === false - ) - continue; if ( tool.toolName === 'Skill' || isShellActivity(tool) || diff --git a/src/web-ui/src/flow_chat/tool-cards/SkillDisplay.test.tsx b/src/web-ui/src/flow_chat/tool-cards/SkillDisplay.test.tsx index d8c6abe49f..e14a663da2 100644 --- a/src/web-ui/src/flow_chat/tool-cards/SkillDisplay.test.tsx +++ b/src/web-ui/src/flow_chat/tool-cards/SkillDisplay.test.tsx @@ -4,6 +4,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { createRoot, type Root } from 'react-dom/client'; import { JSDOM } from 'jsdom'; +import { isToolCardVisible } from '../utils/flowItemVisibility'; import { SkillDisplay } from './SkillDisplay'; import type { FlowToolItem, ToolCardConfig } from '../types/flow-chat'; @@ -124,7 +125,7 @@ describe('SkillDisplay', () => { [{ skill_name: 'deep-research', source_label: ' ', source_id: null }, 'Load skill: deep-research'], [{ skill_name: 'deep-research', source_label: 42, source_id: {}, source_slot: 'codex' }, 'Load skill: deep-research'], [{ skill_name: {}, name: 'legacy-name' }, 'Load skill: legacy-name'], - [{}, 'Load skill: Unknown skill'], + [{}, 'Load skill: deep-research'], ])('renders compatible tool result %j', (result, summary) => { const toolItem: FlowToolItem = { id: 'compatible-skill', @@ -143,6 +144,18 @@ describe('SkillDisplay', () => { expect(container.textContent).toBe(summary); }); + it.each([undefined, null, ' ', 42])('replays a skill alias with command %s', (command) => { + const toolItem: FlowToolItem = { + id: 'failed-skill', type: 'tool', toolName: 'Skill', status: 'error', timestamp: 0, + toolCall: { id: 'call', input: { command, skill: 'visualize' } }, + toolResult: { id: 'call', result: {}, success: false, error: 'command is required', timestamp: 0 }, + }; + act(() => { root.render(<SkillDisplay toolItem={toolItem} config={skillConfig} />); }); + expect(container.textContent).toBe('Load skill: visualize'); + expect(isToolCardVisible(toolItem)).toBe(true); + expect(isToolCardVisible({ ...toolItem, toolResult: undefined, status: 'running' })).toBe(false); + }); + it.each([ ['running', 'Load skill: deep-research'], ['pending', 'Load skill: deep-research'], diff --git a/src/web-ui/src/flow_chat/tool-cards/SkillDisplay.tsx b/src/web-ui/src/flow_chat/tool-cards/SkillDisplay.tsx index 312660d8e5..9853f64a13 100644 --- a/src/web-ui/src/flow_chat/tool-cards/SkillDisplay.tsx +++ b/src/web-ui/src/flow_chat/tool-cards/SkillDisplay.tsx @@ -2,7 +2,7 @@ * Skill tool display — compact row (same pattern as Read file). */ -import React, { useMemo } from 'react'; +import React from 'react'; import { useTranslation } from 'react-i18next'; import { SkillToolCard, @@ -10,30 +10,14 @@ import { } from '@openbitfun/ui/flow-chat'; import { getToolCardStatus, getToolCardStatusDescription } from './toolCardStatus'; import type { ToolCardProps } from '../types/flow-chat'; - -function nonEmptyString(value: unknown): string | undefined { - return typeof value === 'string' ? value.trim() || undefined : undefined; -} +import { skillInvocationName } from '../utils/toolInvocationIdentity'; export const SkillDisplay: React.FC<ToolCardProps> = React.memo(({ toolItem }) => { const { t } = useTranslation('flow-chat'); - const { toolCall, toolResult } = toolItem; + const { toolResult } = toolItem; const status = getToolCardStatus(toolItem); - - const loadedSkillName = useMemo(() => { - if (!toolResult?.result) return null; - const result = toolResult.result as Record<string, unknown>; - return nonEmptyString(result.skill_name) - || nonEmptyString(result.name) - || t('toolCards.skill.unknownSkill'); - }, [toolResult?.result, t]); - - const commandName = - (toolCall?.input?.command as string | undefined) || - (toolCall?.input?.skill_name as string | undefined) || - t('toolCards.skill.unknown'); - - const displayName = status === 'completed' && loadedSkillName ? loadedSkillName : commandName; + const displayName = skillInvocationName(toolItem) + ?? (status === 'completed' ? t('toolCards.skill.unknownSkill') : t('toolCards.skill.unknown')); return ( <SkillToolCard diff --git a/src/web-ui/src/flow_chat/tool-cards/toolCapsuleModel.ts b/src/web-ui/src/flow_chat/tool-cards/toolCapsuleModel.ts index 18b60b36d0..51d261c012 100644 --- a/src/web-ui/src/flow_chat/tool-cards/toolCapsuleModel.ts +++ b/src/web-ui/src/flow_chat/tool-cards/toolCapsuleModel.ts @@ -1,6 +1,6 @@ import { getToolCardStatus } from './toolCardStatus'; import type { FlowToolItem } from '../types/flow-chat'; -import { projectEffectiveToolItem } from '../utils/toolInvocationIdentity'; +import { projectEffectiveToolItem, skillInvocationName } from '../utils/toolInvocationIdentity'; function text(...values: unknown[]): string { return values.find((value): value is string => typeof value === 'string' && value.trim().length > 0)?.trim() ?? ''; @@ -64,7 +64,7 @@ export function getToolCapsuleSummary(item: FlowToolItem, t: (key: string, optio break; case 'Skill': kind = t('toolCapsule.skill'); - target = label = text(result.skill_name, result.name, input.command, input.skill_name); + target = label = skillInvocationName(effective) ?? ''; break; case 'GetToolSpec': kind = t('toolCards.getToolSpec.title'); diff --git a/src/web-ui/src/flow_chat/utils/flowItemVisibility.ts b/src/web-ui/src/flow_chat/utils/flowItemVisibility.ts index 12d0f44be2..58ce372bd8 100644 --- a/src/web-ui/src/flow_chat/utils/flowItemVisibility.ts +++ b/src/web-ui/src/flow_chat/utils/flowItemVisibility.ts @@ -2,7 +2,7 @@ import type { FlowItem, FlowTextItem, FlowToolItem } from '../types/flow-chat'; import { READ_TOOL_NAMES, SEARCH_TOOL_NAMES } from '../tool-cards/toolCardMetadata'; import { splitFilePathAndContent } from '@/shared/utils/partialJsonParser'; import { isCodexVisualUri, visualArtifactUri, visualSidecarUri, visualReceiptPath, visualArtifactRoot } from '@/tools/visualize/reference'; -import { DEFERRED_TOOL_GATEWAY_NAME, projectEffectiveToolItem } from './toolInvocationIdentity'; +import { DEFERRED_TOOL_GATEWAY_NAME, projectEffectiveToolItem, skillInvocationName } from './toolInvocationIdentity'; /** Shared by leaf cards and transcript composition so hidden cards leave no row. */ export function isToolCardVisible(toolItem: FlowToolItem): boolean { @@ -23,7 +23,7 @@ export function isToolCardVisible(toolItem: FlowToolItem): boolean { return false; } if (item.toolName === 'Skill') { - const skill = item.toolResult?.result?.skill_name ?? input?.command; + const skill = skillInvocationName(item); if (item.isParamsStreaming && (!skill || 'visualize'.startsWith(skill))) return false; return skill !== 'visualize'; } diff --git a/src/web-ui/src/flow_chat/utils/toolInvocationIdentity.ts b/src/web-ui/src/flow_chat/utils/toolInvocationIdentity.ts index a88d1d086a..ab71013a12 100644 --- a/src/web-ui/src/flow_chat/utils/toolInvocationIdentity.ts +++ b/src/web-ui/src/flow_chat/utils/toolInvocationIdentity.ts @@ -3,6 +3,14 @@ import { importedToolDisplayName, projectImportedToolItem } from './importedTool export const DEFERRED_TOOL_GATEWAY_NAME = 'CallDeferredTool'; +/** Recover the same skill identity from live parameters and historical receipts. */ +export function skillInvocationName(item: FlowToolItem): string | undefined { + const input = item.toolCall?.input; + const result = item.toolResult?.result; + return [result?.skill_name, result?.name, input?.command, input?.skill, input?.skill_name] + .find((name): name is string => typeof name === 'string' && name.trim().length > 0)?.trim(); +} + export interface EffectiveToolInvocation { toolName: string; input: unknown; diff --git a/src/web-ui/src/infrastructure/api/adapters/figureInspectionBridge.ts b/src/web-ui/src/infrastructure/api/adapters/figureInspectionBridge.ts index e332e38c40..8c4918e7a0 100644 --- a/src/web-ui/src/infrastructure/api/adapters/figureInspectionBridge.ts +++ b/src/web-ui/src/infrastructure/api/adapters/figureInspectionBridge.ts @@ -1,10 +1,10 @@ -import type { FigureOwner } from '@/tools/visualize/visualBindings'; +import type { WrittenFigureRequest } from '@/tools/visualize/backgroundInspection'; import { getActiveSurfaceId, isLocalSurface } from '@/infrastructure/peer-device/deviceSurface'; import { emit, listen } from '@tauri-apps/api/event'; /** Local to the producing desktop. No relay, peer routing or selected chat is involved. */ export function listenForFigureChecks() { - return listen<{ id: string; request: FigureOwner & { source: string } }>('figure-check', async ({ payload }) => { + return listen<{ id: string; request: WrittenFigureRequest }>('figure-check', async ({ payload }) => { const { id, request } = payload; if (!/^[0-9a-f-]{36}$/i.test(id) || typeof request?.source !== 'string') return; let result; @@ -13,7 +13,7 @@ export function listenForFigureChecks() { const checked = await inspectWrittenFigure(request, async sessionId => { const { flowChatStore } = await import('@/flow_chat/store/FlowChatStore'); if (isLocalSurface(getActiveSurfaceId())) return flowChatStore.getState().sessions.get(sessionId); - }); + }, () => emit('figure-check-stage-' + id)); result = { Ok: checked }; } catch (e) { result = { Err: e instanceof Error ? e.message : String(e) }; diff --git a/src/web-ui/src/locales/en-US/flow-chat.json b/src/web-ui/src/locales/en-US/flow-chat.json index c317837937..b6a371a5be 100644 --- a/src/web-ui/src/locales/en-US/flow-chat.json +++ b/src/web-ui/src/locales/en-US/flow-chat.json @@ -1,6 +1,8 @@ { "visualize": { "mapCredit": "Map data · Natural Earth", + "deferredStreetMap": "This reply has no saved street basemap. Loading it here sends public stop names and the map area to the map providers.", + "loadStreetMap": "Load street map", "mapNorth": "N", "mapKilometers": "km", "mapMeters": "m", diff --git a/src/web-ui/src/locales/zh-CN/flow-chat.json b/src/web-ui/src/locales/zh-CN/flow-chat.json index 05a26e9dbf..54a5ec4634 100644 --- a/src/web-ui/src/locales/zh-CN/flow-chat.json +++ b/src/web-ui/src/locales/zh-CN/flow-chat.json @@ -1,6 +1,8 @@ { "visualize": { "mapCredit": "底图数据 Natural Earth", + "deferredStreetMap": "这条回复未保存街道底图。在这台设备加载时,地图服务商会收到公开站点名称和地图范围。", + "loadStreetMap": "加载街道地图", "mapNorth": "北", "mapKilometers": "公里", "mapMeters": "米", diff --git a/src/web-ui/src/locales/zh-TW/flow-chat.json b/src/web-ui/src/locales/zh-TW/flow-chat.json index 76703dca74..24c60e06ee 100644 --- a/src/web-ui/src/locales/zh-TW/flow-chat.json +++ b/src/web-ui/src/locales/zh-TW/flow-chat.json @@ -1,6 +1,8 @@ { "visualize": { "mapCredit": "底圖資料 Natural Earth", + "deferredStreetMap": "這則回覆未儲存街道底圖。在此裝置載入時,地圖服務商會收到公開站點名稱和地圖範圍。", + "loadStreetMap": "載入街道地圖", "mapNorth": "北", "mapKilometers": "公里", "mapMeters": "公尺", diff --git a/src/web-ui/src/shared/types/session-history.ts b/src/web-ui/src/shared/types/session-history.ts index 7776a558e3..672cb5e972 100644 --- a/src/web-ui/src/shared/types/session-history.ts +++ b/src/web-ui/src/shared/types/session-history.ts @@ -239,7 +239,7 @@ export interface SessionTurnCatalog { } export type FigureBindingSnapshot = Record<string, { kind: 'json' | 'file' | 'image'; value: unknown }>; -export type PublishedFigures = Record<string, { source: string; contentBytes: number; bindings?: FigureBindingSnapshot } | null>; +export type PublishedFigures = Record<string, { source: string; contentBytes: number; inspectionDeferred?: boolean; bindings?: FigureBindingSnapshot } | null>; export interface DialogTurnData { publishedFigures?: PublishedFigures; diff --git a/src/web-ui/src/tools/visualize/CodexFigure.tsx b/src/web-ui/src/tools/visualize/CodexFigure.tsx index 61130711ef..442d8c8077 100644 --- a/src/web-ui/src/tools/visualize/CodexFigure.tsx +++ b/src/web-ui/src/tools/visualize/CodexFigure.tsx @@ -13,7 +13,8 @@ import { z } from 'zod/v4'; import { useI18n } from '@/infrastructure/i18n'; import { useHostAppearance } from '@/infrastructure/appearance/hooks/useHostAppearance'; import { buildCodexFrame } from './codexFrameDocument'; -import { watchFigureInspection } from './figureInspection'; +import { watchFigureInspection, type MapPublicationCheck } from './figureInspection'; +import { mapPublication } from './mapPublication'; import { readFigureStyle } from './figureAppearance'; import { readWidgetState } from './widgetStateStorage'; @@ -63,12 +64,14 @@ export default function CodexFigure({ uri, write, checkedBindings, + deferredRequest, getSession, subscribe, }: { uri: string; write: FlowToolItem | FlowTextItem; checkedBindings?: BoundValues; + deferredRequest?: string; getSession: () => Session | undefined; subscribe: (notify: () => void) => () => void; html: string; @@ -109,6 +112,8 @@ export default function CodexFigure({ { name: string; title?: string; description?: string }[] >([]); const [loadError, setLoadError] = useState(false); + const [mapNeedsConsent, setMapNeedsConsent] = useState(false); + const [onlineRevision, setOnlineRevision] = useState<string>(); const [action, setAction] = useState<{ href?: string; prompt?: string; title?: string } | null>(null); const [attempt, setAttempt] = useState(0); const [ready, setReady] = useState(0); @@ -166,13 +171,13 @@ export default function CodexFigure({ useEffect(() => { if (ready && Object.keys(liveValues).length) presentation.current?.postMessage({ values: liveValues }); }, [ready, liveValues]); - const [built, setBuilt] = useState<{ html: string; document: string; version: number }>(); + const [built, setBuilt] = useState<{ html: string; document: string; version: number; publication?: MapPublicationCheck; saveMap?: () => void }>(); const source = built?.html === html ? built.document : ''; const [passed, setPassed] = useState(0); const [issue, setIssue] = useState<string | null>(null); useEffect(() => { let mounted = true; - let state = null; + let state: ReturnType<typeof parseSnapshot> | null = null; try { state = parseSnapshot(readWidgetState(localStorage, storageKey, legacyKey, roundId) ?? 'null'); } catch { @@ -180,17 +185,43 @@ export default function CodexFigure({ } modelContent.current = state?.modelContent ?? null; const initialStyle = readFigureStyle(wrapper.current ?? document.documentElement); - buildCodexFrame(html, state, currentLanguage, declarations ?? {}, bindings, initialStyle) - .then((document) => { - if (mounted) setBuilt(prev => ({ html, document, version: (prev?.version ?? 0) + 1 })); - }) - .catch(() => { - if (mounted) setLoadError(true); - }); + setMapNeedsConsent(false); + setLoadError(false); + setIssue(null); + async function build() { + let fragment = html; + let saveMap: (() => void) | undefined; + if (html.includes('data-visual-atlas')) { + const { recoverReaderStreetMap } = await import('./regionalAtlas'); + if (!mounted) return; + const allowNames = !!declarations && Object.values(declarations).every(entry => entry.type === 'image'); + let recovery; + try { + recovery = await recoverReaderStreetMap(html, storageKey, allowNames, onlineRevision === html); + } catch (error) { + if (mounted && error instanceof Error) setIssue(error.message); + throw error; + } + if (!mounted) return; + if (!recovery) { + setMapNeedsConsent(true); + return; + } + fragment = recovery.html; + saveMap = recovery.save; + } + const publication = deferredRequest === undefined ? undefined : + { ...await mapPublication(uri, deferredRequest, fragment), replay: true }; + const document = await buildCodexFrame(fragment, state, currentLanguage, declarations ?? {}, bindings, initialStyle); + if (mounted) setBuilt(prev => ({ html, document, publication, saveMap, version: (prev?.version ?? 0) + 1 })); + } + void build().catch(() => { + if (mounted) setLoadError(true); + }); return () => { mounted = false; }; - }, [html, storageKey, legacyKey, roundId, currentLanguage, attempt, declarations, bindings]); + }, [html, storageKey, legacyKey, roundId, currentLanguage, attempt, declarations, bindings, onlineRevision, uri, deferredRequest]); const changes = useMemo( () => groups.map((g) => ({ @@ -239,8 +270,11 @@ export default function CodexFigure({ setIssue(null); const stopChecking = watchFigureInspection(frame, problem => { if (problem) setIssue(problem); - else setPassed(built!.version); - }); + else { + built!.saveMap?.(); + setPassed(built!.version); + } + }, built?.publication); setGroups([]); setEdits({}); setLoadError(false); @@ -646,7 +680,13 @@ export default function CodexFigure({ })} </section> )} - {!verified && !loadError && !issue && <small role="status">{t('visualize.loading')}</small>} + {mapNeedsConsent && ( + <div role="status"> + {t('visualize.deferredStreetMap')}{' '} + <button type="button" onClick={() => setOnlineRevision(html)}>{t('visualize.loadStreetMap')}</button> + </div> + )} + {!mapNeedsConsent && !verified && !loadError && !issue && <small role="status">{t('visualize.loading')}</small>} {(loadError || issue) && ( <div role="status"> {t('visualize.resourceError')} {issue}{' '} diff --git a/src/web-ui/src/tools/visualize/VisualAnswer.tsx b/src/web-ui/src/tools/visualize/VisualAnswer.tsx index 112b815271..5f9e12f01b 100644 --- a/src/web-ui/src/tools/visualize/VisualAnswer.tsx +++ b/src/web-ui/src/tools/visualize/VisualAnswer.tsx @@ -426,6 +426,8 @@ export default function VisualAnswer({ {summary} <small role="status">{t('visualize.unavailable')}</small> </p> ); + const deferred = turn.publishedFigures?.[uri]?.inspectionDeferred || + (revision?.type === 'tool' && revision.toolResult?.result?.visualization_check_deferred === true); const stableKey = surface.key('visualize.v2', owner, turn.id, round.id, (item.type === 'tool' ? item.toolCall.id : item.id), uri); return ( <Suspense fallback={<p>{summary}</p>}> @@ -440,6 +442,7 @@ export default function VisualAnswer({ follow={follow} uri={uri} checkedBindings={checkedBindings} + deferredRequest={deferred ? turn.userMessage.content : undefined} write={revision ?? item} getSession={getSession} subscribe={subscribe} diff --git a/src/web-ui/src/tools/visualize/backgroundInspection.ts b/src/web-ui/src/tools/visualize/backgroundInspection.ts index 95f8ab36ed..81e78736ab 100644 --- a/src/web-ui/src/tools/visualize/backgroundInspection.ts +++ b/src/web-ui/src/tools/visualize/backgroundInspection.ts @@ -1,12 +1,13 @@ +import { mapPublication } from './mapPublication'; import { buildCodexFrame } from './codexFrameDocument'; -import { watchFigureInspection } from './figureInspection'; +import { watchFigureInspection, type MapPublicationCheck } from './figureInspection'; import { readBindings, inspectionBindings, usesRecordedData, type BoundValues, type FigureOwner } from './visualBindings'; import type { Session } from '@/flow_chat/types/flow-chat'; import { snapshotFigureScripts } from './figureResources'; import { localizeFigureImages } from './figureImages'; import { readFigureStyle } from './figureAppearance'; -export async function inspectInBackground(source: string, snapshot?: BoundValues): Promise<string | null> { +export async function inspectInBackground(source: string, snapshot?: BoundValues, publication?: MapPublicationCheck): Promise<string | null> { const declarations = readBindings(source); if (!declarations) return 'The visualization binding declaration is invalid.'; const values = snapshot ?? inspectionBindings(declarations, undefined); @@ -51,7 +52,7 @@ export async function inspectInBackground(source: string, snapshot?: BoundValues window.addEventListener('message', ready); try { return await new Promise((resolve) => { - watchFigureInspection(frame, resolve); + watchFigureInspection(frame, resolve, publication); frame.srcdoc = doc; document.body.append(frame); }); @@ -68,9 +69,15 @@ interface CheckedFigure { deferred?: boolean; } +export interface WrittenFigureRequest extends FigureOwner { + source: string; + userRequest?: string | null; +} + export async function inspectWrittenFigure( - request: FigureOwner & { source: string }, + request: WrittenFigureRequest, getProducer: (id: string) => Promise<Session | undefined>, + onStage?: () => Promise<void>, ): Promise<CheckedFigure> { const declarations = readBindings(request.source); if (!declarations) throw Error('Invalid visualization bindings.'); @@ -82,8 +89,9 @@ export async function inspectWrittenFigure( const hasImages = entries.some(entry => entry.type === 'image'); if (hasImages) bindings = await localizeFigureImages(declarations, bindings); const { materializeRegionalMaps } = await import('./regionalAtlas'); - const html = await snapshotFigureScripts(await materializeRegionalMaps(request.source), declarations); - const issue = await inspectInBackground(html, bindings); + const allowPublicLookup = entries.every(entry => entry.type === 'image'); + const html = await snapshotFigureScripts(await materializeRegionalMaps(request.source, true, allowPublicLookup, onStage), declarations); + const issue = await inspectInBackground(html, bindings, await mapPublication(request.uri, request.userRequest, html)); if (issue) throw Error(issue); return { renderedSource: html === request.source ? undefined : html, diff --git a/src/web-ui/src/tools/visualize/cachedVisualRuntime.ts b/src/web-ui/src/tools/visualize/cachedVisualRuntime.ts index ccf3e42a79..d2405b2cf9 100644 --- a/src/web-ui/src/tools/visualize/cachedVisualRuntime.ts +++ b/src/web-ui/src/tools/visualize/cachedVisualRuntime.ts @@ -1,9 +1,9 @@ import release from '../../../../../scripts/visualize-runtime.json'; -const loaders = import.meta.glob<string>('../../../../../.visualize-runtime/*.{js,json}', { +const loaders = import.meta.env.OPENBITFUN_VISUALIZE_DEV_KIT ? import.meta.glob<string>('../../../../../.visualize-runtime/*.{js,json}', { query: '?raw', import: 'default', -}); +}) : {}; const bundled = import.meta.glob<string>('../../../../../third_party/visualize/runtime/*.{js,json}', { query: '?raw', import: 'default', }); @@ -16,10 +16,16 @@ const inline = (body: string) => '<script>' + body.replace(/<\/script/gi, '<\\/s // release URL as its logical identity even when its bytes are served offline. export async function cachedVisualRuntime(fragment: string) { const selected = new Set<Asset>(['lucide.js', 'floating-core.js', 'floating-dom.js']); - if (fragment.includes('/charts/1.6.5/')) { - selected.add('chart.js'); - selected.add('chart-element.js'); - selected.add('chart-runtime.js'); + const chartReleases = [ + ['chart-v1.8.1.js', 'chart-element-v1.8.1.js', 'chart-runtime-v1.8.1.js'], + ['chart.js', 'chart-element.js', 'chart-runtime.js'], + ] as const; + for (const [entry, element, runtime] of chartReleases) { + if (fragment.includes(release[entry].url) || fragment.includes(release[element].url)) { + selected.add(entry); + selected.add(element); + selected.add(runtime); + } } for (const name of names) if (fragment.includes(release[name].url)) selected.add(name); const urls = fragment.match(/https:\/\/[^\s"'<>\x60]+/g) ?? []; @@ -103,7 +109,7 @@ export async function cachedVisualRuntime(fragment: string) { esm.push([aliases[key] ?? [], '@exports:' + lib]); } } else { - if (key === 'chart.js') + if (chartReleases.some(([entry]) => entry === key)) code = code.replace(/"\.\.\/[^"]+"/g, (path) => JSON.stringify(new URL(JSON.parse(path), url).href)); if (key.endsWith('.json')) code = 'export default ' + code; else code = code.replace(/import\.meta\.url/g, () => JSON.stringify(url)); diff --git a/src/web-ui/src/tools/visualize/codexFrameDocument.ts b/src/web-ui/src/tools/visualize/codexFrameDocument.ts index 6fd28f0fd6..04345aade8 100644 --- a/src/web-ui/src/tools/visualize/codexFrameDocument.ts +++ b/src/web-ui/src/tools/visualize/codexFrameDocument.ts @@ -9,13 +9,14 @@ import hostStyles from './figureHost.css?raw'; import chartTheme from './figureChartTheme.js?raw'; import extras from './codexFrameExtras.js?raw'; import inspection from './inspectRenderedFigure.js?raw'; +import release from '../../../../../scripts/visualize-runtime.json'; // The kit is loaded only for v2 figures. One local import supplies both the // runtime and the exact same authoring resources to the installed skill. -const raw = import.meta.glob<string>( +const raw = import.meta.env.OPENBITFUN_VISUALIZE_DEV_KIT ? import.meta.glob<string>( '../../../../crates/assembly/core/builtin_skills/visualize/upstream/{assets/*.js,skills/visualize/assets/*}', { eager: true, import: 'default', query: '?raw' }, -); +) : {}; const assets = Object.fromEntries( Object.entries(raw).map(([file, source]) => [file.split('/').pop()!, source]), ); @@ -59,8 +60,9 @@ export async function buildCodexFrame( // Use Tweak's existing hidden-attribute observer for tabs and conditional content as well as variants. const tweak = assets['tweak.js']?.split('.viz-carousel > [data-variant][hidden]').join('[hidden]') ?? builtInControls; let extensions = script(assets['standalone-host-bridge.js'] ?? builtInGuest) + script(extras) + script(inspection) + script(tweak); - if (hasReferenceKit && /<viz-calendar\b/i.test(fragment) && !/\/widgets\/[^/]+\/(?:calendar|element)\.js/.test(fragment)) { - extensions += script(assets['calendar.js']); + if (/<viz-calendar\b/i.test(fragment) && !/\/widgets\/[^/]+\/(?:calendar|element)\.js/.test(fragment)) { + if (hasReferenceKit) extensions += script(assets['calendar.js']); + else fragment = '<script type="module">import "' + release['calendar.js'].url + '"</script>' + fragment; } if ( hasReferenceKit && /<(?:experimental-)?dash-[a-z-]+\b|<dash-sources\b|data-dashboard-|class=["'][^"']*dashboard-/.test( diff --git a/src/web-ui/src/tools/visualize/figureInspection.ts b/src/web-ui/src/tools/visualize/figureInspection.ts index 48c2c5adc1..e94a78159f 100644 --- a/src/web-ui/src/tools/visualize/figureInspection.ts +++ b/src/web-ui/src/tools/visualize/figureInspection.ts @@ -1,5 +1,13 @@ +export interface MapPublicationCheck { + mapExpected: boolean; + replay?: boolean; + mapRequested?: boolean; + itineraryExpected?: boolean; +} + // The listener is scoped to one frame/document, including cancellation on retry or unmount. -export function watchFigureInspection(frame: HTMLIFrameElement, done: (issue: string | null) => void) { +export function watchFigureInspection(frame: HTMLIFrameElement, done: (issue: string | null) => void, + publication?: MapPublicationCheck) { let port: MessagePort | undefined; const timer = window.setTimeout(onTimeout, 12000); const stop = () => { @@ -27,7 +35,7 @@ export function watchFigureInspection(frame: HTMLIFrameElement, done: (issue: st port.onmessage = ({ data }) => { if (data === null || typeof data === 'string') finish(data); }; - port.postMessage('inspect'); + port.postMessage(publication ?? 'inspect'); } window.addEventListener('message', receive); return stop; diff --git a/src/web-ui/src/tools/visualize/geographicRequestPatterns.json b/src/web-ui/src/tools/visualize/geographicRequestPatterns.json new file mode 100644 index 0000000000..adf61fdbfe --- /dev/null +++ b/src/web-ui/src/tools/visualize/geographicRequestPatterns.json @@ -0,0 +1,14 @@ +{ + "en": { + "schematic": "\\b(?:schematic|diagram|concepts?|mind map|roadmap|heat\\s?map|codebase|architecture|dependenc(?:ies|y)|repository)\\b", + "map": "^(?:please\\s+)?(?:draw|show|make|create)\\s+(?:me\\s+)?(?:an?\\s+|the\\s+)?(?:geographic|geographical|street|walking|route|city|regional|travel|world)\\s+map\\b|^where(?:\\s+is|['’]s)\\s+(?:the|my|our)\\s+map\\s*$", + "itinerary": "\\b(?:itinerary|walking (?:tour|route)|day trip)\\b", + "otherDeliverable": "\\b(?:and|also|plus|with)\\s+(?:(?:then|separately|also)\\s+)*(?:(?:draw|show|build|make|create|provide|give|animate|plot|include|add)\\b|(?:an?\\s+|the\\s+)?(?:calendar|schedule|table|chart|timeline|dashboard|checklist|list)\\b)" + }, + "zh": { + "schematic": "示意|概念|思[维維]|[热熱]力|架[构構]|依[赖賴]|[项項]目[结結][构構]", + "map": "^(?:请|請|帮我|幫我|给我|給我|麻烦|麻煩|\\s)*(?:[画畫绘繪]|展示|显示|顯示)[^,,;;]{0,64}地[图圖]|^地[图圖](?:[给給](?:我|我们|我們))?(?:看看|看一下|看下|展示一下)|^(?:(?:那|那[张張]|路[线線]|行程)\\s*)?地[图圖]\\s*呢\\s*$", + "itinerary": "[游遊]玩路[线線]|一日[游遊]|一日出行|出行[规規][划劃]|步行路[线線]|徒步路[线線]", + "otherDeliverable": "(?:[并並再另]|以及|同[时時]|外加|附[上带帶]?|[还還]要|和)[^。!?.!?,,;;\\n]{0,8}(?:[给給画畫绘繪做列]|生成|制作|展示|提供|日[历曆]|行程表|[时時][间間][线線表]|[图圖]表|表格|清[单單])" + } +} diff --git a/src/web-ui/src/tools/visualize/inspectRenderedFigure.js b/src/web-ui/src/tools/visualize/inspectRenderedFigure.js index b205d503e5..93971d5d6a 100644 --- a/src/web-ui/src/tools/visualize/inspectRenderedFigure.js +++ b/src/web-ui/src/tools/visualize/inspectRenderedFigure.js @@ -41,9 +41,54 @@ r.height, ].join('|'); } - function inspect() { + // On new writes the map destination carries the author's intent even if its + // fragment lost the atlas declaration. Also inspect the accessible name of + // an existing free-form route drawing. Historical replay uses its old check. + function inspectMapPublication({ mapExpected, mapRequested, itineraryExpected }) { + const mapRepair = 'Missing geographic basemap. Use data-visual-atlas="1" with detail:"street", city, country and the public stop names for a local itinerary. ' + + (mapRequested ? 'The reader asked for a map. If publication fails, report the specific error.' : + 'If street data is unavailable, show a labeled stop list without a scale.'); + let found = false; + let cityOnly = false; + let hasScale = false; + for (const el of document.querySelectorAll('svg,canvas,img,[data-visual-map]')) { + if (!bound(el) || el.parentElement?.closest('svg,[data-visual-map]')) continue; + const title = el.getAttribute('aria-label') || el.getAttribute('alt') || el.querySelector('title')?.textContent || ''; + const scale = el.querySelector('[data-map-scale]') || [...el.querySelectorAll('text')].some( + text => /^\s*\d+(?:[.,]\d+)?\s*(?:km|mi|千米|公里|英里)\s*$/iu.test(text.textContent)); + if (scale) hasScale = true; + const route = /\b(?:route|itinerary|street|walking|geographic)\s+map\b|地[图圖]|[路航][线線](?:示意)?[图圖]/iu.test(title); + const declared = el.hasAttribute('data-visual-map'); + if (!declared && !route && !mapExpected) continue; + // A labeled, unscaled schematic may accompany an itinerary. It does not + // satisfy a requested geographic map. + if (!declared && !mapExpected && !scale && /schematic|示意/iu.test(title)) continue; + const source = el.getAttribute('data-map-source') || ''; + const geometry = el.querySelectorAll('[data-map-layer], [data-street-layer], [data-visual-region]'); + if ((el.hasAttribute('data-visual-atlas-map') || /^https?:\/\//i.test(source)) && + [...geometry].some(shape => shape instanceof SVGGeometryElement && bound(shape) && shape.getTotalLength() > 0)) { + if (itineraryExpected && el.hasAttribute('data-visual-atlas-map') && el.querySelector('[data-map-layer="land"]')) { + const positions = Array.from(el.querySelectorAll('[data-map-stop] circle'), dot => + dot.getAttribute('cx') + ',' + dot.getAttribute('cy')); + if (new Set(positions).size === 1) { cityOnly = true; continue; } + } + found = true; + } else if (declared || route) return mapRepair; + } + if (cityOnly && !found) return 'This itinerary map shows only a city center. Plot the public stops in visit order using detail:"street", city and places.'; + if (mapExpected && !found) { + const stops = document.querySelector('ol[data-visual-itinerary]'); + if (mapRequested || !stops || !bound(stops) || !stops.querySelector('li') || hasScale || [...document.querySelectorAll('[data-map-scale]')].some(bound)) return mapRepair; + } + return ''; + } + function inspect(publication) { if (issues.length) return issues[0]; if (document.readyState === 'loading') return 'Markup is still loading'; + if (publication && typeof publication === 'object') { + const problem = inspectMapPublication(publication); + if (problem) return problem; + } const map = document.querySelector('[data-visual-map]'); if (map) { const shapes = [...map.querySelectorAll('[data-visual-region], path[d], polygon[points]')]; @@ -72,12 +117,27 @@ } return 'The fragment produced no visible content.'; } - async function run() { + // Exercise the change handler in the disposable publishing frame. Reading a + // saved figure must never change the reader's selection or persisted state. + async function checkSelects() { + for (const select of document.querySelectorAll('select:enabled:not([multiple])')) { + if (!bound(select)) continue; + const option = Array.from(select.options).find(o => + o.value !== select.value && !o.hidden && !o.matches(':disabled')); + if (!option) continue; + select.selectedIndex = option.index; + select.dispatchEvent(new Event('input', { bubbles: true })); + select.dispatchEvent(new Event('change', { bubbles: true })); + await delay(0); + if (issues.length) return 'Selecting "' + option.label.slice(0, 60) + '" failed: ' + issues[0]; + } + } + async function run(publication) { let issue; const start = Date.now(); do { await delay(100); - issue = inspect(); + issue = inspect(publication); } while (issue && !issues.length && Date.now() - start < 5500); if (!issue && !matchMedia('(prefers-reduced-motion: reduce)').matches) { const movers = [...document.querySelectorAll('[data-visual-motion]')].filter(bound); @@ -95,12 +155,16 @@ issue = 'The marked simulation is stationary. Check its animation target and startup state.'; } } + if (!issue && !issues.length && publication && typeof publication === 'object' && !publication.replay) { + const selectionError = await checkSelects(); + if (selectionError) return selectionError; + } return issues[0] || issue || null; } - reply.onmessage = async () => { + reply.onmessage = async ({ data }) => { reply.onmessage = null; try { - reply.postMessage(await run()); + reply.postMessage(await run(data)); } catch (e) { reply.postMessage(String(e?.message || e).slice(0, 240)); } finally { diff --git a/src/web-ui/src/tools/visualize/mapPublication.ts b/src/web-ui/src/tools/visualize/mapPublication.ts new file mode 100644 index 0000000000..4c71847ecf --- /dev/null +++ b/src/web-ui/src/tools/visualize/mapPublication.ts @@ -0,0 +1,32 @@ +import patterns from './geographicRequestPatterns.json'; +import type { MapPublicationCheck } from './figureInspection'; + +const geographicRequests = Object.values(patterns).map(({ map, schematic, itinerary, otherDeliverable }) => ({ + map: new RegExp(map, 'iu'), schematic: new RegExp(schematic, 'iu'), + itinerary: new RegExp(itinerary, 'iu'), otherDeliverable: new RegExp(otherDeliverable, 'iu'), +})); + +export async function mapPublication(uri: string, userRequest: string | null | undefined, html: string): Promise<MapPublicationCheck> { + const message = userRequest?.trim() ?? ''; + const first = message.split(/[。!?.!?\n]/u, 1)[0]; + let geographic = geographicRequests.some(({ map, schematic }) => map.test(first) && !schematic.test(first)); + const schematic = geographicRequests.some(pattern => pattern.schematic.test(first)); + const subject = first.match(/^(?:please\s+)?(?:draw|show|make|create)\s+(?:me\s+)?(?:(?:a|the)\s+)?map of (.+?)(?:[,;]|\s+and\b|$)/iu)?.[1].trim(); + if (subject && !schematic && !geographic) { + const { isAtlasPlace } = await import('./regionalAtlas'); + geographic = await isAtlasPlace(subject); + } + const ambiguous = !!subject && !geographic; + const fragment = document.createElement('template'); + fragment.innerHTML = html; + const declared = !!fragment.content.querySelector('[data-visual-map],script[data-visual-atlas]'); + const mapArtifact = declared || (!ambiguous && !schematic && /\/map-[^/]+\.html$/.test(uri)); + const multipleOutputs = geographicRequests.some(pattern => pattern.otherDeliverable.test(message)); + const mapRequested = geographic && (mapArtifact || !multipleOutputs); + const headings = Array.from(fragment.content.querySelectorAll('h1,h2,h3'), el => el.textContent).join(' '); + return { + mapExpected: mapArtifact || mapRequested, + mapRequested, + itineraryExpected: declared && geographicRequests.some(pattern => pattern.itinerary.test(message + ' ' + headings)), + }; +} diff --git a/src/web-ui/src/tools/visualize/regionalAtlas.ts b/src/web-ui/src/tools/visualize/regionalAtlas.ts index 512ccb7caa..64b24550c8 100644 --- a/src/web-ui/src/tools/visualize/regionalAtlas.ts +++ b/src/web-ui/src/tools/visualize/regionalAtlas.ts @@ -11,12 +11,47 @@ import seas from '../../../../../third_party/visualize/atlas/seas.json?url&no-in const layers = { land, rivers, lakes, places, borders, provinces, seas }; -async function project(spec: unknown, language: string): Promise<string> { +let placeNames: Promise<Set<string>> | undefined; + +/** An unqualified "map of X" needs positive geographic evidence before it can + * invalidate an otherwise legitimate figure. Reuse the shipped gazetteer. */ +export async function isAtlasPlace(name: string) { + placeNames ??= fetch(places).then(async response => { + if (!response.ok) throw Error('Regional place names unavailable'); + const rows: [string, string, string[]][] = await response.json(); + const names = new Set<string>(); + const countries = new Set<string>(); + for (const [country, region, aliases] of rows) { + for (const alias of aliases) names.add(alias.toLowerCase()); + if (region) names.add(region.toLowerCase()); + if (/^[A-Z]{2}$/.test(country)) countries.add(country); + } + const english = new Intl.DisplayNames(['en'], { type: 'region' }); + for (const code of countries) names.add((english.of(code) ?? code).toLowerCase()); + return names; + }).catch(error => { + placeNames = undefined; + throw error; + }); + return (await placeNames).has(name.toLowerCase()); +} + +type AtlasRequest = { detail?: string; places?: (string | { coordinates?: unknown } | null)[] }; + +function needsPlaceLookup(spec: AtlasRequest) { + return spec?.detail === 'street' && Array.isArray(spec.places) && + spec.places.some(place => typeof place === 'string' || !place?.coordinates); +} + +async function project(spec: AtlasRequest, language: string): Promise<string> { const urls = Object.fromEntries(Object.entries(layers).map(([key, url]) => [key, new URL(url, location.href).href])); const worker = new Worker(new URL('./regionalProjection.js', import.meta.url), { type: 'module' }); try { return await new Promise<string>((resolve, reject) => { - const timeout = window.setTimeout(() => reject(Error('The regional map projection timed out.')), 10000); + // Named stops may require a cold geocoding connection before the tile requests. + const workerBudget = needsPlaceLookup(spec) ? 60000 : 15000; + const deadline = Date.now() + workerBudget; + const timeout = window.setTimeout(() => reject(Error('Map data did not load in time. For an unavailable street map, show a clearly labeled stop list without a scale.')), workerBudget); worker.onmessage = ({ data }) => { clearTimeout(timeout); if (typeof data.svg === 'string') resolve(data.svg); @@ -32,14 +67,14 @@ async function project(spec: unknown, language: string): Promise<string> { km: i18nService.t('flow-chat:visualize.mapKilometers', options), m: i18nService.t('flow-chat:visualize.mapMeters', options), }; - worker.postMessage({ spec, urls, language, labels }); + worker.postMessage({ spec, urls, language, labels, deadline: deadline - 1000, lookupDeadline: deadline - 15000 }); }); } finally { worker.terminate(); } } -export async function materializeRegionalMaps(source: string) { +export async function materializeRegionalMaps(source: string, publishing = false, allowPublicLookup = true, onStage?: () => Promise<void>) { if (!source.includes('data-visual-atlas')) return source; const fragment = document.createElement('template'); fragment.innerHTML = source; @@ -47,7 +82,12 @@ export async function materializeRegionalMaps(source: string) { if (!maps.length) return source; for (const declaration of maps) { const spec = JSON.parse(declaration.textContent || 'null'); + if (spec?.detail === 'street' && !publishing) + throw Error('This street map has no saved basemap. Publish it on the producing host before replay.'); + if (!allowPublicLookup && needsPlaceLookup(spec)) + throw Error('A figure with bound data cannot send stop names to a public geocoder. Supply sourced coordinates or local projected geometry.'); const language = spec?.language || document.documentElement.lang || i18nService.getCurrentLocale(); + await onStage?.(); const svg = await project(spec, language); const result = document.createElement('template'); result.innerHTML = svg; @@ -55,12 +95,50 @@ export async function materializeRegionalMaps(source: string) { style.textContent = atlasStyle; result.content.querySelector('svg')!.prepend(style); const credit = document.createElement('small'); - credit.textContent = i18nService.t('flow-chat:visualize.mapCredit', { lng: language }); + if (spec.detail === 'street') { + const attribution = document.createElement('a'); + attribution.href = 'https://www.openstreetmap.org/copyright'; + attribution.textContent = '© OpenStreetMap contributors'; + credit.append(attribution, ' · VersaTiles'); + } else { + credit.textContent = i18nService.t('flow-chat:visualize.mapCredit', { lng: language }); + } credit.className = 'text-muted'; declaration.replaceWith(result.content, credit); } + await onStage?.(); const html = fragment.innerHTML; if (new TextEncoder().encode(html).length > 1024 * 1024) throw Error('The projected map exceeds the figure size limit. Use a regional extent.'); return html; } + +// A headless producer may save only the declaration. Opening history must not +// disclose its places or coordinates from a different reader's network. +export async function recoverReaderStreetMap(source: string, ownerKey: string, allowNames: boolean, loadOnline: boolean): Promise<{ html: string; save?: () => void } | undefined> { + const fragment = document.createElement('template'); + fragment.innerHTML = source; + const declarations = Array.from(fragment.content.querySelectorAll('script[type="application/json"][data-visual-atlas="1"]'), + el => JSON.parse(el.textContent || 'null')); + if (!declarations.some(spec => spec?.detail === 'street')) return { html: source }; + if (!allowNames && declarations.some(needsPlaceLookup)) + throw Error('This bound street map requires sourced stop coordinates or a basemap saved by its producer.'); + const key = 'visualize.street-map:' + ownerKey; + // Cache the generated markup under the original source revision. Readers + // can still open the figure when persistent storage is unavailable. + const digest = globalThis.crypto?.subtle && new Uint8Array(await crypto.subtle.digest('SHA-256', new TextEncoder().encode(source))).join('.'); + if (digest) { + try { + const saved = JSON.parse(localStorage.getItem(key) ?? 'null'); + if (saved?.revision === digest && typeof saved.html === 'string') return { html: saved.html }; + } catch { /* Missing or unreadable cache. */ } + } + if (!loadOnline) return undefined; + const html = await materializeRegionalMaps(source, true, allowNames); + // Save only after the actual frame passes the reader's render check. + return { html, save: () => { + if (!digest) return; + try { localStorage.setItem(key, JSON.stringify({ revision: digest, html })); } + catch { /* Keep displaying the current result when storage is full. */ } + } }; +} diff --git a/src/web-ui/src/tools/visualize/regionalProjection.js b/src/web-ui/src/tools/visualize/regionalProjection.js index 8ba4098c4b..477616a69e 100644 --- a/src/web-ui/src/tools/visualize/regionalProjection.js +++ b/src/web-ui/src/tools/visualize/regionalProjection.js @@ -1,6 +1,8 @@ import * as d3 from 'd3-geo'; import * as topojson from 'topojson-client'; -self.onmessage = async ({ data: { spec, urls, language, labels: strings } }) => { +import { streetBasemap } from './streetBasemap.js'; +import { resolveStreetStops } from './streetPlaces.js'; +self.onmessage = async ({ data: { spec, urls, language, labels: strings, lookupDeadline, deadline } }) => { const esc = (s) => String(s).replace( /[&<>"']/g, @@ -12,13 +14,17 @@ self.onmessage = async ({ data: { spec, urls, language, labels: strings } }) => return r.json(); }; try { - const { places, title, bounds } = spec; - if (!Array.isArray(places) || !places.length || places.length > 64) throw Error('Provide 1–64 places.'); - const gazetteer = await load('places'); + const { title, bounds } = spec; + if (!Array.isArray(spec.places) || !spec.places.length || spec.places.length > 64) throw Error('Provide 1–64 places.'); + let places = spec.places.map(p => typeof p === 'string' ? { name: p } : p); + if (places.some(p => !p || typeof p.name !== 'string' || !p.name.trim())) throw Error('A map marker needs a name.'); + const street = spec.detail === 'street'; + if (spec.detail !== undefined && !street && spec.detail !== 'regional') + throw Error('Unknown map detail. Use street or regional.'); + const needsCoordinates = places.some(place => !place.coordinates); + const gazetteer = needsCoordinates ? await load('places') : []; + if (street && needsCoordinates) places = await resolveStreetStops(places, spec, gazetteer, lookupDeadline); const points = places.map((place) => { - if (typeof place === 'string') place = { name: place }; - if (!place || typeof place.name !== 'string' || !place.name.trim()) - throw Error('A map marker needs a name.'); let xy = place.coordinates; if (!xy) { const match = (s) => s.toLowerCase() === (place.city || place.name).toLowerCase(); @@ -38,7 +44,7 @@ self.onmessage = async ({ data: { spec, urls, language, labels: strings } }) => place.name + '" matched ' + rows.length + - ' records. Specify country, region or sourced coordinates.', + ' records. Specify country or region for cities. For sights and streets use detail:"street", city and the public stop names.', ); xy = rows[0][3]; } @@ -50,7 +56,7 @@ self.onmessage = async ({ data: { spec, urls, language, labels: strings } }) => Math.abs(xy[1]) > 85 ) throw Error('Invalid longitude/latitude for ' + place.name); - return { name: place.name, xy }; + return { name: place.name, xy, osmId: street ? place.osmId : undefined, osmName: place.osmName, osmTag: place.osmTag }; }); const width = 720, height = 480, @@ -78,8 +84,8 @@ self.onmessage = async ({ data: { spec, urls, language, labels: strings } }) => const y0 = projection([midLon, north])[1], y1 = projection([midLon, south])[1]; const scale = Math.min( - (width * (bounds ? 1 : 0.46)) / Math.max(0.013, (dx * Math.PI) / 180), - (height * (bounds ? 1 : 0.41)) / Math.max(0.009, y1 - y0), + (width * (bounds ? 1 : street ? 0.77 : 0.46)) / Math.max(street ? 0.00012 : 0.013, (dx * Math.PI) / 180), + (height * (bounds ? 1 : street ? 0.72 : 0.41)) / Math.max(street ? 0.00008 : 0.009, y1 - y0), ); projection .scale(scale) @@ -147,13 +153,23 @@ self.onmessage = async ({ data: { spec, urls, language, labels: strings } }) => } return paths.join(''); } - const [land, lakes, rivers, borders] = await Promise.all( + let streetSvg = ''; + if (street) { + try { + streetSvg = await streetBasemap(projection, width, height, + languageKey === 'name_zht' ? 'zh-Hant' : languageKey === 'name_zh' ? 'zh-Hans' : language, terrainLabels, deadline); + } catch (error) { + const confirmed = points.filter(point => point.osmId).map(point => point.name + ' ' + JSON.stringify(point.xy)); + throw Error(error.message + (confirmed.length ? ' Confirmed: ' + confirmed.join('; ') : '')); + } + } + const [land, lakes, rivers, borders] = street ? [] : await Promise.all( ['land', 'lakes', 'rivers', 'borders', 'provinces', 'seas'].map(layer), ); - if (!land) throw Error('No land intersects this regional map. Check the extent or use ocean geometry.'); + if (!street && !land) throw Error('No land intersects this regional map. Check the extent or use ocean geometry.'); const labels = []; const markers = points - .map(({ name, xy }) => { + .map(({ name, xy, osmId, osmName, osmTag }, index) => { const [x, y] = projection(xy); if (x < 0 || x > width || y < 0 || y > height) throw Error('The requested bounds exclude ' + name); const w = Math.min(width / 2, Array.from(name).length * 11 + 8); @@ -176,9 +192,9 @@ self.onmessage = async ({ data: { spec, urls, language, labels: strings } }) => ) || candidates[0]; labels.push(label); return ( - '<g data-visual-marker=""><title>' + - esc(name) + - '' + + esc(name) + '' + (osmId && osmName ? '' + esc(osmName) + ' · ' + esc(osmTag) + '' : '') + '' + - '' + + (street ? '' + streetSvg : '' + '' + @@ -249,7 +266,7 @@ self.onmessage = async ({ data: { spec, urls, language, labels: strings } }) => '"/>' + '' + + '" fill="none" stroke-width=".85" opacity=".45"/>') + context.join('') + markers + '').join(''); +} + +export async function streetBasemap(projection, width, height, language, terrainLabels, deadline = Date.now() + 15000) { + const world = projection.scale() * 2 * Math.PI; + // The public Shortbread tiles end at z14. Project their vector geometry at + // the requested scale for close-up walking routes. + const zoom = Math.min(14, Math.floor(Math.log2(world / 256))); + if (zoom < 9) throw Error('Street maps need a city-scale extent. Use the regional atlas for distant places.'); + const count = 2 ** zoom; + const tileSize = world / count; + const [lon, lat] = projection.invert([0, 0]); + const x0 = (lon + 180) / 360 * count; + const y0 = (1 - Math.asinh(Math.tan(lat * Math.PI / 180)) / Math.PI) / 2 * count; + const tiles = []; + for (let y = Math.max(0, Math.floor(y0)); y < Math.min(count, Math.ceil(y0 + height / tileSize)); y++) { + for (let x = Math.floor(x0); x < Math.ceil(x0 + width / tileSize); x++) tiles.push({ x, y }); + } + const output = []; + // Bounded concurrency and deterministic paint/label order, even on slow links. + for (let i = 0; i < tiles.length; i += 4) { + const batch = await Promise.all(tiles.slice(i, i + 4).map(async ({ x, y }) => { + const url = tileOrigin + '/' + zoom + '/' + ((x + count) % count) + '/' + y; + const tile = new VectorTile(new PbfReader(await tileBytes(url, deadline))); + const left = (x - x0) * tileSize, top = (y - y0) * tileSize; + const labels = []; + const paths = tilePaths(tile, tileSize, left, top, width, height, language, labels); + return { svg: '' + paths + '', labels }; + })); + for (const { svg, labels } of batch) { + output.push(svg); + terrainLabels.push(...labels); + } + } + const svg = output.join(''); + if (!svg.includes('data-street-layer=')) throw Error(unavailable); + return svg; +} diff --git a/src/web-ui/src/tools/visualize/streetBasemap.test.js b/src/web-ui/src/tools/visualize/streetBasemap.test.js new file mode 100644 index 0000000000..27595de287 --- /dev/null +++ b/src/web-ui/src/tools/visualize/streetBasemap.test.js @@ -0,0 +1,64 @@ +import { afterEach, expect, it, vi } from 'vitest'; +import { geoMercator } from 'd3-geo'; +import { PbfWriter } from 'pbf'; +import { streetBasemap } from './streetBasemap.js'; + +const projection = geoMercator().center([120, 30]).scale(240000).translate([360, 240]); + +// Minimal MVT containing a real linestring command sequence in the streets layer. +function streetTile() { + const pbf = new PbfWriter(); + pbf.writeMessage(3, (_, layer) => { + layer.writeVarintField(15, 2); + layer.writeStringField(1, 'streets'); + layer.writeMessage(2, (_, feature) => { + feature.writeVarintField(3, 2); + feature.writePackedVarint(4, [9, 0, 0, 10, 8192, 8192]); + }); + }); + return pbf.finish(); +} + +afterEach(() => vi.unstubAllGlobals()); + +it('projects only fixed public tile requests and emits offline street geometry', async () => { + const fetch = vi.fn(async () => new Response(streetTile())); + vi.stubGlobal('fetch', fetch); + const svg = await streetBasemap(projection, 720, 480, 'en', []); + expect(svg).toContain('data-street-layer="streets" d="M'); + expect(svg).not.toContain('https://'); + for (const [url, options] of fetch.mock.calls) { + expect(url).toMatch(/^https:\/\/tiles\.versatiles\.org\/tiles\/osm-vector\/\d+\/\d+\/\d+$/); + expect(options).toMatchObject({ credentials: 'omit', referrerPolicy: 'no-referrer', redirect: 'error' }); + } +}); + +it('uses the highest available source level for a close-up walking route', async () => { + const closeUp = geoMercator().center([120.15, 30.26]).scale(2100000).translate([360, 240]); + const fetch = vi.fn(async url => new Response(url.includes('/14/') ? streetTile() : null, + { status: url.includes('/14/') ? 200 : 404 })); + vi.stubGlobal('fetch', fetch); + + await expect(streetBasemap(closeUp, 720, 480, 'zh', [])).resolves.toContain('data-street-layer="streets"'); + expect(fetch.mock.calls.every(([url]) => url.includes('/14/'))).toBe(true); +}); + +it('rejects an incomplete viewport when one street tile is offline', async () => { + let unavailableTile; + const fetch = vi.fn(async url => { + unavailableTile ??= url; + if (url === unavailableTile) throw new Error('Offline'); + return new Response(streetTile()); + }); + vi.stubGlobal('fetch', fetch); + await expect(streetBasemap(projection, 720, 480, 'en', [])).rejects.toThrow('Street basemap unavailable'); +}); + +it('recovers a map after a transient failure without refetching the whole viewport', async () => { + const fetch = vi.fn(async () => new Response(streetTile())); + fetch.mockResolvedValueOnce(new Response(null, { status: 503 })); + vi.stubGlobal('fetch', fetch); + await expect(streetBasemap(projection, 720, 480, 'en', [])).resolves.toContain('data-street-layer="streets"'); + const urls = fetch.mock.calls.map(([url]) => url); + expect(urls.length).toBe(new Set(urls).size + 1); +}); diff --git a/src/web-ui/src/tools/visualize/streetPlaces.js b/src/web-ui/src/tools/visualize/streetPlaces.js new file mode 100644 index 0000000000..86093a92c8 --- /dev/null +++ b/src/web-ui/src/tools/visualize/streetPlaces.js @@ -0,0 +1,143 @@ +// Resolve authored public stop names on the publishing host. The reader gets +// the selected coordinates in the SVG and never runs these queries. +const normalized = value => typeof value === 'string' ? value.normalize('NFKC').trim().toLowerCase() : ''; +const limit = 16; + +function searchArea([lon, lat]) { + // A city-centre itinerary can include nearby suburbs. Also check returned + // city and country: proximity alone cannot establish a stop's identity. + const dy = 30 / 111.2; + const dx = dy / Math.cos(lat * Math.PI / 180); + return [Math.max(-180, lon - dx), Math.max(-85, lat - dy), Math.min(180, lon + dx), Math.min(85, lat + dy)]; +} + +function candidates(features, city, bounds) { + const [west, south, east, north] = bounds; + const rows = new Map(); + for (const { properties: p, geometry } of features) { + if (!p || geometry?.type !== 'Point' || !Array.isArray(geometry.coordinates)) continue; + const [x, y] = geometry.coordinates; + if (!Number.isFinite(x) || !Number.isFinite(y) || x < west || x > east || y < south || y > north) continue; + if (normalized(p.countrycode) !== normalized(city[0]) || + (p.city && !city[2].some(alias => normalized(alias) === normalized(p.city)))) continue; + if (typeof p.name !== 'string' || !['N', 'W', 'R'].includes(p.osm_type) || !Number.isSafeInteger(p.osm_id)) continue; + const id = p.osm_type + p.osm_id; + rows.set(id, { id, name: p.name, tag: p.osm_key + ':' + p.osm_value, + area: [p.district, p.locality, p.street].filter(Boolean).join(', '), xy: [x, y] }); + } + return [...rows.values()]; +} + +async function search(city, bounds, query, tag, deadline) { + const url = new URL('https://photon.komoot.io/api/'); + const [lon, lat] = city[3]; + url.search = new URLSearchParams({ q: query, lon, lat, bbox: bounds.join(','), limit }); + if (tag) url.searchParams.set('osm_tag', tag); + for (let attempt = 0; ; attempt++) { + const retry = attempt === 0; + const remaining = deadline - Date.now(); + if (remaining <= 0) throw Error('Photon place lookup deadline reached'); + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), Math.min(remaining, retry ? 10000 : 18000)); + let response; + try { + response = await fetch(url, { credentials: 'omit', redirect: 'error', referrerPolicy: 'no-referrer', signal: controller.signal }); + if (response.ok) { + const features = (await response.json())?.features; + if (!Array.isArray(features)) throw new SyntaxError('Expected a GeoJSON feature collection'); + return { rows: candidates(features, city, bounds), capped: features.length >= limit }; + } + } catch (error) { + if (response && error instanceof SyntaxError) throw Error('Photon returned an invalid place response'); + if (retry) continue; + if (controller.signal.aborted) throw Error('Photon request timed out'); + throw Error('Photon network request failed (' + error.message + ')'); + } finally { + clearTimeout(timer); + } + if (retry && [408, 500, 502, 503, 504].includes(response.status) && !response.headers.has('retry-after')) continue; + throw Error('Photon returned HTTP ' + response.status + (response.status === 429 ? ' (rate limited)' : '')); + } +} + +export async function resolveStreetStops(places, spec, gazetteer, deadline = Date.now() + 45000) { + const cities = gazetteer.filter(([country, , names]) => + (!spec.country || normalized(country) === normalized(spec.country)) && + names.some(name => normalized(name) === normalized(spec.city))); + if (cities.length !== 1) throw Error('For named street stops, set city to their city name and country to its ISO code. The city must identify one atlas record, or supply sourced stop coordinates.'); + const city = cities[0]; + const bounds = searchArea(city[3]); + const cache = new Map(); + const resolved = [...places]; + const issues = []; + const pending = places.entries(); + function lookup(query, tag) { + const key = JSON.stringify([normalized(query), tag]); + if (!cache.has(key)) cache.set(key, search(city, bounds, query, tag, deadline)); + return cache.get(key); + } + async function resolveNext() { + for (const [index, place] of pending) { + if (place.coordinates) continue; + const query = place.query ?? place.name; + if (!normalized(query) || query.length > 160 || (place.osmTag && !/^[a-z_]+:[a-z_]+$/.test(place.osmTag))) + throw Error('Invalid street search name or OSM tag for ' + place.name); + let result; + try { + result = await lookup(query, place.osmTag); + // A display name can be better indexed than an expanded query such as + // "station name + subway station". Use these hits as suggestions; the + // exact-match check below still applies to the original query. + if (!result.rows.length && normalized(query) !== normalized(place.name) && place.name.length <= 160) + result = await lookup(place.name, place.osmTag); + // An incorrect tag can hide the intended place entirely. Keep the + // unfiltered suggestions for repair, while enforcing the tag below. + if (!result.rows.length && place.osmTag) result = await lookup(query); + } + catch (error) { + issues[index] = { unavailable: 'Street place lookup unavailable for "' + query + '": ' + error.message }; + continue; + } + const matches = result.rows.filter(row => + (place.osmId ? normalized(row.id) === normalized(place.osmId) : normalized(row.name) === normalized(query)) && + (!place.osmTag || row.tag === place.osmTag)); + if (matches.length === 1 && (place.osmId || !result.capped)) { + const match = matches[0]; + resolved[index] = { ...place, coordinates: match.xy, osmId: match.id, osmName: match.name, osmTag: match.tag }; + continue; + } + issues[index] = { name: place.name, reason: matches.length ? 'ambiguous' : 'no exact match', + candidates: matches.length ? matches : result.rows }; + } + } + await Promise.all(Array.from({ length: Math.min(3, places.length) }, resolveNext)); + if (issues.length) { + const missing = issues.filter(Boolean); + // Write returns at most 500 characters. Give each stop useful candidates. + const messages = missing.map(({ name, reason, candidates, unavailable }) => { + if (unavailable) return unavailable; + // Several segments of a through-road may rank ahead of a same-named + // pedestrian street. Show distinct feature types before duplicates. + const tags = new Set(); + const distinct = candidates.filter(row => { + if (tags.has(row.tag)) return false; + tags.add(row.tag); + return true; + }); + const ranked = [...new Set([...distinct, ...candidates])]; + const hints = ranked.slice(0, 4).map(row => { + const showArea = candidates.some(other => other.id !== row.id && other.name === row.name && other.tag === row.tag); + return [row.id, row.name !== name && row.name, row.tag, showArea && row.area, + JSON.stringify(row.xy)].filter(Boolean).join(' '); + }); + return name + ' (' + reason + ')' + (hints.length ? ': ' + hints.join(' / ') : ''); + }); + const confirmed = resolved.filter((place, index) => !places[index].coordinates && place.coordinates) + .map(place => place.name + ' ' + JSON.stringify(place.coordinates)); + const hint = missing.some(issue => issue.unavailable) + ? 'Find missing public stops in a map source with WebSearch or WebFetch. Resubmit with its WGS84 coordinates [lon,lat]. ' + : 'Use the matching candidate coordinates [lon,lat] or osmId. '; + throw Error(hint + (confirmed.length ? 'Confirmed: ' + confirmed.join('; ') + '. ' : '') + messages.join('; ')); + } + return resolved; +} diff --git a/src/web-ui/src/tools/visualize/streetPlaces.test.js b/src/web-ui/src/tools/visualize/streetPlaces.test.js new file mode 100644 index 0000000000..08cd63a0a0 --- /dev/null +++ b/src/web-ui/src/tools/visualize/streetPlaces.test.js @@ -0,0 +1,137 @@ +import { afterEach, expect, it, vi } from 'vitest'; +import { resolveStreetStops } from './streetPlaces.js'; + +const city = ['CN', 'Zhejiang', ['Hangzhou', '杭州', '杭州市'], [120.168, 30.252]]; +const spec = { city: '杭州', country: 'CN' }; +function feature(id, tag, xy = [120.159, 30.257], cityName = '杭州市') { + const [osm_key, osm_value] = tag.split(':'); + return { geometry: { type: 'Point', coordinates: xy }, properties: { + name: '龙翔桥', osm_type: 'N', osm_id: id, osm_key, osm_value, city: cityName, countrycode: 'cn', + } }; +} +function serve(features) { + const fetch = vi.fn(async () => Response.json({ features })); + vi.stubGlobal('fetch', fetch); + return fetch; +} +afterEach(() => vi.unstubAllGlobals()); + +it('recovers a named stop after a dropped connection or a temporary upstream failure', async () => { + for (const transient of [new TypeError('Load failed'), new Response(null, { status: 503 })]) { + const fetch = serve([feature(17, 'railway:station')]); + if (transient instanceof Error) fetch.mockRejectedValueOnce(transient); + else fetch.mockResolvedValueOnce(transient); + const [stop] = await resolveStreetStops([{ name: '龙翔桥' }], spec, [city]); + expect(stop.osmId).toBe('N17'); + expect(fetch).toHaveBeenCalledTimes(2); + } +}); + +it('preserves the failed query and HTTP or decoding reason without retrying permanent responses', async () => { + for (const [response, reason] of [ + [new Response(null, { status: 429 }), 'Photon returned HTTP 429 (rate limited)'], + [new Response(null, { status: 403 }), 'Photon returned HTTP 403'], + [new Response(null, { status: 503, headers: { 'retry-after': '120' } }), 'Photon returned HTTP 503'], + [new Response('not JSON'), 'Photon returned an invalid place response'], + ]) { + const fetch = vi.fn().mockResolvedValue(response); + vi.stubGlobal('fetch', fetch); + await expect(resolveStreetStops([{ name: '西湖', query: '涌金公园' }], spec, [city])) + .rejects.toThrow('lookup unavailable for "涌金公园": ' + reason); + expect(fetch).toHaveBeenCalledTimes(1); + } +}); + +it('keeps verified locations available for repair when another public place cannot be queried', async () => { + vi.stubGlobal('fetch', async url => url.searchParams.get('q') === '龙翔桥' + ? Response.json({ features: [feature(17, 'railway:station')] }) + : new Response(null, { status: 503 })); + await expect(resolveStreetStops([{ name: '龙翔桥' }, { name: 'Museum' }], spec, [city])) + .rejects.toThrow(/Confirmed: 龙翔桥 \[120\.159,30\.257\].*Museum.*HTTP 503/); +}); + +it('resolves the requested object within its city and retains supplied points and stop order', async () => { + const station = feature(17, 'railway:station'); + const fetch = serve([station, station, feature(18, 'highway:bus_stop')]); + const first = { name: 'First stop', coordinates: [120.15, 30.24] }; + const stop = { name: '龙翔桥地铁站', query: '龙翔桥', osmTag: 'railway:station' }; + const result = await resolveStreetStops([first, stop, stop], spec, [city]); + expect(result).toEqual([first, ...[stop, stop].map(s => ({ ...s, coordinates: [120.159, 30.257], osmId: 'N17', osmName: '龙翔桥' }))]); + expect(fetch).toHaveBeenCalledTimes(1); + const [[url, options]] = fetch.mock.calls; + expect(url.origin).toBe('https://photon.komoot.io'); + expect(url.searchParams.get('q')).toBe('龙翔桥'); + expect(url.searchParams.get('osm_tag')).toBe('railway:station'); + expect(url.searchParams.has('bbox')).toBe(true); + expect(options).toMatchObject({ credentials: 'omit', redirect: 'error', referrerPolicy: 'no-referrer' }); +}); + +it('reports ambiguous and missing stops together and accepts an explicitly selected candidate', async () => { + serve([feature(17, 'railway:station'), feature(18, 'highway:bus_stop')]); + await expect(resolveStreetStops([{ name: '龙翔桥' }, { name: 'Missing' }], spec, [city])) + .rejects.toThrow(/龙翔桥.*ambiguous.*N17.*N18.*Missing.*no exact match/); + const [selected] = await resolveStreetStops([{ name: 'Transit stop', query: 'Local name alias', osmId: 'n18' }], spec, [city]); + expect(selected.osmId).toBe('N18'); +}); + +it('rejects nearby homonyms from another city, distant suburbs and a truncated result set', async () => { + serve([feature(1, 'railway:station', [120.159, 30.257], 'Another City'), feature(2, 'railway:station', [119.77, 30.26])]); + await expect(resolveStreetStops([{ name: '龙翔桥' }], spec, [city])).rejects.toThrow('no exact match'); + serve(Array(16).fill(feature(1, 'railway:station'))); + await expect(resolveStreetStops([{ name: '龙翔桥' }], spec, [city])).rejects.toThrow('ambiguous'); +}); + +it('only reports confirmed coordinates when an authored OSM id is missing', async () => { + serve([feature(17, 'railway:station')]); + const places = [{ name: 'Verified', query: '龙翔桥', osmId: 'N17' }, { name: 'Unresolved', osmId: 'N99' }]; + await expect(resolveStreetStops(places, spec, [city])) + .rejects.toThrow(/Confirmed: Verified \[120\.159,30\.257\]\. Unresolved \(no exact match\)/); +}); + +it('suggests a station by its display name when the expanded query misses, without guessing the stop', async () => { + const fetch = vi.fn(async url => Response.json({ + features: url.searchParams.get('q') === '龙翔桥' ? [feature(17, 'railway:station')] : [], + })); + vi.stubGlobal('fetch', fetch); + const stop = { name: '龙翔桥', query: '龙翔桥 地铁站', osmTag: 'railway:station' }; + await expect(resolveStreetStops([stop], spec, [city])).rejects.toThrow(/no exact match.*N17.*railway:station/); + const [confirmed] = await resolveStreetStops([{ ...stop, osmId: 'n17' }], spec, [city]); + expect(confirmed.osmId).toBe('N17'); +}); + +it('includes a pedestrian segment in repair hints when the first road segments all have the same type', async () => { + serve([1, 2, 3, 4, 5].map(id => feature(id, 'highway:tertiary')).concat(feature(6, 'highway:pedestrian'))); + await expect(resolveStreetStops([{ name: '龙翔桥' }], spec, [city])).rejects.toThrow(/N1.*N6.*highway:pedestrian/); +}); + +it('requires an identified city before going online and never substitutes its center when offline', async () => { + const fetch = vi.fn().mockRejectedValue(Error('Offline')); + vi.stubGlobal('fetch', fetch); + await expect(resolveStreetStops([{ name: '龙翔桥' }], {}, [city])).rejects.toThrow('set city'); + expect(fetch).not.toHaveBeenCalled(); + await expect(resolveStreetStops([{ name: '龙翔桥' }], spec, [city])).rejects.toThrow('Photon network request failed (Offline)'); + expect(fetch).toHaveBeenCalledTimes(2); +}); + +it('returns confirmed coordinates and ambiguous candidates when another lookup exhausts the shared deadline', async () => { + vi.useFakeTimers(); + try { + const fetch = vi.fn(async (url, options) => { + const q = url.searchParams.get('q'); + if (q === 'Slow') return new Promise((_, reject) => { + options.signal.addEventListener('abort', () => reject(Error('aborted')), { once: true }); + }); + return Response.json({ features: q === '龙翔桥' ? [feature(17, 'railway:station')] : + [feature(81, 'tourism:attraction'), feature(82, 'tourism:museum')] }); + }); + vi.stubGlobal('fetch', fetch); + const result = resolveStreetStops([{ name: '龙翔桥' }, { name: 'Ambiguous' }, { name: 'Slow' }], spec, [city], Date.now() + 11000) + .catch(error => error.message); + await vi.advanceTimersByTimeAsync(11000); + const message = await result; + expect(message).toContain('Confirmed: 龙翔桥 [120.159,30.257]'); + expect(message).toContain('N81'); + expect(message).toContain('Slow'); + expect(fetch).toHaveBeenCalledTimes(4); + } finally { vi.useRealTimers(); } +}); diff --git a/src/web-ui/tests/e2e/visualize.tsx b/src/web-ui/tests/e2e/visualize.tsx index b9f9e5d0c6..6cad571e23 100644 --- a/src/web-ui/tests/e2e/visualize.tsx +++ b/src/web-ui/tests/e2e/visualize.tsx @@ -288,7 +288,12 @@ const session = { { id: 'turn', sessionId: 'visual-test', - userMessage: { id: 'user', content: 'Compare latency and load', timestamp: 0 }, + userMessage: { id: 'user', content: params.get('deferred-request') ?? 'Compare latency and load', timestamp: 0 }, + ...(params.has('deferred-request') && { publishedFigures: Object.fromEntries(items.map(item => { + const uri = item.toolResult!.result.file_path; + const source = sessionStorage.getItem('visualize-test-fragment')!; + return [uri, { source, contentBytes: new TextEncoder().encode(source).length, inspectionDeferred: true }]; + })) }), status: 'completed', startTime: 0, modelRounds: [ diff --git a/src/web-ui/vite.config.ts b/src/web-ui/vite.config.ts index 3108689737..c62c34b6cd 100644 --- a/src/web-ui/vite.config.ts +++ b/src/web-ui/vite.config.ts @@ -19,6 +19,7 @@ import { resolveDevServerPorts } from '../../scripts/dev-server-ports.mjs'; const host = process.env.TAURI_DEV_HOST; const requireFromWebUi = createRequire(path.join(__dirname, 'package.json')); const { port: devPort, hmrPort } = resolveDevServerPorts(); +const { visualizeDevGuide } = requireFromWebUi('../../scripts/visualize-dev-kit.cjs'); const designSystemUiSourceDirectory = path.resolve( __dirname, '../../design-system/packages/ui/src', @@ -161,6 +162,9 @@ export default defineConfig(({ mode, command }) => { } return { + define: { + 'import.meta.env.OPENBITFUN_VISUALIZE_DEV_KIT': command === 'serve' && !!visualizeDevGuide(path.resolve(__dirname, '../..')), + }, // Independent dev ports must not replace each other's optimized deps. // In-process generators use their own config and cache directory. cacheDir: path.resolve(__dirname, 'node_modules/.vite', `dev-${devPort}`),