Repository navigation
Extractor helpers for Node: request and result, with nothing installed - #125
Open
agorischek wants to merge 8 commits into
Open
agorischek wants to merge 8 commits into
agorischek wants to merge 8 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Restrictive umasks can leave cached helpers inaccessible to rootless containers, while the Node 22.14 fallback lacks required worker permission, both blocking extractor startup.
2 open issues
What changed in this PR
Adds zero-install Node extractor helpers, bringing JavaScript and TypeScript extractors closer to the existing Python API while keeping the JSON protocol underneath.
Changes:
- Adds lazy request accessors, synchronous request reading, and shared Python/Node validation fixtures.
- Embeds and preloads the Node helper through the CLI, including container integration and cache permission changes.
- Adopts the helpers in starter templates, examples, self-projections, and documentation.
| File | Description |
|---|---|
tools/scripts/check-boundaries/layers.mjs |
Allows the CLI’s build-time SDK dependency. |
packages/sdk/test/extractor.test.ts |
Updates validation expectations and keyed-ID coverage. |
packages/sdk/src/extractor/result.ts |
Aligns validation messages and optional-field handling. |
packages/sdk/src/extractor/request.ts |
Adds lazy request accessors and synchronous reading. |
packages/sdk/src/extractor/messages.ts |
Centralizes JSON-oriented diagnostic wording. |
packages/sdk/src/extractor.ts |
Exports the new request API and schema type. |
packages/sdk/scripts/extractor-module.mjs |
Generates and checks the standalone helper bundle. |
packages/sdk/README.md |
Documents runner-provided extractor helpers. |
packages/sdk/package.json |
Adds bundle generation and freshness checks. |
packages/schemas/fixtures/extractor-helpers.json |
Defines shared Python/Node conformance cases. |
packages/cli/test/extractor-helpers/driver.py |
Runs conformance cases through Python. |
packages/cli/test/extractor-helpers/driver.mjs |
Runs conformance cases through Node. |
packages/cli/test/extractor-helpers.test.mjs |
Compares both helpers against shared expectations. |
packages/cli/templates/SKILL.md |
Updates authoring guidance and manual execution commands. |
packages/cli/templates/extract.mjs |
Uses helpers in the Node starter. |
packages/cli/src/runtime/extract.rs |
Preloads helpers for custom bare-Node commands. |
packages/cli/src/runtime/container.rs |
Mounts and preloads helpers in containers. |
packages/cli/src/node.rs |
Implements helper unpacking, argument injection, and tests. |
packages/cli/src/main.rs |
Adds the hidden preload-path command. |
packages/cli/src/lib.rs |
Exposes the Node integration module. |
packages/cli/src/cache.rs |
Adjusts helper permissions for container readability. |
packages/cli/python/tests/test_projections.py |
Updates diagnostic assertions. |
packages/cli/python/projections/__init__.py |
Aligns Python diagnostics with Node. |
packages/cli/package.json |
Declares the SDK development dependency. |
packages/cli/node/register.cjs |
Registers resolution hooks for the helper import. |
packages/cli/node/hooks.mjs |
Supplies the asynchronous resolution-hook fallback. |
packages/cli/node/extractor.mjs |
Adds the generated standalone helper bundle. |
package-lock.json |
Records the CLI’s SDK dependency. |
examples/extractors/typescript-common/basics.mjs |
Uses shared request and result helpers. |
examples/extractors/typescript-common/architecture.mjs |
Leaves protocol-version insertion to the helper. |
examples/extractors/test-support.mjs |
Preloads helpers when running sample extractors. |
examples/extractors/README.md |
Explains zero-install helper availability. |
eslint.config.js |
Excludes the generated bundle from linting. |
docs/runner/environments.md |
Documents the container helper mount. |
docs/guides/how-it-works.md |
Uses lazy requests in the generation example. |
docs/guides/first-extractor.md |
Adds a Node helper example and debugging command. |
docs/contracts/extractors.md |
Documents preload behavior and import resolution. |
docs/agents/create-projection.md |
Updates generated authoring guidance. |
.github/projections/self-topology/extract.mjs |
Adopts request and result helpers. |
.github/projections/self-authoring-api/extract.mjs |
Adopts request and result helpers. |
.github/projections/agent-tools/extract.mjs |
Adopts request and result helpers. |
.considered.yaml |
Excludes the generated bundle from measurement. |
Review effort: Balanced
`@githubnext/projections/extractor` gains the Python module's request API:
`request` (read from stdin the first time it is used, with `root`, `project`,
`repositoryRoot`, `checkout`, `id`, `definition`, `inputs`, `upstream` and
`trusted`), `readRequest()` with the same checks and messages, and `result()`
and `extraction()` worded as the Python module's are.
The CLI embeds that entry, bundled (packages/cli/node/extractor.mjs, generated
by packages/sdk/scripts/extractor-module.mjs), unpacks it into its cache like
the Python module, and preloads a resolve hook into a custom extractor whose
program is the bare name `node` (`node --require register.cjs`), so
`import {request, result} from '@githubnext/projections/extractor'` works in
.mjs, .js and .ts extractors with nothing installed. `projections
__node-require` prints the preload, to run one by hand.
Shared fixtures (packages/schemas/fixtures/extractor-helpers.json) run through
both helpers keep them in step. The scaffold template, the self-projections,
the TypeScript samples and the docs use the helpers.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Under Node's permission model the preload also allows reading the module's directory (its real path on Unix). Unpacked helper sets are readable by others, so a rootless container engine's mapped user can read them. A keyed item's own undefined id no longer replaces its key. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…des, permission fallback Register packages/cli/node/extractor.mjs in the generated-file registry. Helper sets get mode 755/644 explicitly, whatever the umask, and verification replaces a set without them. Before Node 22.15, the off-thread hook is skipped under the permission model unless workers are allowed, rather than failing the extractor. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
agorischek
force-pushed
the
js-extractor-api
branch
from
November 21, 2026 01:41
e23ad3b to
83b6aaf
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

JavaScript and TypeScript extractors hand-rolled the extractor protocol (
JSON.parse(readFileSync(0, 'utf8')), thenprocess.stdout.write(JSON.stringify(...))), while Python extractors getfrom projections import request, resultwith nothing installed. This brings the same API to Node, with the same zero-install property.API
@githubnext/projections/extractor(packages/sdk) already hadresult()/extraction(). It gains:request: the request on stdin, read the first time it is used, so importingresultalone reads nothing (as Python's module__getattr__does). Accessors match Python's in camelCase:root,project,repositoryRoot,checkout,id,definition,inputs,upstream,trusted, plus the protocol's own fields (repository,projection,capabilities,generations);JSON.stringify(request)writes the request as it came.readRequest(fd = 0): now synchronous (await readRequest()still works) with Python's checks: a TTY on stdin, not JSON, not an object, anotherprotocolVersion.Request, typed from the TypeBoxExtractorRequest,DefinitionandExtractiontypes in packages/schemas;ExtractorRequestis now that full type.result()'s andread_request()'s messages in both languages are reworded in JSON's terms (entities[0] is an array, not an object,The entity "a" needs label, a string), so they can be identical.Delivery: a resolve hook the runner preloads into a bare
nodeES modules don't search
NODE_PATH, so the Python trick (prepend a directory to a search path) has no equivalent. Options weighed:import … fromnode_modulesawait import(process.env.X))NODE_PATH+createRequireinit./projections.mjs).d.mts) to keep current; hand-written extractors and existing ones don't get itNODE_OPTIONS=--import …--requireon a barenode's argvThe CLI embeds three files (packages/cli/node) and unpacks them into the verified per-user cache like the Python module. For a custom extractor whose program is the bare name
node(the case the runner already special-cases forpython3), it inserts--require <cache>/register.cjsafternode.register.cjsregisters a resolve hook (module.registerHooks, ormodule.registerwithhooks.mjsbefore Node 22.15) for exactly one specifier,@githubnext/projections/extractor, pointing at the cached bundle; every other import resolves as usual. So:import {request, result} from '@githubnext/projections/extractor'works with nothing installed, in.mjs,.jsand.ts(type stripping), on macOS, Linux and Windows (a--requirepath, no URL quoting), and under--isolation container(mounted read-only at/projections/node);@githubnext/projectionsadds its types and makes the import work for other launchers (npx tsx), without changing the specifier;PYTHONPATH, the runner's copy wins over an installed one, so the module always speaks the runner's protocol;--permission) also gets--allow-fs-read=<module directory>, so the preload never stops it; on Unix the directory is its real path, so Node resolves no symlinks on the way.The unpacked helper sets are now readable by others: directories 755, files 644, set explicitly whatever the umask. Verification requires those modes. The cache base and
helpers/stay 700, and verification still rejects group or other write bits. Under rootless Podman or Docker the container's user maps away from the owner, so a 0600 helper mounted into the container was unreadable: harmless forPYTHONPATH, fatal for--require. This fixes both.projections __node-requireprints the preload, to run an extractor by hand:node --require "$(projections __node-require)" extract.mjs < request.json.The bundle is
packages/sdk/src/extractor.tsbuilt bypackages/sdk/scripts/extractor-module.mjsintopackages/cli/node/extractor.mjs(committed, since the CLI embeds it; registered intools/cli/src/lib/generated.ts, and the SDK'stestscript runs it with--check). Under the permission model before Node 22.15, the off-threadmodule.registerfallback needs--allow-worker; without it the preload registers nothing rather than failing the extractor. The CLI now declares@githubnext/projectionsas a development dependency, and the layer table allows it.Shared behaviour
packages/schemas/fixtures/extractor-helpers.jsonholds 28 cases (accessors at the root and in a nested project, omitted fields,git-objectsmode, bad requests, sorting in UTF-16 order, optional fields, and every item error).packages/cli/test/extractor-helpers.test.mjsruns each through the Python module onPYTHONPATHand the Node module through the runner's preload, and requires both to give the expected values, extraction or message.Adopted
projections initNode starter (templates/extract.mjs), now parallel to the Python one.agent-tools,self-authoring-apiandself-topology: extractions verified identical before and after on this tree.examples/extractors:typescript-commonusesreadRequest/result; the tests run the samples with the runner's preload. (examples/polyglotstays hand-rolled on purpose: it shows the raw protocol in both languages.)first-extractor, the extractor contract (process environment), container isolation,how-it-works, the SDK README, and the authoring skill (templates/SKILL.md, anddocs/agents/create-projection.mdfrom it). Edits are small, to rebase easily over Docs: humanize hand-authored documentation #97.Verification (local)
tools check --fast; the conformance test (28 cases, both helpers); the Python unittest; the SDK extractor tests;cargo testfornode::and the authoring test that scaffolds the Node starter and runs it throughprojections show; the generator's--check. CI runs the rest.