Skip to content

Extractor helpers for Node: request and result, with nothing installed - #125

Open
agorischek wants to merge 8 commits into
mainfrom
js-extractor-api
Open

agorischek wants to merge 8 commits into
mainfrom
js-extractor-api

Conversation

@agorischek

@agorischek agorischek commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

JavaScript and TypeScript extractors hand-rolled the extractor protocol (JSON.parse(readFileSync(0, 'utf8')), then process.stdout.write(JSON.stringify(...))), while Python extractors get from projections import request, result with nothing installed. This brings the same API to Node, with the same zero-install property.

import {request, result} from '@githubnext/projections/extractor'

result({entities: {[`project:${request.project}`]: {kind: 'project', label: request.root}}})

API

@githubnext/projections/extractor (packages/sdk) already had result()/extraction(). It gains:

  • request: the request on stdin, read the first time it is used, so importing result alone reads nothing (as Python's module __getattr__ does). Accessors match Python's in camelCase: root, project, repositoryRoot, checkout, id, definition, inputs, upstream, trusted, plus the protocol's own fields (repository, projection, capabilities, generations); JSON.stringify(request) writes the request as it came.
  • readRequest(fd = 0): now synchronous (await readRequest() still works) with Python's checks: a TTY on stdin, not JSON, not an object, another protocolVersion.
  • Request, typed from the TypeBox ExtractorRequest, Definition and Extraction types in packages/schemas; ExtractorRequest is now that full type.

result()'s and read_request()'s messages in both languages are reworded in JSON's terms (entities[0] is an array, not an object, The entity "a" needs label, a string), so they can be identical.

Delivery: a resolve hook the runner preloads into a bare node

ES modules don't search NODE_PATH, so the Python trick (prepend a directory to a search path) has no equivalent. Options weighed:

Option Static import … from Zero install Notes
Subpath export only yes no needs the package installed; materialized revisions have no node_modules
Env var with the module's path no (await import(process.env.X)) yes untyped, and Windows needs a file URL
NODE_PATH + createRequire no yes boilerplate in every ESM extractor
Vendored copy scaffolded by init yes (./projections.mjs) yes a copy per projection (plus a .d.mts) to keep current; hand-written extractors and existing ones don't get it
NODE_OPTIONS=--import … yes yes leaks into every Node process the extractor spawns, including tools run by Python extractors
--require on a bare node's argv yes yes chosen

The CLI embeds three files (packages/cli/node) and unpacks them into the verified per-user cache like the Python module. For a custom extractor whose program is the bare name node (the case the runner already special-cases for python3), it inserts --require <cache>/register.cjs after node. register.cjs registers a resolve hook (module.registerHooks, or module.register with hooks.mjs before Node 22.15) for exactly one specifier, @githubnext/projections/extractor, pointing at the cached bundle; every other import resolves as usual. So:

  • the same import {request, result} from '@githubnext/projections/extractor' works with nothing installed, in .mjs, .js and .ts (type stripping), on macOS, Linux and Windows (a --require path, no URL quoting), and under --isolation container (mounted read-only at /projections/node);
  • installing @githubnext/projections adds its types and makes the import work for other launchers (npx tsx), without changing the specifier;
  • like the Python module first on PYTHONPATH, the runner's copy wins over an installed one, so the module always speaks the runner's protocol;
  • it is argv only: tools the extractor spawns are unaffected, and an extractor that doesn't import it pays one tiny preload;
  • an extractor that sandboxes itself with Node's permission model (--permission) also gets --allow-fs-read=<module directory>, so the preload never stops it; on Unix the directory is its real path, so Node resolves no symlinks on the way.

The unpacked helper sets are now readable by others: directories 755, files 644, set explicitly whatever the umask. Verification requires those modes. The cache base and helpers/ stay 700, and verification still rejects group or other write bits. Under rootless Podman or Docker the container's user maps away from the owner, so a 0600 helper mounted into the container was unreadable: harmless for PYTHONPATH, fatal for --require. This fixes both.

projections __node-require prints the preload, to run an extractor by hand: node --require "$(projections __node-require)" extract.mjs < request.json.

The bundle is packages/sdk/src/extractor.ts built by packages/sdk/scripts/extractor-module.mjs into packages/cli/node/extractor.mjs (committed, since the CLI embeds it; registered in tools/cli/src/lib/generated.ts, and the SDK's test script runs it with --check). Under the permission model before Node 22.15, the off-thread module.register fallback needs --allow-worker; without it the preload registers nothing rather than failing the extractor. The CLI now declares @githubnext/projections as a development dependency, and the layer table allows it.

Shared behaviour

packages/schemas/fixtures/extractor-helpers.json holds 28 cases (accessors at the root and in a nested project, omitted fields, git-objects mode, bad requests, sorting in UTF-16 order, optional fields, and every item error). packages/cli/test/extractor-helpers.test.mjs runs each through the Python module on PYTHONPATH and the Node module through the runner's preload, and requires both to give the expected values, extraction or message.

Adopted

  • projections init Node starter (templates/extract.mjs), now parallel to the Python one.
  • The self-projections agent-tools, self-authoring-api and self-topology: extractions verified identical before and after on this tree.
  • examples/extractors: typescript-common uses readRequest/result; the tests run the samples with the runner's preload. (examples/polyglot stays hand-rolled on purpose: it shows the raw protocol in both languages.)
  • Docs: first-extractor, the extractor contract (process environment), container isolation, how-it-works, the SDK README, and the authoring skill (templates/SKILL.md, and docs/agents/create-projection.md from it). Edits are small, to rebase easily over Docs: humanize hand-authored documentation #97.

Verification (local)

tools check --fast; the conformance test (28 cases, both helpers); the Python unittest; the SDK extractor tests; cargo test for node:: and the authoring test that scaffolds the Node starter and runs it through projections show; the generator's --check. CI runs the rest.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Restrictive umasks can leave cached helpers inaccessible to rootless containers, while the Node 22.14 fallback lacks required worker permission, both blocking extractor startup.

2 open issues
What changed in this PR

Adds zero-install Node extractor helpers, bringing JavaScript and TypeScript extractors closer to the existing Python API while keeping the JSON protocol underneath.

Changes:

  • Adds lazy request accessors, synchronous request reading, and shared Python/Node validation fixtures.
  • Embeds and preloads the Node helper through the CLI, including container integration and cache permission changes.
  • Adopts the helpers in starter templates, examples, self-projections, and documentation.
File Description
tools/​scripts/​check-boundaries/​layers.mjs Allows the CLI’s build-time SDK dependency.
packages/​sdk/​test/​extractor.test.ts Updates validation expectations and keyed-ID coverage.
packages/​sdk/​src/​extractor/​result.ts Aligns validation messages and optional-field handling.
packages/​sdk/​src/​extractor/​request.ts Adds lazy request accessors and synchronous reading.
packages/​sdk/​src/​extractor/​messages.ts Centralizes JSON-oriented diagnostic wording.
packages/​sdk/​src/​extractor.ts Exports the new request API and schema type.
packages/​sdk/​scripts/​extractor-module.mjs Generates and checks the standalone helper bundle.
packages/​sdk/​README.md Documents runner-provided extractor helpers.
packages/​sdk/​package.json Adds bundle generation and freshness checks.
packages/​schemas/​fixtures/​extractor-helpers.json Defines shared Python/Node conformance cases.
packages/​cli/​test/​extractor-helpers/​driver.py Runs conformance cases through Python.
packages/​cli/​test/​extractor-helpers/​driver.mjs Runs conformance cases through Node.
packages/​cli/​test/​extractor-helpers.test.mjs Compares both helpers against shared expectations.
packages/​cli/​templates/​SKILL.md Updates authoring guidance and manual execution commands.
packages/​cli/​templates/​extract.mjs Uses helpers in the Node starter.
packages/​cli/​src/​runtime/​extract.rs Preloads helpers for custom bare-Node commands.
packages/​cli/​src/​runtime/​container.rs Mounts and preloads helpers in containers.
packages/​cli/​src/​node.rs Implements helper unpacking, argument injection, and tests.
packages/​cli/​src/​main.rs Adds the hidden preload-path command.
packages/​cli/​src/​lib.rs Exposes the Node integration module.
packages/​cli/​src/​cache.rs Adjusts helper permissions for container readability.
packages/​cli/​python/​tests/​test_projections.py Updates diagnostic assertions.
packages/​cli/​python/​projections/​__init__.py Aligns Python diagnostics with Node.
packages/​cli/​package.json Declares the SDK development dependency.
packages/​cli/​node/​register.cjs Registers resolution hooks for the helper import.
packages/​cli/​node/​hooks.mjs Supplies the asynchronous resolution-hook fallback.
packages/​cli/​node/​extractor.mjs Adds the generated standalone helper bundle.
package-lock.json Records the CLI’s SDK dependency.
examples/​extractors/​typescript-common/​basics.mjs Uses shared request and result helpers.
examples/​extractors/​typescript-common/​architecture.mjs Leaves protocol-version insertion to the helper.
examples/​extractors/​test-support.mjs Preloads helpers when running sample extractors.
examples/​extractors/​README.md Explains zero-install helper availability.
eslint.config.js Excludes the generated bundle from linting.
docs/​runner/​environments.md Documents the container helper mount.
docs/​guides/​how-it-works.md Uses lazy requests in the generation example.
docs/​guides/​first-extractor.md Adds a Node helper example and debugging command.
docs/​contracts/​extractors.md Documents preload behavior and import resolution.
docs/​agents/​create-projection.md Updates generated authoring guidance.
.github/​projections/​self-topology/​extract.mjs Adopts request and result helpers.
.github/​projections/​self-authoring-api/​extract.mjs Adopts request and result helpers.
.github/​projections/​agent-tools/​extract.mjs Adopts request and result helpers.
.considered.yaml Excludes the generated bundle from measurement.

Review effort: Balanced

Comment thread packages/cli/node/register.cjs Resolved
Comment thread packages/cli/src/cache.rs Resolved
agorischek and others added 3 commits November 20, 2026 17:40
`@githubnext/projections/extractor` gains the Python module's request API:
`request` (read from stdin the first time it is used, with `root`, `project`,
`repositoryRoot`, `checkout`, `id`, `definition`, `inputs`, `upstream` and
`trusted`), `readRequest()` with the same checks and messages, and `result()`
and `extraction()` worded as the Python module's are.

The CLI embeds that entry, bundled (packages/cli/node/extractor.mjs, generated
by packages/sdk/scripts/extractor-module.mjs), unpacks it into its cache like
the Python module, and preloads a resolve hook into a custom extractor whose
program is the bare name `node` (`node --require register.cjs`), so
`import {request, result} from '@githubnext/projections/extractor'` works in
.mjs, .js and .ts extractors with nothing installed. `projections
__node-require` prints the preload, to run one by hand.

Shared fixtures (packages/schemas/fixtures/extractor-helpers.json) run through
both helpers keep them in step. The scaffold template, the self-projections,
the TypeScript samples and the docs use the helpers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Under Node's permission model the preload also allows reading the module's
directory (its real path on Unix). Unpacked helper sets are readable by others,
so a rootless container engine's mapped user can read them. A keyed item's own
undefined id no longer replaces its key.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…des, permission fallback

Register packages/cli/node/extractor.mjs in the generated-file registry. Helper
sets get mode 755/644 explicitly, whatever the umask, and verification replaces a
set without them. Before Node 22.15, the off-thread hook is skipped under the
permission model unless workers are allowed, rather than failing the extractor.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants