Skip to content

feat(appkit): add template and docs for the database development flow - #675

Draft
ditadi wants to merge 30 commits into
stack/database-dev-flow/06-db-verifyfrom
stack/database-dev-flow/07-apps-init
Draft

ditadi wants to merge 30 commits into
stack/database-dev-flow/06-db-verifyfrom
stack/database-dev-flow/07-apps-init

Conversation

@ditadi

@ditadi ditadi commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

🥞 Stacked PR (generated by Stack)

Use this link to review incremental changes.


Stacked on #674. The design is #677.

What did you change, and why?

The commands from #654 to #674 need template support and docs before the plugin can be offered in databricks apps init. This PR adds what the template renders for an app that selects the plugin, and documents the development flow.

It does not offer the plugin. The manifest keeps hidden: true, and beta.ts still exports database by hand because the generated beta barrel skips hidden plugins. template/appkit.plugins.json does not list the plugin, and appkit db stays hidden from appkit --help. The earlier revision of this PR removed hidden: true, and this revision puts it back. Offering the plugin is a later, separate change.

For an app that selects the plugin, the template now renders these files:

  • app.yaml sets APPKIT_DB_MIGRATE_ON_START=true, so the deployed app applies the committed migrations on start.
  • config/database/schema.ts says to edit tables there and run npx appkit db migrate.
  • The README gets a development-workflow section, and a verify --deployed step after the production deploy.

For every app, server/server.ts exits 1 when createApp fails, so a failed migration stops the app instead of leaving it up without routes. _gitignore skips config/database/*.bak, the previous schema.ts that introspect keeps. drizzle-kit is not a template dependency, because appkit db init (#672) installs it.

This PR also makes these changes:

  • The manifest's project field sets bundleIgnore, so the project goes to .env and gets no databricks.yml variable.
  • The owner-mismatch error covers an app recreated with a new service principal. It names databricks postgres delete-role <its role> --reassign-owned-to <new role>, run by someone with CAN MANAGE on the Lakebase project.
  • appkit db verify warns when .env points at a branch appkit db init did not set up for you. While a databricks.yml reference is unresolved, the warning names it and leaves out the init advice.
  • database.md gets a Development workflow section under a note that appkit db is hidden and beta. It covers branch setup, the commands, the deploy guard, the history lock, changing tables in use, migrations on deploy, existing apps, rolling back, schema ownership, verify --deployed and CI. It warns that a new branch copies the default branch's rows, production data included.
  • The earlier revision also told every Lakebase app, in the GA lakebase.md, to run the beta npx appkit db init --schema none. This revision drops that change, so lakebase.md matches main.

Review round, 2026-11-09:

  • The rollback section promises only that commits whose code works with the current schema start again; a contract migration ends rollbacks to code that needs what it removed.
  • "Set up your branch", "Existing apps", the branch caution, the verify section and the template README describe the new init default, which schema introspect reads, and what verify --deployed checks.

Fix round for fjakobs's review of #677, 2026-11-15:

  • The owner error names the two-apps case first and gives delete-role --reassign-owned-to with a check of what the role owns, never ALTER ... OWNER TO (L503, L260).
  • Docs and the template README describe migrate on start as the PGUSER identity rule (L307), and what a deploy waits for per compute type with the 300 s health check (L350). app.yaml.tmpl gains a comment on that.
  • They also describe a failure runbook that repairs rows with a DATABRICKS_SUPERUSER role and never edits an applied migration (L424), long backfills (L327), and a stuck-start runbook that names the role (pg_cancel_backend item).
  • Two apps on one database are unsupported (L260). The docs also cover the full ownership inventory before delete-role, the fork-copy rule, the fork source and the drizzle-kit install.

How do you know it works?

  • verify.test.ts and the CLI's verify.test.ts cover the warning on the default branch and on a dev branch, each with an unresolved reference. ownership.test.ts asserts the delete-role hint.
  • The template smoke passed with pnpm on a fresh store and with npm. The app built, passed its health check and served the client. The smoke strips the plugin conditionals and uses a minimal server, so it does not render a database app. An earlier revision rendered app.yaml.tmpl with Go text/template for a database app and two others, and only the database app got the flag. That block is unchanged since.
  • The full vitest suite passed against a local PostgreSQL 14. A clean build, the size gate, the docs build and typecheck passed. The packed tarballs export the same root and ./beta names as main, and ./beta/database-tooling is the only new subpath. appkit --help does not list db.
  • This revision was not run against a real Lakebase workspace.

Review round, 2026-11-09: each fix has a failing-then-passing test in its PR, and every layer's appkit and shared tests ran against Postgres 14 after the sync. At the top of the stack, pnpm build, pnpm test (6502 passed, 1 skipped), pnpm -r typecheck, pnpm check, pnpm knip and pnpm docs:build pass.

Fix round for fjakobs's review, 2026-11-15: each fix has a failing-then-passing test in its PR, and every layer's database, plugin and CLI tests ran against Postgres 17 after the sync. A throwaway Lakebase fork confirmed that the developer role can write the service principal's rows but cannot alter its tables. At the top of the stack, pnpm test (6602 passed, 1 skipped), pnpm -r typecheck, pnpm build and pnpm docs:build pass, and oxlint and oxfmt are clean on every changed file.

The DatabasePlugin is no longer hidden from `databricks apps init`. The
template ships config/database/schema.ts in the app schema, and app.yaml
sets APPKIT_DB_MIGRATE_ON_START=true for apps that select the plugin, so
the deployed app applies its committed migrations on start. The CLI
manifest renders only valueFrom entries, so the literal lives in
app.yaml.tmpl.

drizzle-kit stays out of the template and its lockfiles: `appkit db init`
installs the pinned version when the app lacks it, so apps without the
plugin do not get it.

`appkit db verify` warns when .env points at a branch `appkit db init` did
not set up for you, such as the deploy branch the scaffolded .env names
until `db init` runs. The template's server exits 1 when startup fails, so
a failed migration stops the app instead of leaving it up without its
routes. The owner-mismatch error names the recovery for an app recreated
with a new service principal: `databricks postgres delete-role
--reassign-owned-to`.

The plugin docs and the template README describe the flow end to end:
`appkit db init` once after scaffolding, `appkit db migrate` while you
develop, migrations on deploy with their per-migration transactions and
lock limits, two-step changes for tables in use, rollback, existing apps,
`appkit db verify --deployed -t <target>` in CI, and one owner per branch.
The Lakebase docs start local development on a dev branch.

The template's .gitignore skips config/database/*.bak, the copy of
schema.ts that `appkit db introspect` keeps.

Signed-off-by: ditadi <victordperd@gmail.com>
@ditadi
ditadi force-pushed the stack/database-dev-flow/07-apps-init branch from 7db9fa1 to 81e2e8f Compare October 20, 2026 12:22
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 41109406247 -R databricks/appkit -n appkit-template-0.89.0-pr.7de3087-stack-database-dev-flow-07-apps-init-675 -D appkit-pr-675 \
  && unzip -o "appkit-pr-675/appkit-template-0.89.0-pr.7de3087-stack-database-dev-flow-07-apps-init-675.zip" -d "appkit-pr-675" \
  && databricks apps init --template "appkit-pr-675"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

@databricks/appkit

npm tarball (packed): 1.4 MB (+19 KB) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 1.4 MB (+21 KB) 522 KB (+6.6 KB)
Type declarations 482 KB (+4.0 KB) 176 KB (+1.4 KB)
Source maps 2.9 MB (+42 KB) 970 KB (+12 KB)
Other 11 KB 3.7 KB
Total 4.8 MB (+67 KB) 1.6 MB (+20 KB)
Per-entry composition (own code — deps external (as shipped))
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
. 106 KB 2.5 KB 108 KB external 343 KB
./beta 108 KB (+174 B) 485 B 108 KB (+174 B) external 324 KB (+474 B)
./beta/database-tooling 49 KB (+4.0 KB) 0 B 49 KB (+4.0 KB) external 145 KB (+12 KB)
./testing 45 KB 32 KB 77 KB external 223 KB
./tsdown 520 B 0 B 520 B external 813 B
./type-generator 26 KB 0 B 26 KB external 74 KB

Chunks:

Entry Chunk Load Size (gz)
. index.js initial 101 KB
. utils.js initial 4.6 KB
. remote-tunnel-manager.js lazy 2.5 KB
./beta beta.js initial 88 KB
./beta modular.js initial 6.0 KB
./beta stream-manager.js initial 5.9 KB
./beta service-context.js initial 4.2 KB
./beta databricks.js initial 3.3 KB
./beta client.js initial 647 B
./beta index.js initial 20 B
./beta supervisor-api.js lazy 192 B
./beta databricks.js lazy 178 B
./beta index.js lazy 115 B
./beta/database-tooling index.js initial 49 KB
./testing manifest.js initial 32 KB
./testing index.js initial 10 KB
./testing wide-event-emitter.js initial 2.9 KB
./testing index.js lazy 28 KB
./testing remote-tunnel-manager.js lazy 2.5 KB
./testing utils.js lazy 1.8 KB
./tsdown index.js initial 520 B
./type-generator index.js initial 26 KB

@databricks/appkit-ui

npm tarball (packed): 397 KB (+4 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 429 KB 147 KB
Type declarations 267 KB 98 KB (+1 B)
Source maps 853 KB 286 KB
CSS 16 KB 3.2 KB
Total 1.5 MB 534 KB (+1 B)
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
./js 5.3 KB 49 KB 55 KB 208 KB 14 KB
./js/beta 2.1 KB 0 B 2.1 KB 0 B 5.0 KB
./react 433 KB 49 KB 481 KB 1.3 MB 177 KB
./react/beta 4.9 KB 0 B 4.9 KB 0 B 12 KB

Chunks:

Entry Chunk Load Size (gz)
./js index.js initial 5.2 KB
./js chunk initial 120 B
./js apache-arrow lazy 49 KB
./js/beta beta.js initial 2.1 KB
./react index.js initial 431 KB
./react tslib initial 2.1 KB
./react apache-arrow lazy 49 KB
./react/beta beta.js initial 4.9 KB

Conflicts: template/_gitignore keeps both the introspect backup and the
history lock entries. database.md keeps PR675's development workflow section
and moves the beta note, the deploy-target check and the history lock from
the merged side into it. The workflow text now matches the merged code:
--schema none hands over no schema and reads no schema.ts, schema public is
never handed over, session branches are bound to their user, and verify
--deployed compares the deployed journal entries and SQL and fails when it
cannot read them.

Signed-off-by: ditadi <victordperd@gmail.com>
The GA Lakebase guide told every Lakebase app to run the beta
`npx appkit db init --schema none`, without saying it is beta, and said that
copied objects keep their owner and that --schema none leaves the schemas as
they are. The database development flow is beta and hidden, and it is
documented on the Database plugin page, so the Lakebase guide goes back to its
main text.

Signed-off-by: ditadi <victordperd@gmail.com>
PR675 removed hidden from the database manifest, so the generated beta barrel
and the template manifest picked the plugin up, and showed appkit db in
appkit --help. The database development flow stays hidden while it is beta:
the manifest is hidden again, beta.ts exports database by hand because the
generated barrel skips hidden plugins, the template manifest no longer offers
the plugin, and appkit db is hidden from --help again. The project field keeps
bundleIgnore.

Signed-off-by: ditadi <victordperd@gmail.com>
Signed-off-by: ditadi <victordperd@gmail.com>
With a reference databricks.yml leaves unresolved, every branch counts as a
deploy target and init refuses too, so the local verify warning sent you to a
command that cannot help. The report now says whether the refusal covers every
branch, and the warning then says which commands refuse and leaves out init.

Signed-off-by: ditadi <victordperd@gmail.com>
…es a migration

Signed-off-by: ditadi <victordperd@gmail.com>
Signed-off-by: ditadi <victordperd@gmail.com>
… branch

refusesEveryBranch is now the guard's every-branch refusal text, not a flag.
The warning prints it after the branch's own reason, so a .env on the default
or a protected branch still names the reference and its file, and it lists
introspect among the commands that refuse.

Signed-off-by: ditadi <victordperd@gmail.com>
…ation checks

verify --deployed reads the branch from variable-overrides.json and fails on a
deployed migration the checkout does not commit. On a branch that takes no
writes, migrate also regenerates shared/appkit-types/database.d.ts.

Signed-off-by: ditadi <victordperd@gmail.com>
ditadi added a commit that referenced this pull request Oct 30, 2026
The design doc promised behavior the corrected stack does not ship. It
now describes what the code does:

- The tooling export is @databricks/appkit/beta/database-tooling, beta
  and internal to the AppKit CLI.
- migrate, introspect, clean and init hold config/database/.appkit-db.lock.
  init holds one lock for its whole run, and init --schema none, seed,
  verify and clean --delete-branch take none. The doc says how to
  recover a stale lock file.
- init --schema none hands off nothing, and schema public is never
  handed off.
- Values in variable-overrides.json count for the deploy boundary, and
  an unresolved branch or database reference refuses every write.
- Introspect keeps .private() on declared columns, writes new columns
  private, and refuses a schema.ts that does not load or a foreign key
  it does not declare.
- verify --deployed compares the committed migrations with the source
  snapshot of the active deployment, and fails when it cannot read it.
- Migrations log one line per run. There is no span and no metric.
- #675 keeps the plugin and appkit db hidden. A reveal is a later,
  separate change.

The appkit db check section stays as it is until #678 settles.

Signed-off-by: ditadi <victordperd@gmail.com>
@ditadi ditadi changed the title feat(appkit): offer the database plugin in databricks apps init feat(appkit): add template and docs for the database development flow Oct 30, 2026
ditadi and others added 3 commits November 9, 2026 13:05
…schema

Redeploying an older commit applies nothing, and the startup check then fails
when a table or column that commit declares is gone. A migration that drops or
renames one ends rollbacks to the commits that need it, so the docs no longer
promise that two-step changes keep every older commit starting.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
A branch db init creates now starts with the app schema rebuilt from the
committed migrations, or from the baseline introspect writes, so it holds no
copied rows. --schema migrate keeps them. Existing apps with a schema.ts adopt
their tables with init --schema introspect, because a plain init on a new branch
drops tables the migrations do not create.

Introspect reads the schema that schema.ts declares. Without schema.ts, init and
introspect read schema app, so an app on public runs init --schema none and then
introspect --schema-name public.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
ditadi and others added 17 commits November 9, 2026 14:11
…ed migrations

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…kebase-only recovery

The hint for a schema another role owns must say first that two apps on one
database each need their own schemaName, then give the delete-role recovery
for a recreated app or a re-added resource, with the warning that it moves
every object the role owns. It must not suggest ALTER OWNER, which fails
with "must be able to SET ROLE" on a Lakebase service principal role.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…pps case first

Schema "app" owned by another role has two causes, and the hint now orders
them. Two apps on one database each need their own schemaName. Otherwise
the app was recreated or its resource re-added, and Lakebase handed the
objects to another role. No Postgres role can hand them back, because
ALTER ... OWNER TO a service principal role fails with "must be able to
SET ROLE", so the hint gives only the platform's delete-role
--reassign-owned-to command, covers the re-add case where the owner is the
remover's role, and warns that it moves every object the role owns and
deletes the role.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…very for database

Two apps on one database with the same schema are unsupported; each needs
its own schemaName. A deploy succeeds 2 s after the process starts unless
app.yaml declares a health_check, and then /health must answer within
300 s, which AppKit binds only after migrations. Classic compute stops the
old process first, no-downtime compute keeps it serving, a crashed process
is not restarted, and scale-to-zero apps rerun the start on every cold
start. A start migrates only as the app's own service principal, so a
local start or run-local never migrates, and a deploy start refuses an
edited migration or a diverged history. A failed migration is repaired in
the rows, never by editing the migration; a DATABRICKS_SUPERUSER role on
the deploy branch reads and writes the service principal's rows, runs a
batched backfill between an expand and a contract migration, and can
cancel a stuck appkit-migrate session. The owner recovery drops ALTER
OWNER, which fails with "must be able to SET ROLE", lists what the role
owns first, and gives delete-role --reassign-owned-to. init decides from
the branch's schema owner instead of APPKIT_DB_NEW_BRANCH, tags the
copied journal until the rebuild finishes, and forks the branch the
default databricks.yml target binds.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
….yaml has health_check

The template README no longer says init forks the project's default
branch or that any new branch is rebuilt; init forks the branch the app
deploys to and rebuilds the app schema when it still belongs to the
copied service principal. It says the deployed app migrates only as its
own service principal and that the deploy waits up to 300 s for /health.
The app.yaml comment records why the health check is there, so nobody
removes it and turns a failed migration into a deploy that passed.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
# Conflicts:
#	docs/docs/plugins/database.md
#	packages/appkit/src/database/migrations/ownership.ts
#	packages/appkit/src/database/migrations/tests/ownership.test.ts
A start migrates only when PGUSER is set and equals DATABRICKS_CLIENT_ID.
The database plugin docs and the template README said a local start never
migrates because it is not a deployment. Say it as the identity rule
instead, and that db init writes the developer's PGUSER, so a local start
or run-local connects as them and skips.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
The one owner per branch runbook listed only pg_class objects before the
role delete. Use the inventory db init takes before a hand-off instead:
every object the role owns in the current database through pg_shdepend and
pg_identify_object, run in each database, plus what it owns or is granted
in other databases and shared objects. Say that the delete also drops the
role's memberships and grants, and to resolve anything unrelated to the
app first.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…--schema clean after a keep

Add three facts of the fork-copy rule to the database docs and the
template README. A branch the same init run created is rebuilt even when
its app schema was already yours. migrate, introspect and seed refuse an
untouched copy as well as a tagged one. After keeping the copied rows, the
schema is yours and appkit db init --schema clean removes them. Fold the
two keep-copy paragraphs that the merge of the init layer left side by
side into one.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
# Conflicts:
#	docs/docs/plugins/database.md
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…nit's resolver

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant