Skip to content

feat(appkit): apply committed database migrations on deploy start - #654

Draft
ditadi wants to merge 25 commits into
mainfrom
stack/database-dev-flow/01-migrations-on-deploy
Draft

ditadi wants to merge 25 commits into
mainfrom
stack/database-dev-flow/01-migrations-on-deploy

Conversation

@ditadi

@ditadi ditadi commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

🥞 Stacked PR (generated by Stack)

Use this link to review incremental changes.


First implementation PR of the DatabasePlugin development flow stack. The design is #677. The plugin stays hidden from databricks apps init through the whole stack.

What did you change, and why?

A deployed app has to apply its committed migrations by itself, because App Builder apps have no human or CI to run them (design 001).

database() applies the drizzle migrations under config/database/migrations after the pool opens, before select 1 and the declared-schema check. It runs only when APPKIT_DB_MIGRATE_ON_START=true and the service principal's credentials are set, so a local run never migrates on start. The service principal runs the migrations and owns the app schema.

  • Each migration runs and is recorded in its own transaction. A failure keeps the migrations before it.
  • The journal is reconciled with the recorded rows. A migration older than the last recorded one that never ran fails startup and changes nothing. An edited migration, or a recorded row with no committed file, logs a warning.
  • A start with nothing pending takes no lock. Otherwise a per-schema advisory lock makes instances apply in turn. The wait is capped at 5 minutes and logs the holder's pid and application_name. Each table lock waits 5 s, with three tries per migration.
  • A dev-… or session-… branch is skipped on start. A -pooler host is refused, because the session lock does not hold through a transaction-mode pooler.
  • An unrecorded 0000_baseline whose tables and columns all exist is recorded as applied. A partial match fails startup.
  • Every failure names the step and the SQLSTATE. On 42501 the error also names the schema owner and the connecting role. Clients only see Database setup failed. One summary line per start logs the counts, the lock wait and the duration.

This revision changes two things:

  • Two journal entries that share a when each keep their own recorded row. Before, the entry that never ran read as edited and was never applied. Now it is pending or stranded.
  • knip.json declares three tools/ scripts and packages/appkit/bin/appkit.js as entry points. The pre-commit knip run had reported the bin, and the esbuild, ora and zod imports of those scripts, as unused.

Nothing migrates until an app sets APPKIT_DB_MIGRATE_ON_START=true, so this PR is safe to deploy alone. It adds no span and no metric for migrations.

Review round, 2026-11-09:

  • Migrations that share a journal timestamp are matched to recorded rows one to one. A history the migrator cannot pair unambiguously is refused (MigrationHistoryError) instead of skipping a migration that never ran.
  • The migrator uses __drizzle_migrations only when it is an ordinary table: it records only into a table whose owner it trusts (its own role or a role it inherits, or the schema's owner), and reads another role's table only when that table has no row-level security. schemaName: "public" apps, whose journal the SP owns in a schema it does not own, keep working.
  • When the server rejects client_connection_check_interval, the migrator logs a warning that a killed instance's statement keeps its locks until it ends, and how to end it.

Fix round for fjakobs's review of #677, 2026-11-15:

  • A start migrates only when PGUSER is set and equals DATABRICKS_CLIENT_ID, so a local or databricks apps run-local start never migrates as another identity. With the flag on and no match, it logs why (L307 and the run-local item).
  • The name-based skip is gone. A deployed app bound to a branch with the exact shape of a dev or session branch refuses to start (L307).
  • A deploy start refuses an edited migration, and a diverged history (committed migrations unrecorded next to recorded rows the checkout lacks). The CLI still only warns. A file checked out with CRLF or LF counts as the same file (L424, L456).
  • The ownership errors stop suggesting ALTER ... OWNER TO, which Lakebase refuses with must be able to SET ROLE, and name the two-apps case first (L503, L260). The startup check no longer tells an operator to set a flag that is already set.

How do you know it works?

  • apply.test.ts, history.test.ts and migrate-on-start.test.ts cover the transactions, each reconcile case including two entries with one when, the locks, the message for each failed step, the flag, the branch skip and the pooler refusal.
  • apply.pg.test.ts runs against Postgres when APPKIT_TEST_PG_URL is set, and CI sets it with a postgres:17 service. It covers the enum case that fails with 55P04 in one transaction, a stranded migration, three instances starting together, a schema owned by another role, and baseline adoption.
  • At this PR's head, the history tests passed, and a start against a local PostgreSQL 14 applied both migrations while a second start applied none. The full vitest suite, a clean build, the docs build and typecheck passed at the top of the stack (feat(appkit): add appkit db check #678), which contains this PR, and its packed tarballs export the same root and ./beta names as main. CI runs the full suite on this PR.
  • An earlier revision ran on a Lakebase project with a throwaway deploy branch and app. This revision was not run against a real Lakebase workspace.

Review round, 2026-11-09: each fix has a failing-then-passing test in its PR, and every layer's appkit and shared tests ran against Postgres 14 after the sync. At the top of the stack, pnpm build, pnpm test (6502 passed, 1 skipped), pnpm -r typecheck, pnpm check, pnpm knip and pnpm docs:build pass.

Fix round for fjakobs's review, 2026-11-15: each fix has a failing-then-passing test in its PR, and every layer's database, plugin and CLI tests ran against Postgres 17 after the sync. A throwaway Lakebase fork confirmed that the developer role can write the service principal's rows but cannot alter its tables. At the top of the stack, pnpm test (6602 passed, 1 skipped), pnpm -r typecheck, pnpm build and pnpm docs:build pass, and oxlint and oxfmt are clean on every changed file.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

@databricks/appkit

npm tarball (packed): 1.3 MB (+25 KB) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 1.3 MB (+28 KB) 468 KB (+10 KB)
Type declarations 464 KB 168 KB (+2 B)
Source maps 2.6 MB (+51 KB) 873 KB (+17 KB)
Other 11 KB 3.7 KB
Total 4.3 MB (+80 KB) 1.5 MB (+28 KB)
Per-entry composition (own code — deps external (as shipped))
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
. 106 KB (-1 B) 2.5 KB (+1 B) 108 KB external 343 KB
./beta 107 KB (+4.8 KB) 485 B 108 KB (+4.8 KB) external 322 KB (+13 KB)
./testing 45 KB (+1 B) 32 KB (-3 B) 77 KB (-2 B) external 223 KB
./tsdown 520 B 0 B 520 B external 813 B
./type-generator 26 KB 0 B 26 KB external 74 KB

Chunks:

Entry Chunk Load Size (gz)
. index.js initial 101 KB
. utils.js initial 4.6 KB
. remote-tunnel-manager.js lazy 2.5 KB
./beta beta.js initial 87 KB
./beta modular.js initial 6.0 KB
./beta stream-manager.js initial 5.9 KB
./beta service-context.js initial 4.2 KB
./beta databricks.js initial 3.3 KB
./beta client.js initial 647 B
./beta index.js initial 20 B
./beta supervisor-api.js lazy 192 B
./beta databricks.js lazy 178 B
./beta index.js lazy 115 B
./testing manifest.js initial 32 KB
./testing index.js initial 10 KB
./testing wide-event-emitter.js initial 2.9 KB
./testing index.js lazy 28 KB
./testing remote-tunnel-manager.js lazy 2.5 KB
./testing utils.js lazy 1.8 KB
./tsdown index.js initial 520 B
./type-generator index.js initial 26 KB

@databricks/appkit-ui

npm tarball (packed): 397 KB (+4 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 429 KB 147 KB
Type declarations 267 KB 98 KB (+1 B)
Source maps 853 KB 286 KB
CSS 16 KB 3.2 KB
Total 1.5 MB 534 KB (+1 B)
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
./js 5.3 KB 49 KB 55 KB 208 KB 14 KB
./js/beta 2.1 KB 0 B 2.1 KB 0 B 5.0 KB
./react 433 KB 49 KB 481 KB 1.3 MB 177 KB
./react/beta 4.9 KB 0 B 4.9 KB 0 B 12 KB

Chunks:

Entry Chunk Load Size (gz)
./js index.js initial 5.2 KB
./js chunk initial 120 B
./js apache-arrow lazy 49 KB
./js/beta beta.js initial 2.1 KB
./react index.js initial 431 KB
./react tslib initial 2.1 KB
./react apache-arrow lazy 49 KB
./react/beta beta.js initial 4.9 KB

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 41109403947 -R databricks/appkit -n appkit-template-0.89.0-pr.16ec3ee-stack-database-dev-flow-01-migrations-on-deploy-654 -D appkit-pr-654 \
  && unzip -o "appkit-pr-654/appkit-template-0.89.0-pr.16ec3ee-stack-database-dev-flow-01-migrations-on-deploy-654.zip" -d "appkit-pr-654" \
  && databricks apps init --template "appkit-pr-654"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

A deployed DatabasePlugin app now applies the migrations committed under
config/database/migrations before its startup checks, as its own service
principal and under a per-schema advisory lock. Pending migrations run in
one transaction, and a table lock held for over 30s fails the migration
instead of stalling the table. A 0000_baseline whose tables already exist
is recorded, in its own transaction, instead of run. Any failure stops
startup, and the server log names it: the tags and SQLSTATE, or the
missing migration file. Local runs never migrate on start.

Signed-off-by: ditadi <victordperd@gmail.com>
- Run and record each pending migration in its own transaction. An enum value
  added by one migration can now be used by the next (55P04), and a failure
  keeps the migrations before it.
- Compare the committed journal with the recorded rows by hash and created_at.
  Fail closed on a migration older than the last recorded one; warn on edited
  migrations and on recorded rows with no committed file.
- Read the history without the lock first. Take the lock only when something
  is pending, then read again.
- Bound the wait for the migration lock to 5 minutes and name its holder. Set
  application_name and client_connection_check_interval. Lower lock_timeout to
  5s with three tries per migration.
- Migrate on start only with APPKIT_DB_MIGRATE_ON_START=true and service
  principal credentials. Skip dev- and session- branches; refuse a -pooler
  host.
- Name the failed step and the SQLSTATE or system code. On 42501, name the
  schema owner and the connecting role.
- Add real-Postgres tests gated on APPKIT_TEST_PG_URL; CI runs them against a
  postgres:17 service.

Signed-off-by: ditadi <victordperd@gmail.com>
@ditadi
ditadi force-pushed the stack/database-dev-flow/01-migrations-on-deploy branch from 5505de7 to 19adf06 Compare October 20, 2026 12:23
ditadi and others added 16 commits October 30, 2026 12:52
Signed-off-by: ditadi <victordperd@gmail.com>
knip.json ignores tools/**, so knip reported esbuild, ora and zod as unused even though
tools/bundle-size.ts, tools/generate-json-schema.ts and tools/playground/deploy-playground.ts
import them. It also reported packages/appkit/bin/appkit.js as an unused file. Declare these
as entry points so the pre-commit knip check passes without suppressing any finding.

Signed-off-by: ditadi <victordperd@gmail.com>
reconcileMigrations kept one recorded hash per created_at, so a committed migration whose
journal entry shares a when with an applied sibling was reported as edited and never ran.
Keep every recorded hash per when, and classify an entry whose sibling owns the recorded
row as pending or stranded.

Signed-off-by: ditadi <victordperd@gmail.com>
…imestamp

Two committed migrations with one `when`, neither matching the row
recorded at it, are both classified edited today, so the one that never
ran is skipped with only a warning. A second recorded row at a committed
`when` is reported as neither edited nor unknown. These tests expect each
timestamp to be reconciled one to one, and the migrator to refuse when
the pairing is ambiguous.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
… ambiguous history

reconcileMigrations keyed each `when` on a set of recorded hashes, so two
files sharing a timestamp with one recorded row were both called edited
(the one that never ran was skipped with a warning), and a second recorded
row at a committed `when` was neither edited nor unknown.

Each `when` is now a bucket: exact hash matches are consumed first; one
file left with one row left is edited; files left with no row are pending
or stranded by lastRecorded; rows left with no file are unknown. Any other
remainder is ambiguous, and the migrator refuses it with a
MigrationHistoryError that names the timestamp, the files and the row
count, before taking the lock. MigrationHistoryError takes the refusal as
an object, with a `stranded` and an `ambiguous` field.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…s table

The migrator finds the journal by name in pg_class and then selects from
and inserts into it, so a view or another role's table at that name would
run as the migrator. These tests expect the catalog read to return the
relation's kind and owner, the read and the write paths to refuse
anything but an ordinary table owned by the schema's owner, the journal
to be created only after that check and without IF NOT EXISTS, and the
refusal to become a setup failure on start.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…r's table

readRecordedMigrations found the journal by name in pg_class and then
selected from it, and ensureJournalTable created it with IF NOT EXISTS,
so a view or another role's table at that name would have been read and
written with the migrator's privileges.

One check now fronts both paths. The catalog lookup returns the relation's
kind, its owner and the schema's owner; anything but an ordinary table
owned by the schema's owner is refused with a MigrationJournalError that
names the relation, its kind and both owners. A missing table still means
nothing recorded. The journal is created only after that check, without
IF NOT EXISTS, so a relation that appears in between fails the
transaction instead of receiving the record. migrateOnStart turns the
refusal into a setup failure like the other migration errors.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…ating

statement_timeout is 0 for the migration session, and
client_connection_check_interval is what lets the server cancel a
statement whose instance died. When the server rejects that setting the
migrator only logs at debug, so an operator never learns that a killed
instance's statement keeps its locks until it ends. This test expects a
warning that says so and names pg_terminate_backend.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…ejected

The migration session runs with statement_timeout = 0 and relies on
client_connection_check_interval for the server to cancel a statement
whose instance died. When the server rejects that setting, the message
was logged at debug only. It is now a warning that says a killed
instance's statement keeps its locks until it ends, and how to end it
with pg_terminate_backend on the pid found in pg_stat_activity. No
statement deadline is added; migrations must still fit the start window.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
journalTableExists refuses the journal unless its owner is the schema's
owner. On `public` the schema belongs to the database owner (or to
pg_database_owner on Postgres 15 and Lakebase) while the app's service
principal creates and owns the journal there, so every app with
schemaName "public" fails its second start, and local tooling reading an
SP-owned journal in `public` fails with it.

These tests expect the journal's owner to be trusted when the connecting
role owns it or inherits it, or when it is the schema's owner. A table of
an untrusted owner is still read when it has no row-level security, since
a plain SELECT runs no code of its owner's, and it is refused when the
migrator would record into it, naming both roles and the fix. Row-level
security on such a table refuses the read. A view is refused as before.
The Postgres test builds the `public` shape with real roles and expects
two starts to read and record through the app's own journal.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…wner

journalTableExists accepted the journal only when its owner was the
schema's owner. On `public` that owner is the database owner (or
pg_database_owner on Postgres 15 and Lakebase) while the app's service
principal creates and owns the journal there, so every app with
schemaName "public" failed its second start, and local tooling reading
an SP-owned journal in `public` failed with it.

The threat is a relation at the journal's name that another role controls
and that runs code when the migrator reads or records: a view, a
row-security policy, triggers, rules and column defaults. The catalog
query now also returns whether the connecting role owns the relation or
inherits its owner (pg_has_role with USAGE), or the owner is the schema's;
either makes the owner trusted. Anything but an ordinary table is still
refused. A trusted owner's table is read and recorded into. An untrusted
owner's table is read when it has no row-level security, since a plain
SELECT runs none of its owner's code, and refused for the read when it
does; recording into it is refused, with a message that names the owner
and the connecting role and says to have the owner run the migration or
transfer the table with ALTER TABLE … OWNER TO. MigrationJournalError
takes the refusal next to the relation. The missing-table path and the
CREATE without IF NOT EXISTS are unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…olerating line endings

A recorded migration whose committed file changed, or pending migrations
next to recorded rows this checkout lacks, must stop a deploy start before
anything runs. A file read back with the other line endings is the same
migration. A pure code rollback and a CLI-style call keep warning.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…across line endings

applyPendingMigrations takes onDrift. "warn" is today's behavior, so a
developer can still rework an unapplied migration and rerun. migrateOnStart
passes "refuse": a recorded migration whose file changed, or pending
migrations next to recorded rows this checkout lacks, throw
MigrationHistoryError before anything runs. The error names each tag and
the way back: restore the file and add a new migration, or repair rows on
the deploy branch and redeploy the unchanged file; deploy a commit whose
journal has the recorded migrations, or merge them into main unchanged
first. A pure rollback (unknown rows, nothing pending) still only warns.

Each file now carries the hashes of its LF and CRLF variants, so a checkout
with core.autocrlf matches the row another checkout recorded. The as-is hash
is still what gets recorded. MigrationHistoryError exposes the refusal
union instead of parallel stranded and ambiguous arrays.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
Measured on Lakebase: ALTER ... OWNER TO the service principal's role fails
with "must be able to SET ROLE", and nobody holds ADMIN or SET on an Apps
service principal role. The hints name the shared-database case first and
say only Lakebase can move objects between roles.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
Measured on Lakebase: ALTER ... OWNER TO the service principal's role fails
with "must be able to SET ROLE", and nobody holds ADMIN or SET on an Apps
service principal role. The schema hint now names the shared-database case
first (give each app its own schemaName), then the recreated-app case, and
says no Postgres role can hand the objects back; only Lakebase can move
them. The generic hint and the journal-table messages say the same.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
The decision is a pure function of the environment: off without the flag,
skipped with a reason when a credential is missing or PGUSER is not the
app's client id, migrate otherwise. A skipped start warns once with that
reason, and the schema check stops telling an operator to set a flag that
is already set. A deployed app bound to a branch with the exact shape of a
developer or agent session branch refuses to start instead of skipping
silently; session-store and session-k3x9q2m7 do not have that shape.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
ditadi and others added 7 commits November 15, 2026 20:19
decideMigrateOnStart replaces shouldMigrateOnStart with a pure decision:
off without the flag, skipped with a reason when DATABRICKS_CLIENT_ID or
DATABRICKS_CLIENT_SECRET is missing or PGUSER is not the app's client id,
migrate otherwise. Databricks Apps sets PGUSER to the service principal's
client id for a bound database, so a local start or `databricks apps
run-local` with exported M2M credentials keeps the developer's PGUSER and
no longer migrates as them. The lifecycle logs one warning with the reason
when the flag is on but the start is skipped.

The schema check takes the decision and words its advice by it: set the
flag when it is off, read the startup warning when the start was skipped,
commit a migration when the start migrated and the schema still differs.

A deployed app bound to a branch whose id has the exact shape of a
developer branch (dev-<slug>-<6 hex>, at most 63 characters) or an agent
session branch (session-<6 hex>-<id>) fails its start with the branch name
and the fix, instead of skipping silently; session-store and
session-k3x9q2m7 do not match and migrate. The message no longer names a
CLI command this layer does not ship.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…nown rows as diverged

When the branch deployed first carries the newer timestamp, main's own
migration is stranded instead of pending, and today's stranded message
tells the developer to delete and regenerate it. With rows this checkout
lacks, the right way back is the diverged runbook. The same history keeps
the stranded refusal without the option, as the CLI callers expect.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…own rows as diverged

Under onDrift "refuse", a history with stranded entries and recorded rows
this checkout lacks is a divergence, not a merge mishap: the branch that
deployed first carried the newer timestamp, so main's own migration lands
behind it. The stranded message would tell the developer to delete and
regenerate that migration, which is the wrong way back. The diverged
refusal now covers it, with the same runbook: deploy a commit whose journal
has the recorded migrations, or merge them into main unchanged first.
Without the option the stranded refusal stays, so CLI callers see no change.

The diverged refusal's tag list is renamed from pending to unrecorded, since
it names every committed migration the database never recorded, stranded or
pending, oldest first. Nothing outside apply.ts read the field.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
Exported M2M credentials with no PGUSER skip instead of migrating. The
Lakebase pool falls back to DATABRICKS_CLIENT_ID for the user, so a local
process in that state would otherwise migrate as the service principal.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
decideMigrateOnStart migrated when PGUSER was unset. The Lakebase pool
then falls back to DATABRICKS_CLIENT_ID for the user, so a local process
with exported M2M credentials and no PGUSER connected as the service
principal and migrated. Databricks Apps sets PGUSER to the client id for
every app with a bound database, so an unset PGUSER now skips with a
reason that says where the deployed app gets it.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
The comment audit of this fix round flagged 24 comments in this layer
that restate a name, a type, a body or a nearby message. They go. The
kept comments each explain a foreign fact the code cannot show, such as
drizzle's hashing and separator, git autocrlf, Lakebase's ALTER OWNER
refusal, and the PGUSER that Databricks Apps sets.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
A stranded migration next to rows this checkout lacks means a branch
deployed first, and a deploy start refuses that as diverged rather than
as stranded, so the fix is the diverged runbook and not a new timestamp.
That rule lived in a comment over a bare boolean. It is now the predicate
branchDeployedFirst, true when unrecorded migrations here sit next to
recorded rows this checkout lacks, and the onDrift check stays at the
call site. Behavior is unchanged. The migration tests pin the stranded,
edited and diverged refusals.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant