Skip to content

Offline mode (ContextSchemaPath) improvements - #884

Open
Thorium wants to merge 2 commits into
fsprojects:masterfrom
Thorium:offline-schema-improvements
Open

Thorium wants to merge 2 commits into
fsprojects:masterfrom
Thorium:offline-schema-improvements

Conversation

@Thorium

@Thorium Thorium commented Nov 20, 2026

Copy link
Copy Markdown
Member

Offline mode (ContextSchemaPath) improvements: Individuals to a separate json file, and add schema descriptions to another one.

…ate json file, and add schema descriptions to another one.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Several moderate active issues remain in schema-qualified metadata lookups, concurrency guarding, and PostgreSQL identifier handling, risking incorrect or missing descriptions and duplicate expensive reads.

5 open issues
What changed in this PR

This PR improves offline ContextSchemaPath support by separating schema metadata, descriptions, and sampled individuals into adjacent files, while adding provider-specific bulk description loading and new full-schema design-time commands.

Changes:

  • Stores individuals and table/column descriptions in separate JSON files with backward-compatible loading and merge support.
  • Adds lazy offline loading for descriptions and individuals, plus commands for saving the full schema or sampled rows.
  • Adds bulk column-description queries for PostgreSQL, Oracle, MySQL, SQL Server, Firebird, and DuckDB, with accompanying tests and documentation.
File Description
tests/​SqlProvider.Tests/​SqlProvider.Tests.fsproj Includes the new schema cache persistence tests.
tests/​SqlProvider.Tests/​SchemaCacheFileTests.fs Tests separate individuals and descriptions files, compatibility, serialization, and merging.
src/​SQLProvider.Runtime/​Providers.Postgresql.fs Adds bulk PostgreSQL column-description loading and quoted table handling.
src/​SQLProvider.Runtime/​Providers.Oracle.fs Adds bulk Oracle column-description loading.
src/​SQLProvider.Runtime/​Providers.MySql.fs Adds bulk MySQL column-description loading.
src/​SQLProvider.Runtime/​Providers.MsSqlServer.fs Adds bulk SQL Server column-description loading.
src/​SQLProvider.Runtime/​Providers.MsSqlServer.Dynamic.fs Adds bulk dynamic SQL Server column-description loading.
src/​SQLProvider.Runtime/​Providers.Firebird.fs Adds bulk Firebird column-description loading.
src/​SQLProvider.Runtime/​Providers.DuckDb.fs Adds bulk DuckDB column-description loading.
src/​SQLProvider.DesignTime/​SqlDesignTime.fs Loads offline metadata lazily and adds full-schema and individuals-saving commands.
src/​SQLProvider.Common/​Utils.fs Adds shared reader logic for column descriptions.
src/​SQLProvider.Common/​SqlRuntime.Common.fs Implements adjacent-file persistence, compatibility loading, merging, and the bulk-description interface.
docs/​RELEASE_NOTES.md Documents the new offline files, commands, compatibility, and provider support.
docs/​content/​core/​parameters.fsx Documents the offline file layout, commands, and usage workflow.

Review effort: Lite

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Active moderate issues remain in description merging, primary-key type preservation, and quoted table-name handling across several providers, so the changes are not yet ready for approval.

No open issues 🎉

6 previously missed issues

In code that hasn't changed since last review

Medium severity Preserve updated descriptions when merging existing file entries

src/​SQLProvider.Common/​SqlRuntime.Common.fs:1004

SaveDescriptions first calls LoadDescriptions, and LoadDescriptions uses TryAdd for every entry from the existing file. Therefore an existing description wins over the value currently held in this, so saving an updated online comment silently restores the stale on-disk value. The added test saving the descriptions read so far keeps the others in the file expects "Changed" here and will fail; merge the old file only for keys that are absent from the current cache, while letting the current cache overwrite matching keys.

Medium severity Keep primary key values in their original type

src/​SQLProvider.Common/​SqlRuntime.Common.fs:1044

The conversion is applied to the primary key as well as ordinary column values. For a primary key type such as DateTimeOffset, IndividualsFileValue converts the key to text; offline createIndividualsType then embeds that string in GetIndividual, which binds it using the database's original primary-key mapping and can fail or select no row. Only non-key display values can safely be downgraded to text; preserve the key's type or restore it from the cached column mapping when loading.

Medium severity Unquote table and schema names before SQL Server metadata lookup

src/​SQLProvider.Runtime/​Providers.MsSqlServer.Dynamic.fs:585

This dynamic SQL Server implementation has the same identifier issue as the regular provider: Table.FullName wraps whitespace-containing names in double quotes, but those wrappers are bound directly to sys.tables.name/sys.schemas.name. The new full-schema description export will silently miss tables such as Order Details; strip the wrapper quotes from both parsed names before binding them.

Medium severity Unquote table and schema names before SQL Server metadata lookup

src/​SQLProvider.Runtime/​Providers.MsSqlServer.fs:506

Table.FullName also quotes names containing spaces, so tn/sn can contain the literal " wrapper. Comparing those values to sys.tables.name and sys.schemas.name makes descriptions for tables such as Order Details return no rows, including in the new all-columns query. Remove the wrapper quotes before binding the metadata names.

Medium severity Unquote Oracle identifiers before querying description metadata

src/​SQLProvider.Runtime/​Providers.Oracle.fs:232

Table.FullName adds double-quote wrappers around identifiers containing spaces, but descriptionOwnerAndTable only escapes apostrophes and leaves those wrappers in the values compared with ALL_TAB_COMMENTS/ALL_COL_COMMENTS. An Oracle table such as Order Details therefore has no matching description in the new offline export. Strip the wrapper quotes before applying sqlText (the escaping should happen after unquoting).

Medium severity Unquote PostgreSQL identifiers before regclass lookup

src/​SQLProvider.Runtime/​Providers.Postgresql.fs:859

Table.FullName wraps identifiers containing spaces in double quotes, but this path passes those wrappers to PostgreSQL's %I formatter. For a table such as Order Details, tn is already "Order Details", so %I quotes the quote characters and the regclass lookup misses the real table; SaveContextSchemaFull consequently omits its descriptions. Strip the provider-added quote wrappers before passing schema/table names (as the MySQL and DuckDB implementations do).

5 resolved since last review

Review effort: Lite

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants