Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/RELEASE_NOTES.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,16 @@
### 1.5.34 - (unreleased)
- Offline mode (ContextSchemaPath) saves three files beside each other: schema.json for the schema, schema.descriptions.json for table and column descriptions (comments), schema.individuals.json for individuals. Builds read only the schema file; the descriptions file is read for the first tooltip and the individuals file when Individuals are first opened
- Offline tooltips show the saved table and column descriptions, instead of only "Offline mode." SaveContextSchema saves the descriptions of what you hovered over online
- A saved schema no longer carries individuals, so it stays small and loads fast. A schema file with individuals in it still loads them, and re-saving it moves them to the individuals file. Older versions can read the new schema files
- New design-time commands next to SaveContextSchema, online only, that read the whole database instead of what was used so far: SaveContextSchemaFull (every table, column, relationship and stored procedure, and every table and column description) and SaveContextSchemaIndividuals (individuals of every table with a single primary key)
- SaveContextSchemaFull reads the column descriptions of a table in one query instead of one per column (MS SQL Server, PostgreSQL, MySQL, Oracle, Firebird, DuckDB)
- OfflineTools.mergeCacheFiles merges descriptions and individuals files too
- Fix: after a failed SaveContextSchema, design-time commands no longer stop working until the IDE restarts
- Fix: saving individuals no longer fails on a column value such as a DateTimeOffset or an array; such values are saved as text
- Fix: PostgreSQL tables with capital letters or an apostrophe in their name show their descriptions in tooltips; reading them failed before
- Fix: MS SQL Server descriptions of a table come from its own schema; with tables of the same name in two schemas, either could show the other's
- Fix: Oracle descriptions of tables owned by another schema (Owner) are read; only the current user's tables had them before

### 1.5.33 - 19.11.2026
- Fix: a saved schema (ContextSchemaPath) loads on both .NET and .NET Framework, whichever saved it. Visual Studio, whose editor and compiler run on .NET Framework, failed on a schema saved by dotnet with "Value cannot be null. Parameter name: key"

Expand Down
38 changes: 38 additions & 0 deletions docs/content/core/parameters.fsx
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,45 @@ is saved, the outcome of the method execution is stored in memory so the file wi
In case the database schema changes and the schema file must be updated, remove the outdated file, reload
the solution and retype or uncomment a call to `SaveContextSchema` to regenerate the schema file.

The offline mode uses up to three files beside each other:

| File | Contents | Read |
|---|---|---|
| `schema.json` (`ContextSchemaPath`) | Tables, columns, relationships, stored procedures | Every build |
| `schema.descriptions.json` | Table and column descriptions (comments), shown in tooltips | For the first tooltip |
| `schema.individuals.json` | Sample rows of the [Individuals](individuals.html) feature | When `Individuals` is first opened |

Builds read only the schema file, so descriptions and individuals don't slow them down.

To save the whole database instead of the parts used so far, use these commands while online (before the schema file
exists). Each reads the whole database and may take a while on a large one:

```fsharp
// Every table, column, relationship and stored procedure, to schema.json, and every table
// and column description, to schema.descriptions.json. No rows of data.
ctx.``Design Time Commands``.SaveContextSchemaFull
// IndividualsAmount individuals of every table with a single primary key, to schema.individuals.json
ctx.``Design Time Commands``.SaveContextSchemaIndividuals
```

These two commands exist only while online. Remove the line after saving: once the schema file exists, the
solution is offline and a line that still calls one of them doesn't compile.

Individuals are rows of your data, so mind GDPR/PII before committing the individuals file to version control.

When the database schema changes, remove all three files, not only the schema file, before saving again. The
descriptions and individuals files are otherwise kept as they are, and may then describe tables and columns that
no longer exist.

`SaveContextSchema` also saves the descriptions of the tables and columns you hovered over, and keeps the descriptions
already in the descriptions file. Offline, a table or column whose description wasn't saved shows "Offline mode." in
its tooltip.

Schema files saved by older versions with individuals in them still work: re-saving such a file moves its individuals
to the individuals file.

There is a tool method FSharp.Data.Sql.Common.OfflineTools.mergeCacheFiles to merge multiple files together.
It merges their descriptions and individuals files as well.

*)

Expand Down
140 changes: 131 additions & 9 deletions src/SQLProvider.Common/SqlRuntime.Common.fs
Original file line number Diff line number Diff line change
Expand Up @@ -932,7 +932,13 @@ and SchemaCache =
SprocsParams : ConcurrentDictionary<string,QueryParameter list> //sproc name and params
Packages : ResizeArray<CompileTimeSprocDefinition>
Individuals : ConcurrentDictionary<string, (obj * IDictionary<string,obj>) array> //table name and entities
IsOffline : bool }
IsOffline : bool
/// Table descriptions (comments) for tooltips, by table name. Saved in their own file (SaveDescriptions), not in the schema file.
[<field: NonSerialized>]
TableDescriptions : ConcurrentDictionary<string,string>
/// Column descriptions (comments) for tooltips, by table name and column name. Saved like TableDescriptions.
[<field: NonSerialized>]
ColumnDescriptions : ConcurrentDictionary<string,ConcurrentDictionary<string,string>> }
with
static member Empty = {
PrimaryKeys = ConcurrentDictionary<string,string list>()
Expand All @@ -943,22 +949,124 @@ and SchemaCache =
SprocsParams = ConcurrentDictionary<string,QueryParameter list>()
Packages = ResizeArray()
Individuals = ConcurrentDictionary<string, _>()
IsOffline = false }
IsOffline = false
TableDescriptions = ConcurrentDictionary<string,string>()
ColumnDescriptions = ConcurrentDictionary<string,ConcurrentDictionary<string,string>>() }
static member Load(filePath) =
use ms = new MemoryStream(Encoding.UTF8.GetBytes(File.ReadAllText(filePath)))
let ser = System.Runtime.Serialization.Json.DataContractJsonSerializer(typeof<SchemaCache>)
{ (ser.ReadObject(ms) :?> SchemaCache) with IsOffline = true }
// Schema files saved before 1.5.34 may carry individuals in the schema file itself: they're kept.
let loaded = ser.ReadObject(ms) :?> SchemaCache
{ loaded with
IsOffline = true
Individuals = if isNull loaded.Individuals then ConcurrentDictionary<string, _>() else loaded.Individuals
TableDescriptions = if isNull loaded.TableDescriptions then ConcurrentDictionary<string,string>() else loaded.TableDescriptions
ColumnDescriptions = if isNull loaded.ColumnDescriptions then ConcurrentDictionary<string,ConcurrentDictionary<string,string>>() else loaded.ColumnDescriptions }
static member LoadOrEmpty(filePath) =
if String.IsNullOrEmpty(filePath) || (not(File.Exists filePath)) then
if String.IsNullOrEmpty(filePath) || (not(File.Exists filePath)) then
SchemaCache.Empty
else
SchemaCache.Load(filePath)
member this.Save(filePath) =
static member private BesideFilePath(filePath:string, kind:string) =
let ext = Path.GetExtension filePath
Path.Combine(Path.GetDirectoryName filePath, Path.GetFileNameWithoutExtension filePath + "." + kind + ext)
/// Individuals are saved beside the schema file: schema.json -> schema.individuals.json
static member IndividualsFilePath(filePath:string) = SchemaCache.BesideFilePath(filePath, "individuals")
/// Descriptions are saved beside the schema file: schema.json -> schema.descriptions.json
static member DescriptionsFilePath(filePath:string) = SchemaCache.BesideFilePath(filePath, "descriptions")
static member private WriteJson(filePath, value:obj) =
use ms = new MemoryStream()
let ser = System.Runtime.Serialization.Json.DataContractJsonSerializer(this.GetType())
ser.WriteObject(ms, { this with IsOffline = true });
let json = ms.ToArray();
let ser = System.Runtime.Serialization.Json.DataContractJsonSerializer(value.GetType())
ser.WriteObject(ms, value)
let json = ms.ToArray()
File.WriteAllText(filePath, Encoding.UTF8.GetString(json, 0, json.Length))
static member private ReadJson<'T>(filePath) =
use ms = new MemoryStream(Encoding.UTF8.GetBytes(File.ReadAllText filePath))
System.Runtime.Serialization.Json.DataContractJsonSerializer(typeof<'T>).ReadObject(ms) :?> 'T
/// Saves the schema, and the descriptions to their own file. Individuals are not in the schema file:
/// SaveIndividuals saves them to their own file. Individuals read from saved files (offline mode) are saved to
/// the individuals file, so a re-save doesn't lose them.
member this.Save(filePath) =
// The empty Individuals is kept so that older SQLProvider versions can read the file.
SchemaCache.WriteJson(filePath, { this with IsOffline = true; Individuals = ConcurrentDictionary<string, _>() })
this.SaveDescriptions filePath |> ignore
if this.IsOffline && this.HasIndividuals then
// An individuals file already there is newer than individuals of an old schema file
this.LoadIndividuals filePath
this.SaveIndividuals filePath |> ignore
member this.HasDescriptions =
this.TableDescriptions.Count > 0 || this.ColumnDescriptions.Count > 0
/// Saves the descriptions to the descriptions file beside the schema file and returns its path; None when there
/// are none. Descriptions already in the file are kept unless read again, so a save of the parts used so far
/// doesn't drop the others.
member this.SaveDescriptions(filePath) =
this.LoadDescriptions filePath
if not this.HasDescriptions then None else
let descriptionsFile = SchemaCache.DescriptionsFilePath filePath
SchemaCache.WriteJson(descriptionsFile, { SchemaDescriptions.TableComments = this.TableDescriptions; ColumnComments = this.ColumnDescriptions })
Some descriptionsFile
/// Reads the descriptions file beside the schema file, if there is one. Descriptions already read are kept.
member this.LoadDescriptions(filePath) =
if not(String.IsNullOrEmpty filePath) then
let descriptionsFile = SchemaCache.DescriptionsFilePath filePath
if File.Exists descriptionsFile then
let loaded = SchemaCache.ReadJson<SchemaDescriptions> descriptionsFile
if not (isNull loaded.TableComments) then
for KeyValue(table, description) in loaded.TableComments do
this.TableDescriptions.TryAdd(table, description) |> ignore
if not (isNull loaded.ColumnComments) then
for KeyValue(table, columns) in loaded.ColumnComments do
let known = this.ColumnDescriptions.GetOrAdd(table, fun _ -> ConcurrentDictionary<string,string>())
for KeyValue(column, description) in columns do
known.TryAdd(column, description) |> ignore
member this.HasIndividuals =
this.Individuals |> Seq.exists (fun kv -> not (Array.isEmpty kv.Value))
/// A column value as the individuals file can hold it. The serializer writes only some types when the value
/// is an obj: one DateTimeOffset or string array would fail the whole file. Offline the values of other
/// columns than the primary key are only shown as text, so the others are saved as their text.
static member private IndividualsFileValue(value:obj) : obj =
match value with
| null -> null
| :? string | :? bool | :? char | :? decimal | :? DateTime | :? Guid | :? TimeSpan | :? (byte[]) -> value
| _ when value.GetType().IsPrimitive -> value
| :? Array as items -> box [| for item in items -> SchemaCache.IndividualsFileValue item |]
| _ -> box (value.ToString())
/// Saves the individuals to the individuals file beside the schema file and returns its path.
/// Tables without individuals are left out.
member this.SaveIndividuals(filePath) =
let individualsFile = SchemaCache.IndividualsFilePath filePath
let toSave = ConcurrentDictionary<string, (obj * IDictionary<string,obj>) array>()
for KeyValue(table, entities) in this.Individuals do
if not (Array.isEmpty entities) then
toSave.[table] <-
entities |> Array.map (fun (pk, columns) ->
SchemaCache.IndividualsFileValue pk,
dict [ for KeyValue(column, value) in columns -> column, SchemaCache.IndividualsFileValue value ])
SchemaCache.WriteJson(individualsFile, toSave)
individualsFile
/// Reads the individuals file beside the schema file, if there is one. Its individuals replace the same tables' older ones.
member this.LoadIndividuals(filePath) =
if not(String.IsNullOrEmpty filePath) then
let individualsFile = SchemaCache.IndividualsFilePath filePath
if File.Exists individualsFile then
let loaded = SchemaCache.ReadJson<ConcurrentDictionary<string, (obj * IDictionary<string,obj>) array>> individualsFile
if not (isNull loaded) then
for KeyValue(table, entities) in loaded do
this.Individuals.[table] <- entities

/// Table and column descriptions as saved in the descriptions file beside the schema file
and SchemaDescriptions =
{ /// Table name to description
TableComments : ConcurrentDictionary<string,string>
/// Table name to column name to description
ColumnComments : ConcurrentDictionary<string,ConcurrentDictionary<string,string>> }

/// A provider that can read the descriptions of all the columns of a table in one query implements this too.
/// Saving the descriptions of a whole database is then one query per table, not one per column.
type IColumnDescriptionsProvider =
/// Column name to description, for the columns of the table that have a description.
/// The descriptions are the same as ISqlProvider.GetColumnDescription gives.
abstract GetColumnDescriptions : IDbConnection * string -> IDictionary<string,string>

/// GroupResultItems is an item to create key-igrouping-structure.
/// From the select group-by projection, aggregate operations like Enumerable.Count()
Expand Down Expand Up @@ -1243,8 +1351,15 @@ module public OfflineTools =
else
if File.Exists targetfile then
File.Delete targetfile
for besideFile in [ SchemaCache.IndividualsFilePath targetfile; SchemaCache.DescriptionsFilePath targetfile ] do
if File.Exists besideFile then
File.Delete besideFile
let s1 = SchemaCache.Load sourcefile1
let s2 = SchemaCache.Load sourcefile2
s1.LoadIndividuals sourcefile1
s2.LoadIndividuals sourcefile2
s1.LoadDescriptions sourcefile1
s2.LoadDescriptions sourcefile2
let merged =
{ PrimaryKeys = ConcurrentDictionary(
Seq.concat [|s1.PrimaryKeys ; s2.PrimaryKeys |] |> Seq.distinctBy(fun d -> d.Key));
Expand All @@ -1267,7 +1382,14 @@ module public OfflineTools =
Packages = ResizeArray(Seq.concat [| s1.Packages ; s2.Packages |] |> Seq.distinctBy(fun s -> s.ToString()));
Individuals = ConcurrentDictionary(
Seq.concat [|s1.Individuals ; s2.Individuals |] |> Seq.distinctBy(fun d -> d.Key));
IsOffline = s1.IsOffline || s2.IsOffline}
IsOffline = s1.IsOffline || s2.IsOffline
TableDescriptions = ConcurrentDictionary(
Seq.concat [|s1.TableDescriptions ; s2.TableDescriptions |] |> Seq.distinctBy(fun d -> d.Key))
ColumnDescriptions = ConcurrentDictionary(
Seq.concat [|s1.ColumnDescriptions ; s2.ColumnDescriptions |]
|> Seq.groupBy(fun d -> d.Key)
|> Seq.map(fun (table, descs) ->
KeyValuePair(table, ConcurrentDictionary(descs |> Seq.collect(fun d -> d.Value) |> Seq.distinctBy(fun c -> c.Key))))) }
merged.Save targetfile
"Merge saved " + targetfile + " at " + DateTime.Now.ToString("hh:mm:ss")

Expand Down
12 changes: 12 additions & 0 deletions src/SQLProvider.Common/Utils.fs
Original file line number Diff line number Diff line change
Expand Up @@ -849,6 +849,18 @@ module Sql =
let inline dbUnboxWithDefault<'a> def (v:obj) : 'a =
if Convert.IsDBNull(v) then def else unbox v

/// Reads rows of a name and its description to a dictionary. Rows without a description are left out,
/// and of rows with the same name the first one counts.
let readDescriptions (reader:IDataReader) =
let descriptions = System.Collections.Generic.Dictionary<string,string>()
while reader.Read() do
if not (reader.IsDBNull 0) && not (reader.IsDBNull 1) then
let name = reader.GetValue(0).ToString()
let description = reader.GetValue(1).ToString()
if description <> "" && not (descriptions.ContainsKey name) then
descriptions.[name] <- description
descriptions :> System.Collections.Generic.IDictionary<string,string>

/// Note: SQLProvider reuses the connection through multiple instances, so you can't dispose it here.
/// Instead it's created with ISQLProvider's CreateConnection method, and that is having always "use" to ensure it is disposed properly on "finally".
let connect (con:IDbConnection) f =
Expand Down
Loading
Loading