diff --git a/docs/RELEASE_NOTES.md b/docs/RELEASE_NOTES.md index 8e40f4f1..e493942c 100644 --- a/docs/RELEASE_NOTES.md +++ b/docs/RELEASE_NOTES.md @@ -1,3 +1,16 @@ +### 1.5.34 - (unreleased) +- Offline mode (ContextSchemaPath) saves three files beside each other: schema.json for the schema, schema.descriptions.json for table and column descriptions (comments), schema.individuals.json for individuals. Builds read only the schema file; the descriptions file is read for the first tooltip and the individuals file when Individuals are first opened +- Offline tooltips show the saved table and column descriptions, instead of only "Offline mode." SaveContextSchema saves the descriptions of what you hovered over online +- A saved schema no longer carries individuals, so it stays small and loads fast. A schema file with individuals in it still loads them, and re-saving it moves them to the individuals file. Older versions can read the new schema files +- New design-time commands next to SaveContextSchema, online only, that read the whole database instead of what was used so far: SaveContextSchemaFull (every table, column, relationship and stored procedure, and every table and column description) and SaveContextSchemaIndividuals (individuals of every table with a single primary key) +- SaveContextSchemaFull reads the column descriptions of a table in one query instead of one per column (MS SQL Server, PostgreSQL, MySQL, Oracle, Firebird, DuckDB) +- OfflineTools.mergeCacheFiles merges descriptions and individuals files too +- Fix: after a failed SaveContextSchema, design-time commands no longer stop working until the IDE restarts +- Fix: saving individuals no longer fails on a column value such as a DateTimeOffset or an array; such values are saved as text +- Fix: PostgreSQL tables with capital letters or an apostrophe in their name show their descriptions in tooltips; reading them failed before +- Fix: MS SQL Server descriptions of a table come from its own schema; with tables of the same name in two schemas, either could show the other's +- Fix: Oracle descriptions of tables owned by another schema (Owner) are read; only the current user's tables had them before + ### 1.5.33 - 19.11.2026 - Fix: a saved schema (ContextSchemaPath) loads on both .NET and .NET Framework, whichever saved it. Visual Studio, whose editor and compiler run on .NET Framework, failed on a schema saved by dotnet with "Value cannot be null. Parameter name: key" diff --git a/docs/content/core/parameters.fsx b/docs/content/core/parameters.fsx index e98876be..327d1a96 100644 --- a/docs/content/core/parameters.fsx +++ b/docs/content/core/parameters.fsx @@ -160,7 +160,45 @@ is saved, the outcome of the method execution is stored in memory so the file wi In case the database schema changes and the schema file must be updated, remove the outdated file, reload the solution and retype or uncomment a call to `SaveContextSchema` to regenerate the schema file. +The offline mode uses up to three files beside each other: + +| File | Contents | Read | +|---|---|---| +| `schema.json` (`ContextSchemaPath`) | Tables, columns, relationships, stored procedures | Every build | +| `schema.descriptions.json` | Table and column descriptions (comments), shown in tooltips | For the first tooltip | +| `schema.individuals.json` | Sample rows of the [Individuals](individuals.html) feature | When `Individuals` is first opened | + +Builds read only the schema file, so descriptions and individuals don't slow them down. + +To save the whole database instead of the parts used so far, use these commands while online (before the schema file +exists). Each reads the whole database and may take a while on a large one: + +```fsharp +// Every table, column, relationship and stored procedure, to schema.json, and every table +// and column description, to schema.descriptions.json. No rows of data. +ctx.``Design Time Commands``.SaveContextSchemaFull +// IndividualsAmount individuals of every table with a single primary key, to schema.individuals.json +ctx.``Design Time Commands``.SaveContextSchemaIndividuals +``` + +These two commands exist only while online. Remove the line after saving: once the schema file exists, the +solution is offline and a line that still calls one of them doesn't compile. + +Individuals are rows of your data, so mind GDPR/PII before committing the individuals file to version control. + +When the database schema changes, remove all three files, not only the schema file, before saving again. The +descriptions and individuals files are otherwise kept as they are, and may then describe tables and columns that +no longer exist. + +`SaveContextSchema` also saves the descriptions of the tables and columns you hovered over, and keeps the descriptions +already in the descriptions file. Offline, a table or column whose description wasn't saved shows "Offline mode." in +its tooltip. + +Schema files saved by older versions with individuals in them still work: re-saving such a file moves its individuals +to the individuals file. + There is a tool method FSharp.Data.Sql.Common.OfflineTools.mergeCacheFiles to merge multiple files together. +It merges their descriptions and individuals files as well. *) diff --git a/src/SQLProvider.Common/SqlRuntime.Common.fs b/src/SQLProvider.Common/SqlRuntime.Common.fs index 8c43ac72..ebf4eab5 100644 --- a/src/SQLProvider.Common/SqlRuntime.Common.fs +++ b/src/SQLProvider.Common/SqlRuntime.Common.fs @@ -932,7 +932,13 @@ and SchemaCache = SprocsParams : ConcurrentDictionary //sproc name and params Packages : ResizeArray Individuals : ConcurrentDictionary) array> //table name and entities - IsOffline : bool } + IsOffline : bool + /// Table descriptions (comments) for tooltips, by table name. Saved in their own file (SaveDescriptions), not in the schema file. + [] + TableDescriptions : ConcurrentDictionary + /// Column descriptions (comments) for tooltips, by table name and column name. Saved like TableDescriptions. + [] + ColumnDescriptions : ConcurrentDictionary> } with static member Empty = { PrimaryKeys = ConcurrentDictionary() @@ -943,22 +949,124 @@ and SchemaCache = SprocsParams = ConcurrentDictionary() Packages = ResizeArray() Individuals = ConcurrentDictionary() - IsOffline = false } + IsOffline = false + TableDescriptions = ConcurrentDictionary() + ColumnDescriptions = ConcurrentDictionary>() } static member Load(filePath) = use ms = new MemoryStream(Encoding.UTF8.GetBytes(File.ReadAllText(filePath))) let ser = System.Runtime.Serialization.Json.DataContractJsonSerializer(typeof) - { (ser.ReadObject(ms) :?> SchemaCache) with IsOffline = true } + // Schema files saved before 1.5.34 may carry individuals in the schema file itself: they're kept. + let loaded = ser.ReadObject(ms) :?> SchemaCache + { loaded with + IsOffline = true + Individuals = if isNull loaded.Individuals then ConcurrentDictionary() else loaded.Individuals + TableDescriptions = if isNull loaded.TableDescriptions then ConcurrentDictionary() else loaded.TableDescriptions + ColumnDescriptions = if isNull loaded.ColumnDescriptions then ConcurrentDictionary>() else loaded.ColumnDescriptions } static member LoadOrEmpty(filePath) = - if String.IsNullOrEmpty(filePath) || (not(File.Exists filePath)) then + if String.IsNullOrEmpty(filePath) || (not(File.Exists filePath)) then SchemaCache.Empty else SchemaCache.Load(filePath) - member this.Save(filePath) = + static member private BesideFilePath(filePath:string, kind:string) = + let ext = Path.GetExtension filePath + Path.Combine(Path.GetDirectoryName filePath, Path.GetFileNameWithoutExtension filePath + "." + kind + ext) + /// Individuals are saved beside the schema file: schema.json -> schema.individuals.json + static member IndividualsFilePath(filePath:string) = SchemaCache.BesideFilePath(filePath, "individuals") + /// Descriptions are saved beside the schema file: schema.json -> schema.descriptions.json + static member DescriptionsFilePath(filePath:string) = SchemaCache.BesideFilePath(filePath, "descriptions") + static member private WriteJson(filePath, value:obj) = use ms = new MemoryStream() - let ser = System.Runtime.Serialization.Json.DataContractJsonSerializer(this.GetType()) - ser.WriteObject(ms, { this with IsOffline = true }); - let json = ms.ToArray(); + let ser = System.Runtime.Serialization.Json.DataContractJsonSerializer(value.GetType()) + ser.WriteObject(ms, value) + let json = ms.ToArray() File.WriteAllText(filePath, Encoding.UTF8.GetString(json, 0, json.Length)) + static member private ReadJson<'T>(filePath) = + use ms = new MemoryStream(Encoding.UTF8.GetBytes(File.ReadAllText filePath)) + System.Runtime.Serialization.Json.DataContractJsonSerializer(typeof<'T>).ReadObject(ms) :?> 'T + /// Saves the schema, and the descriptions to their own file. Individuals are not in the schema file: + /// SaveIndividuals saves them to their own file. Individuals read from saved files (offline mode) are saved to + /// the individuals file, so a re-save doesn't lose them. + member this.Save(filePath) = + // The empty Individuals is kept so that older SQLProvider versions can read the file. + SchemaCache.WriteJson(filePath, { this with IsOffline = true; Individuals = ConcurrentDictionary() }) + this.SaveDescriptions filePath |> ignore + if this.IsOffline && this.HasIndividuals then + // An individuals file already there is newer than individuals of an old schema file + this.LoadIndividuals filePath + this.SaveIndividuals filePath |> ignore + member this.HasDescriptions = + this.TableDescriptions.Count > 0 || this.ColumnDescriptions.Count > 0 + /// Saves the descriptions to the descriptions file beside the schema file and returns its path; None when there + /// are none. Descriptions already in the file are kept unless read again, so a save of the parts used so far + /// doesn't drop the others. + member this.SaveDescriptions(filePath) = + this.LoadDescriptions filePath + if not this.HasDescriptions then None else + let descriptionsFile = SchemaCache.DescriptionsFilePath filePath + SchemaCache.WriteJson(descriptionsFile, { SchemaDescriptions.TableComments = this.TableDescriptions; ColumnComments = this.ColumnDescriptions }) + Some descriptionsFile + /// Reads the descriptions file beside the schema file, if there is one. Descriptions already read are kept. + member this.LoadDescriptions(filePath) = + if not(String.IsNullOrEmpty filePath) then + let descriptionsFile = SchemaCache.DescriptionsFilePath filePath + if File.Exists descriptionsFile then + let loaded = SchemaCache.ReadJson descriptionsFile + if not (isNull loaded.TableComments) then + for KeyValue(table, description) in loaded.TableComments do + this.TableDescriptions.TryAdd(table, description) |> ignore + if not (isNull loaded.ColumnComments) then + for KeyValue(table, columns) in loaded.ColumnComments do + let known = this.ColumnDescriptions.GetOrAdd(table, fun _ -> ConcurrentDictionary()) + for KeyValue(column, description) in columns do + known.TryAdd(column, description) |> ignore + member this.HasIndividuals = + this.Individuals |> Seq.exists (fun kv -> not (Array.isEmpty kv.Value)) + /// A column value as the individuals file can hold it. The serializer writes only some types when the value + /// is an obj: one DateTimeOffset or string array would fail the whole file. Offline the values of other + /// columns than the primary key are only shown as text, so the others are saved as their text. + static member private IndividualsFileValue(value:obj) : obj = + match value with + | null -> null + | :? string | :? bool | :? char | :? decimal | :? DateTime | :? Guid | :? TimeSpan | :? (byte[]) -> value + | _ when value.GetType().IsPrimitive -> value + | :? Array as items -> box [| for item in items -> SchemaCache.IndividualsFileValue item |] + | _ -> box (value.ToString()) + /// Saves the individuals to the individuals file beside the schema file and returns its path. + /// Tables without individuals are left out. + member this.SaveIndividuals(filePath) = + let individualsFile = SchemaCache.IndividualsFilePath filePath + let toSave = ConcurrentDictionary) array>() + for KeyValue(table, entities) in this.Individuals do + if not (Array.isEmpty entities) then + toSave.[table] <- + entities |> Array.map (fun (pk, columns) -> + SchemaCache.IndividualsFileValue pk, + dict [ for KeyValue(column, value) in columns -> column, SchemaCache.IndividualsFileValue value ]) + SchemaCache.WriteJson(individualsFile, toSave) + individualsFile + /// Reads the individuals file beside the schema file, if there is one. Its individuals replace the same tables' older ones. + member this.LoadIndividuals(filePath) = + if not(String.IsNullOrEmpty filePath) then + let individualsFile = SchemaCache.IndividualsFilePath filePath + if File.Exists individualsFile then + let loaded = SchemaCache.ReadJson) array>> individualsFile + if not (isNull loaded) then + for KeyValue(table, entities) in loaded do + this.Individuals.[table] <- entities + +/// Table and column descriptions as saved in the descriptions file beside the schema file +and SchemaDescriptions = + { /// Table name to description + TableComments : ConcurrentDictionary + /// Table name to column name to description + ColumnComments : ConcurrentDictionary> } + +/// A provider that can read the descriptions of all the columns of a table in one query implements this too. +/// Saving the descriptions of a whole database is then one query per table, not one per column. +type IColumnDescriptionsProvider = + /// Column name to description, for the columns of the table that have a description. + /// The descriptions are the same as ISqlProvider.GetColumnDescription gives. + abstract GetColumnDescriptions : IDbConnection * string -> IDictionary /// GroupResultItems is an item to create key-igrouping-structure. /// From the select group-by projection, aggregate operations like Enumerable.Count() @@ -1243,8 +1351,15 @@ module public OfflineTools = else if File.Exists targetfile then File.Delete targetfile + for besideFile in [ SchemaCache.IndividualsFilePath targetfile; SchemaCache.DescriptionsFilePath targetfile ] do + if File.Exists besideFile then + File.Delete besideFile let s1 = SchemaCache.Load sourcefile1 let s2 = SchemaCache.Load sourcefile2 + s1.LoadIndividuals sourcefile1 + s2.LoadIndividuals sourcefile2 + s1.LoadDescriptions sourcefile1 + s2.LoadDescriptions sourcefile2 let merged = { PrimaryKeys = ConcurrentDictionary( Seq.concat [|s1.PrimaryKeys ; s2.PrimaryKeys |] |> Seq.distinctBy(fun d -> d.Key)); @@ -1267,7 +1382,14 @@ module public OfflineTools = Packages = ResizeArray(Seq.concat [| s1.Packages ; s2.Packages |] |> Seq.distinctBy(fun s -> s.ToString())); Individuals = ConcurrentDictionary( Seq.concat [|s1.Individuals ; s2.Individuals |] |> Seq.distinctBy(fun d -> d.Key)); - IsOffline = s1.IsOffline || s2.IsOffline} + IsOffline = s1.IsOffline || s2.IsOffline + TableDescriptions = ConcurrentDictionary( + Seq.concat [|s1.TableDescriptions ; s2.TableDescriptions |] |> Seq.distinctBy(fun d -> d.Key)) + ColumnDescriptions = ConcurrentDictionary( + Seq.concat [|s1.ColumnDescriptions ; s2.ColumnDescriptions |] + |> Seq.groupBy(fun d -> d.Key) + |> Seq.map(fun (table, descs) -> + KeyValuePair(table, ConcurrentDictionary(descs |> Seq.collect(fun d -> d.Value) |> Seq.distinctBy(fun c -> c.Key))))) } merged.Save targetfile "Merge saved " + targetfile + " at " + DateTime.Now.ToString("hh:mm:ss") diff --git a/src/SQLProvider.Common/Utils.fs b/src/SQLProvider.Common/Utils.fs index c9698563..d6d87f1d 100644 --- a/src/SQLProvider.Common/Utils.fs +++ b/src/SQLProvider.Common/Utils.fs @@ -849,6 +849,18 @@ module Sql = let inline dbUnboxWithDefault<'a> def (v:obj) : 'a = if Convert.IsDBNull(v) then def else unbox v + /// Reads rows of a name and its description to a dictionary. Rows without a description are left out, + /// and of rows with the same name the first one counts. + let readDescriptions (reader:IDataReader) = + let descriptions = System.Collections.Generic.Dictionary() + while reader.Read() do + if not (reader.IsDBNull 0) && not (reader.IsDBNull 1) then + let name = reader.GetValue(0).ToString() + let description = reader.GetValue(1).ToString() + if description <> "" && not (descriptions.ContainsKey name) then + descriptions.[name] <- description + descriptions :> System.Collections.Generic.IDictionary + /// Note: SQLProvider reuses the connection through multiple instances, so you can't dispose it here. /// Instead it's created with ISQLProvider's CreateConnection method, and that is having always "use" to ensure it is disposed properly on "finally". let connect (con:IDbConnection) f = diff --git a/src/SQLProvider.DesignTime/SqlDesignTime.fs b/src/SQLProvider.DesignTime/SqlDesignTime.fs index c000e90e..abb81eb0 100644 --- a/src/SQLProvider.DesignTime/SqlDesignTime.fs +++ b/src/SQLProvider.DesignTime/SqlDesignTime.fs @@ -139,19 +139,47 @@ module DesignTimeUtils = let transactionOptions = TransactionOptions.Default - let createIndividualsType (con:IDbConnection option) (prov:ISqlProvider) (table:Table) (designTimeDc:Lazy<_>) dbVendor individualsAmount tableTypeDef = - let t = ProvidedTypeDefinition($"{table.Schema}.{table.Name}.Individuals", Some typeof, isErased=true) - let individualsTypes = ResizeArray<_>() - individualsTypes.Add t - - t.AddXmlDocDelayed(fun _ -> $"A sample of %s{table.Name} individuals from the SQL object as supplied in the static parameters") - t.AddMember(ProvidedConstructor([ProvidedParameter("dataContext", typeof)], empty)) - t.AddMembersDelayed( fun _ -> + /// A table's description (comment). Online it's read from the database and kept in the schema cache, so a + /// save keeps it; offline it comes from the descriptions file, read on first use. None when it wasn't saved. + let tableDescription (con:IDbConnection option) (prov:ISqlProvider) (offlineDescriptions:Lazy) (tableName:string) = + let descriptions = prov.GetSchemaCache().TableDescriptions + match con with + | Some con -> + let description = match prov.GetTableDescription(con, tableName) with null -> "" | d -> d + descriptions.[tableName] <- description + Some description + | None -> + offlineDescriptions.Force() + match descriptions.TryGetValue tableName with + | true, description -> Some description + | false, _ -> None + + /// A column's description (comment), kept in the schema cache like tableDescription. + let columnDescription (con:IDbConnection option) (prov:ISqlProvider) (offlineDescriptions:Lazy) (tableName:string) (columnName:string) = + let descriptions = prov.GetSchemaCache().ColumnDescriptions + match con with + | Some con -> + let description = match prov.GetColumnDescription(con, tableName, columnName) with null -> "" | d -> d + descriptions.GetOrAdd(tableName, fun _ -> System.Collections.Concurrent.ConcurrentDictionary()).[columnName] <- description + Some description + | None -> + offlineDescriptions.Force() + match descriptions.TryGetValue tableName with + | true, columns -> + match columns.TryGetValue columnName with + | true, description -> Some description + | false, _ -> None + | false, _ -> None + + /// The primary key and individuals of a table with a single primary key. Read from the database once and kept + /// in the schema cache; offline they come from the individuals file. + let getIndividuals (con:IDbConnection option) (prov:ISqlProvider) (table:Table) (designTimeDc:Lazy<_>) (offlineIndividuals:Lazy) dbVendor individualsAmount = let columns = match con with | Some con -> prov.GetColumns(con,table) | None -> prov.GetSchemaCache().Columns.TryGetValue(table.FullName) |> function | true,cols -> cols | false, _ -> Map.empty match prov.GetPrimaryKey table with + | None -> None | Some pkName -> let rec (|FixedType|_|) (o:obj) = @@ -169,6 +197,7 @@ module DesignTimeUtils = | _, _ -> None let dcDone = designTimeDc.Force() + if con.IsNone then offlineIndividuals.Force() let entities = prov.GetSchemaCache().Individuals.GetOrAdd(($"{table.FullName}_{pkName}"), fun k -> match con with @@ -186,6 +215,18 @@ module DesignTimeUtils = mapped | None -> [||] ) + Some (pkName, entities) + + let createIndividualsType (con:IDbConnection option) (prov:ISqlProvider) (table:Table) (designTimeDc:Lazy<_>) (offlineIndividuals:Lazy) dbVendor individualsAmount tableTypeDef = + let t = ProvidedTypeDefinition($"{table.Schema}.{table.Name}.Individuals", Some typeof, isErased=true) + let individualsTypes = ResizeArray<_>() + individualsTypes.Add t + + t.AddXmlDocDelayed(fun _ -> $"A sample of %s{table.Name} individuals from the SQL object as supplied in the static parameters") + t.AddMember(ProvidedConstructor([ProvidedParameter("dataContext", typeof)], empty)) + t.AddMembersDelayed( fun _ -> + match getIndividuals con prov table designTimeDc offlineIndividuals dbVendor individualsAmount with + | Some (pkName, entities) -> if Array.isEmpty entities then [] else // for each column in the entity except the primary key, create a new type that will read ``As Column 1`` etc // inside that type the individuals will be listed again but with the text for the relevant column as the name @@ -227,13 +268,16 @@ module DesignTimeUtils = // this next bit is just side effect to populate the "As Column" types for the supported columns for colName, colValue in columnValues |> Seq.map(fun kvp -> kvp.Key, kvp.Value) do if colName <> pkName then - let colDefinition, _ = propertyMap.[colName] - colDefinition.AddMemberDelayed(fun() -> - ProvidedProperty( sprintf "%s, %s" (prettyPrint pkValue) (prettyPrint colValue) - , tableTypeDef - , getterCode = getterCode - ) - ) + // An individuals file saved before the schema changed may have columns the schema doesn't + match propertyMap.TryFind colName with + | Some (colDefinition, _) -> + colDefinition.AddMemberDelayed(fun() -> + ProvidedProperty( sprintf "%s, %s" (prettyPrint pkValue) (prettyPrint colValue) + , tableTypeDef + , getterCode = getterCode + ) + ) + | None -> () // return the primary key property Some <| ProvidedProperty(prettyPrint pkValue , tableTypeDef @@ -252,7 +296,7 @@ module DesignTimeUtils = | None -> []) individualsTypes :> seq<_> - let createColumnProperty (con:IDbConnection option) (prov:ISqlProvider) useOptionTypes key (c:Column) = + let createColumnProperty (con:IDbConnection option) (prov:ISqlProvider) (offlineDescriptions:Lazy) useOptionTypes key (c:Column) = let nullable = if c.IsNullable then useOptionTypes else NullableColumnType.NO_OPTION let ty = Utilities.getType c.TypeMapping.ClrType @@ -296,15 +340,13 @@ module DesignTimeUtils = let nfo = c.TypeInfo let typeInfo = match nfo with ValueNone -> "" | ValueSome x -> x.ToString() - match con with - | Some con -> - prop.AddXmlDocDelayed(fun () -> - let details = prov.GetColumnDescription(con, key, name).Replace("<","<").Replace(">",">") + prop.AddXmlDocDelayed(fun () -> + match columnDescription con prov offlineDescriptions key name with + | Some description -> + let details = description.Replace("<","<").Replace(">",">") let separator = if (String.IsNullOrWhiteSpace typeInfo) || (String.IsNullOrWhiteSpace details) then "" else "/" - sprintf "%s %s %s" (String.concat ": " [|name; details|]) separator typeInfo) - | None -> - prop.AddXmlDocDelayed(fun () -> $"Offline mode. %s{typeInfo}") - () + sprintf "%s %s %s" (String.concat ": " [|name; details|]) separator typeInfo + | None -> $"Offline mode. %s{typeInfo}") prop let generateSprocMethod (container:ProvidedTypeDefinition) (con:IDbConnection option) (prov:ISqlProvider) (sproc:CompileTimeSprocDefinition) = @@ -469,66 +511,93 @@ module DesignTimeUtils = |> Seq.map snd | sproc::rest -> generateTypeTree con prov (walkSproc con prov [] None createdTypes sproc) rest - let createDesignTimeCommands (prov:ISqlProvider) contextSchemaPath recreate (invalidate: _ -> Unit)= + let createDesignTimeCommands (prov:ISqlProvider) contextSchemaPath recreate (invalidate: _ -> Unit) (onlineCommands: (string * string * (unit -> string)) list) = let designTimeCommandsContainer = ProvidedTypeDefinition("DesignTimeCommands", Some typeof, isErased=true) let designTime = ProvidedProperty("Design Time Commands", designTimeCommandsContainer, getterCode = empty) designTime.AddXmlDocDelayed(fun () -> "Developer's design time commands to TypeProvider.") + /// Runs a design-time command, unless one is already running (None then): another thread holds the lock, + /// or the command itself asks for a command on this thread (the lock is re-entrant, so the flag tells). + /// The lock and the flag are released even if the command fails. + let runCommand (command: unit -> string) = + if not (System.Threading.Monitor.TryEnter mySaveLock) then None else + try + if saveInProcess then None else + saveInProcess <- true + try Some (command()) + finally saveInProcess <- false + finally System.Threading.Monitor.Exit mySaveLock + + let commandResult result = + match result with + | Some message -> [ ProvidedProperty(message, typeof, getterCode = empty) :> MemberInfo ] + | None -> [] + let saveResponse = ProvidedTypeDefinition("SaveContextResponse", Some typeof, isErased=true) saveResponse.AddMember(ProvidedConstructor([], empty)) saveResponse.AddMembersDelayed(fun () -> - if not saveInProcess then - let result = - if not(String.IsNullOrEmpty contextSchemaPath) then - try - lock mySaveLock (fun() -> - saveInProcess <- true - prov.GetSchemaCache().Save contextSchemaPath - saveInProcess <- false - "Saved " + contextSchemaPath + " at " + DateTime.Now.ToString("hh:mm:ss") - ) - with - | e -> "Save failed: " + e.Message - else "ContextSchemaPath is not defined" - [ ProvidedProperty(result,typeof, getterCode = empty) :> MemberInfo ] - else [] + if String.IsNullOrEmpty contextSchemaPath then commandResult (Some "ContextSchemaPath is not defined") else + runCommand (fun () -> + try + prov.GetSchemaCache().Save contextSchemaPath + "Saved " + contextSchemaPath + " at " + DateTime.Now.ToString("hh:mm:ss") + with + | e -> "Save failed: " + e.Message) + |> commandResult ) let m = ProvidedProperty("SaveContextSchema", (saveResponse :> Type), getterCode = empty) let mOld = ProvidedMethod("SaveContextSchema", [], (saveResponse :> Type), invokeCode = empty) m.AddXmlDocComputed(fun () -> if String.IsNullOrEmpty contextSchemaPath then "ContextSchemaPath static parameter has to be defined to use this function." - else "Schema location: " + contextSchemaPath + ". Write dot after SaveContextSchema to save the schema at design time." + else "Schema location: " + contextSchemaPath + ". Write dot after SaveContextSchema to save the schema used so far at design time, " + + "and the descriptions read so far to " + SchemaCache.DescriptionsFilePath contextSchemaPath + ". " + + "Online, SaveContextSchemaFull saves the whole database schema and SaveContextSchemaIndividuals the individuals." ) let expirationMessage = "Expired, moved under: .``Design Time Commands``" mOld.AddXmlDocComputed(fun () -> expirationMessage) mOld.AddObsoleteAttribute expirationMessage + /// Adds a command: a property whose type gets its member when you type a dot after it. Computing the member + /// runs the command, and the member's name tells how it went. + let addCommand name (tooltip: string) (command: unit -> string) = + let response = ProvidedTypeDefinition(name + "Response", Some typeof, isErased=true) + response.AddMember(ProvidedConstructor([], empty)) + response.AddMembersDelayed(fun () -> + runCommand (fun () -> + try command() + with e -> name + " failed: " + e.Message) + |> commandResult) + let property = ProvidedProperty(name, (response :> Type), getterCode = empty) + property.AddXmlDocComputed(fun () -> tooltip) + designTimeCommandsContainer.AddMember response + designTimeCommandsContainer.AddMember property + + for name, tooltip, command in onlineCommands do + addCommand name tooltip command + // ClearDatabaseSchemaCache only in online-mode if String.IsNullOrEmpty contextSchemaPath then let invalidateActionResponse = ProvidedTypeDefinition("InvalidateResponse", Some typeof, isErased=true) invalidateActionResponse.AddMember(ProvidedConstructor([], empty)) invalidateActionResponse.AddMembersDelayed(fun () -> - if not saveInProcess then - let result = - lock mySaveLock (fun() -> - saveInProcess <- true - let schemacache = prov.GetSchemaCache() - schemacache.PrimaryKeys.Clear() - schemacache.Tables.Clear() - schemacache.Columns.Clear() - schemacache.Relationships.Clear() - schemacache.Sprocs.Clear() - schemacache.SprocsParams.Clear() - schemacache.Packages.Clear() - schemacache.Individuals.Clear() - invalidate() - let pf = recreate() - saveInProcess <- false - "Database schema cache cleared.") - [ProvidedProperty(result,typeof, getterCode = empty) :> MemberInfo] - else [] + runCommand (fun () -> + let schemacache = prov.GetSchemaCache() + schemacache.PrimaryKeys.Clear() + schemacache.Tables.Clear() + schemacache.Columns.Clear() + schemacache.Relationships.Clear() + schemacache.Sprocs.Clear() + schemacache.SprocsParams.Clear() + schemacache.Packages.Clear() + schemacache.Individuals.Clear() + schemacache.TableDescriptions.Clear() + schemacache.ColumnDescriptions.Clear() + invalidate() + recreate() + "Database schema cache cleared.") + |> commandResult ) let m2 = ProvidedProperty("ClearDatabaseSchemaCache", (invalidateActionResponse :> Type), getterCode = empty) m2.AddXmlDocComputed(fun () -> @@ -624,6 +693,20 @@ module DesignTimeUtils = | None -> prov.GetSchemaCache().Sprocs |> Seq.toList let getTableData name = tableColumns.Force().[name].Force() + // Offline, individuals are in their own file, read when an Individuals type is first expanded + let offlineIndividuals = + lazy + if con.IsNone then + // Individuals are optional: an unreadable individuals file just leaves them out + try prov.GetSchemaCache().LoadIndividuals contextSchemaPath + with _ -> () + // Offline, descriptions are in their own file, read for the first tooltip: builds never need them + let offlineDescriptions = + lazy + if con.IsNone then + // Descriptions are optional: an unreadable descriptions file just leaves them out + try prov.GetSchemaCache().LoadDescriptions contextSchemaPath + with _ -> () let designTimeDc = lazy SqlDataContext(rootTypeName, conString, dbVendor, resolutionPath, config.ReferencedAssemblies, config.RuntimeAssembly, owner, caseSensitivity, tableNames, contextSchemaPath, odbcquote, sqliteLibrary, transactionOptions, None, SelectOperations.DotNetSide, ssdtPath, true) // first create all the types so we are able to recursively reference them in each other's definitions let baseTypes = @@ -676,7 +759,7 @@ module DesignTimeUtils = let ct = ProvidedTypeDefinition(name, Some typeof,isErased=true) ct.AddInterfaceImplementationsDelayed( fun () -> [ProvidedTypeBuilder.MakeGenericType(typedefof>,[et :> Type]); typeof]) let tableTypeDef,_,_,_ = baseTypes.Force().[table.FullName] - let it = createIndividualsType con prov table designTimeDc dbVendor individualsAmount tableTypeDef + let it = createIndividualsType con prov table designTimeDc offlineIndividuals dbVendor individualsAmount tableTypeDef yield name,(ct,it) | false, _ -> ()] @@ -685,7 +768,7 @@ module DesignTimeUtils = t.AddMembersDelayed(fun () -> let (columns,(children,parents)) = getTableData key let attProps = - let createCols = createColumnProperty con prov useOptionTypes key + let createCols = createColumnProperty con prov offlineDescriptions useOptionTypes key List.map createCols (columns |> Seq.map (fun kvp -> kvp.Value) |> Seq.toList) let relProps = let getRelationshipName = Utilities.uniqueName() @@ -745,6 +828,113 @@ module DesignTimeUtils = let getOrAddSchema (name:string) = schemaTypes.GetOrAdd(name, fun nme -> ProvidedTypeDefinition(nme + "Schema", Some typeof, isErased=true)) + // Design-time commands that read the whole database at once, instead of what IntelliSense opened so far, + // and save it beside ContextSchemaPath. Only online: offline there's no database to read. + let onlineCommands = + match con with + | Some connection when not (String.IsNullOrEmpty contextSchemaPath) -> + let schemaCache () = prov.GetSchemaCache() + let savedAt file = "Saved " + file + " at " + DateTime.Now.ToString("hh:mm:ss") + ":" + let failures (failed: ResizeArray) = + if failed.Count = 0 then "" else $" {failed.Count} failed, the first: {failed.[0]}." + /// Runs read for every table; a table that fails doesn't stop the others. + let forTables (failed: ResizeArray) (read: Table -> bool) = + let mutable count = 0 + for table in tables.Force() do + try if read table then count <- count + 1 + with e -> failed.Add(table.FullName + ": " + e.Message) + count + + // A stored procedure's parameters and return columns are read when IntelliSense opens its type, and + // saved under the name of that type: overloads get different names. So open the types the same way. + let declared = BindingFlags.Public ||| BindingFlags.NonPublic ||| BindingFlags.Instance ||| BindingFlags.Static ||| BindingFlags.DeclaredOnly + let rec openTypes (failed: ResizeArray) (t: Type) = + // Listing the nested types adds the stored procedures' types, which may show only in the next listing + let rec nestedTypes (known: Type[]) = + let listed = t.GetNestedTypes declared + if listed.Length = known.Length then listed else nestedTypes listed + let nested = + try nestedTypes [||] + with e -> + let name = if isNull t.DeclaringType then t.Name else t.DeclaringType.Name + "." + t.Name + failed.Add(name + ": " + e.Message) + [||] + for n in nested do + openTypes failed n + + let readFullSchema (failed: ResizeArray) = + // Columns, primary keys and relationships go to the schema cache as they're read + let tableCount = forTables failed (fun table -> getTableData table.FullName |> ignore; true) + for container in containers do + openTypes failed (container :> Type) + $" {tableCount} tables and {schemaCache().SprocsParams.Count} stored procedures." + + let readDescriptions (failed: ResizeArray) = + let tableCount = + forTables failed (fun table -> + let columns, _ = getTableData table.FullName + tableDescription con prov offlineDescriptions table.FullName |> ignore + match prov with + | :? IColumnDescriptionsProvider as allColumns -> + // One query for the table, instead of one for each column + let described = allColumns.GetColumnDescriptions(connection, table.FullName) + let known = schemaCache().ColumnDescriptions.GetOrAdd(table.FullName, fun _ -> System.Collections.Concurrent.ConcurrentDictionary()) + for KeyValue(_, column) in columns do + known.[column.Name] <- + match described.TryGetValue column.Name with + | true, description when not (isNull description) -> description + | _ -> "" + | _ -> + for KeyValue(_, column) in columns do + columnDescription con prov offlineDescriptions table.FullName column.Name |> ignore + true) + $" descriptions of {tableCount} tables." + + let readIndividuals (failed: ResizeArray) = + let tableCount = + forTables failed (fun table -> + match getIndividuals con prov table designTimeDc offlineIndividuals dbVendor individualsAmount with + | Some (_, entities) -> not (Array.isEmpty entities) + | None -> false) + $" individuals of {tableCount} tables." + + /// The whole schema and all the descriptions: metadata only, no rows. + let saveFull () = + let failed = ResizeArray() + let schema = readFullSchema failed + let descriptions = readDescriptions failed + let cache = schemaCache() + cache.Save contextSchemaPath + let files = + [ yield contextSchemaPath + if cache.HasDescriptions then yield SchemaCache.DescriptionsFilePath contextSchemaPath ] + savedAt (String.concat ", " files) + schema + descriptions + failures failed + + /// Rows of data: a step of its own, apart from the schema, because of GDPR/PII. + let saveIndividuals () = + let failed = ResizeArray() + let read = readIndividuals failed + if schemaCache().HasIndividuals then savedAt (schemaCache().SaveIndividuals contextSchemaPath) + read + failures failed + else "No individuals to save: no table with a single primary key has rows." + failures failed + + // SSDT reads a .dacpac: it has no rows + let hasIndividuals = dbVendor <> DatabaseProviderTypes.MSSQLSERVER_SSDT + + let individualsFile = SchemaCache.IndividualsFilePath contextSchemaPath + let descriptionsFile = SchemaCache.DescriptionsFilePath contextSchemaPath + [ yield "SaveContextSchemaFull", + "Saves every table, column, relationship and stored procedure of the database to " + contextSchemaPath + + ", not only the ones used so far, and the descriptions (comments) of every table and column to " + + descriptionsFile + ". No rows of data. Write dot after SaveContextSchemaFull to save. May take a while.", + saveFull + if hasIndividuals then + yield "SaveContextSchemaIndividuals", + "Saves IndividualsAmount individuals (rows) of every table with a single primary key to " + individualsFile + + ". Write dot after SaveContextSchemaIndividuals to save. May take a while. " + + "Individuals are rows of your data: mind GDPR/PII before committing the file.", + saveIndividuals ] + | _ -> [] + let addServiceTypeMembers (isReadonly:bool) = [ if not isReadonly then @@ -1002,15 +1192,13 @@ module DesignTimeUtils = let a0 = args.[0] <@@ ((%%a0 : obj) :?> ISqlDataContext).CreateEntities key @@> ) let tname = ct.Name - match con with - | Some con -> - prop.AddXmlDocDelayed (fun () -> - let details = prov.GetTableDescription(con, tname).Replace("<","<").Replace(">",">") + prop.AddXmlDocDelayed (fun () -> + match tableDescription con prov offlineDescriptions tname with + | Some description -> + let details = description.Replace("<","<").Replace(">",">") let separator = if (String.IsNullOrWhiteSpace desc) || (String.IsNullOrWhiteSpace details) then "" else "/" - $"%s{details} %s{separator} %s{desc}") - | None -> - prop.AddXmlDocDelayed (fun () -> "Offline mode.") - () + $"%s{details} %s{separator} %s{desc}" + | None -> $"Offline mode. %s{desc}") schemaType.AddMember ct if not (schemaType.DeclaredMembers |> Seq.exists(fun m -> m.Name = prop.Name)) then schemaType.AddMember prop @@ -1047,7 +1235,7 @@ module DesignTimeUtils = if not isReadonly then let recreate = fun () -> createTypes rootType serviceType readServiceType config sqlRuntimeInfo invalidate registerClose (connectionString, conStringName,dbVendor,resolutionPath,individualsAmount,useOptionTypes,owner,caseSensitivity, tableNames, contextSchemaPath, odbcquote, sqliteLibrary, ssdtPath, rootTypeName) let designTimeCommandsContainer, saveResponse, mOld, designTime, invalidateActionResponse = - createDesignTimeCommands prov contextSchemaPath recreate invalidate + createDesignTimeCommands prov contextSchemaPath recreate invalidate onlineCommands designTimeCommandsContainer.AddMember saveResponse designTimeCommandsContainer.AddMember templateContainer diff --git a/src/SQLProvider.Runtime/Providers.DuckDb.fs b/src/SQLProvider.Runtime/Providers.DuckDb.fs index e36115b3..8d33ec38 100644 --- a/src/SQLProvider.Runtime/Providers.DuckDb.fs +++ b/src/SQLProvider.Runtime/Providers.DuckDb.fs @@ -538,6 +538,20 @@ type internal DuckDbProvider(resolutionPath, contextSchemaPath, owner:string, re #if REFLECTIONLOAD DuckDb.referencedAssemblies <- referencedAssemblies #endif + interface IColumnDescriptionsProvider with + member __.GetColumnDescriptions(con,tableName) = + let sn = tableName.Substring(0,tableName.LastIndexOf('.')) + let tn = tableName.Substring(tableName.LastIndexOf('.')+1) + let baseQuery = @"SELECT COLUMN_NAME, COLUMN_COMMENT + FROM INFORMATION_SCHEMA.COLUMNS + WHERE TABLE_SCHEMA = $schema AND TABLE_NAME = $table" + use com = (this:>ISqlProvider).CreateCommand(con,baseQuery) + com.Parameters.Add((this:>ISqlProvider).CreateCommandParameter(QueryParameter.Create("schema", 0), sn)) |> ignore + com.Parameters.Add((this:>ISqlProvider).CreateCommandParameter(QueryParameter.Create("table", 1), (DuckDb.ripQuotes tn))) |> ignore + if con.State <> ConnectionState.Open then con.Open() + use reader = com.ExecuteReader() + Sql.readDescriptions reader + interface ISqlProvider with member __.CloseConnectionAfterQuery = true member __.DesignConnection = true diff --git a/src/SQLProvider.Runtime/Providers.Firebird.fs b/src/SQLProvider.Runtime/Providers.Firebird.fs index 0c40a6f2..308341ed 100644 --- a/src/SQLProvider.Runtime/Providers.Firebird.fs +++ b/src/SQLProvider.Runtime/Providers.Firebird.fs @@ -692,6 +692,18 @@ type internal FirebirdProvider(resolutionPath, contextSchemaPath, owner, referen #if REFLECTIONLOAD Firebird.referencedAssemblies <- referencedAssemblies #endif + interface IColumnDescriptionsProvider with + member __.GetColumnDescriptions(con,tableName) = + let tn = tableName.Substring(tableName.LastIndexOf('.')+1) + let baseQuery = @"SELECT trim(RDB$FIELD_NAME), RDB$DESCRIPTION + FROM RDB$RELATION_FIELDS + WHERE RDB$RELATION_NAME = @table" + use com = (this:>ISqlProvider).CreateCommand(con,baseQuery) + com.Parameters.Add((this:>ISqlProvider).CreateCommandParameter(QueryParameter.Create("@table", 1), (Firebird.ripQuotes tn))) |> ignore + if con.State <> ConnectionState.Open then con.Open() + use reader = com.ExecuteReader() + Sql.readDescriptions reader + interface ISqlProvider with member __.CloseConnectionAfterQuery = true member __.DesignConnection = true diff --git a/src/SQLProvider.Runtime/Providers.MsSqlServer.Dynamic.fs b/src/SQLProvider.Runtime/Providers.MsSqlServer.Dynamic.fs index 09d1ac33..24bad5cd 100644 --- a/src/SQLProvider.Runtime/Providers.MsSqlServer.Dynamic.fs +++ b/src/SQLProvider.Runtime/Providers.MsSqlServer.Dynamic.fs @@ -579,22 +579,45 @@ type internal MSSqlServerDynamicProvider(resolutionPath, contextSchemaPath, refe MSSqlServerDynamic.resolutionPath <- resolutionPath MSSqlServerDynamic.referencedAssemblies <- referencedAssemblies + interface IColumnDescriptionsProvider with + member __.GetColumnDescriptions(con,tableName) = + let tn = tableName.Substring(tableName.LastIndexOf('.')+1) + let sn = if tableName.Contains "." then tableName.Substring(0, tableName.LastIndexOf('.')) else "" + let baseq = + """select sc.name, sep.value + from sys.tables st + inner join sys.columns sc on st.object_id = sc.object_id + inner join sys.extended_properties sep on st.object_id = sep.major_id + and sc.column_id = sep.minor_id + and sep.name = 'MS_Description' + inner join sys.schemas ss on st.schema_id = ss.schema_id + where st.name = @TableName and (@SchemaName = '' or ss.name = @SchemaName)""" + use com = (this:>ISqlProvider).CreateCommand(con,baseq) + com.Parameters.Add((this:>ISqlProvider).CreateCommandParameter(QueryParameter.Create("@TableName", 0), tn)) |> ignore + com.Parameters.Add((this:>ISqlProvider).CreateCommandParameter(QueryParameter.Create("@SchemaName", 2), sn)) |> ignore + if con.State <> ConnectionState.Open then con.Open() + use reader = com.ExecuteReader() + Sql.readDescriptions reader + interface ISqlProvider with member __.CloseConnectionAfterQuery = true member __.DesignConnection = true member __.StoredProcedures = true member __.GetLockObject() = myLock member __.GetTableDescription(con,tableName) = - let tn = tableName.Substring(tableName.LastIndexOf('.')+1) + let tn = tableName.Substring(tableName.LastIndexOf('.')+1) + let sn = if tableName.Contains "." then tableName.Substring(0, tableName.LastIndexOf('.')) else "" let baseq = """select sep.value from sys.tables st left join sys.extended_properties sep on st.object_id = sep.major_id and (sep.minor_id is null or sep.minor_id =0) and sep.name = 'MS_Description' - where st.name = @TableName""" + inner join sys.schemas ss on st.schema_id = ss.schema_id + where st.name = @TableName and (@SchemaName = '' or ss.name = @SchemaName)""" use com = (this:>ISqlProvider).CreateCommand(con,baseq) com.Parameters.Add((this:>ISqlProvider).CreateCommandParameter(QueryParameter.Create("@TableName", 0), tn)) |> ignore + com.Parameters.Add((this:>ISqlProvider).CreateCommandParameter(QueryParameter.Create("@SchemaName", 2), sn)) |> ignore if con.State <> ConnectionState.Open then con.Open() use reader = com.ExecuteReader() @@ -605,7 +628,8 @@ type internal MSSqlServerDynamicProvider(resolutionPath, contextSchemaPath, refe else "" member __.GetColumnDescription(con,tableName,columnName) = - let tn = tableName.Substring(tableName.LastIndexOf('.')+1) + let tn = tableName.Substring(tableName.LastIndexOf('.')+1) + let sn = if tableName.Contains "." then tableName.Substring(0, tableName.LastIndexOf('.')) else "" let baseq = """select sep.value from sys.tables st @@ -613,10 +637,12 @@ type internal MSSqlServerDynamicProvider(resolutionPath, contextSchemaPath, refe left join sys.extended_properties sep on st.object_id = sep.major_id and sc.column_id = sep.minor_id and sep.name = 'MS_Description' - where st.name = @TableName + inner join sys.schemas ss on st.schema_id = ss.schema_id + where st.name = @TableName and (@SchemaName = '' or ss.name = @SchemaName) and sc.name = @ColumnName""" use com = (this:>ISqlProvider).CreateCommand(con,baseq) com.Parameters.Add((this:>ISqlProvider).CreateCommandParameter(QueryParameter.Create("@TableName", 0), tn)) |> ignore + com.Parameters.Add((this:>ISqlProvider).CreateCommandParameter(QueryParameter.Create("@SchemaName", 2), sn)) |> ignore com.Parameters.Add((this:>ISqlProvider).CreateCommandParameter(QueryParameter.Create("@ColumnName", 1), columnName)) |> ignore if con.State <> ConnectionState.Open then con.Open() use reader = com.ExecuteReader() diff --git a/src/SQLProvider.Runtime/Providers.MsSqlServer.fs b/src/SQLProvider.Runtime/Providers.MsSqlServer.fs index 73326e85..dafaa7ab 100644 --- a/src/SQLProvider.Runtime/Providers.MsSqlServer.fs +++ b/src/SQLProvider.Runtime/Providers.MsSqlServer.fs @@ -500,22 +500,45 @@ type internal MSSqlServerProvider(contextSchemaPath, tableNames:string) = // Remembers the version of each instance it connects to let mssqlVersionCache = ConcurrentDictionary>() + interface IColumnDescriptionsProvider with + member __.GetColumnDescriptions(con,tableName) = + let tn = tableName.Substring(tableName.LastIndexOf('.')+1) + let sn = if tableName.Contains "." then tableName.Substring(0, tableName.LastIndexOf('.')) else "" + let baseq = + """select sc.name, sep.value + from sys.tables st + inner join sys.columns sc on st.object_id = sc.object_id + inner join sys.extended_properties sep on st.object_id = sep.major_id + and sc.column_id = sep.minor_id + and sep.name = 'MS_Description' + inner join sys.schemas ss on st.schema_id = ss.schema_id + where st.name = @TableName and (@SchemaName = '' or ss.name = @SchemaName)""" + use com = new SqlCommand(baseq,con:?>SqlConnection) + com.Parameters.AddWithValue("@TableName",tn) |> ignore + com.Parameters.AddWithValue("@SchemaName",sn) |> ignore + if con.State <> ConnectionState.Open then con.Open() + use reader = com.ExecuteReader() + Sql.readDescriptions reader + interface ISqlProvider with member __.CloseConnectionAfterQuery = true member __.DesignConnection = true member __.StoredProcedures = true member __.GetLockObject() = myLock member __.GetTableDescription(con,tableName) = - let tn = tableName.Substring(tableName.LastIndexOf('.')+1) + let tn = tableName.Substring(tableName.LastIndexOf('.')+1) + let sn = if tableName.Contains "." then tableName.Substring(0, tableName.LastIndexOf('.')) else "" let baseq = """select sep.value from sys.tables st left join sys.extended_properties sep on st.object_id = sep.major_id and (sep.minor_id is null or sep.minor_id =0) and sep.name = 'MS_Description' - where st.name = @TableName""" + inner join sys.schemas ss on st.schema_id = ss.schema_id + where st.name = @TableName and (@SchemaName = '' or ss.name = @SchemaName)""" use com = new SqlCommand(baseq,con:?>SqlConnection) com.Parameters.AddWithValue("@TableName",tn) |> ignore + com.Parameters.AddWithValue("@SchemaName",sn) |> ignore if con.State <> ConnectionState.Open then con.Open() use reader = com.ExecuteReader() if reader.Read() then @@ -525,7 +548,8 @@ type internal MSSqlServerProvider(contextSchemaPath, tableNames:string) = else "" member __.GetColumnDescription(con,tableName,columnName) = - let tn = tableName.Substring(tableName.LastIndexOf('.')+1) + let tn = tableName.Substring(tableName.LastIndexOf('.')+1) + let sn = if tableName.Contains "." then tableName.Substring(0, tableName.LastIndexOf('.')) else "" let baseq = """select sep.value from sys.tables st @@ -533,10 +557,12 @@ type internal MSSqlServerProvider(contextSchemaPath, tableNames:string) = left join sys.extended_properties sep on st.object_id = sep.major_id and sc.column_id = sep.minor_id and sep.name = 'MS_Description' - where st.name = @TableName + inner join sys.schemas ss on st.schema_id = ss.schema_id + where st.name = @TableName and (@SchemaName = '' or ss.name = @SchemaName) and sc.name = @ColumnName""" use com = new SqlCommand(baseq,con:?>SqlConnection) com.Parameters.AddWithValue("@TableName",tn) |> ignore + com.Parameters.AddWithValue("@SchemaName",sn) |> ignore com.Parameters.AddWithValue("@ColumnName",columnName) |> ignore if con.State <> ConnectionState.Open then con.Open() use reader = com.ExecuteReader() diff --git a/src/SQLProvider.Runtime/Providers.MySql.fs b/src/SQLProvider.Runtime/Providers.MySql.fs index 1a5da092..805ceaae 100644 --- a/src/SQLProvider.Runtime/Providers.MySql.fs +++ b/src/SQLProvider.Runtime/Providers.MySql.fs @@ -594,6 +594,20 @@ type internal MySqlProvider(resolutionPath, contextSchemaPath, owner:string, ref #if REFLECTIONLOAD MySql.referencedAssemblies <- referencedAssemblies #endif + interface IColumnDescriptionsProvider with + member __.GetColumnDescriptions(con,tableName) = + let sn = tableName.Substring(0,tableName.LastIndexOf('.')) + let tn = tableName.Substring(tableName.LastIndexOf('.')+1) + let baseQuery = @"SELECT COLUMN_NAME, COLUMN_COMMENT + FROM INFORMATION_SCHEMA.COLUMNS + WHERE TABLE_SCHEMA = @schema AND TABLE_NAME = @table" + use com = (this:>ISqlProvider).CreateCommand(con,baseQuery) + com.Parameters.Add((this:>ISqlProvider).CreateCommandParameter(QueryParameter.Create("@schema", 0), sn)) |> ignore + com.Parameters.Add((this:>ISqlProvider).CreateCommandParameter(QueryParameter.Create("@table", 1), (MySql.ripQuotes tn))) |> ignore + if con.State <> ConnectionState.Open then con.Open() + use reader = com.ExecuteReader() + Sql.readDescriptions reader + interface ISqlProvider with member __.CloseConnectionAfterQuery = true member __.DesignConnection = true diff --git a/src/SQLProvider.Runtime/Providers.Oracle.fs b/src/SQLProvider.Runtime/Providers.Oracle.fs index c4e6c7a1..5b2f591c 100644 --- a/src/SQLProvider.Runtime/Providers.Oracle.fs +++ b/src/SQLProvider.Runtime/Providers.Oracle.fs @@ -221,6 +221,16 @@ module internal Oracle = new Oracle.ManagedDataAccess.Client.OracleCommand(commandText, (connection :?> Oracle.ManagedDataAccess.Client.OracleConnection) ) :> IDbCommand #endif + /// A value for inside '...' in SQL text: an apostrophe in it doubled. + let sqlText (value: string) = value.Replace("'", "''") + + /// The owner and name of a table, from OWNER.TABLE, for the comments queries. The owner is "" without one: the + /// queries then read the current user's. + let descriptionOwnerAndTable (tableName: string) = + let dot = tableName.LastIndexOf '.' + let owner = if dot < 0 then "" else tableName.Substring(0, dot) + sqlText owner, sqlText (tableName.Substring(dot + 1)) + let readParameter (parameter:IDbDataParameter) = #if REFLECTIONLOAD let parameterType = parameterType.Value @@ -721,21 +731,36 @@ type internal OracleProvider(resolutionPath, contextSchemaPath, owner, reference #endif Oracle.resolutionPath <- resolutionPath + interface IColumnDescriptionsProvider with + member __.GetColumnDescriptions(con,tableName) = + let sn, tn = Oracle.descriptionOwnerAndTable tableName + Sql.connect con (fun con -> + let query = + sprintf """SELECT COLUMN_NAME, COMMENTS + FROM all_col_comments + WHERE OWNER = NVL('%s', USER) + AND TABLE_NAME = '%s' + AND COMMENTS <> '-' + """ sn tn + use com = Oracle.createCommand query con + use reader = com.ExecuteReader() + Sql.readDescriptions reader) + interface ISqlProvider with member __.CloseConnectionAfterQuery = true member __.DesignConnection = true member __.StoredProcedures = true member __.GetLockObject() = myLock member __.GetTableDescription(con,tableName) = - let sn = tableName.Substring(0,tableName.LastIndexOf('.')) - let tn = tableName.Substring(tableName.LastIndexOf('.')+1) - Sql.connect con (fun con -> + let sn, tn = Oracle.descriptionOwnerAndTable tableName + Sql.connect con (fun con -> let comment = - sprintf """SELECT COMMENTS - FROM user_tab_comments - WHERE TABLE_NAME = '%s' + sprintf """SELECT COMMENTS + FROM all_tab_comments + WHERE OWNER = NVL('%s', USER) + AND TABLE_NAME = '%s' AND COMMENTS <> '-' - """ tn + """ sn tn |> Oracle.read con (fun row -> Sql.dbUnbox row.[0]) |> Seq.toList @@ -745,16 +770,16 @@ type internal OracleProvider(resolutionPath, contextSchemaPath, owner, reference "" ) member __.GetColumnDescription(con,tableName,columnName) = - let sn = tableName.Substring(0,tableName.LastIndexOf('.')) - let tn = tableName.Substring(tableName.LastIndexOf('.')+1) - Sql.connect con (fun con -> + let sn, tn = Oracle.descriptionOwnerAndTable tableName + Sql.connect con (fun con -> let comment = - sprintf """SELECT COMMENTS - FROM user_col_comments - WHERE TABLE_NAME = '%s' + sprintf """SELECT COMMENTS + FROM all_col_comments + WHERE OWNER = NVL('%s', USER) + AND TABLE_NAME = '%s' AND COLUMN_NAME = '%s' AND COMMENTS <> '-' - """ tn columnName + """ sn tn (Oracle.sqlText columnName) |> Oracle.read con (fun row -> Sql.dbUnbox row.[0]) |> Seq.toList diff --git a/src/SQLProvider.Runtime/Providers.Postgresql.fs b/src/SQLProvider.Runtime/Providers.Postgresql.fs index ce855861..f8a8060e 100644 --- a/src/SQLProvider.Runtime/Providers.Postgresql.fs +++ b/src/SQLProvider.Runtime/Providers.Postgresql.fs @@ -323,6 +323,13 @@ module PostgreSQL = new Npgsql.NpgsqlCommand(commandText, connection :?> Npgsql.NpgsqlConnection) :> IDbCommand #endif + /// Runs a query with the values as parameters instead of in the SQL text, so that any name works in it. + let executeSqlWith commandText (parameters: (string * obj) list) (connection:IDbConnection) = + use com = createCommand commandText connection + parameters |> List.iteri (fun i (name, value) -> + createCommandParameter (QueryParameter.Create(name, i)) value |> com.Parameters.Add |> ignore) + com.ExecuteReader() + let readParameter (parameter:IDbDataParameter) = #if REFLECTIONLOAD match parameter.DbType, (dbTypeGetter.Value.Invoke(parameter, [||]) :?> int) with @@ -822,6 +829,23 @@ type internal PostgresqlProvider(resolutionPath, contextSchemaPath, owner, refer owner.Split(';', ',', ' ', '\n', '\r') |> Array.filter (not << String.IsNullOrWhiteSpace) + interface IColumnDescriptionsProvider with + member __.GetColumnDescriptions(con,tableName) = + Sql.connect con (fun _ -> + let sn = tableName.Substring(0,tableName.LastIndexOf('.')) + let tn = tableName.Substring(tableName.LastIndexOf('.')+1) + use reader = + PostgreSQL.executeSqlWith + """SELECT c.column_name, d.description + FROM pg_description d + LEFT JOIN information_schema.columns c + ON c.ordinal_position = d.objsubid + WHERE objoid = format('%I.%I', @schema, @table)::regclass + AND c.table_schema = @schema + AND c.table_name = @table;""" + [ "@schema", box sn; "@table", box tn ] con + Sql.readDescriptions reader) + interface ISqlProvider with member __.CloseConnectionAfterQuery = true member __.DesignConnection = true @@ -829,12 +853,16 @@ type internal PostgresqlProvider(resolutionPath, contextSchemaPath, owner, refer member __.GetLockObject() = myLock member __.GetTableDescription(con,tableName) = Sql.connect con (fun _ -> - use reader = - Sql.executeSql PostgreSQL.createCommand ( - sprintf """SELECT description - FROM pg_description - WHERE objoid = '%s'::regclass AND objsubid=0; - """ tableName) con + // The names are quoted (%I), so that a table with capital letters in its name is found. They are + // parameters, so that a name with an apostrophe in it works too. + let sn = tableName.Substring(0,tableName.LastIndexOf('.')) + let tn = tableName.Substring(tableName.LastIndexOf('.')+1) + use reader = + PostgreSQL.executeSqlWith + """SELECT description + FROM pg_description + WHERE objoid = format('%I.%I', @schema, @table)::regclass AND objsubid=0;""" + [ "@schema", box sn; "@table", box tn ] con if reader.Read() then let comment = Sql.dbUnbox reader.["description"] if isNull comment then "" else comment @@ -844,17 +872,17 @@ type internal PostgresqlProvider(resolutionPath, contextSchemaPath, owner, refer Sql.connect con (fun _ -> let sn = tableName.Substring(0,tableName.LastIndexOf('.')) let tn = tableName.Substring(tableName.LastIndexOf('.')+1) - use reader = - Sql.executeSql PostgreSQL.createCommand ( - sprintf """SELECT description - FROM pg_description d - LEFT JOIN information_schema.columns c - ON c.ordinal_position = d.objsubid - WHERE objoid = '%s'::regclass - AND c.table_schema = '%s' - AND c.table_name = '%s' - AND c.column_name = '%s'; - """ tableName sn tn columnName) con + use reader = + PostgreSQL.executeSqlWith + """SELECT description + FROM pg_description d + LEFT JOIN information_schema.columns c + ON c.ordinal_position = d.objsubid + WHERE objoid = format('%I.%I', @schema, @table)::regclass + AND c.table_schema = @schema + AND c.table_name = @table + AND c.column_name = @column;""" + [ "@schema", box sn; "@table", box tn; "@column", box columnName ] con if reader.Read() then let comment = Sql.dbUnbox reader.["description"] if isNull comment then "" else comment diff --git a/tests/SqlProvider.Tests/SchemaCacheFileTests.fs b/tests/SqlProvider.Tests/SchemaCacheFileTests.fs new file mode 100644 index 00000000..9b1c57e4 --- /dev/null +++ b/tests/SqlProvider.Tests/SchemaCacheFileTests.fs @@ -0,0 +1,224 @@ +module SchemaCacheFileTests + +// The individuals of a saved schema (ContextSchemaPath) are in their own file beside it. + +open System +open System.IO +open System.Text +open System.Collections.Generic +open System.Collections.Concurrent +open FSharp.Data.Sql.Common +open NUnit.Framework + +let private tempSchemaPath () = + let dir = Path.Combine(Path.GetTempPath(), "SqlProviderSchemaTests", Guid.NewGuid().ToString("N")) + Directory.CreateDirectory dir |> ignore + Path.Combine(dir, "schema.json") + +let private customers () : (obj * IDictionary) array = + [| for i in 1 .. 3 -> box i, (dict [ "Id", box i; "Name", box (sprintf "Customer %d" i) ]) |] + +let private withCustomers (cache:SchemaCache) = + cache.Tables.["main.Customers"] <- { Schema = "main"; Name = "Customers"; Type = "TABLE" } + cache.Individuals.["main.Customers_Id"] <- customers () + cache + +/// A schema file as versions before 1.5.34 saved it: individuals in the schema file itself +let private saveOldFormat (cache:SchemaCache) (filePath:string) = + use ms = new MemoryStream() + let ser = System.Runtime.Serialization.Json.DataContractJsonSerializer(typeof) + ser.WriteObject(ms, { cache with IsOffline = true }) + File.WriteAllText(filePath, Encoding.UTF8.GetString(ms.ToArray())) + +let private names (entities:(obj * IDictionary) array) = + entities |> Array.map (fun (_, cols) -> string cols.["Name"]) + +[] +let ``the individuals file is beside the schema file`` () = + Assert.AreEqual(Path.Combine("c:\\db", "schema.individuals.json"), SchemaCache.IndividualsFilePath(Path.Combine("c:\\db", "schema.json"))) + Assert.AreEqual("schema.individuals", SchemaCache.IndividualsFilePath "schema") + +[] +let ``saving the schema leaves out the individuals of an online cache`` () = + let path = tempSchemaPath () + let cache = withCustomers SchemaCache.Empty + cache.Save path + Assert.IsFalse(File.Exists(SchemaCache.IndividualsFilePath path)) + let loaded = SchemaCache.Load path + Assert.IsTrue(loaded.Tables.ContainsKey "main.Customers") + Assert.AreEqual(0, loaded.Individuals.Count) + StringAssert.DoesNotContain("Customer 1", File.ReadAllText path) + +[] +let ``saved individuals are read from their own file`` () = + let path = tempSchemaPath () + let cache = withCustomers SchemaCache.Empty + cache.Save path + let individualsFile = cache.SaveIndividuals path + Assert.AreEqual(SchemaCache.IndividualsFilePath path, individualsFile) + let loaded = SchemaCache.Load path + Assert.AreEqual(0, loaded.Individuals.Count) + loaded.LoadIndividuals path + CollectionAssert.AreEqual(names (customers ()), names loaded.Individuals.["main.Customers_Id"]) + +[] +let ``tables without individuals are not saved`` () = + let path = tempSchemaPath () + let cache = withCustomers SchemaCache.Empty + cache.Individuals.["main.Orders_Id"] <- [||] + cache.SaveIndividuals path |> ignore + let loaded = SchemaCache.Empty + loaded.LoadIndividuals path + Assert.IsFalse(loaded.Individuals.ContainsKey "main.Orders_Id") + Assert.IsTrue(loaded.Individuals.ContainsKey "main.Customers_Id") + +[] +let ``column values the file can't hold as such are saved as text`` () = + let path = tempSchemaPath () + let cache = SchemaCache.Empty + let changed = DateTimeOffset(2026, 11, 20, 12, 0, 0, TimeSpan.FromHours 2.) + cache.Individuals.["main.Customers_Id"] <- + [| box 1, (dict [ "Id", box 1; "Name", box "Customer 1"; "Changed", box changed + "Tags", box [| "a"; "b" |]; "Address", box (Net.IPAddress.Parse "10.0.0.1") + "Missing", box DBNull.Value; "Nothing", null ]) |] + cache.SaveIndividuals path |> ignore + let loaded = SchemaCache.Empty + loaded.LoadIndividuals path + let columns = snd loaded.Individuals.["main.Customers_Id"].[0] + Assert.AreEqual("Customer 1", columns.["Name"]) + Assert.AreEqual(changed.ToString(), columns.["Changed"]) + CollectionAssert.AreEqual([| "a"; "b" |], columns.["Tags"] :?> obj[]) + Assert.AreEqual("10.0.0.1", columns.["Address"]) + Assert.AreEqual("", columns.["Missing"]) + Assert.IsNull(columns.["Nothing"]) + // The cache itself keeps the values as they were read + Assert.AreEqual(box changed, (snd cache.Individuals.["main.Customers_Id"].[0]).["Changed"]) + +[] +let ``a schema file with individuals in it still loads them`` () = + let path = tempSchemaPath () + saveOldFormat (withCustomers SchemaCache.Empty) path + let loaded = SchemaCache.Load path + CollectionAssert.AreEqual(names (customers ()), names loaded.Individuals.["main.Customers_Id"]) + +[] +let ``re-saving an old schema file moves its individuals to their own file`` () = + let path = tempSchemaPath () + saveOldFormat (withCustomers SchemaCache.Empty) path + let loaded = SchemaCache.Load path + loaded.Save path + StringAssert.DoesNotContain("Customer 1", File.ReadAllText path) + let reloaded = SchemaCache.Load path + reloaded.LoadIndividuals path + CollectionAssert.AreEqual(names (customers ()), names reloaded.Individuals.["main.Customers_Id"]) + +[] +let ``the individuals file wins over individuals of an old schema file`` () = + let path = tempSchemaPath () + saveOldFormat (withCustomers SchemaCache.Empty) path + let newer = SchemaCache.Empty + newer.Individuals.["main.Customers_Id"] <- [| box 9, (dict [ "Id", box 9; "Name", box "Newer" ]) |] + newer.SaveIndividuals path |> ignore + let loaded = SchemaCache.Load path + loaded.Save path + let reloaded = SchemaCache.Load path + reloaded.LoadIndividuals path + CollectionAssert.AreEqual([| "Newer" |], names reloaded.Individuals.["main.Customers_Id"]) + +[] +let ``merging schema files merges their individuals files`` () = + let path1, path2, target = tempSchemaPath (), tempSchemaPath (), tempSchemaPath () + let cache1 = withCustomers SchemaCache.Empty + cache1.Save path1 + cache1.SaveIndividuals path1 |> ignore + let cache2 = SchemaCache.Empty + cache2.Individuals.["main.Orders_Id"] <- [| box 1, (dict [ "Id", box 1; "Name", box "Order 1" ]) |] + cache2.Save path2 + cache2.SaveIndividuals path2 |> ignore + OfflineTools.mergeCacheFiles(path1, path2, target) |> ignore + StringAssert.DoesNotContain("Customer 1", File.ReadAllText target) + let merged = SchemaCache.Load target + merged.LoadIndividuals target + Assert.IsTrue(merged.Individuals.ContainsKey "main.Customers_Id") + Assert.IsTrue(merged.Individuals.ContainsKey "main.Orders_Id") + +// Table and column descriptions (comments) are in their own file too, for tooltips in offline mode. + +let private columnDescriptions (pairs: (string * string) list) = + ConcurrentDictionary(pairs |> Seq.map (fun (k, v) -> KeyValuePair(k, v))) + +let private describedCustomers () = + let cache = withCustomers SchemaCache.Empty + cache.TableDescriptions.["main.Customers"] <- "People who buy" + cache.TableDescriptions.["main.Orders"] <- "" + cache.ColumnDescriptions.["main.Customers"] <- columnDescriptions [ "Name", "Full name"; "Id", "" ] + cache + +[] +let ``the descriptions file is beside the schema file`` () = + Assert.AreEqual(Path.Combine("c:\\db", "schema.descriptions.json"), SchemaCache.DescriptionsFilePath(Path.Combine("c:\\db", "schema.json"))) + +[] +let ``saving the schema saves the descriptions to their own file`` () = + let path = tempSchemaPath () + (describedCustomers ()).Save path + StringAssert.DoesNotContain("People who buy", File.ReadAllText path) + let loaded = SchemaCache.Load path + Assert.AreEqual(0, loaded.TableDescriptions.Count) + loaded.LoadDescriptions path + Assert.AreEqual("People who buy", loaded.TableDescriptions.["main.Customers"]) + Assert.AreEqual("", loaded.TableDescriptions.["main.Orders"]) + Assert.AreEqual("Full name", loaded.ColumnDescriptions.["main.Customers"].["Name"]) + Assert.AreEqual("", loaded.ColumnDescriptions.["main.Customers"].["Id"]) + +[] +let ``without descriptions there is no descriptions file`` () = + let path = tempSchemaPath () + (withCustomers SchemaCache.Empty).Save path + Assert.IsFalse(File.Exists(SchemaCache.DescriptionsFilePath path)) + Assert.AreEqual(None, SchemaCache.Empty.SaveDescriptions path) + +[] +let ``saving the descriptions read so far keeps the others in the file`` () = + let path = tempSchemaPath () + (describedCustomers ()).Save path + let later = SchemaCache.Empty + later.TableDescriptions.["main.Customers"] <- "Changed" + later.ColumnDescriptions.["main.Customers"] <- columnDescriptions [ "Phone", "Phone number" ] + later.SaveDescriptions path |> ignore + let loaded = SchemaCache.Empty + loaded.LoadDescriptions path + Assert.AreEqual("Changed", loaded.TableDescriptions.["main.Customers"]) + Assert.AreEqual("", loaded.TableDescriptions.["main.Orders"]) + Assert.AreEqual("Full name", loaded.ColumnDescriptions.["main.Customers"].["Name"]) + Assert.AreEqual("Phone number", loaded.ColumnDescriptions.["main.Customers"].["Phone"]) + +[] +let ``re-saving an offline schema keeps its descriptions`` () = + let path = tempSchemaPath () + (describedCustomers ()).Save path + // Offline, descriptions are read only for the first tooltip: a save before that must not lose them + (SchemaCache.Load path).Save path + let loaded = SchemaCache.Load path + loaded.LoadDescriptions path + Assert.AreEqual("People who buy", loaded.TableDescriptions.["main.Customers"]) + +[] +let ``merging schema files merges their descriptions files`` () = + let path1, path2, target = tempSchemaPath (), tempSchemaPath (), tempSchemaPath () + let cache1 = SchemaCache.Empty + cache1.TableDescriptions.["main.Customers"] <- "People who buy" + cache1.ColumnDescriptions.["main.Customers"] <- columnDescriptions [ "Name", "Full name" ] + cache1.Save path1 + let cache2 = SchemaCache.Empty + cache2.TableDescriptions.["main.Orders"] <- "What they buy" + cache2.ColumnDescriptions.["main.Customers"] <- columnDescriptions [ "Phone", "Phone number" ] + cache2.Save path2 + OfflineTools.mergeCacheFiles(path1, path2, target) |> ignore + let merged = SchemaCache.Load target + merged.LoadDescriptions target + Assert.AreEqual("People who buy", merged.TableDescriptions.["main.Customers"]) + Assert.AreEqual("What they buy", merged.TableDescriptions.["main.Orders"]) + Assert.AreEqual("Full name", merged.ColumnDescriptions.["main.Customers"].["Name"]) + Assert.AreEqual("Phone number", merged.ColumnDescriptions.["main.Customers"].["Phone"]) + diff --git a/tests/SqlProvider.Tests/SqlProvider.Tests.fsproj b/tests/SqlProvider.Tests/SqlProvider.Tests.fsproj index d1f30559..a8e72855 100644 --- a/tests/SqlProvider.Tests/SqlProvider.Tests.fsproj +++ b/tests/SqlProvider.Tests/SqlProvider.Tests.fsproj @@ -43,6 +43,7 @@ + Always