Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
01d532c
Add migration overview documentation for RavenDB to SQL transition
warwickschroeder Sep 4, 2026
e0676d8
Refine migration overview documentation for RavenDB to SQL transition
warwickschroeder Sep 10, 2026
4db14b8
Clarify documentation on RavenDB source requirements for containerize…
warwickschroeder Sep 10, 2026
5891c62
Enhance migration overview documentation for RavenDB to SQL transitio…
warwickschroeder Sep 11, 2026
94d13ba
Implement the foundation for migration engine
warwickschroeder Sep 11, 2026
42e91ab
Add unit tests for migration engine functionality and improve migrati…
warwickschroeder Sep 15, 2026
0afd1e1
Clarify migration overview by detailing additional skip conditions fo…
warwickschroeder Sep 15, 2026
f116664
PR review fixes
warwickschroeder Sep 15, 2026
746617c
PR review
warwickschroeder Sep 16, 2026
7c88e2d
Enhance migration engine to handle benign skips and update related tests
warwickschroeder Sep 17, 2026
b42522f
Enhance Migration Engine and Unit Tests
warwickschroeder Sep 17, 2026
c72590e
Refactor migration settings and update documentation to remove RavenD…
warwickschroeder Sep 30, 2026
7908498
Clarify event log handling and data retention during migration from R…
warwickschroeder Sep 30, 2026
54a6571
Update migration documentation to clarify integration events and erro…
warwickschroeder Sep 30, 2026
54befb3
Refine migration documentation for clarity on data retention and copy…
warwickschroeder Sep 30, 2026
1cd87eb
[Migration Engine Part 2] Add a migration checkpoint table to SQL Ser…
warwickschroeder Oct 9, 2026
09021ee
Migration engine: Copy unresolved failed messages and their bodies to…
warwickschroeder Nov 12, 2026
5a6d619
Migration engine: Copy licensing throughput history to SQL (#6063)
warwickschroeder Nov 12, 2026
ea4bbb9
Change offline migration step to an interactive cli action (#6106)
rbev Nov 13, 2026
1068147
Fix typo in documentation: change 'DisableAutoIndexCreation' to 'Disa…
warwickschroeder Nov 20, 2026
071a36d
Add CreatedDate property to licensing endpoint mappings and tests
warwickschroeder Nov 20, 2026
49f63e6
Remove unused ServiceControl.Persistence namespace from ImportFailedE…
warwickschroeder Nov 20, 2026
a338a6d
Copy redirects, subscriptions, retry history and settings to SQL
warwickschroeder Nov 16, 2026
eda1aa6
Refactor migration documentation for clarity and accuracy
warwickschroeder Nov 16, 2026
33b1b0f
Update migration overview to clarify custom ServicePulse UI for migra…
warwickschroeder Nov 20, 2026
5f62c89
Trigger CI
warwickschroeder Nov 20, 2026
731848e
Copy the last four required categories from RavenDB to SQL: pending i…
warwickschroeder Nov 17, 2026
e79b1e9
Remove logging of skipped IDs from migration engine processing
warwickschroeder Nov 18, 2026
750d3ed
Migration Engine: Copy redirects, subscriptions, retry history and se…
warwickschroeder Nov 20, 2026
500291c
Merge pull request #6156 from Particular/migration-engine-8
warwickschroeder Nov 20, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
# Categories are declared by the <TestCategory> property in each test project. A category can
# span several projects that share infrastructure, so that the container is provisioned once
# and the union of their build closures is compiled once.
test-category: [ DefaultCore, DefaultMonitoring, Raven, SqlServer, PostgreSql, RabbitMQ, AzureServiceBus, AzureStorageQueues, MSMQ, SQS, IBMMQ ]
test-category: [ DefaultCore, DefaultMonitoring, Migration, Raven, SqlServer, PostgreSql, RabbitMQ, AzureServiceBus, AzureStorageQueues, MSMQ, SQS, IBMMQ ]
include:
- os: windows-latest
os-name: Windows
Expand Down Expand Up @@ -96,7 +96,7 @@ jobs:
id: select
run: ./tools/select-test-projects.ps1 -Category ${{ matrix.test-category }}
- name: Download RavenDB Server
if: matrix.test-category == 'DefaultCore'
if: contains(fromJSON('["DefaultCore", "Migration"]'), matrix.test-category)
run: ./tools/download-ravendb-server.ps1
- name: Build
id: build
Expand Down
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ It's also possible to [locally test containers built from PRs in GitHub Containe
If the instance is executed for the first time, it must set up the required infrastructure. To do so, once the instance is configured to use the selected transport and persister, run it in setup mode. This can be done by using the `Setup {instance name}` launch profile that is defined in
the `launchSettings.json` file of each instance. When started in setup mode, the instance will start as usual, execute the setup process, and exit. At this point the instance can be run normally by using the non-setup launch profile.

## Migrating from RavenDB to SQL Server or PostgreSQL

- [Migrate from RavenDB to SQL Server or PostgreSQL](docs/migration/ravendb-to-sql-migration-instructions.md): what an operator can run today, and what is not built yet.

## Secrets

Testing using the [CI workflow](/.github/workflows/ci.yml) depends on the following secrets. The Particular values for these secrets are stored in the secure note named **ServiceControl Repo Secrets**.
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ This section points to sources that explain why ServiceControl is designed the w
- [Message body storage](message-body-storage.md) — where the Audit SQL Server and PostgreSQL persisters keep message bodies, how retention removes them, and which settings are optional
- [Bulk retries design](bulk-retries-design.md) — how ServiceControl retries failed messages in bulk
- [Retries over Azure Storage Queues transport](retries-asq-transport.md) — how retries behave with one or several storage accounts
- [RavenDB-to-SQL migration design](migration/ravendb-to-sql-migration-overview.md) — why required data copies offline and persisted checkpoints drive online completion
- [Data versioning design](data-versioning-design.md) — which invariant keeps a cached API response from going stale
- [Event log design](eventlog-design.md) — what the event log is and what it records
- [Multiple ServiceControl instances communication](multipleservicecontrolinstancescommunication.md) — how primary, audit, and monitoring instances talk to each other
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
269 changes: 269 additions & 0 deletions docs/migration/ravendb-to-sql-migration-instructions.md

Large diffs are not rendered by default.

540 changes: 540 additions & 0 deletions docs/migration/ravendb-to-sql-migration-overview.md

Large diffs are not rendered by default.

302 changes: 302 additions & 0 deletions docs/migration/ravendb-to-sql-migration-system-design.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@
<ItemGroup>
<ProjectReference Include="..\ServiceControl.AcceptanceTesting\ServiceControl.AcceptanceTesting.csproj" />
<ProjectReference Include="..\ServiceControl.Persistence.EFCore.PostgreSql\ServiceControl.Persistence.EFCore.PostgreSql.csproj" />
<!-- Built so its manifest is where DevelopmentPersistenceLocations looks; not copied, or its persistence.manifest would collide with this project's own. -->
<ProjectReference Include="..\ServiceControl.Persistence.RavenDB\ServiceControl.Persistence.RavenDB.csproj" ReferenceOutputAssembly="false" Private="false" />
<ProjectReference Include="..\ServiceControl.RavenDB\ServiceControl.RavenDB.csproj" />
<ProjectReference Include="..\ServiceControl.Transports.Learning\ServiceControl.Transports.Learning.csproj" />
<ProjectReference Include="..\ServiceControl\ServiceControl.csproj" />
<ProjectReference Include="..\TestHelper\TestHelper.csproj" />
Expand All @@ -20,6 +23,7 @@
<ItemGroup>
<PackageReference Include="GitHubActionsTestLogger" />
<PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" />
<PackageReference Include="Microsoft.Extensions.TimeProvider.Testing" />
<PackageReference Include="Microsoft.NET.Test.Sdk" />
<PackageReference Include="NServiceBus.Heartbeat" />
<PackageReference Include="NServiceBus.SagaAudit" />
Expand All @@ -31,6 +35,9 @@

<ItemGroup>
<Compile Include="..\ServiceControl.AcceptanceTests\**\*.cs" LinkBase="Shared" />
<Compile Include="..\ServiceControl.Migration.AcceptanceTests\**\*.cs" LinkBase="Migration" />
<Compile Include="..\ServiceControl.Persistence.Tests\IngestedFailure.cs" />
<Compile Include="..\ServiceControl.Persistence.Tests\EFCore\InMemoryBodyStoragePersistence.cs" />
<Compile Include="..\ServiceControl.Persistence.Tests.PostgreSql\PostgreSqlSharedContainer.cs" />
<Compile Include="..\ServiceControl.Persistence.Tests.PostgreSql\TestSchema.cs" />
<Compile Include="..\ServiceControl.Persistence.Tests.PostgreSql\StopSharedPostgreSql.cs" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@
<ItemGroup>
<ProjectReference Include="..\ServiceControl.AcceptanceTesting\ServiceControl.AcceptanceTesting.csproj" />
<ProjectReference Include="..\ServiceControl.Persistence.EFCore.SqlServer\ServiceControl.Persistence.EFCore.SqlServer.csproj" />
<!-- Built so its manifest is where DevelopmentPersistenceLocations looks; not copied, or its persistence.manifest would collide with this project's own. -->
<ProjectReference Include="..\ServiceControl.Persistence.RavenDB\ServiceControl.Persistence.RavenDB.csproj" ReferenceOutputAssembly="false" Private="false" />
<ProjectReference Include="..\ServiceControl.RavenDB\ServiceControl.RavenDB.csproj" />
<ProjectReference Include="..\ServiceControl.Transports.Learning\ServiceControl.Transports.Learning.csproj" />
<ProjectReference Include="..\ServiceControl\ServiceControl.csproj" />
<ProjectReference Include="..\TestHelper\TestHelper.csproj" />
Expand All @@ -20,6 +23,7 @@
<ItemGroup>
<PackageReference Include="GitHubActionsTestLogger" />
<PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" />
<PackageReference Include="Microsoft.Extensions.TimeProvider.Testing" />
<PackageReference Include="Microsoft.NET.Test.Sdk" />
<PackageReference Include="NServiceBus.Heartbeat" />
<PackageReference Include="NServiceBus.SagaAudit" />
Expand All @@ -31,6 +35,9 @@

<ItemGroup>
<Compile Include="..\ServiceControl.AcceptanceTests\**\*.cs" LinkBase="Shared" />
<Compile Include="..\ServiceControl.Migration.AcceptanceTests\**\*.cs" LinkBase="Migration" />
<Compile Include="..\ServiceControl.Persistence.Tests\IngestedFailure.cs" />
<Compile Include="..\ServiceControl.Persistence.Tests\EFCore\InMemoryBodyStoragePersistence.cs" />
<Compile Include="..\ServiceControl.Persistence.Tests.SqlServer\SqlServerSharedContainer.cs" />
<Compile Include="..\ServiceControl.Persistence.Tests.SqlServer\TestSchema.cs" />
<Compile Include="..\ServiceControl.Persistence.Tests.SqlServer\StopSharedSqlServer.cs" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,9 @@ namespace ServiceControl.AcceptanceTests.Recoverability
using ServiceControl.Infrastructure;
using ServiceControl.MessageFailures;
using ServiceControl.Operations;
using ServiceControl.Persistence.DataMigration;
using ServiceControl.Persistence.EFCore.DbContexts;
using ServiceControl.Persistence.EFCore.Entities;
using ServiceControl.Persistence.EFCore.Infrastructure;
using ServiceControl.Recoverability;
using ServiceControl.Transports;

class When_hosting_error_ingestion_only : AcceptanceTest
Expand Down Expand Up @@ -98,9 +97,37 @@ public async Task Should_default_the_concurrency_to_10_when_none_is_configured()
"InternalCustomChecksHostedService", // reports this node's ingestion health to the database
"MetricsReporterHostedService",
"HealthCheckPublisherHostedService", // inert, no IHealthCheckPublisher is registered
"ExternalIntegrationRequestsDataStore" // its drain is inert here, nothing calls Subscribe
"ExternalIntegrationRequestsDataStore", // its drain is inert here, nothing calls Subscribe
"FinishedCopyBeforeAnIngestionNodeOpens", // keeps this node out of a database a copy has not finished filling
"RecordHostOpenedOnTarget" // idempotent upsert of a marker only read for presence, safe on every node
];

[Test]
public async Task Should_refuse_to_start_while_a_copy_into_the_database_is_unfinished()
{
var settings = await CreateSettings();

await new SetupCommand().Execute(new HostArguments([]), settings);

var host = ErrorIngestionOnlyCommand.BuildHost(settings);

try
{
await host.Services.GetRequiredService<IMigrationCheckpointStore>().Upsert(
new MigrationCheckpoint(MigrationCategoryIds.KnownEndpoints, MigrationCategoryState.Halted, null, 0, 0, null, null, null, null, null, "The source stopped responding."));

var exception = Assert.ThrowsAsync<Exception>(() => host.StartAsync());

Assert.That(exception.Message, Does.Contain("has not finished")
.And.Contain(MigrationCategoryIds.KnownEndpoints)
.And.Contain("--migration-abandon KnownEndpoints"));
}
finally
{
await host.DisposeAsync();
}
}

[Test]
public void Should_refuse_to_start_against_unsupported_storage()
{
Expand Down Expand Up @@ -205,6 +232,7 @@ public async Task Should_serve_health_over_https_with_the_configured_certificate
await new SetupCommand().Execute(new HostArguments([]), settings);

host = ErrorIngestionOnlyCommand.BuildHost(settings);
host.Urls.Clear();
host.Urls.Add("https://127.0.0.1:0");

await host.StartAsync();
Expand Down
6 changes: 6 additions & 0 deletions src/ServiceControl.Migration.AcceptanceTests/.editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[*.cs]

# Justification: Test project
dotnet_diagnostic.CA2007.severity = none
dotnet_diagnostic.PS0013.severity = none
dotnet_diagnostic.PS0018.severity = none
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
namespace ServiceControl.Migration.AcceptanceTests;

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using NUnit.Framework;
using ServiceControl.Contracts.CustomChecks;
using ServiceControl.Operations;
using ServiceControl.Persistence;
using ServiceControl.Persistence.DataMigration;
using ServiceControl.Persistence.Infrastructure;

[TestFixture]
[NonParallelizable]
class CustomCheckMigrationTests : MigrationAcceptanceTest
{
static readonly EndpointDetails Endpoint = new() { Name = "Sales", Host = "MACHINE", HostId = Guid.NewGuid() };

static Guid IdOf(EndpointDetails endpoint, string customCheckId) =>
new CustomCheckDetail { OriginatingEndpoint = endpoint, CustomCheckId = customCheckId, Category = "Ping" }.GetDeterministicId();

async Task<IList<CustomCheckView>> CopiedChecks() => (await CustomChecks.GetStats(new PagingInfo())).Results!;

// One of each status, because Status.Fail is the enum's zero and a status that never deserializes would read as Fail.
[Test]
public async Task Copies_a_failing_and_a_passing_custom_check_with_their_deterministic_ids_and_reported_status()
{
await SeedSourceCustomCheck(Endpoint, "Sales.HealthCheck", "Ping", failed: true, reason: "timed out");
await SeedSourceCustomCheck(Endpoint, "Sales.DiskCheck", "Disk", failed: false);

var result = await CopyCategory("CustomChecks");

var copied = await CopiedChecks();
var failing = copied.Single(check => check.CustomCheckId == "Sales.HealthCheck");
var passing = copied.Single(check => check.CustomCheckId == "Sales.DiskCheck");

using (Assert.EnterMultipleScope())
{
Assert.That(result.Copied, Is.EqualTo(2));
Assert.That(await Target.Count(MigrationCategoryRegistry.Find(MigrationCategoryIds.CustomChecks)!), Is.EqualTo(2), "Count is what --migration-verify reads");
Assert.That(Guid.Parse(failing.Id!), Is.EqualTo(IdOf(Endpoint, "Sales.HealthCheck")), "a live report of the same check upserts on this id, so it replaces the copied row rather than adding a second");
Assert.That(failing.Status, Is.EqualTo(Status.Fail));
Assert.That(failing.Category, Is.EqualTo("Ping"));
Assert.That(failing.FailureReason, Is.EqualTo("timed out"));
Assert.That(failing.ReportedAt, Is.EqualTo(SeededReportedAt));
Assert.That((failing.OriginatingEndpoint!.Name, failing.OriginatingEndpoint.Host, failing.OriginatingEndpoint.HostId), Is.EqualTo((Endpoint.Name, Endpoint.Host, Endpoint.HostId)));
Assert.That(Guid.Parse(passing.Id!), Is.EqualTo(IdOf(Endpoint, "Sales.DiskCheck")));
Assert.That(passing.Status, Is.EqualTo(Status.Pass), "a status read back as the enum's zero would show every copied check red");
Assert.That(passing.Category, Is.EqualTo("Disk"));
Assert.That(passing.FailureReason, Is.Null);
}
}

[Test]
public async Task A_check_with_no_originating_endpoint_is_skipped_as_a_required_value_missing()
{
await SeedSource(($"CustomChecks/{Guid.NewGuid()}", new { CustomCheckId = "Sales.HealthCheck", Category = "Ping", Status = (int)Status.Pass, ReportedAt = DateTime.UtcNow }));

var result = await CopyCategory("CustomChecks");
var checkpoint = await ReadCheckpoint("CustomChecks");

using (Assert.EnterMultipleScope())
{
Assert.That(result.Saved.State, Is.EqualTo(MigrationCategoryState.CompleteWithErrors), "one fault skip ends a required category Failed");
Assert.That(checkpoint.CopiedCount, Is.Zero);
Assert.That(checkpoint.SkipReasons?.GetValueOrDefault(MigrationSkipReason.RequiredValueMissing), Is.EqualTo(1), "CustomChecks.OriginatingEndpointName is NOT NULL, and a throw would stop the batch rather than count the row");
}
}

[Test]
public async Task Only_a_check_whose_id_and_category_match_a_RavenDB_persister_check_is_skipped_as_harmless_and_the_category_ends_done()
{
var serviceControl = new EndpointDetails { Name = "Particular.ServiceControl", Host = "MACHINE", HostId = Guid.NewGuid() };
var audit = new EndpointDetails { Name = "Particular.ServiceControl.Audit", Host = "MACHINE", HostId = Guid.NewGuid() };
await SeedSourceCustomCheck(serviceControl, "Error Database Index Lag", "ServiceControl Health", failed: true, reason: "index lag");
await SeedSourceCustomCheck(audit, "Audit Database Index Lag", "ServiceControl.Audit Health", failed: true, reason: "index lag");
await SeedSourceCustomCheck(audit, "RavenDB dirty memory", "ServiceControl.Audit Health", failed: true, reason: "dirty memory");
await SeedSourceCustomCheck(Endpoint, "Sales.HealthCheck", "Ping", failed: true, reason: "timed out");
await SeedSourceCustomCheck(Endpoint, "Message Ingestion Process", "Sales", failed: true, reason: "stuck");

var result = await CopyCategory("CustomChecks");

using (Assert.EnterMultipleScope())
{
Assert.That(result.Saved.State, Is.EqualTo(MigrationCategoryState.Complete), "a harmless skip leaves a required category Done, so ServiceControl can open");
Assert.That(result.Saved.SkipReasons, Is.EqualTo(new Dictionary<MigrationSkipReason, long> { [MigrationSkipReason.SourceStorageCheck] = 1 }), "only the primary's index lag check is the RavenDB persister's own");
Assert.That((await CopiedChecks()).Select(check => (check.CustomCheckId, check.Category)), Is.EquivalentTo(new[]
{
("Audit Database Index Lag", "ServiceControl.Audit Health"),
("RavenDB dirty memory", "ServiceControl.Audit Health"),
("Sales.HealthCheck", "Ping"),
("Message Ingestion Process", "Sales")
}), "the skip matches the id and the category, so the audit instance's checks and a customer check with a colliding id keep their failing results");
}
}
}
Loading
Loading