Skip to content

[IN PROGRESS] Migration engine for the primary instance - #5894

Draft
warwickschroeder wants to merge 30 commits into
masterfrom
warwick/migration-engine-1
Draft

warwickschroeder wants to merge 30 commits into
masterfrom
warwick/migration-engine-1

Conversation

@warwickschroeder

@warwickschroeder warwickschroeder commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

What this adds

The migration engine moves an error instance's data from RavenDB to SQL Server or PostgreSQL. --start-data-migration copies the data ServiceControl needs to open while it is stopped, then ServiceControl starts on SQL and copies the rest in the background. This is the base branch for the whole engine. It is not finished, and a Release build still refuses to copy anything until the last milestone below.

  • --start-data-migration copies the required data with ServiceControl stopped, and a second run after Ctrl+C or a crash resumes from the last saved batch.
  • Copied so far: known endpoints, endpoint settings, unresolved and retry-issued failed messages with their groups and bodies, and the licensing throughput history.
  • Each batch saves its rows and its progress together, in a new progress table on SQL Server and PostgreSQL, so a stop never loses or doubles a row.
  • Startup checks refuse before the first row moves, and each refusal says what to fix.
  • ServiceControl, ingestion-only workers and --import-failed-errors refuse to start while any required data is still unfinished.
  • The first time a host opens on SQL is recorded, because going back to RavenDB is only free before that.
  • A background copier starts with ServiceControl for the optional data, but there is no optional data it can copy yet.
  • RavenDB is only ever read, and --migration-source-report prints its version, location, databases and row counts.
  • RavenDB to SQL Server or PostgreSQL is the only allowed pair, though the engine talks only to a source and a target and can take other pairs later.
  • A Debug build copies for real only when SERVICECONTROL_MIGRATION_ALLOW_UNRELEASED is true.
  • Docs describe the finished migration in the overview, instructions and system design, and CI gains a Migration test category.

Milestones

Each milestone lands on this branch as its own PR. This branch goes to master once the last one is in.

Tests

  • ServiceControl.UnitTests/Migration: the engine, its halt and resume rules, each startup check, the refusal messages, the background copier and the Debug switch, against fakes and a fake clock.
  • ServiceControl.UnitTests/Hosting: parsing --migration-source-report and --start-data-migration.
  • ServiceControl.UnitTests/ApprovalFiles: the fixed category order.
  • ServiceControl.Persistence.Tests/EFCore: the progress table, the target and its writers, on SQL Server and PostgreSQL.
  • ServiceControl.Persistence.Tests.RavenDB/DataMigration: the read-only source and its readers against an embedded server.
  • ServiceControl.Persistence.Tests.SqlServer: endpoint settings keys that differ only in letter case, value length checks and the batch insert.
  • ServiceControl.Persistence.Tests.PostgreSql: value length checks.
  • ServiceControl.Migration.Tests (new): the source report, both persisters in one process, and every reader having a matching writer.
  • ServiceControl.Migration.AcceptanceTests (new): the offline command end to end on both providers, including a killed copy, each refusal, ServiceControl opening on SQL, failed messages with bodies and the licensing data.

@rbev rbev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partial review - adding as comments for now.

@warwickschroeder warwickschroeder changed the title Implement the foundation for migration engine [Migration Engine Part 2] Implement the foundation for migration engine Sep 16, 2026
@warwickschroeder warwickschroeder changed the title [Migration Engine Part 2] Implement the foundation for migration engine [Migration Engine Part 1] Implement the foundation for migration engine Sep 16, 2026

@johnsimons johnsimons left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few question and comments but overall, lets keep going

@warwickschroeder
warwickschroeder force-pushed the warwick/migration-engine-1 branch from 8949344 to 0dcf9c8 Compare September 30, 2026 01:24
@warwickschroeder
warwickschroeder removed this pull request from stack #5898 October 9, 2026 01:45
@warwickschroeder warwickschroeder changed the title [Migration Engine Part 1] Implement the foundation for migration engine [IN PROGRESS] Migration engine for the primary instance Oct 9, 2026
@rbev
rbev added this pull request to stack #6109 November 11, 2026 13:17
@rbev
rbev marked this pull request as ready for review November 11, 2026 13:17
@rbev
rbev force-pushed the warwick/migration-engine-1 branch 2 times, most recently from af5b8ed to 6ae519b Compare November 11, 2026 14:00
@warwickschroeder
warwickschroeder marked this pull request as draft November 12, 2026 07:07
@warwickschroeder
warwickschroeder removed this pull request from stack #6109 November 12, 2026 08:10
@warwickschroeder
warwickschroeder force-pushed the warwick/migration-engine-1 branch from f770bf8 to f76aa71 Compare November 20, 2026 04:08
@warwickschroeder
warwickschroeder removed this pull request from stack #6157 November 20, 2026 04:26
…n, clarifying strategies, goals, and migration limitations.
…on source reporting

- Implemented MigrationEngineFailurePathTests to validate behavior during write failures and halts.
- Created MigrationEngineHaltTests to ensure categories halt correctly on systemic failures.
- Added MigrationEngineOptionsTests to verify default and environment variable configurations.
- Developed MigrationEngineOrderingTests to check execution order of migration categories.
- Introduced MigrationEngineResumeTests to confirm no duplicates or gaps after restarts.
- Added MigrationEngineRunCategoriesTests to ensure all categories run in specified order.
- Implemented MigrationEngineSkipReasonTests to validate skip reasons and their aggregation.
- Enhanced MigrationEngineThrottleTests to verify pause behavior for optional categories.
- Updated MigrationSourceReportCommand for improved output and clarity.
- Modified Help.txt and HostArguments.cs for better command descriptions.
- Refined Settings.cs and PersistenceFactory.cs for clearer migration source configuration.
warwickschroeder and others added 16 commits November 20, 2026 12:29
- Introduced `BenignSkipped` property in `MigrationWriteResult` to track rows that the target would have deleted anyway.
- Updated `MigrationEngine` logic to ensure benign skips do not count towards the halt threshold.
- Added tests to verify that benign skips are handled correctly and do not cause category halts.
- Updated skip reasons to include `PastRetention` for better clarity on skipped rows.
- Created approval files for category selection tests to ensure correct order and configuration.
- Improved error logging in MigrationEngine to provide clearer context on exceptions.
- Added functionality to InMemoryMigrationSource to simulate shutdown behavior during body reads.
- Enhanced InMemoryMigrationTarget to allow custom exceptions for failure scenarios.
- Introduced TimerRecordingTimeProvider to facilitate testing of timer-related functionality.
- Expanded HaltThresholdTests to cover edge cases involving mixed benign and fault skips.
- Updated MigrationEngineBodyRetryTests to ensure proper handling of shutdowns during body reads.
- Added tests to verify behavior when categories are empty or already completed.
- Implemented checks for checkpoint conflicts and cancellation scenarios in MigrationEngineFailurePathTests.
- Enhanced MigrationEngineHaltTests to evaluate mixed skip scenarios and their impact on halting.
- Improved MigrationEngineOrderingTests to ensure blocked categories behave correctly.
- Added tests to MigrationEngineResumeTests to verify that restart behavior maintains original start time.
- Updated MigrationEngineRunCategoriesTests to handle cases where no categories are selected.
- Enhanced MigrationEngineSkipReasonTests to ensure accurate recording of multiple skip reasons.
- Added tests to MigrationEngineThrottleTests to verify behavior during pauses and shutdowns.
…ver and PostgreSQL (#5897)

* feat: Add Migration Checkpoints to EF Core

- Introduced a new table `MigrationCheckpoints` to track migration states and progress.
- Implemented `MigrationCheckpointEntity` to represent the checkpoint data structure.
- Added `EFMigrationCheckpointStore` for managing checkpoint data with methods for reading and upserting checkpoints.
- Created `MigrationCheckpointConfiguration` for configuring the entity in the DbContext.
- Enhanced `ServiceControlDbContext` to include the new `MigrationCheckpoints` DbSet.
- Updated `EfCoreExtensions` to provide an `UpsertCheckpoint` method for managing checkpoint persistence.
- Added unit tests for `EFMigrationCheckpointStore` and `MigrationCheckpointTable` to ensure correct functionality.
- Implemented transaction handling in `MigrationCheckpointTransactionTests` to verify checkpoint behavior during commits and rollbacks.
- Updated RavenDB integration to support embedded server readiness checks.

* feat: Enhance migration engine with improved error handling and configuration options

* feat: Refactor migration checkpoint handling and add UpsertCheckpoint method

* docs: Update migration documentation for clarity on RavenDB to SQL migration process

* [Migration Engine Part 3] Implement the RavenDB to SQL migration machinery (#5911)

* Update migration documentation

* feat: run the required migration copy at host start

Adds the RavenDB source and the EF Core target for the KnownEndpoints and EndpointSettings categories, the startup checks that refuse an unsupported or unready migration, and the stall watchdog that stops a copy committing nothing for 30 minutes. Documents the migration contracts.

* test: cover the RavenDB to SQL migration end to end

Adds unit tests for the startup checks, the stall watchdog and the refusals, target and reader tests for both persisters, a SQL Server collation test for the endpoint settings key, and acceptance tests for a copy that is killed, restarted and opened on. Approves the two new migration settings.

* tests: ensure DB schema is setup before the host starts

* refactor: update BatchSizeFor method to return Task<int> for async compatibility across migration targets

* docs: update migration instructions and system design to clarify settings and behavior

* Refactor migration checks and improve error handling

* Refactor migration commands and error handling

* PR review

* Fix flakey test
… SQL (#6005)

* Add support for failed message migration with bodys

* Enhance migration process and tests: improve dry run estimates, add handling for failure details without timestamps, and optimize SQL Server and PostgreSQL dialects for better performance.

* Add details on handling unresolved and retry-issued failed messages in migration documentation

* Fix migration process for failed messages: add retry claims for retry-issued messages, update tests for accurate staging behavior, and refine documentation on message handling.

* Clarify message skip reasons in migration documentation: differentiate between fault and harmless skips, and specify conditions for retries.

* Remove unnecessary phrasing from migration overview documentation

* Refactor FailedMessagesWriter to use FailedMessageDocumentTranslator

* Add test for handling null content type in FailedMessageDocumentTranslator

* Refactor indexing wait methods to allow error states in tests and update exception handling in migration tests

* Add error handling for RavenDB index queries in FailedMessagesReader
* Add licensing data migration

* Update migration docs

* Clarify licensing endpoint handling in migration overview documentation

* Enhance licensing endpoints and throughput readers to support ordinal document-id order and cursor resumption; add tests for endpoint reading behavior.

* Refactor licensing throughput handling to aggregate counts for endpoints with names differing only in case and trailing spaces; update related tests and documentation.

* Enhance SQL Server migration dialect to support key comparison ignoring trailing spaces; introduce PaddedKeyComparer for improved equality checks.

* Enhance RetentionSweepTests to support cancellation during database operations; add delay handling for command execution.

* Clarify open decisions in migration documentation; update references to gaps and data handling in SQL migration process.
* Change offline migration step to an interactive cli action

* Split step

* Refactor migration cancellation token handling in MigrateCommand and OptionalCategoryCopier

* Refine migration documentation by removing redundant statements and clarifying instructions

* Remove redundant exception handling in MigrateCommand

* Update exception message in TargetHoldsNoServiceControlDataCheck for clarity on migration steps

* Remove CheckpointMigrationState and IMigrationState interfaces along with related tests. Not needed now with the --migrate command

* Enhance migration documentation and error messages for clarity

* Add "RecordHostOpenedOnTarget" to the list of services in When_hosting_error_ingestion_only test for idempotent upsert

* Update migration commands from --migrate to --start-data-migration across documentation and codebase

---------

Co-authored-by: Warwick Schroeder <warwick.schroeder@particular.net>
@warwickschroeder
warwickschroeder force-pushed the warwick/migration-engine-1 branch from f76aa71 to 49f63e6 Compare November 20, 2026 04:39
- Copy message redirects, subscriptions, retry operations, notification settings and the trial end date
- Report rows merged by the SQL key instead of dropping them silently
- Delete store-keyed rows when a category is reset for a retry
…ntegration events, custom checks, failed error imports with their bodies, and group comments. With them, the offline copy covers all sixteen required categories.
…ttings to SQL (#6137)

* Copy redirects, subscriptions, retry history and settings to SQL

- Copy message redirects, subscriptions, retry operations, notification settings and the trial end date
- Report rows merged by the SQL key instead of dropping them silently
- Delete store-keyed rows when a category is reset for a retry

* Refactor migration documentation for clarity and accuracy

* Update migration overview to clarify custom ServicePulse UI for migration status

* Trigger CI
Migration Engine: Copy the remaining required categories to SQL
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants