Repository navigation
[IN PROGRESS] Migration engine for the primary instance - #5894
Draft
warwickschroeder wants to merge 30 commits into
Draft
warwickschroeder wants to merge 30 commits into
warwickschroeder wants to merge 30 commits into
Conversation
rbev
reviewed
Sep 15, 2026
rbev
left a comment
Contributor
There was a problem hiding this comment.
Partial review - adding as comments for now.
rbev
reviewed
Sep 16, 2026
johnsimons
approved these changes
Sep 17, 2026
johnsimons
left a comment
Member
There was a problem hiding this comment.
A few question and comments but overall, lets keep going
warwickschroeder
force-pushed
the
warwick/migration-engine-1
branch
from
September 30, 2026 01:24
8949344 to
0dcf9c8
Compare
rbev
marked this pull request as ready for review
November 11, 2026 13:17
rbev
force-pushed
the
warwick/migration-engine-1
branch
2 times, most recently
from
November 11, 2026 14:00
af5b8ed to
6ae519b
Compare
warwickschroeder
marked this pull request as draft
November 12, 2026 07:07
warwickschroeder
force-pushed
the
warwick/migration-engine-1
branch
from
November 20, 2026 04:08
f770bf8 to
f76aa71
Compare
…d ServiceControl migration
…n, clarifying strategies, goals, and migration limitations.
…on source reporting - Implemented MigrationEngineFailurePathTests to validate behavior during write failures and halts. - Created MigrationEngineHaltTests to ensure categories halt correctly on systemic failures. - Added MigrationEngineOptionsTests to verify default and environment variable configurations. - Developed MigrationEngineOrderingTests to check execution order of migration categories. - Introduced MigrationEngineResumeTests to confirm no duplicates or gaps after restarts. - Added MigrationEngineRunCategoriesTests to ensure all categories run in specified order. - Implemented MigrationEngineSkipReasonTests to validate skip reasons and their aggregation. - Enhanced MigrationEngineThrottleTests to verify pause behavior for optional categories. - Updated MigrationSourceReportCommand for improved output and clarity. - Modified Help.txt and HostArguments.cs for better command descriptions. - Refined Settings.cs and PersistenceFactory.cs for clearer migration source configuration.
…r SQL migration from RavenDB
- Introduced `BenignSkipped` property in `MigrationWriteResult` to track rows that the target would have deleted anyway. - Updated `MigrationEngine` logic to ensure benign skips do not count towards the halt threshold. - Added tests to verify that benign skips are handled correctly and do not cause category halts. - Updated skip reasons to include `PastRetention` for better clarity on skipped rows. - Created approval files for category selection tests to ensure correct order and configuration.
- Improved error logging in MigrationEngine to provide clearer context on exceptions. - Added functionality to InMemoryMigrationSource to simulate shutdown behavior during body reads. - Enhanced InMemoryMigrationTarget to allow custom exceptions for failure scenarios. - Introduced TimerRecordingTimeProvider to facilitate testing of timer-related functionality. - Expanded HaltThresholdTests to cover edge cases involving mixed benign and fault skips. - Updated MigrationEngineBodyRetryTests to ensure proper handling of shutdowns during body reads. - Added tests to verify behavior when categories are empty or already completed. - Implemented checks for checkpoint conflicts and cancellation scenarios in MigrationEngineFailurePathTests. - Enhanced MigrationEngineHaltTests to evaluate mixed skip scenarios and their impact on halting. - Improved MigrationEngineOrderingTests to ensure blocked categories behave correctly. - Added tests to MigrationEngineResumeTests to verify that restart behavior maintains original start time. - Updated MigrationEngineRunCategoriesTests to handle cases where no categories are selected. - Enhanced MigrationEngineSkipReasonTests to ensure accurate recording of multiple skip reasons. - Added tests to MigrationEngineThrottleTests to verify behavior during pauses and shutdowns.
…r imports handling
…ver and PostgreSQL (#5897) * feat: Add Migration Checkpoints to EF Core - Introduced a new table `MigrationCheckpoints` to track migration states and progress. - Implemented `MigrationCheckpointEntity` to represent the checkpoint data structure. - Added `EFMigrationCheckpointStore` for managing checkpoint data with methods for reading and upserting checkpoints. - Created `MigrationCheckpointConfiguration` for configuring the entity in the DbContext. - Enhanced `ServiceControlDbContext` to include the new `MigrationCheckpoints` DbSet. - Updated `EfCoreExtensions` to provide an `UpsertCheckpoint` method for managing checkpoint persistence. - Added unit tests for `EFMigrationCheckpointStore` and `MigrationCheckpointTable` to ensure correct functionality. - Implemented transaction handling in `MigrationCheckpointTransactionTests` to verify checkpoint behavior during commits and rollbacks. - Updated RavenDB integration to support embedded server readiness checks. * feat: Enhance migration engine with improved error handling and configuration options * feat: Refactor migration checkpoint handling and add UpsertCheckpoint method * docs: Update migration documentation for clarity on RavenDB to SQL migration process * [Migration Engine Part 3] Implement the RavenDB to SQL migration machinery (#5911) * Update migration documentation * feat: run the required migration copy at host start Adds the RavenDB source and the EF Core target for the KnownEndpoints and EndpointSettings categories, the startup checks that refuse an unsupported or unready migration, and the stall watchdog that stops a copy committing nothing for 30 minutes. Documents the migration contracts. * test: cover the RavenDB to SQL migration end to end Adds unit tests for the startup checks, the stall watchdog and the refusals, target and reader tests for both persisters, a SQL Server collation test for the endpoint settings key, and acceptance tests for a copy that is killed, restarted and opened on. Approves the two new migration settings. * tests: ensure DB schema is setup before the host starts * refactor: update BatchSizeFor method to return Task<int> for async compatibility across migration targets * docs: update migration instructions and system design to clarify settings and behavior * Refactor migration checks and improve error handling * Refactor migration commands and error handling * PR review * Fix flakey test
… SQL (#6005) * Add support for failed message migration with bodys * Enhance migration process and tests: improve dry run estimates, add handling for failure details without timestamps, and optimize SQL Server and PostgreSQL dialects for better performance. * Add details on handling unresolved and retry-issued failed messages in migration documentation * Fix migration process for failed messages: add retry claims for retry-issued messages, update tests for accurate staging behavior, and refine documentation on message handling. * Clarify message skip reasons in migration documentation: differentiate between fault and harmless skips, and specify conditions for retries. * Remove unnecessary phrasing from migration overview documentation * Refactor FailedMessagesWriter to use FailedMessageDocumentTranslator * Add test for handling null content type in FailedMessageDocumentTranslator * Refactor indexing wait methods to allow error states in tests and update exception handling in migration tests * Add error handling for RavenDB index queries in FailedMessagesReader
* Add licensing data migration * Update migration docs * Clarify licensing endpoint handling in migration overview documentation * Enhance licensing endpoints and throughput readers to support ordinal document-id order and cursor resumption; add tests for endpoint reading behavior. * Refactor licensing throughput handling to aggregate counts for endpoints with names differing only in case and trailing spaces; update related tests and documentation. * Enhance SQL Server migration dialect to support key comparison ignoring trailing spaces; introduce PaddedKeyComparer for improved equality checks. * Enhance RetentionSweepTests to support cancellation during database operations; add delay handling for command execution. * Clarify open decisions in migration documentation; update references to gaps and data handling in SQL migration process.
* Change offline migration step to an interactive cli action * Split step * Refactor migration cancellation token handling in MigrateCommand and OptionalCategoryCopier * Refine migration documentation by removing redundant statements and clarifying instructions * Remove redundant exception handling in MigrateCommand * Update exception message in TargetHoldsNoServiceControlDataCheck for clarity on migration steps * Remove CheckpointMigrationState and IMigrationState interfaces along with related tests. Not needed now with the --migrate command * Enhance migration documentation and error messages for clarity * Add "RecordHostOpenedOnTarget" to the list of services in When_hosting_error_ingestion_only test for idempotent upsert * Update migration commands from --migrate to --start-data-migration across documentation and codebase --------- Co-authored-by: Warwick Schroeder <warwick.schroeder@particular.net>
warwickschroeder
force-pushed
the
warwick/migration-engine-1
branch
from
November 20, 2026 04:39
f76aa71 to
49f63e6
Compare
- Copy message redirects, subscriptions, retry operations, notification settings and the trial end date - Report rows merged by the SQL key instead of dropping them silently - Delete store-keyed rows when a category is reset for a retry
…ntegration events, custom checks, failed error imports with their bodies, and group comments. With them, the offline copy covers all sixteen required categories.
…ttings to SQL (#6137) * Copy redirects, subscriptions, retry history and settings to SQL - Copy message redirects, subscriptions, retry operations, notification settings and the trial end date - Report rows merged by the SQL key instead of dropping them silently - Delete store-keyed rows when a category is reset for a retry * Refactor migration documentation for clarity and accuracy * Update migration overview to clarify custom ServicePulse UI for migration status * Trigger CI
Migration Engine: Copy the remaining required categories to SQL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
The migration engine moves an error instance's data from RavenDB to SQL Server or PostgreSQL.
--start-data-migrationcopies the data ServiceControl needs to open while it is stopped, then ServiceControl starts on SQL and copies the rest in the background. This is the base branch for the whole engine. It is not finished, and a Release build still refuses to copy anything until the last milestone below.--start-data-migrationcopies the required data with ServiceControl stopped, and a second run after Ctrl+C or a crash resumes from the last saved batch.--import-failed-errorsrefuse to start while any required data is still unfinished.--migration-source-reportprints its version, location, databases and row counts.SERVICECONTROL_MIGRATION_ALLOW_UNRELEASEDis true.Migrationtest category.Milestones
Each milestone lands on this branch as its own PR. This branch goes to
masteronce the last one is in.--start-data-migrationruns the required copy with ServiceControl stopped, and the progress table in SQL holds the migration state.--migration-retryand--migration-abandon, so a Failed category can be copied again or given up.--migration-status,--migration-verifyand the dry run.Tests
ServiceControl.UnitTests/Migration: the engine, its halt and resume rules, each startup check, the refusal messages, the background copier and the Debug switch, against fakes and a fake clock.ServiceControl.UnitTests/Hosting: parsing--migration-source-reportand--start-data-migration.ServiceControl.UnitTests/ApprovalFiles: the fixed category order.ServiceControl.Persistence.Tests/EFCore: the progress table, the target and its writers, on SQL Server and PostgreSQL.ServiceControl.Persistence.Tests.RavenDB/DataMigration: the read-only source and its readers against an embedded server.ServiceControl.Persistence.Tests.SqlServer: endpoint settings keys that differ only in letter case, value length checks and the batch insert.ServiceControl.Persistence.Tests.PostgreSql: value length checks.ServiceControl.Migration.Tests(new): the source report, both persisters in one process, and every reader having a matching writer.ServiceControl.Migration.AcceptanceTests(new): the offline command end to end on both providers, including a killed copy, each refusal, ServiceControl opening on SQL, failed messages with bodies and the licensing data.