Skip to content

Split failed messages into offline and online stages - #6162

Open
rbev wants to merge 3 commits into
warwick/migration-engine-1from
migration-split-error-migration
Open

rbev wants to merge 3 commits into
warwick/migration-engine-1from
migration-split-error-migration

Conversation

@rbev

@rbev rbev commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

This pull request updates the migration documentation to clarify how unresolved and retry-issued failed messages are handled during the migration from RavenDB to SQL, and to document new requirements and blockers for a successful migration. The changes improve operator guidance, specify which messages are copied during outage and background phases, and explain how unfinished operations are handled.

Clarifications and updates to migration process:

  • The documentation now explains that unresolved failed messages are copied in the background after ServiceControl opens, and only retry-issued messages and those with retry claims are required during the outage. A large unresolved backlog no longer lengthens the outage, but does extend the background copy. [1] [2]
  • Operators are instructed to let retries, archive, and unarchive operations finish on RavenDB before migration; migration now refuses to proceed if any persisted operation documents remain. Steps to resolve unfinished operations are provided. [1] [2] [3]

Documentation of migration blockers and release readiness:

  • The overview and instructions now explicitly reference unresolved release blockers for the failed-message split, and link to their listing in the documentation. [1] [2]

Detailed category and data handling changes:

  • The number and definition of migration categories are updated: unresolved failed messages are now an optional category copied after opening, and the required category is retry-issued messages and those with retry claims. The handling of group comments and retry claims is clarified, and the refusal to migrate with in-flight operation documents is documented. [1] [2]
  • The documentation now specifies that unresolved failed messages have no window and are always copied after opening, regardless of age. [1] [2]

Behavioral clarifications for data writes:

  • The migration now documents that existing SQL rows always win (no updates are made), and that message bodies are immutable and only written if absent.
  • If the unresolved failed messages category fails, the unresolved backlog remains only in RavenDB.

@rbev
rbev added this pull request to stack #6157 November 19, 2026 04:42
@rbev rbev changed the title Add pre-flight check for incomplete batch operations Split failed messages into offline and online stages Nov 19, 2026
@rbev
rbev requested a review from johnsimons November 20, 2026 01:04
@rbev
rbev marked this pull request as ready for review November 20, 2026 01:04
@warwickschroeder
warwickschroeder removed this pull request from stack #6157 November 20, 2026 04:26
Base automatically changed from migration-engine-8 to warwick/migration-engine-1 November 20, 2026 05:38
@rbev
rbev force-pushed the migration-split-error-migration branch from ad60baf to 9a63050 Compare November 20, 2026 05:54

@warwickschroeder warwickschroeder left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main issue is that group comments would be cleaned up by the retention sweep if they are left as required and migrated before failed messages

Comment on lines 3 to +4
> [!NOTE]
> This build does not carry the whole migration yet. This page describes it as it will be when it ships.
> This build does not carry the whole migration yet. This page describes it as it will be when it ships. `MigrationIsReleasedCheck` refuses production copies until it does.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can probable remove this entire note now.

| Custom checks, with their last status | Required | Not every check reports again. An endpoint that is down never does, and a check with no repeat interval reports only when its endpoint starts. Leaving one out could hide a known failure until that endpoint restarts. Results of the six checks the RavenDB persister runs about its own database, matched on id and category, are left out, as harmless skips |
| Failed error imports, with bodies | Required | Messages taken off the error queue but not ingested. They exist nowhere else and never expire. After the move the "Error Message Ingestion" custom check keeps flagging them and `--import-failed-errors` imports them into SQL. Importing them on RavenDB first is better still |
| Group comments | Required | Copied after unresolved failed messages, so a note like "do not retry" is there when ServiceControl opens. Blank comments are not copied. Some comments are removed after ServiceControl opens, see [Open decisions](#rows-removed-after-the-copy) |
| Group comments | Required | Copied after the required failed messages and never waiting for an optional category. Blank comments are not copied. A comment on a group whose messages have not reached SQL yet is removed shortly after ServiceControl opens, an accepted edge case, see [Rows removed after the copy](#rows-removed-after-the-copy) |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| Group comments | Required | Copied after the required failed messages and never waiting for an optional category. Blank comments are not copied. A comment on a group whose messages have not reached SQL yet is removed shortly after ServiceControl opens, an accepted edge case, see [Rows removed after the copy](#rows-removed-after-the-copy) |
| Group comments | Required | Copied after the retry-issued and claim-holding failed messages, and never waiting for an optional category. Blank comments are not copied. A comment on a group whose messages have not reached SQL yet is removed shortly after ServiceControl opens, an accepted edge case, see [Rows removed after the copy](#rows-removed-after-the-copy) |

| Failed error imports, with bodies | Required | Messages taken off the error queue but not ingested. They exist nowhere else and never expire. After the move the "Error Message Ingestion" custom check keeps flagging them and `--import-failed-errors` imports them into SQL. Importing them on RavenDB first is better still |
| Group comments | Required | Copied after unresolved failed messages, so a note like "do not retry" is there when ServiceControl opens. Blank comments are not copied. Some comments are removed after ServiceControl opens, see [Open decisions](#rows-removed-after-the-copy) |
| Group comments | Required | Copied after the required failed messages and never waiting for an optional category. Blank comments are not copied. A comment on a group whose messages have not reached SQL yet is removed shortly after ServiceControl opens, an accepted edge case, see [Rows removed after the copy](#rows-removed-after-the-copy) |
| Unresolved failed messages, with bodies | Optional | Copied once ServiceControl has opened, whatever the age of the message, because an operator is waiting on each one. It has no window and no setting turns it off. A message that already holds a retry claim copied with the required category, so it counts as already present. An existing SQL row always wins, see [Writing to SQL](#writing-to-sql). Group retry, archive and unarchive act only on the messages copied so far, an accepted edge case |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It has no window and no setting turns it off

Optional categories should be able to be abandonded, stopped, or continued. The other optional categories havent been added yet but it should be true for them too. Setting a category to abandonded should be available with the PR im working on.

@@ -44,8 +50,9 @@ public static class MigrationCategoryRegistry
new(MigrationCategoryIds.GroupComments, MigrationCategoryKind.Required, CarriesBodies: false, Order: 16, MustFollow: MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GroupComments still copies offline, but the groups that only unclaimed unresolved messages belong to now have no FailedMessageGroups row until the background copy reaches them.

RetentionSweeper.SweepOrphanedGroupComments runs one minute after the host opens and deletes every comment whose group has no row, so most comments on unresolved groups are lost on a normal migration. I think GroupComments could become optional with MustFollow: UnresolvedFailedMessages.

(spaces.Groups.Single().Id, " "),
("group-with-no-copied-message", "Raised with the shipping team"));
await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages);
await CopyCategory(MigrationCategoryIds.UnresolvedFailedMessages);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to my other comment RE leaving group comments as required. The test isnt currently reflecting how prod will run.

2. Read why the category failed. The reason is in the startup refusal and in the output of `--migration-status`.
3. Correct the cause. It is usually outside the migration, for example the body storage cannot be reached, a certificate expired, a database is down, or a disk is full.
4. Run `--migration-retry <category>` to reset the migration progress for that category.
5. For a required category, run `--start-data-migration` again before starting ServiceControl. For an optional category, start ServiceControl. The category copies again from the start.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For an optional category, start ServiceControl. The category copies again from the start.

This probably isnt what we want. Optional categories should also start from their checkpoint cursor, and initiated by the API via SP.

using ServiceControl.Persistence.RavenDB.Recoverability;
using ServiceControl.Recoverability;

class UnfinishedOperationsCheckTests : RavenMigrationSourceTestBase

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests run ContributedChecks() themselves, so nothing proves the migrate command runs this check before copying. Could one acceptance test seed a RetryBatches/ document, run the command, and assert it refuses with nothing copied?

/// The retry-issued messages and the messages that hold a retry claim, copied before the host opens: every retry-issued message and
/// every other message that still holds a retry claim.
/// </summary>
public const string UnresolvedAndRetryIssuedFailedMessages = nameof(UnresolvedAndRetryIssuedFailedMessages);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probably rename this category now as it doesnt copy unresolved failed messages, apart from claim holders. Rename to RetryIssuedAndClaimedFailedMessages?

}

[Test]
public async Task A_comment_on_a_group_with_no_copied_message_is_reported_as_a_change_the_sweep_will_delete()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tests intent was to catch the odd group comment when the failed message wasnt there for some edgecase reason, which is flagged as harmless. Missing group comments because the failed message hasnt been migrated yes is not harmless.


[Test]
public void The_two_failed_message_categories_split_every_status_between_them()
public void The_unresolved_failed_messages_are_copied_after_the_host_opens_and_the_group_comments_do_not_wait_for_them()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, I think group comments should me migrated after them

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants