Repository navigation
Conversation
e0e150c to
e79b1e9
Compare
ad60baf to
9a63050
Compare
warwickschroeder
left a comment
There was a problem hiding this comment.
The main issue is that group comments would be cleaned up by the retention sweep if they are left as required and migrated before failed messages
| > [!NOTE] | ||
| > This build does not carry the whole migration yet. This page describes it as it will be when it ships. | ||
| > This build does not carry the whole migration yet. This page describes it as it will be when it ships. `MigrationIsReleasedCheck` refuses production copies until it does. |
There was a problem hiding this comment.
We can probable remove this entire note now.
| | Custom checks, with their last status | Required | Not every check reports again. An endpoint that is down never does, and a check with no repeat interval reports only when its endpoint starts. Leaving one out could hide a known failure until that endpoint restarts. Results of the six checks the RavenDB persister runs about its own database, matched on id and category, are left out, as harmless skips | | ||
| | Failed error imports, with bodies | Required | Messages taken off the error queue but not ingested. They exist nowhere else and never expire. After the move the "Error Message Ingestion" custom check keeps flagging them and `--import-failed-errors` imports them into SQL. Importing them on RavenDB first is better still | | ||
| | Group comments | Required | Copied after unresolved failed messages, so a note like "do not retry" is there when ServiceControl opens. Blank comments are not copied. Some comments are removed after ServiceControl opens, see [Open decisions](#rows-removed-after-the-copy) | | ||
| | Group comments | Required | Copied after the required failed messages and never waiting for an optional category. Blank comments are not copied. A comment on a group whose messages have not reached SQL yet is removed shortly after ServiceControl opens, an accepted edge case, see [Rows removed after the copy](#rows-removed-after-the-copy) | |
There was a problem hiding this comment.
| | Group comments | Required | Copied after the required failed messages and never waiting for an optional category. Blank comments are not copied. A comment on a group whose messages have not reached SQL yet is removed shortly after ServiceControl opens, an accepted edge case, see [Rows removed after the copy](#rows-removed-after-the-copy) | | |
| | Group comments | Required | Copied after the retry-issued and claim-holding failed messages, and never waiting for an optional category. Blank comments are not copied. A comment on a group whose messages have not reached SQL yet is removed shortly after ServiceControl opens, an accepted edge case, see [Rows removed after the copy](#rows-removed-after-the-copy) | |
| | Failed error imports, with bodies | Required | Messages taken off the error queue but not ingested. They exist nowhere else and never expire. After the move the "Error Message Ingestion" custom check keeps flagging them and `--import-failed-errors` imports them into SQL. Importing them on RavenDB first is better still | | ||
| | Group comments | Required | Copied after unresolved failed messages, so a note like "do not retry" is there when ServiceControl opens. Blank comments are not copied. Some comments are removed after ServiceControl opens, see [Open decisions](#rows-removed-after-the-copy) | | ||
| | Group comments | Required | Copied after the required failed messages and never waiting for an optional category. Blank comments are not copied. A comment on a group whose messages have not reached SQL yet is removed shortly after ServiceControl opens, an accepted edge case, see [Rows removed after the copy](#rows-removed-after-the-copy) | | ||
| | Unresolved failed messages, with bodies | Optional | Copied once ServiceControl has opened, whatever the age of the message, because an operator is waiting on each one. It has no window and no setting turns it off. A message that already holds a retry claim copied with the required category, so it counts as already present. An existing SQL row always wins, see [Writing to SQL](#writing-to-sql). Group retry, archive and unarchive act only on the messages copied so far, an accepted edge case | |
There was a problem hiding this comment.
It has no window and no setting turns it off
Optional categories should be able to be abandonded, stopped, or continued. The other optional categories havent been added yet but it should be true for them too. Setting a category to abandonded should be available with the PR im working on.
| @@ -44,8 +50,9 @@ public static class MigrationCategoryRegistry | |||
| new(MigrationCategoryIds.GroupComments, MigrationCategoryKind.Required, CarriesBodies: false, Order: 16, MustFollow: MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages), | |||
There was a problem hiding this comment.
GroupComments still copies offline, but the groups that only unclaimed unresolved messages belong to now have no FailedMessageGroups row until the background copy reaches them.
RetentionSweeper.SweepOrphanedGroupComments runs one minute after the host opens and deletes every comment whose group has no row, so most comments on unresolved groups are lost on a normal migration. I think GroupComments could become optional with MustFollow: UnresolvedFailedMessages.
| (spaces.Groups.Single().Id, " "), | ||
| ("group-with-no-copied-message", "Raised with the shipping team")); | ||
| await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); | ||
| await CopyCategory(MigrationCategoryIds.UnresolvedFailedMessages); |
There was a problem hiding this comment.
Related to my other comment RE leaving group comments as required. The test isnt currently reflecting how prod will run.
| 2. Read why the category failed. The reason is in the startup refusal and in the output of `--migration-status`. | ||
| 3. Correct the cause. It is usually outside the migration, for example the body storage cannot be reached, a certificate expired, a database is down, or a disk is full. | ||
| 4. Run `--migration-retry <category>` to reset the migration progress for that category. | ||
| 5. For a required category, run `--start-data-migration` again before starting ServiceControl. For an optional category, start ServiceControl. The category copies again from the start. |
There was a problem hiding this comment.
For an optional category, start ServiceControl. The category copies again from the start.
This probably isnt what we want. Optional categories should also start from their checkpoint cursor, and initiated by the API via SP.
| using ServiceControl.Persistence.RavenDB.Recoverability; | ||
| using ServiceControl.Recoverability; | ||
|
|
||
| class UnfinishedOperationsCheckTests : RavenMigrationSourceTestBase |
There was a problem hiding this comment.
These tests run ContributedChecks() themselves, so nothing proves the migrate command runs this check before copying. Could one acceptance test seed a RetryBatches/ document, run the command, and assert it refuses with nothing copied?
| /// The retry-issued messages and the messages that hold a retry claim, copied before the host opens: every retry-issued message and | ||
| /// every other message that still holds a retry claim. | ||
| /// </summary> | ||
| public const string UnresolvedAndRetryIssuedFailedMessages = nameof(UnresolvedAndRetryIssuedFailedMessages); |
There was a problem hiding this comment.
Should probably rename this category now as it doesnt copy unresolved failed messages, apart from claim holders. Rename to RetryIssuedAndClaimedFailedMessages?
| } | ||
|
|
||
| [Test] | ||
| public async Task A_comment_on_a_group_with_no_copied_message_is_reported_as_a_change_the_sweep_will_delete() |
There was a problem hiding this comment.
This tests intent was to catch the odd group comment when the failed message wasnt there for some edgecase reason, which is flagged as harmless. Missing group comments because the failed message hasnt been migrated yes is not harmless.
|
|
||
| [Test] | ||
| public void The_two_failed_message_categories_split_every_status_between_them() | ||
| public void The_unresolved_failed_messages_are_copied_after_the_host_opens_and_the_group_comments_do_not_wait_for_them() |
There was a problem hiding this comment.
Again, I think group comments should me migrated after them
This pull request updates the migration documentation to clarify how unresolved and retry-issued failed messages are handled during the migration from RavenDB to SQL, and to document new requirements and blockers for a successful migration. The changes improve operator guidance, specify which messages are copied during outage and background phases, and explain how unfinished operations are handled.
Clarifications and updates to migration process:
Documentation of migration blockers and release readiness:
Detailed category and data handling changes:
Behavioral clarifications for data writes: