Repository navigation
Conversation
recalculateDomainResourceCount() only selects existing resource_count rows since apache#8903. When a domain has no row for the type and tag, for example after a tag is added to resource.limit.storage.tags or resource.limit.host.tags, domainRC stays null. As soon as the accounts of the domain have usage for that tag, domainRC.setCount() throws a NullPointerException. updateResourceCount fails, and the periodic ResourceCountCheckTask stops at that domain on every run. Persist a new domain row with the recalculated count when none exists, like recalculateAccountResourceCount() already does for accounts. Fixes apache#14729
|
Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR makes
recalculateDomainResourceCount()create the domain'sresource_countrow when it is missing, instead of failing with aNullPointerException.Since #8903 the method only selects existing rows for the domain, its immediate child domains and its accounts. If the domain has no row for the type and tag,
domainRCstaysnull. That is the case for a tag that was added toresource.limit.storage.tagsorresource.limit.host.tagsafter the domain was created. As soon as the domain's accounts have any usage for that tag,oldResourceCount != newResourceCountanddomainRC.setCount()throws.recalculateAccountResourceCount()already persists a new row in the same situation for accounts. This change does the same for the domain: a new row with the recalculated count, including 0. When the row exists, nothing changes. The discrepancy warning is still logged when the count changes.Fixes: #14729
How I found it
I started from #14729 and checked it against the code. The method, and the
// TODO domainRC may be null ...comment right above the failing line, are the same in 4.20.3.1, on the 4.20 branch and on main. Before #8903,lockDomainRows()went throughlistAllRowsToUpdate(), which created missing rows. The new code only selects rows that already exist.Where this can show up in practice
This is my estimate from the code. The only field report I know of is #14729.
updateResourceCountfor that domain then fails with error 530.ResourceCountCheckTaskruns every 300 seconds by default (resourcecount.check.interval) and has no try/catch around its loop over types and domains. It stops at the first domain that hits this, so the types and domains after it, and the root accounts, are not rechecked. This repeats on every run until a resource with that tag is allocated or released in the domain, which creates the row throughlistRowsToUpdateForDomain().Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
On macOS with JDK 27, without a Maven build. I compiled
ResourceLimitManagerImplandResourceLimitManagerImplTestfrom the 4.20.3.1 sources against the server jar from the 4.20.3.1cloudstack-managementpackage, and ran them with JUnit 4.13.2 and Mockito 4.11.0. To run on JDK 27 I used Byte Buddy 1.18.17 with-Dnet.bytebuddy.experimental=trueinstead of the version Mockito 4.11 depends on.Three new tests in
ResourceLimitManagerImplTest, each with one account that has usage for a storage tag:testRecalculateDomainResourceCountCreatesMissingTaggedRow: no domain row, account usage 10. Without the fix it fails with the error from [4.20.3.1] Tagged domain resource-count recalculation fails when domain counter is missing #14729 (Cannot invoke "com.cloud.configuration.ResourceCountVO.setCount(long)" because "domainRC" is null). With the fix, the method returns 10, a domain row with that type, tag and count 10 is persisted, andupdate()is not called.testRecalculateDomainResourceCountCreatesMissingTaggedRowWithZeroCount: no domain row, usage 0. Without the fix it does not throw, but nothing is persisted. With the fix, a 0 row is persisted.testRecalculateDomainResourceCountUpdatesExistingRow(control): existing domain row with count 5. It passes with and without the fix: the row is updated to 10 and nothing is persisted.Results: before the fix, 56 of 58 tests pass (the first two above fail). After the fix, all 58 pass (55 existing and 3 new).
I then ran the same check on the 4.20 branch head (e6f193d), where the method is unchanged. The head uses
ResourceLimitDao.listByDomainIdsAndTypeAndTag(), which is not in the 4.20.3.1 jar, so I compiledResourceLimitDaoandResourceLimitDaoImplfrom the head in front of the jar.ResourceLimitManagerImplTestfrom the head with the three new tests: 64 of 66 pass without the fix (the same two fail), and 66 pass with it.Not run: the Maven build, checkstyle, the simulator and Marvin tests, and a real management server with MySQL.
How did you try to break this feature and the system with this change?
Going through the cases listed in #14729:
findActiveAccountsForDomain()only leaves out removed accounts, so their usage is counted. This PR does not change that.ResourceCheckTaskglobal lock, but the row can also be created at the same time by a manualupdateResourceCount, or bylistRowsToUpdateForDomain()when a resource with that tag is allocated or released on any management server. That path also uses a plainpersist()without a lock.schema-41910to42000.sqladds the unique indexi_resource_count__type_tag_domainIdon(type, tag, domain_id), so the second insert fails with a duplicate key error and no second row is created. If the recalculation is the one that fails, its transaction is rolled back and that checker run stops at this domain, as it does today with the NPE, but the next run finds the row and updates it. If the allocation fails, it fails the same way it already can when two allocations create the row at once. I did not add locking or a retry, to keep this the same as the existing create-on-miss path. If you want a retry on duplicate key here, I can add one.tagNULL, which the unique index does not cover, but those rows are created together with the domain, so this path should not normally need to create them.