Skip to content

Order clusters by size when picking hosts for agent load balancing - #14737

Open
MuhammetSoil wants to merge 1 commit into
apache:4.22from
MuhammetSoil:fix-agentlb-cluster-sort
Open

MuhammetSoil wants to merge 1 commit into
apache:4.22from
MuhammetSoil:fix-agentlb-cluster-sort

Conversation

@MuhammetSoil

Copy link
Copy Markdown

Description

This PR fixes the cluster ordering used by agent load balancing (#14625).

ClusterBasedAgentLoadBalancerPlanner.getHostsToRebalance() groups the Up routing hosts of a
management server by cluster and then walks the clusters in the order returned by
sortByClusterSize(), taking whole clusters until it has given away enough hosts. That walk
assumes the clusters come largest first.

The comparator in sortByClusterSize() does not implement that order:

  • it returns 1 when the first cluster is smaller, but 0 (equal) when it is larger, so it
    never says that one cluster comes before another;
  • it is not symmetric (compare(a, b) and compare(b, a) can both be 0 or one of them 1),
    which breaks the Comparator contract;
  • it throws a NullPointerException if only the second cluster has no host list.

As a result, Collections.sort() mostly keeps the HashMap iteration order (cluster id order),
and which clusters a management server hands over during rebalancing depends on that order rather
than on cluster size. For example, clusters with 1, 3 and 2 hosts (ids 1, 2, 3) come back as
[1, 2, 3] instead of [2, 3, 1].

The comparator is replaced with one that orders by host count, largest first, with clusters
without a host list last. sort is stable, so clusters of the same size keep their previous
relative order. Nothing else in the planner changes.

Unit tests are added for the ordering, the null handling and that all entries are kept. The first
test fails without the fix.

The code is the same on 4.20, 4.22 and main; this targets 4.22 so it can be merged forward.

Fixes: #14625

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

  • Added ClusterBasedAgentLoadBalancerPlannerTest; ran it with
    mvn -pl engine/orchestration test -Dtest=ClusterBasedAgentLoadBalancerPlannerTest on JDK 11.
  • Checked that sortByClusterSizePutsLargestClusterFirst fails against the previous comparator
    (it returns [1, 2, 3]) and passes with this change.

How did you try to break this feature and the system with this change?

Tested with ten clusters in random size order, and a cluster
mapped to a null host list (which made the previous comparator throw when it was the second
argument).

The comparator used by ClusterBasedAgentLoadBalancerPlanner.sortByClusterSize
never returned a negative value and returned 0 when the first cluster was
larger, so it broke the Comparator contract and left the clusters mostly in
HashMap order. It also threw a NullPointerException when only the second
cluster had no host list.

Order the clusters by host count, largest first, with clusters without a host
list last, as getHostsToRebalance expects. Add unit tests.

Fixes apache#14625
@boring-cyborg

boring-cyborg Bot commented Nov 19, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant