Repository navigation
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #14735 +/- ##
============================================
- Coverage 20.81% 20.81% -0.01%
Complexity 20923 20923
============================================
Files 6053 6053
Lines 548451 548451
Branches 73279 73279
============================================
- Hits 114138 114137 -1
- Misses 420948 420949 +1
Partials 13365 13365
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
kiwiflyer
force-pushed
the
ovn-extension
branch
from
November 19, 2026 12:44
bfcb8af to
81dfa7d
Compare
Guest networking for KVM on an OVN logical fabric, driven entirely by a script
through ovn-nbctl. It needs no changes to CloudStack and nothing on the
hypervisor beyond ovn-controller.
The KVM data path already works. OvsVifDriver handles BroadcastDomainType
Lswitch by emitting libvirt's <parameters interfaceid='<nic uuid>'/>, libvirt
turns that into external_ids:iface-id on the br-int port, and ovn-controller
binds it to the Logical_Switch_Port of that name -- the port-binding convention
OVN inherited from NVP. HypervisorGuruBase.toNicTO() copies the same NIC uuid
the extension is handed in prepare-nic, so the two names agree by construction.
Services, across 25 script commands:
Connectivity, Dhcp logical switch per network, native OVN DHCP
SourceNat, StaticNat NAT rows on a per-network logical router
PortForwarding Load_Balancer VIPs; OVN NAT has no L4 translation
Firewall ACL rows on the guest switch
VPC, NetworkACL one router per VPC, a leg and ACLs per tier
EVPN is optional and off by default. When enabled it sets the external_ids a
companion ovn-bgp-agent reads; the script does not touch FRR.
Private gateways, VPC static routes and private-gateway ACLs are implemented
but depend on a companion change to NetworkExtensionElement, which throws
UnsupportedOperationException for all four today. Stock CloudStack never
dispatches those commands, so the script is unaffected without it.
Also included:
test-with-ovn.sh integration test against a real OVN northbound database,
with --payloads and --concurrency modes. Not a unit test
and not run in CI: it needs an OVN to talk to.
ovn-host-setup.sh the host-side prerequisites that have no CloudStack API
behind them -- installing OVN, pointing ovn-controller at
the southbound database, a stable chassis name, the
provider bridge and its ovn-bridge-mappings entry, and
the agent's VIF driver. Three idempotent roles, with
--dry-run and --verify.
Verified against a real OVN 24.03 with ovn-northd and a live ovn-controller
chassis: 87 assertions over the full lifecycle with no northd errors; VIF
binding by external_ids:iface-id alone, with the NB ports going up; real
packets forwarded between VIFs and OVN answering DHCP itself; a second chassis
on a Geneve tunnel with BFD up both ways; 21 assertions replaying payloads
captured from NetworkExtensionElement; and 13 under concurrency.
Testing against a real database found four bugs, all fixed here and all kept as
regression tests:
* ACL and Logical_Router_Static_Route are not root tables, so a row created
in a transaction of its own is garbage collected before the follow-up
attach runs. That broke every firewall rule, every VPC network ACL and
every static route. Both sites now create and attach in one transaction
with --id. DHCP_Options and Load_Balancer are root tables, which is why the
same pattern was correct for them and the fault was invisible against a
stub.
* ovn-nbctl --may-exist is not atomic. Only Logical_Switch_Port and
Logical_Router_Port carry a name index, so eight parallel ls-add calls
produce three switches of one name. ensure_dhcp_options is reached from
both implement-network and prepare-nic, so a network coming up while
instances were deploying produced up to eight DHCP_Options rows with
instances split across them -- silently, since a later config-dhcp-subnet
updates only one and the rest keep serving stale settings. Fixed with a
per-command lock keyed on the network or VPC, a second lock for the
external switch shared across a public VLAN, and deterministic collapse of
duplicate rows for the case a file lock cannot reach.
* dhcpv4_options is a weak reference, so collapsing duplicates silently left
instances with no DHCP when their row was destroyed. Ports are re-pointed
before any row goes, and a sweep repairs any orphaned by a concurrent
collapse.
* ovs-vsctl add-port succeeds for a device that does not exist, leaving the
interface in an error state, so a typo in ovn-host-setup.sh --public-nic
looked like success and would have surfaced much later as no north-south
traffic. It now checks the interface error column and says how to undo it.
What remains unsafe is documented rather than papered over: two management
servers creating the same logical switch at the same instant can still produce
two of them, splitting the network. Preventing that needs a compare-and-swap,
which means OVSDB wait operations and a native client; the script logs it
loudly instead of leaving it to be discovered.
Added to the Debian package file list, which names extension scripts
explicitly. The RPM specs copy extensions/* wholesale and need no change.
kiwiflyer
force-pushed
the
ovn-extension
branch
from
November 19, 2026 14:45
81dfa7d to
03bb3ca
Compare
Contributor
|
I'm labbing it as an extension to the toolkit. Pretty straight forward, I dont expect any issues. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds an OVN network extension: guest networking for KVM on an
OVN logical fabric, driven entirely by a script through
ovn-nbctl.It needs no changes to CloudStack and nothing on the hypervisor beyond
ovn-controller. It is a new extension underextensions/OVN/, alongside the existingProxmox,HyperVandMaaSsamples, plus one line in the Debian package file list.Nothing outside
extensions/is touched, so no existing behaviour changes: adeployment that does not create and register this extension is unaffected.
Why the data path needs nothing new.
OvsVifDriveralready handlesBroadcastDomainType.Lswitchby emitting libvirt's<virtualport type='openvswitch'><parameters interfaceid='...'/></virtualport>, whichlibvirt turns into
external_ids:iface-idon thebr-intport, andovn-controllerbinds that to the
Logical_Switch_Portof the same name.HypervisorGuruBase.toNicTO()copies
NicVO.getUuid()intoNicTO— under a comment still referring to Niciraintegration — so the uuid the extension is handed in
prepare-nicis the same one theagent stamps on the VIF. OVN inherited NVP's port-binding convention, so the plumbing
CloudStack shipped for NSX-MH is exactly what OVN needs.
Services implemented, across 25 script commands:
Connectivity,DhcpSourceNat,StaticNatNATrows on a per-network logical routerPortForwardingLoad_BalancerVIPs — OVN's NAT has no L4 translationFirewallACLrows on the guest switchNetworkACLexternal_idstags read by a companionovn-bgp-agentAlso included:
test-with-ovn.sh— an integration test that runs the extension against a real OVNnorthbound database. It is not a unit test and does not run in CI; it needs an OVN to
talk to.
ovn-host-setup.sh— the host-side prerequisites, which have no CloudStack APIbehind them: installing OVN, pointing
ovn-controllerat the southbound database,giving a chassis a stable name, the provider bridge and its
ovn-bridge-mappingsentry, and switching the agent to
OvsVifDriver. Three roles, idempotent, with--dry-runand--verify.Private gateways, VPC static routes and private gateway ACLs are implemented in the
script but depend on a companion change to
NetworkExtensionElement, which throwsUnsupportedOperationExceptionfor all four today. On stock CloudStack those commandsare simply never dispatched and the script behaves exactly as it does without them.
There is a PR already out there for the Network Extension additions: #14734
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
n/a
How Has This Been Tested?
Against a real OVN 24.03 — northbound database,
ovn-northd, and liveovn-controllerchassis — not against a mock.
./test-with-ovn.shreproduces all of it.Full lifecycle, 87 assertions. Every one of the 25 commands, asserting on the
resulting NB objects and on
ovn-northd's log, because the NB database will happilystore a configuration northd cannot translate into logical flows — that last check is
the one that proves anything. 0 northd errors, 326 logical flows generated.
Port binding on a real chassis.
ovs-vswitchdandovn-controllerwere brought upand registered as a chassis. Two OVS ports carrying nothing but
external_ids:iface-id=<NIC uuid>— precisely what libvirt writes — were bound byovn-controllerto the switch ports the script created, and the NB ports wentup:The controller also claimed the distributed gateway port on the chassis
ovn_gateway_chassisnamed, without prompting.Real packets. Frames injected with
ovs-ofctl packet-outand the flow counterswatched, so this is the installed datapath rather than a simulation of it:
n_packets=0 -> 3,183 bytes, which is 3 x the 61-byte packet)
port. The other VIF's counter never moved, so those are OVN's own DHCP OFFERs rather
than the broadcast being flooded — there is no DHCP server in the topology at all
ovn-traceconfirms the options come from the script's own configuration:put_dhcp_opts(offerip = 10.5.1.10, dns_server = {8.8.8.8}, lease_time = 86400, mtu = 1442, netmask = 255.255.255.0, router = 10.5.1.1, server_id = 10.5.1.1), and thatVM-to-internet reaches
ct_snaton the gateway port and leaves via the localnet port.Two chassis. A second chassis on a real Geneve tunnel, BFD up both ways with
forwarding=true.ovn_gateway_chassisproducedGateway_Chassisrows at priority 100and 90 in the order given, northd built the
HA_Chassis_Group, and the highest-prioritychassis claimed the distributed gateway port. On host loss the survivor's BFD reported
Control Detection Time Expired.Payload replay, 21 assertions. With the companion
NetworkExtensionElementchange,the payloads that class actually produces are captured by a unit test and replayed here
against real OVN, substituting only
physical-network-extension-details. Unit tests onone side and script tests on the other are each checked against fixtures written by the
same hand, so both would pass if a key name drifted; this closes that.
Concurrency, 13 assertions. Covered below.
Not tested: : no CloudStack management server driving a real
zone, no libvirt-managed VIF (packets were injected at the OpenFlow layer, so this
exercises the convention the agent relies on rather than the agent itself), no
ovn-bgp-agentso EVPN remains tag-only, shared networks, and IPv6.How did you try to break this feature and the system with this change?
Testing against a real database found four bugs that a mock would not have. All are
fixed in this PR, with the reproduction kept as a regression test.
1. OVSDB garbage-collects rows in non-root tables. The script created an
ACLorLogical_Router_Static_Routerow and attached it to its parent in a secondovn-nbctlcall. A row in a non-root table that nothing references by the end of thetransaction that created it is deleted, so it was gone before the second call ran:
That broke every firewall rule, every VPC network ACL and every static route. Both
sites now create and attach in one transaction with
--id=@name.DHCP_OptionsandLoad_Balancerare root tables, which is why the identical pattern was correct forthem and the fault stayed invisible against a stub.
2.
--may-existis not atomic. OnlyLogical_Switch_PortandLogical_Router_Portcarry a name index in the NB schema.Logical_Switch,Logical_Router,DHCP_Options,Load_BalancerandACLdo not, so OVSDB will notrefuse a duplicate:
ensure_dhcp_optionsis called from bothimplement-networkandprepare-nic, so anetwork coming up while instances are already deploying hit this routinely — up to
8
DHCP_Optionsrows for one network with instances bound across 2 or 3 of them. Thatfailure is silent: a later
config-dhcp-subnetupdates one row and the instances on theothers keep being handed stale DNS, MTU and lease settings indefinitely.
Fixed with a per-command file lock keyed on the network — or on the VPC, since tiers
share a router — plus a second lock for the external switch, which is shared by every
network on a public VLAN. Duplicate
DHCP_Optionsrows are additionally collapseddeterministically, for the case a file lock cannot reach: a second management server.
3. My first version of that fix was also wrong, and the test caught it.
dhcpv4_optionsis a weak reference, so destroying a row silently empties the columnon every port pointing at it. Because OVSDB uuids are random, two racers can briefly
disagree about which row wins — 3 of 10 instances ended with no DHCP at all. Ports are
now re-pointed before any row is destroyed, and a sweep repairs anything a concurrent
collapse orphaned.
4.
ovs-vsctl add-portsucceeds for a device that does not exist. It records theport and leaves the interface in an error state, so a typo in
ovn-host-setup.sh --public-nicreported success and would have surfaced much later as "north-southtraffic does not work". It now checks the interface's
errorcolumn and fails with thedel-portcommand to undo it.What is still not safe:
Two management servers creating the same logical switch at the same instant can still
produce two of them, which splits the network. Nothing
ovn-nbctloffers can preventthat — it needs a compare-and-swap, which means OVSDB
waitoperations and a nativeclient. The script now logs it loudly instead of leaving it to be discovered. In
practice CloudStack serialises most per-network work with its own locks, so this is
defence in depth rather than a routine hazard.
Other adversarial cases, all regression tests: missing
ovn_nb_db; missing requiredpayload fields; malformed payload JSON; an unwritable log directory;
ovn-nbctlfailing;
ovn-nbctlwriting noise to stdout and stderr (the management server mergesthem and JSON-parses the result, so one stray line breaks
implement-network— thescript writes its result to a saved fd 3 that a redirecting call site cannot swallow);
NAT arriving before the router exists; port-forward ranges that are over-cap,
mismatched or use an unsupported protocol; both egress default policies; and unknown
commands, which return success rather than wedging a deployment with an over-broad
offering.
ovn-host-setup.shwas exercised too: dry run changes nothing, a real run converges, asecond run reports everything already set, bridge mappings merge rather than overwrite
another provider, and it refuses to enslave the NIC carrying the host's default route
without
--force, because doing so cuts the host off.Operational note for reviewers who test failover: BFD is answered by
ovs-vswitchd, notovn-controller. Stoppingovn-controlleron the active gatewaychassis fails nothing over — the peer still sees BFD up and correctly leaves the port
where it is. The switch or the host has to go down. This is easy to mistake for the
feature being broken.