Skip to content

OVN support using the new network extension framework - #14735

Open
kiwiflyer wants to merge 1 commit into
apache:mainfrom
kiwiflyer:ovn-extension
Open

kiwiflyer wants to merge 1 commit into
apache:mainfrom
kiwiflyer:ovn-extension

Conversation

@kiwiflyer

Copy link
Copy Markdown
Contributor

Description

This PR adds an OVN network extension: guest networking for KVM on an
OVN logical fabric, driven entirely by a script through
ovn-nbctl.

It needs no changes to CloudStack and nothing on the hypervisor beyond
ovn-controller. It is a new extension under extensions/OVN/, alongside the existing
Proxmox, HyperV and MaaS samples, plus one line in the Debian package file list.
Nothing outside extensions/ is touched, so no existing behaviour changes: a
deployment that does not create and register this extension is unaffected.

Why the data path needs nothing new. OvsVifDriver already handles
BroadcastDomainType.Lswitch by emitting libvirt's
<virtualport type='openvswitch'><parameters interfaceid='...'/></virtualport>, which
libvirt turns into external_ids:iface-id on the br-int port, and ovn-controller
binds that to the Logical_Switch_Port of the same name. HypervisorGuruBase.toNicTO()
copies NicVO.getUuid() into NicTO — under a comment still referring to Nicira
integration — so the uuid the extension is handed in prepare-nic is the same one the
agent stamps on the VIF. OVN inherited NVP's port-binding convention, so the plumbing
CloudStack shipped for NSX-MH is exactly what OVN needs.

Services implemented, across 25 script commands:

Service Implemented with
Connectivity, Dhcp a logical switch per network, native OVN DHCP
SourceNat, StaticNat NAT rows on a per-network logical router
PortForwarding Load_Balancer VIPs — OVN's NAT has no L4 translation
Firewall ACL rows on the guest switch
VPC + NetworkACL one logical router per VPC, a leg and ACLs per tier
EVPN (optional, off by default) external_ids tags read by a companion ovn-bgp-agent

Also included:

  • test-with-ovn.sh — an integration test that runs the extension against a real OVN
    northbound database. It is not a unit test and does not run in CI; it needs an OVN to
    talk to.
  • ovn-host-setup.sh — the host-side prerequisites, which have no CloudStack API
    behind them: installing OVN, pointing ovn-controller at the southbound database,
    giving a chassis a stable name, the provider bridge and its ovn-bridge-mappings
    entry, and switching the agent to OvsVifDriver. Three roles, idempotent, with
    --dry-run and --verify.

Private gateways, VPC static routes and private gateway ACLs are implemented in the
script but depend on a companion change to NetworkExtensionElement, which throws
UnsupportedOperationException for all four today. On stock CloudStack those commands
are simply never dispatched and the script behaves exactly as it does without them.

There is a PR already out there for the Network Extension additions: #14734

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

n/a

How Has This Been Tested?

Against a real OVN 24.03 — northbound database, ovn-northd, and live ovn-controller
chassis — not against a mock. ./test-with-ovn.sh reproduces all of it.

Full lifecycle, 87 assertions. Every one of the 25 commands, asserting on the
resulting NB objects and on ovn-northd's log, because the NB database will happily
store a configuration northd cannot translate into logical flows — that last check is
the one that proves anything. 0 northd errors, 326 logical flows generated.

Port binding on a real chassis. ovs-vswitchd and ovn-controller were brought up
and registered as a chassis. Two OVS ports carrying nothing but
external_ids:iface-id=<NIC uuid> — precisely what libvirt writes — were bound by
ovn-controller to the switch ports the script created, and the NB ports went up:

nic-bind-0001    Port_Binding.chassis=chassis-a    Logical_Switch_Port up=up
nic-bind-0002    Port_Binding.chassis=chassis-a    Logical_Switch_Port up=up

The controller also claimed the distributed gateway port on the chassis
ovn_gateway_chassis named, without prompting.

Real packets. Frames injected with ovs-ofctl packet-out and the flow counters
watched, so this is the installed datapath rather than a simulation of it:

  • three ICMP echo requests on one VIF's port came out the other's (n_packets=0 -> 3,
    183 bytes, which is 3 x the 61-byte packet)
  • two DHCP DISCOVERs of 292 bytes produced two 328-byte replies back to the client
    port. The other VIF's counter never moved, so those are OVN's own DHCP OFFERs rather
    than the broadcast being flooded — there is no DHCP server in the topology at all

ovn-trace confirms the options come from the script's own configuration:
put_dhcp_opts(offerip = 10.5.1.10, dns_server = {8.8.8.8}, lease_time = 86400, mtu = 1442, netmask = 255.255.255.0, router = 10.5.1.1, server_id = 10.5.1.1), and that
VM-to-internet reaches ct_snat on the gateway port and leaves via the localnet port.

Two chassis. A second chassis on a real Geneve tunnel, BFD up both ways with
forwarding=true. ovn_gateway_chassis produced Gateway_Chassis rows at priority 100
and 90 in the order given, northd built the HA_Chassis_Group, and the highest-priority
chassis claimed the distributed gateway port. On host loss the survivor's BFD reported
Control Detection Time Expired.

Payload replay, 21 assertions. With the companion NetworkExtensionElement change,
the payloads that class actually produces are captured by a unit test and replayed here
against real OVN, substituting only physical-network-extension-details. Unit tests on
one side and script tests on the other are each checked against fixtures written by the
same hand, so both would pass if a key name drifted; this closes that.

Concurrency, 13 assertions. Covered below.

Not tested: : no CloudStack management server driving a real
zone, no libvirt-managed VIF (packets were injected at the OpenFlow layer, so this
exercises the convention the agent relies on rather than the agent itself), no
ovn-bgp-agent so EVPN remains tag-only, shared networks, and IPv6.

How did you try to break this feature and the system with this change?

Testing against a real database found four bugs that a mock would not have. All are
fixed in this PR, with the reproduction kept as a regression test.

1. OVSDB garbage-collects rows in non-root tables. The script created an ACL or
Logical_Router_Static_Route row and attached it to its parent in a second
ovn-nbctl call. A row in a non-root table that nothing references by the end of the
transaction that created it is deleted, so it was gone before the second call ran:

ovn-nbctl: applying "create" command to table ACL without --id option will have no effect
transaction error: ... references nonexistent row ... "error":"referential integrity violation"

That broke every firewall rule, every VPC network ACL and every static route. Both
sites now create and attach in one transaction with --id=@name. DHCP_Options and
Load_Balancer are root tables, which is why the identical pattern was correct for
them and the fault stayed invisible against a stub.

2. --may-exist is not atomic. Only Logical_Switch_Port and
Logical_Router_Port carry a name index in the NB schema. Logical_Switch,
Logical_Router, DHCP_Options, Load_Balancer and ACL do not, so OVSDB will not
refuse a duplicate:

8 x ovn-nbctl --may-exist ls-add dupetest   ->   3 switches named dupetest

ensure_dhcp_options is called from both implement-network and prepare-nic, so a
network coming up while instances are already deploying hit this routinely — up to
8 DHCP_Options rows for one network with instances bound across 2 or 3 of them. That
failure is silent: a later config-dhcp-subnet updates one row and the instances on the
others keep being handed stale DNS, MTU and lease settings indefinitely.

Fixed with a per-command file lock keyed on the network — or on the VPC, since tiers
share a router — plus a second lock for the external switch, which is shared by every
network on a public VLAN. Duplicate DHCP_Options rows are additionally collapsed
deterministically, for the case a file lock cannot reach: a second management server.

3. My first version of that fix was also wrong, and the test caught it.
dhcpv4_options is a weak reference, so destroying a row silently empties the column
on every port pointing at it. Because OVSDB uuids are random, two racers can briefly
disagree about which row wins — 3 of 10 instances ended with no DHCP at all. Ports are
now re-pointed before any row is destroyed, and a sweep repairs anything a concurrent
collapse orphaned.

4. ovs-vsctl add-port succeeds for a device that does not exist. It records the
port and leaves the interface in an error state, so a typo in ovn-host-setup.sh --public-nic reported success and would have surfaced much later as "north-south
traffic does not work". It now checks the interface's error column and fails with the
del-port command to undo it.

What is still not safe:
Two management servers creating the same logical switch at the same instant can still
produce two of them, which splits the network. Nothing ovn-nbctl offers can prevent
that — it needs a compare-and-swap, which means OVSDB wait operations and a native
client. The script now logs it loudly instead of leaving it to be discovered. In
practice CloudStack serialises most per-network work with its own locks, so this is
defence in depth rather than a routine hazard.

Other adversarial cases, all regression tests: missing ovn_nb_db; missing required
payload fields; malformed payload JSON; an unwritable log directory; ovn-nbctl
failing; ovn-nbctl writing noise to stdout and stderr (the management server merges
them and JSON-parses the result, so one stray line breaks implement-network — the
script writes its result to a saved fd 3 that a redirecting call site cannot swallow);
NAT arriving before the router exists; port-forward ranges that are over-cap,
mismatched or use an unsupported protocol; both egress default policies; and unknown
commands, which return success rather than wedging a deployment with an over-broad
offering.

ovn-host-setup.sh was exercised too: dry run changes nothing, a real run converges, a
second run reports everything already set, bridge mappings merge rather than overwrite
another provider, and it refuses to enslave the NIC carrying the host's default route
without --force, because doing so cuts the host off.

Operational note for reviewers who test failover: BFD is answered by
ovs-vswitchd, not ovn-controller. Stopping ovn-controller on the active gateway
chassis fails nothing over — the peer still sees BFD up and correctly leaves the port
where it is. The switch or the host has to go down. This is easy to mistake for the
feature being broken.

@codecov

codecov Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 20.81%. Comparing base (6a0fcd6) to head (03bb3ca).

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #14735      +/-   ##
============================================
- Coverage     20.81%   20.81%   -0.01%     
  Complexity    20923    20923              
============================================
  Files          6053     6053              
  Lines        548451   548451              
  Branches      73279    73279              
============================================
- Hits         114138   114137       -1     
- Misses       420948   420949       +1     
  Partials      13365    13365              
Flag Coverage Δ
uitests 3.86% <ø> (ø)
unittests 22.24% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Guest networking for KVM on an OVN logical fabric, driven entirely by a script
through ovn-nbctl. It needs no changes to CloudStack and nothing on the
hypervisor beyond ovn-controller.

The KVM data path already works. OvsVifDriver handles BroadcastDomainType
Lswitch by emitting libvirt's <parameters interfaceid='<nic uuid>'/>, libvirt
turns that into external_ids:iface-id on the br-int port, and ovn-controller
binds it to the Logical_Switch_Port of that name -- the port-binding convention
OVN inherited from NVP. HypervisorGuruBase.toNicTO() copies the same NIC uuid
the extension is handed in prepare-nic, so the two names agree by construction.

Services, across 25 script commands:

  Connectivity, Dhcp      logical switch per network, native OVN DHCP
  SourceNat, StaticNat    NAT rows on a per-network logical router
  PortForwarding          Load_Balancer VIPs; OVN NAT has no L4 translation
  Firewall                ACL rows on the guest switch
  VPC, NetworkACL         one router per VPC, a leg and ACLs per tier

EVPN is optional and off by default. When enabled it sets the external_ids a
companion ovn-bgp-agent reads; the script does not touch FRR.

Private gateways, VPC static routes and private-gateway ACLs are implemented
but depend on a companion change to NetworkExtensionElement, which throws
UnsupportedOperationException for all four today. Stock CloudStack never
dispatches those commands, so the script is unaffected without it.

Also included:

  test-with-ovn.sh    integration test against a real OVN northbound database,
                      with --payloads and --concurrency modes. Not a unit test
                      and not run in CI: it needs an OVN to talk to.
  ovn-host-setup.sh   the host-side prerequisites that have no CloudStack API
                      behind them -- installing OVN, pointing ovn-controller at
                      the southbound database, a stable chassis name, the
                      provider bridge and its ovn-bridge-mappings entry, and
                      the agent's VIF driver. Three idempotent roles, with
                      --dry-run and --verify.

Verified against a real OVN 24.03 with ovn-northd and a live ovn-controller
chassis: 87 assertions over the full lifecycle with no northd errors; VIF
binding by external_ids:iface-id alone, with the NB ports going up; real
packets forwarded between VIFs and OVN answering DHCP itself; a second chassis
on a Geneve tunnel with BFD up both ways; 21 assertions replaying payloads
captured from NetworkExtensionElement; and 13 under concurrency.

Testing against a real database found four bugs, all fixed here and all kept as
regression tests:

  * ACL and Logical_Router_Static_Route are not root tables, so a row created
    in a transaction of its own is garbage collected before the follow-up
    attach runs. That broke every firewall rule, every VPC network ACL and
    every static route. Both sites now create and attach in one transaction
    with --id. DHCP_Options and Load_Balancer are root tables, which is why the
    same pattern was correct for them and the fault was invisible against a
    stub.

  * ovn-nbctl --may-exist is not atomic. Only Logical_Switch_Port and
    Logical_Router_Port carry a name index, so eight parallel ls-add calls
    produce three switches of one name. ensure_dhcp_options is reached from
    both implement-network and prepare-nic, so a network coming up while
    instances were deploying produced up to eight DHCP_Options rows with
    instances split across them -- silently, since a later config-dhcp-subnet
    updates only one and the rest keep serving stale settings. Fixed with a
    per-command lock keyed on the network or VPC, a second lock for the
    external switch shared across a public VLAN, and deterministic collapse of
    duplicate rows for the case a file lock cannot reach.

  * dhcpv4_options is a weak reference, so collapsing duplicates silently left
    instances with no DHCP when their row was destroyed. Ports are re-pointed
    before any row goes, and a sweep repairs any orphaned by a concurrent
    collapse.

  * ovs-vsctl add-port succeeds for a device that does not exist, leaving the
    interface in an error state, so a typo in ovn-host-setup.sh --public-nic
    looked like success and would have surfaced much later as no north-south
    traffic. It now checks the interface error column and says how to undo it.

What remains unsafe is documented rather than papered over: two management
servers creating the same logical switch at the same instant can still produce
two of them, splitting the network. Preventing that needs a compare-and-swap,
which means OVSDB wait operations and a native client; the script logs it
loudly instead of leaving it to be discovered.

Added to the Debian package file list, which names extension scripts
explicitly. The RPM specs copy extensions/* wholesale and need no change.
@alexandremattioli

Copy link
Copy Markdown
Contributor

I'm labbing it as an extension to the toolkit. Pretty straight forward, I dont expect any issues.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants