diff --git a/debian/cloudstack-management.install b/debian/cloudstack-management.install index befc7049c30e..ce6992358af6 100644 --- a/debian/cloudstack-management.install +++ b/debian/cloudstack-management.install @@ -25,6 +25,8 @@ /etc/cloudstack/extensions/Proxmox/proxmox.sh /etc/cloudstack/extensions/HyperV/hyperv.py /etc/cloudstack/extensions/MaaS/maas.py +/etc/cloudstack/extensions/OVN/ovn.sh +/etc/cloudstack/extensions/OVN/ovn-host-setup.sh /etc/default/cloudstack-management /etc/security/limits.d/cloudstack-limits.conf /etc/sudoers.d/cloudstack diff --git a/extensions/OVN/README.md b/extensions/OVN/README.md new file mode 100644 index 000000000000..9356b9eab7ff --- /dev/null +++ b/extensions/OVN/README.md @@ -0,0 +1,777 @@ + + +# OVN NetworkOrchestrator extension for Apache CloudStack + +Puts guest networks on an [OVN](https://www.ovn.org/) logical fabric, driven entirely +by this script through `ovn-nbctl`. It needs **no changes to CloudStack** and no agent +on the hypervisor beyond `ovn-controller`. + +| Service | Implemented with | +| --- | --- | +| `Connectivity`, `Dhcp` | a logical switch per network, native OVN DHCP | +| `SourceNat`, `StaticNat` | `NAT` rows on a per-network logical router | +| `PortForwarding` | `Load_Balancer` VIPs — OVN's NAT has no L4 translation | +| `Firewall` | `ACL` rows on the guest switch | +| VPC + `NetworkACL` | one logical router per VPC, a leg and ACLs per tier | +| EVPN (optional, off by default) | `external_ids` tags read by a companion `ovn-bgp-agent` | + +Private gateways, VPC static routes and private-gateway ACLs are implemented as well, +but depend on a CloudStack change that is not in main — see §7. + +Requires the network-extension framework +([PR #13032](https://github.com/apache/cloudstack/pull/13032), merged April 2026 — +main / 24.0.0-SNAPSHOT at the time of writing). + +> **Status:** exercised against a real OVN 24.03 — northbound database, `ovn-northd` +> and a live `ovn-controller` chassis. All 25 commands run clean (`./test-with-ovn.sh`, +> 87 assertions), and a VIF carrying `external_ids:iface-id` binds to the logical +> switch port exactly as §1 claims, with real packets forwarded between VIFs and OVN +> answering DHCP itself. It has **not** been run against a real CloudStack management +> server. §9 is precise about where that boundary falls. + +--- + +## 1. Why this works without patching CloudStack + +```text + implement-network ──► returns broadcast_domain_type = "Lswitch" + broadcast_uri = "lswitch:cs-net-" + │ + ▼ + CloudStack stamps that onto the network and every NIC + │ + prepare-nic ──► creates OVN Logical_Switch_Port named + │ + ▼ + OvsVifDriver sees BroadcastDomainType.Lswitch and emits + + ← same uuid + + │ + libvirt ──► external_ids:iface-id on the br-int port + │ + ovn-controller ──► binds it to the LSP of that name +``` + +The two uuids are the same one: `HypervisorGuruBase.toNicTO()` copies +`NicVO.getUuid()` into `NicTO`, under a comment that still says *"the UUID we need for +Nicira integration"*. OVN inherited NVP's port-binding convention, so the plumbing +CloudStack shipped for NSX-MH a decade ago is exactly what OVN needs. + +## 2. Topology + +One logical router per isolated network; one external logical switch shared by every +router on the same public VLAN. + +```text + internet + │ + ┌──────────────┴──────────────┐ + │ cs-ext- │ shared external logical switch + │ cs-ext-ln- │ localnet, via ovn-bridge-mappings + └──────────────┬──────────────┘ + │ cs-ext-lsp- (LSP type=router) + ┌──────────────┴──────────────┐ + │ cs-rtr- │ logical router + │ cs-ext-lrp- │ distributed gateway port, + │ │ holds the source-NAT IP + └──────────────┬──────────────┘ + │ cs-lrp- (guest gateway IP) + │ cs-lsp-rtr- (LSP type=router) + ┌──────────────┴──────────────┐ + │ cs-net- │ guest logical switch + │ … │ one LSP per VM NIC + └─────────────────────────────┘ +``` + +| CloudStack concept | OVN object | +| --- | --- | +| Guest network | `Logical_Switch cs-net-` | +| VM NIC | `Logical_Switch_Port ` | +| DHCP | `DHCP_Options` tagged `cs-network-id`, attached per LSP | +| Network gateway | `Logical_Router cs-rtr-` + `cs-lrp-` | +| Source NAT | `NAT type=snat`, external IP on the gateway port | +| Static NAT | `NAT type=dnat_and_snat` | +| Port forwarding | `Load_Balancer`, one VIP entry per forwarded port | +| Egress/ingress firewall | `ACL` rows on the guest switch | +| VPC | one `Logical_Router cs-vpc-rtr-` shared by every tier | +| VPC tier | its own `cs-net-` switch and its own leg on that router | +| VPC network ACL | `ACL` rows, priority `30000 - rule number` | + +--- + +## 3. Prerequisites + +`ovn-host-setup.sh` does all of this, idempotently, and is the quickest way to get it +right. Everything it does is spelled out below so you can do it yourself, or express it +in whatever configuration management you already run — which, for anything beyond a +lab, you should. + +```bash +# on each management server +./ovn-host-setup.sh --role management --nb-db tcp:10.0.0.1:6641 + +# on each KVM host +./ovn-host-setup.sh --role host --sb-db tcp:10.0.0.1:6642 --encap-ip 10.1.0.11 + +# on the hosts that may carry north-south traffic +./ovn-host-setup.sh --role gateway --sb-db tcp:10.0.0.1:6642 --encap-ip 10.1.0.11 \ + --public-nic eth1 --physnet physnet1 + +# or drive the whole estate over ssh, one host per line ("name" or "name encap-ip") +./ovn-host-setup.sh --hosts kvm-hosts.txt --role host --sb-db tcp:10.0.0.1:6642 + +# check an existing host without changing anything +./ovn-host-setup.sh --verify +``` + +`--dry-run` prints what would change and touches nothing. Re-running is safe: it only +acts on what differs and says what the previous value was. It will not enslave the NIC +carrying the host's default route without `--force`, will not restart `cloudstack-agent` +without `--restart-agent`, merges into any existing `ovn-bridge-mappings` instead of +overwriting another provider, and backs up `agent.properties` before editing it. + +It deliberately stops short of the CloudStack API: creating the extension, registering +it on the physical network and setting the traffic label are §4, not host setup. + +**OVN control plane** — an OVN cluster reachable from every management server. +Three `ovsdb-server` nodes in RAFT for NB and SB is the normal shape. The NB DB must +accept connections, e.g. `ovn-nbctl set-connection ptcp:6641`. Use SSL outside a lab. + +**Management servers** — `ovn-nbctl` (package `ovn-common` / `ovn-central`) and `jq` +on **every** management server. The script runs there, not on the hosts. + +**KVM hosts** — `openvswitch-switch` + `ovn-host`: + +```bash +ovs-vsctl set open_vswitch . \ + external_ids:ovn-remote="tcp:10.0.0.1:6642,tcp:10.0.0.2:6642,tcp:10.0.0.3:6642" \ + external_ids:ovn-encap-type=geneve \ + external_ids:ovn-encap-ip= \ + external_ids:system-id= +systemctl enable --now ovn-controller +``` + +**Gateway chassis** — the hosts that may carry north-south traffic need a bridge +mapping from the provider network name to the OVS bridge holding the public NIC: + +```bash +ovs-vsctl add-br br-ex +ovs-vsctl add-port br-ex eth1 # the public NIC +ovs-vsctl set open_vswitch . external_ids:ovn-bridge-mappings=physnet1:br-ex +``` + +`physnet1` must match `ovn_public_physnet` (§4.4). Note the caveat in §7 — the gateway +port lives on **one** chassis at a time. + +**CloudStack agent** — `agent.properties` on each KVM host must select the OVS VIF +driver, or `plug()` will never take the `Lswitch` branch: + +```properties +libvirt.vif.driver=com.cloud.hypervisor.kvm.resource.OvsVifDriver +network.bridge.type=openvswitch +``` + +**Physical network traffic label** — set the guest traffic label to `br-int`. +`OvsVifDriver.getPifs()` registers every bridge from `ovs-vsctl list-br` under its own +name, so `br-int` resolves and VIFs land on the integration bridge. + +--- + +## 4. Install and register + +### 4.1 Install + +On every management server: + +```bash +install -d -m 0755 /usr/share/cloudstack-management/extensions/ovn +install -m 0755 ovn.sh /usr/share/cloudstack-management/extensions/ovn/ovn.sh +install -d -o cloud -g cloud -m 0755 /var/log/cloudstack/management/extensions +``` + +Directory and script name are derived from the extension name: +`Extension.getDirectoryName("OVN")` lowercases it, so the relative path CloudStack +resolves is `ovn/ovn.sh` — note the lowercase directory, even though this source tree +ships the file under `extensions/OVN/` to match its siblings. CloudStack checksums the +file and marks the extension **Not Ready** if it differs between management servers, so +deploy it with config management. + +### 4.2 Create the extension + +```bash +cmk create extension name=OVN type=NetworkOrchestrator \ + description='OVN logical networking' \ + details[0].key=network.services \ + details[0].value=Connectivity,Dhcp,SourceNat,StaticNat,PortForwarding,Firewall \ + details[1].key=network.allocate.extension.ip \ + details[1].value=false +``` + +`network.services` decides which services the synthesised provider advertises. Keep it +to what the script implements — anything else routes into `cmd_unsupported`, which +logs and returns success rather than wedging the deployment. + +`network.allocate.extension.ip=false` stops CloudStack reserving a placeholder IP from +the subnet pool. This is the exact case the framework documents for OVN: *"the +extension handles its own data-plane addressing … (e.g. OVN, which programs port +bindings in-fabric)"*. + +### 4.3 Register on the physical network + +```bash +cmk register extension id= \ + resourcetype=PhysicalNetwork resourceid= \ + details[0].key=network.isolation.method details[0].value=NetworkExtension \ + details[1].key=ovn_nb_db details[1].value='tcp:10.0.0.1:6641,tcp:10.0.0.2:6641,tcp:10.0.0.3:6641' \ + details[2].key=ovn_public_physnet details[2].value='physnet1' \ + details[3].key=ovn_gateway_chassis details[3].value='kvm01,kvm02' \ + details[4].key=ovn_dns_servers details[4].value='8.8.8.8,8.8.4.4' \ + details[5].key=ovn_mtu details[5].value='1442' +``` + +`network.isolation.method=NetworkExtension` is **required**. It is what makes +`NetworkExtensionGuestNetworkGuru.canHandle()` claim the network; without it the guru +never engages and the broadcast URI the script returns is silently ignored. + +`ovn_gateway_chassis` is a comma-separated list of chassis `system-id`s in priority +order. Without it the distributed gateway port is never scheduled and north-south +traffic will not work — the script logs a warning but does not fail. + +Then enable the provider: + +```bash +cmk list networkserviceproviders physicalnetworkid= name=OVN +cmk update networkserviceprovider id= state=Enabled +``` + +### 4.4 `physical-network-extension-details` reference + +| Key | Required | Default | Meaning | +| --- | --- | --- | --- | +| `ovn_nb_db` | **yes** | — | NB DB target(s), `ovn-nbctl --db` syntax. Comma-separate a RAFT cluster. | +| `ovn_public_physnet` | for L3 | `physnet1` | Provider name in each host's `ovn-bridge-mappings` | +| `ovn_gateway_chassis` | for L3 | — | Chassis `system-id`s, highest priority first | +| `ovn_nb_private_key` | no | — | SSL client key (`--private-key`) | +| `ovn_nb_certificate` | no | — | SSL client cert (`--certificate`) | +| `ovn_nb_ca_cert` | no | — | SSL CA cert (`--ca-cert`) | +| `ovn_dns_servers` | no | the gateway | DHCP `dns_server` option | +| `ovn_mtu` | no | unset | DHCP `mtu` option | +| `ovn_dhcp_lease_time` | no | `86400` | DHCP `lease_time` option | +| `ovn_port_security` | no | `true` | Set `port_security` on each LSP | +| `ovn_pf_max_ports` | no | `128` | Max VIPs one port-forward rule may expand into | +| `ovn_private_physnet` | no | `ovn_public_physnet` | Provider name for VPC private gateways | +| `ovn_evpn_enabled` | no | `false` | Tag switches for `ovn-bgp-agent` | +| `ovn_evpn_type` | no | `l3` | `l3` (route the subnet into a VRF) or `l2` (stretch the segment) | +| `ovn_evpn_vni_base` | no | `0` | VNI = base + VPC id (L3) or + network id (L2) | +| `ovn_evpn_key_prefix` | no | `neutron_bgpvpn` | `external_ids` key prefix the agent reads | + +### 4.5 Offering and network + +```bash +cmk create networkoffering name=OVN-Isolated displaytext='OVN isolated network' \ + guestiptype=Isolated traffictype=Guest \ + supportedservices=Connectivity,Dhcp,SourceNat,StaticNat,PortForwarding,Firewall \ + serviceproviderlist[0].service=Connectivity serviceproviderlist[0].provider=OVN \ + serviceproviderlist[1].service=Dhcp serviceproviderlist[1].provider=OVN \ + serviceproviderlist[2].service=SourceNat serviceproviderlist[2].provider=OVN \ + serviceproviderlist[3].service=StaticNat serviceproviderlist[3].provider=OVN \ + serviceproviderlist[4].service=PortForwarding serviceproviderlist[4].provider=OVN \ + serviceproviderlist[5].service=Firewall serviceproviderlist[5].provider=OVN \ + specifyvlan=false egressdefaultpolicy=true state=Enabled + +cmk create network name=ovn-test displaytext='OVN test network' \ + networkofferingid= zoneid= \ + gateway=10.1.1.1 netmask=255.255.255.0 startip=10.1.1.10 endip=10.1.1.200 +``` + +`egressdefaultpolicy` maps straight onto the ACL base policy: `true` → permissive base +with explicit egress rules as denies; `false` → base drop with explicit rules as allows. + +--- + +## 5. Verify + +```bash +NB='--db=tcp:10.0.0.1:6641' + +ovn-nbctl $NB show # whole topology +ovn-nbctl $NB lsp-list cs-net- # a port per NIC +ovn-nbctl $NB lr-nat-list cs-rtr- # snat + dnat_and_snat +ovn-nbctl $NB lr-lb-list cs-rtr- # port forwards +ovn-nbctl $NB acl-list cs-net- # firewall +ovn-nbctl $NB lrp-get-gateway-chassis cs-ext-lrp- + +# On the KVM host +ovs-vsctl --columns=name,external_ids list interface | grep iface-id +ovn-sbctl --db=tcp:10.0.0.1:6642 show # port bindings per chassis +``` + +The decisive check for L2 is that `external_ids:iface-id` on the host equals a +`Logical_Switch_Port.name` in the NB DB, with `ovn-sbctl show` listing that port under +the chassis the VM landed on. For L3, check that `lrp-get-gateway-chassis` returns a +chassis and that a VM can reach the internet through the source-NAT IP. + +Script activity, including every `ovn-nbctl` argv and any stderr it produced, goes to +`/var/log/cloudstack/management/extensions/ovn.log`. + +--- + +## 6. Design notes + +**Naming is derived from `network_id`.** The payload carries `network_id` (the internal +long), **not** `network_uuid`, so that is the only stable network identity available. +Logical switch ports are named with `nic_uuid` — that name is load-bearing, not +cosmetic: it must equal `external_ids:iface-id`. + +**Port forwarding uses load balancers, not NAT.** OVN's NAT table has no L4 port +translation. The standard answer — also what OpenStack does for floating-IP port +forwarding — is a `Load_Balancer` VIP with a single backend. OVN has no port-range +DNAT either, so a range expands into one VIP per port, capped by `ovn_pf_max_ports` +(default 128). Beyond the cap the script fails with an explicit message rather than +quietly creating thousands of entries. + +**Ingress firewall rules are resolved through NAT.** CloudStack gives an ingress rule a +`publicIp`, but ACLs on the guest switch see post-DNAT traffic. The script reads the +router's `dnat_and_snat` entries back to find the private IP behind that public IP and +targets the rule at that VM. If no mapping exists, it widens the rule to the guest CIDR +and says so in the log. + +**Firewall state is rebuilt, not diffed.** CloudStack sends the complete active rule +set on every `apply-fw-rules` call, so the script deletes every ACL it owns (tagged +`external_ids:cs-network-id`) and recreates them. DHCP is allowed at a priority above +everything so a restrictive egress policy can never break addressing. + +**`shutdown-network` is deliberately a no-op.** CloudStack shuts a network down while +its NICs may still exist, and there is no guarantee `prepare-nic` runs again for +already-running VMs on restart. Deleting the logical switch there would blackhole them. +An idle logical switch costs nothing in OVN, so cleanup happens only in +`destroy-network`. + +**NAT rules fail loudly if L3 is not up.** CloudStack applies the source-NAT IP during +`implement()`, so a static NAT arriving before the router exists is a real ordering +problem. Half-configuring it would be worse than stopping. + +**Everything is idempotent.** `--may-exist` / `--if-exists` throughout, multi-command +`ovn-nbctl` transactions so related changes land atomically, and `prepare-nic` +recreates the logical switch if it is missing. + +**stdout discipline.** The management server merges the script's stderr into stdout +(`ProcessBuilder.redirectErrorStream(true)`) and JSON-parses the whole thing, so a +single stray log line breaks `implement-network`. The script writes its result JSON to +a saved fd 3, never fd 1 — a call site that redirects stdout cannot swallow it — and +captures `ovn-nbctl`'s stderr into the log file. + +--- + +## 7. Known gaps and caveats + +**Gateway concentration.** OVN NAT and load balancing require exactly one distributed +gateway port per router. North-south traffic for a network is therefore pinned to one +chassis; east-west routing stays distributed. `ovn_gateway_chassis` sets the failover +order, but a chassis loss affects every network homed there. Capacity-plan and +load-test this before production — it is the single biggest operational difference from +a per-network virtual router. + +When you do test failover, note that **BFD is answered by `ovs-vswitchd`, not by +`ovn-controller`**. Stopping `ovn-controller` on the active gateway chassis does not +fail anything over: the peer still sees BFD up and correctly leaves the port where it +is. Take down the switch, the host, or the tunnel path instead. This cost real time to +work out and is easy to mistake for the feature being broken. + +**No VPN.** OVN has no IPsec site-to-site or remote-access VPN. Do not put `Vpn` in the +offering. + +**No user data or metadata.** OVN has no metadata service. Use **ConfigDrive** as the +`UserData` provider; it is a first-class CloudStack provider and keeps the VR out of +the path entirely. + +**Load balancing is not wired up.** OVN's load balancer is L4 only, with IPv4-only +health checks and none of the stickiness or SSL-offload capabilities a CloudStack LB +offering advertises. `Lb` is deliberately absent from `network.services`. + +**IPv4 only.** `network_ip6_gateway` / `network_ip6_cidr` arrive in the payload but no +DHCPv6 options, RA config or IPv6 ACLs are created. + +**Shared networks are untested.** The LS/LSP path should work, but +`NetworkExtensionGuestNetworkGuru` delegates shared-network allocation to +`DirectNetworkGuru` and that combination has not been exercised. + +**Security groups are out of scope.** CloudStack implements them with host-side +iptables/ebtables on KVM; OVN would use `Port_Group` + `ACL`. Running both is at best +redundant. Use isolated networks only. + +**No reconciliation.** If `release-nic` is missed on some VM-destroy path, the LSP +leaks. A sweeper comparing NB ports tagged `external_ids:cs-network-id` against +CloudStack's NIC table is still to be written. + +**Private gateways, VPC static routes and private-gateway ACLs need a CloudStack +patch.** `NetworkExtensionElement.createPrivateGateway()`, +`deletePrivateGateway()`, `applyStaticRoutes()` and `applyACLItemsToPrivateGw()` +throw `UnsupportedOperationException` in stock CloudStack, so no network +extension can offer them, whatever its backend. + +The script implements the four verbs anyway — `create-private-gateway`, +`delete-private-gateway`, `apply-static-routes`, `apply-private-gw-acl` — against a +companion change that wires those methods up the same way as their neighbours, adding +the four script commands and a `NetworkExtensionElementTest` covering them. **That +change is not in main and has not been submitted upstream.** On stock CloudStack these +four verbs are simply never dispatched, so the script behaves exactly as it does +without them — nothing here depends on the patch being present. + +Private-gateway ACLs are scoped to the gateway's own subnet, derived from its +address and netmask. OVN masks the host bits, so a match written as +`ip4.src == 172.16.0.5/24` means the enclosing `/24`. + +--- + +## 8. EVPN + +Optional, off by default. OVN keeps using Geneve between chassis; EVPN is about +exposing those networks into a physical BGP/EVPN fabric. + +### How it works + +The translation is done out of band by **`ovn-bgp-agent`** (`nb_ovn_bgp_driver`, +`exposing_method = vrf`) running on each host. It watches the OVN NB DB and drives +FRR plus Linux VRF and VXLAN devices. The CMS's entire job is to set two +`external_ids` on a logical switch, so that is all this script does: + +```text +external_ids:neutron_bgpvpn\:type = l3 | l2 +external_ids:neutron_bgpvpn\:vni = +``` + +Those key names are fixed by the agent. A +[Neutron RFE](https://bugs.launchpad.net/neutron/+bug/2067401) notes that writing +them with `ovn-nbctl` is currently the only supported way to set them — which is +exactly what an extension is able to do. `ovn_evpn_key_prefix` exists so a rename +upstream does not require editing the script. + +### L3 vs L2 + +| | `ovn_evpn_type=l3` | `ovn_evpn_type=l2` | +| --- | --- | --- | +| Tagged object | the provider switch `cs-ext--vni` | the guest switch `cs-net-` | +| VNI scope | one per VPC (one per network when standalone) | one per logical switch | +| Effect | the tenant's subnets are routed into a VRF in the fabric | the L2 segment is stretched into the fabric | + +L3 is the default and is what the OpenStack example documents. Because +`ovn-bgp-agent` maps **one provider network to one VNI**, enabling L3 EVPN changes +the external switch from being shared per public VLAN to being **scoped per VRF** — +`cs-ext-101` becomes `cs-ext-101-vni100007`. That is what gives each VPC its own +routing domain in the fabric rather than dumping every tenant into one VRF. With +EVPN off, naming is unchanged. + +### Setup + +```bash +cmk update registered extension id= resourcetype=PhysicalNetwork \ + resourceid= \ + details[0].key=ovn_evpn_enabled details[0].value=true \ + details[1].key=ovn_evpn_type details[1].value=l3 \ + details[2].key=ovn_evpn_vni_base details[2].value=100000 +``` + +VNIs are then `100000 + ` (L3) or `100000 + ` (L2). Reserve +that range in the fabric. On each host, `ovn-bgp-agent` and FRR must be configured +per the [OpenStack example](https://docs.openstack.org/ovn-bgp-agent/latest/examples/nb_evpn_vrf.html); +this script does not touch FRR. + +### Relationship to CloudStack's existing VXLAN/EVPN support + +CloudStack already ships a [VXLAN plugin with EVPN](http://docs.cloudstack.apache.org/en/latest/plugins/vxlan.html) +(`scripts/vm/network/vnet/modifyvxlan-evpn.sh` plus FRR on each host). **It is not +reusable here and the two cannot run on the same host**, because that path requires +`BridgeVifDriver` and Linux bridges, while this extension requires `OvsVifDriver` +and `br-int`. `libvirt.vif.driver` is a single per-host setting. The VXLAN plugin +also owns its own VNI allocation through the physical network's VNI range, whereas +an extension network uses `Lswitch` isolation and is allocated no VNI at all — +hence `ovn_evpn_vni_base`. + +It is still worth reading as prior art: it shows the FRR configuration shape +(BGP unnumbered to ToRs, `advertise-all-vni`, per-hypervisor ASN, /32 loopback +VTEP) that the underlay needs either way. + +### Caveats + +- **Untested.** The tagging is verified against a stub; no `ovn-bgp-agent` has + seen it. +- **VNI derivation is arithmetic, not allocation.** Nothing checks the VNI is free + in the fabric or that two zones do not overlap. Pick disjoint bases per zone. +- **No API surface.** VNIs cannot be chosen per network; CloudStack passes no such + field and the extension framework has nowhere to put one short of a custom action. +- **L3 EVPN changes external switch naming**, so turning it on for an existing + deployment strands the old `cs-ext-` switch. Enable it before building + networks, or clean up by hand. +- The OpenStack docs' limitations apply: one flat provider network per VNI, no + reusing a VLAN id within a VNI, and tenant networks behind a gateway cannot be + split into separate VRFs without address-scope discipline. + +--- + +## 9. Testing status + +### Against a real OVN + +`test-with-ovn.sh` drives the script through a full isolated-network and VPC lifecycle +against a live OVN 24.03 northbound database, asserts on the resulting NB objects, and +then checks `ovn-northd`'s log — the NB DB will happily store a configuration that +northd cannot translate, so that last check is the one that proves anything. 87 +assertions, all passing, 0 northd errors, 223 logical flows generated. To reproduce: + +```bash +# in a container or VM with ovn-central installed +/usr/share/ovn/scripts/ovn-ctl start_northd +./test-with-ovn.sh +``` + +This found two real bugs that the stub could not, both the same mistake: + +**`ACL` and `Logical_Router_Static_Route` are not root tables.** The script created a +row and then attached it to its parent in a second `ovn-nbctl` call. OVSDB garbage +collects a row in a non-root table that nothing references by the end of the +transaction that created it, so the row was gone before the second call ran: + +```text +ovn-nbctl: applying "create" command to table ACL without --id option will have no effect +transaction error: ... references nonexistent row ... "error":"referential integrity violation" +``` + +That broke **every firewall rule and every network ACL**, and every VPC static route — +most of the `Firewall` and `NetworkACL` services. Both now create and attach in a +single transaction using `--id=@name`. `DHCP_Options` and `Load_Balancer` are root +tables, which is why the same pattern was correct for them and the fault was invisible +until a real database enforced it. + +**What this does not cover:** no CloudStack management server, no hypervisor, no +`ovn-controller`, no VM and no traffic. It proves the script speaks OVSDB correctly and +that northd accepts what it writes. It proves nothing about port binding to a real VIF, +gateway-chassis failover, or whether packets actually flow. One observation from the +teardown path: deleting a private gateway leaves any static route whose next hop sat +behind it unresolvable, and northd says so — harmless here because CloudStack revokes +those routes itself, but worth knowing. + +### Against payloads CloudStack actually produced + +Everything above feeds the script payloads written by hand. That leaves one seam +untested: the Java element writes the JSON, this script reads it, and each side's +tests are checked against a fixture written by the same person. Both suites would +still pass if a key name drifted. + +`test-with-ovn.sh --payloads ` closes it. CloudStack's +`NetworkExtensionElementPayloadContractTest` writes the payloads +`NetworkExtensionElement` really produces to +`framework/extensions/target/extension-payloads/`; this mode replays those files +against a live OVN, substituting only `physical-network-extension-details`, since +that describes the backend rather than the contract. Everything under `.payload` is +the element's own output, untouched. + +```bash +# in the CloudStack tree +mvn -pl framework/extensions test -Dtest=NetworkExtensionElementPayloadContractTest +# then, where OVN is running +./test-with-ovn.sh --payloads /path/to/framework/extensions/target/extension-payloads +``` + +Six commands are captured and replayed as a whole VPC lifecycle — 21 assertions, all +passing. The element's `implement-vpc` payload creates the router (with no source NAT +IP, the state a VPC is first implemented in); its `apply-network-acl` payload produces +the tier's ACLs, with rule 10 ahead of rule 20 at OVN priority 29990 and the source +CIDRs intact; its `create-private-gateway` payload produces a router port carrying its +address; its `apply-static-routes` payload installs routes `ovn-northd` resolves; its +`apply-private-gw-acl` payload builds the gateway ACLs; and its +`delete-private-gateway` payload removes the port. + +Two steps in that run are still harness-built — the tier's `implement-network`, and +nothing captures `update-vpc-source-nat-ip` — so the replay is not yet the complete +VPC path. The four private gateway and static route commands need the companion +CloudStack change described in §7: they do not exist in stock CloudStack, so there is +nothing to capture without it. + +### Across two chassis + +A second chassis was added and the two were put on a real Geneve tunnel — BFD up in +both directions, `forwarding=true` — so this is genuine host-to-host overlay rather +than one machine talking to itself. + +Confirmed: + +- `ovn_gateway_chassis=chassis-a,chassis-b` produces `Gateway_Chassis` rows at + priority 100 and 90 in the order given, and `ovn-northd` turns them into an + `HA_Chassis_Group` for the port. +- The highest-priority chassis claims the distributed gateway port: + `cr-cs-ext-lrp-1000 -> chassis-a`. +- When that chassis is taken down, the survivor's BFD session reports + `Control Detection Time Expired` and `state=down`, which is the signal OVN acts on. + +**Automatic takeover was not demonstrated**, and the reason is the lab rather than the +configuration: the backup's `ovn-controller` never finishes its startup handshake with +`ovs-vswitchd` in a container, logging `OVS frw cleared, restarting flow-restore-wait +sequence` in a loop, and a controller stuck in `flow-restore-wait` claims nothing. The +pieces OVN needs are all present and correct in the databases; what is missing is a +backup chassis healthy enough to act on them. Proving failover end to end needs two +real hosts. + +### Under concurrency + +```bash +./test-with-ovn.sh --concurrency 12 +``` + +Fires N callers at the same network at once — `implement-network` and +`prepare-nic` together, which is what a network coming up while instances are +already deploying looks like — then checks that exactly one of each per-network +singleton exists and that every port shares one `DHCP_Options` row. + +This found the worst bug so far. **Only `Logical_Switch_Port` and +`Logical_Router_Port` have a name index in the NB schema.** `Logical_Switch`, +`Logical_Router`, `DHCP_Options`, `Load_Balancer` and `ACL` do not, so OVSDB will +not refuse a duplicate, and `ovn-nbctl --may-exist` is a read followed by a write +with nothing in between to stop two callers both reading "absent". Eight parallel +`ovn-nbctl --may-exist ls-add` calls produce three switches of the same name. That +is not this script misusing the tool; it is what the tool offers. + +Left alone it produced up to eight `DHCP_Options` rows for one network with +instances bound across two or three of them, which is quiet rather than loud: a +later `config-dhcp-subnet` updates one row, and the instances on the others keep +being handed stale DNS, MTU or lease settings forever. + +The fix has two halves, because they cover different failure domains: + +- **A per-command file lock**, keyed on the network — or on the VPC, since a + VPC's tiers share one router and would otherwise both create it. This makes a + single management server correct, which is the ordinary case. +- **Deterministic collapse of duplicate `DHCP_Options` rows**, because a second + management server cannot see the first one's lock file. Every racer picks the + same winner and re-points ports onto it. `dhcpv4_options` is a *weak* + reference, so a port pointing at a row that is destroyed is silently left with + no DHCP at all — ports are re-pointed before the row goes, and a sweep repairs + any that a concurrent collapse orphaned anyway. + +**What is still not safe, and cannot be made safe here:** two management servers +creating the same logical switch at the same instant can still produce two +switches of that name, which splits the network in half. The script now says so +in its log rather than leaving it to be discovered. A real fix needs a +compare-and-swap, which means OVSDB `wait` operations and a native client — +precisely the argument for building this as a Java plugin instead. In practice +CloudStack serialises most per-network work with its own locks, so this is +defence in depth rather than a routine hazard. + +### Against a real chassis + +`ovn-controller` and `ovs-vswitchd` (userspace datapath) were then brought up in the +same container and registered as chassis `chassis-a`, which is what turns the NB +configuration into forwarding state. + +**The port binding convention §1 rests on holds.** Two OVS internal ports were created +on `br-int` carrying nothing but `external_ids:iface-id=` — precisely what +libvirt writes when `OvsVifDriver` emits ``. +`ovn-controller` bound both to the logical switch ports the script had created, and the +NB `Logical_Switch_Port`s went `up`: + +```text + nic-bind-0001 Port_Binding.chassis=chassis-a Logical_Switch_Port up=up + nic-bind-0002 Port_Binding.chassis=chassis-a Logical_Switch_Port up=up +``` + +`ovn_gateway_chassis` works too: the controller claimed the distributed gateway port +on its own — `cr-cs-vpc-ext-lrp-8: Claiming 02:03:00:00:00:08 192.0.2.30/24`. + +`ovn-trace` then confirms the pipelines OVN built from the script's configuration: + +- **VM to VM** on a tier is delivered to the right port, through port security. +- **DHCP** is answered by OVN itself, with every value traceable to the script's + input — `put_dhcp_opts(offerip = 10.5.1.10, dns_server = {8.8.8.8}, + lease_time = 86400, mtu = 1442, netmask = 255.255.255.0, router = 10.5.1.1, + server_id = 10.5.1.1)`. `offerip` comes from `prepare-nic`, `dns_server` and `mtu` + from the physical network details, the rest from the network's gateway and CIDR. +- **VM to the internet** reaches `ct_snat(192.0.2.60)` on the gateway port and leaves + through the localnet port into `physnet1`. +- **Port forwarding** is programmed as + `ct_lb_mark(backends=10.5.1.10:80)` at `lr_in_dnat`, and the router answers ARP for + the VIP. (`ovn-trace` itself shows a plain `ct_dnat` here because it cannot model + conntrack state; the generated logical flow is the thing to read.) + +Traces describe intent; the flow counters show what the switch actually did. Packets +built by hand and injected with `ovs-ofctl packet-out` were forwarded by the installed +datapath: + +- **Three ICMP echo requests** sent in on the first VIF's OpenFlow port came out the + second's — `n_packets=0 → 3`, `n_bytes=183`, which is 3 × the 61-byte packet. +- **Two DHCP DISCOVERs** (292 bytes each) produced two **328-byte** packets delivered + back to the client port, `n_bytes=656`. The other VIF's counter never moved, so these + are OVN's own DHCP OFFERs rather than the broadcast being flooded — the responder + runs inside `ovn-controller`, with no DHCP server anywhere in the picture. + +Final state: 326 logical flows, 898 OpenFlow rules on `br-int`, 4 bound port bindings, +and **zero errors from both `ovn-northd` and `ovn-controller`**. + +**Still not covered.** Packets were injected at the OpenFlow layer rather than sent by +a guest OS through a libvirt-managed VIF, so nothing here exercises CloudStack's agent +or `OvsVifDriver` itself — only the convention they rely on. There is no CloudStack +management server, no second chassis (so no gateway-chassis failover), and no +`ovn-bgp-agent` (so EVPN remains tag-only). + +### Against a stub + +`ovn.sh` was also exercised against a stateful stub `ovn-nbctl` that records every +invocation and models routers, NAT entries, load balancers and ACLs. Covered: + +- all 25 commands the script dispatches, each returning exactly one JSON object +- source-NAT assignment end to end: router, gateway port, localnet, default route, SNAT +- static NAT add/delete, including re-pointing an existing external IP +- port forwarding: single port, range expansion with correct port pairing, over-cap + range rejection, unsupported protocol, mismatched range sizes +- firewall ACL generation under both egress default policies, with source/dest CIDR + sets, port ranges, ICMP type/code, and public-IP-to-private-IP resolution +- teardown: partial and final VIP removal with load-balancer cleanup, source-NAT + release, full network destroy +- failure modes: missing `ovn_nb_db`, missing required payload fields, malformed + payload, unwritable log directory, `ovn-nbctl` failing, `ovn-nbctl` writing noise to + both stdout and stderr, and NAT arriving before the router exists +- VPC: one shared router across two tiers with distinct legs and MACs, network ACL + priority inversion, tier teardown leaving the router intact, `shutdown-vpc` + removing it last, and source-NAT IP re-pointing +- EVPN: L3 tagging on a per-VRF provider switch, L2 tagging per guest switch, + correct VNI scoping in both modes, and a clean no-op when disabled + +Five bugs were found and fixed this way, most of which would have been quiet +misconfigurations rather than visible errors: + +1. `jq '.default_egress_allow // true'` — the alternative operator treats a literal + `false` as empty, so a **restrictive egress policy was being applied as permissive**. +2. `IFS=$'\t'` with `read` — tab is an IFS whitespace character, so bash collapsed runs + of tabs and every field after an empty one shifted left. Source CIDRs and public IPs + were silently dropped from firewall rules. Now joined with `\u001f`. +3. `emit_failure` wrote to stdout, which call sites redirect to `/dev/null` — failures + returned an empty body. Now written to a saved fd 3. +4. The gateway leg was only created by `assign-ip`, so **a VPC tier that never + received a public IP of its own had no gateway at all** — DHCP advertised a + router address that nothing answered. `implement-network` now attaches it. +5. L2 EVPN derived its VNI from the VPC id, which would have given every tier of a + VPC the same L2 VNI and merged them into one broadcast domain in the fabric. + L2 VNIs are now always per logical switch. + +The stub proves sequencing and I/O discipline. It cannot prove that `ovn-nbctl` +accepts the transaction syntax — as the two bugs above show, it accepted syntax a real +database rejects — and it says nothing about whether traffic flows. A CloudStack +management server driving a real zone, with VMs on real hypervisors, is still the next +step. diff --git a/extensions/OVN/ovn-host-setup.sh b/extensions/OVN/ovn-host-setup.sh new file mode 100755 index 000000000000..82fb548e9303 --- /dev/null +++ b/extensions/OVN/ovn-host-setup.sh @@ -0,0 +1,508 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# --------------------------------------------------------------------------- +# Prepare hosts for the CloudStack OVN network extension. +# +# The extension programs the OVN northbound database and nothing else. It does +# not install OVN, does not configure a hypervisor, and does not edit +# agent.properties. Everything in this script is the part CloudStack will never +# do for you, collected in one place so it can be run the same way on every +# host. +# +# Roles: +# +# management ovn-nbctl and jq, so the extension script can run, plus the +# log directory it writes to. +# host Open vSwitch and ovn-controller, pointed at the OVN southbound +# database, with a stable chassis name. Also sets the CloudStack +# agent's VIF driver, without which the Lswitch branch in +# OvsVifDriver is never reached. +# gateway A host that may additionally carry north-south traffic: adds +# the provider bridge and the ovn-bridge-mappings entry that +# ovn_public_physnet refers to. +# +# Everything is idempotent. Run it again after a change and it converges. +# +# Usage: +# ovn-host-setup.sh --role management --nb-db tcp:10.0.0.1:6641 +# ovn-host-setup.sh --role host --sb-db tcp:10.0.0.1:6642 --encap-ip 10.1.0.11 +# ovn-host-setup.sh --role gateway --sb-db tcp:10.0.0.1:6642 --encap-ip 10.1.0.11 \ +# --public-nic eth1 --physnet physnet1 +# ovn-host-setup.sh --verify +# ovn-host-setup.sh --hosts hosts.txt --role host --sb-db tcp:10.0.0.1:6642 +# +# --hosts fans the script out over ssh: it copies itself to each host listed +# (one per line, "name" or "name encap-ip") and runs it there with the same +# arguments. Without --hosts it configures the machine it is running on. +# +# This is a convenience, not a replacement for configuration management. If you +# run Ansible or Puppet, read this for the list of things that need doing and +# express them there instead. +# --------------------------------------------------------------------------- + +set -u +set -o pipefail + +ROLE="" +SB_DB="" +NB_DB="" +ENCAP_IP="" +ENCAP_TYPE="geneve" +PHYSNET="physnet1" +PUBLIC_NIC="" +BRIDGE="br-ex" +SYSTEM_ID="" +AGENT_PROPERTIES="/etc/cloudstack/agent/agent.properties" +EXTENSION_LOG_DIR="/var/log/cloudstack/management/extensions" +HOSTS_FILE="" +SSH_USER="root" +DRY_RUN=0 +VERIFY_ONLY=0 +FORCE=0 +RESTART_AGENT=0 + +RED=""; GREEN=""; YELLOW=""; RESET="" +if [ -t 1 ]; then RED=$'\033[31m'; GREEN=$'\033[32m'; YELLOW=$'\033[33m'; RESET=$'\033[0m'; fi + +say() { printf ' %s\n' "$*"; } +ok() { printf ' %sok%s %s\n' "$GREEN" "$RESET" "$*"; } +warn() { printf ' %swarn%s %s\n' "$YELLOW" "$RESET" "$*" >&2; } +die() { printf ' %serror%s %s\n' "$RED" "$RESET" "$*" >&2; exit 1; } +head2() { printf '\n== %s ==\n' "$*"; } + +# run -- echoes under --dry-run, executes otherwise. +run() { + if [ "${DRY_RUN}" -eq 1 ]; then + printf ' would run: %s\n' "$*" + return 0 + fi + "$@" +} + +usage() { sed -n '19,60p' "$0" | sed 's/^# \{0,1\}//'; exit "${1:-0}"; } + +while [ $# -gt 0 ]; do + case "$1" in + --role) ROLE="${2:?}"; shift 2 ;; + --sb-db) SB_DB="${2:?}"; shift 2 ;; + --nb-db) NB_DB="${2:?}"; shift 2 ;; + --encap-ip) ENCAP_IP="${2:?}"; shift 2 ;; + --encap-type) ENCAP_TYPE="${2:?}"; shift 2 ;; + --physnet) PHYSNET="${2:?}"; shift 2 ;; + --public-nic) PUBLIC_NIC="${2:?}"; shift 2 ;; + --bridge) BRIDGE="${2:?}"; shift 2 ;; + --system-id) SYSTEM_ID="${2:?}"; shift 2 ;; + --agent-properties) AGENT_PROPERTIES="${2:?}"; shift 2 ;; + --hosts) HOSTS_FILE="${2:?}"; shift 2 ;; + --ssh-user) SSH_USER="${2:?}"; shift 2 ;; + --restart-agent) RESTART_AGENT=1; shift ;; + --dry-run|-n) DRY_RUN=1; shift ;; + --verify) VERIFY_ONLY=1; shift ;; + --force) FORCE=1; shift ;; + -h|--help) usage 0 ;; + *) die "unknown argument: $1 (try --help)" ;; + esac +done + +# --------------------------------------------------------------------------- +# ssh fan-out +# --------------------------------------------------------------------------- + +if [ -n "${HOSTS_FILE}" ]; then + [ -r "${HOSTS_FILE}" ] || die "cannot read host list ${HOSTS_FILE}" + SELF="$(cd "$(dirname "$0")" && pwd)/$(basename "$0")" + rc=0 + while read -r host hostip _; do + case "${host}" in ''|\#*) continue ;; esac + printf '\n######## %s ########\n' "${host}" + args=(--role "${ROLE}") + [ -n "${SB_DB}" ] && args+=(--sb-db "${SB_DB}") + [ -n "${NB_DB}" ] && args+=(--nb-db "${NB_DB}") + [ -n "${PUBLIC_NIC}" ] && args+=(--public-nic "${PUBLIC_NIC}") + [ -n "${PHYSNET}" ] && args+=(--physnet "${PHYSNET}") + # Per-host encap IP from the second column, else the shared one, else + # let the remote side work it out from its own route to the SB DB. + if [ -n "${hostip}" ]; then args+=(--encap-ip "${hostip}") + elif [ -n "${ENCAP_IP}" ]; then args+=(--encap-ip "${ENCAP_IP}") + fi + [ "${DRY_RUN}" -eq 1 ] && args+=(--dry-run) + [ "${VERIFY_ONLY}" -eq 1 ] && args+=(--verify) + [ "${FORCE}" -eq 1 ] && args+=(--force) + [ "${RESTART_AGENT}" -eq 1 ] && args+=(--restart-agent) + if ! scp -q "${SELF}" "${SSH_USER}@${host}:/tmp/ovn-host-setup.sh"; then + warn "${host}: could not copy the script"; rc=1; continue + fi + # shellcheck disable=SC2029 + if ! ssh "${SSH_USER}@${host}" "chmod +x /tmp/ovn-host-setup.sh && /tmp/ovn-host-setup.sh ${args[*]}"; then + warn "${host}: setup failed"; rc=1 + fi + done < "${HOSTS_FILE}" + exit "${rc}" +fi + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +need_root() { + [ "$(id -u)" -eq 0 ] || die "must run as root (or under sudo)" +} + +pkg_manager() { + for m in apt-get dnf yum zypper; do + command -v "$m" >/dev/null 2>&1 && { printf '%s' "$m"; return 0; } + done + return 1 +} + +# install_packages -- skips entirely if the commands that +# matter are already present, so this is safe on a host built by other means. +install_packages() { + local purpose="$1"; shift + local mgr + if ! mgr="$(pkg_manager)"; then + warn "no supported package manager found; install ${purpose} yourself: $*" + return 0 + fi + say "installing ${purpose}: $*" + case "${mgr}" in + apt-get) run env DEBIAN_FRONTEND=noninteractive apt-get install -y "$@" ;; + dnf|yum) run "${mgr}" install -y "$@" ;; + zypper) run zypper --non-interactive install "$@" ;; + esac +} + +service_enable() { + local svc="$1" + if command -v systemctl >/dev/null 2>&1; then + run systemctl enable --now "${svc}" || warn "could not enable ${svc}" + else + warn "no systemctl; start ${svc} yourself" + fi +} + +ovs_set() { # ovs_set -- idempotent, reports only real changes + local key="$1" value="$2" current + current="$(ovs-vsctl --if-exists get open_vswitch . "external_ids:${key}" 2>/dev/null | tr -d '"')" + if [ "${current}" = "${value}" ]; then + ok "${key} already ${value}" + return 0 + fi + run ovs-vsctl set open_vswitch . "external_ids:${key}=${value}" \ + || die "failed to set ${key}" + ok "${key} = ${value}${current:+ (was ${current})}" +} + +# Pick the source address the kernel would use to reach the southbound DB, so +# the Geneve tunnels come up on the same path as the control connection. +derive_encap_ip() { + local target="${SB_DB#*:}"; target="${target%:*}" + case "${target}" in + ''|*[!0-9.]*) return 1 ;; + esac + ip -4 route get "${target}" 2>/dev/null | sed -n 's/.* src \([0-9.]*\).*/\1/p' | head -1 +} + +# A chassis name must be stable across reboots: ovn-controller uses it to find +# its own Chassis row, and a new name on every boot leaks rows and strands +# port bindings. +resolve_system_id() { + local existing + existing="$(ovs-vsctl --if-exists get open_vswitch . external_ids:system-id 2>/dev/null | tr -d '"')" + if [ -n "${SYSTEM_ID}" ]; then printf '%s' "${SYSTEM_ID}"; return; fi + if [ -n "${existing}" ]; then printf '%s' "${existing}"; return; fi + if [ -r /etc/openvswitch/system-id.conf ]; then + cat /etc/openvswitch/system-id.conf; return + fi + hostname -s +} + +# Set a key in a java .properties file, in place, idempotently. +set_property() { + local file="$1" key="$2" value="$3" current + [ -w "$(dirname "${file}")" ] || true + if [ ! -f "${file}" ]; then + warn "${file} not found; skipping ${key} (is this host a CloudStack agent?)" + return 0 + fi + current="$(sed -n "s/^[[:space:]]*${key}[[:space:]]*=[[:space:]]*//p" "${file}" | tail -1)" + if [ "${current}" = "${value}" ]; then + ok "${key} already ${value}" + return 0 + fi + if [ "${DRY_RUN}" -eq 1 ]; then + printf ' would set %s=%s in %s (currently %s)\n' "${key}" "${value}" "${file}" "${current:-unset}" + return 0 + fi + cp -n "${file}" "${file}.before-ovn" 2>/dev/null || true + if [ -n "${current}" ]; then + sed -i "s|^[[:space:]]*${key}[[:space:]]*=.*|${key}=${value}|" "${file}" + else + printf '%s=%s\n' "${key}" "${value}" >> "${file}" + fi + ok "${key} = ${value}${current:+ (was ${current})} [backup: ${file}.before-ovn]" +} + +# --------------------------------------------------------------------------- +# Roles +# --------------------------------------------------------------------------- + +do_management() { + head2 "management server" + need_root + if command -v ovn-nbctl >/dev/null 2>&1; then + ok "ovn-nbctl present ($(ovn-nbctl --version 2>/dev/null | head -1))" + else + install_packages "the OVN client" ovn-common || install_packages "the OVN client" ovn + fi + if command -v jq >/dev/null 2>&1; then ok "jq present"; else install_packages "jq" jq; fi + + run mkdir -p "${EXTENSION_LOG_DIR}" + if id cloud >/dev/null 2>&1; then + run chown cloud:cloud "${EXTENSION_LOG_DIR}" || warn "could not chown ${EXTENSION_LOG_DIR}" + fi + ok "extension log directory ${EXTENSION_LOG_DIR}" + + if [ -n "${NB_DB}" ]; then + if [ "${DRY_RUN}" -eq 1 ]; then + say "would test the northbound connection to ${NB_DB}" + elif ovn-nbctl --db="${NB_DB}" --timeout=10 show >/dev/null 2>&1; then + ok "northbound database reachable at ${NB_DB}" + else + warn "cannot reach the northbound database at ${NB_DB}" + warn " check 'ovn-nbctl set-connection ptcp:6641' on the OVN central nodes" + fi + else + say "no --nb-db given; skipping the northbound reachability check" + fi + + cat <<'NOTE' + + Still to do by hand, because they are CloudStack API calls rather than host setup: + * install ovn.sh as /ovn/ovn.sh on EVERY management server + (note the lowercase directory; see README section 4.1) + * cmk create extension / register extension / enable the provider + * set the guest traffic label on the physical network to br-int +NOTE +} + +do_host() { + head2 "hypervisor" + need_root + [ -n "${SB_DB}" ] || die "--sb-db is required for --role host (e.g. tcp:10.0.0.1:6642)" + + if command -v ovs-vsctl >/dev/null 2>&1 && command -v ovn-controller >/dev/null 2>&1; then + ok "openvswitch and ovn-controller already installed" + else + install_packages "Open vSwitch and ovn-controller" openvswitch-switch ovn-host \ + || install_packages "Open vSwitch and ovn-controller" openvswitch ovn-host + fi + + service_enable openvswitch-switch 2>/dev/null || service_enable openvswitch + + if ! ovs-vsctl show >/dev/null 2>&1; then + if [ "${DRY_RUN}" -eq 1 ]; then + say "would configure Open vSwitch once ovsdb-server is running" + return 0 + fi + die "ovs-vsctl cannot reach ovsdb-server; is openvswitch running?" + fi + + local sysid encap + sysid="$(resolve_system_id)" + [ -n "${sysid}" ] || die "could not determine a chassis system-id; pass --system-id" + + encap="${ENCAP_IP}" + if [ -z "${encap}" ]; then + encap="$(derive_encap_ip || true)" + [ -n "${encap}" ] && say "derived encap ip ${encap} from the route to the southbound db" + fi + [ -n "${encap}" ] || die "could not determine the overlay address; pass --encap-ip" + + ovs_set system-id "${sysid}" + ovs_set ovn-remote "${SB_DB}" + ovs_set ovn-encap-type "${ENCAP_TYPE}" + ovs_set ovn-encap-ip "${encap}" + + service_enable ovn-controller + + head2 "CloudStack agent" + # Without these two the agent plugs VIFs with the bridge driver and the + # Lswitch branch in OvsVifDriver is never reached, so nothing binds. + set_property "${AGENT_PROPERTIES}" "libvirt.vif.driver" \ + "com.cloud.hypervisor.kvm.resource.OvsVifDriver" + set_property "${AGENT_PROPERTIES}" "network.bridge.type" "openvswitch" + if [ "${RESTART_AGENT}" -eq 1 ]; then + service_restart_agent + elif [ -f "${AGENT_PROPERTIES}" ]; then + say "restart cloudstack-agent for the VIF driver change to take effect" + say " (re-run with --restart-agent to do that here; it disconnects the host briefly)" + fi +} + +service_restart_agent() { + if command -v systemctl >/dev/null 2>&1; then + run systemctl restart cloudstack-agent || warn "could not restart cloudstack-agent" + ok "cloudstack-agent restarted" + else + warn "no systemctl; restart cloudstack-agent yourself" + fi +} + +do_gateway() { + do_host + head2 "gateway chassis" + [ -n "${PUBLIC_NIC}" ] || die "--public-nic is required for --role gateway" + + # Moving the NIC that carries the default route into a bridge will cut this + # host off unless the address moves with it. Refuse unless told twice. + local defnic + defnic="$(ip -4 route show default 2>/dev/null | sed -n 's/.* dev \([^ ]*\).*/\1/p' | head -1)" + if [ "${PUBLIC_NIC}" = "${defnic}" ] && [ "${FORCE}" -ne 1 ]; then + die "${PUBLIC_NIC} carries this host's default route; enslaving it will cut the host off. + Move the address onto ${BRIDGE} in your network configuration first, or pass --force + if you have already arranged for that." + fi + + if ovs-vsctl br-exists "${BRIDGE}" 2>/dev/null; then + ok "bridge ${BRIDGE} exists" + else + run ovs-vsctl add-br "${BRIDGE}" || die "could not create ${BRIDGE}" + ok "bridge ${BRIDGE} created" + fi + + if ovs-vsctl list-ports "${BRIDGE}" 2>/dev/null | grep -qx "${PUBLIC_NIC}"; then + ok "${PUBLIC_NIC} already on ${BRIDGE}" + else + run ovs-vsctl add-port "${BRIDGE}" "${PUBLIC_NIC}" || die "could not add ${PUBLIC_NIC} to ${BRIDGE}" + ok "${PUBLIC_NIC} added to ${BRIDGE}" + fi + + # add-port succeeds even when the device does not exist -- OVS records the + # port and leaves the interface in an error state. A typo in --public-nic + # would otherwise look like success and only surface as "no north-south + # traffic" much later. + if [ "${DRY_RUN}" -ne 1 ]; then + local iface_err + iface_err="$(ovs-vsctl --if-exists get interface "${PUBLIC_NIC}" error 2>/dev/null | tr -d '"')" + if [ -n "${iface_err}" ] && [ "${iface_err}" != "[]" ]; then + die "${PUBLIC_NIC} is on ${BRIDGE} but the interface is in error: ${iface_err} + Check the device name. Remove the port with: + ovs-vsctl del-port ${BRIDGE} ${PUBLIC_NIC}" + fi + ok "${PUBLIC_NIC} interface is healthy" + fi + + # Merge into any existing mappings rather than overwriting another provider. + local existing new + existing="$(ovs-vsctl --if-exists get open_vswitch . external_ids:ovn-bridge-mappings 2>/dev/null | tr -d '"')" + if printf '%s' "${existing}" | tr ',' '\n' | grep -qx "${PHYSNET}:${BRIDGE}"; then + ok "bridge mapping ${PHYSNET}:${BRIDGE} already present" + else + if [ -n "${existing}" ]; then new="${existing},${PHYSNET}:${BRIDGE}"; else new="${PHYSNET}:${BRIDGE}"; fi + ovs_set ovn-bridge-mappings "${new}" + fi + + say "add this chassis to ovn_gateway_chassis in the physical network details:" + say " $(resolve_system_id)" +} + +# --------------------------------------------------------------------------- +# Verify +# --------------------------------------------------------------------------- + +do_verify() { + head2 "verification" + local fail=0 + # A management server runs the extension script, not a hypervisor: it has no + # Open vSwitch and no chassis, and checking for them would only produce + # warnings that are correct to ignore. + if [ "${ROLE}" = "management" ]; then + if command -v ovn-nbctl >/dev/null 2>&1; then ok "ovn-nbctl present"; else warn "ovn-nbctl missing"; fail=1; fi + if command -v jq >/dev/null 2>&1; then ok "jq present"; else warn "jq missing"; fail=1; fi + if [ -d "${EXTENSION_LOG_DIR}" ]; then ok "log directory present"; else warn "log directory missing"; fail=1; fi + if [ "${fail}" -eq 0 ]; then printf '\n %sall checks passed%s\n' "$GREEN" "$RESET" + else printf '\n %ssome checks failed%s\n' "$RED" "$RESET"; fi + return "${fail}" + fi + if command -v ovs-vsctl >/dev/null 2>&1; then ok "ovs-vsctl present"; else warn "ovs-vsctl missing"; fail=1; fi + if command -v ovn-controller >/dev/null 2>&1; then ok "ovn-controller present"; else warn "ovn-controller missing"; fail=1; fi + + if ovs-vsctl show >/dev/null 2>&1; then + ok "ovsdb-server reachable" + local ids + ids="$(ovs-vsctl get open_vswitch . external_ids 2>/dev/null)" + for key in system-id ovn-remote ovn-encap-type ovn-encap-ip; do + if printf '%s' "${ids}" | grep -q "${key}"; then ok "${key} set" + else warn "${key} NOT set"; fail=1; fi + done + if ovs-vsctl br-exists br-int 2>/dev/null; then + ok "br-int exists (ovn-controller has connected at least once)" + else + warn "br-int does not exist yet; ovn-controller has not connected" + fail=1 + fi + if printf '%s' "${ids}" | grep -q "ovn-bridge-mappings"; then + ok "bridge mappings: $(ovs-vsctl get open_vswitch . external_ids:ovn-bridge-mappings 2>/dev/null)" + else + say "no bridge mappings (fine unless this is a gateway chassis)" + fi + else + warn "cannot reach ovsdb-server"; fail=1 + fi + + if pgrep -x ovn-controller >/dev/null 2>&1; then ok "ovn-controller running" + else warn "ovn-controller not running"; fail=1; fi + + if [ -f "${AGENT_PROPERTIES}" ]; then + local drv btype + drv="$(sed -n 's/^[[:space:]]*libvirt.vif.driver[[:space:]]*=[[:space:]]*//p' "${AGENT_PROPERTIES}" | tail -1)" + btype="$(sed -n 's/^[[:space:]]*network.bridge.type[[:space:]]*=[[:space:]]*//p' "${AGENT_PROPERTIES}" | tail -1)" + if [ "${drv}" = "com.cloud.hypervisor.kvm.resource.OvsVifDriver" ]; then ok "agent VIF driver is OvsVifDriver" + else warn "agent VIF driver is '${drv:-unset}', must be com.cloud.hypervisor.kvm.resource.OvsVifDriver"; fail=1; fi + if [ "${btype}" = "openvswitch" ]; then ok "agent bridge type is openvswitch" + else warn "agent bridge type is '${btype:-unset}', must be openvswitch"; fail=1; fi + else + say "no ${AGENT_PROPERTIES}; not a CloudStack agent host" + fi + + if [ "${fail}" -eq 0 ]; then printf '\n %sall checks passed%s\n' "$GREEN" "$RESET" + else printf '\n %ssome checks failed%s\n' "$RED" "$RESET"; fi + return "${fail}" +} + +# --------------------------------------------------------------------------- + +if [ "${VERIFY_ONLY}" -eq 1 ]; then + do_verify + exit $? +fi + +[ -n "${ROLE}" ] || usage 1 +[ "${DRY_RUN}" -eq 1 ] && printf '\n (dry run: nothing will be changed)\n' + +case "${ROLE}" in + management) do_management ;; + host) do_host ;; + gateway) do_gateway ;; + *) die "unknown role '${ROLE}' (management | host | gateway)" ;; +esac + +printf '\n' +[ "${DRY_RUN}" -eq 1 ] || do_verify || true diff --git a/extensions/OVN/ovn.sh b/extensions/OVN/ovn.sh new file mode 100755 index 000000000000..da9c246939c2 --- /dev/null +++ b/extensions/OVN/ovn.sh @@ -0,0 +1,1756 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# --------------------------------------------------------------------------- +# OVN NetworkOrchestrator extension for Apache CloudStack +# +# Drives an OVN northbound database through ovn-nbctl to provide guest +# networking on KVM: +# +# * L2 and native OVN DHCP for isolated and shared guest networks +# * L3 for isolated networks: source NAT, static NAT, port forwarding and +# firewall rules on a per-network logical router +# * VPCs: one logical router per VPC, a tier per logical switch, and network +# ACLs applied to the tier's router port +# * optionally, EVPN/VXLAN export of routed and bridged networks, driven by a +# companion ovn-bgp-agent rather than by this script +# +# Nothing extra runs on the hypervisor beyond ovn-controller. CloudStack's +# OvsVifDriver already binds a guest NIC to a Logical_Switch_Port named after +# the NIC's UUID when the network's broadcast domain type is Lswitch — the +# port-binding convention OVN inherited from NVP — so the KVM data path needs +# no changes. +# +# Invoked by NetworkExtensionElement as: +# ovn.sh +# +# CONTRACT NOTES (verified against cloudstack main @ 24.0.0-SNAPSHOT): +# * The management server merges our stderr into stdout +# (ProcessBuilder.redirectErrorStream(true)) and then JSON-parses the whole +# thing. Therefore NOTHING may be written to stdout or stderr except the +# single result JSON object. All diagnostics go to the log file. +# * A "status" field that is present and not "success" fails the operation +# even on exit code 0. +# * implement-network MUST return network.broadcast_domain_type and +# network.broadcast_uri or the management server throws. +# --------------------------------------------------------------------------- + +set -u +set -o pipefail + +COMMAND="${1:-}" +PAYLOAD_FILE="${2:-}" +TIMEOUT="${3:-60}" + +LOG_DIR="${CS_OVN_LOG_DIR:-/var/log/cloudstack/management/extensions}" +LOG_FILE="${LOG_DIR}/ovn.log" + +# Save the real stdout on fd 3. Result JSON is always written to fd 3, never to +# fd 1, so that a call site which redirects stdout (nbctl_or_fail ... >/dev/null) +# cannot swallow the result object on the way out. +exec 3>&1 + +# --------------------------------------------------------------------------- +# Logging — file only, never stdout/stderr (see CONTRACT NOTES) +# --------------------------------------------------------------------------- + +if ! mkdir -p "${LOG_DIR}" 2>/dev/null || ! touch "${LOG_FILE}" 2>/dev/null; then + LOG_FILE=/dev/null +fi + +log() { + printf '%s [%s] [%s] %s\n' \ + "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$$" "${COMMAND:-?}" "$*" >> "${LOG_FILE}" 2>/dev/null || true +} + +# --------------------------------------------------------------------------- +# Result emission — the ONLY thing allowed on stdout +# --------------------------------------------------------------------------- + +emit_json() { + printf '%s\n' "$1" >&3 +} + +emit_success() { + # $@ : zero or more extra "key=value" pairs added to the result object + local json pair key value + json="$(jq -nc '{status:"success"}')" + for pair in "$@"; do + key="${pair%%=*}" + value="${pair#*=}" + json="$(printf '%s' "${json}" | jq -c --arg k "${key}" --arg v "${value}" '. + {($k): $v}')" + done + log "OK ${json}" + emit_json "${json}" + exit 0 +} + +emit_failure() { + local message="$1" + log "FAIL ${message}" + emit_json "$(jq -nc --arg e "${message}" '{status:"failure", error:$e}')" + exit 1 +} + +trap 'rc=$?; if [ ${rc} -ne 0 ]; then log "aborted rc=${rc}"; fi' EXIT + +# --------------------------------------------------------------------------- +# Preflight +# --------------------------------------------------------------------------- + +command -v jq >/dev/null 2>&1 || { + printf '{"status":"failure","error":"jq is not installed on the management server"}\n' + exit 1 +} + +[ -n "${COMMAND}" ] || emit_failure "no command given" +[ -n "${PAYLOAD_FILE}" ] || emit_failure "no payload file given" +[ -r "${PAYLOAD_FILE}" ] || emit_failure "payload file ${PAYLOAD_FILE} is not readable" + +command -v ovn-nbctl >/dev/null 2>&1 || \ + emit_failure "ovn-nbctl not found — install the ovn-common package on every management server" + +PAYLOAD_JSON="$(cat "${PAYLOAD_FILE}")" +echo "${PAYLOAD_JSON}" | jq -e . >/dev/null 2>&1 || emit_failure "payload file is not valid JSON" + +log "payload ${PAYLOAD_JSON}" + +# --------------------------------------------------------------------------- +# Payload accessors +# +# phys -> physical-network-extension-details. (admin-registered) +# netd -> network-extension-details. (our own blob, +# returned by ensure-network-device and persisted by CloudStack) +# arg -> payload. (per-operation) +# --------------------------------------------------------------------------- + +_get() { + local root="$1" key="$2" + printf '%s' "${PAYLOAD_JSON}" | jq -r --arg r "${root}" --arg k "${key}" \ + '((.[$r] // {}) | .[$k]) // empty' 2>/dev/null +} + +phys() { _get 'physical-network-extension-details' "$1"; } +netd() { _get 'network-extension-details' "$1"; } +arg() { _get 'payload' "$1"; } + +# require_arg +# +# Assigns into a named variable rather than echoing. emit_failure must run in +# the main shell: inside $( ) its exit would only end the subshell and the +# script would carry on with the error JSON as the variable's value. +require_arg() { + local __var="$1" __key="$2" __val + __val="$(arg "${__key}")" + [ -n "${__val}" ] || emit_failure "payload.${__key} is required for ${COMMAND}" + printf -v "${__var}" '%s' "${__val}" +} + +# --------------------------------------------------------------------------- +# OVN Northbound connection +# --------------------------------------------------------------------------- + +NB_DB="$(phys ovn_nb_db)" +[ -n "${NB_DB}" ] || emit_failure \ + "ovn_nb_db is not set in the physical network extension details (e.g. tcp:10.0.0.1:6641)" + +NB_ARGS=(--db="${NB_DB}" --timeout="${TIMEOUT}") + +NB_PRIVKEY="$(phys ovn_nb_private_key)" +NB_CERT="$(phys ovn_nb_certificate)" +NB_CACERT="$(phys ovn_nb_ca_cert)" +[ -n "${NB_PRIVKEY}" ] && NB_ARGS+=(--private-key="${NB_PRIVKEY}") +[ -n "${NB_CERT}" ] && NB_ARGS+=(--certificate="${NB_CERT}") +[ -n "${NB_CACERT}" ] && NB_ARGS+=(--ca-cert="${NB_CACERT}") + +# nbctl: run a transaction, capture stdout, log stderr, never leak either. +nbctl() { + local out rc err_file + err_file="$(mktemp)" + out="$(ovn-nbctl "${NB_ARGS[@]}" "$@" 2>"${err_file}")" + rc=$? + if [ -s "${err_file}" ]; then + log "ovn-nbctl stderr: $(tr '\n' ' ' < "${err_file}")" + fi + rm -f "${err_file}" + if [ ${rc} -ne 0 ]; then + log "ovn-nbctl failed rc=${rc}: $*" + return ${rc} + fi + printf '%s' "${out}" + return 0 +} + +nbctl_or_fail() { + local out + if ! out="$(nbctl "$@")"; then + emit_failure "ovn-nbctl failed: $* (see ${LOG_FILE})" + fi + printf '%s' "${out}" +} + +# --------------------------------------------------------------------------- +# Naming +# +# The logical switch is derived from the CloudStack network id, which is stable +# and is the only network identity in the payload (network_uuid is NOT passed). +# +# The logical switch port name MUST equal the NIC uuid: OvsVifDriver emits +# and +# libvirt turns that into external_ids:iface-id, which is the key ovn-controller +# matches against Logical_Switch_Port.name to bind the port. +# --------------------------------------------------------------------------- + +ls_name() { printf 'cs-net-%s' "$1"; } # guest logical switch (per tier/network) +lrp_guest() { printf 'cs-lrp-%s' "$1"; } # router port facing the guest LS +lsp_guest_rtr(){ printf 'cs-lsp-rtr-%s' "$1"; } # guest LS port facing the router + +# A VPC has ONE router shared by all its tiers; a standalone isolated network has +# its own. VPC_ID is set from the payload, so these pick the right scope on their own. +lr_name() { + if [ -n "${VPC_ID}" ]; then printf 'cs-vpc-rtr-%s' "${VPC_ID}" + else printf 'cs-rtr-%s' "$1"; fi +} +lrp_ext() { # router port facing the external LS (the distributed gateway port) + if [ -n "${VPC_ID}" ]; then printf 'cs-vpc-ext-lrp-%s' "${VPC_ID}" + else printf 'cs-ext-lrp-%s' "$1"; fi +} +lsp_ext_rtr() { # external LS port facing the router + if [ -n "${VPC_ID}" ]; then printf 'cs-vpc-ext-lsp-%s' "${VPC_ID}" + else printf 'cs-ext-lsp-%s' "$1"; fi +} + +# External/provider logical switch. Normally shared by every router on the same +# public VLAN; with L3 EVPN it is scoped per VRF instead (see evpn_vni below), +# because ovn-bgp-agent maps one provider network to one VNI. +ext_ls_name() { + local vlan="${1:-untagged}" vni="${2:-}" + if [ -n "${vni}" ]; then printf 'cs-ext-%s-vni%s' "${vlan}" "${vni}" + else printf 'cs-ext-%s' "${vlan}"; fi +} +ext_ln_name() { + local vlan="${1:-untagged}" vni="${2:-}" + if [ -n "${vni}" ]; then printf 'cs-ext-ln-%s-vni%s' "${vlan}" "${vni}" + else printf 'cs-ext-ln-%s' "${vlan}"; fi +} + +# Deterministic, locally-administered MACs. The second octet namespaces the role +# so the three MACs derived from one network id never collide. +_derived_mac() { + local role_octet="$1" network_id="$2" hex + hex="$(printf '%08x' $(( network_id & 0xFFFFFFFF )))" + printf '02:%s:%s:%s:%s:%s' "${role_octet}" \ + "${hex:0:2}" "${hex:2:2}" "${hex:4:2}" "${hex:6:2}" +} +dhcp_server_mac() { _derived_mac 00 "$1"; } +guest_lrp_mac() { _derived_mac 01 "$1"; } +pgw_lrp_mac() { _derived_mac 04 "$1"; } + +# VPC private gateway objects, keyed by the gateway id. +pgw_ls_name() { printf 'cs-pgw-%s' "$1"; } +pgw_ln_name() { printf 'cs-pgw-ln-%s' "$1"; } +pgw_lrp_name() { printf 'cs-pgw-lrp-%s' "$1"; } +pgw_lsp_name() { printf 'cs-pgw-lsp-%s' "$1"; } +# Role octet 02 for a standalone network's gateway port, 03 for a VPC's, so a +# network and a VPC that happen to share a numeric id never derive the same MAC. +ext_lrp_mac() { + if [ -n "${VPC_ID}" ]; then _derived_mac 03 "${VPC_ID}" + else _derived_mac 02 "$1"; fi +} + +# Set for every VPC tier (addNetworkToPayload) and for the VPC-level commands +# (addVpcToPayload). Empty means a standalone isolated network. +VPC_ID="$(arg vpc_id)" + +PORT_SECURITY="$(phys ovn_port_security)" +[ -n "${PORT_SECURITY}" ] || PORT_SECURITY="true" + +# Name of the provider network in each host's ovn-bridge-mappings, used as the +# localnet port's network_name. +PUBLIC_PHYSNET="$(phys ovn_public_physnet)" +[ -n "${PUBLIC_PHYSNET}" ] || PUBLIC_PHYSNET="physnet1" + +# Comma-separated chassis (system-id) names, highest priority first, that may +# host the distributed gateway port. +GATEWAY_CHASSIS="$(phys ovn_gateway_chassis)" + +# Upper bound on how many individual OVN load-balancer VIPs one port-forwarding +# rule may expand into (OVN has no port-range DNAT). +PF_MAX_PORTS="$(phys ovn_pf_max_ports)" +[ -n "${PF_MAX_PORTS}" ] || PF_MAX_PORTS="128" + +# --------------------------------------------------------------------------- +# EVPN (optional) +# +# OVN keeps using Geneve between chassis; EVPN is how those networks are exposed +# into a physical BGP/EVPN fabric. The work is done out of band by ovn-bgp-agent +# running on each host, which watches the NB DB and drives FRR plus Linux VRF and +# VXLAN devices. The CMS's entire job is to tag objects with external_ids, which +# is what this script does. +# +# The key names are fixed by ovn-bgp-agent (neutron_bgpvpn:type / :vni) and there +# is an open Neutron RFE noting that ovn-nbctl is currently the only supported way +# to set them. ovn_evpn_key_prefix exists so a fork or a renamed upstream key can +# be accommodated without editing the script. +# --------------------------------------------------------------------------- + +EVPN_ENABLED="$(phys ovn_evpn_enabled)" +[ -n "${EVPN_ENABLED}" ] || EVPN_ENABLED="false" + +EVPN_TYPE="$(printf '%s' "$(phys ovn_evpn_type)" | tr '[:upper:]' '[:lower:]')" +[ -n "${EVPN_TYPE}" ] || EVPN_TYPE="l3" + +EVPN_VNI_BASE="$(phys ovn_evpn_vni_base)" +[ -n "${EVPN_VNI_BASE}" ] || EVPN_VNI_BASE="0" + +EVPN_KEY_PREFIX="$(phys ovn_evpn_key_prefix)" +[ -n "${EVPN_KEY_PREFIX}" ] || EVPN_KEY_PREFIX="neutron_bgpvpn" + +evpn_on() { [ "${EVPN_ENABLED}" = "true" ]; } + +# An L3 VNI identifies a routing domain, so all tiers of a VPC share one. An L2 +# VNI identifies a broadcast domain, so it is always per logical switch — giving +# a VPC's tiers the same L2 VNI would merge them into one L2 segment. +# Both are offset from a configured base so the range can be reserved in the fabric. +evpn_vni() { + local network_id="${1:-}" scope + if [ "${EVPN_TYPE}" = "l2" ] || [ -z "${VPC_ID}" ]; then + scope="${network_id}" + else + scope="${VPC_ID}" + fi + case "${scope}${EVPN_VNI_BASE}" in *[!0-9]*) printf ''; return 1 ;; esac + [ -n "${scope}" ] || { printf ''; return 1; } + printf '%s' "$(( EVPN_VNI_BASE + scope ))" +} + +# Tag a logical switch for ovn-bgp-agent. Escaping matters: the key contains a +# colon, so ovn-nbctl needs it backslash-escaped inside the column:key=value form. +evpn_tag_switch() { + local ls="$1" type="$2" vni="$3" + [ -n "${vni}" ] || { log "evpn: no VNI resolved for ${ls}, not tagging"; return 1; } + log "evpn: tagging ${ls} as type=${type} vni=${vni}" + nbctl set Logical_Switch "${ls}" \ + "external_ids:${EVPN_KEY_PREFIX}\\:type=${type}" \ + "external_ids:${EVPN_KEY_PREFIX}\\:vni=${vni}" >/dev/null || return 1 + return 0 +} + +# ACL priorities +ACL_PRIO_BASE=1000 # default policy +ACL_PRIO_RULE=2000 # explicit firewall rules (isolated networks) + +# VPC network ACLs carry an explicit rule number: lower number = higher precedence +# in CloudStack, higher priority = evaluated first in OVN, so we invert. +ACL_PRIO_VPC_TOP=30000 + +# --------------------------------------------------------------------------- +# DHCP_Options helpers +# --------------------------------------------------------------------------- + +find_dhcp_options() { + local network_id="$1" + nbctl --bare --columns=_uuid find DHCP_Options \ + "external_ids:cs-network-id=\"${network_id}\"" | head -n1 +} + +# --------------------------------------------------------------------------- +# Per-network serialisation +# +# A DHCP_Options row is a singleton per network, but the NB schema has no index +# on that table, so OVSDB cannot refuse a duplicate, and "create only if absent" +# cannot be expressed in one ovn-nbctl transaction. Two callers therefore both +# look, both find nothing, and both create. +# +# This is not hypothetical: CloudStack calls ensure_dhcp_options from both +# implement-network and prepare-nic, so a network coming up while instances are +# already being deployed produces several rows. VMs then bind to different ones, +# and a later config-dhcp-subnet updates only one of them -- the instances on +# the others keep being handed stale DNS, MTU or lease settings indefinitely, +# with nothing logged. +# +# The lock below removes the race within one management server, which is the +# common case. dedupe_dhcp_options converges the rest, because a cluster has +# several management servers and a file lock is local to one of them. +# --------------------------------------------------------------------------- + +LOCK_DIR="${CS_OVN_LOCK_DIR:-/var/run/cloudstack-ovn}" + +# lock_acquire ; lock_release +# +# fd 9 is the command-wide lock, fd 8 the external switch. They are separate +# because the external switch is shared by every network on a public VLAN, +# which is a different scope from the network or VPC the command names. +lock_acquire() { + local fd="$1" key="$2" + command -v flock >/dev/null 2>&1 || return 0 + mkdir -p "${LOCK_DIR}" 2>/dev/null || return 0 + eval "exec ${fd}>\"${LOCK_DIR}/${key}.lock\"" 2>/dev/null || return 0 + flock -w 30 "${fd}" 2>/dev/null \ + || log "warning: timed out waiting for the ${key} lock; continuing unlocked" + return 0 +} + +lock_release() { + local fd="$1" + command -v flock >/dev/null 2>&1 || return 0 + flock -u "${fd}" 2>/dev/null || true + eval "exec ${fd}>&-" 2>/dev/null || true + return 0 +} + +# Collapse duplicate DHCP_Options rows for a network onto one, and echo it. +# +# Every caller picks the same winner (lowest uuid), so management servers that +# raced each other converge instead of fighting. Logical_Switch_Port's +# dhcpv4_options is a WEAK reference: destroying a row silently empties the +# column on every port that pointed at it, leaving those instances with no DHCP +# at all. Ports must therefore be re-pointed BEFORE the row they reference is +# destroyed, never after. +dedupe_dhcp_options() { + local network_id="$1" uuids winner uuid lsp + uuids="$(nbctl --bare --columns=_uuid find DHCP_Options \ + "external_ids:cs-network-id=\"${network_id}\"" 2>/dev/null | grep . | sort)" + winner="$(printf '%s\n' "${uuids}" | head -1)" + [ -n "${winner}" ] || return 1 + for uuid in ${uuids}; do + [ "${uuid}" = "${winner}" ] && continue + log "collapsing duplicate DHCP_Options ${uuid} for network ${network_id} onto ${winner}" + for lsp in $(nbctl --bare --columns=name find Logical_Switch_Port \ + "dhcpv4_options=${uuid}" 2>/dev/null); do + nbctl lsp-set-dhcpv4-options "${lsp}" "${winner}" >/dev/null 2>&1 \ + || log "warning: could not re-point ${lsp} onto ${winner}" + done + nbctl --if-exists destroy DHCP_Options "${uuid}" >/dev/null 2>&1 || true + done + + # Two racers can disagree about the winner for a moment, because OVSDB uuids + # are random and a row created later can sort lower than one created + # earlier. A port pointed at a row in that window loses its reference when + # the row goes, and the reference is weak so nothing says so. Sweep this + # network's ports and give any that ended up with nothing the winner. + local port type + for port in $(nbctl lsp-list "$(ls_name "${network_id}")" 2>/dev/null | sed -n 's/.*(\(.*\))$/\1/p'); do + type="$(nbctl lsp-get-type "${port}" 2>/dev/null)" + [ -n "${type}" ] && continue # router and localnet ports take no DHCP + [ -n "$(nbctl --bare --columns=dhcpv4_options list Logical_Switch_Port "${port}" 2>/dev/null)" ] && continue + log "repairing ${port}: left with no DHCP_Options by a concurrent collapse" + nbctl lsp-set-dhcpv4-options "${port}" "${winner}" >/dev/null 2>&1 || true + done + + printf '%s' "${winner}" +} + +# Create or update the DHCP_Options row for this network's subnet. +# Echoes the row uuid. +ensure_dhcp_options() { + local network_id="$1" cidr="$2" gateway="$3" + local uuid server_mac dns mtu lease opts + + [ -n "${cidr}" ] || return 1 + [ -n "${gateway}" ] || return 1 + + server_mac="$(dhcp_server_mac "${network_id}")" + dns="$(phys ovn_dns_servers)" + [ -n "${dns}" ] || dns="${gateway}" + mtu="$(phys ovn_mtu)" + lease="$(phys ovn_dhcp_lease_time)" + [ -n "${lease}" ] || lease="86400" + + # ovn-nbctl renders set-valued DHCP options (such as dns_server) as {a,b} + opts=( + "server_id=\"${gateway}\"" + "server_mac=\"${server_mac}\"" + "lease_time=\"${lease}\"" + "router=\"${gateway}\"" + "dns_server=\"{${dns}}\"" + ) + [ -n "${mtu}" ] && opts+=("mtu=\"${mtu}\"") + + uuid="$(find_dhcp_options "${network_id}")" + if [ -n "${uuid}" ]; then + log "updating DHCP_Options ${uuid} for network ${network_id}" + nbctl set DHCP_Options "${uuid}" "cidr=${cidr}" \ + "options={$(IFS=,; echo "${opts[*]}")}" >/dev/null || return 1 + else + log "creating DHCP_Options for network ${network_id} cidr=${cidr}" + uuid="$(nbctl create DHCP_Options "cidr=${cidr}" \ + "options={$(IFS=,; echo "${opts[*]}")}" \ + "external_ids:cs-network-id=${network_id}")" || return 1 + fi + + # Another management server may have created its own row in the same window. + # Every racer collapses onto the same winner, so this converges. + local winner + winner="$(dedupe_dhcp_options "${network_id}")" && [ -n "${winner}" ] && uuid="${winner}" + + printf '%s' "${uuid}" +} + +destroy_dhcp_options() { + local network_id="$1" uuid + uuid="$(find_dhcp_options "${network_id}")" + if [ -n "${uuid}" ]; then + log "destroying DHCP_Options ${uuid}" + nbctl destroy DHCP_Options "${uuid}" >/dev/null || true + fi +} + +# --------------------------------------------------------------------------- +# L3 helpers +# +# Topology per isolated network: +# +# internet +# │ +# ┌───────┴────────┐ cs-ext- shared external logical switch +# │ localnet port │ cs-ext-ln- mapped via ovn-bridge-mappings +# └───────┬────────┘ +# │ cs-ext-lsp- (type=router) +# ┌───────┴────────┐ cs-ext-lrp- DGP, holds the source-NAT IP +# │ cs-rtr- │ logical router +# └───────┬────────┘ cs-lrp- holds the guest gateway IP +# │ cs-lsp-rtr- (type=router) +# ┌───────┴────────┐ +# │ cs-net- │ guest logical switch +# └────────────────┘ +# --------------------------------------------------------------------------- + +# "255.255.255.0" -> "24". Private gateways carry a netmask, not a CIDR. +prefix_from_netmask() { + local mask="$1" bits=0 octet + local IFS=. + for octet in ${mask}; do + case "${octet}" in + 255) bits=$(( bits + 8 )) ;; 254) bits=$(( bits + 7 )) ;; + 252) bits=$(( bits + 6 )) ;; 248) bits=$(( bits + 5 )) ;; + 240) bits=$(( bits + 4 )) ;; 224) bits=$(( bits + 3 )) ;; + 192) bits=$(( bits + 2 )) ;; 128) bits=$(( bits + 1 )) ;; + 0) ;; + *) return 1 ;; + esac + done + printf '%s' "${bits}" +} + +# Prefix length out of "10.1.1.0/24" -> "24" +prefix_from_cidr() { + local cidr="$1" + case "${cidr}" in + */*) printf '%s' "${cidr#*/}" ;; + *) printf '' ;; + esac +} + +# Create just the router. For a VPC this is the one router every tier hangs off, +# so it is also created by implement-vpc before any tier exists. +ensure_lr() { + local network_id="${1:-}" lr tag + lr="$(lr_name "${network_id}")" + if [ -n "${VPC_ID}" ]; then tag="external_ids:cs-vpc-id=${VPC_ID}" + else tag="external_ids:cs-network-id=${network_id}"; fi + log "ensuring logical router ${lr}" + nbctl -- --may-exist lr-add "${lr}" \ + -- set Logical_Router "${lr}" "${tag}" >/dev/null || return 1 + warn_if_duplicated Logical_Router "${lr}" + return 0 +} + +# Logical_Switch, Logical_Router, DHCP_Options, Load_Balancer and ACL have no +# index on name in the NB schema -- only the two port tables do. ovn-nbctl's +# --may-exist is therefore a read followed by a write with nothing to stop two +# callers both reading "absent", and OVSDB will store both rows happily. +# +# The per-command lock below prevents that within one management server. A +# second management server has no visibility of that lock, so this at least +# makes the damage loud instead of silent: two switches of the same name split +# a network in half, and the instances on the loser are quietly isolated. +warn_if_duplicated() { + local table="$1" name="$2" n + n="$(nbctl --bare --columns=_uuid find "${table}" "name=${name}" 2>/dev/null | grep -c .)" + [ "${n}" -le 1 ] 2>/dev/null && return 0 + log "WARNING: ${n} rows in ${table} are named ${name}. ${table} has no name index," \ + "so concurrent creation from more than one management server can duplicate it." \ + "Resolve by hand: ovn-nbctl find ${table} name=${name}" + return 0 +} + +# Create the router and attach this network's (or VPC tier's) guest-side leg. +ensure_router() { + local network_id="$1" cidr="$2" gateway="$3" + local lr lrp lsp prefix mac + + lr="$(lr_name "${network_id}")" + lrp="$(lrp_guest "${network_id}")" + lsp="$(lsp_guest_rtr "${network_id}")" + prefix="$(prefix_from_cidr "${cidr}")" + + [ -n "${gateway}" ] || { log "ensure_router: no gateway, skipping"; return 1; } + [ -n "${prefix}" ] || { log "ensure_router: no prefix in cidr '${cidr}'"; return 1; } + + ensure_lr "${network_id}" || return 1 + + mac="$(guest_lrp_mac "${network_id}")" + + log "attaching ${gateway}/${prefix} to ${lr} via ${lrp}" + nbctl -- --may-exist lrp-add "${lr}" "${lrp}" "${mac}" "${gateway}/${prefix}" \ + -- set Logical_Router_Port "${lrp}" "external_ids:cs-network-id=${network_id}" \ + -- --may-exist lsp-add "$(ls_name "${network_id}")" "${lsp}" \ + -- lsp-set-type "${lsp}" router \ + -- lsp-set-addresses "${lsp}" router \ + -- lsp-set-options "${lsp}" "router-port=${lrp}" >/dev/null || return 1 + return 0 +} + +# Create the shared external switch and this router's distributed gateway port. +# Only called for the source-NAT IP: that IP is the router's own external address. +ensure_external() { + local network_id="$1" public_ip="$2" public_vlan="$3" public_gateway="$4" public_cidr="$5" + local lr ext_ls ln lrp lsp prefix mac chassis prio vni + + lr="$(lr_name "${network_id}")" + prefix="$(prefix_from_cidr "${public_cidr}")" + [ -n "${prefix}" ] || { log "ensure_external: no prefix in public_cidr '${public_cidr}'"; return 1; } + [ -n "${public_gateway}" ] || { log "ensure_external: no public_gateway"; return 1; } + + # With L3 EVPN the provider switch is scoped per VRF, so each VPC (or each + # standalone network) gets its own provider network and therefore its own VNI. + vni="" + if evpn_on && [ "${EVPN_TYPE}" = "l3" ]; then + vni="$(evpn_vni "${network_id}")" || vni="" + fi + ext_ls="$(ext_ls_name "${public_vlan}" "${vni}")" + ln="$(ext_ln_name "${public_vlan}" "${vni}")" + lrp="$(lrp_ext "${network_id}")" + lsp="$(lsp_ext_rtr "${network_id}")" + mac="$(ext_lrp_mac "${network_id}")" + + log "ensuring external switch ${ext_ls} (physnet=${PUBLIC_PHYSNET} vlan=${public_vlan:-untagged})" + # Shared by every network on this public VLAN, so it needs a lock of its own: + # the command lock is scoped to one network or VPC and two different networks + # reach here concurrently. + lock_acquire 8 "ext-${PUBLIC_PHYSNET}-${public_vlan:-untagged}${vni:+-vni${vni}}" + nbctl -- --may-exist ls-add "${ext_ls}" \ + -- set Logical_Switch "${ext_ls}" "external_ids:cs-managed=true" \ + -- --may-exist lsp-add "${ext_ls}" "${ln}" \ + -- lsp-set-type "${ln}" localnet \ + -- lsp-set-addresses "${ln}" unknown \ + -- lsp-set-options "${ln}" "network_name=${PUBLIC_PHYSNET}" >/dev/null \ + || { lock_release 8; return 1; } + warn_if_duplicated Logical_Switch "${ext_ls}" + lock_release 8 + + if [ -n "${vni}" ]; then + evpn_tag_switch "${ext_ls}" "l3" "${vni}" \ + || log "warning: could not tag ${ext_ls} for EVPN" + fi + + # Tag the localnet port only for a real VLAN id. + case "${public_vlan}" in + ''|untagged|*[!0-9]*) : ;; + *) nbctl set Logical_Switch_Port "${ln}" "tag_request=${public_vlan}" >/dev/null \ + || log "warning: could not set tag_request=${public_vlan} on ${ln}" ;; + esac + + log "ensuring gateway port ${lrp} with ${public_ip}/${prefix}" + nbctl -- --may-exist lrp-add "${lr}" "${lrp}" "${mac}" "${public_ip}/${prefix}" \ + -- --may-exist lsp-add "${ext_ls}" "${lsp}" \ + -- lsp-set-type "${lsp}" router \ + -- lsp-set-addresses "${lsp}" router \ + -- lsp-set-options "${lsp}" "router-port=${lrp}" >/dev/null || return 1 + + # Pin the distributed gateway port to chassis, highest priority first. + if [ -n "${GATEWAY_CHASSIS}" ]; then + prio=100 + local IFS=',' + for chassis in ${GATEWAY_CHASSIS}; do + unset IFS + chassis="$(printf '%s' "${chassis}" | tr -d '[:space:]')" + [ -n "${chassis}" ] || continue + nbctl lrp-set-gateway-chassis "${lrp}" "${chassis}" "${prio}" >/dev/null \ + || log "warning: could not pin ${lrp} to chassis ${chassis}" + prio=$(( prio - 10 )) + IFS=',' + done + unset IFS + else + log "warning: ovn_gateway_chassis is not set — the gateway port will not be scheduled" + fi + + # Default route out of the router. + nbctl -- --may-exist lr-route-add "${lr}" "0.0.0.0/0" "${public_gateway}" >/dev/null \ + || log "warning: could not add default route via ${public_gateway}" + return 0 +} + +# Fail loudly if L3 has not been set up yet. CloudStack applies the source-NAT IP +# during implement(), so anything else arriving first is a real ordering problem +# and silently half-configuring NAT would be worse than stopping. +require_router() { + local network_id="$1" lr + lr="$(lr_name "${network_id}")" + if [ -z "$(nbctl --bare --columns=name find Logical_Router "name=${lr}")" ]; then + emit_failure "logical router ${lr} does not exist — the source NAT IP must be assigned before NAT rules" + fi +} + +# Private IP behind a public IP, read back from the router's dnat_and_snat rules. +# Used to target ingress firewall rules at the right VM instead of the whole subnet. +nat_logical_ip() { + local network_id="$1" public_ip="$2" + nbctl --bare --columns=logical_ip find NAT \ + "type=dnat_and_snat" "external_ip=${public_ip}" 2>/dev/null | head -n1 +} + +# All load balancers this network owns, optionally filtered by public IP. +lb_find() { + local network_id="$1" public_ip="${2:-}" + if [ -n "${public_ip}" ]; then + nbctl --bare --columns=_uuid find Load_Balancer \ + "external_ids:cs-network-id=\"${network_id}\"" \ + "external_ids:cs-public-ip=\"${public_ip}\"" + else + nbctl --bare --columns=_uuid find Load_Balancer \ + "external_ids:cs-network-id=\"${network_id}\"" + fi +} + +# Expand "22" or "22-25" into a space-separated list of ports. +expand_ports() { + local range="$1" start end i out="" + case "${range}" in + *-*) start="${range%%-*}"; end="${range##*-}" ;; + *) start="${range}"; end="${range}" ;; + esac + case "${start}${end}" in *[!0-9]*) return 1 ;; esac + [ "${start}" -le "${end}" ] || return 1 + if [ $(( end - start + 1 )) -gt "${PF_MAX_PORTS}" ]; then + return 2 + fi + i="${start}" + while [ "${i}" -le "${end}" ]; do + out="${out}${i} " + i=$(( i + 1 )) + done + printf '%s' "${out% }" + return 0 +} + +# --------------------------------------------------------------------------- +# Firewall ACL helpers +# --------------------------------------------------------------------------- + +# Remove every ACL this script owns on the network's guest switch. +# acl_clear [switch] [tag-key] +# acl_add [switch] [tag-key] +# +# The trailing arguments default to the guest switch of tagged +# cs-network-id, which is what isolated networks and VPC tiers want. Private +# gateways pass their own switch and cs-gateway-id instead. +acl_clear() { + local scope_id="$1" ls="${2:-}" tag_key="${3:-cs-network-id}" uuid + [ -n "${ls}" ] || ls="$(ls_name "${scope_id}")" + for uuid in $(nbctl --bare --columns=_uuid find ACL \ + "external_ids:${tag_key}=\"${scope_id}\"" 2>/dev/null); do + nbctl -- remove Logical_Switch "${ls}" acls "${uuid}" >/dev/null 2>&1 || true + nbctl destroy ACL "${uuid}" >/dev/null 2>&1 || true + done +} + +acl_add() { + local scope_id="$1" direction="$2" priority="$3" match="$4" action="$5" + local ls="${6:-}" tag_key="${7:-cs-network-id}" + [ -n "${ls}" ] || ls="$(ls_name "${scope_id}")" + # ACL is not a root table in the NB schema. A row created in a transaction + # of its own is unreferenced when that transaction commits, so OVSDB garbage + # collects it immediately and the follow-up "add" fails with a referential + # integrity violation. Create and attach have to be one transaction, with + # --id to carry the new row's uuid between the two commands. + nbctl -- --id=@acl create ACL "direction=${direction}" "priority=${priority}" \ + "match=\"$(printf '%s' "${match}" | sed 's/"/\\"/g')\"" "action=${action}" \ + "external_ids:${tag_key}=${scope_id}" \ + -- add Logical_Switch "${ls}" acls @acl >/dev/null || return 1 + return 0 +} + +# Build the L3/L4 portion of an ACL match from a rule's protocol and ports. +acl_proto_match() { + local proto="$1" port_start="$2" port_end="$3" icmp_type="$4" icmp_code="$5" + local m="" + case "$(printf '%s' "${proto}" | tr '[:upper:]' '[:lower:]')" in + tcp|udp) + m="${proto}" + if [ -n "${port_start}" ] && [ "${port_start}" != "null" ]; then + if [ -n "${port_end}" ] && [ "${port_end}" != "null" ] && [ "${port_end}" != "${port_start}" ]; then + m="${m} && ${proto}.dst >= ${port_start} && ${proto}.dst <= ${port_end}" + else + m="${m} && ${proto}.dst == ${port_start}" + fi + fi + ;; + icmp) + m="icmp4" + [ -n "${icmp_type}" ] && [ "${icmp_type}" != "null" ] && [ "${icmp_type}" -ge 0 ] 2>/dev/null \ + && m="${m} && icmp4.type == ${icmp_type}" + [ -n "${icmp_code}" ] && [ "${icmp_code}" != "null" ] && [ "${icmp_code}" -ge 0 ] 2>/dev/null \ + && m="${m} && icmp4.code == ${icmp_code}" + ;; + # "all" means every protocol; the caller already constrains the addresses + # with ip4.src/ip4.dst, so an extra "ip4" term would just be noise. + all|''|null) m="" ;; + *) m="" ;; + esac + printf '%s' "${m}" +} + +# Join non-empty match terms with " && ". +match_join() { + local out="" term + for term in "$@"; do + [ -n "${term}" ] || continue + if [ -z "${out}" ]; then out="${term}"; else out="${out} && ${term}"; fi + done + printf '%s' "${out}" +} + +# --------------------------------------------------------------------------- +# Commands +# --------------------------------------------------------------------------- + +cmd_ensure_network_device() { + # OVN is a distributed fabric: there is no per-network host to select. + # We use this hook as a Northbound reachability probe and to hand CloudStack + # a stable details blob that is echoed back on every later call. + local network_id nb_schema + require_arg network_id network_id + + if ! nb_schema="$(nbctl --bare get NB_Global . external_ids 2>/dev/null)"; then + # get on an empty table can legitimately fail; fall back to a plain list + if ! nbctl list NB_Global >/dev/null; then + emit_failure "cannot reach the OVN northbound database at ${NB_DB}" + fi + fi + + log "northbound ${NB_DB} reachable (NB_Global external_ids=${nb_schema:-none})" + emit_json "$(jq -nc --arg db "${NB_DB}" --arg ls "$(ls_name "${network_id}")" \ + '{nb_db:$db, logical_switch:$ls, status:"success"}')" + exit 0 +} + +cmd_implement_network() { + local network_id ls cidr gateway dhcp_uuid + require_arg network_id network_id + ls="$(ls_name "${network_id}")" + cidr="$(arg cidr)" + gateway="$(arg gateway)" + + log "implementing logical switch ${ls} cidr=${cidr:-none} gateway=${gateway:-none}" + + nbctl_or_fail -- --may-exist ls-add "${ls}" \ + -- set Logical_Switch "${ls}" \ + "external_ids:cs-network-id=${network_id}" \ + "external_ids:cs-zone-id=$(arg zone_id)" >/dev/null + warn_if_duplicated Logical_Switch "${ls}" + + # OVN native DHCP. If the network has no CIDR (shared network with upstream + # addressing) we simply skip it. + if [ -n "${cidr}" ] && [ -n "${gateway}" ]; then + if dhcp_uuid="$(ensure_dhcp_options "${network_id}" "${cidr}" "${gateway}")"; then + log "DHCP_Options ${dhcp_uuid} ready for ${ls}" + else + emit_failure "failed to configure DHCP options for network ${network_id}" + fi + else + log "no cidr/gateway in payload — skipping DHCP_Options for ${ls}" + fi + + # L2 EVPN stretches the guest subnet itself into the fabric, so the tag goes + # on the guest switch rather than the provider switch. + if evpn_on && [ "${EVPN_TYPE}" = "l2" ]; then + evpn_tag_switch "${ls}" "l2" "$(evpn_vni "${network_id}")" \ + || log "warning: could not tag ${ls} for L2 EVPN" + fi + + # Attach the gateway leg here rather than waiting for assign-ip. DHCP already + # advertises this gateway, so the router port should exist as soon as the + # network does — and a VPC tier that never receives a public IP of its own + # would otherwise have no gateway at all. + if [ -n "${cidr}" ] && [ -n "${gateway}" ]; then + ensure_router "${network_id}" "${cidr}" "${gateway}" \ + || emit_failure "failed to attach the gateway leg for network ${network_id}" + fi + + # REQUIRED by applyNetworkUpdateFromScriptOutput(). BroadcastDomainType.Lswitch + # is what makes OvsVifDriver take the interfaceid path on KVM. + emit_json "$(jq -nc --arg uri "lswitch:${ls}" '{ + status: "success", + "network.broadcast_domain_type": "Lswitch", + "network.broadcast_uri": $uri + }')" + exit 0 +} + +cmd_prepare_nic() { + local network_id ls nic_uuid mac ip addresses dhcp_uuid cidr gateway + require_arg network_id network_id + ls="$(ls_name "${network_id}")" + require_arg nic_uuid nic_uuid + require_arg mac mac + ip="$(arg ip)" + cidr="$(arg cidr)" + gateway="$(arg gateway)" + + # The logical switch may not exist yet if implement-network was skipped + # (persistent networks, restarts). Make this path self-sufficient. + nbctl_or_fail -- --may-exist ls-add "${ls}" \ + -- set Logical_Switch "${ls}" "external_ids:cs-network-id=${network_id}" >/dev/null + warn_if_duplicated Logical_Switch "${ls}" + + if [ -n "${ip}" ]; then + addresses="${mac} ${ip}" + else + addresses="${mac} dynamic" + fi + + log "binding port ${nic_uuid} on ${ls} addresses='${addresses}'" + + local txn=( + -- --may-exist lsp-add "${ls}" "${nic_uuid}" + -- lsp-set-addresses "${nic_uuid}" "${addresses}" + -- set Logical_Switch_Port "${nic_uuid}" + "external_ids:cs-network-id=${network_id}" + "external_ids:cs-nic-uuid=${nic_uuid}" + ) + if [ "${PORT_SECURITY}" = "true" ]; then + txn+=(-- lsp-set-port-security "${nic_uuid}" "${addresses}") + fi + + nbctl_or_fail "${txn[@]}" >/dev/null + + # Attach DHCP options if we have a subnet for them. + if [ -n "${cidr}" ] && [ -n "${gateway}" ]; then + dhcp_uuid="$(find_dhcp_options "${network_id}")" + if [ -z "${dhcp_uuid}" ]; then + dhcp_uuid="$(ensure_dhcp_options "${network_id}" "${cidr}" "${gateway}")" || dhcp_uuid="" + fi + if [ -n "${dhcp_uuid}" ]; then + nbctl lsp-set-dhcpv4-options "${nic_uuid}" "${dhcp_uuid}" >/dev/null \ + || log "warning: could not attach DHCP options ${dhcp_uuid} to ${nic_uuid}" + fi + fi + + # Wait for ovn-northd to compile the change into the southbound DB so the + # flows exist by the time libvirt plugs the VIF. + nbctl --wait=sb sync >/dev/null || log "warning: --wait=sb sync timed out" + + emit_success +} + +cmd_release_nic() { + local nic_uuid + require_arg nic_uuid nic_uuid + log "releasing port ${nic_uuid}" + nbctl_or_fail -- --if-exists lsp-del "${nic_uuid}" >/dev/null + emit_success +} + +cmd_shutdown_network() { + # Deliberately a no-op against the fabric. + # + # CloudStack shuts a network down while its NICs may still exist, and + # implement-network is not guaranteed to be followed by prepare-nic for + # already-running VMs. Deleting the logical switch here would blackhole + # them. An idle logical switch costs nothing in OVN, so we keep it and let + # destroy-network do the cleanup. + local network_id + require_arg network_id network_id + log "shutdown-network for ${network_id}: keeping logical switch $(ls_name "${network_id}") in place" + emit_success +} + +cmd_destroy_network() { + local network_id ls lr uuid + require_arg network_id network_id + ls="$(ls_name "${network_id}")" + lr="$(lr_name "${network_id}")" + + # Load balancers first: they are referenced by the router. + for uuid in $(lb_find "${network_id}"); do + nbctl -- --if-exists lr-lb-del "${lr}" "${uuid}" >/dev/null 2>&1 || true + nbctl destroy Load_Balancer "${uuid}" >/dev/null 2>&1 || true + done + + acl_clear "${network_id}" + + if [ -n "${VPC_ID}" ]; then + # A VPC tier: detach this tier's leg but leave the shared router alone — + # sibling tiers are still using it. shutdown-vpc removes the router. + log "destroying VPC tier ${ls} (router ${lr} belongs to VPC ${VPC_ID}, kept)" + nbctl -- --if-exists lrp-del "$(lrp_guest "${network_id}")" >/dev/null 2>&1 || true + else + log "destroying logical router ${lr} and logical switch ${ls}" + # lr-del and ls-del cascade to their ports; the external-side LSP lives on + # the shared external switch, so it is removed by name first. + nbctl -- --if-exists lsp-del "$(lsp_ext_rtr "${network_id}")" >/dev/null 2>&1 || true + nbctl -- --if-exists lr-del "${lr}" >/dev/null 2>&1 || true + fi + + nbctl_or_fail -- --if-exists ls-del "${ls}" >/dev/null + destroy_dhcp_options "${network_id}" + emit_success +} + +# --------------------------------------------------------------------------- +# L3: NAT, port forwarding, firewall +# --------------------------------------------------------------------------- + +cmd_assign_ip() { + local network_id public_ip source_nat cidr gateway public_vlan public_gateway public_cidr lr + require_arg network_id network_id + require_arg public_ip public_ip + source_nat="$(arg source_nat)" + cidr="$(arg cidr)" + gateway="$(arg gateway)" + public_vlan="$(arg public_vlan)" + public_gateway="$(arg public_gateway)" + public_cidr="$(arg public_cidr)" + lr="$(lr_name "${network_id}")" + + ensure_router "${network_id}" "${cidr}" "${gateway}" \ + || emit_failure "failed to create logical router for network ${network_id}" + + if [ "${source_nat}" = "true" ]; then + ensure_external "${network_id}" "${public_ip}" "${public_vlan}" \ + "${public_gateway}" "${public_cidr}" \ + || emit_failure "failed to configure external connectivity for ${public_ip}" + + log "adding snat ${cidr} -> ${public_ip} on ${lr}" + nbctl -- --may-exist lr-nat-add "${lr}" snat "${public_ip}" "${cidr}" >/dev/null \ + || emit_failure "failed to add source NAT ${cidr} -> ${public_ip}" + else + # A plain public IP becomes reachable once a static NAT or port forward + # references it; OVN answers ARP for NAT external IPs on the gateway port. + log "public IP ${public_ip} assigned, no source NAT requested — nothing to program yet" + fi + + nbctl --wait=sb sync >/dev/null || log "warning: --wait=sb sync timed out" + emit_success +} + +cmd_release_ip() { + local network_id public_ip source_nat cidr lr lrp lsp uuid + require_arg network_id network_id + require_arg public_ip public_ip + source_nat="$(arg source_nat)" + cidr="$(arg cidr)" + lr="$(lr_name "${network_id}")" + + # Drop any NAT entry that uses this address, whichever kind it is. + nbctl -- --if-exists lr-nat-del "${lr}" dnat_and_snat "${public_ip}" >/dev/null 2>&1 || true + + # Tear down load balancers (port forwards) bound to this address. + for uuid in $(lb_find "${network_id}" "${public_ip}"); do + nbctl -- --if-exists lr-lb-del "${lr}" "${uuid}" >/dev/null 2>&1 || true + nbctl destroy Load_Balancer "${uuid}" >/dev/null 2>&1 || true + done + + if [ "${source_nat}" = "true" ]; then + log "releasing source NAT IP ${public_ip} and the gateway port" + [ -n "${cidr}" ] && nbctl -- --if-exists lr-nat-del "${lr}" snat "${cidr}" >/dev/null 2>&1 || true + nbctl -- --if-exists lr-route-del "${lr}" "0.0.0.0/0" >/dev/null 2>&1 || true + lrp="$(lrp_ext "${network_id}")" + lsp="$(lsp_ext_rtr "${network_id}")" + nbctl -- --if-exists lsp-del "${lsp}" >/dev/null 2>&1 || true + nbctl -- --if-exists lrp-del "${lrp}" >/dev/null 2>&1 || true + fi + + emit_success +} + +cmd_add_static_nat() { + local network_id public_ip private_ip lr + require_arg network_id network_id + require_arg public_ip public_ip + require_arg private_ip private_ip + require_router "${network_id}" + lr="$(lr_name "${network_id}")" + + log "adding dnat_and_snat ${public_ip} <-> ${private_ip} on ${lr}" + # Replace any stale mapping for this external IP first so a re-point is clean. + nbctl -- --if-exists lr-nat-del "${lr}" dnat_and_snat "${public_ip}" >/dev/null 2>&1 || true + nbctl -- lr-nat-add "${lr}" dnat_and_snat "${public_ip}" "${private_ip}" >/dev/null \ + || emit_failure "failed to add static NAT ${public_ip} -> ${private_ip}" + + nbctl --wait=sb sync >/dev/null || log "warning: --wait=sb sync timed out" + emit_success +} + +cmd_delete_static_nat() { + local network_id public_ip lr + require_arg network_id network_id + require_arg public_ip public_ip + lr="$(lr_name "${network_id}")" + log "removing dnat_and_snat ${public_ip} from ${lr}" + nbctl -- --if-exists lr-nat-del "${lr}" dnat_and_snat "${public_ip}" >/dev/null 2>&1 || true + emit_success +} + +# One Load_Balancer per (network, public IP, protocol); each forwarded port is a +# vips map entry. OVN has no port-range DNAT, so a range expands into one entry +# per port, capped by ovn_pf_max_ports. +_pf_lb_uuid() { + local network_id="$1" public_ip="$2" proto="$3" create="$4" uuid lr + uuid="$(nbctl --bare --columns=_uuid find Load_Balancer \ + "external_ids:cs-network-id=\"${network_id}\"" \ + "external_ids:cs-public-ip=\"${public_ip}\"" \ + "external_ids:cs-protocol=\"${proto}\"" 2>/dev/null | head -n1)" + if [ -z "${uuid}" ] && [ "${create}" = "create" ]; then + lr="$(lr_name "${network_id}")" + uuid="$(nbctl create Load_Balancer "name=cs-pf-${network_id}-${public_ip}-${proto}" \ + "protocol=${proto}" \ + "external_ids:cs-network-id=${network_id}" \ + "external_ids:cs-vpc-id=${VPC_ID:-none}" \ + "external_ids:cs-public-ip=${public_ip}" \ + "external_ids:cs-protocol=${proto}")" || return 1 + [ -n "${uuid}" ] || return 1 + nbctl lr-lb-add "${lr}" "${uuid}" >/dev/null 2>&1 || true + fi + printf '%s' "${uuid}" +} + +cmd_add_port_forward() { + local network_id public_ip private_ip public_port private_port proto + local pub_list priv_list uuid rc i n pub priv + require_arg network_id network_id + require_arg public_ip public_ip + require_arg private_ip private_ip + require_arg public_port public_port + require_arg private_port private_port + proto="$(printf '%s' "$(arg protocol)" | tr '[:upper:]' '[:lower:]')" + case "${proto}" in + tcp|udp|sctp) : ;; + *) emit_failure "unsupported port-forwarding protocol '${proto}' — OVN load balancers support tcp, udp and sctp" ;; + esac + require_router "${network_id}" + + pub_list="$(expand_ports "${public_port}")"; rc=$? + [ ${rc} -eq 2 ] && emit_failure "public port range ${public_port} exceeds ovn_pf_max_ports=${PF_MAX_PORTS}; OVN has no port-range DNAT" + [ ${rc} -ne 0 ] && emit_failure "invalid public port range '${public_port}'" + priv_list="$(expand_ports "${private_port}")"; rc=$? + [ ${rc} -eq 2 ] && emit_failure "private port range ${private_port} exceeds ovn_pf_max_ports=${PF_MAX_PORTS}" + [ ${rc} -ne 0 ] && emit_failure "invalid private port range '${private_port}'" + + set -- ${pub_list}; n=$# + set -- ${priv_list} + [ $# -eq ${n} ] || emit_failure "public range ${public_port} and private range ${private_port} differ in size" + + uuid="$(_pf_lb_uuid "${network_id}" "${public_ip}" "${proto}" create)" \ + || emit_failure "failed to create load balancer for ${public_ip}/${proto}" + [ -n "${uuid}" ] || emit_failure "failed to resolve load balancer for ${public_ip}/${proto}" + + i=1 + for pub in ${pub_list}; do + priv="$(printf '%s' "${priv_list}" | cut -d' ' -f${i})" + log "port forward ${public_ip}:${pub} -> ${private_ip}:${priv} (${proto})" + nbctl set Load_Balancer "${uuid}" \ + "vips:\"${public_ip}:${pub}\"=\"${private_ip}:${priv}\"" >/dev/null \ + || emit_failure "failed to add vip ${public_ip}:${pub}" + i=$(( i + 1 )) + done + + nbctl --wait=sb sync >/dev/null || log "warning: --wait=sb sync timed out" + emit_success +} + +cmd_delete_port_forward() { + local network_id public_ip public_port proto pub_list uuid pub remaining lr + require_arg network_id network_id + require_arg public_ip public_ip + require_arg public_port public_port + proto="$(printf '%s' "$(arg protocol)" | tr '[:upper:]' '[:lower:]')" + lr="$(lr_name "${network_id}")" + + uuid="$(_pf_lb_uuid "${network_id}" "${public_ip}" "${proto}" nocreate)" + if [ -z "${uuid}" ]; then + log "no load balancer for ${public_ip}/${proto} — nothing to remove" + emit_success + fi + + pub_list="$(expand_ports "${public_port}")" || pub_list="${public_port}" + for pub in ${pub_list}; do + log "removing vip ${public_ip}:${pub} (${proto})" + nbctl remove Load_Balancer "${uuid}" vips "\"${public_ip}:${pub}\"" >/dev/null 2>&1 || true + done + + # Drop the load balancer once its last vip is gone. + remaining="$(nbctl --bare --columns=vips list Load_Balancer "${uuid}" 2>/dev/null | tr -d '[:space:]')" + if [ -z "${remaining}" ] || [ "${remaining}" = "{}" ]; then + log "load balancer ${uuid} has no vips left — destroying it" + nbctl -- --if-exists lr-lb-del "${lr}" "${uuid}" >/dev/null 2>&1 || true + nbctl destroy Load_Balancer "${uuid}" >/dev/null 2>&1 || true + fi + + emit_success +} + +cmd_apply_fw_rules() { + local network_id cidr fw default_allow base_action line + local rtype proto pstart pend itype icode pubip scidrs dcidrs + local match proto_match target src_match dst_match action prio + require_arg network_id network_id + + fw="$(printf '%s' "${PAYLOAD_JSON}" | jq -c '.payload.fw_rules // {}')" + cidr="$(printf '%s' "${fw}" | jq -r '.cidr // empty')" + [ -n "${cidr}" ] || cidr="$(arg cidr)" + [ -n "${cidr}" ] || emit_failure "apply-fw-rules needs the network cidr" + # NOTE: not `.default_egress_allow // true` — jq's alternative operator treats + # a literal false as empty, so that idiom silently turns a restrictive egress + # policy into a permissive one. + default_allow="$(printf '%s' "${fw}" | jq -r 'if .default_egress_allow == false then "false" else "true" end')" + + log "rebuilding ACLs for network ${network_id} cidr=${cidr} default_egress_allow=${default_allow}" + + # CloudStack sends the complete active rule set every time, so a full rebuild + # is both correct and the simplest way to stay in sync. + acl_clear "${network_id}" + + # Let DHCP through in both directions regardless of policy: OVN answers DHCP + # itself and we must not let a restrictive egress policy break addressing. + acl_add "${network_id}" from-lport $(( ACL_PRIO_RULE + 100 )) \ + "udp && udp.src == 68 && udp.dst == 67" allow || true + acl_add "${network_id}" to-lport $(( ACL_PRIO_RULE + 100 )) \ + "udp && udp.src == 67 && udp.dst == 68" allow || true + + # Default egress policy. + if [ "${default_allow}" = "true" ]; then + base_action="allow-related" + else + base_action="drop" + fi + acl_add "${network_id}" from-lport "${ACL_PRIO_BASE}" "ip4.src == ${cidr}" "${base_action}" \ + || emit_failure "failed to apply the default egress policy" + + # Default ingress policy: anything arriving from outside the subnet is dropped + # unless an explicit rule allows it. Replies to VM-initiated flows still pass, + # because the egress ACL above is allow-related. + acl_add "${network_id}" to-lport "${ACL_PRIO_BASE}" \ + "ip4.dst == ${cidr} && ip4.src != ${cidr}" drop \ + || emit_failure "failed to apply the default ingress policy" + + # Fields are joined with US (0x1f), not tab: tab is an IFS *whitespace* + # character, so bash collapses runs of them and silently shifts every field + # after an empty one. US is not whitespace, so empty fields survive. + while IFS=$'\037' read -r rtype proto pstart pend itype icode pubip scidrs dcidrs; do + [ -n "${rtype}" ] || continue + proto_match="$(acl_proto_match "${proto}" "${pstart}" "${pend}" "${itype}" "${icode}")" + prio="${ACL_PRIO_RULE}" + + if [ "${rtype}" = "egress" ]; then + # Explicit egress rules invert the default policy. + if [ "${default_allow}" = "true" ]; then action="drop"; else action="allow-related"; fi + src_match="ip4.src == ${cidr}" + if [ -n "${scidrs}" ] && [ "${scidrs}" != "0.0.0.0/0" ]; then + src_match="ip4.src == {${scidrs}}" + fi + dst_match="" + if [ -n "${dcidrs}" ] && [ "${dcidrs}" != "0.0.0.0/0" ]; then + dst_match="ip4.dst == {${dcidrs}}" + fi + match="$(match_join "${src_match}" "${dst_match}" "${proto_match}")" + acl_add "${network_id}" from-lport "${prio}" "${match}" "${action}" \ + || log "warning: could not apply egress rule (${match})" + else + # Ingress rules are always allow rules sitting above the default drop. + # Resolve the public IP back to the VM behind it so the rule targets + # that VM rather than the whole subnet. + target="" + [ -n "${pubip}" ] && target="$(nat_logical_ip "${network_id}" "${pubip}")" + if [ -n "${target}" ]; then + dst_match="ip4.dst == ${target}" + else + dst_match="ip4.dst == ${cidr}" + [ -n "${pubip}" ] && log "no static NAT found for ${pubip}; widening ingress rule to ${cidr}" + fi + src_match="" + if [ -n "${scidrs}" ] && [ "${scidrs}" != "0.0.0.0/0" ]; then + src_match="ip4.src == {${scidrs}}" + fi + match="$(match_join "${src_match}" "${dst_match}" "${proto_match}")" + acl_add "${network_id}" to-lport "${prio}" "${match}" allow-related \ + || log "warning: could not apply ingress rule (${match})" + fi + done </dev/null || log "warning: --wait=sb sync timed out" + emit_success +} + +# --------------------------------------------------------------------------- +# VPC and network ACLs +# +# One logical router per VPC (cs-vpc-rtr-), shared by every tier. Each +# tier keeps its own logical switch and its own leg on that router, so the +# isolated-network code paths work unchanged — lr_name()/lrp_ext()/ext_lrp_mac() +# switch scope on their own when VPC_ID is set. +# --------------------------------------------------------------------------- + +cmd_implement_vpc() { + local vpc_id vpc_cidr public_ip source_nat public_vlan public_gateway public_cidr lr + require_arg vpc_id vpc_id + vpc_cidr="$(arg vpc_cidr)" + public_ip="$(arg public_ip)" + source_nat="$(arg source_nat)" + public_vlan="$(arg public_vlan)" + public_gateway="$(arg public_gateway)" + public_cidr="$(arg public_cidr)" + lr="$(lr_name "")" + + ensure_lr "" || emit_failure "failed to create logical router for VPC ${vpc_id}" + + # CloudStack includes the source NAT IP here only if one is already allocated; + # otherwise it arrives later via update-vpc-source-nat-ip or assign-ip. + if [ -n "${public_ip}" ] && [ "${source_nat}" = "true" ]; then + ensure_external "" "${public_ip}" "${public_vlan}" "${public_gateway}" "${public_cidr}" \ + || emit_failure "failed to configure external connectivity for VPC ${vpc_id}" + if [ -n "${vpc_cidr}" ]; then + log "adding vpc-wide snat ${vpc_cidr} -> ${public_ip} on ${lr}" + nbctl -- --may-exist lr-nat-add "${lr}" snat "${public_ip}" "${vpc_cidr}" >/dev/null \ + || emit_failure "failed to add source NAT ${vpc_cidr} -> ${public_ip}" + else + log "no vpc_cidr in payload — skipping the VPC-wide source NAT rule" + fi + else + log "VPC ${vpc_id} implemented without a source NAT IP yet" + fi + + nbctl --wait=sb sync >/dev/null || log "warning: --wait=sb sync timed out" + emit_success +} + +cmd_shutdown_vpc() { + local vpc_id lr lsp lrp uuid + require_arg vpc_id vpc_id + lr="$(lr_name "")" + lsp="$(lsp_ext_rtr "")" + lrp="$(lrp_ext "")" + + # CloudStack calls destroy-network for every tier before this, so only the + # VPC-wide objects are left. + for uuid in $(nbctl --bare --columns=_uuid find Load_Balancer \ + "external_ids:cs-vpc-id=\"${vpc_id}\"" 2>/dev/null); do + nbctl -- --if-exists lr-lb-del "${lr}" "${uuid}" >/dev/null 2>&1 || true + nbctl destroy Load_Balancer "${uuid}" >/dev/null 2>&1 || true + done + + log "shutting down VPC ${vpc_id}: removing ${lsp}, ${lrp} and ${lr}" + nbctl -- --if-exists lsp-del "${lsp}" >/dev/null 2>&1 || true + nbctl -- --if-exists lrp-del "${lrp}" >/dev/null 2>&1 || true + nbctl -- --if-exists lr-del "${lr}" >/dev/null 2>&1 || true + emit_success +} + +cmd_update_vpc_source_nat_ip() { + local vpc_id vpc_cidr public_ip public_vlan public_gateway public_cidr lr lrp prefix + require_arg vpc_id vpc_id + require_arg public_ip public_ip + vpc_cidr="$(arg vpc_cidr)" + public_vlan="$(arg public_vlan)" + public_gateway="$(arg public_gateway)" + public_cidr="$(arg public_cidr)" + lr="$(lr_name "")" + lrp="$(lrp_ext "")" + prefix="$(prefix_from_cidr "${public_cidr}")" + + ensure_lr "" || emit_failure "failed to ensure logical router for VPC ${vpc_id}" + + # Drop the old VPC-wide SNAT. For type=snat the IP argument is the *logical* + # (internal) IP, so the VPC CIDR identifies the rule regardless of which + # external address it used. + if [ -n "${vpc_cidr}" ]; then + nbctl -- --if-exists lr-nat-del "${lr}" snat "${vpc_cidr}" >/dev/null 2>&1 || true + fi + + # Re-point the gateway port at the new address, creating it if this is the + # first source NAT IP the VPC has had. + if [ -n "$(nbctl --bare --columns=name find Logical_Router_Port "name=${lrp}")" ] \ + && [ -n "${prefix}" ]; then + log "re-pointing ${lrp} at ${public_ip}/${prefix}" + nbctl set Logical_Router_Port "${lrp}" "networks=\"${public_ip}/${prefix}\"" >/dev/null \ + || emit_failure "failed to update ${lrp} to ${public_ip}/${prefix}" + else + ensure_external "" "${public_ip}" "${public_vlan}" "${public_gateway}" "${public_cidr}" \ + || emit_failure "failed to configure external connectivity for VPC ${vpc_id}" + fi + + if [ -n "${vpc_cidr}" ]; then + log "adding vpc-wide snat ${vpc_cidr} -> ${public_ip} on ${lr}" + nbctl -- --may-exist lr-nat-add "${lr}" snat "${public_ip}" "${vpc_cidr}" >/dev/null \ + || emit_failure "failed to add source NAT ${vpc_cidr} -> ${public_ip}" + fi + + nbctl --wait=sb sync >/dev/null || log "warning: --wait=sb sync timed out" + emit_success +} + +cmd_apply_network_acl() { + local network_id cidr rules count prio + local number action rtype proto pstart pend itype icode scidrs + local proto_match src_match dst_match match ovn_action + require_arg network_id network_id + cidr="$(arg cidr)" + [ -n "${cidr}" ] || emit_failure "apply-network-acl needs the tier cidr" + + rules="$(printf '%s' "${PAYLOAD_JSON}" | jq -c '.payload.acl_rules // []')" + count="$(printf '%s' "${rules}" | jq 'length')" + log "rebuilding network ACLs for tier ${network_id} cidr=${cidr} rules=${count}" + + acl_clear "${network_id}" + + # DHCP above everything, as for isolated networks. + acl_add "${network_id}" from-lport $(( ACL_PRIO_VPC_TOP + 100 )) \ + "udp && udp.src == 68 && udp.dst == 67" allow || true + acl_add "${network_id}" to-lport $(( ACL_PRIO_VPC_TOP + 100 )) \ + "udp && udp.src == 67 && udp.dst == 68" allow || true + + # VPC tiers default to deny in both directions; CloudStack's "default allow" + # ACL is expressed as explicit allow rules, so an empty list really does mean + # deny everything. + acl_add "${network_id}" from-lport "${ACL_PRIO_BASE}" "ip4.src == ${cidr}" drop \ + || emit_failure "failed to apply the default egress policy" + acl_add "${network_id}" to-lport "${ACL_PRIO_BASE}" \ + "ip4.dst == ${cidr} && ip4.src != ${cidr}" drop \ + || emit_failure "failed to apply the default ingress policy" + + while IFS=$'\037' read -r number action rtype proto pstart pend itype icode scidrs; do + [ -n "${rtype}" ] || continue + + # CloudStack: lower rule number wins. OVN: higher priority wins. + prio=$(( ACL_PRIO_VPC_TOP - number )) + [ "${prio}" -gt "${ACL_PRIO_BASE}" ] || prio=$(( ACL_PRIO_BASE + 1 )) + + case "${action}" in + allow) ovn_action="allow-related" ;; + *) ovn_action="drop" ;; + esac + + proto_match="$(acl_proto_match "${proto}" "${pstart}" "${pend}" "${itype}" "${icode}")" + + # The VR applies an ingress rule's CIDR as the source and an egress rule's + # CIDR as the destination (systemvm configure.py: "-s" vs "-d"), so the + # single sourceCidrs field means different things per direction. + if [ "${rtype}" = "egress" ]; then + src_match="ip4.src == ${cidr}" + dst_match="" + if [ -n "${scidrs}" ] && [ "${scidrs}" != "0.0.0.0/0" ]; then + dst_match="ip4.dst == {${scidrs}}" + fi + match="$(match_join "${src_match}" "${dst_match}" "${proto_match}")" + acl_add "${network_id}" from-lport "${prio}" "${match}" "${ovn_action}" \ + || log "warning: could not apply egress ACL rule ${number} (${match})" + else + src_match="" + if [ -n "${scidrs}" ] && [ "${scidrs}" != "0.0.0.0/0" ]; then + src_match="ip4.src == {${scidrs}}" + fi + dst_match="ip4.dst == ${cidr}" + match="$(match_join "${src_match}" "${dst_match}" "${proto_match}")" + acl_add "${network_id}" to-lport "${prio}" "${match}" "${ovn_action}" \ + || log "warning: could not apply ingress ACL rule ${number} (${match})" + fi + done </dev/null || log "warning: --wait=sb sync timed out" + emit_success +} + +# --------------------------------------------------------------------------- +# VPC private gateways and static routes +# +# These need the matching verbs in NetworkExtensionElement; upstream CloudStack +# currently throws UnsupportedOperationException for all four. See the patch in +# ../cloudstack (branch network-extension-vpc-gateways-routes). +# --------------------------------------------------------------------------- + +PRIVATE_PHYSNET="$(phys ovn_private_physnet)" +[ -n "${PRIVATE_PHYSNET}" ] || PRIVATE_PHYSNET="${PUBLIC_PHYSNET}" + +cmd_create_private_gateway() { + local gateway_id ip netmask vlan prefix lr ls ln lrp lsp mac + require_arg gateway_id gateway_id + require_arg ip gateway_ip_address + netmask="$(arg gateway_netmask)" + vlan="$(arg gateway_vlan)" + + prefix="$(prefix_from_netmask "${netmask}")" \ + || emit_failure "invalid netmask '${netmask}' on private gateway ${gateway_id}" + [ -n "${prefix}" ] || emit_failure "could not derive a prefix from netmask '${netmask}'" + + ensure_lr "" || emit_failure "failed to ensure the VPC router for private gateway ${gateway_id}" + + lr="$(lr_name "")" + ls="$(pgw_ls_name "${gateway_id}")" + ln="$(pgw_ln_name "${gateway_id}")" + lrp="$(pgw_lrp_name "${gateway_id}")" + lsp="$(pgw_lsp_name "${gateway_id}")" + mac="$(pgw_lrp_mac "${gateway_id}")" + + log "creating private gateway ${gateway_id}: ${ip}/${prefix} on vlan ${vlan:-untagged}" + + nbctl_or_fail -- --may-exist ls-add "${ls}" \ + -- set Logical_Switch "${ls}" "external_ids:cs-gateway-id=${gateway_id}" \ + -- --may-exist lsp-add "${ls}" "${ln}" \ + -- lsp-set-type "${ln}" localnet \ + -- lsp-set-addresses "${ln}" unknown \ + -- lsp-set-options "${ln}" "network_name=${PRIVATE_PHYSNET}" >/dev/null + warn_if_duplicated Logical_Switch "${ls}" + + case "${vlan}" in + ''|untagged|*[!0-9]*) : ;; + *) nbctl set Logical_Switch_Port "${ln}" "tag_request=${vlan}" >/dev/null \ + || log "warning: could not tag ${ln} with vlan ${vlan}" ;; + esac + + nbctl_or_fail -- --may-exist lrp-add "${lr}" "${lrp}" "${mac}" "${ip}/${prefix}" \ + -- set Logical_Router_Port "${lrp}" "external_ids:cs-gateway-id=${gateway_id}" \ + -- --may-exist lsp-add "${ls}" "${lsp}" \ + -- lsp-set-type "${lsp}" router \ + -- lsp-set-addresses "${lsp}" router \ + -- lsp-set-options "${lsp}" "router-port=${lrp}" >/dev/null + + nbctl --wait=sb sync >/dev/null || log "warning: --wait=sb sync timed out" + emit_success +} + +cmd_delete_private_gateway() { + local gateway_id ls lrp + require_arg gateway_id gateway_id + ls="$(pgw_ls_name "${gateway_id}")" + lrp="$(pgw_lrp_name "${gateway_id}")" + + log "deleting private gateway ${gateway_id}" + acl_clear "${gateway_id}" "${ls}" "cs-gateway-id" + nbctl -- --if-exists lrp-del "${lrp}" >/dev/null 2>&1 || true + nbctl_or_fail -- --if-exists ls-del "${ls}" >/dev/null + emit_success +} + +cmd_apply_static_routes() { + local vpc_id lr routes count uuid cidr next_hop + require_arg vpc_id vpc_id + ensure_lr "" || emit_failure "failed to ensure the VPC router for ${vpc_id}" + lr="$(lr_name "")" + + routes="$(printf '%s' "${PAYLOAD_JSON}" | jq -c '.payload.static_routes // []')" + count="$(printf '%s' "${routes}" | jq 'length')" + log "rebuilding ${count} static route(s) on ${lr}" + + # Only routes this script created are cleared. The default route that + # ensure_external adds with lr-route-add carries no external_ids, so a static + # route rebuild can never strip the VPC's path to the internet. + for uuid in $(nbctl --bare --columns=_uuid find Logical_Router_Static_Route \ + "external_ids:cs-vpc-id=\"${vpc_id}\"" 2>/dev/null); do + nbctl -- remove Logical_Router "${lr}" static_routes "${uuid}" >/dev/null 2>&1 || true + nbctl destroy Logical_Router_Static_Route "${uuid}" >/dev/null 2>&1 || true + done + + while IFS=$'\037' read -r cidr next_hop; do + [ -n "${cidr}" ] || continue + if [ -z "${next_hop}" ]; then + log "skipping static route ${cidr}: no next hop" + continue + fi + # Logical_Router_Static_Route is not a root table either, so the row + # has to be created and attached in a single transaction (see acl_add). + nbctl -- --id=@route create Logical_Router_Static_Route \ + "ip_prefix=${cidr}" "nexthop=${next_hop}" \ + "external_ids:cs-vpc-id=${vpc_id}" \ + -- add Logical_Router "${lr}" static_routes @route >/dev/null \ + || emit_failure "failed to add static route ${cidr} via ${next_hop} to ${lr}" + log "static route ${cidr} via ${next_hop}" + done </dev/null || log "warning: --wait=sb sync timed out" + emit_success +} + +cmd_apply_private_gw_acl() { + local gateway_id ls cidr ip prefix netmask rules count prio + local number action rtype proto pstart pend itype icode scidrs + local proto_match src_match dst_match match ovn_action + require_arg gateway_id gateway_id + require_arg ip gateway_ip_address + netmask="$(arg gateway_netmask)" + ls="$(pgw_ls_name "${gateway_id}")" + + prefix="$(prefix_from_netmask "${netmask}")" || prefix="" + [ -n "${prefix}" ] || emit_failure "invalid netmask '${netmask}' on private gateway ${gateway_id}" + cidr="${ip}/${prefix}" + + rules="$(printf '%s' "${PAYLOAD_JSON}" | jq -c '.payload.acl_rules // []')" + count="$(printf '%s' "${rules}" | jq 'length')" + log "rebuilding ${count} ACL rule(s) on private gateway ${gateway_id} (${ls})" + + acl_clear "${gateway_id}" "${ls}" "cs-gateway-id" + + acl_add "${gateway_id}" from-lport "${ACL_PRIO_BASE}" "ip4.src == ${cidr}" drop \ + "${ls}" "cs-gateway-id" || emit_failure "failed to apply the default egress policy" + acl_add "${gateway_id}" to-lport "${ACL_PRIO_BASE}" "ip4.dst == ${cidr}" drop \ + "${ls}" "cs-gateway-id" || emit_failure "failed to apply the default ingress policy" + + while IFS=$'\037' read -r number action rtype proto pstart pend itype icode scidrs; do + [ -n "${rtype}" ] || continue + prio=$(( ACL_PRIO_VPC_TOP - number )) + [ "${prio}" -gt "${ACL_PRIO_BASE}" ] || prio=$(( ACL_PRIO_BASE + 1 )) + case "${action}" in + allow) ovn_action="allow-related" ;; + *) ovn_action="drop" ;; + esac + proto_match="$(acl_proto_match "${proto}" "${pstart}" "${pend}" "${itype}" "${icode}")" + + if [ "${rtype}" = "egress" ]; then + src_match="ip4.src == ${cidr}" + dst_match="" + if [ -n "${scidrs}" ] && [ "${scidrs}" != "0.0.0.0/0" ]; then + dst_match="ip4.dst == {${scidrs}}" + fi + match="$(match_join "${src_match}" "${dst_match}" "${proto_match}")" + acl_add "${gateway_id}" from-lport "${prio}" "${match}" "${ovn_action}" \ + "${ls}" "cs-gateway-id" \ + || log "warning: could not apply egress ACL rule ${number} (${match})" + else + src_match="" + if [ -n "${scidrs}" ] && [ "${scidrs}" != "0.0.0.0/0" ]; then + src_match="ip4.src == {${scidrs}}" + fi + dst_match="ip4.dst == ${cidr}" + match="$(match_join "${src_match}" "${dst_match}" "${proto_match}")" + acl_add "${gateway_id}" to-lport "${prio}" "${match}" "${ovn_action}" \ + "${ls}" "cs-gateway-id" \ + || log "warning: could not apply ingress ACL rule ${number} (${match})" + fi + done </dev/null || log "warning: --wait=sb sync timed out" + emit_success +} + +cmd_unsupported() { + # CloudStack should only dispatch the verbs backing the services this + # extension declares. Succeed loudly in the log rather than failing the + # operation, so an over-broad network offering does not wedge a deployment. + log "command '${COMMAND}' is not implemented by this extension — returning success" + emit_success +} + +# --------------------------------------------------------------------------- +# Dispatch +# +# Everything below mutates per-network or per-VPC singletons that the NB schema +# cannot keep unique for us, so the whole command runs under one lock. +# +# A VPC's tiers share one logical router, so VPC work locks on the VPC rather +# than the tier: two tiers coming up at once would otherwise both create +# cs-vpc-rtr-. A standalone network locks on itself. +# --------------------------------------------------------------------------- + +LOCK_KEY="" +if [ -n "${VPC_ID}" ]; then + LOCK_KEY="vpc-${VPC_ID}" +elif [ -n "$(arg network_id)" ]; then + LOCK_KEY="net-$(arg network_id)" +fi +if [ -n "${LOCK_KEY}" ]; then + trap 'lock_release 9; lock_release 8' EXIT + lock_acquire 9 "${LOCK_KEY}" +fi + +case "${COMMAND}" in + ensure-network-device) cmd_ensure_network_device ;; + implement-network) cmd_implement_network ;; + prepare-nic) cmd_prepare_nic ;; + release-nic) cmd_release_nic ;; + shutdown-network) cmd_shutdown_network ;; + destroy-network) cmd_destroy_network ;; + + # Declared-but-delegated DHCP verbs. prepare-nic already does the work and + # implement-network already created the subnet options, so these reconcile. + add-dhcp-entry) cmd_prepare_nic ;; + remove-dhcp-entry) emit_success ;; + config-dhcp-subnet) cmd_implement_network ;; + remove-dhcp-subnet) emit_success ;; + + # L3: NAT, port forwarding, firewall + assign-ip) cmd_assign_ip ;; + release-ip) cmd_release_ip ;; + add-static-nat) cmd_add_static_nat ;; + delete-static-nat) cmd_delete_static_nat ;; + add-port-forward) cmd_add_port_forward ;; + delete-port-forward) cmd_delete_port_forward ;; + apply-fw-rules) cmd_apply_fw_rules ;; + + # VPC and network ACLs + implement-vpc) cmd_implement_vpc ;; + shutdown-vpc) cmd_shutdown_vpc ;; + update-vpc-source-nat-ip) cmd_update_vpc_source_nat_ip ;; + apply-network-acl) cmd_apply_network_acl ;; + + # VPC private gateways and static routes. These require the companion patch to + # NetworkExtensionElement; stock CloudStack never dispatches them. + create-private-gateway) cmd_create_private_gateway ;; + delete-private-gateway) cmd_delete_private_gateway ;; + apply-static-routes) cmd_apply_static_routes ;; + apply-private-gw-acl) cmd_apply_private_gw_acl ;; + + *) cmd_unsupported ;; +esac diff --git a/extensions/OVN/test-with-ovn.sh b/extensions/OVN/test-with-ovn.sh new file mode 100755 index 000000000000..06408976c4c7 --- /dev/null +++ b/extensions/OVN/test-with-ovn.sh @@ -0,0 +1,555 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# --------------------------------------------------------------------------- +# Integration test for ovn.sh against a REAL OVN northbound database. +# +# It drives the script through a full isolated-network and VPC lifecycle, then +# asserts on the resulting NB objects and on ovn-northd's log -- the NB DB will +# happily store a configuration that northd cannot turn into logical flows, so +# checking northd is the part that actually proves anything. +# +# Nothing here needs CloudStack, a hypervisor or a VM; it tests the control +# plane only. See the README for what that does and does not cover. +# +# Usage, in a container or VM with ovn-central installed and running: +# +# /usr/share/ovn/scripts/ovn-ctl start_northd +# ./test-with-ovn.sh +# +# Environment: +# OVN_SCRIPT path to ovn.sh (default: alongside this script) +# OVN_NB_DB northbound db target (default: the local unix socket) +# OVN_NORTHD_LOG ovn-northd logfile (default: /var/log/ovn/ovn-northd.log) +# --------------------------------------------------------------------------- +# Drive ovn.sh against a real OVN northbound database. +# Every step asserts: exactly one JSON object on stdout, status=success, +# and the expected objects present in the NB DB. +SCRIPT="${OVN_SCRIPT:-$(cd "$(dirname "$0")" && pwd)/ovn.sh}" +DB="${OVN_NB_DB:-unix:/var/run/ovn/ovnnb_db.sock}" +NORTHD_LOG="${OVN_NORTHD_LOG:-/var/log/ovn/ovn-northd.log}" +NB="ovn-nbctl --db=$DB" + +# ovn-northd appends to one logfile for the life of the daemon, so a warning from +# an earlier run of this script is still sitting in it. Remember how long the log +# is now and only ever look at what this run adds. +NORTHD_BASELINE="$(wc -l < "${NORTHD_LOG}" 2>/dev/null || echo 0)" +PASS=0; FAIL=0; STEP=0 + +PHYS='"ovn_nb_db":"unix:/var/run/ovn/ovnnb_db.sock","ovn_public_physnet":"physnet1","ovn_gateway_chassis":"chassis-a,chassis-b","ovn_dns_servers":"8.8.8.8,8.8.4.4","ovn_mtu":"1442"' + +ok() { PASS=$((PASS+1)); printf ' \033[32mok\033[0m %s\n' "$1"; } +bad() { FAIL=$((FAIL+1)); printf ' \033[31mFAIL\033[0m %s\n' "$1"; } + +# run