1. neutron l3 ha 实现拓扑
参考: https://docs.openstack.org/networking-ovn/latest/admin/routing.html
关于BFD的两端，一端就是就是lrp，lrp可以在多个gw chassis node漂移，另一端可以是物理节点上的一个网卡|网桥（基于bfdd实现bfd的另一端）。
所以L3的HA，即基于ecmp（或bgp）的路由基于bfd的链路检测做路由变更或者路由决策调。那么HA包含router LRP的高可用以及LRP的对端（节点上的网桥|网卡）的高可用。
网桥会对外暴露主机，而ns内的网卡（internal port不会）。
关于neutron ovn实现的l3 HA的可靠解释全文：https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/networking_with_open_virtual_network/index#:~:text=to%20master.-,2.5.%20Layer%203%20high%20availability%20with%20OVN,-OVN%20supports%20Layer
2.5. Layer 3 high availability with OVN
OVN supports Layer 3 high availability (L3 HA) without any special configuration. OVN automatically schedules the router port to all available gateway nodes that can act as an L3 gateway on the specified external network. OVN L3 HA uses the gateway_chassis column in the OVN Logical_Router_Port table. Most functionality is managed by OpenFlow rules with bundled active_passive outputs. The ovn-controller handles the Address Resolution Protocol (ARP) responder and router enablement and disablement. Gratuitous ARPs for FIPs and router external addresses are also periodically sent by the ovn-controller.
ovn 不需要额外配置 即支持l3 lrp的HA，它会自动将lrp漂移到功能正常的节点。lrp 表中有一列用于记录lrp所在的gw chassis。这个功能是基于openflow规则结合捆绑的主备 output端口列表实现的。
ovn-controller 负责处理lrp是否响应arp。 以及fip的免费arp周期性的对外响应。
Note
L3HA uses OVN to balance the routers back to the original gateway nodes to avoid any nodes becoming a bottleneck.
L2 HA 使用ovn来平衡网关节点的公网流量。如果之前的网关节点挂了，然后又恢复了，流量会自动切回到原来的gw node。
BFD monitoring
OVN uses the Bidirectional Forwarding Detection (BFD) protocol to monitor the availability of the gateway nodes. This protocol is encapsulated on top of the Geneve tunnels established from node to node.
ovn 使用bfd协议来监控所有gw node的可用性。该协议封装与geneve隧道协议之上，在node之间互相检测。
Each gateway node monitors all the other gateway nodes in a star topology in the deployment. Gateway nodes also monitor the compute nodes to let the gateways enable and disable routing of packets and ARP responses and announcements.
在部署上，每个网关节点都 基于星型拓扑监控其他网关节点。同时网关节点也监控其他计算节点。 以便控制网关启用|禁用包的路由记忆arp的响应和宣告。
Each compute node uses BFD to monitor each gateway node and automatically steers external traffic, such as source and destination Network Address Translation (SNAT and DNAT), through the active gateway node for a given router. Compute nodes do not need to monitor other compute nodes.
每个计算节点也使用BFD来监控每一个网关节点并且自动化轮转到外部的流量，比如snat 和 dnat的流量。这些流量需要基于活跃网关节点的给定软路由的lrp才能转发出去。计算节点不监控其他计算节点。
Note
External network failures are not detected as would happen with an ML2-OVS configuration.
无法实现像ML2-OVS配置那样检测到外部网络的故障。
L3 HA for OVN supports the following failure modes:
- The gateway node becomes disconnected from the network (tunneling interface).
- ovs-vswitchd stops (ovs-switchd is responsible for BFD signaling)
- ovn-controller stops (ovn-controller removes itself as a registered node).
L3 HA 支持容忍以下故障模式:
- 网关节点断联（tunnel网卡故障）
- ovs-vswitchd 进程down了。 该服务负责BFD信号
- ovn-controller 进程down了。 该服务负责将其自身从注册node中移除（hypervisor gw chassis相关的资源记录）。
Note
This BFD monitoring mechanism only works for link failures, not for routing failures.
BFD机制仅能用于链路检测，而非路由故障。
1.1 分析neutron ovn的功能列表
github neutron\doc\source\admin\ovn\features.rst
Open Virtual Network (OVN) offers the following virtual network
services:
-
Layer-2 (switching)
Native implementation. Replaces the conventional Open vSwitch (OVS)
agent. -
Layer-3 (routing)
Native implementation that supports distributed routing. Replaces the
conventional Neutron L3 agent. This includes transparent L3HA :doc::routing
support, based on BFD monitorization integrated in core OVN.
可以看到neutron “透明L3 HA的实现”是基于 内置于ovn BFD 动力
-
DHCP
Native distributed implementation. Replaces the conventional Neutron DHCP
agent. Note that the native implementation does not yet support DNS
features. -
DPDK
OVN and the OVN mechanism driver may be used with OVS using either the Linux
kernel datapath or the DPDK datapath. -
Trunk driver
Uses OVN's functionality of parent port and port tagging to support trunk
service plugin. One has to enable the 'trunk' service plugin in neutron
configuration files to use this feature. -
VLAN tenant networks
The OVN driver does support VLAN tenant networks when used
with OVN version 2.11 (or higher). -
DNS
Native implementation. Since the version 2.8 OVN contains a built-in
DNS implementation. -
Port Forwarding
The OVN driver supports port forwarding as an extension of floating
IPs. Enable the 'port_forwarding' service plugin in neutron configuration
files to use this feature. -
Packet Logging
Packet logging service is designed as a Neutron plug-in that captures network
packets for relevant resources when the registered events occur. OVN supports
this feature based on security groups. -
Segments
Allows for Network segments ranges to be used with OVN. Requires OVN
version 20.06 or higher.
.. TODO What about tenant networks?
-
Routed provider networks
Allows for multiple localnet ports to be attached to a single Logical
Switch entry. This work also assumes that only a single localnet
port (of the same Logical Switch) is actually mapped to a given
hypervisor. Requires OVN version 20.06 or higher.
+----------------------------------+---------------------------------+
| Extension Name | Extension Alias |
+==================================+=================================+
| Allowed Address Pairs | allowed-address-pairs |
+----------------------------------+---------------------------------+
| Auto Allocated Topology Services | auto-allocated-topology |
+----------------------------------+---------------------------------+
| Availability Zone | availability_zone |
+----------------------------------+---------------------------------+
| Default Subnetpools | default-subnetpools |
+----------------------------------+---------------------------------+
| DNS Integration | dns-integration |
+----------------------------------+---------------------------------+
| DNS domain for ports | dns-domain-ports |
+----------------------------------+---------------------------------+
| DNS domain names with keywords | dns-integration-domain-keywords |
+----------------------------------+---------------------------------+
| Subnet DNS publish fixed IP | subnet-dns-publish-fixed-ip |
+----------------------------------+---------------------------------+
| Multi Provider Network | multi-provider |
+----------------------------------+---------------------------------+
| Network IP Availability | network-ip-availability |
+----------------------------------+---------------------------------+
| Network Segment | segment |
+----------------------------------+---------------------------------+
| Neutron external network | external-net |
+----------------------------------+---------------------------------+
| Neutron Extra DHCP opts | extra_dhcp_opt |
+----------------------------------+---------------------------------+
| Neutron Extra Route | extraroute |
+----------------------------------+---------------------------------+
| Neutron L3 external gateway | ext-gw-mode |
+----------------------------------+---------------------------------+
| Neutron L3 Router | router |
+----------------------------------+---------------------------------+
| Network MTU | net-mtu |
+----------------------------------+---------------------------------+
| Packet Logging | logging |
+----------------------------------+---------------------------------+
| Port Binding | binding |
+----------------------------------+---------------------------------+
| Port Bindings Extended | binding-extended |
+----------------------------------+---------------------------------+
| Port Forwarding | port_forwarding |
+----------------------------------+---------------------------------+
| Port MAC address Regenerate | port-mac-address-regenerate |
+----------------------------------+---------------------------------+
| Port Security | port-security |
+----------------------------------+---------------------------------+
| Provider Network | provider |
+----------------------------------+---------------------------------+
| Quality of Service | qos |
+----------------------------------+---------------------------------+
| Quota management support | quotas |
+----------------------------------+---------------------------------+
| RBAC Policies | rbac-policies |
+----------------------------------+---------------------------------+
| Resource revision numbers | standard-attr-revisions |
+----------------------------------+---------------------------------+
| security-group | security-group |
+----------------------------------+---------------------------------+
| standard-attr-description | standard-attr-description |
+----------------------------------+---------------------------------+
| Subnet Allocation | subnet_allocation |
+----------------------------------+---------------------------------+
| Subnet service types | subnet-service-types |
+----------------------------------+---------------------------------+
| Tag support | standard-attr-tag |
+----------------------------------+---------------------------------+
| Time Stamp Fields | standard-attr-timestamp |
+----------------------------------+---------------------------------+
2. 分析 neutron如何支持bfd
neutron 关于 支持bfd的提议记录这里(2021-04-27): https://blueprints.launchpad.net/neutron/+spec/bfd-support-for-neutron
最初的bug讨论： https://bugs.launchpad.net/neutron/+bug/1907089
大致描述是：
基于bfd来实现router间的链路故障。主要用于以下两个场景：
- 如果下一跳路由活跃|故障，则调整响应的路由。
- 基于路由协议ecmp或者bgp 基于bfd反馈的链路状态改变路由策略。
关于提议的主要部分：
添加关于bfd_monitors 资源对象相关的api接口
POST /v2.0/bfd_monitors
GET /v2.0/bfd_monitors
GET /v2.0/bfd_monitors/{monitor_uuid}
DELETE POST /v2.0/bfd_monitors/{monitor_uuid}
PUT /v2.0/bfd_monitors/{monitor_uuid}支持获取bfd-monitor的状态。
GET /v2.0/bfd_monitors/{monitor_uuid}/monitor_status修改router的api支持将bfd_monitor 绑定到 路由
PUT /v2.0/routers/{router_uuid}/add_extraroutes OR PUT /v2.0/routers/{router_id}
{"router" : {"routes" : [{ "destination" : "10.0.3.0/24", "nexthop" : "10.0.0.13" , "bfd": <bfd_monitor_id>}]}}
- 支持查看router的详情
GET /v2.0/routers/{router_id}/routes_status
BFD 不仅仅会给出监控信息，并且要求对链路状态做出快速响应。
在neutron的产经中，可能需要根据bfd的监控信息，来将“故障”路由从路由表中移除，并且在它恢复后将路由表加回（目前kube-ovn的ecmp多nexthop设计基本类似）。
其他的场景中够可以基于交换|路由机制来实现更为成熟的解决方案。
简单的后端开源实现可以依赖于ovn，ovs可以负责实现BFD monitor。
关于上面的关键词解释 bfd_monitor 大概可以解释为对应的是bfd rfc的中的bfd会话，对应的是ovn nbdb中的bfd 表，而neutron采用ovn方案，应该是通过ovn来直接对接bfd的资源需求。
小结一下： neutron想搞一套面向bfd 会话资源维护的接口。这个接口后面的bfd资源可以为l3 ha对应的路由变更和决策提供链路检测。
当然有了这些接口，也可以在外部实现一些监控自动化为维护工具。
neutron 实现BFD
https://opendev.org/openstack/neutron-specs/commit/e35a6606f093cd87f72becabbdfcba0729187d18
具体代码在参考部分
References
==========
.. [1] https://tools.ietf.org/html/rfc5880
.. [2] https://tools.ietf.org/html/rfc5880#section-6.1
.. [3] https://tools.ietf.org/html/rfc5880#section-6.8.1
.. [4] https://tools.ietf.org/html/draft-wang-bfd-one-arm-use-case-00
.. [5] https://tools.ietf.org/html/rfc5880#section-6.7
.. [6] https://review.opendev.org/c/openstack/neutron-lib/+/778859
.. [7] https://review.opendev.org/c/openstack/neutron-specs/+/767337/7/specs/wallaby/bfd_support.rst#377
在序号7的参考文档中: 有关于lsp ovs支持bfd的直接示例
OVS is able to handle BFD, see [3]_, it's possible to set BFD on an
interface, and check the status of it.
The manual process is something like this:
.. code-block:: bash
$ sudo ovs-vsctl set interface ens5 bfd:bfd_src_ip=192.168.122.237 bfd:bfd_dst_ip=192.168.122.57 bfd:enable=true
$ #.... Do the same on the remote end ....
$ sudo ovs-vsctl get interface ens5 bfd_status
{diagnostic="Control Detection Time Expired", flap_count="7", forwarding="true", remote_diagnostic="Neighbor Signaled Session Down", remote_state=up, state=up}
# 包括最关键的如何正确使用的部分:
该文档不是简要完整分析，分析过程在kube-ovn bfd相关pr中附注
