ovn控制器初探-526互联

一．OVN架构介绍

OVN由以下组件构成：

northbound database：存储逻辑交换机、路由器、ACL、端口等的信息，目前基于ovsdb-server。

ovn-northd: 集中式控制器，负责把northbound database数据分发到各个ovn-controller。

ovn-controller: 运行在每台机器上的本地SDN控制器

southbound database：基于ovsdb-server，包含三类数据

物理网络数据，比如VM的IP地址和隧道封装格式

逻辑网络数据，比如报文转发方式

物理网络和逻辑网络的绑定关系

一．实验拓扑

本次OVN实验研究的是二层转发，配置基本的二层拓扑，采用三台设备，全部用的vmware下的虚拟机模拟，没有部署CMS，通过命令配置的方式进行逻辑网络创建和虚拟机创建。采用一个中央控制节点，管理两台controller节点。环境基于VMware下安装的centos7系统。基础实验环境上centos系统上配置的接口地址如下：

注意：在HV节点配置的时候启动时需要做hostname的修改。

一．实验配置

OpenVswitch的安装

openvswitch采用源码编译的安装方式，三台设备上分别下载openvswitch2.9.2的源码进行编译和安装。

服务启用

2.1 启动中央控制节点

中央节点是总控节点，上面需要部署northd进程进行南向和北向两个数据库的数据守护和维护更新操作。

Northd启动：

export PATH=$PATH:/usr/local/share/openvswitch/scripts

ovn-ctl start_northd

中央控制节点上当ovn-ctl脚本执行start_northd会分别启动南北向数据库。

南北向数据库监听端口设置：

ovn-nbctl set-connection ptcp:6641

ovn-sbctl set-connection ptcp:6642

2.2 HV节点上的controller服务启动

HV1和HV2都执行如下命令启动ovn-controller进程。

export PATH=$PATH:/usr/local/share/openvswitch/scripts

ovs-ctl start --system-id=random

ovn-ctl start_controller

HV1建立与central节点数据库的连接，配置隧道封装类型域隧道封装地址

ovs-vsctl set open . external-ids:ovn-remote=tcp:192.168.29.171:6642

ovs-vsctl set open . external-ids:ovn-encap-type=geneve

ovs-vsctl set open . external-ids:ovn-encap-ip=192.168.29.136

HV2建立与central节点数据库的连接，配置隧道封装类型域隧道封装地址

ovs-vsctl set open . external-ids:ovn-remote=tcp:192.168.29.171:6642

ovs-vsctl set open . external-ids:ovn-encap-type=geneve

ovs-vsctl set open . external-ids:ovn-encap-ip=192.168.29.172

ovn-controller启动之后用netstat -atnp查看到南向数据库的建立是否是established状态。

3. 逻辑拓扑创建

逻辑拓扑是在中央控制节点上进行的，配置需要在central node上进行。

# 创建logical switch

ovn-nbctl ls-add ls1

# 创建 logical port

ovn-nbctl lsp-add ls1 ls1-vm1

ovn-nbctl lsp-set-addresses ls1-vm1 02:ac:10:ff:00:11

ovn-nbctl lsp-set-port-security ls1-vm1 02:ac:10:ff:00:11

# 创建 logical port

ovn-nbctl lsp-add ls1 ls1-vm2

ovn-nbctl lsp-set-addresses ls1-vm2 02:ac:10:ff:00:22

ovn-nbctl lsp-set-port-security ls1-vm2 02:ac:10:ff:00:22

4. 物理设备创建

HV上创建br-int

br-int的建立：

ovs-vsctl add-br br-int -- set Bridge br-int fail-mode=secure

ovs-vsctl list-br

本次实验采用的是创建linux namespace模拟虚拟机，VM1和VM2配置如下：

HV1节点上配置VM1

ip netns add vm1

ovs-vsctl add-port br-int vm1 -- set interface vm1 type=internal

ip link set vm1 netns vm1

ip netns exec vm1 ip link set vm1 address 02:ac:10:ff:00:11

ip netns exec vm1 ip addr add 172.16.255.11/24 dev vm1

ip netns exec vm1 ip link set vm1 up

#映射关系的建立

ovs-vsctl set Interface vm1 external_ids:iface-id=ls1-vm1

HV2节点配置VM2

ip netns add vm2

ovs-vsctl add-port br-int vm2 -- set interface vm2 type=internal

ip link set vm2 netns vm2

ip netns exec vm2 ip link set vm2 address 02:ac:10:ff:00:22

ip netns exec vm2 ip addr add 172.16.255.22/24 dev vm2

ip netns exec vm2 ip link set vm2 up

#映射关系的建立

ovs-vsctl set Interface vm2 external_ids:iface-id=ls1-vm2

四.实验结果分析

1 逻辑拓扑及数据库相关表信息

经过上述配置之后，逻辑网络创建，通过查询北向数据库看到逻辑网络的信息。

[root@bogon ~]# ovn-nbctl show

switch 42c57499-97ab-44ec-bd38-a84262897e84 (ls1)

port ls1-vm2

addresses: ["02:ac:10:ff:00:22"]

port ls1-vm1

addresses: ["02:ac:10:ff:00:11"]

[root@bogon ~]#

查询NB中逻辑设备，ports列有两个端口信息。

接口状态已经是UP的状态。这是根据SB中的端口绑定信息，由northd进程设置的，当VM创建之后，端口up并且与逻辑端口建立了映射关系之后，northd设置该列up。

关于接口up时如何监测的：

查看南向数据库信息，南向当前显示出有OVN-controller写入的HV1，HV2节点的信息，注意：在HV节点配置的时候启动ovs需要有system-id和hostname的修改。

[root@bogon ~]# ovn-sbctl show

Chassis "709a59d3-9e62-43d8-9e1f-91e7845ed55f"

hostname: "slave2.localdomain"

Encap geneve

ip: "192.168.29.172"

options: {csum="true"}

Port_Binding "ls1-vm2"

Chassis "2c2e87ff-eac1-4131-be2c-90b1de79e084"

hostname: "slave1.localdomain"

Encap geneve

ip: "192.168.29.136"

options: {csum="true"}

Port_Binding "ls1-vm1"

HV节点与SB建立连接之后查询数据库有以下数据被填写：

查询SB的端口绑定详细信息：

Chassis列已经填写，在HV上创建了VM之后，SB的port_binding相应的列会被ovn-controller设置。Tunnel-key字段就是逻辑接口的id，后面流表和报文封装的genneve中会使用到该字段。这个表建立了逻辑端口和VM上的物理端口的映射关系。

Logical switch也会有一个相应的datapathid，tunnel-key值，对应其datapath的id值。

2. 逻辑流表信息

流表分为ingress和egress，逻辑流表分为switch和router的流表：

查看ovs2.9 中ovn的逻辑流表定义：

router的流表：

3. 物理流表信息

流表中用到的几个ovs寄存器：

物理流表是HV节点上的controller进程根据逻辑流表进行的翻译。逻辑流表与物理流表由相应的映射关系。从源码可以看到这样的映射关系

Logical_Flow tables 0 through 23 become OpenFlow tables 8 through 31。

ovn-controller uses the first 32 bits of the logical flow’s UUID as the cookie for its OpenFlow flow or flows. (This is not necessarily unique, since the first 32 bits of a logical flow’s UUID is not necessarily unique.)

到这里看下真是的物理流表是什么样的，以VM1ping Vm2进行流表的分析：

[root@slave1 ~]# ovs-ofctl dump-flows br-int

//分析vm1 ping 远端vm2发起的流程

cookie=0x0, duration=17827.498s, table=0, n_packets=132, n_bytes=12292, priority=100,in_port="ovn-709a59-0" actions=move:NXM_NX_TUN_ID[0..23]->OXM_OF_METADATA[0..23],move:NXM_NX_TUN_METADATA0[16..30]->NXM_NX_REG14[0..14],move:NXM_NX_TUN_METADATA0[0..15]->NXM_NX_REG15[0..15],resubmit(,33)

//vm1 接口进入 datapath 为1，存入metadata ，入端口 logical inport 为1 存入reg14，跳转到8号表

cookie=0x0, duration=17589.121s, table=0, n_packets=132, n_bytes=12292, priority=100,in_port=vm1 actions=load:0x1->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],resubmit(,8)

cookie=0x5387d306, duration=17589.121s, table=8, n_packets=0, n_bytes=0, priority=100,metadata=0x1,vlan_tci=0x1000/0x1000 actions=drop

cookie=0x460de5fb, duration=17589.121s, table=8, n_packets=0, n_bytes=0, priority=100,metadata=0x1,dl_src=01:00:00:00:00:00/01:00:00:00:00:00 actions=drop

// 匹配源mac地址为02:ac:10:ff:00:11 跳转到9号表

cookie=0x6ce7f9b7, duration=17589.121s, table=8, n_packets=132, n_bytes=12292, priority=50,reg14=0x1,metadata=0x1,dl_src=02:ac:10:ff:00:11 actions=resubmit(,9)

判断metadata==1 跳转到10号表

cookie=0x3ee71401, duration=17589.121s, table=9, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,10)

metadata == 1 源mac = 02:ac:10:ff:00:11 arp报文跳转到11，icmp跳转到11号表

cookie=0x3397e37b, duration=17589.121s, table=10, n_packets=11, n_bytes=462, priority=90,arp,reg14=0x1,metadata=0x1,dl_src=02:ac:10:ff:00:11,arp_sha=02:ac:10:ff:00:11 actions=resubmit(,11)

cookie=0xafc1e1b, duration=17589.121s, table=10, n_packets=0, n_bytes=0, priority=90,icmp6,reg14=0x1,metadata=0x1,dl_src=02:ac:10:ff:00:11,nw_ttl=255,icmp_type=135,icmp_code=0,nd_sll=00:00:00:00:00:00 actions=resubmit(,11)

cookie=0xafc1e1b, duration=17589.121s, table=10, n_packets=0, n_bytes=0, priority=90,icmp6,reg14=0x1,metadata=0x1,dl_src=02:ac:10:ff:00:11,nw_ttl=255,icmp_type=136,icmp_code=0,nd_tll=02:ac:10:ff:00:11 actions=resubmit(,11)

cookie=0xafc1e1b, duration=17589.121s, table=10, n_packets=0, n_bytes=0, priority=90,icmp6,reg14=0x1,metadata=0x1,dl_src=02:ac:10:ff:00:11,nw_ttl=255,icmp_type=136,icmp_code=0,nd_tll=00:00:00:00:00:00 actions=resubmit(,11)

cookie=0x366f63d3, duration=17589.121s, table=10, n_packets=0, n_bytes=0, priority=80,arp,reg14=0x1,metadata=0x1 actions=drop

cookie=0x366f63d3, duration=17589.121s, table=10, n_packets=0, n_bytes=0, priority=80,icmp6,reg14=0x1,metadata=0x1,nw_ttl=255,icmp_type=136,icmp_code=0 actions=drop

cookie=0x366f63d3, duration=17589.121s, table=10, n_packets=0, n_bytes=0, priority=80,icmp6,reg14=0x1,metadata=0x1,nw_ttl=255,icmp_type=135,icmp_code=0 actions=drop

cookie=0x9d27e530, duration=17589.121s, table=10, n_packets=121, n_bytes=11830, priority=0,metadata=0x1 actions=resubmit(,11)

//继续跳转到12 ，13

cookie=0x998059e5, duration=17589.121s, table=11, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,12)

cookie=0x54ed9686, duration=17589.121s, table=12, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,13)

cookie=0x5542fd28, duration=17589.121s, table=13, n_packets=0, n_bytes=0, priority=100,ipv6,reg0=0x1/0x1,metadata=0x1 actions=ct(table=14,zone=NXM_NX_REG13[0..15])

//ipv4 匹配，跳转到14号表 ct是个什么action connect traction，无关紧要

cookie=0x5542fd28, duration=17589.121s, table=13, n_packets=0, n_bytes=0, priority=100,ip,reg0=0x1/0x1,metadata=0x1 actions=ct(table=14,zone=NXM_NX_REG13[0..15])

cookie=0xdbf913bf, duration=17589.121s, table=13, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,14)

cookie=0x56eda2d2, duration=17589.121s, table=14, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,15)

cookie=0x6118745b, duration=17589.121s, table=15, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,16)

cookie=0x25f7085f, duration=17589.121s, table=16, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,17)

cookie=0x5c24e7dc, duration=17589.121s, table=17, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,18)

跳转到18号表

cookie=0x89f7d1ba, duration=17589.121s, table=18, n_packets=0, n_bytes=0, priority=100,ip,reg0=0x4/0x4,metadata=0x1 actions=ct(table=19,zone=NXM_NX_REG13[0..15],nat)

cookie=0x89f7d1ba, duration=17589.121s, table=18, n_packets=0, n_bytes=0, priority=100,ipv6,reg0=0x4/0x4,metadata=0x1 actions=ct(table=19,zone=NXM_NX_REG13[0..15],nat)

cookie=0x6ff5a15c, duration=17589.121s, table=18, n_packets=0, n_bytes=0, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,19)

cookie=0x6ff5a15c, duration=17589.121s, table=18, n_packets=0, n_bytes=0, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,19)

cookie=0x3cbe87be, duration=17589.121s, table=18, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,19)

cookie=0x92a8e293, duration=17589.121s, table=19, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,20)

cookie=0x297b7816, duration=17589.121s, table=20, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,21)

cookie=0xe4907671, duration=17589.121s, table=21, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,22)

cookie=0xf1796bec, duration=17589.121s, table=22, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,23)

cookie=0x26fa6760, duration=17589.121s, table=23, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,24)

//24号表 dst mac匹配广播，arp报文的时候处理匹配到改表,广播报文，此时填reg15为 0xffff; 如果是icmp则匹配dl_dst=02:ac:10:ff:00:22，重新填下reg15 为2（出端口的logical port tunnel key) ,跳转到32 ，32号表作为remote转发表

cookie=0x395fddad, duration=17589.121s, table=24, n_packets=2, n_bytes=112, priority=100,metadata=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=load:0xffff->NXM_NX_REG15[],resubmit(,32)

cookie=0x73d4b044, duration=17589.121s, table=24, n_packets=130, n_bytes=12180, priority=50,metadata=0x1,dl_dst=02:ac:10:ff:00:22 actions=load:0x2->NXM_NX_REG15[],resubmit(,32)

cookie=0x15f7c302, duration=17589.121s, table=24, n_packets=0, n_bytes=0, priority=50,metadata=0x1,dl_dst=02:ac:10:ff:00:11 actions=load:0x1->NXM_NX_REG15[],resubmit(,32)

//跳转到32号表

cookie=0x0, duration=17827.498s, table=32, n_packets=0, n_bytes=0, priority=150,reg10=0x2/0x2 actions=resubmit(,33)

cookie=0x0, duration=17589.121s, table=32, n_packets=0, n_bytes=0, priority=150,reg10=0x10/0x10,reg15=0xffff,metadata=0x1 actions=resubmit(,33)

//arp广播报文匹配，隧道出接口为隧道出口转发广播报文，同时应该由报文复制另一支进入33号表，做本地广播

cookie=0x0, duration=17526.523s, table=32, n_packets=1, n_bytes=42, priority=100,reg15=0xffff,metadata=0x1 actions=load:0x1->NXM_NX_TUN_ID[0..23],set_field:0xffff->tun_metadata0,move:NXM_NX_REG14[0..14]->NXM_NX_TUN_METADATA0[16..30],output:"ovn-709a59-0",resubmit(,33)

//icmp单播报文匹配，reg15 保存的是逻辑出端口，action 保存datapath 1 到NXM_NX_TUN_ID，填写tunnel metadata字段，包含datapathid 出端口，reg14中保存到in port到写到tun metadata中，执行output操作，出端口为geneve的隧道出口

cookie=0x0, duration=17526.523s, table=32, n_packets=130, n_bytes=12180, priority=100,reg15=0x2,metadata=0x1 actions=load:0x1->NXM_NX_TUN_ID[0..23],set_field:0x2->tun_metadata0,move:NXM_NX_REG14[0..14]->NXM_NX_TUN_METADATA0[16..30],output:"ovn-709a59-0"

cookie=0x0, duration=17827.498s, table=32, n_packets=1, n_bytes=70, priority=0 actions=resubmit(,33)

cookie=0x0, duration=17589.121s, table=33, n_packets=131, n_bytes=12222, priority=100,reg15=0x1,metadata=0x1 actions=load:0x1->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],resubmit(,34)

cookie=0x0, duration=17589.121s, table=33, n_packets=3, n_bytes=182, priority=100,reg15=0xffff,metadata=0x1 actions=load:0x1->NXM_NX_REG13[],load:0x1->NXM_NX_REG15[],resubmit(,34),load:0xffff->NXM_NX_REG15[]

cookie=0x0, duration=17589.121s, table=34, n_packets=2, n_bytes=112, priority=100,reg10=0/0x1,reg14=0x1,reg15=0x1,metadata=0x1 actions=drop

cookie=0x0, duration=17827.498s, table=34, n_packets=132, n_bytes=12292, priority=0 actions=load:0->NXM_NX_REG0[],load:0->NXM_NX_REG1[],load:0->NXM_NX_REG2[],load:0->NXM_NX_REG3[],load:0->NXM_NX_REG4[],load:0->NXM_NX_REG5[],load:0->NXM_NX_REG6[],load:0->NXM_NX_REG7[],load:0->NXM_NX_REG8[],load:0->NXM_NX_REG9[],resubmit(,40)

cookie=0x22599f4, duration=17589.121s, table=40, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,41)

cookie=0x8cf78299, duration=17589.121s, table=41, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,42)

cookie=0xcb58095c, duration=17589.121s, table=42, n_packets=0, n_bytes=0, priority=100,ipv6,reg0=0x1/0x1,metadata=0x1 actions=ct(table=43,zone=NXM_NX_REG13[0..15])

cookie=0xcb58095c, duration=17589.121s, table=42, n_packets=0, n_bytes=0, priority=100,ip,reg0=0x1/0x1,metadata=0x1 actions=ct(table=43,zone=NXM_NX_REG13[0..15])

cookie=0xb73860b0, duration=17589.121s, table=42, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,43)

cookie=0x38bf2923, duration=17589.121s, table=43, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,44)

cookie=0x2eb77eb8, duration=17589.121s, table=44, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,45)

cookie=0x4011e263, duration=17589.121s, table=45, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,46)

cookie=0x35126859, duration=17589.121s, table=46, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,47)

提交到47号表

cookie=0x63225283, duration=17589.121s, table=47, n_packets=0, n_bytes=0, priority=100,ip,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)

cookie=0x63225283, duration=17589.121s, table=47, n_packets=0, n_bytes=0, priority=100,ipv6,reg0=0x2/0x2,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG13[0..15],exec(load:0->NXM_NX_CT_LABEL[0])),resubmit(,48)

cookie=0x62565c17, duration=17589.121s, table=47, n_packets=0, n_bytes=0, priority=100,ipv6,reg0=0x4/0x4,metadata=0x1 actions=ct(table=48,zone=NXM_NX_REG13[0..15],nat)

cookie=0x62565c17, duration=17589.121s, table=47, n_packets=0, n_bytes=0, priority=100,ip,reg0=0x4/0x4,metadata=0x1 actions=ct(table=48,zone=NXM_NX_REG13[0..15],nat)

cookie=0x205f2b29, duration=17589.121s, table=47, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,48)

cookie=0xf590f2f3, duration=17589.121s, table=48, n_packets=132, n_bytes=12292, priority=0,metadata=0x1 actions=resubmit(,49)

//匹配所有广播报文，提交到64号表

cookie=0x7540bdec, duration=17589.121s, table=49, n_packets=1, n_bytes=70, priority=100,metadata=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,64)

cookie=0x960b864f, duration=17589.121s, table=49, n_packets=131, n_bytes=12222, priority=50,reg15=0x1,metadata=0x1,dl_dst=02:ac:10:ff:00:11 actions=resubmit(,64)

cookie=0x0, duration=17589.121s, table=64, n_packets=0, n_bytes=0, priority=100,reg10=0x1/0x1,reg15=0x1,metadata=0x1 actions=push:NXM_OF_IN_PORT[],load:0->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]

cookie=0x0, duration=17827.499s, table=64, n_packets=132, n_bytes=12292, priority=0 actions=resubmit(,65)

cookie=0x0, duration=17589.121s, table=65, n_packets=132, n_bytes=12292, priority=100,reg15=0x1,metadata=0x1 actions=output:vm1

抓包分析，第一个arp报文时广播请求：