Support for VXLAN with BGP EVPN / MP-BGP

Bug Report

Version of OpenNebula

• 5.2.2
• 5.4.0
• 5.4.1
• 5.4.2
• 5.4.3
• 5.4.4
• 5.4.5
• 5.4.6
• 5.4.7
• 5.4.8
• 5.4.9
• 5.4.10
• 5.4.11
• Development build

Component

• Authorization (LDAP, x509 certs…)
• Command Line Interface (CLI)
• Contextualization
• Documentation
• Federation and HA
• Host, Clusters and Monitorization
• KVM
• Networking
• Orchestration (OpenNebula Flow)
• Packages
• Scheduler
• Storage & Images
• Sunstone
• Upgrades
• User, Groups, VDCs and ACL
• vCenter

Description

Expected Behavior

Actual Behavior

How to reproduce

Enhancement Request

Description

From my point of view a modern datacenter fabric has to scale and if based on VXLAN a control plane is one of the secrets to gain scalability and manageability for a resilent and performant multi-tenant datacenter and/or cloud network.

Therefore i did some PoC for adding VXLAN with BGP-EVPN support to OpenNebula. For sure, OpenNebula is the ideal platform to extend and integrate with SDN and other tools (pre,post,clean).

My PoC (now also running in a medium sized cloud environment / datacenter is based on

• VXLAN (RFC7348)
• BGP4 (RFC4271)
• BGP MPLS-Based Ethernet-VPN (RFC 7432)
• draft-ietf-bess-evpn-overlay-08
• MP-BGP (RFC4760)

The advantages over VXLAN with Multicast are:

• traffic engineering
• more scalability
• better manageability
• easier to handle than IP-Multicast in a large scale network

In my setup BGP EVPN and the whole VXLAN is based on classic Linux bridging (one Bridge per VNI) and Quagga (Cumulusnetworks) inside a Docker container on the VMM hosts.

On the (mostly redundant) OpenNebula frontends there are BGP Route Reflectors installed (also dockerized Quagga).

Example for the RR:

docker pull cumulusnetworks/quagga
docker run -t -d --net=host --privileged --name BGP-EVPN-RR1 cumulusnetworks/quagga:latest

The RR BGP config part:

router bgp 64512
  bgp router-id 172.24.201.8
  bgp cluster-id 172.24.201.8
  bgp log-neighbor-changes
  no bgp default ipv4-unicast
  neighbor onefabric peer-group
  neighbor onefabric remote-as 64512
  neighbor onefabric capability extended-nexthop
  neighbor onefabric update-source 172.24.201.8
  bgp listen range 172.24.201.0/24 peer-group onefabric
  !
  address-family evpn
   neighbor onefabric activate
   neighbor onefabric route-reflector-client
  exit-address-family
  !

The VTEP BGP config part:

router bgp 64512
 bgp router-id 172.24.201.102
 no bgp default ipv4-unicast
 neighbor onefabric peer-group
 neighbor onefabric remote-as 64512
 neighbor onefabric update-source vmnet0
 neighbor onefabric capability extended-nexthop
 neighbor 172.24.201.8 peer-group onefabric
 !
 address-family evpn
  neighbor onefabric activate
  advertise-all-vni
 exit-address-family
!

In the last step the bridges and the VXLAN interfaces has to be created:

ip link add vxlan${vni} type vxlan id ${vni} dstport 4789 local $ip-of-vtep nolearning
brctl addbr vxlan-br-${vni}
brctl addif vxlan-br-${vni} vxlan${vni}
brctl stp vxlan-br-${vni} off
ip link set up dev vxlan-br-${vni}
ip link set up dev vxlan${vni}

Now you`re able to use these bridges as standard “Bridged network” inside OpenNebula, and VXLAN with BGP-EVPN. Of course you had to take care about the MTU for the vNIC (1450 Bytes for example) cause of the VXLAN header and so on. But this works.

It would be really nice to integrate (and optimize) this kind of networking model into OpenNebula. For example to create the networks directly out of OpenNebula. Therefore new vnm drivers had to be created and the UI (Sunstone) has to be changed for the new values like VNI, bridge-name, VTEP-IP.

Use case

• scalable multi-tenancy datacenter and cloud-networks
• better manageability of large networks
• micro segmentation
• SDN
• Layer2 multi-tenancy

For the moment its just an idea. And if i were able to develop ill do it. If you need any more informations, access to my lab, etc. - let me know.

Interface Changes

Sunstone:

• Wizzard for the new network model
• VNM drivers

Progress Status

• Branch created
• Code committed to development branch
• Testing - QA
• Documentation (Add details on how to setup a basic BGP EVPN)
• Release notes - resolved issues, compatibility, known issues
• Code committed to upstream release/hotfix branches
• Documentation committed to upstream release/hotfix branches

[smangelkramer]

See a small blogpost from my (private, thus as pdf and not as link) cloud blog describing this topic more precise:

bgp_evpn_vxlan_opennebula.pdf

[hydro-b]

You can find the presentation here: https://www.youtube.com/watch?v=kpVrEYBFwZ0