BGP with Cilium

31/8/22 update: if you want to try it yourself, go and check out our new online lab “Getting Started with BGP on Cilium” here. Hope you like it!

I have recently been looking at containerized networking solutions, as you might have seen in my containerlab blog post,. I have been particularly focusing on running BGP with Cilium.

As I wrote on the official Isovalent 1.12 release, BGP is the standard networking protocol within data centers and it made absolute sense for Cilium to support BGP to 1) interconnect Cilium-managed clusters with an existing network and 2) remove the need for a 3rd party tool to achieve that.

When BGP was initially introduced in Cilium in 1.10, Cilium used under the hood MetalLB. By all accounts, it’s a great platform – it’s just that, as Cilium users started asking for IPv6 features, MetalLB was actually limited on that front. Meanwhile, GoBGP natively supported features such as BGP over IPv6 and it came clear that it was worth the effort with moving away from MetalLB to GoBGP over a couple of Cilium releases.

Enabling BGP is pretty straight-forward with the flag --enable-bgp-control-plane=true and then you just have to deploy a BGP CiliumBGPPeeringPolicy.

---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
  name: rack0
spec:
  nodeSelector:
    matchLabels:
      rack: rack0
  virtualRouters:
  - localASN: 65010
    exportPodCIDR: true
    neighbors:
    - peerAddress: "10.0.0.1/32"
      peerASN: 65010
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
  name: rack1
spec:
  nodeSelector:
    matchLabels:
      rack: rack1
  virtualRouters:
  - localASN: 65011
    exportPodCIDR: true
    neighbors:
    - peerAddress: "10.0.0.2/32"
      peerASN: 65011

Just specify:

• the remote peer IP and AS Number
• your own local AS Number

And you’ll be good to go!

You either advertise all the Pod CIDRs with exportPodCIDR: true or you don’t advertise any at all. At this moment in time, even if the Cilium virtual BGP routers learn some routes, it won’t do anything with these routes yet. It won’t modify the datapath and the egress traffic from Pods will be masqueraded and will follow the host’s routing table.

The purpose of BGP in the Cilium context is mainly to tell the rest of the networking world about the Pod CIDRs (and when supported on GoBGP, Service IP for Load Balancing.)

It’s very simple but I suspect some users will eventually want more functionality like the ability to set filtering and have the means to operate and troubleshoot. As it stands, you can do it on the other side of the BGP session.

I imagine it will come in due time but again, it depends on customer demands so raise it on the Cilium GitHub page if you have a specific requirement.

This video walks through in more details BGP on Cilium:

When I get the chance, I plan to follow up with a video on BGP with IPv6 and a longer blog post on isovalent.com.

Thanks for reading.