31/8/22 update: if you want to try it yourself, go and check out our new online lab “Getting Started with BGP on Cilium” here. Hope you like it!
I have recently been looking at containerized networking solutions, as you might have seen in my containerlab blog post,. I have been particularly focusing on running BGP with Cilium.
As I wrote on the official Isovalent 1.12 release, BGP is the standard networking protocol within data centers and it made absolute sense for Cilium to support BGP to 1) interconnect Cilium-managed clusters with an existing network and 2) remove the need for a 3rd party tool to achieve that.
When BGP was initially introduced in Cilium in 1.10, Cilium used under the hood MetalLB. By all accounts, it’s a great platform – it’s just that, as Cilium users started asking for IPv6 features, MetalLB was actually limited on that front. Meanwhile, GoBGP natively supported features such as BGP over IPv6 and it came clear that it was worth the effort with moving away from MetalLB to GoBGP over a couple of Cilium releases.
Enabling BGP is pretty straight-forward with the flag
--enable-bgp-control-plane=true and then you just have to deploy a BGP
--- apiVersion: "cilium.io/v2alpha1" kind: CiliumBGPPeeringPolicy metadata: name: rack0 spec: nodeSelector: matchLabels: rack: rack0 virtualRouters: - localASN: 65010 exportPodCIDR: true neighbors: - peerAddress: "10.0.0.1/32" peerASN: 65010 --- apiVersion: "cilium.io/v2alpha1" kind: CiliumBGPPeeringPolicy metadata: name: rack1 spec: nodeSelector: matchLabels: rack: rack1 virtualRouters: - localASN: 65011 exportPodCIDR: true neighbors: - peerAddress: "10.0.0.2/32" peerASN: 65011
- the remote peer IP and AS Number
- your own local AS Number
And you’ll be good to go!
You either advertise all the Pod CIDRs with
exportPodCIDR: true or you don’t advertise any at all. At this moment in time, even if the Cilium virtual BGP routers learn some routes, it won’t do anything with these routes yet. It won’t modify the datapath and the egress traffic from Pods will be masqueraded and will follow the host’s routing table.
The purpose of BGP in the Cilium context is mainly to tell the rest of the networking world about the Pod CIDRs (and when supported on GoBGP, Service IP for Load Balancing.)
It’s very simple but I suspect some users will eventually want more functionality like the ability to set filtering and have the means to operate and troubleshoot. As it stands, you can do it on the other side of the BGP session.
I imagine it will come in due time but again, it depends on customer demands so raise it on the Cilium GitHub page if you have a specific requirement.
This video walks through in more details BGP on Cilium:
When I get the chance, I plan to follow up with a video on BGP with IPv6 and a longer blog post on isovalent.com.
Thanks for reading.
6 thoughts on “BGP with Cilium”
Would you please share the kind config file and the containerlab config file here. thanks a lot.
Thanks for the feedback! I will post the kind config and the containerlab topology next week.
Great video, thx. We’re waiting for the kind & containerlab file to experiment by ourselves… 🙂
“Enabling BGP is pretty straight-forward with the flag –enable-bgp-control-plane=true” … I’m installed Cilium on AKS BYOCNI with the enable flag set to true. It however, refuses to set to true but remains as false. I’m try to get Ingress to announce the ExternalIP – but the ADDRESS is empty.
Hi Ike –
I haven’t tried BGP on top of AKS BYOCNI – to be honest, it’s more something we would typically see on-prem to announce a prefix to a typical top of rack network. It may work though – it’s weird that the flag remains false.
it’s a bit hard to help via WordPress Comments though- maybe hop over to the Cilium Slack channel instead?