Deep Dive into Kubernetes Gateway API

What I find fascinating with Kubernetes is that, despite its popularity, it is always changing and evolving. Large-scale adoption brings more complex requirements and challenges. But bringing new technical features isn’t always the most difficult challenge – the complexity is in adapting Kubernetes to fit its operational model.

That’s why the Kubernetes Gateway APIs were created.


Ingress API vs Kubernetes Gateway API

One of the first tasks to complete, once you’ve deployed your applications on your Kubernetes clusters, is to provide access to these applications. This is typically known as the “North-South” access into your cluster (with “East-West” within pods in your cluster).

The Ingress API has been the standard way to address this requirement. Ingress API is the common way to route traffic from your external load-balancing into your Kubernetes services.

The problem with the Ingress API is that it didn’t turn out to 1) provide the advanced load-balancing requirements users wanted to define and 2) was impractical for users to manage.

Tech vendors have largely addressed the lack of functionality by using annotations in order to extend the platform. But annotations end up creating inconsistency.

So the Special Interest Group for K8S networking started to work towards a new API to provide additional functionalities (for more details, read here) and a different consumption model. It’s interesting that an API is defined based on the role of the users that may use it.

This short video explains well the need for a new API:

One of the benefits of these new APIs is that the Ingress API is essentially split into separate functions – one to describe the Gateway and one for the Routes to the back-end services. By splitting these two functions, it gives operators the ability to change and swap gateways but keep the same routing configuration (if I had to compare it with something I know well – it’s as if you swap out a Cisco router for a Juniper one while keeping the same configuration).

Another benefit is simply based on the operational model. Platform owners might be the ones owning the deployment and the type of Gateways while developers and operators might be the ones updating the routes to the services they want to expose. So you can apply different levels of role based access control to these components.

Finally, the Gateway API adds supports for more sophisticated LB features like weighted traffic splitting (as we will see in the lab shortly) and HTTP header-based matching and manipulation.

As the Gateway API came out with K8S v1.21, many vendors started updating their platforms to support not just the Ingress APIs but also the new Gateway APIs. That includes my employer: HashiCorp released an update to Consul to support these new specifications, it gives me an excuse to test it out. Let’s check it out in the lab.

Lab Time!

Let’s get our hands on with the Consul API Gateway. I’m going to go through the Learn@HashiCorp tutorial. The guides on HashiCorp Learn are seriously excellent but they can sometimes skip some basics so I am going to provide as much context as I can.

The steps are:

  1. Deploy a k8s with kind
  2. Deploy Consul and the Consul API Gateway
  3. Deploy the sample applications that leverages that gateway

Let’s start with cloning the repo:

% git clone https://github.com/hashicorp/learn-consul-kubernetes.git
Cloning into 'learn-consul-kubernetes'...
remote: Enumerating objects: 1648, done.
remote: Counting objects: 100% (1648/1648), done.
remote: Compressing objects: 100% (1051/1051), done.
remote: Total 1648 (delta 818), reused 1289 (delta 541), pack-reused 0
Receiving objects: 100% (1648/1648), 606.72 KiB | 4.04 MiB/s, done.
Resolving deltas: 100% (818/818), done.

% cd learn-consul-kubernetes 
% git checkout v0.0.12
Note: switching to 'v0.0.12'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 8df861b update README

% cd api-gateway/

Next, I’m going to use kind to build a Kubernetes cluster. It’s a change from EKS which I used in my eBPF posts. I surprisingly didn’t have kind yet on my laptop so I had to install it:

 % kind create cluster --config=kind/cluster.yaml
zsh: command not found: kind
% brew install kind
Running `brew update --preinstall`...
==> Auto-updated Homebrew!
Updated 2 taps (hashicorp/tap and homebrew/core).
==> New Formulae
alpscore               elixir-ls              iodine                 linux-headers@5.16     spago                  vermin
ascii2binary           elvis                  jless                  netmask                sshs                   vkectl
asyncapi               esphome                juliaup                numdiff                tagref                 webp-pixbuf-loader
atlas                  fb303                  kdoctor                odo-dev                tarlz                  websocketpp
bk                     fbthrift               kopia                  oh-my-posh             terminalimageviewer    weggli
boost@1.76             fdroidcl               kubescape              pinot                  terraform-lsp          xidel
brev                   ffmpeg@4               kyverno                reshape                textidote              yamale
canfigger              ghcup                  libadwaita             roapi                  tfschema               zk
cpptoml                http-prompt            librasterlite2         ruby@3.0               tidy-viewer
csview                 inotify-tools          linode-cli             rure                   usbutils
==> Updated Formulae
Updated 1513 formulae.
==> Renamed Formulae
annie -> lux
==> Deleted Formulae
carina           go@1.10          go@1.11          go@1.12          go@1.9           gr-osmosdr       hornetq          path-extractor

==> Downloading https://ghcr.io/v2/homebrew/core/kind/manifests/0.11.1
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/kind/blobs/sha256:116a1749c6aee8ad7282caf3a3d2616d11e6193c839c8797cde045cddd0e1138
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:116a1749c6aee8ad7282caf3a3d2616d11e6193c839c8797cde0
######################################################################## 100.0%
==> Pouring kind--0.11.1.big_sur.bottle.tar.gz
==> Caveats
zsh completions have been installed to:
  /usr/local/share/zsh/site-functions
==> Summary
๐Ÿบ  /usr/local/Cellar/kind/0.11.1: 8 files, 8.4MB

OK, I am ready to build my local k8s cluster:

% kind create cluster --config=kind/cluster.yaml
Creating cluster "kind" ...
 โœ“ Ensuring node image (kindest/node:v1.21.1) ๐Ÿ–ผ 
 โœ“ Preparing nodes ๐Ÿ“ฆ  
 โœ“ Writing configuration ๐Ÿ“œ 
 โœ“ Starting control-plane ๐Ÿ•น๏ธ 
 โœ“ Installing CNI ๐Ÿ”Œ 
 โœ“ Installing StorageClass ๐Ÿ’พ 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community ๐Ÿ™‚

That was easy (and cheaper than using EKS or GKE…). Now, let’s install Consul via Helm to automatically configure the Consul and Kubernetes integration to run within an existing Kubernetes cluster.

% helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" already exists with the same configuration, skipping
 % 
 % helm install --values consul/config.yaml consul hashicorp/consul --version "0.40.0"
Error: INSTALLATION FAILED: failed to download "hashicorp/consul" at version "0.40.0"

Unsurprisingly, I had already added the HashiCorp Helm chart (from doing a previous HashiCorp tutorial) but it was before the Consul API Gateway was live. I just had to do a Helm repo update to get started.

% helm repo update          
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
...Successfully got an update from the "cilium" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. โŽˆHappy Helming!โŽˆ

Right, let’s go and install Consul.

% helm install --values consul/config.yaml consul hashicorp/consul --version "0.40.0"
NAME: consul
LAST DEPLOYED: Tue Feb 22 15:56:56 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Consul!

Now that you have deployed Consul, you should look over the docs on using 
Consul with Kubernetes available here: 

https://www.consul.io/docs/platform/k8s/index.html


Your release is named consul.

To learn more about the release, run:

  $ helm status consul
  $ helm get all consul

We’re now ready to deploy the sample applications. HashiCup is one of the standard HashiCorp demo apps and leverages micro-services and Consul Service Mesh to connect them all together:

From https://learn.hashicorp.com/tutorials/consul/kubernetes-api-gateway

Let’s go!

% kubectl apply --filename two-services/
servicedefaults.consul.hashicorp.com/echo-1 created
service/echo-1 created
deployment.apps/echo-1 created
servicedefaults.consul.hashicorp.com/echo-2 created
service/echo-2 created
deployment.apps/echo-2 created
servicedefaults.consul.hashicorp.com/frontend created
service/frontend created
serviceaccount/frontend created
configmap/nginx-configmap created
deployment.apps/frontend created
serviceintentions.consul.hashicorp.com/public-api created
serviceintentions.consul.hashicorp.com/postgres created
serviceintentions.consul.hashicorp.com/payments created
serviceintentions.consul.hashicorp.com/product-api created
service/payments created
serviceaccount/payments created
servicedefaults.consul.hashicorp.com/payments created
deployment.apps/payments created
service/postgres created
servicedefaults.consul.hashicorp.com/postgres created
serviceaccount/postgres created
deployment.apps/postgres created
service/product-api created
serviceaccount/product-api created
servicedefaults.consul.hashicorp.com/product-api created
configmap/db-configmap created
deployment.apps/product-api created
service/public-api created
serviceaccount/public-api created
servicedefaults.consul.hashicorp.com/public-api created
deployment.apps/public-api created
clusterrolebinding.rbac.authorization.k8s.io/consul-api-gateway-tokenreview-binding created
clusterrole.rbac.authorization.k8s.io/consul-api-gateway-auth created
clusterrolebinding.rbac.authorization.k8s.io/consul-api-gateway-auth-binding created
clusterrolebinding.rbac.authorization.k8s.io/consul-auth-binding created

Everything’s working great:

% kubectl get pods
NAME                                             READY   STATUS    RESTARTS   AGE
consul-api-gateway-controller-5c46947749-8c24r   1/1     Running   1          95m
consul-client-rmjqs                              1/1     Running   0          95m
consul-connect-injector-5c6557976b-dvj9q         1/1     Running   0          95m
consul-connect-injector-5c6557976b-mmcnj         1/1     Running   0          95m
consul-controller-84b676b9bb-plntb               1/1     Running   0          95m
consul-server-0                                  1/1     Running   0          95m
consul-webhook-cert-manager-8595bff784-22mmf     1/1     Running   0          95m
echo-1-79b597d656-476nl                          2/2     Running   0          28m
echo-2-94b68d65b-z7rlz                           2/2     Running   0          28m
frontend-5c54674c4-tgwbk                         2/2     Running   0          28m
payments-77598ddb8f-bphzg                        2/2     Running   0          28m
postgres-8479965456-s8mhj                        2/2     Running   0          28m
product-api-dcf898744-zbfv9                      2/2     Running   1          28m
public-api-7f67d79fb6-hpdh5                      2/2     Running   0          28m

Let’s deploy our API Gateway:

% kubectl apply --filename api-gw/consul-api-gateway.yaml
gateway.gateway.networking.k8s.io/example-gateway created

Now, let’s take a look at this file we’ve just applied and let’s start digging into it:

---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  name: example-gateway
spec:
  gatewayClassName: consul-api-gateway
  listeners:
  - protocol: HTTPS
    port: 8443
    name: https
    allowedRoutes:
      namespaces:
        from: Same
    tls:
      certificateRefs:
        - name: consul-server-cert

The Gateway above (appropriately named example-gateway) is actually derived from a GatewayClass (deployed earlier with Helm):

% kubectl describe GatewayClass
Name:         consul-api-gateway
API Version:  gateway.networking.k8s.io/v1alpha2
Kind:         GatewayClass
Spec:
  Controller Name:  hashicorp.com/consul-api-gateway-controller
  Parameters Ref:
    Group:  api-gateway.consul.hashicorp.com
    Kind:   GatewayClassConfig
    Name:   consul-api-gateway
Status:
  Conditions:
    Last Transition Time:  2022-02-22T15:58:32Z
    Message:               Accepted
    Observed Generation:   1
    Reason:                Accepted
    Status:                True
    Type:                  Accepted
Events:                    <none>

The GatewayClass is a type of Gateway that can be deployed: in other words, it is a template. This is done in a way to let infrastructure providers offer different types of Gateways. Users can then choose the Gateway they like.

Here, the GatewayClass is based on the Consul API Gateway.

And when I create a Gateway, it will be based on the parameters specified in the GatewayClass resource specifications.

One of these parameters is the GatewayClassConfig. That’s used to provide some Consul-specific details, like the API Gateway and Envoy images. We’re using Custom Resource Definitions (CRD) for most of this configuration.

 % kubectl describe GatewayClassConfig
Name:         consul-api-gateway
Namespace:    
API Version:  api-gateway.consul.hashicorp.com/v1alpha1
Kind:         GatewayClassConfig
Spec:
  Consul:
    Authentication:
    Ports:
      Grpc:  8502
      Http:  8501
    Scheme:  https
  Copy Annotations:
  Image:
    Consul API Gateway:  hashicorp/consul-api-gateway:0.1.0-beta
    Envoy:               envoyproxy/envoy-alpine:v1.20.1
  Log Level:             info
  Service Type:          NodePort
  Use Host Ports:        true
Events:                  <none>

Now we understand a bit better the Gateway configuration: it was based on the GatewayClass (and its GatewayClassConfig for specific Consul parameters).

If we look back at the spec:

spec:
  gatewayClassName: consul-api-gateway
  listeners:
  - protocol: HTTPS
    port: 8443
    name: https
    allowedRoutes:
      namespaces:
        from: Same
    tls:
      certificateRefs:
        - name: consul-server-cert

It’s actually pretty clear: we are listening on port 8443 for HTTPS traffic (going back to the North/South requirement: that’s the traffic coming southbound into the cluster). The allowedRoutes is here to specify the namespaces from which Routes may be attached to this Listener. By default, it’s “Same” – ie we only use the namespace of the Gateway by default.

Interestingly, when we decide to use “All” instead of “Same” – to bind this gateway to routes in any namespace – it enables us to use a single gateway across multiple namespaces that may be managed by different teams. Again I would compare it my networking terms to MPLS or more specifically to VRF Lite.

It’s probably now a good time to look at these Routes – more precisely, these HTTPRoutes:

---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: example-route-1
spec:
  parentRefs:
  - name: example-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /echo
    backendRefs:
    - kind: Service
      name: echo-1
      port: 8080
      weight: 50
    - kind: Service
      name: echo-2
      port: 8090
      weight: 50
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: example-route-2
spec:
  parentRefs:
  - name: example-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - kind: Service
      name: frontend
      port: 80

With all the sophistication around Kubernetes, well in the end we’re back to the ways of creating static routes ๐Ÿ˜€

This is essentially what the spec above does: specify the routing but also some load-balancing requirements:

  • The routes are both attached to the “example-gateway” Gateway we built earlier.
  • They have Rules to describe which traffic we’re going to send where. The PathPrefix type means we’re matching based on the URL path.
    • If there’s “echo” in the path, we’ll send the traffic over to the echo services. Given the 50/50 weight, we’ll just load-balance the traffic between the two echo servers.
    • For everything else, we’ll send the traffic over to the frontend service.
  • When we actually go ahead and access the the load balancer over port 8443 (remember – that’s the port we are listening on), the traffic will be load-balanced between our two echo devices.
Load Balancing

When I change the weights to 99 and 1 to introduce canary deployments and re-apply the routes, almost all the traffic goes to the echo-1 service as I expect.

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: example-route-1
spec:
  parentRefs:
  - name: example-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /echo
    backendRefs:
    - kind: Service
      name: echo-1
      port: 8080
      weight: 99
    - kind: Service
      name: echo-2
      port: 8090
      weight: 1

Let’s check it works by running a loop with “while”:

% while true; do curl -s -k "https://localhost:8443/echo" >> curlresponses.txt ;done

% more curlresponses.txt| grep -c "Hostname: echo-1"
1179
% more curlresponses.txt| grep -c "Hostname: echo-2"
9

What’s also cool is that we could specify different namespaces in the HTTPRoutes – therefore, for example, you could send the traffic to https://acme.com/payments to a namespace where the payment app is deployed and https://acme.com/ads to a namespace used by the ads team for their application.

Once the traffic enters the cluster from North to South, it would then, in most cases, need to go East-West across the Service Mesh cluster. Obviously here, that’s using the Consul Service Mesh functionality.


That’s it! In summary, the Gateway API addresses some of the deficiencies in the Ingress APIs and it’s clear most vendors in the API Gateway/Service Mesh/Load-Balancing world will adapt their platform to this new model.

Thanks for reading.

Categories: K8S

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s