GKE Mixed TCP/UDP Loadbalancer

Recently I’ve been experimenting with Google Kubernetes Engine (GKE) and in particular the ingress from external to the GKE cluster. There are essentially 3 network options, fairly well described.

1. ClusterIP (service exposed internally to cluster)
2. NodePort (port opened and mapped on the node)
3. LoadBalancer (external IP created and forwarded to application. This can both be at a TCP/UDP level or at a layer 7 http/s level)

LoadBalancer is the probably the type you want if you want if you want external ingress into your application

I wanted to run a Go Ethereum (geth) client on GKE. For peer discovery and communication you want to an external IP that will send TCP 30303 and UDP 30303 to your application.

No worries, i’ve made a bunch of LoadBalancer types before. Something like this (adapted from my helm3 chart)

apiVersion: v1
kind: Service
metadata:
name: lit-lb
labels:
helm.sh/chart: lnd-lit-0.1.0
app.kubernetes.io/name: lnd-lit
app.kubernetes.io/instance: lit
app.kubernetes.io/version: "v0.3.1-alpha"
app.kubernetes.io/managed-by: Helm
spec:
type: LoadBalancer
selector:
app.kubernetes.io/name: lnd-lit
app.kubernetes.io/instance: lit
ports:
    – protocol: TCP
port: 443
loadBalancerSourceRanges:
– 129.205.33.85/32

You can have the last one for free (loadBalancerSourceRanges) – That actually does the GKE firewall which can limit which IP can access your ingress. (This can be super useful for an app which you want to access, but its not ideal for the rest of the world to access)

The above code speaks to GKE and created a Google Loadbalancer (you can see it in your console) and connect it to the service running on the node. What you will also probably notice is that this is only for TCP. I need both UDP and TCP. No worries, i’ll just add that in via another – protocol: UDP section under the ports. Right?

THE ISSUE

What happens is, you run into this “cannot create an external load balancer  with mix protocols” – essentially saying you cant (well, GKE) cant create a mixed type TCP/UDP loadbalancer. This upstream issue references it.

This causes a problem as geth expects both the TCP and UDP to arrive to it on 30303. It also seemed that you couldn’t change this to be different.

HOW WE FIXED IT

HIGHLIGHTS

1. Switch to using NodePort
2. Create a static IP on Google Network services
3. Use kubectl to create a 2x LoadBalancer’s and pass in the loadBalancerIP: “StaticIPhere” (one for UDP and one for TCP in my application)
4. Monitor in Google console and kubectl that the services are created and bound

IN DETAIL

1. Switch to using NodePort

The initial thinking was to switch from LoadBalancer to NodePort. This would listen on the node to any incoming traffic on 30303 TCP and UDP.

apiVersion: v1
kind: Service
metadata:
name: geth-np
labels:
helm.sh/chart: geth-0.1.0
app.kubernetes.io/name: geth
app.kubernetes.io/instance: geth
app.kubernetes.io/version: "v1.9.24"
app.kubernetes.io/managed-by: Helm
spec:
type: NodePort
selector:
app.kubernetes.io/name: geth
app.kubernetes.io/instance: geth
ports:
- name: geth-tcp
port: 30303
targetPort: 30303
protocol: TCP
nodePort: 30303
- name: geth-udp
port: 30303
targetPort: 30303
protocol: UDP
nodePort: 30303


This should be simple enough. This will open a node port from TCP 30303 to 30303 on the node. Confirm you have this with <kubectl get svc>

2. Create a Static IP on Google Network Services

Create a static IP. Put it in the same region as your GKE servers and attached to none

3. User Kubectl to Create Two Load Balancers

This is an important step as we couldnt get it working doing it via the gui. Kubectl does some further magic I haven’t quiet worked out yet. Be sure to replace your staticIP with one you reserved

apiVersion: v1
kind: Service
metadata:
name: geth-lb-tcp
spec:
selector:
app.kubernetes.io/instance: geth
app.kubernetes.io/name: geth
ports:
- port: 30303
targetPort: 30303
protocol: TCP
type: LoadBalancer
loadBalancerIP: "StaticIPHere"
---
apiVersion: v1
kind: Service
metadata:
name: geth-lb-udp
spec:
selector:
app.kubernetes.io/instance: geth
app.kubernetes.io/name: geth
ports:
- port: 30303
targetPort: 30303
protocol: UDP
type: LoadBalancer
loadBalancerIP: "StaticIPHere"

4. Monitor That it’s Working

geth-lb-tcp          LoadBalancer   192.168.65.47     StaticHere              30303:31232/TCP                   3d9h
geth-lb-udp          LoadBalancer   192.168.95.91    StaticHere              30303:31571/UDP                   3d10h

THAT SHOULD HOPEFULLY BE IT