K8s Going Deeper

In following posts, I started with my k8s cluster at home:

In this post I want to go a little deeper and try out some things on my k8s cluster. I don’t want it to just work, I also want to see it fail and do its thing do fix it, like throwing tons of requests on it with jmeter or bringing down entire nodes etc.



This time I started reading the documentation1, which is not always what I do first… There one can read up that the HorizontalPodAutoscaler(short HPA) acts on following formula:

desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]

The HPA needs a metric-server, which may not be there if you set up your k8s cluster by yourself. I installed a metric server as follows:

git clone https://github.com/kubernetes-incubator/metrics-server.git
cd charts/metrics-server

There you find the helm chart for the metric server. The requirements for it are described in the README. At the follwoing point in the reqirements, I took the easy way:

Kubelet certificate needs to be signed by cluster Certificate Authority (or disable certificate validation by passing –kubelet-insecure-tls to Metrics Server)

In the values.yml I added the last point under defaultArgs:

  - --cert-dir=/tmp
  - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
  - --kubelet-use-node-status-port
  - --metric-resolution=15s
  - --kubelet-insecure-tls

Then install it with:

helm install metrics-server .

I will use a special spring-boot-app, which I quickly made for this purpose. It only has a GET request with a parameter count. This requests performs a for loop as big as the number count, and fills an ArrayList to also fill up the memory.

You can find it here on docker hub.

With the metrics-server installed one can check the metrics with kubectl (here in a certain namespace). This can be useful for the next tests with auto-scaling enabled.

kubectl top pod -n comments

I changed my helm chart as following to enable autoscaling:

  enabled: true
  minReplicas: 1
  maxReplicas: 3
  targetCPUUtilizationPercentage: 80
  targetMemoryUtilizationPercentage: 80

And one also needs to specify the resources the pods should request, otherwise the HPA does not know any limits…

    cpu: 500m
    memory: 1Gi
    cpu: 200m
    memory: 1Gi

m stands for milli cores of the CPU, so 100m means 0.1 x 1 core.

With these settings the HPA should spawn new replicas as soon as it exceeds the targets.

First it did not work as I expected, so I had to calculate it myself with the above formula. I got the current metrics as follows:

kubectl top pod -n comments
NAME                                    CPU(cores)   MEMORY(bytes)
comments-test-77c9c54498-zr8ns          1m           132Mi

So I had one replica and want the system to spawn at least one more. That is what I get with the formula at the moment for memory:

1*(132/ (256*0.55)) = 0.9374999999999999

With these settings with a request of count=10000000, I could easily provoke another replicas and after a certain time it scaled down again, prefect…

Cluster autoscaler

TODO, if node fails…. test it…


auto deployment

argocd, blue-green / canary deployment?

helm charts repo

simpler webserver irgendwie siehe doku.

Stateful Sets

For example: Galera Cluster, Sharded MongoDb or Kafka