K8s Going Deeper
In following posts, I started with my k8s cluster at home:
In this post I want to go a little deeper and try out some things on my k8s cluster. I don’t want it to just work, I also want to see it fail and do its thing do fix it, like throwing tons of requests on it with jmeter or bringing down entire nodes etc.
This time I started reading the documentation1, which is not always what I do first… There one can read up that the HorizontalPodAutoscaler(short HPA) acts on following formula:
desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]
The HPA needs a metric-server, which may not be there if you set up your k8s cluster by yourself. I installed a metric server as follows:
git clone https://github.com/kubernetes-incubator/metrics-server.git
There you find the helm chart for the metric server. The requirements for it are described in the README. At the follwoing point in the reqirements, I took the easy way:
Kubelet certificate needs to be signed by cluster Certificate Authority (or disable certificate validation by passing –kubelet-insecure-tls to Metrics Server)
values.yml I added the last point under defaultArgs:
Then install it with:
helm install metrics-server .
I will use a special spring-boot-app, which I quickly made for this purpose.
It only has a GET request with a parameter
count. This requests performs
a for loop as big as the number
count, and fills an ArrayList to also
fill up the memory.
You can find it here on docker hub.
With the metrics-server installed one can check the metrics with kubectl (here in a certain namespace). This can be useful for the next tests with auto-scaling enabled.
kubectl top pod -n comments
I changed my helm chart as following to enable autoscaling:
And one also needs to specify the resources the pods should request, otherwise the HPA does not know any limits…
m stands for milli cores of the CPU, so
0.1 x 1 core.
With these settings the HPA should spawn new replicas as soon as it exceeds the targets.
First it did not work as I expected, so I had to calculate it myself with the above formula. I got the current metrics as follows:
kubectl top pod -n comments
NAME CPU(cores) MEMORY(bytes)
comments-test-77c9c54498-zr8ns 1m 132Mi
So I had one replica and want the system to spawn at least one more. That is what I get with the formula at the moment for memory:
1*(132/ (256*0.55)) = 0.9374999999999999
With these settings with a request of
I could easily provoke another replicas and after a certain time it scaled down again, prefect…
TODO, if node fails…. test it…
argocd, blue-green / canary deployment?
helm charts repo
simpler webserver irgendwie siehe doku.
For example: Galera Cluster, Sharded MongoDb or Kafka