K8s Going Deeper
In following posts, I started with my k8s cluster at home:
In this post I want to go a little deeper and try out some things on my k8s cluster. I don’t want it to just work, I also want to see it fail and do its thing do fix it, like throwing tons of requests on it with jmeter or bringing down entire nodes etc.
Autoscaling
HorizontalPodAutoscaler
This time I started reading the documentation1, which is not always what I do first… There one can read up that the HorizontalPodAutoscaler(short HPA) acts on following formula:
desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]
The HPA needs a metric-server, which may not be there if you set up your k8s cluster by yourself. I installed a metric server as follows:
git clone https://github.com/kubernetes-incubator/metrics-server.git
cd charts/metrics-server
There you find the helm chart for the metric server. The requirements for it are described in the README. At the follwoing point in the reqirements, I took the easy way:
Kubelet certificate needs to be signed by cluster Certificate Authority (or disable certificate validation by passing –kubelet-insecure-tls to Metrics Server)
In the values.yml I added the last point under defaultArgs:
defaultArgs:
- --cert-dir=/tmp
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=15s
- --kubelet-insecure-tls
Then install it with:
helm install metrics-server .
I will use a special spring-boot-app, which I quickly made for this purpose.
It only has a GET request with a parameter count. This requests performs
a for loop as big as the number count, and fills an ArrayList to also
fill up the memory.
You can find it here on docker hub.
With the metrics-server installed one can check the metrics with kubectl (here in a certain namespace). This can be useful for the next tests with auto-scaling enabled.
kubectl top pod -n comments
I changed my helm chart as following to enable autoscaling:
autoscaling:
enabled: true
minReplicas: 1
maxReplicas: 3
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
And one also needs to specify the resources the pods should request, otherwise the HPA does not know any limits…
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: 200m
memory: 1Gi
m stands for milli cores of the CPU, so 100m means 0.1 x 1 core.
With these settings the HPA should spawn new replicas as soon as it exceeds the targets.
First it did not work as I expected, so I had to calculate it myself with the above formula. I got the current metrics as follows:
kubectl top pod -n comments
NAME CPU(cores) MEMORY(bytes)
comments-test-77c9c54498-zr8ns 1m 132Mi
So I had one replica and want the system to spawn at least one more. That is what I get with the formula at the moment for memory:
1*(132/ (256*0.55)) = 0.9374999999999999
With these settings with a request of count=10000000,
I could easily provoke another replicas and after a certain time it scaled down again, prefect…
Cluster autoscaler
TODO, if node fails…. test it…
Next;
auto deployment
argocd, blue-green / canary deployment?
helm charts repo
simpler webserver irgendwie siehe doku.
Stateful Sets
For example: Galera Cluster, Sharded MongoDb or Kafka