tobi exploring, learning and having fun

Reverse Tunneling

2026-01-06T00:00:00+01:00

I previously kept my backup NAS in the same location as my primary NAS, connected to the same power grid and without a UPS. I have now been offered a new location for the backup NAS and plan to move it there to introduce geographic redundancy.

The idea is to have the backup NAS operate in pull mode. It will connect as a VPN client and replicate datasets incrementally using ZFS. In addition, I plan to deploy an old Raspberry Pi at the new location. The Pi will run 24/7 and wake the backup NAS once a week so it can start the replication process.

I do not want to set up a site-to-site VPN tunnel, as I prefer not to modify the router at the new location. Instead, I plan to keep a persistent reverse tunnel from the Raspberry Pi to a jump host in my homelab for maintenance purposes. This setup will allow me to connect to the Pi via SSH, wake the backup NAS, and access the backup NAS web interface. The Raspberry Pi hardware should be more than sufficient for these lightweight maintenance tasks, while the VPN handling the heavy data transfer will run on the much more powerful backup NAS hardware.

There are certainly other ways to achieve this. The reasons I chose this architecture are:

It relies on tools I already know and have configured
The tools are available on almost any Linux system
I have long wanted to experiment with reverse tunnels, both out of curiosity and to better understand malicious intrusions and data exfiltration techniques

If you can think of a better approach, feel free to let me know. SSH tunneling is certainly not the most performant solution, especially on a Raspberry Pi.

The following diagram visualizes a simple sample setup and introduces the names that I will reuse later.

Preparing the Pi and the Reverse Jump

Requirements:

The system should be highly isolated, as it needs to be exposed to the internet.
The SSH server running on it must be hardened.
The local firewall must be enabled.

To meet these requirements, I placed the system in my DMZ VLAN and added a very restrictive firewall rule set. By default, it is not allowed to connect to anything: not hosts in the same subnet, not other internal subnets, and not the internet, except the Debian repos to update.

In addition, I hardened the SSH server on both the Raspberry Pi and the Reverse Jump as follows:

AllowUsers user-reverse-jump user-pi
PermitRootLogin no
PasswordAuthentication no
VersionAddendum none

For testing, I initiated the reverse tunnel on the Raspberry Pi using: ssh -p 231 -R 2222:localhost:22 user-reverse-jump@reverse-jump. I then connected to it from the Reverse Jump host with: ssh -p 2222 user-pi@localhost.

Tunnel http traffic through

To access the web interface of my backup NAS, I also need to proxy web requests from the Reverse Jump through the tunnel, effectively allowing me to browse as if I were on the Raspberry Pi. To achieve this, I start a SOCKS¹ v5 proxy on the Pi and expose it through the tunnel.

What I did to make this work:

Started the SOCKS proxy on the Pi: ssh -N -D 1080 localhost.
Exposed the proxy through the tunnel by running the following on the Pi: ssh -p 231 -R 1080:localhost:1080 toob@10.0.0.220.
On the Reverse Jump, I run a lightweight desktop environment with Firefox configured to use the SOCKS proxy.

Finalizing the Setup

Raspberry Pi

On the Raspberry Pi side, I needed to make everything persistent and ensure it starts automatically on boot. It also needed to reconnect automatically in case of network outages. While looking for a solution, I came across autossh, which describes itself in its man page as follows:

autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic.

autossh -M 0 -N -p 231 \
-R 2222:localhost:22 \
-R 1080:localhost:1080 \
user-reverse-jump@reverse-jump

Using systemd units, I configured everything to start at boot. This makes the setup easy to restart, redeploy, or move to a new location, and it ensures that I always regain a reverse shell from that network as long as outbound internet access is permitted.

/etc/systemd/system/autossh-reverse.service

[Unit]
Description=AutoSSH reverse SSH + reverse SOCKS
After=network-online.target ssh-socks.service
Wants=network-online.target ssh-socks.service

[Service]
User=YOUR_LOCAL_USERNAME
Environment="AUTOSSH_GATETIME=0"
ExecStart=/usr/bin/autossh \
  -M 0 -N \
  -o ServerAliveInterval=30 \
  -o ServerAliveCountMax=3 \
  -o ExitOnForwardFailure=yes \
  -p 231 \
  -R 2222:localhost:22 \
  -R 1080:localhost:1080 \
  toob@10.0.9.102
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

/etc/systemd/system/ssh-socks.service

[Unit]
Description=Local SSH SOCKS proxy
After=network.target

[Service]
User=YOUR_LOCAL_USERNAME
ExecStart=/usr/bin/ssh -N -D 127.0.0.1:1080 localhost
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Using the Reverse Jump as a Proxy

To allow other machines on the same network as the Reverse Jump to connect to the tunnel (SSH and SOCKS), I needed to make a few small adjustments.

On the Reverse Jump, I updated the SSH daemon configuration as follows:

AllowTcpForwarding yes
GatewayPorts yes

A SOCKS server is a proxy that forwards network traffic at the transport layer, allowing applications to route their connections through another host without being aware of the underlying network topology. Its main benefits are flexibility and protocol independence, making it useful for securely accessing services across restricted or segmented networks. ↩

Autopsy On Debian

2025-09-18T00:00:00+02:00

I am happily running Debian on my personal laptop and want to run the newest Autopsy version on it with its nice new GUI. Unfortunately there is only a nice Windows installer and a Snap package. Even though I am no fan of Canonical’s Snap’s I quickly tried it, but it did not work. So I decided to build and install it myself. I did this before the release of Debian 13 (Trixie), but also after the upgrade to Trixie I did it one more time. Because the dependency SleuthKit has a dependancy to a Debian Java 17 package it got slightly more complex.

Clone the source code: https://github.com/sleuthkit/autopsy.git or download the bundled release you like; https://github.com/sleuthkit/autopsy/releases/tag/autopsy-4.22.1
Run the script: unix_setup.sh -j /opt/java17 and do what it reports…, which is:
1. Install or provide java 17. In Debian 12 the default java version was 17, but in 13 it is Java 21, that is why I installed Java 21 manually. See here; https://adoptium.net/temurin/releases/?variant=openjdk17&version=17&os=any&arch=any
2. Download the sleuthkit java library, because it is not in the debian repos. One will only find the sleuthkit in the debian repos, which is the c++ CLI tool, but not what autopsy needs… I did it with there provided .deb package here: https://github.com/sleuthkit/sleuthkit/releases/download/sleuthkit-4.14.0/sleuthkit-java_4.14.0-1_amd64.deb
3. Install it; sudo apt install /opt/sleuthkit/sleuthkit-java_4.14.0-1_amd64.deb. Unfortunately following occured:
```
 sudo apt install /opt/sleuthkit/sleuthkit-java_4.14.0-1_amd64.deb 

 Error! Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: Unsatisfied dependencies: sleuthkit-java : Depends: openjdk-17-jre but it is not installable Error: Unable to correct problems, you have held broken packages. Error: The following information from --solver 3.0 may provide additional context: 
 Unable to satisfy dependencies. Reached two conflicting decisions: 
 1. sleuthkit-java:amd64=4.14.0-1 is selected for install 
 2. sleuthkit-java:amd64 Depends openjdk-17-jre but none of the choices are installable: [no choices]
```
4. I already installed a java 17 version, so we just need the install to move on. For this problem i used the equivs tool to mimic a Debian package (maybe there is a cleaner solution?):
  1. Install and generate the build file:
    sudo apt install equivs equivs-control openjdk17-jre
  2. Modify the file openjdk17.jre as follows:
    Package: openjdk-17-jre Version: 17.0 Provides: openjdk-17-jre Description: Dummy package to satisfy sleuthkit-java dep This is a fake package because I already have JDK 17 installed manually.
  3. Install it equivs-build openjdk17-jre && sudo apt install ./openjdk-17-jre_17.0_all.deb
5. After this the install worked, see 3.
Rerun unix_setup.sh -j /opt/java17
Run Autopsy (from the directory you cloned unpacked); bin/autopsy --jdkhome /opt/java/jdk-17.0.12+7

That should be it.

Podman Reloaded

2025-01-27T00:00:00+01:00

In an earlier post, I wrote about Podman. See also Podman. Now, I wanted to migrate some applications to Podman and received a deprecation warning with the command: podman generate systemd --new --files --name test. It is suggested to adapt to Quadlet, which provides a nicer workflow. So I did, but it took some reading to figure it out, which is why I’m summarizing it here.

This change offers a clean and elegant solution, seamlessly integrating Podman with systemd. While it took me some time as a user to get everything running smoothly again, the effort was absolutely worth it.

This is another blog I found helpful: https://mo8it.com/blog/quadlet/.

Quadlet

Quadlet is integrated into Podman and allows users to create systemd services in a declarative way. You can find more details here:
https://www.redhat.com/en/blog/quadlet-podman.

I placed the quadlet files in .config/containers/systemd/. And a simple app can look like that:

# some-app.container
[Container]
AutoUpdate=registry
ContainerName=some-app
Environment=TZ=Europe/Zurich
Image=lscr.io/linuxserver/some-app:latest
Pod=some-app-stack.pod
PodmanArgs=--tty
Volume=some-app_data:/config

[Service]
Restart=always

[Install]
WantedBy=default.target

With following command one can verify how the generated files look like:

/usr/libexec/podman/quadlet -dryrun -user

Output:

quadlet-generator[4645]: Loading source unit file /home/toob/.config/containers/systemd/some-app.container
---some-app.service---
# some-app.container
[X-Container]
AutoUpdate=registry
ContainerName=some-app
Environment=TZ=Europe/Zurich
Image=lscr.io/linuxserver/some-app:latest
Pod=some-app-stack.pod
PodmanArgs=--tty
Volume=some-app_data:/config

[Service]
Restart=always
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStop=/usr/bin/podman rm -v -f -i --cidfile=%t/%N.cid
ExecStopPost=-/usr/bin/podman rm -v -f -i --cidfile=%t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name some-app --cidfile=%t/%N.cid --replace --rm --cgroups=split --sdnotify=conmon -d -v some-app_data:/config --label io.containers.autoupdate=registry --env TZ=Europe/Zurich --pod-id-file %t/some-app-stack-pod.pod-id --tty lscr.io/linuxserver/some-app:latest

[Install]
WantedBy=default.target

[Unit]
Wants=podman-user-wait-network-online.service
After=podman-user-wait-network-online.service
SourcePath=/home/toob/.config/containers/systemd/some-app.container
RequiresMountsFor=%t/containers
BindsTo=some-app-stack-pod.service
After=some-app-stack-pod.service

Podlet

Podlet helps to generate such files. One can find it here on github; https://github.com/containers/podlet?tab=readme-ov-file.

It supports many different modes, including generating from already running resources, which is the closest replacement for the old podman generate command. However, it also supports interesting features like generating from Docker Compose files.

Here’s an example of how I used it to generate a Quadlet file from an existing pod:

podlet generate pod some-app-stack

I think the functionality of the generate option hasn’t received much attention. It only supports a small subset of creation options and may fail. In my case, I had to recreate almost everything manually.

Troubleshooting

Initially, I encountered problems with the AutoUpdate=registry setting for my Grafana instance. Since I want it to auto-update for easier maintenance, I included this argument in my quadlet file. However, with this setting, Grafana failed to start. When I removed the argument, it started without any issues.

Upon checking the service logs, I noticed a warning about the image reference, indicating that it should be a fully qualified domain name (FQDN).

To resolve this, I updated my image reference from Image=grafana/grafana:latest to Image=docker.io/grafana/grafana:latest. After this change, everything worked as expected. In hindsight, this makes sense because Podman does not assume Docker Hub as the default registry. Still, it was a bit of a headache to figure out.

Auto-Updates and Rollbacks

The auto-updates and rollbacks continue to function as before. You can refer to my previous post for more details.

Minecraft Server

2024-03-23T00:00:00+01:00

Hosting a Minecraft Server

I am a fan of Debian, so that is the base of the server.

Fabric loader

I was following the official fabric documentation:

Install curl
Install java 17 apt install openjdk-17-jre-headless
Choose version you want…
Download curl -OJ https://meta.fabricmc.net/v2/versions/loader/1.20.4/0.15.7/1.0.0/server/jar
run the jar once, so that all necessary files and folders get generated
change eula.txt, server.properties
copy fabric API jar and mods in place

Make it a service

With screen you can start and interact with the server. For more information see here.

Service File

[Unit]
Description=Minecraft Server: %i
After=network.target

[Service]
WorkingDirectory=/opt/%i

User=minecraft
Group=minecraft

Restart=always

ExecStart=/usr/bin/screen -DmS mc-%i /usr/bin/java -Xmx6G -jar fabric-server-mc.1.20.4-loader.0.15.7-launcher.1.0.0.jar nogui

ExecStop=/usr/bin/screen -p 0 -S mc-%i -X eval 'stuff "say SERVER SHUTTING DOWN IN 20 SECONDS."\015'
ExecStop=/bin/sleep 1
ExecStop=/usr/bin/screen -p 0 -S mc-%i -X eval 'stuff "save-all"\015'
ExecStop=/usr/bin/screen -p 0 -S mc-%i -X eval 'stuff "say  SAVED EVERYTHING, STOP IN 10 SECONDS..."\015'
ExecStop=/bin/sleep 10
ExecStop=/usr/bin/screen -p 0 -S mc-%i -X eval 'stuff "stop"\015'


[Install]
WantedBy=multi-user.target

Enable the service: systemctl enable minecraft@hermitcraft. Start the service: systemctl stop minecraft@hermitcraft.

See defined WorkingDirectory in service unit. in this working directory i have servers in folder like hermitcraft. Like that i have to write only one service file.

Make sure everything gets saved

One could propably achieve this via systemd as well, but I found it easier like that…

Place a shutdown script in /etc/init.d/ which takes care of saving the world before a shutdown or a reboot. It only needs to stop the service and give it some time to do its job.

#! /bin/sh

systemctl stop minecraft@hermitcraft

# Wait for the service to be stopped
while systemctl is-active --quiet "$service_name"; do
     sleep 5
     done

exit 0

Create soft links to the proper runlevel (for shutdown and reboot, 0 and 6); /etc/rc0.d/, /etc/rc6.d/

Backup

A minecraft server by default saves all maps to disc every 5 minutes. See answer from some AI:

In Minecraft, the save-all command is used to manually trigger the server to save the current state of the world to disk. However, by default, Minecraft servers also have an auto-save feature enabled, which automatically saves the world at regular intervals. The frequency of these auto-saves can be configured in the server settings. By default, Minecraft servers save the world every 6000 ticks, which is approximately every 5 minutes. This auto-save feature helps prevent data loss in case of unexpected server crashes or shutdowns. However, server administrators may still use the save-all command manually, especially before making significant changes to the world or performing maintenance tasks.

This means that i can take zfs snapshots of the entire lxc and handle it with the hypervisor.

K8s Going Deeper

2023-10-28T00:00:00+02:00

In following posts, I started with my k8s cluster at home:

In this post I want to go a little deeper and try out some things on my k8s cluster. I don’t want it to just work, I also want to see it fail and do its thing do fix it, like throwing tons of requests on it with jmeter or bringing down entire nodes etc.

Autoscaling

HorizontalPodAutoscaler

This time I started reading the documentation¹, which is not always what I do first… There one can read up that the HorizontalPodAutoscaler(short HPA) acts on following formula:

desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]

The HPA needs a metric-server, which may not be there if you set up your k8s cluster by yourself. I installed a metric server as follows:

git clone https://github.com/kubernetes-incubator/metrics-server.git
cd charts/metrics-server

There you find the helm chart for the metric server. The requirements for it are described in the README. At the follwoing point in the reqirements, I took the easy way:

Kubelet certificate needs to be signed by cluster Certificate Authority (or disable certificate validation by passing –kubelet-insecure-tls to Metrics Server)

In the values.yml I added the last point under defaultArgs:

defaultArgs:
  - --cert-dir=/tmp
  - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
  - --kubelet-use-node-status-port
  - --metric-resolution=15s
  - --kubelet-insecure-tls

Then install it with:

helm install metrics-server .

I will use a special spring-boot-app, which I quickly made for this purpose. It only has a GET request with a parameter count. This requests performs a for loop as big as the number count, and fills an ArrayList to also fill up the memory.

You can find it here on docker hub.

With the metrics-server installed one can check the metrics with kubectl (here in a certain namespace). This can be useful for the next tests with auto-scaling enabled.

kubectl top pod -n comments

I changed my helm chart as following to enable autoscaling:

autoscaling:
  enabled: true
  minReplicas: 1
  maxReplicas: 3
  targetCPUUtilizationPercentage: 80
  targetMemoryUtilizationPercentage: 80

And one also needs to specify the resources the pods should request, otherwise the HPA does not know any limits…

resources:
  requests:
    cpu: 500m
    memory: 1Gi
  limits:
    cpu: 200m
    memory: 1Gi

m stands for milli cores of the CPU, so 100m means 0.1 x 1 core.

With these settings the HPA should spawn new replicas as soon as it exceeds the targets.

First it did not work as I expected, so I had to calculate it myself with the above formula. I got the current metrics as follows:

kubectl top pod -n comments
NAME                                    CPU(cores)   MEMORY(bytes)
comments-test-77c9c54498-zr8ns          1m           132Mi

So I had one replica and want the system to spawn at least one more. That is what I get with the formula at the moment for memory:

1*(132/ (256*0.55)) = 0.9374999999999999

With these settings with a request of count=10000000, I could easily provoke another replicas and after a certain time it scaled down again, prefect…

Cluster autoscaler

TODO, if node fails…. test it…

Next;

auto deployment

argocd, blue-green / canary deployment?

helm charts repo

simpler webserver irgendwie siehe doku.

Stateful Sets

For example: Galera Cluster, Sharded MongoDb or Kafka

k8s autoscale: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ ↩

Home Automation

2023-09-10T00:00:00+02:00

Approximately a year ago I started to use Node-RED a little, see following post; Home Energy.

Recently I found a new use case. I am still a little old school and rather watch “Live TV” over dbv-c. I prefer it that way, because it is almost free, and I can still schedule recordings etc. In Switzerland, it is also in discussion to make ads not shippable anymore in the internet tc offerings, which is one more reason to record the shows I want to watch myself.

I am running a “TV-Headend Server”(handling my dvb-c connection) on my Home Theater PC(short HTPC) , but I don’t want to run it 24/7. That is where node-red comes in to place.

TV recordings with smart turn on/off

The requirements:

The HTPC should not run all the time
The HTPC should turn on only long enough to record the wanted program on live tv.
The HTPC should only turn off, if we are not already watching something in parallel (Kodi).
Turning it on should not interfere with current watching.

Node-RED

Turning it on

I was already using the Wake on LAN nodes to wake up my devices over the dashboard, which is a very simple way to accomplish it. It’s also no problem if it is already on. If one uses Node-RED as a docker container, make sure the devices are reachable by adding this; network_mode: host.

I used the “Inject” node to schedule it. There are more complex timers available, but I preferred the simplest, which is already in the basic Node-RED installation. One may also wonder why I wake two devices, because my HTPC has two network interfaces and, I never know which one is plugged in :D

Turning it OFF

That is the slightly more complex task, because it was not trivial to me how to detect, if someone is already watching something. I found the node ssh-v3, which let you run any command you would be able to tun over ssh. The relevant services are always started and there is no indicator on the HTPC to check if it’s just recording something or someone is watching a movie in parallel.

Then I remembered that I am monitoring the power of the whole entertainment system. The power consumption is the perfect indicator, because if only the HTPC is running, it consumes about 25 W and if the TV and the audio receiver are also running, it is somewhere between 100 and 150 W. Here cou can read up how I managed to install these power measurements.

Let me try to explain a little. I start the flow with an “Inject”. In this case I have two of them, because on Tuesdays I want to record something additional. These first nodes inject a variable in msg.command, which is already the shutdown now instruction for the ssh connection later. In the next Influx DB node I get the latest value of the power measurement with:

SELECT last("value") FROM "power_sens2"

Then in the next connected function node I put the logic like that:

if (msg.payload[0].last > 50){
    msg.payload = 'uptime';
}else{
    msg.payload = msg.command;
}

return msg;

Which should speak for itself. I replace the shutdown command by something innocent(uptime), but there may be better solutions which just interrupt the flow there.

Finally, it goes to the ssh node. And for debug reasons I added two debug nodes (one for the payload and one for the session of the ssh connection).

TV-Headend / Kodi

Map recordings to NAS

My HTPC has limited resources, so I wondered if I can somehow mount a nfs-share from my NAS. I am using LibreELEC and their documentation provides good instructions, see here.

The only thing I changed was to set Before=kodi.service to Before=service.tvheadend42.service, because in my case it turned out to be more stable, otherwise the recording were only partially loaded. Following my entire service file:

[Unit]
Description=nfs share for tv recordings
Requires=network-online.service
After=network-online.service
Before=service.tvheadend42.service

[Mount]
What=10.32.1.5:/mnt/tralala/kodi_recordings
Where=/storage/recordings
Options=
Type=nfs

[Install]
WantedBy=multi-user.target

Schedule the recording in TV-Headend

I simply created a timer in TV-Headend, which can be done from the Kodi interface or over the web frontend f TV-Headend. It might be easier to use the web frontend..

Like that I covered all my requirements and enjoy the setup ever since :D

Ansible

2023-08-01T00:00:00+02:00

Ansible in general

I introduced Ansible at work as well as in my home and came to appreciate it a lot. If hosts are set up with ansible it is easy to replicate and keep track of all instances. At the same time I consider it documented, if set up purely with ansible.

To get started I can recommend this book. It is a perfect starting point and if you want to go deeper just use the official ansible documentation.

Run ansible tasks in parallel

I am scheduling some lxc container with awx and ansible on my proxmox node. Now I added a new K8s development cluster, which consumes a lot of power, while running, so I also want to make sure, that this is shutdown most of the time.

With the containers it was easy with simple tasks, because they are started and stopped quickly.

Following my playbook for containers, which is fast enough:

- name: Start/stop containers
  hosts: proxmox
  become: yes

  vars:
  ids:
  - '202'
  - '203'
  - '204'
  - '206'
  - '207'
  state: started

  tasks:
    - name: " LXC's"
      proxmox:
      api_host: 10.0.0.3
      api_user: root@pam
      api_password: !vault |
      $ANSIBLE_VAULT;1.1;AES256
      62623164336537346264373264386431356534636162343439393734303233386437656365623161
      3035376339316363353764626566343832653834656638650a353166326630366563363362396633
          NOT A REAL HASH, EVEN WITH SECRET OF NO USE
      3735663235646638360a653961366538383036663035666134303231346562323732306334373965
      30323435616630306639396335386133326431663066333539616636393466653764
      node: pve
      vmid: ""
      state: ""
      loop: ""

The K8s cluster is composed of virtual machines, which also mounts nfs-shares, and they take some time shutting down. That is why I want to run the commands in parallel.

- name: Start/stop K8s related vms
  hosts: proxmox
  become: yes

  vars:
    ids:
      - '301'
      - '310'
      - '311'
      - '312'
    state: started

  tasks:
    - name: " vm's"
      proxmox_kvm:
        api_host: 10.0.0.3
        api_user: root@pam
        api_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          62623164336537346264373264386431356534636162343439393734303233386437656365623161
          3035376339316363353764626566343832653834656638650a353166326630366563363362396633
          NOT A REAL HASH, EVEN WITH SECRET OF NO USE
          3735663235646638360a653961366538383036663035666134303231346562323732306334373965
          30323435616630306639396335386133326431663066333539616636393466653764
        node: pve
        vmid: ""
        state: ""
        timeout: 30
        force: true
      async: 175 # proxmox forcefully terminates a vm after 120 secondsi
      poll: 0 # moves on to the next task immediately without checking back (concurency)
      loop: ""
      register: result

    - name: Check async tasks status
      async_status:
        jid: ""
        loop: ""
        loop_control:
        loop_var: "async_result_item"
      register: async_poll_results
      until: async_poll_results.finished
      retries: 200

The option poll: 0 makes sure to not check if a task was successful and ansible goes immediately to the next task. The task with the module async_status allows me to check back on them in this case.

Like that one has a nice mechanic to parallelize long-running tasks.

Snapshots on proxmox

For some experiments, with multiple vm’s (4 for the k8s cluster), I had to quickly take snapshots at the same time. That is how I do it over ansible, respectively with awx:

https://docs.ansible.com/ansible/latest/collections/community/general/proxmox_snap_module.html

AWX

I use an older version of AWX because I decided to not run it on a K8s cluster, because I only start the cluster on certain occasions to save energy. There is an easy install possibility with docker-compose on a docker instance see here.

If you have a properly set up ansible project it is super simple to automated it in AWX. As soon as multiple users deploy scripts on servers it gets a lot easier to track, in what state the servers are.

More to come…

Helm

2023-07-30T00:00:00+02:00

Create a helm-chart

I will use my publicly available comments app to build a helm-chart. Get the default scaffolding from helm with:

helm create comments

What I changed to begin with:

In Charts.yaml I changed a few:

appVersion: “53” (so that it matches the tag of the image I want to deploy)
name: comments
Description: A Helm chart for the comments stack

In values.yaml:

replicaCount: 3 (because I have 3 worker nodes on my cluster)
image.repository: toubivankenoubi/comments (my image is available on docker hub)

Furthermore, some snippets from the values.yaml with a little explanation:

service:
  type: ClusterIP
  port: 8080
  
ingress:
  enabled: true
  className: "nginx"
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  hosts:
    - host: k8s-node-1.tobisyurt.local
      paths:
        - path: /comments
          pathType: Prefix

    - host: k8s-node-2.tobisyurt.local
      paths:
        - path: /comments
          pathType: Prefix
    - host: k8s-node-3.tobisyurt.local
      paths:
        - path: /comments
          pathType: Prefix
    - host: tobisyurt.net
      paths:
        - path: /comments
          pathType: Prefix
  tls: []

I only changed the port of the clusterIp service, because I was used to 8080.

In the Ingress I added a lot of hosts. The important one is tobisyurt.net which is the public facing. Because I want to be able to test it locally too, I added my internal(private network) DNS entries for the 3 k8s nodes.

helm dependencies

The comments app is dependent on a mongodb, so we add that to the dependencies in the Chart.yaml:

...

dependencies:
- name: mongodb
  version: "12.15.4"
  repository: "https://registry-2.docker.io/bitnamicharts"

If you deploy this chart as is, it would deploy the subchart mongodb with all its defaults. Normally you need to override some of these values, so that your pods talk to each other properly.

In the values.yml in your main chart you can override values of your subcharts as follows:

... other values of your chart ...

mongodb:
  persistence:
    size: 499Mi
  auth:
    rootPassword: "a password..."

Using the helm chart

Deployed with:

helm install comments-test ./comments -n comments

Uninstalled with:

helm uninstall comments-test -n comments

Test it without deploying it on the cluster (dry run):

helm install --debug --dry-run comments-test ./comments

Upgrade to the latest chart release:

helm upgrade comments-test . -n comments

K8s

2023-06-25T00:00:00+02:00

I decided to install a kubernetes (short K8s) cluster in my homelab. Just to be able to mess around with it as I please. By setting it up myself I also learn the architecture of it, its benefits , as well as its limitations or where cloud providers make your life easier for certain costs…

Because I only fnd time in the evenings it took me quite some time, but one can break it down in several milestones, which are manageable in short time slots. This way It was always fun!

Some useful commands

To delete something previously deployed by kubectl apply some-file.yml:

kubectl delete -f test-replica-set.yml

Persistent Volumes

I will provide persistent volumes by my TrueNas server with NFS. I used the official k8s documentation and some other blog as reference¹.

Add nfs subdir external provisioner:

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

kubectl create ns nfs-provisioner

NFS_SERVER=192.168.x.y
NFS_EXPORT_PATH=/mnt/sysdataset/k8s-nfs-1

helm -n  nfs-provisioner install nfs-provisioner-01 nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
    --set nfs.server=$NFS_SERVER \
    --set nfs.path=$NFS_EXPORT_PATH \
    --set storageClass.defaultClass=true \
    --set replicaCount=1 \
    --set storageClass.name=nfs-01 \
   --set storageClass.provisionerName=nfs-provisioner-01  

Theoretically that should work by now. But as always it did not -.-

Troubleshooting

Firstly the pods could not mount the volumes, because nfs-common was missing on each worker node. After installing that it could mount the volumes.

The next problem was then container related. I tried to install mongodb and my worker node vm missed a cpu instruction (The default one was missing some newer ones). So I changed my processor type of the worker nodes in proxmox to “host”. Finally all worked as expected.

ingress

The ingress controller is mapped to the nodeport range because of the bare metal setup. One can change that, if it is important and the nodes should expose ports 80 and 443 directly. See here in the documentation: https://kubernetes.github.io/ingress-nginx/deploy/baremetal/#over-a-nodeport-service

80:32355/TCP http
443:32748/TCP https

My router resolves to host names of my vms like hostname.tobisyurt.local. So in my case I can access ingress on node 1 like that: http://k8s-node-1.tobisyurt.home:32355/comments/ For me that is good enough for testing purposes. Especially because I will use an external load balancer, so the port numbers do not matter anyways.

External Load Balancer

I simply used my nginx reverse proxy / waf, which I already use for all other stuff to route traffic to the cluster.

I will recommend reading it up on the nginx documentation, but that is how I set it up:

I defined the cluster in the nginx.conf

upstream k8s {
        server k8s-node-1.tobisyurt.home:32355;
        server k8s-node-2.tobisyurt.home:32355;
        server k8s-node-3.tobisyurt.home:32355;
    }

The default load balancing is Round Robin, which is what I want in this case, because it is a stateless application. If the app is stateful, they do not share all information (for example; the deployment does not have a shared session storage) you can set ‘ip_hash’. Normally you would also want to configure a proper health check.

In the vhost config it looks as following:

location /example {
#       proxy_set_header Host k8s-node-x.tobisyurt.home;
        include snippets/proxy-params.conf;
        proxy_pass http://k8s;
        access_log /var/log/nginx/k8s-test.access.log json_analytics;
        error_log /var/log/nginx/k8s-test.error.log info;
}

It is important to make sure the Host header corresponds to the host in the ingress configuration. Otherwise, the requests will not get forwarded to the right apps in your k8s cluster.

Pull images from private registries

You need to create a k8s secret for the private registry to enable your k8s cluster to pull images from it². Here an example how to generate a secret named regcred for a private registry:

kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

Configure NFS as Kubernetes Persistent Volume Storage: https://computingforgeeks.com/configure-nfs-as-kubernetes-persistent-volume-storage/ ↩
k8s documentation: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ ↩

Backup Strategy follow-up

2023-02-25T00:00:00+01:00

With my Backup-Nas, and other small changes the backup strategy changed a little.

These are my previous posts for reference:

With my previous strategy I had one incident, which led me to download around 4 TB of movies again. Which took some time and some movies my system could hardly find. That is why I decided that with the newly built Backup-Nas, I would also back up all my media.

Therefore, I decided to improve my storage in two steps:

Put the media on a ZFS RAIDZ1¹ Pool, so that I can also benefit from the self-healing abilities of ZFS
Backup everything on the Backup-Nas, also on RAIDZ1.

Improvement 1 - RAIDZ1

For that to be power efficient I invested in 3 18 TB drives, which leaves me with about 33 TB of usable space in a RADIZ1 array. They consume about 5 W each in standby, wich was acceptable. All the other drives I would use for the Backup-Nas media pool.

Improvement 2 - Backup

Unfortunately I needed lots of small drives to match the big capacity of the media pool of my Main-Nas. So I asked around and finally filled up my Backup-Nas with a lot of old 4 and 3 TB drives. In the end I had the following pools ready:

Pool 1 = (4 * 4 TB in RAIDZ1) + (5 * 3 TB in RAIDZ1)
Pool 2 = 2 * 10 TB in a mirror

In total about 32 TB.

Like that I was able to hit approximately the same capacity as my Main-Nas for the media. It does consume some power, but it is only running about 2 times an hour a week to synchronize.

The rest of the backup strategy I didn’t change.

RAID-Level Summary: https://en.wikipedia.org/wiki/ZFS ↩