So last week, I once and for all let go of that mindset and started deploying OpenClaw. OpenClaw has been a hot topic in the tech space since the beginning of this year. It’s been all over the internet —on every tech news site, every forum, and every computer-related reddit thread. To be honest, I mostly ignored the articles I saw, though I did google it once out of curiosity. But the AI trend was real, and fast forward to today —I’ve managed to convert my shell scripts into a complete end-to-end stack of tools for measuring and analyzing network performance.

OpenClaw

If you’re not already aware, OpenClaw is an AI agent that can interact directly with your files, browse the web, execute shell commands (and yes, even create cron jobs!), read your entire email inbox (and respond when you like), manage your calendar, and more. I haven’t gone too far with it yet, but I’ve been using it for light coding —converting shell scripts and assembling the full stack of tools needed to publish network test results to a database with visualization.

There are many concerns on security and the extensive permissions it’s been granted to do almost anything. But from my perspective, OpenClaw is open-source anyway, and I do trust the open-source community. That doesn’t mean I don’t value security. But I believe anyone who wants and is able to deploy the software, should also have the capability to assess how much data and what level of confidentiality they are willing to risk. Fo my use case of light coding and research, it fits the bill!

That being said, there are also plenty of recommendations online to have it running on an isolated environment such as a VPS or separate machine. For some people, they have it deployed on the base model of a Mac mini. In my environment, it’s deployed on my Proxmox cluster, and on an isolated VLAN where only access to the internet and another VM (sandbox) are allowed. I haven’t faced issues allocating 2 CPU and 4GB RAM for it, but I would suggest having sufficient disk space in case you intend to build applications within the same environment.

Openclaw is just the agent —a carriage without a horse. I still needed a model that will do the thinking and suggest the next actions. There was an option to use the subscription Oauth token from Claude or Gemini, but the companies behind them started cracking down on users who use this approach insteading of paying for the API access. So I hopped onto Openrouter’s rankings and found Step 3.5 Flash (Free) sitting there on the Top 10. Since then I’ve been using the same model and even signed up for the actual API subscription from the company’s official website after exhausting the limits from Openrouter.

Design

The project idea was very simple and there’s nothing special about the design. It follows the same flow as any other download speed test app: users perform the test, results are sent to an API endpoint. But I’m a sucker for visually appealing diagrams, so I made one anyway!

The infrastructure already existed. The only new components here are OpenClaw, the sandbox environment, and the applications —the Android app, FastAPI, PostgreSQL, and Apache Superset. During development, I had OpenClaw access the sandbox environment to polish the configuration. FastAPI, PostgreSQL, and Superset were all initially deployed on a single VM, but the plan from the start for the “prod” setup was to isolate them in containers. FastAPI wasn’t dockerized yet, so it was deployed in an LXC container. This will definitely be migrated to the k8s cluster, but for now I just wanted to beat the clock and get something working quickly. Superset, from the beginning, was already running in Docker. However, I wanted more processing horsepower for a snappy visualization experience —and for that, only my Unraid host was suitable. As for PostgreSQL, it’s deployed in my k8s cluster using the cnpg helm chart with a Ceph RBD backend storage.

Building the APK and bringing life into my shell scripts

I’m not a developer, but I can work with shell scripts. Before all this my tool set consisted of separate shell scripts I made in the past. My challenge is that not everyone understands shell, and it’s quite difficult to keep explaining to anyone who wants to perform tests how to execute shell scripts from a mobile phone. Performance-wise, for most of the time they did a good job. But there were times when one of the tests would deviate a little from the expectation, and I would only find out after analyzing the trace. The idea was to have an app that will run natively on the device’s OS that brings along a whole different ball game of flexibility and usability. Then came the idea of building an actual app in Android Studio with the help of AI. I definitely hit the API limits of Gemini and Claude for this one. So the first project I used OpenClaw for was to assist in reviewing and modifying the code. The app is written in Kotlin, and its main goal is to perform network throughput tests. The conditions are simple:

1. There are three different types of test: download throughput, TCP latency, and UDP latency.
2. For download test, user should be able to specify the download URL.
3. User should also be able to specify how many URLs it wants to be part of the download test.
4. For UDP and TCP latency test, user should be able to specify the IP address or FQDN.
5. There should be an option to group these tests and perform multiple iterations.
6. Network conditions are necessary to be captured at the beginning of every test.
7. Timestamps are to be captured as well before and after each test.
8. There should be a field to specify an API URL endpoint.
9. At the end of every global iteration, the test results should be captured in JSON format and should be sent to the specified API URL.
10. There should also be an option to save the JSON output locally.
11. The start button should be greyed out whenever there is an ongoing test.

There are more conditions specific to each test but I’ll save that for later.

ANTT: Automated Network Testing Tool

I was working on this during my free time at home, and, after a few days, I was able to tick off everything in the checklist! You can see the sample snaps of the app below:

JSON output

The app has the feature to run multiple iterations so there are two UUIDs generated. The true UUID across all tests is printed as the first key-value pair. The secondary UUID or run_id is only unique for each press of the start button. It will persist on multiple iterations. Timestamps are also recorded before and after every iteration for easier mapping of traces. The rest are pretty much self-explanatory.

{
  "uuid": "d81ba2d4-a21e-4da8-9657-f2f2fdc42366",
  "run_id": "8a6f53de-42f4-472b-95aa-c60f07db6f88",
  "iteration": 1,
  "timestamp_start": "2026-03-28T20:57:19",
  "timestamp_end": "2026-03-28T20:57:38",
  "location": "36.11950402, 140.00541702",
  "netinfo_Timestamp": "2026-03-28T20:57:19",
  "netinfo_Coordinates": "40.61950402, 123.60541702",
  "netinfo_Connection": "4G/LTE",
  "netinfo_IPv4": "172.16.0.31",
  "netinfo_IPv6": "2020:dead:beef:188d:cd48:3c9e:e489:2cd",
  "netinfo_MSISDN": "N/A",
  "netinfo_IMSI": "N/A (Restricted)",
  "netinfo_Operator": "REDACTED",
  "netinfo_Cell_ID": "71989000",
  "netinfo_RSRP": "-89 dBm",
  "netinfo_RSRQ": "-7 dB",
  "downloadtest1_url": "https://jiyugaoka.ensaymada.net/800MB.bin",
  "downloadtest1_streams": 4,
  "downloadtest1_downloaded_mb": 35,
  "downloadtest1_avg_mbps": 28.93510269426006,
  "downloadtest1_peak_mbps": 70.07809900990098,
  "downloadtest2_url": "https://jiyugaoka.ensaymada.net/3MB.bin",
  "downloadtest2_streams": 1,
  "downloadtest2_downloaded_mb": 5,
  "downloadtest2_avg_mbps": 21.3559266802444,
  "downloadtest2_peak_mbps": 38.55058823529412,
  "downloadtest3_url": "https://jiyugaoka.ensaymada.net/1MB.bin",
  "downloadtest3_streams": 1,
  "downloadtest3_downloaded_mb": 2,
  "downloadtest3_avg_mbps": 20.140715486194477,
  "downloadtest3_peak_mbps": 27.252594059405936,
  "downloadtest4_url": null,
  "downloadtest4_streams": null,
  "downloadtest4_downloaded_mb": null,
  "downloadtest4_avg_mbps": null,
  "downloadtest4_peak_mbps": null,
  "downloadtest5_url": null,
  "downloadtest5_streams": null,
  "downloadtest5_downloaded_mb": null,
  "downloadtest5_avg_mbps": null,
  "downloadtest5_peak_mbps": null,
  "tcptest_avg_ms": 37.4,
  "tcptest_median_ms": 37,
  "tcptest_min_ms": 34,
  "tcptest_max_ms": 41,
  "udptest_success_percent": 100,
  "udptest_avg_ms": 33.96,
  "udptest_median_ms": 32,
  "udptest_min_ms": 25,
  "udptest_max_ms": 81,
  "id": 7
}

FastAPI, PostgreSQL, and Apache Superset for visualization

I heard about fastAPI at the time of building a static website for another automation project. From a beginner’s standpoint, the learning curve wasn’t really that steep. Or maybe it’s easy to say that because my use cases were pretty simple. But overall dealing with FastAPI was pretty straight forward. You define the paths and the corresponding python functions you want to call. It was a fairly easy task for OpenClaw to create the required files to parse the JSON data and publish to a postgres backend.

Talking about Postgres, as I mentioned earlier, the CNPG helm chart was used to deploy this in my k8s cluster. Though it was deployed through ArgoCD since I always keep practice of infrastructure-as-code. First I had to deploy the cnpg operator. Below is a sample snip of the ArgoCD manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cnpg
  namespace: argocd
spec:
  destination:
    namespace: cnpg-system
    server: https://kubernetes.default.svc
  project: default
  sources:
    - repoURL: 'https://cloudnative-pg.github.io/charts'
      chart: cloudnative-pg
      targetRevision: 0.27.1
      helm:
        releaseName: cnpg
        valueFiles:
          - $values/cnpg/values.yaml
    - repoURL: git@github.com:mygituser/tokyo-prod.git
      targetRevision: HEAD
      ref: values
    - repoURL: git@github.com:mygituser/tokyo-prod.git
      path: cnpg/manifests
      targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

Then the actual cnpg cluster:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cnpg-cluster-dev
  namespace: argocd
spec:
  destination:
    namespace: cnpg-cluster-dev
    server: https://kubernetes.default.svc
  project: default
  sources:
    - repoURL: git@github.com:mygituser/tokyo-prod.git
      path: cnpg-cluster-dev/manifests
      targetRevision: HEAD
  syncPolicy:
    automated:
      prune: false
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

As for Apache Superset, this might be a little bit overkill for the job but it’s still nice to have nevertheless. I faced a few hiccups here and there but so far there’s nothing AI couldn’t fix for this setup. What I actually did was demand OpenClaw to have Superset docker instance running and gave a requirement to have the compose files to connect to my remote postgresql database. It asked for the connection information then proceeded to create the yaml file. It also identified that the setup required the installation of required packages to connect to a postgres backend so it also provided a Dockerfile.

From within the superset dashboard, the queries were failing at first. After sharing the error message to OpenClaw, it then identified a minor misconfiguration on the RedisDB. I was surprised it proactively tried to fix the configuration – restarting the containers as well. For every change and restart, it also verified whether the error has been cleared and if the query would function. After a few container restarts, a bit more of modification requests, everything was working perfectly.

Since it performed the changes proactively, it was a pain to keep track of them from the chat window. I asked it to create a README.md file of all the changes it made with a short simple description for the reason. It did a pretty good job covering everything from the creation of the custom Docker image until the SQL data type revisions I had towards the end.

The wrap up and what’s in the roadmap

The past few late night work I’ve had for this project was all filled in with curiousity and excitement. Every press of the enter button for each prompt always had me thinking if the agent would give proper results. It did and even exceeded my expectations! Those times I would share the error message asking for a fix, it performed proactively without asking, and verifying whether the changes worked. When it hit a new error during verification, it looped back to trying to find the next solution. Yes it’s pretty scary especially if you can’t keep track of the changes in real time. Definitely, a risk. But really, it just works. It’s up to the user how to manage this. And apart from version control, there should already be common and general practices how to deal with this. Also, Openclaw is not just the only autonomous agent out there. There are other options like nanobot built with a smaller footprint of python that can be easier for the user to understand and perform modifications on top of. For those cocnerned with privacy, then there’s also the self-hosted option.

It’s just the start, but I can’t wait to see what else and others can build with AI!

Talking about the app and the complete stack, these are in the very initial phase. However, there are still a few things I want to cover later on. Off the top of my head:

1. Containerize FastAPI and migrate it to the Kubernetes cluster.
2. Maybe migrate Superset to the Kubernetes cluster as well.
3. Run tcpdump while tests are running (not even sure if this is possible, lol)
4. Add the ability to import and save configuration (maybe using XML).
5. Add a scheduling or interval feature for the global iteration loop.
6. Add authentication to the API requests.

Second, the old subscription only had IPv4. Don’t get me wrong, IPv4 isn’t a bad thing. In fact, a good majority of the internet still runs on IPv4. But recently, I’ve gained interest in exploring the capabilities and other features that come with IPv6. More on this later, but the main trigger point for this is because IPv6 is already widely used in Japan. If I’m not mistaken, IPv6 adoption has been enforced by the Japan Ministry of Communications for some time now.

Third, since the beginning of last year, I’ve taken on a new role focused on optimizing customer experience performance, where speed is one of the main concerns. To better support this and overcome the limitations of my current toolset, having a speed test server (with multi-gigabit bandwidth) that I have full control of is a fundamental first step!

Hardware

Certain Lenovo Tiny PCs are popular machines for having a PCIe slot that can be fitted with 10G NICs, so that’s what I went with. I was able to get a secondhand one with an Intel i5-8500T 6C CPU. This should be more than enough for my needs, even if I wanted to host other services on this machine. As for the NIC, I had the option of going with Mellanox or Intel ones. To be more specific, I was choosing between a Mellanox ConnectX-4 and an Intel X710-DA2. Mellanox seems to be more popular in terms of compatibility and stability, but it’s also known to have elevated power consumption since it’s not able to achieve higher C-states even when idle. The X710-DA2, on the other hand, was known to have compatibility issues, especially when using the OEM-branded ones. I went with Intel to have peace of mind knowing that I tried my best to save electricity costs (as if all the other machines in the homelab justify all the power they require!).

Unfortunately, I assumed the compatibility issue was something generic and could easily be fixed by cross-flashing the original Intel firmware. While this might have worked for others, this isn’t the case when specifically using a Dell-branded X710 NIC with a Lenovo M920q Tiny PC. So, if anyone intends to follow this route, take note! I learned this the hard way because I purchased a Dell-branded one and went through the whole cross-flashing process only to find out that my machine will only boot up successfully if it’s coming from an unplugged state. Yup, that means it will only boot up after unplugging and plugging the power cable. Anyway, I ended up purchasing an Intel-branded one and just flashed it with the latest firmware. For anyone interested, the articles I followed are also shared at the end of this post!

Lenovo M920q Tiny PCs require a PCIe riser for plugging in your cards. This can easily be bought for about ¥2000~3000 online. There are different part numbers in terms of the supported bus speed, so I had to ensure I got the one with P/N: 01AJ940, which supports x8.

As for the 10G switch, for the time being, I got a 4x2.5 + 2x10G switch from Horaco (AliExpress). I already have an 8x2.5G switch from the same brand and it’s been rock stable. There was no really good reason for me to try out another brand. If there was one thing where you have to manage your risk, it’s when purchasing networking equipment from AliExpress!

10G SFP+ to copper transceivers are known to run hot. And because I keep my networking equipment in an unventilated closet, I had to shed an extra couple of bucks for the premium ones—the version that supports up to 80m of copper. For the transceivers, I got them from a brand called ZYOPM.

Managing the temperature inside an enclosed space is a priority, so I had to make sure the temps are at a minimum whenever possible. For the upstream connection of my switch to the router, I used a DAC cable. I’ve never used one before and actually thought of using optical transceivers in the beginning.

On to the Wireless AP, I also upgraded from my not-so-old TP-Link EAP610 to an EAP773. The former only supported Wi-Fi 6 over 2.4/5 GHz bands with 1G upstream. For “future-proofing,” at least for the next few years, and to make use of the upgraded upstream, I needed something which at least supported Wi-Fi 6E over 6 GHz.

I also had to get a PoE++ 90W injector since the new switch didn’t support PoE+.

Software

There are mixed opinions on whether you should virtualize your router or not. But if you live somewhere where space is a luxury, then you already have the best reason to go with virtualization (but regardless of that, I just really like complicating things, lol). So of course, we go back to our good old friend, Proxmox. The last time I had to spin up a new Proxmox machine was about 2 years ago when I started using CEPH as my default backend storage. A lot of good things must have been added within these 2 years, but I just really needed an easily managed KVM host. No one can beat Proxmox on that.

As for the router itself, I made the big switch from OPNsense to OpenWRT. And I couldn’t be happier. I’ve had OPNsense for about 3 years now and while I didn’t really complain about anything since it was more of a “set and forget” setup, whenever I had to modify something in the configuration, sometimes it just felt like I had to explore and familiarize myself again. With OpenWRT, the GUI just felt a bit more natural and warm to my eyes. Maybe it’s because of fewer sections or tabs to go through, or, I don’t know, configuration just seemed a little more straightforward this time compared to when I just started with the other.

But in fact, I don’t think I would have bothered checking out OpenWRT if only OPNsense supported MAP-E connections used by my new ISP. There is a way to get it working with OPNsense, but it was more of a workaround than a natively supported feature.

My choice of ISP

The primary factor in choosing the ISP is the monthly cost. The switch to 10G wouldn’t be justifiable if we had to pay 50% more than our old subscription. Second would be the option to have a fixed IPv4 address since I host some services for family and friends. Fortunately, I came across a post on Reddit suggesting En Hikari. En Hikari uses NTT FLET’S HIKARI as the backbone provider, so you can be assured of the same quality of connectivity used by most ISPs. At the time of writing, the monthly cost is about JPY 4,917 (tax included) plus an additional JPY 770 for the optional fixed IPv4 address. That’s a total of JPY 5,687, which comes out even cheaper than our current basic 1G subscription at JPY 5,720 (dynamic IPv4 address only).

Preparations

When I got the Intel X710 card, it only had v6.01 installed. I didn’t find a way to upgrade directly from v6.01 to the latest one (v9.56 as of this writing) and had to go through it stepwise. The upgrade path in my case was from 6.01 –> 8.6 –> 9.0 –> 9.10 –> 9.56. All done under Debian (Proxmox).

My Intel card came with unlocked vendor support, so I didn’t have to run the unlocker script. I guess this is only applicable to the OEM versions.

I actually ran into some errors initially that prevented the second Ethernet port from functioning. Honestly, I hit a wall and gave up on it for the night. Then, I’m not sure how, but it just got resolved the following morning after a restart.

After a few more rounds of restarts and going through kernel dmesg logs, I finally decided to go with a fresh install of OpenWRT. For some reason, I couldn’t figure out how to create the VLAN interfaces from the GUI. I ended up with a successful attempt when I tried to do it from the text file configuration. To those who are in the same boat, below is a sample /etc/config/network for configuring VLANs.

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd00:7808:88c3::/48'
	option packet_steering '1'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'

config interface 'lan'
	option device 'br-lan.1'
	option proto 'static'
	option ipaddr '192.168.0.1'
	option netmask '255.255.255.0'
	option ip6assign '64'
	list dns '192.168.0.1'
	option ip6hint '00'

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'
	option keepalive '5 10'

config interface 'wan6'
	option device 'eth1'
	option proto 'dhcpv6'
	option reqaddress 'try'
	option reqprefix 'auto'
	option norelease '1'

config bridge-vlan
	option device 'br-lan'
	option vlan '1'
	list ports 'eth0'

config bridge-vlan
	option device 'br-lan'
	option vlan '10'
	list ports 'eth0:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '20'
	list ports 'eth0:t'

config interface 'vlan10'
	option proto 'static'
	option device 'br-lan.10'
	option ipaddr '192.168.10.1'
	option netmask '255.255.255.0'
	list dns '192.168.10.0.1'

config interface 'vlan20'
	option proto 'static'
	option device 'br-lan.20'
	option ipaddr '192.168.20.1'
	option netmask '255.255.255.0'
	list dns '192.168.20.0.1'

/etc/init.d/network restart to apply the config.

For now, I just wanted a stable internet connection and didn’t even bother to think about having SR-IOV, so it’s been decided to just make use of the existing Proxmox Linux bridges for both the WAN and LAN interface.

En Hikari configuration on OpenWRT

References: https://note.com/arunya/n/n7d81e0de9db7 https://www.ficusonline.com/ja/posts/openwrt-v6-plus-map-e

I found these two articles that provide step-by-step instructions to get En Hikari working with v6 plus and the fixed IPv4. The initial steps to have DHCPv6 working are in the first one. In my case, I installed the required packages first:

• luci-proto-ipv6
• map
• ds-lite
• ip-full

As I know Hikari Cross utilizes MAP-E. I am not sure if ds-lite is really required, but I just installed it anyway.

I created a new WAN interface, named it ‘wan6’, and set the protocol to DHCPv6 client. Under DHCP Server > IPv6 Settings, the RA-Service, DHCPv6 Service, and NDP Proxy were all set to disabled. After a few seconds, I was assigned an IPv6 /56 subnet.

For the clients to connect over IPv6, I had to enable IPv6 under Interfaces > LAN > Advanced Settings. IPv6 assignment length was set to 64 and hint to 00 so my LAN gets assigned the first smaller subnet of the /56 block.

Under LAN > DHCP Server > IPv6 Settings, RA-Service and DHCPv6 Service should be set to server and NDP Proxy to relay. Under IPv6 RA Settings, I made sure SLAAC was enabled. The rest were set to default. After applying the changes, my devices started having an IPv6 address, the majority by SLAAC and a very few through DHCPv6.

For the IPv4 configuration, I just followed the article. Though I am still sharing my snaps here for future reference:

Info that was provided by the ISP:

• V4アドレス
• インターフェスID
• BRアドレス
• ユーサID
• パスワード

Quick speed test

I don’t have any other equipment (yet, lol) that supports 10G traffic, so I spun up an LXC container on the same PVE hosting OpenWRT and ran a simple iPerf test from there. I selected one of the JP servers from this list maintained by @R0GGER (Big thanks to him!).

The result, DL and UL speed respectively:

The result:

DL SPEED

UL SPEED

I may not be getting the full 10G bandwidth, but I’m certainly getting ~5x the speed compared to my previous connection for the same price.

What’s next

In addition to the benefits a multi-gigabit upstream brings, being assigned an IPv6 subnet from my ISP will allow me to host multiple services from within my local network since the hosts will now have publicly routable addresses. This means a few things:

1. No need to keep track and configure ports for port-forwarding to different VMs or containers.
2. Multiple services can be hosted on the same port.
3. I can create a DMZ network to completely expose hosts to the public internet.
4. P2P connectivity can be established without the requirement of NAT-ing.

References

• Cross-flashing X710 NICs
• Disabling vendor-lock of X710 NICs
• Explanation of above
• OpenWRT on Proxmox
• OpenWRT configuration for En Hikari with v6 plus and Fixed IP
• Intel X710 drivers

This might come in handy to those who are thinking of setting up their own VPS to connect to their K8s cluster, whether it be hosted at home or in the cloud, or maybe to those who just simply want to connect two mutually remote devices running on two different networks with a firewall.

To start off, first, keep in mind that my NAS in the backend is exported via NFS by an LXC container hosted on Proxmox, and the remote service (Deluge in this case) that would be mounting the share is running as a docker container.

Tailscale pre-requisites for LXC containers on PVE

On the NFS server end, the container needs access to /dev/net/tun on the host. To allow this we need two lines in the container configuration (reference: tailscale kb article).

lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
lxc.cgroup2.devices.allow: c 10:200 rwm

Stop the container and hop on to your PVE host’s terminal. Modify /etc/pve/lxc/1201.conf and the above two lines.

Tailscale installation on the LXC container and remote server

Next up is installation of tailscale. The installation is fairly easy. An installation command can be generated together with a pre-approved authentication. Go to your Tailscale account > Machines > Add device > Generate install script. Ensure to create a reusable key. There is no need to add tags if the device is intended for a single user. Execute on both the LXC container and remote server

e.g.

curl -fsSL https://tailscale.com/install.sh | sh && sudo tailscale up --auth-key=tskey-auth-kHD2rsVNiv11CNTRL-188WY38WSG2iqTHTZs1oF2e3LMSmMYuXF

You should now be able to see your device under the Machines tab. The tailscale IP will also be visible from there. If you want to check the status of tailscale: sudo systemctl status tailscaled

To avoid manual intervention in the future, you can disable the key expiry for each device in the Machines tab.

Configure NFS

Allow the tailscale IP in your /etc/exports file:

/mnt/nfs-share 100.22.112.36/32(rw,sync,no_subtree_check,all_squash,anonuid=1000,anongid=1000,insecure)

On the remote server, ensure the containers are stopped and modify your compose file to mount the nfs share to the /downloads directory.

version: "2.1"
services:
  deluge:
    image: lscr.io/linuxserver/deluge:latest
    container_name: deluge
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Asia/Manila
      - DELUGE_LOGLEVEL=error
    volumes:
      - .:/config
      - nfsdir:/downloads
    ports:
      - 8112:8112
      - 6881:6881
      - 6881:6881/udp
    restart: unless-stopped
volumes:
  nfsdir:
    driver: local
    driver_opts:
      type: nfs
      o: addr=100.22.112.36,rw,vers=4.1
      device: ":/mnt/nfs-share"

Start the container and try to create a file in the NFS share.

Install tailscale-operator on k8s

It will be easier for you to refer to the official documentation so I am sharing the actual KB article here.

In summary, two items are needed to be configured from the Tailscale Management page — tags and OAuth client credentials. Complete these steps and carry on with the helm installation. Helm installation steps are also shared below.

Installation with helm

Replace the clientId and clientSecret variables:

helm repo add tailscale https://pkgs.tailscale.com/helmcharts
helm repo update
helm upgrade \
  --install \
  tailscale-operator \
  tailscale/tailscale-operator \
  --namespace=tailscale \
  --create-namespace \
  --set-string oauth.clientId="<OAauth client ID>" \
  --set-string oauth.clientSecret="<OAuth client secret>" \
  --wait

Create a ProxyGroup of type egress.

apiVersion: tailscale.com/v1alpha1
kind: ProxyGroup
metadata:
  name: ts-proxies
spec:
  type: egress
  replicas: 3

Installation with ArgoCD

If you are tidy, maintain your infrastructure as code, and use ArgoCD, then go ahead and download the values.yaml file and update the clientId and clientSecret fields.

helm show values tailscale/tailscale-operator > values.yaml

...
oauth:
  clientId: "kq7LLBiMeL11CNTRL"
  clientSecret: "tskey-client-kq7LLBiMeL11CNTRL-rPFdRtSCj74REgir443J84CYp12Lo8Vce"
...

Create the manifest for your ArgoCD application (replace yourgituser and yourrepo).

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: tailscale
  namespace: argocd
spec:
  destination:
    namespace: tailscale
    server: https://kubernetes.default.svc
  project: default
  sources:
    - repoURL: 'https://pkgs.tailscale.com/helmcharts'
      chart: tailscale-operator
      targetRevision: 1.78.3 # This is the latest version as of the time of this post.
      helm:
        releaseName: tailscale-operator
        valueFiles:
          - $values/tailscale/values.yaml
    - repoURL: git@github.com:yourgithubuser/yourrepo.git
      targetRevision: HEAD
      ref: values
    - repoURL: git@github.com:yourgithubuser/yourrepo.git
      path: tailscale/manifests
      targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
    syncOptions:
      - CreateNamespace=true

Don’t forget to create the egress ProxyGroup manifest!

Testing connectivity from a pod to the remote server

Login to any pod and perform a curl to the other end. If you don’t have a test pod, then you can deploy an alpine image.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: alpine
  name: alpine
  namespace: alpine
spec:
  replicas: 1
  selector:
    matchLabels:
      run: alpine
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        run: alpine
    spec:
      containers:
      - image: alpine:latest
        command:
          - /bin/sh
          - "-c"
          - "apk --update add curl && sleep 1440m"
        imagePullPolicy: IfNotPresent
        name: alpine

kubectl -n alpine exec -it alpine-9c7cb4b95-nx7b4 -- curl http://10.43.59.234:8112

Pod to tailscale communication

Now that I have a secondary network attachd to my cluster, I can utilize my ingress controller, that is Traefik, to provide HTTPS access to devices on that network.

To do so traefik must be allowed to reach ExternalNames. Set providers.allowExternalNameServices to true in the values.yaml file:

...
providers:
  kubernetesIngress:
    enabled: true
    allowExternalNameServices: true
...

Create the ingress resource as you would normally do, and also create a service of type ExternalName. You can either provide the IP or the tailscale FQDN.

In case of the latter:

apiVersion: v1
kind: Service
metadata:
  annotations:
    tailscale.com/tailnet-fqdn: "voyager.taile9a23.ts.net"
    tailscale.com/proxy-group: "ts-proxies"
  name: ts-egress-voyager
spec:
  externalName: voyager-deluge # any value - will be overwritten by the operator
  type: ExternalName
  ports:
  - port: 8112
    protocol: TCP
    name: web

Note that you have to specify the ProxyGroup that was created in the previous step. Also an additional clusterIP service will be created on top of this:

NAME                         TYPE           CLUSTER-IP      EXTERNAL-IP                                      PORT(S)    AGE
ts-egress-voyager            ExternalName   <none>          ts-ts-egress-voyager-mgmz2.tailscale.svc.local   8112/TCP   2s
ts-ts-egress-voyager-mgmz2   ClusterIP      10.43.231.143   <none>                                           8112/TCP   2s

This is the final step and you should now be able to access Deluge on the remote server via HTTPS.

Tailscale Lovin'

What I love about tailscale is how easy it is to setup and the how the folks over there continue to offer a free plan for personal use. I still remember how difficult it was to setup a VPN with friends 15-20 years ago just to be able to play games like Counter-Strike or Starcraft (maybe I was not just as experienced yet? I don’t know!) But with tailscale today, it’s totally a game changer. I also want to mention here that tailscale offer Mullvad VPN access for $5/month per 5 devices. Although I am not sure if this is supported by the tailscale-operator, at the least all your other devices can utilize this (you can even install tailescale on the AppleTV!). I probably will consider subscribing to this in the not so distant future if I don’t find any other use case for remote Deluge server.

Ceph on its own is a huge topic. It has so many moving parts-monitors, metadata servers, OSDs, placement groups to name a few. And yes I am completely new to it. The good thing in today’s age every beautiful piece of software together with the community backing it stems from how well the user documenation is written. Here I am not just talking about Ceph but also Proxmox. Both applications have very well-written documentation to get you started, and if anything just goes haywire, you have a plethora of resources to point you back to the right direction.

That said, Proxmox also nicely integrates Ceph and the complete installation can be done from the GUI itself. All I had to do was create the VLANs and specify the interface to be used for the public and private network. Though to be honest this part took me some time and a fair amount of ceph reinstallations to understand a bit more on the networking side.

Today’s agenda is to walk through how I was able to integrate my K8S cluster to an externally managed ceph storage, and how I am actually migrating to this new cluster by performing a disaster recovery procedure. I am also sharing here whatever resources were used for this operation.

References:

• https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster
• https://rook.io/docs/rook/latest-release/CRDs/Cluster/external-cluster/
• https://github.com/catalogicsoftware/velero-docs/blob/main/velero-intro-pvs/velero-intro.md
• https://velero.io/docs/v1.13/migration-case/
• https://velero.io/docs/v1.13/contributions/minio/

Current situation

At the moment I have an RKE2 cluster running Longhorn as my storage solution for persistent volumes. It has worked wonderfully -safeguarding my data with replication. But the long term goal had always been to leverage Proxmox’s built-in ceph management tool. Back then I was not having the appropriate hardware but with recent “reasonable” upgrades, I think now is the right time. There aren’t any heavy workload and majority of what are running are media management tools. The problem is the virtual disk is 150GB which takes up about a third of my 500GB SSD. Apart from this Longhorn storage is running on top of the same disk where the RKE2 ETCD DB is running and all just converge on the same physical disk where Proxmox is also installed. The cluster makes use of a single storageclass for both RWO and RWX PVCs. This cluster is also running in the same vlan where the rest of my VMs are also running. A single vNIC is attached for both k8s traffic, and longhorn replication traffic.

Planning and logistics

The second cluster will have about 100GB total size (about 80GB total of ephemeral storage after OS). Rook will be used to connect to the Proxmox-managed ceph cluster for persistent volumes. With this the cluster will have two storageclasses — ceph-rbd for RWO PVCs, and cephfs for RWX. Each VM will have two vNICs attached. First vNIC will be assigned a new VLAN3 to be used solely for k8s traffic, and the second vNIC will be running in another VLAN dedicated for ceph traffic. Velero will be used to migrate to the new cluster. MinIO will be used as the S3-compatible storage for storing the backups. Existing Cilium LB IPs will be re-used by re-learning BGP routes from the main router with the new VLAN.

I am working on a diagram for the network at home and although it’s still work in progress, it should be more than enough to help you understand how things are connected and segmented.

To expound more on this, on high-level we have two clusters - cluster 1 (old) and cluster 2 (new). The migration will go as follows:

• First we will prepare cluster 2 by integrating to an external ceph cluster.
• Then install velero on both k8s clusters.
• On cluster 1, perform a backup of the resources.
• Go back to cluster 2 and perform the restore. RWO PVCs directly should be migrated to ceph-rbd, and the RWX PVCs to be migrated to cephfs.
• Block cluster 1 IPs (on VLAN20) from internet access to avoid issues with cert-manager.
• Enable BGP peering on cluster 2 (VLAN3) and reconfigure BGP on OPNsense to route externally exposed IPs to this cluster.

Installing Rook and integrating with an external cluster

Download create-external-cluster-resources.py:

curl -s https://raw.githubusercontent.com/rook/rook/release-1.13/deploy/examples/create-external-cluster-resources.py > create-external-cluster-resources.py

Without enabling prometheus you might get the following error after the next step:

ExecutionFailureException: can't find monitoring_endpoint, prometheus module might not be enabled, enable the module by running 'ceph mgr module enable prometheus'

To avoid this you can either enable prometheus with the command below or add --skip-monitoring-endpoint when executing the python script.

ceph mgr module enable prometheus

Execute the script to generate the required environment variables:

python3 create-external-cluster-resources.py --rbd-data-pool-name tokyoceph --namespace rook-ceph-external --format bash

Sample output:

export NAMESPACE=rook-ceph-external
export ROOK_EXTERNAL_FSID=31833ea9-541d-435d-96ef-e9c653e97d2b
export ROOK_EXTERNAL_USERNAME=client.healthchecker
export ROOK_EXTERNAL_CEPH_MON_DATA=pve1=10.22.0.11:6789
export ROOK_EXTERNAL_USER_SECRET=AQAigfdlyQocHhAAtBlaM6EMLp9N6ysQoUUN2A==
export CSI_RBD_NODE_SECRET=AQAigfdlfB/OHhAAeVpSf6Ow4OE8pWjOU2XPkA==
export CSI_RBD_NODE_SECRET_NAME=csi-rbd-node
export CSI_RBD_PROVISIONER_SECRET=AQAigfdld8WCHxBBtakNxF2axoyXbghVzsLROA==
export CSI_RBD_PROVISIONER_SECRET_NAME=csi-rbd-provisioner
export CEPHFS_POOL_NAME=tokyocephfs_data
export CEPHFS_METADATA_POOL_NAME=tokyocephfs_metadata
export CEPHFS_FS_NAME=tokyocephfs
export CSI_CEPHFS_NODE_SECRET=AQA6gvdlNsZGKxAAXbWkhSawMStb+CGwV7Lurw==
export CSI_CEPHFS_PROVISIONER_SECRET=AQA6gvdlP3MhLBAA+vykqaFjLh20bagnGNwGsg==
export CSI_CEPHFS_NODE_SECRET_NAME=csi-cephfs-node
export CSI_CEPHFS_PROVISIONER_SECRET_NAME=csi-cephfs-provisioner
export MONITORING_ENDPOINT=10.22.0.12
export MONITORING_ENDPOINT_PORT=9283
export RBD_POOL_NAME=tokyoceph
export RGW_POOL_PREFIX=default

Paste the output in your shell environment either in the CLI or in the shell configuration file e.g. ~/.bashrc or ~/.zshrc. If you are pasting in your shell configuration file, don’t forge to reload.

Download the import script and run it:

curl -s https://raw.githubusercontent.com/rook/rook/release-1.13/deploy/examples/import-external-cluster.sh > import-external-cluster.sh
./import-external-cluster.sh

This import script will read the environment variables and accordingly create the necessary resources to connect to your external ceph cluster. The following resources should get created:

namespace/rook-ceph-external created
secret/rook-ceph-mon created
configmap/rook-ceph-mon-endpoints created
secret/rook-csi-rbd-node created
secret/rook-csi-rbd-provisioner created
secret/rook-csi-cephfs-node created
secret/rook-csi-cephfs-provisioner created
storageclass.storage.k8s.io/ceph-rbd created
storageclass.storage.k8s.io/cephfs created

Now it’s time to install rook! Take note of the first two lines. Here we specify the namespace where the rook operator will be running in and the namespace for the external ceph cluster. You can have a common namespace if you wish.

export operatorNamespace="rook-ceph"
export clusterNamespace="rook-ceph-external"
curl -s https://raw.githubusercontent.com/rook/rook/release-1.13/deploy/charts/rook-ceph/values.yaml > values.yaml
curl -s https://raw.githubusercontent.com/rook/rook/release-1.13/deploy/charts/rook-ceph-cluster/values-external.yaml > values-external.yaml
helm install --create-namespace --namespace $operatorNamespace rook-ceph rook-release/rook-ceph -f values.yaml
helm install --create-namespace --namespace $clusterNamespace rook-ceph-cluster \
--set operatorNamespace=$operatorNamespace rook-release/rook-ceph-cluster -f values-external.yaml

Logging the important nodes post helm installation (this is for personal reference. This part can be skipped):

# Operator installation
# Important Notes:
# - You must customize the 'CephCluster' resource in the sample manifests for your cluster.
# - Each CephCluster must be deployed to its own namespace, the samples use `rook-ceph` for the namespace.
# - The sample manifests assume you also installed the rook-ceph operator in the `rook-ceph` namespace.
# - The helm chart includes all the RBAC required to create a CephCluster CRD in the same namespace.
# - Any disk devices you add to the cluster in the 'CephCluster' must be empty (no filesystem and no partitions).

# CephCluster installation
# Important Notes:
# - Visit https://rook.io/docs/rook/latest/CRDs/ceph-cluster-crd/ for more information about the Ceph CRD.
# - You can only deploy a single cluster per namespace
# - If you wish to delete this cluster and start fresh, you will also have to wipe the OSD disks using `sfdisk`

In a few minutes you should be able to see a successful connection to your ceph cluster. You can also query the storageclasses created as part of the import script.

❯ k -n rook-ceph-external get cephcluster
NAME                 DATADIRHOSTPATH   MONCOUNT   AGE     PHASE       MESSAGE                          HEALTH      EXTERNAL   FSID
rook-ceph-external   /var/lib/rook     3          2m53s   Connected   Cluster connected successfully   HEALTH_OK   true       31833ea9-541d-435d-96ef-e9c653e97d2b

❯ k get sc
NAME       PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
ceph-rbd   rook-ceph.rbd.csi.ceph.com      Delete          Immediate           true                   3h10m
cephfs     rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   3h10m

Installing velero on Cluster 1 and Cluster 2

For this I prepared five files:

• credentials-velero
• velero-exclude-nfs-volumepolicy.yaml
• minio-service-endpoint.yaml
• 1_velero-change-sc-configmap.yaml
• 2_velero-change-sc-cephfs-configmap.yaml

Before installation, ensure that you have an S3-compatible storage for storing backups. I am using MinIO hosted on my Openmediavault NAS. To be able to reach this from within my cluster, I have to create a service and endpoint manually pointing to the IP and port that should be reachable by the nodes. You can follow this yaml file and change the IP and port whichever is necessary:

apiVersion: v1
kind: Namespace
metadata:
  name: external
---
apiVersion: v1
kind: Service
metadata:
  namespace: external
  name: minio
spec:
  ports:
  - name: app
    port: 9000
    protocol: TCP
    targetPort: 9000
  - name: web
    port: 9001
    protocol: TCP
    targetPort: 9001
  clusterIP: None
  type: ClusterIP
---
apiVersion: v1
kind: Endpoints
metadata:
  namespace: external
  name: minio
subsets:
- addresses:
  - ip: 10.0.0.8
  ports:
  - name: app
    port: 9000
    protocol: TCP
  - name: web
    port: 9001
    protocol: TCP

Next we install velero on both clusters. To connect to an S3-compatible we must install the velero plugin for AWS apart from specifying it as the provider.

Input your S3 access and secret key in the following format:

[default]
aws_access_key_id = inputs3SecretKeyIdHere
aws_secret_access_key = inputs3SecretKeyHere

In the same install command we specify the backup location and point to the MinIO URL. We can follow the cluster FQDN format of svc-name.namespace.svc.cluster.local to reach this externally hosted service from within the pods.

velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.9.1,velero/velero-plugin-for-csi:v0.7.0 \
    --use-node-agent --features=EnableCSI \
    --bucket velero \
    --secret-file ./credentials-velero \
    --use-volume-snapshots=false \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.external.svc.cluster.local:9000

Execute velero backup-location get or v backup-location get if you also configured auto-completion:

❯ v backup-location get
NAME      PROVIDER   BUCKET/PREFIX   PHASE       LAST VALIDATED                  ACCESS MODE   DEFAULT
default   aws        velero          Available   2024-03-23 08:15:37 +0900 JST   ReadWrite     true

Perform backup on Cluster 1

First we create a configmap to disable backup of NFS volumes. We do not want NFS volumes to be part of the backup since these are expected to be remounted later on.

Create a file with the following content:

version: v1
volumePolicies:
- conditions:
    # nfs could be empty which matches any nfs volume source
    nfs: {}
  action:
    type: skip

kubectl -n velero create cm exclude-nfs-volumepolicy --from-file velero-exclude-nfs-volumepolicy.yaml

Create two backups - one for all RWX volumes, and another one for all RWO volumes. In my case the only application that is using an RWX volume is adguard. I can specify the namespace here. I don’t want the CRDs that are created dynamically by cilium so

v backup create all-rwx-fs-backup --default-volumes-to-fs-backup=true --exclude-resources storageclasses.storage.k8s.io,ciliumendpoints.cilium.io,ciliumidentities.cilium.io --include-namespaces adguard --resource-policies-configmap exclude-nfs-volumepolicy

For RWO volumes:

v backup create all-rwo-fs-backup --default-volumes-to-fs-backup=true --exclude-namespaces kube-system,longhorn-system,velero,adguard --exclude-resources storageclasses.storage.k8s.io,ciliumendpoints.cilium.io,ciliumidentities.cilium.io,ciliumbgppeeringpolicies.cilium.io --resource-policies-configmap exclude-nfs-volumepolicy --parallel-files-upload 10

❯ v backup get
NAME                STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
all-rwo-fs-backup   Completed   0        5          2024-03-23 09:31:50 +0900 JST   29d       default            <none>
all-rwx-fs-backup   Completed   0        0          2024-03-23 09:31:28 +0900 JST   29d       default            <none>

❯ k get pv
NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                              STORAGECLASS   REASON   AGE
adguard-conf-pv        1Gi        RWX            Retain           Bound    adguard/adguard-conf-pvc           longhorn                20d
adguard-work-pv        2Gi        RWX            Retain           Bound    adguard/adguard-work-pvc           longhorn                20d
code-server-pv         500Mi      RWO            Retain           Bound    vsc/code-server-config-pvc         longhorn                20d
dashy-config-pv        200Mi      RWO            Retain           Bound    dashy/dashy-config-pvc             longhorn                20d
grafana-pv             10Gi       RWO            Retain           Bound    grafana/grafana-pvc                longhorn                20d
influxdb-data-pv       10Gi       RWO            Retain           Bound    influxdb/influxdb-data-pvc         longhorn                20d
jellyseerr-config-pv   512Mi      RWO            Retain           Bound    jellyseerr/jellyseerr-config-pvc   longhorn                20d
radarr-config-pv       1Gi        RWO            Retain           Bound    radarr/radarr-config-pvc           longhorn                20d
sabnzbd-config-pv      512Mi      RWO            Retain           Bound    sabnzbd/sabnzbd-config-pvc         longhorn                20d
sonarr-config-pv       1Gi        RWO            Retain           Bound    sonarr/sonarr-config-pvc           longhorn                20d
wikijs-config-pv       512Mi      RWO            Retain           Bound    wikijs/wikijs-config-pvc           longhorn                20d
wikijs-data-pv         10Gi       RWO            Retain           Bound    wikijs/wikijs-data-pvc             longhorn                20d

Perform restoration on Cluster 2

Verify the same backups are reflected. Patch the backup-location as read-only to avoid overwriting the backups:

kubectl patch backupstoragelocation default \
    --namespace velero \
    --type merge \
    --patch '{"spec":{"accessMode":"ReadOnly"}}'

First we will restore all RWO volumes. Since we are changing the storageclass from longhorn to ceph-rbd, we have to create a configmap to allow this conversion. Apply 1_velero-change-sc-configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: change-storage-class-config
  namespace: velero
  labels:
    velero.io/plugin-config: ""
    velero.io/change-storage-class: RestoreItemAction
data:
  longhorn: ceph-rbd

Execute the restoration:

v restore create --from-backup all-rwo-fs-backup

Wait for the pods to go up and running.

Next we update the config map to migrate longhorn storageclasses to cephfs. Apply 2_velero-change-sc-cephfs-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: change-storage-class-config
  namespace: velero
  labels:
    velero.io/plugin-config: ""
    velero.io/change-storage-class: RestoreItemAction
data:
  longhorn: cephfs

v restore create --from-backup all-rwx-fs-backup

You can check the restoration status with

v restore get

Have a look on your PVs and make sure everything is there. Check the storageclass as well.

❯ k get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                              STORAGECLASS   REASON   AGE
pvc-002f450b-1765-4a2d-8900-916f83ac4fec   1Gi        RWO            Delete           Bound    sonarr/sonarr-config-pvc           ceph-rbd                168m
pvc-5613d3f3-473a-47ed-8e3f-8bedd3cb36c4   10Gi       RWO            Delete           Bound    grafana/grafana-pvc                ceph-rbd                168m
pvc-5d088e43-fd9e-4e08-ac4c-c095a681b18b   500Mi      RWO            Delete           Bound    vsc/code-server-config-pvc         ceph-rbd                168m
pvc-80d7d785-9c82-4995-808e-dfb7985c910e   512Mi      RWO            Delete           Bound    jellyseerr/jellyseerr-config-pvc   ceph-rbd                168m
pvc-853f057a-c851-474d-8825-3bfbe2eb6573   10Gi       RWO            Delete           Bound    wikijs/wikijs-data-pvc             ceph-rbd                168m
pvc-8f5a7b54-cffa-4453-909a-519e9dc74f83   512Mi      RWO            Delete           Bound    wikijs/wikijs-config-pvc           ceph-rbd                168m
pvc-a5734f5f-7f10-4e22-bcb0-9c2ab4d95023   10Gi       RWO            Delete           Bound    influxdb/influxdb-data-pvc         ceph-rbd                168m
pvc-b9a70ed6-1a60-4e2b-a1fe-1de19d64c1f5   512Mi      RWO            Delete           Bound    sabnzbd/sabnzbd-config-pvc         ceph-rbd                168m
pvc-c7d935b3-2194-43ac-96b7-5eb184cb716a   200Mi      RWO            Delete           Bound    dashy/dashy-config-pvc             ceph-rbd                168m
pvc-f0d2af55-b1d9-4d76-b929-03007c787529   1Gi        RWO            Delete           Bound    radarr/radarr-config-pvc           ceph-rbd                168m
pvc-f7e848cb-4ce8-47c9-8a81-65775fd1c747   1Gi        RWX            Delete           Bound    adguard/adguard-conf-pvc           cephfs                  161m
pvc-fa9a9b75-48e8-48f0-8e53-75105b80b913   2Gi        RWX            Delete           Bound    adguard/adguard-work-pvc           cephfs                  161m

We should be able to see ceph logs by now continuously generating:

Ceph logs:

2024-03-24T10:25:57.822352+0900 mgr.pve2 (mgr.1064599) 46085 : cluster 0 pgmap v46023: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 17 KiB/s wr, 2 op/s
2024-03-24T10:25:59.822721+0900 mgr.pve2 (mgr.1064599) 46086 : cluster 0 pgmap v46024: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 19 KiB/s wr, 3 op/s
2024-03-24T10:26:01.823161+0900 mgr.pve2 (mgr.1064599) 46087 : cluster 0 pgmap v46025: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 27 KiB/s wr, 4 op/s
2024-03-24T10:26:03.823408+0900 mgr.pve2 (mgr.1064599) 46088 : cluster 0 pgmap v46026: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 19 KiB/s wr, 3 op/s
2024-03-24T10:26:05.823949+0900 mgr.pve2 (mgr.1064599) 46089 : cluster 0 pgmap v46027: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 29 KiB/s wr, 4 op/s
2024-03-24T10:26:07.824176+0900 mgr.pve2 (mgr.1064599) 46090 : cluster 0 pgmap v46028: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 22 KiB/s wr, 2 op/s
2024-03-24T10:26:09.824519+0900 mgr.pve2 (mgr.1064599) 46091 : cluster 0 pgmap v46029: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 23 KiB/s wr, 3 op/s
2024-03-24T10:26:11.824919+0900 mgr.pve2 (mgr.1064599) 46092 : cluster 0 pgmap v46030: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 51 KiB/s wr, 6 op/s
2024-03-24T10:26:13.825139+0900 mgr.pve2 (mgr.1064599) 46093 : cluster 0 pgmap v46031: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 41 KiB/s wr, 5 op/s
2024-03-24T10:26:15.825790+0900 mgr.pve2 (mgr.1064599) 46094 : cluster 0 pgmap v46032: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 53 KiB/s wr, 7 op/s
2024-03-24T10:26:17.826087+0900 mgr.pve2 (mgr.1064599) 46095 : cluster 0 pgmap v46033: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 42 KiB/s wr, 6 op/s
2024-03-24T10:26:19.826454+0900 mgr.pve2 (mgr.1064599) 46096 : cluster 0 pgmap v46034: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 44 KiB/s wr, 6 op/s
2024-03-24T10:26:21.826811+0900 mgr.pve2 (mgr.1064599) 46097 : cluster 0 pgmap v46035: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 50 KiB/s wr, 7 op/s
2024-03-24T10:26:23.827027+0900 mgr.pve2 (mgr.1064599) 46098 : cluster 0 pgmap v46036: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 21 KiB/s wr, 3 op/s
2024-03-24T10:26:25.827427+0900 mgr.pve2 (mgr.1064599) 46099 : cluster 0 pgmap v46037: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 2.7 TiB / 2.7 TiB avail; 30 KiB/s wr, 4 op/s

Migrate the BGP peers

The last but not the least is to remove the BGP peering with the old cluster and configure on the new one! If you are also using OPNsense and given you have the BGP peer configuration resources already created in the cluster, theh you just have to clone the existing BGP configuration (on OPNsense) for each of the old nodes, change the ASN number, and update the router ID to be in the new VLAN.

Onwards with Ceph

This sums up how I was able to migrate the storageclass from Longhorn to Ceph by performing disaster recovery to a new cluster. In the past I’ve done a similar kind of operationbut by simply restoring longhorn volumes with the same storageclass. It was fairly easy and longhorn even allows you to use the last PV and PVC name used. The backups can be stored in either NFS or S3 compatible storage as well. But with Velero it just feels a lot more straightforward. Since operation is via CLI, it allows you to prepare backup and restore commands that can be kept as a playbook for future use. By simply copy-pasting you will be able to restore your entire cluster without having to navigate in the GUI. Though don’t get me wrong - I am only saying this in relation to backup and restore. I haven’t even operated Longhorn long enough!

Talking about Ceph, I don’t really have anything to say yet. I just know it has long been established as the grandfather of distributed storage. The first time I came across it was at work when we started to face DB latency issues which seemed to be due to the bluestore cache and OSD memory facing some kind of memory leak. Personally I just started using it as the storage backend of my RKE2 cluster and so far connecting to it with Rook has been a breeze. With very minimal workload, I still look forward to any new learning I can get running my small cluster at home.

I used to run my k3s cluster on top of my OPNsense box, an HP T630 thin client, and a Lenovo Mini PC. I’ve recently replaced both the former with two other Mini PCs from HP as well running on 6th-gen Intel i5 CPU. So I was again sitting on the fence whether I should just sell the T630 or try to find some other use case for it.

Though I really love the thin client. It’s fanless (it doesn’t emit any noise), has two M.2 SATA slots, and an internal USB3.0 port that can be used for another storage device. The only thing is it used to run quite hot when k3s together with longhorn was running. I guess this was not really meant for continuous workload but only for lightweight tasks such as web browsing… or so I thought a NAS!

Since I now have a small k3s cluster running 24/7, and now that winter is coming (hello higher electriciy rates!) I just thought it will be a good idea to shutdown my main server and NAS to save up on electricity costs. I can then convert the T630 from a Proxmox node to a dedicated NAS running TrueNAS or Openmediavault.

After debating with myself on the two NAS solutions, I finally decided to go with OMV. Reason is I wanted something as lightweight as possible. I don’t really need the additional features of TrueNAS. Asisde from this OMV has a plugin which allows the use of system memory during run time to avoid wear and tear if I want to use a USB flash drive as my boot disk. With this I will be able to use the two M.2 SATA slots for my RAID0-ed drives.

Just to note, at first I actually bought a USB to M.2 NVME adapter to have a mirrored RAID setup with 1 NVME drive and 1 SATA drive. Unfortunately the USB adapter would always reset and it seems like due to the NVME drive overheating.

dmesg output

[Wed Oct 11 00:49:04 2023] usb 3-1: reset SuperSpeed USB device number 2 using xhci_hcd
[Wed Oct 11 00:49:04 2023] sd 2:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=30s
[Wed Oct 11 00:49:04 2023] sd 2:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 20 00
[Wed Oct 11 00:49:04 2023] I/O error, dev sdc, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 2
[Wed Oct 11 00:52:25 2023] usb 3-1: reset SuperSpeed USB device number 2 using xhci_hcd
[Wed Oct 11 00:52:25 2023] sd 2:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=31s
[Wed Oct 11 00:52:25 2023] sd 2:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 3b 9e 12 00 00 00 08 00
[Wed Oct 11 00:52:25 2023] I/O error, dev sdc, sector 1000215040 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[Wed Oct 11 01:01:44 2023] usb 3-1: reset SuperSpeed USB device number 2 using xhci_hcd
[Wed Oct 11 01:02:21 2023] usb 3-1: reset SuperSpeed USB device number 2 using xhci_hcd

I gave up on this and got myself another 500GB SATA SSD. The HP T630 has slots for a M.2 2280 and a 2230 drive. The challenge is finding a 2230 SATA drive that comes from the reliable manufacturers. Though with a 2280 size it’s fairly easy so I decided to go for a second one instead.

How did I screw it on the 2230 slot? Well, I didn’t. I used an insulated tape to put it in place. Many would say it’s unsafe especially if it will be used as a NAS, but I don’t plan of storing any important files here. It will mainly be used as a quick low powered backup solution for my homelab - storing VM disk images, hosting an S3-compatible strage like MinIO, and maybe for running non-critical docker containers for monitoring.

There is no special configuration required to run MinIO or docker containers since both come as readily-available plugins to download within OMV.

Also since this post should have been logged from October of 2023, I am just updating here that I have decided to break the RAID0 and use the two SATA drives separately. One used for docker containers, and the other one for MinIO. Over time I figured there is no good use case for me as of now to configure RAID0 on my drives since I already have my Unraid setup for bigger backups. But with two drives running independently, I do get the benefit of saving extra effort to restore the files if either of the two goes down!

Events:
  Type     Reason       Age                 From     Message
  ----     ------       ----                ----     -------
  Warning  FailedMount  45m (x3 over 56m)   kubelet  Unable to attach or mount volumes: unmounted volumes=[adguard-conf-pv adguard-work-pv], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition
  Warning  FailedMount  16m (x23 over 61m)  kubelet  MountVolume.MountDevice failed for volume "adguard-work-pv" : rpc error: code = Internal desc = mount failed: exit status 32
Mounting command: /usr/local/sbin/nsmounter
Mounting arguments: mount -t nfs -o vers=4.1,noresvport,intr,hard 10.43.20.191:/adguard-work-pv /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/6c472e8b20509432a91f0d78010890b28fd5ebac0d9e85d0554ba7a69c9dccbd/globalmount
Output: mount.nfs: Protocol not supported
  Warning  FailedMount  6m50s (x21 over 61m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[adguard-work-pv adguard-conf-pv], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition
  Warning  FailedMount  67s (x31 over 62m)    kubelet  MountVolume.MountDevice failed for volume "adguard-conf-pv" : rpc error: code = Internal desc = mount failed: exit status 32
Mounting command: /usr/local/sbin/nsmounter
Mounting arguments: mount -t nfs -o vers=4.1,noresvport,intr,hard 10.43.25.161:/adguard-conf-pv /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/e5cf370525e8a07eda18f130064a2fb3712c6f4668904245d8622230e3defd8e/globalmount
Output: mount.nfs: Protocol not supported

At first I was having some doubt on the NFS package — if it got updated and started causing issues with Longhorn. But this wasn’t the case. Googling online landed me to one github issue that was opened fairly recently (link here). andrewheberle shared a comment about facing the same issue after upgrading the kernel to 5.15.0-94.

https://www.kolide.com/features/checks/ubuntu-unattended-upgrades

Checking my apt history, it looks like Ubuntu upgraded automatically last 8th of February: less /var/log/apt/history.log

This looks to be due to unattended upgrades that are enabled by default upon OS installation.

You can check this by executing: cat /etc/apt/apt.conf.d/20auto-upgrades

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

In my case I disabled both updating of package list and unattended upgrades by changing the value from 1 to 0.

Before we start the rollback, let’s check the available linux kernel images with dpkg -l | grep linux-image. We should be able to see our target kernel version. In this case 5.15.0-92 should be there.

After verification then we can start uninstallation of the new kernel image. On the next reboot, the kernel should be running with the next latest available kernel version.

Refeferences:

• https://www.geekersdigest.com/how-to-downgrade-to-a-lower-kernel-version-in-ubuntu-linux/
• https://askubuntu.com/questions/1303568/downgrade-kernel-by-deleting-new-one-from-boot

sudo apt remove linux-headers-5.15.0-97-generic -y
sudo apt remove linux-headers-5.15.0-97 linux-headers-5.15.0-97-generic linux-modules-5.15.0-97-generic linux-image-unsigned-5.15.0-97-generic -y

sudo apt remove linux-headers-5.15.0-94-generic -y
sudo apt remove linux-headers-5.15.0-94 linux-headers-5.15.0-94-generic linux-modules-5.15.0-94-generic linux-image-unsigned-5.15.0-94-generic -y

# then reboot
sudo shutdown -r now

Post reboot check the version: uname -r

The longhorn volumes are also successfully mounted this time with no errors.

P.S. This known issue seems to be logged already in this Longhorn KB article

Today I also want to share how I started exploring Cilium’s other features along side open5gs’ 5G core. Luckily there is a link in the official documentation to available helm charts created and uploaded to github by Gradiant. Their tutorial includes existing working values.yaml files to integrate the open5gs applications together with the UERANSIM simulator that also comes in a separate helm chart provided by them.

In this demonstration we will gradually build up our network policies based on the information we will get from Hubble. We will start by applying a DNS rule to see the connections the NFs are trying to make, then we apply a cilium network policy to whitelist that flow. We will keep doing the same thing (check hubble, define CNP, repeat) until we are able to see a successful UE attach.

Preparing the environment

I recently completed the README of my HA K3S project utilizing Terraform and Ansible but overall the project is a work in progress. I still a have a lot of shell commands that I would need to convert to a module and later on clean up the scripts with roles. You can check the github project here or you can follow ahead.

You can also skip this part if you alread have a working k8s cluster, but here I also noted some points that you might find helpful if you are hosting your cluster at home with VMs.

First we will spin up VMs on Proxmox with the help of Terraform. Each VM will have 4 CPU and 4GB RAM with the CPU type set to host to pass-through the required feature set to run the latest version of mongodb. Each will have a 50GB disk and a single vNIC with the IP address and default gateway automatically configured with cloud-init.

Next we execute a preflight script to install some packages, do a couple of other stuff to prepare the environment for use with Ansible.

Then, we execute ansible-playbook -i inventory.yaml k3s-kubevip-helm-ciliumInstallHelmCli.yaml to install K3S, Kube-vip, helm, and Cilium. If you will be using the same ansible script in the k3s-ha repository, do note that Cilium’s kube-proxy replacement doesn’t support SCTP yet, though I have created the playbook for general K3S use. In this case you have to remove or comment this out in the playbook.

Cilium, Hubble, and Open5gs

Before we start deploying open5gs helm charts, let’s make it a prerequisite to install hubble. You can install cilium cli and hubble from your management host where you normally execute your kubectl commands. Note that by default hubble will not monitor the packets. For that to happen we can either annotate the pods or enforce a CiliumNetworkPolicy. We will go with the latter.

Apply the following CiliumNetworkPolicy, l4-egress-to-dns.yaml:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l4-egress-to-dns"
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/instance: open5gs
  egress:
    - toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*"

Here we will allow DNS traffic on port 53 for resolving the FQDNs of the services to be used with SBI. For now we will not configure the CNP (CiliumNetworkPolicy) for any other port. Note that the L4 DNS rule is also set on the egress of each pod. After the successful DNS resolution by any labeled pod, the succeeding connection attempts should be filtered and dropped by Cilium.

In another terminal window let’s run our hubble command to start monitoring TCP requests that will be initiated for the sending out HTTP POST messages for the NRF registration.

cilium hubble port-forward&
hubble observe -n default -f --protocol TCP --port 7777

Now that is set, let’s deploy our 5G core network:

helm install open5gs oci://registry-1.docker.io/gradiant/open5gs --version 2.2.0 --values https://gradiant.github.io/5g-charts/docs/open5gs-ueransim-gnb/5gSA-values.yaml

Soon enough we should start seeing TCP messages filtered by hubble with the verdict DROPPED since we only allowed DNS until this point. We should be able to see all core components are trying to connect to SCP on port 7777 and SCP trying to connect to NRF on the same port.

Let’s then proceed to create a new L7 CNP that allows this communication. We again configure this on the egress. Note that although CiliumNetworkPolicies are stateful, it only means you can expect the return path of the packet to be allowed as well. Here we need to ensure that the forward path will be allowed.

But just right before that let’s define a label for each core component that is expected to have an SBI. We can then match this label when we apply our network policy. Let’s patch each of those component with the label sbi: enabled

kubectl patch deployment open5gs-amf --patch '{"spec": {"template": {"metadata": {"labels": {"sbi": "enabled"}}}}}'
kubectl patch deployment open5gs-ausf --patch '{"spec": {"template": {"metadata": {"labels": {"sbi": "enabled"}}}}}'
kubectl patch deployment open5gs-bsf --patch '{"spec": {"template": {"metadata": {"labels": {"sbi": "enabled"}}}}}'
kubectl patch deployment open5gs-nrf --patch '{"spec": {"template": {"metadata": {"labels": {"sbi": "enabled"}}}}}'
kubectl patch deployment open5gs-nssf --patch '{"spec": {"template": {"metadata": {"labels": {"sbi": "enabled"}}}}}'
kubectl patch deployment open5gs-pcf --patch '{"spec": {"template": {"metadata": {"labels": {"sbi": "enabled"}}}}}'
kubectl patch deployment open5gs-scp --patch '{"spec": {"template": {"metadata": {"labels": {"sbi": "enabled"}}}}}'
kubectl patch deployment open5gs-smf --patch '{"spec": {"template": {"metadata": {"labels": {"sbi": "enabled"}}}}}'
kubectl patch deployment open5gs-udm --patch '{"spec": {"template": {"metadata": {"labels": {"sbi": "enabled"}}}}}'
kubectl patch deployment open5gs-udr --patch '{"spec": {"template": {"metadata": {"labels": {"sbi": "enabled"}}}}}'
kubectl patch deployment open5gs-upf --patch '{"spec": {"template": {"metadata": {"labels": {"sbi": "enabled"}}}}}'

Let’s start monitoring HTTP POST requests this time with:

hubble observe -n default -f --http-method POST

Apply the following CNP, l7-egress-to-scp.yaml, l7-egress-scp-to-nrf.yaml, making use of the label we just defined:

l7-egress-to-scp.yaml:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l7-egress-to-scp"
spec:
  endpointSelector:
    matchLabels:
      sbi: enabled
  egress: 
    - toEndpoints:
        - matchLabels:
            app.kubernetes.io/name: scp
      toPorts:
        # For SBI communication
        - ports:
            - port: "7777"
              protocol: TCP
          rules:
            http: [{}]

l7-egress-scp-to-core.yaml:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l7-egress-scp-to-core"
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: scp
  egress: 
    - toEndpoints:
        - matchLabels:
            sbi: enabled
      toPorts:
        # For SBI communication
        - ports:
            - port: "7777"
              protocol: TCP
          rules:
            http: [{}]

Let’s also configure another CNP to allow ingress HTTP traffic only if it’s coming from the SCP.

l7-ingress-from-scp.yaml:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l7-ingress-from-scp"
spec:
  endpointSelector:
    matchLabels:
      sbi: enabled
  ingress: 
    - fromEndpoints:
        - matchLabels:
            app.kubernetes.io/name: scp
        - matchLabels:
            sbi: enabled
      toPorts:
        # For SBI communication
        - ports:
            - port: "7777"
              protocol: TCP
          rules:
            http: [{}]

We will also allow port 27017 used by mongodb to be accessed by UDR, PCF, and WebUI.

l4-egress-populate-webui-udr-pcf-to-mongodb.yaml:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l4-egress-populate-to-mongodb"
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/component: populate
  egress:
    - toEndpoints:
        - matchLabels:
            app.kubernetes.io/name: mongodb
      toPorts:
        # For mongodb access
        - ports:
            - port: "27017"
              protocol: TCP
---
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l4-egress-webui-to-mongodb"
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: webui
  egress:
    - toEndpoints:
        - matchLabels:
            app.kubernetes.io/name: mongodb
      toPorts:
        # For mongodb access
        - ports:
            - port: "27017"
              protocol: TCP
---
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l4-egress-udr-to-mongodb"
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: udr
  egress:
    - toEndpoints:
        - matchLabels:
            app.kubernetes.io/name: mongodb
      toPorts:
        # For mongodb access
        - ports:
            - port: "27017"
              protocol: TCP
---
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l4-egress-pcf-to-mongodb"
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: pcf
  egress:
    - toEndpoints:
        - matchLabels:
            app.kubernetes.io/name: mongodb
      toPorts:
        # For mongodb access
        - ports:
            - port: "27017"
              protocol: TCP

kubectl apply -f l7-egress-to-scp.yaml -f l7-ingress-from-scp.yaml -f l7-egress-scp-to-nrf.yaml -f l4-egress-to-mongodb.yaml

HTTP2 messages look fine. Here we can see each application performing NF registration with NRF via SCP (acting as proxy).

We can also check the heartbeat messages by changing the http method to PATCH.

If you also want to see the DNS flow you can execute hubble observe -n kube-system -f --port 53.

Next we deploy the UERANSIM GNB and UE pods.

helm install ueransim-gnb oci://registry-1.docker.io/gradiant/ueransim-gnb --version 0.2.6 --values https://gradiant.github.io/5g-charts/docs/open5gs-ueransim-gnb/gnb-ues-values.yaml

If you follow the instructions you should expect a new tunnel interface inside the UE pod. But you most likely won’t and if you check the gNB logs, you should see an error “Cell selection failure, no suitable or acceptable cell found”.

We can check first if gNB is able to connect successfully to the AMF. gNB connects via SCTP so we can filter that with hubble:

hubble observe -n default -f --protocol sctp

It’s getting dropped due to the L7 ingress rule we applied. You might have thought for a while why it’s dropping the packet if it’s an L7 rule even if SCTP is on L4. Network policies actually work as a whitelist, so everything else not part of the policies we are applying gets dropped by default the moment you apply the rule on either the ingress or egress direction.

Let’s go apply the SCTP rule!

l4-ingress-amf-from-gnb-sctp.yaml:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l4-ingress-from-gnb-sctp"
  namespace: default
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: amf
  ingress:
    - fromEndpoints:
        - matchLabels:
            app.kubernetes.io/name: ueransim-gnb
      toPorts:
        - ports:
          - port: "38412"
            protocol: SCTP

OK, the SCTP association looks established now.

We can also see successful UE registration:

But still no tunneled interface. Let’s check for other dropped packets:

hubble observe -n default -f --verdict DROPPED

What’s getting dropped this time? We forgot about the SMF to UPF communication. From UPF we should also allow all ports to any destination at least on the egress side. Apart from this the gNB should also be allowed to communicate with the UPF for use traffic.

l4-egress-smf-to-upf.yaml:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l4-egress-smf-to-upf"
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: smf
  egress:
    - toEndpoints:
        - matchLabels:
            app.kubernetes.io/name: upf
      toPorts:
        # For SMF-UPF internet traffic
        - ports:
            - port: "8805"
              protocol: UDP

l4-ingress-upf-from-smf.yaml:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l4-ingress-upf-from-smf"
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: upf
  ingress:
    - fromEndpoints:
        - matchLabels:
            app.kubernetes.io/name: smf
      toPorts:
        # For SMF-UPF internet traffic
        - ports:
            - port: "8805"
              protocol: UDP

l4-ingress-upf-from-gnb.yaml:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l4-ingress-upf-from-gnb"
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: upf
  ingress:
    - fromEndpoints:
        - matchLabels:
            app.kubernetes.io/component: gnb
      toPorts:
        # For SMF-UPF internet traffic
        - ports:
            - port: "2152"
              protocol: UDP

l3-egress-upf-to-any.yaml:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l3-egress-upf-to-any"
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: upf
  egress:
    - toCIDRSet:
        - cidr: 0.0.0.0/0
          # except private CIDR
          except:
            - 10.0.0.0/8
            - 172.16.0.0/12
            - 192.168.0.0/16

For the UPF we allow all IP addresses except for the private CIDR blocks.

Let’s try one more time. It should be successful now. This time let’s monitor all traffic related to 8.8.8.8: hubble observe -f --ip 8.8.8.8. From the UE we initiate with ping -I uesimtun0 8.8.8.8.

For the last part we try to attach an unprovisioned SUPI. This time we can filter http status code 404 to verify:

hubble observe -f -n default --http-status 404

Deploy the UE pods with unknown SUPI:

helm install -n default ueransim-ues-not-defined oci://registry-1.docker.io/gradiant/ueransim-ues \
--set gnb.hostname=ueransim-gnb \
--set count=2 \
--set initialMSISDN="0000000003" \

We can see here 404 response received at UDM from UDR, AUSF from UDM, and at AMF back from AUSF.

To sum up all the cilium network policies we applied, you can refer to the below list:

NAME
l3-egress-upf-to-any
l4-egress-pcf-to-mongodb
l4-egress-populate-to-mongodb
l4-egress-smf-to-upf
l4-egress-to-dns
l4-egress-udr-to-mongodb
l4-egress-webui-to-mongodb
l4-ingress-amf-from-gnb-sctp
l4-ingress-upf-from-gnb
l4-ingress-upf-from-smf
l7-egress-scp-to-core
l7-egress-to-scp
l7-ingress-from-scp

Cilium (with Hubble) is awesome

To conclude, we have demonstrated how we can make use of network policies to add another layer of security to our network. We managed to create new policies, applying them on the ingress or egress wherever necessary, making use of L3, L4, and L7 rules to whitelist the traffic. At the same time we also utilized hubble to have a good peek into the network layer, allowing us to understand how network policies work and see which packets are getting allowed, dropped, or rejected. We made use of different filters working on protocol level, http method, http status, port. This demonstration doesn’t cover the full capability of hubble and there is a ton of other different filters you can use for observability. And the good thing is the project is in very active development and the guys over at Isovalent seem to add cool new features every time they have a new release. I will not be surprised if Cilium becomes the primary choice for CNI for any new project. Stay tuned for upcoming posts as we explore more other Cilium features!

I got everything working eventually after multiple iterations of trial and error, and upgrading my traefik helm release countless times. And after finding this Traefik blog post where they explain how it all actually works and can be configured. At this point I’ve migrated all of my docker applications into my k8s cluster, and reading through the Traefik documentation again, it just makes a lot more sense now.

I guess the most important bit of the documentation is the part where they mentioned the requirement of using Cert-manager when using Traefik Proxy in Kubernetes. Quoting from the documentation:

LetsEncrypt Support with the Ingress Provider By design, Traefik is a stateless application, meaning that it only derives its configuration from the environment it runs in, without additional configuration. For this reason, users can run multiple instances of Traefik at the same time to achieve HA, as is a common pattern in the kubernetes ecosystem. When using a single instance of Traefik Proxy with Let’s Encrypt, you should encounter no issues. However, this could be a single point of failure. Unfortunately, it is not possible to run multiple instances of Traefik 2.0 with Let’s Encrypt enabled, because there is no way to ensure that the correct instance of Traefik receives the challenge request, and subsequent responses. Previous versions of Traefik used a KV store to attempt to achieve this, but due to sub-optimal performance that feature was dropped in 2.0. If you need Let’s Encrypt with high availability in a Kubernetes environment, we recommend using Traefik Enterprise which includes distributed Let’s Encrypt as a supported feature. If you want to keep using Traefik Proxy, LetsEncrypt HA can be achieved by using a Certificate Controller such as Cert-Manager. When using Cert-Manager to manage certificates, it creates secrets in your namespaces that can be referenced as TLS secrets in your ingress objects.

Since we are working in K8s, almost everything you can think of can be abstracted. For this instance, the TLS configuration is abstracted from the helm custom values yaml file and instead can be pegged to an Ingress resource when using Cert-manager.

To summarize in a few lines:

• Install Traefik so it can act your Ingress controller.
• Install Cert-manager so it can monitor your Ingress resources, and based on the TLS configuration and with some annotations, it will automatically initiate the DNS challenge, create the TLS certificates sand save them as a secret in the application namespace.
• Create Ingress resources to access your backend servcies while at the same time terminate HTTPS traffic (with Traefik).

Do note that while Traefik is an ingress controller and comes with its own IngressRoute CRD, it cannot be used together with Cert-manager at the moment. In this case we have to use the default Kubernetes Ingress resource.

In this post I will show how I configured Traefik and Cert-manager (patterned to the Traefik blog post!) to be used with Cloudflare. I will also show how you can have another layer of security by having multiple entry points and explicitly defining one during the ingress configuration. Though before proceeding, do note that I assume you already have an idea how to configure dynamic DNS and port-forwarding on Cloudflare and on your router.

Traefik

Recently I’ve started using ArgoCD for managing all my applications including those that are installed via helm charts. For now we will install Traefik using helm.

helm repo add traefik https://traefik.github.io/charts # Add the Traefik repository
helm repo update # Update the repositories
helm show values traefik/traefik > values.yaml # Download the custom values yaml

I would recommend you to browse through the custom values yaml file to get familiarized with the available configuration and other features of Traefik. If you want to get started quickly then you can follow the below yaml configuration:

values.yaml

deployment:
  enabled: true
  kind: Deployment
  replicas: 1
  terminationGracePeriodSeconds: 60
  minReadySeconds: 0

ingressRoute:
  dashboard:
    enabled: true
    matchRule: Host(`traefik.yownowndomain.com`) && PathPrefix(`/dashboard`) || Host(`traefik.yownowndomain.com`) && PathPrefix(`/api`)
    entryPoints: ["traefik","websecure"]

additionalArguments:
  - "--serversTransport.insecureSkipVerify=true"
  - "--providers.kubernetesingress.ingressendpoint.publishedservice=traefik/traefik"

ports:
  traefik:
    port: 9000
    expose: true
    exposedPort: 9000
    protocol: TCP
  web:
    port: 80
    expose: true
    exposedPort: 80
    protocol: TCP
    redirectTo: websecure
  webext:
    port: 8080
    expose: true
    exposedPort: 8080
    protocol: TCP
    redirectTo: websecureexternal
  websecure:
    port: 443
    expose: true
    exposedPort: 443
    protocol: TCP
    http3:
      enabled: false
    tls:
      enabled: true
      options: ""
      certResolver: ""
    middlewares:
      - default-default@kubernetescrd
  websecureext:
    port: 8443
    expose: true
    exposedPort: 8443
    protocol: TCP
    http3:
      enabled: false
    tls:
      enabled: true
      options: ""
      certResolver: ""
    middlewares:
      - default-default@kubernetescrd
  metrics:
    port: 9100
    expose: false
    exposedPort: 9100
    protocol: TCP
  dnsovertls:
    port: 8853
    expose: true
    exposedPort: 853
    protocol: TCP
    tls:
      enabled: false
    middlewares:
      - default-default@kubernetescrd
tlsOptions:
  default:
    minVersion: VersionTLS12
    cipherSuites:
      - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
      - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
      - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

service:
  enabled: true
  single: true
  type: LoadBalancer
  annotations:
    io.cilium/lb-ipam-ips: 10.80.0.2
  labels:
    exposedExternal: "yes"

In my custom yaml file notice that I have four different entryPoints defined. I configured it like this to have separate incoming streams between my internal-only applications against the internet-exposed ones. Having a common entryPoint for both internal and externally exposed applications is a security flaw since hackers might still be able to access your internal-only apps if they guessed the sub-domain and locally configured CNAME forwarding (they don’t need access to your DNS configuration).

Once your custom values yaml file is ready. Hit up the helm install command and proceed to the next step to create a Middleware resource.

default-middleware.yaml

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: traefik
spec:
  headers:
    customResponseHeaders:
      X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
      X-Forwarded-Proto: "https"
      server: ""
    customRequestHeaders:
      X-Forwarded-Proto: "https"
    sslProxyHeaders:
      X-Forwarded-Proto: "https"
    referrerPolicy: "same-origin"
    hostsProxyHeaders:
      - "X-Forwarded-Host"
    contentTypeNosniff: true
    browserXssFilter: true
    forceSTSHeader: true
    stsIncludeSubdomains: true
    stsSeconds: 63072000
    stsPreload: true

helm install traefik/traefik -f values.yaml

kubectl apply -f default-middleware.yaml

Now that you have Traefik up and running, expose the webext and websecureext entryPoints on your router. Simply port-forward 80 to 8080, and 443 to 8443.

Cloudflare

Going to Cloudflare, you have to configure an access token to be used later on when configuring Cert-manager. Login to your Cloudflare account. Navigate to My Profile > API Tokens > Create Token.

Under Permissions, set Zone - DNS - Edit

Under Zone Resources, set Include - Specific zone - “yourowndomain.com”

E.g.

Create and save your token somewhere safe.

Cert-manager

Cert-manager can be installed using helm. Quoting from the official documentation, Cert-Manager together with the CRDs can be installed with the following commands:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.13.2 \
  --set installCRDs=true

After installation, it’s time to configure cert-manager for use with Cloudflare. You simply just have to create a secret containing your cloudflare email and API token:

cloudflare-api-token-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token
  namespace: cert-manager
type: Opaque
stringData:
  email: example@domain.com
  apiToken: AbcD321eFg456_

Next is to create a ClusterIssuer resource. This is one of the CRDs that will come with cert-manager. Here you have to specify your email to be used with Let’s Encrypt as well as the secret you created in the previous step.

clusterissuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cluster-issuer
  namespace: cert-manager
spec:
  acme:
    email: example@domain.com
    # Prod
    server: https://acme-v02.api.letsencrypt.org/directory
    # We use the staging server here for testing to avoid hitting
#    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # if not existing, it will register a new account and stores it
      name: letsencrypt-acc-key
    solvers:
      - dns01:
      # The ingressClass used to create the necessary ingress routes
          cloudflare:
            email: example@domain.com
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: apiToken
        selector:
          dnsZones:
            - 'yourowndomain.com'

If you are doing this the first time, I strongly suggest you first use the staging server of Let’s Encrypt and ensure certificate issuance works fine. To switch to the porduction server all you have to do is just re-create the resource and point to the new URL. Do note in the example above, the resource is pointing to the production server.

Exposing your application

To test this we create a simple nginx deployment that is exposed within the cluster and an ingress resource.

nginx.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
    - name: "http"
      port: 80
      targetPort: 80
  type: ClusterIP

When creating the ingress manifest, you have to ensure that you have the annotation: cert-manager.io/cluster-issuer: "letsencrypt-cluster-issuer".

Apart from this if you want your application to be accessed from a specific traefik entryPoint, then you need to explicitly mention this as another annotation. By default the ingress will be allowed from any entryPoint if you don’t specify this. In the below example manifest, I am keeping this annotation commented out so I can access my Nginx app from both internal and external entryPoints.

nginx-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-cluster-issuer"
    # traefik.ingress.kubernetes.io/router.entrypoints: websecureext
spec:
  tls:
    - hosts:
        - nginx.yownowndomain.com
      secretName: tls-nginx-ingress
  rules:
    - host: nginx.yownowndomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nginx
                port:
                  number: 80

Once you apply these manifests, simply do a kubectl get certificate. This will show that a new certificate and secret resource got automatically created.

kubectl get certificate
NAME                READY   SECRET              AGE
tls-nginx-ingress   False   tls-nginx-ingress   61s

Wait for a few minutes. You should see that cerficicate ready status should be set to True.

You can also describe this certificate to see the events.

Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    91s   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  90s   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "tls-nginx-ingress-2mvlp"
  Normal  Requested  90s   cert-manager-certificates-request-manager  Created new CertificateRequest resource "tls-nginx-ingress-1"
  Normal  Issuing    11s   cert-manager-certificates-issuing          The certificate has been successfully issued

To expose this on the internet, you will have to go back to Cloudflare and configure a CNAME forwarding to the sub-domain that points to your router’s external IP, given that you have already configured dynamic DNS.

You can also test this by configuring a CNAME entry on your local network’s DNS.

Once you ge this working you should be able to see a valid certificate when you access your nginx page.

The certificate will have a validity of 3 months but you don’t have to worry since Cert-manager will renew this automatically.

Sep 29 14:54:58 pve1 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:
                               TDH                  <22>
                               TDT                  <bb>
                               next_to_use          <bb>
                               next_to_clean        <22>
                             buffer_info[next_to_clean]:
                               time_stamp           <100cb7acb>
                               next_to_watch        <23>
                               jiffies              <100cb7d99>
                               next_to_watch.status <0>
                             MAC Status             <80083>
                             PHY Status             <796d>
                             PHY 1000BASE-T Status  <3800>
                             PHY Extended Status    <3000>
                             PCI Status             <10>
...
...
...
Sep 29 14:55:07 pve1 kernel: e1000e 0000:00:1f.6 eno1: Reset adapter unexpectedly
Sep 29 14:55:07 pve1 kernel: vmbr0: port 1(eno1) entered disabled state
Sep 29 14:55:07 pve1 kernel: vmbr0v20: port 1(eno1.20) entered disabled state
Sep 29 14:55:07 pve1 kernel: vmbr0v30: port 1(eno1.30) entered disabled state
Sep 29 14:55:11 pve1 kernel: e1000e 0000:00:1f.6 eno1: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Sep 29 14:55:11 pve1 kernel: vmbr0: port 1(eno1) entered blocking state
Sep 29 14:55:11 pve1 kernel: vmbr0: port 1(eno1) entered forwarding state
Sep 29 14:55:11 pve1 kernel: vmbr0v20: port 1(eno1.20) entered blocking state
Sep 29 14:55:11 pve1 kernel: vmbr0v20: port 1(eno1.20) entered forwarding state
Sep 29 14:55:11 pve1 kernel: vmbr0v30: port 1(eno1.30) entered blocking state
Sep 29 14:55:11 pve1 kernel: vmbr0v30: port 1(eno1.30) entered forwarding state

When I saw these messages I thought my switch was acting up. I was almost ready to purchase a new switch but then upon further googling I landed in this forum post.

Turns out this specific model is known to be freezing from time to time when segmentation is done by the NIC. This comes enabled by default when installing Proxmox. A quick fix suggeested in the post which worked for me was to simply disable the TSO and GSO flags on the specific interface. On runtime this can be disabled by executing ethtool -K eno1 tso off gso off.

To make the changes permenant, the following line should be added in the /etc/network/interfaces file:

post-up /usr/bin/logger -p debug -t ifup "Disabling segmentation offload for eno1" && /sbin/ethtool -K $IFACE tso off gso off && /usr/bin/logger -p debug -t ifup "Disabled offload for eno1"

e.g.

One more issue I faced was very slow throughput when playing videos from Jellyfin.

Noticed that throughput was very low regardless whether I’m using the docker traefik or the one in my kubernetes cluster, while the original traefik container on my Unraid

After drilling down that issue seems to be specific to my Proxmox node, I checked whether the 1Gbit/s speed was properly negotiated as suggested in this other forum post.

Executing ethtool eno1 showed me the following output:

root@pve1:~# ethtool eno1
Settings for eno1:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	Supported pause frame use: No
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10baseT/Full
	Advertised pause frame use: No
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 10Mb/s
	Duplex: Full
	Auto-negotiation: on
	Port: Twisted Pair
	PHYAD: 1
	Transceiver: internal
	MDI-X: on (auto)
	Supports Wake-on: pumbg
	Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
	Link detected: yes

Disabled auto-negotiation and configured 1Gbit manually: ethtool -s eno1 speed 1000 duplex full autoneg off

root@pve1:~# ethtool eno1
Settings for eno1:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	Supported pause frame use: No
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  1000baseT/Full
	Advertised pause frame use: No
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Auto-negotiation: on
	Port: Twisted Pair
	PHYAD: 1
	Transceiver: internal
	MDI-X: off (auto)
	Supports Wake-on: pumbg
	Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
	Link detected: yes

After having these in place, my cluster has been running perfectly fine for almost a month now with nohiccups!

At this time of writing, so far only two services have been migrated from the docker environment into the new K8S cluster. That is my DNS which is also replaced now by AdguardHome (sorry Pihole!), and Traefik, which terminates all external HTTP communication incoming to my Homelab. And for the rest of the other services, those will be migrated in the coming weeks. For now what is important is I have HTTPS running perfectly fine with valid CA certificates even with dockerized services in the backend.

I also want to note here that during the learning process, whatever services that were running in the docker setup, all of those were nearly untouched since I was working inside a staging environment, keeping rigorous activities of countless create-delete of resources in isolation from my home production network.

That said, I received zero complaints from the wife except for the time I lost connectivity to the cluster when I thought everything was already stable. Lol more on this later.

In summary below opensource tools have been used for this project.

• Proxmox - Hypervisor (installed in 3 devices)
• Terraform - VM instantiation
• Ansible - VM configuration, Kubernetes installation
• K3s - Kubernetes platform
• Kube-vip - LB for Kube API
• Longhorn - Block storage
• Cilium - CNI
• Traefik - Reverse Proxy
• Cert-manager - SSL certificate manager

Terraform and Ansible play the major role of orchestrators allowing for quick creation and deletion of virtual resources putting Infrascture-as-Code into practice. The rest are supplementary to learning IaC.

Side node: Unfortunately Terraform will be transitioning away from a completely opensource license and soon will be sitting behind a BSL. It might be a good idea to switch to OpenTF (or OpenTofu) which is a fork of Terraform and now officially a CNCF project as well.

The setup

All development activities were done in a staging environment. This is where the unaccounted events of VM creation-deletion have transpired from testing out the Terraform scripts until the very end when installing Longhorn and Cilium with Ansible, and even when playing around with Traefik in K8s.

The staging environment was deployed on my Unraid box and as for the production build, I am using Tiny PCs that house a total of 32GB RAM each allowing more room for other VMs that I might need in the long run. To give a glimpse of the production HW:

I was also able to source data center-grade Intel S3510 SSD drives from the second-hand market. These should help in the reliability department of these nodes that will be running data replication because of Longhorn. Currently, these are housed in 2 of 3 nodes. Still looking to get another one and pop it in PVE1.

One container to rule them all

Both the Terraform and Ansible controllers are running from a LXC container deployed on PVE1. This is where I do all of the development work. I am running a docker instance of VS code on the same container which enables me to create and modify code easily. The same container is used to communicate with the production environment.

One of the best advantages of using an LXC container is that it’s so lightweight you can easily backup the environment anytime. I confirm Terraform and Ansible works well with the Ubuntu 22.04 template that comes with Proxmox.

Terraform and Ansible

Learning Terraform wasn’t so bad. The Proxmox provider documentation from Telmate is enough to get you started and with the declarative style that is used to write Terraform language, it just makes the learning process a lot more easier to deal with. This is also the part where I had the least modifications and time spent the least.

Once I got the VMs up and working, next thing I worked on is Ansible. Now this is the part where I spent the biggest chunk of my time. I literally am unable to count the times I had to execute terraform destroy and terraform apply to re-create the VMs and test out my Asnbile playbooks. To be fair starting with Ansible wasn’t really hard. It was the amount of automation I wanted to go with that later on will prove useful. Though I have to admit the playbooks are rather simple and which others might find lacking in terms of best practices. But hey, I have to start from somewhere!

The five commands to have a complete working k3s cluster:

terraform apply
ansible-playbook -i inventory.yaml preflight.yaml
ansible-playbook -i inventory.yaml logical-volume-create.yaml
ansible-playbook -i inventory.yaml k3s-kubevip-helm-ciliumInstallHelmCli.yaml
ansible-playbook -i inventory.yaml longhorn-install.yaml

The whole process takes about more or less 15 minutes. Terraform creates 3 VMs, one on each proxmox node. Each VM is set 4 vCPU, 12GB RAM, 50GB boot disk + 200GB longhorn disk. preflight.yaml defines the ssh keys for passwordless authentication and installs the necessarry packages to run k3s-related software. logical-volume-create.yaml as its name suggests, creates the logical volume to be used for longhorn. k3s-kubevip-helm-ciliumInstallHelmCli.yaml installs k3s, kube-vip, helm, and Cilium altogether sequentially. And last but not the least, longhorn-install.yaml installs longhorn via helm.

Cilium

While Calico would be the go-to CNI for most, I opted to go for Cilium. The main reason for this is to start learning eBPF and have a grasp how things move within the kernel space. The installation was straightforward but to get it working in properly was a challenge.

I was able to make BGP control plane work during the initial phase and when I thought everything was already stable, I then started seeing BGP peers getting dropped and at some point even lost connectvivity to my DNS hosted in the cluster. After adding into the parameters one by one, I was able to make it work without disconnects.

helm install cilium cilium/cilium --version 1.14.2 \
--namespace kube-system \
--set bgpControlPlane.enabled=true \
--set tunnel=disabled \
--set ipam.operator.clusterPoolIPv4PodCIDRList=10.42.0.0/16 \
--set kubeProxyReplacement=true \
--set k8sServiceHost={{ _k8sServiceHost }} \
--set k8sServicePort=6443 \
--set routingMode=native \
--set autoDirectNodeRoutes=true \
--set ipv4NativeRoutingCIDR=10.42.0.0/16 \
--set loadBalancer.mode=dsr \
--set ipv4.enabled=true \
--set prometheus.enabled=true \
--set operator.prometheus.enabled=true \
--set hubble.enabled=true \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}"

kubeProxyReplacement is enabled to make use of eBPF instead of the traditional way of iptables. loadBalancerMode is also set to DSR making it possible for nodes to respond directly to the sender, avoiding the need to go via the return path. routingMode set to native and autoDirectNodeRoutes is set to true since all nodes are connected via the same L2 network. You can read more on Cilium routing here. Prometheus and Hubble are also enabled so I can touch on them later once I get more time.

cilium-bgp-peering-policy.yaml:

apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
  name: 00-bgp-peering-policy
spec: # CiliumBGPPeeringPolicySpec
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: k8s-master-0-dev
  virtualRouters: # []CiliumBGPVirtualRouter
    - localASN: 65090
      exportPodCIDR: false
      serviceSelector:
        matchLabels:
          exposedExternal: "yes"
      neighbors: # []CiliumBGPNeighbor
        - peerAddress: '10.20.0.1/32'
          peerASN: 65000
          eBGPMultihopTTL: 10
          connectRetryTimeSeconds: 120
          holdTimeSeconds: 90
          keepAliveTimeSeconds: 30
          gracefulRestart:
            enabled: true
            restartTimeSeconds: 120
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
  name: 01-bgp-peering-policy
spec: # CiliumBGPPeeringPolicySpec
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: k8s-master-1-dev
  virtualRouters: # []CiliumBGPVirtualRouter
    - localASN: 65091
      exportPodCIDR: false
      serviceSelector:
        matchLabels:
          exposedExternal: "yes"
      neighbors: # []CiliumBGPNeighbor
        - peerAddress: '10.20.0.1/32'
          peerASN: 65000
          eBGPMultihopTTL: 10
          connectRetryTimeSeconds: 120
          holdTimeSeconds: 90
          keepAliveTimeSeconds: 30
          gracefulRestart:
            enabled: true
            restartTimeSeconds: 120
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
  name: 02-bgp-peering-policy
spec: # CiliumBGPPeeringPolicySpec
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: k8s-master-2-dev
  virtualRouters: # []CiliumBGPVirtualRouter
    - localASN: 65092
      exportPodCIDR: false
      serviceSelector:
        matchLabels:
          exposedExternal: "yes"
      neighbors: # []CiliumBGPNeighbor
        - peerAddress: '10.20.0.1/32'
          peerASN: 65000
          eBGPMultihopTTL: 10
          connectRetryTimeSeconds: 120
          holdTimeSeconds: 90
          keepAliveTimeSeconds: 30
          gracefulRestart:
            enabled: true
            restartTimeSeconds: 120

As for configuring BGP on OPNsense, all I had to do was download the os-frr plugin and apply the configuration according to the Cilium BGP resources.

Once BGP is configured, I then configured CiliumLoadBalancerIPPool and configured the serviceSelector there so any service with a matching label will be assigned an external IP.

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: externalpool
spec:
  cidrs:
    - cidr: 192.168.100.0/27
  disabled: false
  serviceSelector:
    matchLabels:
      exposedExternal: "yes"

To assign services with external IPs, all you have to do is ensure two things (third one is optional):

1. The service should have a label matching what you defined in your CiliumLoadBalancerIPPool.
2. The service should be of type LoadBalancer.
3. For static external IP assignment, the service should have an annotation of io.cilium/lb-ipam-ips followed by the IP address.

apiVersion: v1
kind: Service
metadata:
  annotations:
    io.cilium/lb-ipam-ips: 192.168.100.7
  labels:
    exposedExternal: "yes"
  name: adguard-ui
  namespace: adguard
...
...
spec:
  type: LoadBalancer
...
...

When all of the above are applied, you should now see an external IP assigned to your service. e.g.

❯ k -n adguard get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
adguard      LoadBalancer   10.43.254.122   192.168.100.8   53:30176/UDP   27d
adguard-ui   LoadBalancer   10.43.38.101    192.168.100.7   80:32108/TCP   27d

You should also be able to check the BGP peering status as well as the learned routes in OPNsense.

Longhorn

Before deciding to go for Longhorn, I was was trying to make Piraeus datastore work (FOSS version of Linstor storage for Kubernetes). I got it to work with ReadWriteOnce but the moment I tried to test ReadWriteMany(RWX), it just wouldn’t. On top of this it also felt that there was a steep learning curve to understand how Piraeus work on a deeper level in case I had to do extra troubleshooting in the future.

Longhorn on the other hand worked well out of the box. Testing out RWX by re-creating a pod on a different node worked well too and since Longhorn seems to use NFS to support this feature, accessing the volume from something external to the cluster e.g. from a VM works out of the box. The only thing is that there seems to be an open issue with Cilium when exposing the volume externally. When I try to mount the share to a VM, I do experience slowdowns when opening a file with vim or even when just browsing through the directories.

Longhorn makes use of a 200Gi second volume that was declared in the Terraform script. The above snapshot shows the amount of available volume.

For reference, below is the playbook task to install longhorn:

- name: install longhorn
  kubernetes.core.helm: 
    name: longhorn
    chart_ref: longhorn/longhorn
    release_namespace: longhorn-system
    create_namespace: true
    update_repo_cache: true
    set_values:
      - value: service.ui.type=LoadBalancer
      - value: defaultSettings.defaultDataPath=/longhorn_vol
      - value: defaultSettings.defaultReplicaCount=3

Exposing services

I took more time than expected when I was trying to expose the services with Traefik now running in Kubernetes. In the docker setup, the configuration was pretty straightforward with minimal reading required of the documentation. Whereas when running treafik in k8s, it came to the point that it already felt like I was digging my own grave with all the research and testing.

Eventually I got it to work by going with the base installation and slowly inching my way through the custom values in the yaml file. Once I got the middleware (for additional security headers) and TLS working via cert-manager, all I had to do was create individual ingress resources for each of the services I wanted to expose. The certificates are automatically created and managed by Cert-manager.

One thing to note, in Docker, the certificates can be managed by Traefik. But when using Traefik in a K8s environment, to make use of Let’s Encrypt, the only option is to use Cert-manager which can only be paired up with the default Kubernetes Ingress resource. Traefik’s Ingress CRD doesn’t support this at the moment.

Below is a sample Ingress resource to reach the Adguard GUI from external to the cluster:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: adguard-ui
 namespace: adguard
 annotations:
   cert-manager.io/cluster-issuer: "letsencrypt-cluster-issuer"
spec:
 tls:
   - hosts:
       - adguard.su-root.net
     secretName: tls-adguard-ui-ingress-dns
 rules:
   - host: adguard.su-root.net
     http:
       paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: adguard-ui
               port: 
                 number: 80