Second, the old subscription only had IPv4. Don’t get me wrong, IPv4 isn’t a bad thing. In fact, a good majority of the internet still runs on IPv4. But recently, I’ve gained interest in exploring the capabilities and other features that come with IPv6. More on this later, but the main trigger point for this is because IPv6 is already widely used in Japan. If I’m not mistaken, IPv6 adoption has been enforced by the Japan Ministry of Communications for some time now.

Third, since the beginning of last year, I’ve taken on a new role focused on optimizing customer experience performance, where speed is one of the main concerns. To better support this and overcome the limitations of my current toolset, having a speed test server (with multi-gigabit bandwidth) that I have full control of is a fundamental first step!

Hardware

Certain Lenovo Tiny PCs are popular machines for having a PCIe slot that can be fitted with 10G NICs, so that’s what I went with. I was able to get a secondhand one with an Intel i5-8500T 6C CPU. This should be more than enough for my needs, even if I wanted to host other services on this machine. As for the NIC, I had the option of going with Mellanox or Intel ones. To be more specific, I was choosing between a Mellanox ConnectX-4 and an Intel X710-DA2. Mellanox seems to be more popular in terms of compatibility and stability, but it’s also known to have elevated power consumption since it’s not able to achieve higher C-states even when idle. The X710-DA2, on the other hand, was known to have compatibility issues, especially when using the OEM-branded ones. I went with Intel to have peace of mind knowing that I tried my best to save electricity costs (as if all the other machines in the homelab justify all the power they require!).

Unfortunately, I assumed the compatibility issue was something generic and could easily be fixed by cross-flashing the original Intel firmware. While this might have worked for others, this isn’t the case when specifically using a Dell-branded X710 NIC with a Lenovo M920q Tiny PC. So, if anyone intends to follow this route, take note! I learned this the hard way because I purchased a Dell-branded one and went through the whole cross-flashing process only to find out that my machine will only boot up successfully if it’s coming from an unplugged state. Yup, that means it will only boot up after unplugging and plugging the power cable. Anyway, I ended up purchasing an Intel-branded one and just flashed it with the latest firmware. For anyone interested, the articles I followed are also shared at the end of this post!

Lenovo M920q Tiny PCs require a PCIe riser for plugging in your cards. This can easily be bought for about ¥2000~3000 online. There are different part numbers in terms of the supported bus speed, so I had to ensure I got the one with P/N: 01AJ940, which supports x8.

As for the 10G switch, for the time being, I got a 4x2.5 + 2x10G switch from Horaco (AliExpress). I already have an 8x2.5G switch from the same brand and it’s been rock stable. There was no really good reason for me to try out another brand. If there was one thing where you have to manage your risk, it’s when purchasing networking equipment from AliExpress!

10G SFP+ to copper transceivers are known to run hot. And because I keep my networking equipment in an unventilated closet, I had to shed an extra couple of bucks for the premium ones—the version that supports up to 80m of copper. For the transceivers, I got them from a brand called ZYOPM.

Managing the temperature inside an enclosed space is a priority, so I had to make sure the temps are at a minimum whenever possible. For the upstream connection of my switch to the router, I used a DAC cable. I’ve never used one before and actually thought of using optical transceivers in the beginning.

On to the Wireless AP, I also upgraded from my not-so-old TP-Link EAP610 to an EAP773. The former only supported Wi-Fi 6 over 2.4/5 GHz bands with 1G upstream. For “future-proofing,” at least for the next few years, and to make use of the upgraded upstream, I needed something which at least supported Wi-Fi 6E over 6 GHz.

I also had to get a PoE++ 90W injector since the new switch didn’t support PoE+.

Software

There are mixed opinions on whether you should virtualize your router or not. But if you live somewhere where space is a luxury, then you already have the best reason to go with virtualization (but regardless of that, I just really like complicating things, lol). So of course, we go back to our good old friend, Proxmox. The last time I had to spin up a new Proxmox machine was about 2 years ago when I started using CEPH as my default backend storage. A lot of good things must have been added within these 2 years, but I just really needed an easily managed KVM host. No one can beat Proxmox on that.

As for the router itself, I made the big switch from OPNsense to OpenWRT. And I couldn’t be happier. I’ve had OPNsense for about 3 years now and while I didn’t really complain about anything since it was more of a “set and forget” setup, whenever I had to modify something in the configuration, sometimes it just felt like I had to explore and familiarize myself again. With OpenWRT, the GUI just felt a bit more natural and warm to my eyes. Maybe it’s because of fewer sections or tabs to go through, or, I don’t know, configuration just seemed a little more straightforward this time compared to when I just started with the other.

But in fact, I don’t think I would have bothered checking out OpenWRT if only OPNsense supported MAP-E connections used by my new ISP. There is a way to get it working with OPNsense, but it was more of a workaround than a natively supported feature.

My choice of ISP

The primary factor in choosing the ISP is the monthly cost. The switch to 10G wouldn’t be justifiable if we had to pay 50% more than our old subscription. Second would be the option to have a fixed IPv4 address since I host some services for family and friends. Fortunately, I came across a post on Reddit suggesting En Hikari. En Hikari uses NTT FLET’S HIKARI as the backbone provider, so you can be assured of the same quality of connectivity used by most ISPs. At the time of writing, the monthly cost is about JPY 4,917 (tax included) plus an additional JPY 770 for the optional fixed IPv4 address. That’s a total of JPY 5,687, which comes out even cheaper than our current basic 1G subscription at JPY 5,720 (dynamic IPv4 address only).

Preparations

When I got the Intel X710 card, it only had v6.01 installed. I didn’t find a way to upgrade directly from v6.01 to the latest one (v9.56 as of this writing) and had to go through it stepwise. The upgrade path in my case was from 6.01 –> 8.6 –> 9.0 –> 9.10 –> 9.56. All done under Debian (Proxmox).

My Intel card came with unlocked vendor support, so I didn’t have to run the unlocker script. I guess this is only applicable to the OEM versions.

I actually ran into some errors initially that prevented the second Ethernet port from functioning. Honestly, I hit a wall and gave up on it for the night. Then, I’m not sure how, but it just got resolved the following morning after a restart.

After a few more rounds of restarts and going through kernel dmesg logs, I finally decided to go with a fresh install of OpenWRT. For some reason, I couldn’t figure out how to create the VLAN interfaces from the GUI. I ended up with a successful attempt when I tried to do it from the text file configuration. To those who are in the same boat, below is a sample /etc/config/network for configuring VLANs.

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd00:7808:88c3::/48'
	option packet_steering '1'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'

config interface 'lan'
	option device 'br-lan.1'
	option proto 'static'
	option ipaddr '192.168.0.1'
	option netmask '255.255.255.0'
	option ip6assign '64'
	list dns '192.168.0.1'
	option ip6hint '00'

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'
	option keepalive '5 10'

config interface 'wan6'
	option device 'eth1'
	option proto 'dhcpv6'
	option reqaddress 'try'
	option reqprefix 'auto'
	option norelease '1'

config bridge-vlan
	option device 'br-lan'
	option vlan '1'
	list ports 'eth0'

config bridge-vlan
	option device 'br-lan'
	option vlan '10'
	list ports 'eth0:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '20'
	list ports 'eth0:t'

config interface 'vlan10'
	option proto 'static'
	option device 'br-lan.10'
	option ipaddr '192.168.10.1'
	option netmask '255.255.255.0'
	list dns '192.168.10.0.1'

config interface 'vlan20'
	option proto 'static'
	option device 'br-lan.20'
	option ipaddr '192.168.20.1'
	option netmask '255.255.255.0'
	list dns '192.168.20.0.1'

/etc/init.d/network restart to apply the config.

For now, I just wanted a stable internet connection and didn’t even bother to think about having SR-IOV, so it’s been decided to just make use of the existing Proxmox Linux bridges for both the WAN and LAN interface.

En Hikari configuration on OpenWRT

References: https://note.com/arunya/n/n7d81e0de9db7 https://www.ficusonline.com/ja/posts/openwrt-v6-plus-map-e

I found these two articles that provide step-by-step instructions to get En Hikari working with v6 plus and the fixed IPv4. The initial steps to have DHCPv6 working are in the first one. In my case, I installed the required packages first:

• luci-proto-ipv6
• map
• ds-lite
• ip-full

As I know Hikari Cross utilizes MAP-E. I am not sure if ds-lite is really required, but I just installed it anyway.

I created a new WAN interface, named it ‘wan6’, and set the protocol to DHCPv6 client. Under DHCP Server > IPv6 Settings, the RA-Service, DHCPv6 Service, and NDP Proxy were all set to disabled. After a few seconds, I was assigned an IPv6 /56 subnet.

For the clients to connect over IPv6, I had to enable IPv6 under Interfaces > LAN > Advanced Settings. IPv6 assignment length was set to 64 and hint to 00 so my LAN gets assigned the first smaller subnet of the /56 block.

Under LAN > DHCP Server > IPv6 Settings, RA-Service and DHCPv6 Service should be set to server and NDP Proxy to relay. Under IPv6 RA Settings, I made sure SLAAC was enabled. The rest were set to default. After applying the changes, my devices started having an IPv6 address, the majority by SLAAC and a very few through DHCPv6.

For the IPv4 configuration, I just followed the article. Though I am still sharing my snaps here for future reference:

Info that was provided by the ISP:

• V4アドレス
• インターフェスID
• BRアドレス
• ユーサID
• パスワード

Quick speed test

I don’t have any other equipment (yet, lol) that supports 10G traffic, so I spun up an LXC container on the same PVE hosting OpenWRT and ran a simple iPerf test from there. I selected one of the JP servers from this list maintained by @R0GGER (Big thanks to him!).

The result, DL and UL speed respectively:

The result:

DL SPEED

UL SPEED

I may not be getting the full 10G bandwidth, but I’m certainly getting ~5x the speed compared to my previous connection for the same price.

What’s next

In addition to the benefits a multi-gigabit upstream brings, being assigned an IPv6 subnet from my ISP will allow me to host multiple services from within my local network since the hosts will now have publicly routable addresses. This means a few things:

1. No need to keep track and configure ports for port-forwarding to different VMs or containers.
2. Multiple services can be hosted on the same port.
3. I can create a DMZ network to completely expose hosts to the public internet.
4. P2P connectivity can be established without the requirement of NAT-ing.

References

• Cross-flashing X710 NICs
• Disabling vendor-lock of X710 NICs
• Explanation of above
• OpenWRT on Proxmox
• OpenWRT configuration for En Hikari with v6 plus and Fixed IP
• Intel X710 drivers

Sep 29 14:54:58 pve1 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:
                               TDH                  <22>
                               TDT                  <bb>
                               next_to_use          <bb>
                               next_to_clean        <22>
                             buffer_info[next_to_clean]:
                               time_stamp           <100cb7acb>
                               next_to_watch        <23>
                               jiffies              <100cb7d99>
                               next_to_watch.status <0>
                             MAC Status             <80083>
                             PHY Status             <796d>
                             PHY 1000BASE-T Status  <3800>
                             PHY Extended Status    <3000>
                             PCI Status             <10>
...
...
...
Sep 29 14:55:07 pve1 kernel: e1000e 0000:00:1f.6 eno1: Reset adapter unexpectedly
Sep 29 14:55:07 pve1 kernel: vmbr0: port 1(eno1) entered disabled state
Sep 29 14:55:07 pve1 kernel: vmbr0v20: port 1(eno1.20) entered disabled state
Sep 29 14:55:07 pve1 kernel: vmbr0v30: port 1(eno1.30) entered disabled state
Sep 29 14:55:11 pve1 kernel: e1000e 0000:00:1f.6 eno1: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Sep 29 14:55:11 pve1 kernel: vmbr0: port 1(eno1) entered blocking state
Sep 29 14:55:11 pve1 kernel: vmbr0: port 1(eno1) entered forwarding state
Sep 29 14:55:11 pve1 kernel: vmbr0v20: port 1(eno1.20) entered blocking state
Sep 29 14:55:11 pve1 kernel: vmbr0v20: port 1(eno1.20) entered forwarding state
Sep 29 14:55:11 pve1 kernel: vmbr0v30: port 1(eno1.30) entered blocking state
Sep 29 14:55:11 pve1 kernel: vmbr0v30: port 1(eno1.30) entered forwarding state

When I saw these messages I thought my switch was acting up. I was almost ready to purchase a new switch but then upon further googling I landed in this forum post.

Turns out this specific model is known to be freezing from time to time when segmentation is done by the NIC. This comes enabled by default when installing Proxmox. A quick fix suggeested in the post which worked for me was to simply disable the TSO and GSO flags on the specific interface. On runtime this can be disabled by executing ethtool -K eno1 tso off gso off.

To make the changes permenant, the following line should be added in the /etc/network/interfaces file:

post-up /usr/bin/logger -p debug -t ifup "Disabling segmentation offload for eno1" && /sbin/ethtool -K $IFACE tso off gso off && /usr/bin/logger -p debug -t ifup "Disabled offload for eno1"

e.g.

One more issue I faced was very slow throughput when playing videos from Jellyfin.

Noticed that throughput was very low regardless whether I’m using the docker traefik or the one in my kubernetes cluster, while the original traefik container on my Unraid

After drilling down that issue seems to be specific to my Proxmox node, I checked whether the 1Gbit/s speed was properly negotiated as suggested in this other forum post.

Executing ethtool eno1 showed me the following output:

root@pve1:~# ethtool eno1
Settings for eno1:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	Supported pause frame use: No
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10baseT/Full
	Advertised pause frame use: No
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 10Mb/s
	Duplex: Full
	Auto-negotiation: on
	Port: Twisted Pair
	PHYAD: 1
	Transceiver: internal
	MDI-X: on (auto)
	Supports Wake-on: pumbg
	Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
	Link detected: yes

Disabled auto-negotiation and configured 1Gbit manually: ethtool -s eno1 speed 1000 duplex full autoneg off

root@pve1:~# ethtool eno1
Settings for eno1:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	Supported pause frame use: No
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  1000baseT/Full
	Advertised pause frame use: No
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Auto-negotiation: on
	Port: Twisted Pair
	PHYAD: 1
	Transceiver: internal
	MDI-X: off (auto)
	Supports Wake-on: pumbg
	Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
	Link detected: yes

After having these in place, my cluster has been running perfectly fine for almost a month now with nohiccups!

At this time of writing, so far only two services have been migrated from the docker environment into the new K8S cluster. That is my DNS which is also replaced now by AdguardHome (sorry Pihole!), and Traefik, which terminates all external HTTP communication incoming to my Homelab. And for the rest of the other services, those will be migrated in the coming weeks. For now what is important is I have HTTPS running perfectly fine with valid CA certificates even with dockerized services in the backend.

I also want to note here that during the learning process, whatever services that were running in the docker setup, all of those were nearly untouched since I was working inside a staging environment, keeping rigorous activities of countless create-delete of resources in isolation from my home production network.

That said, I received zero complaints from the wife except for the time I lost connectivity to the cluster when I thought everything was already stable. Lol more on this later.

In summary below opensource tools have been used for this project.

• Proxmox - Hypervisor (installed in 3 devices)
• Terraform - VM instantiation
• Ansible - VM configuration, Kubernetes installation
• K3s - Kubernetes platform
• Kube-vip - LB for Kube API
• Longhorn - Block storage
• Cilium - CNI
• Traefik - Reverse Proxy
• Cert-manager - SSL certificate manager

Terraform and Ansible play the major role of orchestrators allowing for quick creation and deletion of virtual resources putting Infrascture-as-Code into practice. The rest are supplementary to learning IaC.

Side node: Unfortunately Terraform will be transitioning away from a completely opensource license and soon will be sitting behind a BSL. It might be a good idea to switch to OpenTF (or OpenTofu) which is a fork of Terraform and now officially a CNCF project as well.

The setup

All development activities were done in a staging environment. This is where the unaccounted events of VM creation-deletion have transpired from testing out the Terraform scripts until the very end when installing Longhorn and Cilium with Ansible, and even when playing around with Traefik in K8s.

The staging environment was deployed on my Unraid box and as for the production build, I am using Tiny PCs that house a total of 32GB RAM each allowing more room for other VMs that I might need in the long run. To give a glimpse of the production HW:

I was also able to source data center-grade Intel S3510 SSD drives from the second-hand market. These should help in the reliability department of these nodes that will be running data replication because of Longhorn. Currently, these are housed in 2 of 3 nodes. Still looking to get another one and pop it in PVE1.

One container to rule them all

Both the Terraform and Ansible controllers are running from a LXC container deployed on PVE1. This is where I do all of the development work. I am running a docker instance of VS code on the same container which enables me to create and modify code easily. The same container is used to communicate with the production environment.

One of the best advantages of using an LXC container is that it’s so lightweight you can easily backup the environment anytime. I confirm Terraform and Ansible works well with the Ubuntu 22.04 template that comes with Proxmox.

Terraform and Ansible

Learning Terraform wasn’t so bad. The Proxmox provider documentation from Telmate is enough to get you started and with the declarative style that is used to write Terraform language, it just makes the learning process a lot more easier to deal with. This is also the part where I had the least modifications and time spent the least.

Once I got the VMs up and working, next thing I worked on is Ansible. Now this is the part where I spent the biggest chunk of my time. I literally am unable to count the times I had to execute terraform destroy and terraform apply to re-create the VMs and test out my Asnbile playbooks. To be fair starting with Ansible wasn’t really hard. It was the amount of automation I wanted to go with that later on will prove useful. Though I have to admit the playbooks are rather simple and which others might find lacking in terms of best practices. But hey, I have to start from somewhere!

The five commands to have a complete working k3s cluster:

terraform apply
ansible-playbook -i inventory.yaml preflight.yaml
ansible-playbook -i inventory.yaml logical-volume-create.yaml
ansible-playbook -i inventory.yaml k3s-kubevip-helm-ciliumInstallHelmCli.yaml
ansible-playbook -i inventory.yaml longhorn-install.yaml

The whole process takes about more or less 15 minutes. Terraform creates 3 VMs, one on each proxmox node. Each VM is set 4 vCPU, 12GB RAM, 50GB boot disk + 200GB longhorn disk. preflight.yaml defines the ssh keys for passwordless authentication and installs the necessarry packages to run k3s-related software. logical-volume-create.yaml as its name suggests, creates the logical volume to be used for longhorn. k3s-kubevip-helm-ciliumInstallHelmCli.yaml installs k3s, kube-vip, helm, and Cilium altogether sequentially. And last but not the least, longhorn-install.yaml installs longhorn via helm.

Cilium

While Calico would be the go-to CNI for most, I opted to go for Cilium. The main reason for this is to start learning eBPF and have a grasp how things move within the kernel space. The installation was straightforward but to get it working in properly was a challenge.

I was able to make BGP control plane work during the initial phase and when I thought everything was already stable, I then started seeing BGP peers getting dropped and at some point even lost connectvivity to my DNS hosted in the cluster. After adding into the parameters one by one, I was able to make it work without disconnects.

helm install cilium cilium/cilium --version 1.14.2 \
--namespace kube-system \
--set bgpControlPlane.enabled=true \
--set tunnel=disabled \
--set ipam.operator.clusterPoolIPv4PodCIDRList=10.42.0.0/16 \
--set kubeProxyReplacement=true \
--set k8sServiceHost={{ _k8sServiceHost }} \
--set k8sServicePort=6443 \
--set routingMode=native \
--set autoDirectNodeRoutes=true \
--set ipv4NativeRoutingCIDR=10.42.0.0/16 \
--set loadBalancer.mode=dsr \
--set ipv4.enabled=true \
--set prometheus.enabled=true \
--set operator.prometheus.enabled=true \
--set hubble.enabled=true \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}"

kubeProxyReplacement is enabled to make use of eBPF instead of the traditional way of iptables. loadBalancerMode is also set to DSR making it possible for nodes to respond directly to the sender, avoiding the need to go via the return path. routingMode set to native and autoDirectNodeRoutes is set to true since all nodes are connected via the same L2 network. You can read more on Cilium routing here. Prometheus and Hubble are also enabled so I can touch on them later once I get more time.

cilium-bgp-peering-policy.yaml:

apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
  name: 00-bgp-peering-policy
spec: # CiliumBGPPeeringPolicySpec
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: k8s-master-0-dev
  virtualRouters: # []CiliumBGPVirtualRouter
    - localASN: 65090
      exportPodCIDR: false
      serviceSelector:
        matchLabels:
          exposedExternal: "yes"
      neighbors: # []CiliumBGPNeighbor
        - peerAddress: '10.20.0.1/32'
          peerASN: 65000
          eBGPMultihopTTL: 10
          connectRetryTimeSeconds: 120
          holdTimeSeconds: 90
          keepAliveTimeSeconds: 30
          gracefulRestart:
            enabled: true
            restartTimeSeconds: 120
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
  name: 01-bgp-peering-policy
spec: # CiliumBGPPeeringPolicySpec
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: k8s-master-1-dev
  virtualRouters: # []CiliumBGPVirtualRouter
    - localASN: 65091
      exportPodCIDR: false
      serviceSelector:
        matchLabels:
          exposedExternal: "yes"
      neighbors: # []CiliumBGPNeighbor
        - peerAddress: '10.20.0.1/32'
          peerASN: 65000
          eBGPMultihopTTL: 10
          connectRetryTimeSeconds: 120
          holdTimeSeconds: 90
          keepAliveTimeSeconds: 30
          gracefulRestart:
            enabled: true
            restartTimeSeconds: 120
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
  name: 02-bgp-peering-policy
spec: # CiliumBGPPeeringPolicySpec
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: k8s-master-2-dev
  virtualRouters: # []CiliumBGPVirtualRouter
    - localASN: 65092
      exportPodCIDR: false
      serviceSelector:
        matchLabels:
          exposedExternal: "yes"
      neighbors: # []CiliumBGPNeighbor
        - peerAddress: '10.20.0.1/32'
          peerASN: 65000
          eBGPMultihopTTL: 10
          connectRetryTimeSeconds: 120
          holdTimeSeconds: 90
          keepAliveTimeSeconds: 30
          gracefulRestart:
            enabled: true
            restartTimeSeconds: 120

As for configuring BGP on OPNsense, all I had to do was download the os-frr plugin and apply the configuration according to the Cilium BGP resources.

Once BGP is configured, I then configured CiliumLoadBalancerIPPool and configured the serviceSelector there so any service with a matching label will be assigned an external IP.

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: externalpool
spec:
  cidrs:
    - cidr: 192.168.100.0/27
  disabled: false
  serviceSelector:
    matchLabels:
      exposedExternal: "yes"

To assign services with external IPs, all you have to do is ensure two things (third one is optional):

1. The service should have a label matching what you defined in your CiliumLoadBalancerIPPool.
2. The service should be of type LoadBalancer.
3. For static external IP assignment, the service should have an annotation of io.cilium/lb-ipam-ips followed by the IP address.

apiVersion: v1
kind: Service
metadata:
  annotations:
    io.cilium/lb-ipam-ips: 192.168.100.7
  labels:
    exposedExternal: "yes"
  name: adguard-ui
  namespace: adguard
...
...
spec:
  type: LoadBalancer
...
...

When all of the above are applied, you should now see an external IP assigned to your service. e.g.

❯ k -n adguard get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
adguard      LoadBalancer   10.43.254.122   192.168.100.8   53:30176/UDP   27d
adguard-ui   LoadBalancer   10.43.38.101    192.168.100.7   80:32108/TCP   27d

You should also be able to check the BGP peering status as well as the learned routes in OPNsense.

Longhorn

Before deciding to go for Longhorn, I was was trying to make Piraeus datastore work (FOSS version of Linstor storage for Kubernetes). I got it to work with ReadWriteOnce but the moment I tried to test ReadWriteMany(RWX), it just wouldn’t. On top of this it also felt that there was a steep learning curve to understand how Piraeus work on a deeper level in case I had to do extra troubleshooting in the future.

Longhorn on the other hand worked well out of the box. Testing out RWX by re-creating a pod on a different node worked well too and since Longhorn seems to use NFS to support this feature, accessing the volume from something external to the cluster e.g. from a VM works out of the box. The only thing is that there seems to be an open issue with Cilium when exposing the volume externally. When I try to mount the share to a VM, I do experience slowdowns when opening a file with vim or even when just browsing through the directories.

Longhorn makes use of a 200Gi second volume that was declared in the Terraform script. The above snapshot shows the amount of available volume.

For reference, below is the playbook task to install longhorn:

- name: install longhorn
  kubernetes.core.helm: 
    name: longhorn
    chart_ref: longhorn/longhorn
    release_namespace: longhorn-system
    create_namespace: true
    update_repo_cache: true
    set_values:
      - value: service.ui.type=LoadBalancer
      - value: defaultSettings.defaultDataPath=/longhorn_vol
      - value: defaultSettings.defaultReplicaCount=3

Exposing services

I took more time than expected when I was trying to expose the services with Traefik now running in Kubernetes. In the docker setup, the configuration was pretty straightforward with minimal reading required of the documentation. Whereas when running treafik in k8s, it came to the point that it already felt like I was digging my own grave with all the research and testing.

Eventually I got it to work by going with the base installation and slowly inching my way through the custom values in the yaml file. Once I got the middleware (for additional security headers) and TLS working via cert-manager, all I had to do was create individual ingress resources for each of the services I wanted to expose. The certificates are automatically created and managed by Cert-manager.

One thing to note, in Docker, the certificates can be managed by Traefik. But when using Traefik in a K8s environment, to make use of Let’s Encrypt, the only option is to use Cert-manager which can only be paired up with the default Kubernetes Ingress resource. Traefik’s Ingress CRD doesn’t support this at the moment.

Below is a sample Ingress resource to reach the Adguard GUI from external to the cluster:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: adguard-ui
 namespace: adguard
 annotations:
   cert-manager.io/cluster-issuer: "letsencrypt-cluster-issuer"
spec:
 tls:
   - hosts:
       - adguard.su-root.net
     secretName: tls-adguard-ui-ingress-dns
 rules:
   - host: adguard.su-root.net
     http:
       paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: adguard-ui
               port: 
                 number: 80

The journey continues

The learning doesn’t stop here. What I have achieved so far is a basic understanding of how IaC integrates and can be made to work in different environments depending on the requirement. Another good thing is that this adds up to my confidence knowingly I can spin up my cluster in a matter of minutes even in the event that I have to physically migrate to another environment.

Going further I will continue to enhance the Ansible playbooks and try to make use of industry best practices even if this is only intended for Homelab use. I’m also looking into integrating this with some kind of CICD tool like Jenkins or ArgoCD in the near future.

If you are interested to see more of this project, feel free to check out the repository over at my Github page. A disclaimer though, the README is not updated yet! I will be updating this sooner or later and together with that will try to explain in detail the idea behind each step of the installation process.

Proxmox already supports LXC containers by default and in this case, running Pihole on LXC provides some advantage in terms of flexibility and ease of configuration. This is especially for those who have less experience working with docker but only with Linux in general. I will share one good use case for this later on but for now let’s start setting up Pihole on Proxmox.

The hardware requirements according to the pihole documentation: Minimum of 2GB disk (4GB recommended) 512MB RAM

First you need to download an LXC template. I like using debian as the base image since it’s very lightweight and it’s where Ubuntu is based from. If you don’t have it yet, you can download it by going to Datacenter > pve > storage > CT templates > Templates and search for Debian 11 Bullseye.

Click on Create CT and input a container ID, hostname, and the root password. Check Unprivileged container and nesting (optional). Note that nesting is not really required but in my case the proxmox terminal will keep on printing permission errors if I leave this unchecked.

Select debian as the template.

For disk, I assigned 6GB. For RAM, double the recommended, since I have enough.

Assign a static IP to the container. Gateway should also be defined.

For DNS I will assign my unbound IP which is the same as my OPNsense gateway IP. (Unbound is a DNS caching tool built-into OPNsense). This will basically be my upstream DNS for my Pihole. If you don’t have Unbound running then you can input any public DNS like Google (8.8.8.8, 8.8.4.4) or Cloudflare (1.1.1.1).

Confirm

Enable start on boot flag.

Login as root

Update and upgrade

apt-get update -y && apt-get upgrade -y

Install curl:

apt-get install curl -y

Install Pihole:

curl -sSL https://install.pi-hole.net | bash

Install custom upstream and point to unbound IP

Install the default blacklist.

Install admin interface and lighthttpd.

Query logging or any other option is fine.

Reset the pihole password.

sudo pihole -a -p

Now go to your container IP and append /admin (e.g. http://10.0.0.88/admin)

If you are running it on proxmox like me you’d probably get the same error as in the snap below. You can ignore this especially if you have multi-core host. If you want to be sure you can check your CPU utilization with the top command.

After changing your DNS to pihole, check with nslookup if your device is able to send and receive to and from the Pihole IP.

As for the specs, it’s a fanless appliance running a quad-core Intel 12th-gen J6412 (2 GHz base, 2.6 GHz boost) with 5x Intel i226 2.5Gbe NICs. I chose the bare minimum option so I can avoid the crappy RAM and storage that might come with it. For the RAM I was able to source second-hand 2x8GB Samsung DDR4 SODIMMs and for the storage I didn’t any more bother and just ordered a 240GB SSD from Amazon since these come cheap these days.

Installed Proxmox and spawned a VM running a router software based on freeBSD called OPNsense. I can’t believe I’ve been missing on this piece of software for the longest time. It feels like I’ve rekindled my love for networking ever since I switched to a different role in my career graduating from configuring routers and switches.

The entire setup process was very straightforward. I just got confused a tad bit in the beginning when I was trying to figure out which interface to assign for management access. On my first attempt, I assigned the first interface in the list, expecting that it would be mapped to the first physical port. But when I tried to plug an ethernet cable from my laptop, for some reason I couldn’t ping the management IP I set. Long story short I re-installed Proxmox again only to know that the first installation was already correct. Maybe some lose cable?

Anyway the router has been running stable for the past week. This is with virtio running as a driver for the Intel i226 NICs. On average I’m getting about 500Mbps for internet (I used to get about 700MBps with a TP-Link A10 router and yes I can deal with that 200Mbps difference for now!). I’ll explore physically passing through the NICs later on once I get the time. But as far as I can tell, I’m just loving it. On top of that Proxmox has also been remarkably gaining my interest now that I have a pi-hole running in an LXC container.