Second, the old subscription only had IPv4. Don’t get me wrong, IPv4 isn’t a bad thing. In fact, a good majority of the internet still runs on IPv4. But recently, I’ve gained interest in exploring the capabilities and other features that come with IPv6. More on this later, but the main trigger point for this is because IPv6 is already widely used in Japan. If I’m not mistaken, IPv6 adoption has been enforced by the Japan Ministry of Communications for some time now.

Third, since the beginning of last year, I’ve taken on a new role focused on optimizing customer experience performance, where speed is one of the main concerns. To better support this and overcome the limitations of my current toolset, having a speed test server (with multi-gigabit bandwidth) that I have full control of is a fundamental first step!

Hardware

Certain Lenovo Tiny PCs are popular machines for having a PCIe slot that can be fitted with 10G NICs, so that’s what I went with. I was able to get a secondhand one with an Intel i5-8500T 6C CPU. This should be more than enough for my needs, even if I wanted to host other services on this machine. As for the NIC, I had the option of going with Mellanox or Intel ones. To be more specific, I was choosing between a Mellanox ConnectX-4 and an Intel X710-DA2. Mellanox seems to be more popular in terms of compatibility and stability, but it’s also known to have elevated power consumption since it’s not able to achieve higher C-states even when idle. The X710-DA2, on the other hand, was known to have compatibility issues, especially when using the OEM-branded ones. I went with Intel to have peace of mind knowing that I tried my best to save electricity costs (as if all the other machines in the homelab justify all the power they require!).

Unfortunately, I assumed the compatibility issue was something generic and could easily be fixed by cross-flashing the original Intel firmware. While this might have worked for others, this isn’t the case when specifically using a Dell-branded X710 NIC with a Lenovo M920q Tiny PC. So, if anyone intends to follow this route, take note! I learned this the hard way because I purchased a Dell-branded one and went through the whole cross-flashing process only to find out that my machine will only boot up successfully if it’s coming from an unplugged state. Yup, that means it will only boot up after unplugging and plugging the power cable. Anyway, I ended up purchasing an Intel-branded one and just flashed it with the latest firmware. For anyone interested, the articles I followed are also shared at the end of this post!

Lenovo M920q Tiny PCs require a PCIe riser for plugging in your cards. This can easily be bought for about ¥2000~3000 online. There are different part numbers in terms of the supported bus speed, so I had to ensure I got the one with P/N: 01AJ940, which supports x8.

As for the 10G switch, for the time being, I got a 4x2.5 + 2x10G switch from Horaco (AliExpress). I already have an 8x2.5G switch from the same brand and it’s been rock stable. There was no really good reason for me to try out another brand. If there was one thing where you have to manage your risk, it’s when purchasing networking equipment from AliExpress!

10G SFP+ to copper transceivers are known to run hot. And because I keep my networking equipment in an unventilated closet, I had to shed an extra couple of bucks for the premium ones—the version that supports up to 80m of copper. For the transceivers, I got them from a brand called ZYOPM.

Managing the temperature inside an enclosed space is a priority, so I had to make sure the temps are at a minimum whenever possible. For the upstream connection of my switch to the router, I used a DAC cable. I’ve never used one before and actually thought of using optical transceivers in the beginning.

On to the Wireless AP, I also upgraded from my not-so-old TP-Link EAP610 to an EAP773. The former only supported Wi-Fi 6 over 2.4/5 GHz bands with 1G upstream. For “future-proofing,” at least for the next few years, and to make use of the upgraded upstream, I needed something which at least supported Wi-Fi 6E over 6 GHz.

I also had to get a PoE++ 90W injector since the new switch didn’t support PoE+.

Software

There are mixed opinions on whether you should virtualize your router or not. But if you live somewhere where space is a luxury, then you already have the best reason to go with virtualization (but regardless of that, I just really like complicating things, lol). So of course, we go back to our good old friend, Proxmox. The last time I had to spin up a new Proxmox machine was about 2 years ago when I started using CEPH as my default backend storage. A lot of good things must have been added within these 2 years, but I just really needed an easily managed KVM host. No one can beat Proxmox on that.

As for the router itself, I made the big switch from OPNsense to OpenWRT. And I couldn’t be happier. I’ve had OPNsense for about 3 years now and while I didn’t really complain about anything since it was more of a “set and forget” setup, whenever I had to modify something in the configuration, sometimes it just felt like I had to explore and familiarize myself again. With OpenWRT, the GUI just felt a bit more natural and warm to my eyes. Maybe it’s because of fewer sections or tabs to go through, or, I don’t know, configuration just seemed a little more straightforward this time compared to when I just started with the other.

But in fact, I don’t think I would have bothered checking out OpenWRT if only OPNsense supported MAP-E connections used by my new ISP. There is a way to get it working with OPNsense, but it was more of a workaround than a natively supported feature.

My choice of ISP

The primary factor in choosing the ISP is the monthly cost. The switch to 10G wouldn’t be justifiable if we had to pay 50% more than our old subscription. Second would be the option to have a fixed IPv4 address since I host some services for family and friends. Fortunately, I came across a post on Reddit suggesting En Hikari. En Hikari uses NTT FLET’S HIKARI as the backbone provider, so you can be assured of the same quality of connectivity used by most ISPs. At the time of writing, the monthly cost is about JPY 4,917 (tax included) plus an additional JPY 770 for the optional fixed IPv4 address. That’s a total of JPY 5,687, which comes out even cheaper than our current basic 1G subscription at JPY 5,720 (dynamic IPv4 address only).

Preparations

When I got the Intel X710 card, it only had v6.01 installed. I didn’t find a way to upgrade directly from v6.01 to the latest one (v9.56 as of this writing) and had to go through it stepwise. The upgrade path in my case was from 6.01 –> 8.6 –> 9.0 –> 9.10 –> 9.56. All done under Debian (Proxmox).

My Intel card came with unlocked vendor support, so I didn’t have to run the unlocker script. I guess this is only applicable to the OEM versions.

I actually ran into some errors initially that prevented the second Ethernet port from functioning. Honestly, I hit a wall and gave up on it for the night. Then, I’m not sure how, but it just got resolved the following morning after a restart.

After a few more rounds of restarts and going through kernel dmesg logs, I finally decided to go with a fresh install of OpenWRT. For some reason, I couldn’t figure out how to create the VLAN interfaces from the GUI. I ended up with a successful attempt when I tried to do it from the text file configuration. To those who are in the same boat, below is a sample /etc/config/network for configuring VLANs.

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd00:7808:88c3::/48'
	option packet_steering '1'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'

config interface 'lan'
	option device 'br-lan.1'
	option proto 'static'
	option ipaddr '192.168.0.1'
	option netmask '255.255.255.0'
	option ip6assign '64'
	list dns '192.168.0.1'
	option ip6hint '00'

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'
	option keepalive '5 10'

config interface 'wan6'
	option device 'eth1'
	option proto 'dhcpv6'
	option reqaddress 'try'
	option reqprefix 'auto'
	option norelease '1'

config bridge-vlan
	option device 'br-lan'
	option vlan '1'
	list ports 'eth0'

config bridge-vlan
	option device 'br-lan'
	option vlan '10'
	list ports 'eth0:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '20'
	list ports 'eth0:t'

config interface 'vlan10'
	option proto 'static'
	option device 'br-lan.10'
	option ipaddr '192.168.10.1'
	option netmask '255.255.255.0'
	list dns '192.168.10.0.1'

config interface 'vlan20'
	option proto 'static'
	option device 'br-lan.20'
	option ipaddr '192.168.20.1'
	option netmask '255.255.255.0'
	list dns '192.168.20.0.1'

/etc/init.d/network restart to apply the config.

For now, I just wanted a stable internet connection and didn’t even bother to think about having SR-IOV, so it’s been decided to just make use of the existing Proxmox Linux bridges for both the WAN and LAN interface.

En Hikari configuration on OpenWRT

References: https://note.com/arunya/n/n7d81e0de9db7 https://www.ficusonline.com/ja/posts/openwrt-v6-plus-map-e

I found these two articles that provide step-by-step instructions to get En Hikari working with v6 plus and the fixed IPv4. The initial steps to have DHCPv6 working are in the first one. In my case, I installed the required packages first:

• luci-proto-ipv6
• map
• ds-lite
• ip-full

As I know Hikari Cross utilizes MAP-E. I am not sure if ds-lite is really required, but I just installed it anyway.

I created a new WAN interface, named it ‘wan6’, and set the protocol to DHCPv6 client. Under DHCP Server > IPv6 Settings, the RA-Service, DHCPv6 Service, and NDP Proxy were all set to disabled. After a few seconds, I was assigned an IPv6 /56 subnet.

For the clients to connect over IPv6, I had to enable IPv6 under Interfaces > LAN > Advanced Settings. IPv6 assignment length was set to 64 and hint to 00 so my LAN gets assigned the first smaller subnet of the /56 block.

Under LAN > DHCP Server > IPv6 Settings, RA-Service and DHCPv6 Service should be set to server and NDP Proxy to relay. Under IPv6 RA Settings, I made sure SLAAC was enabled. The rest were set to default. After applying the changes, my devices started having an IPv6 address, the majority by SLAAC and a very few through DHCPv6.

For the IPv4 configuration, I just followed the article. Though I am still sharing my snaps here for future reference:

Info that was provided by the ISP:

• V4アドレス
• インターフェスID
• BRアドレス
• ユーサID
• パスワード

Quick speed test

I don’t have any other equipment (yet, lol) that supports 10G traffic, so I spun up an LXC container on the same PVE hosting OpenWRT and ran a simple iPerf test from there. I selected one of the JP servers from this list maintained by @R0GGER (Big thanks to him!).

The result, DL and UL speed respectively:

The result:

DL SPEED

UL SPEED

I may not be getting the full 10G bandwidth, but I’m certainly getting ~5x the speed compared to my previous connection for the same price.

What’s next

In addition to the benefits a multi-gigabit upstream brings, being assigned an IPv6 subnet from my ISP will allow me to host multiple services from within my local network since the hosts will now have publicly routable addresses. This means a few things:

1. No need to keep track and configure ports for port-forwarding to different VMs or containers.
2. Multiple services can be hosted on the same port.
3. I can create a DMZ network to completely expose hosts to the public internet.
4. P2P connectivity can be established without the requirement of NAT-ing.

References

• Cross-flashing X710 NICs
• Disabling vendor-lock of X710 NICs
• Explanation of above
• OpenWRT on Proxmox
• OpenWRT configuration for En Hikari with v6 plus and Fixed IP
• Intel X710 drivers

I got everything working eventually after multiple iterations of trial and error, and upgrading my traefik helm release countless times. And after finding this Traefik blog post where they explain how it all actually works and can be configured. At this point I’ve migrated all of my docker applications into my k8s cluster, and reading through the Traefik documentation again, it just makes a lot more sense now.

I guess the most important bit of the documentation is the part where they mentioned the requirement of using Cert-manager when using Traefik Proxy in Kubernetes. Quoting from the documentation:

LetsEncrypt Support with the Ingress Provider By design, Traefik is a stateless application, meaning that it only derives its configuration from the environment it runs in, without additional configuration. For this reason, users can run multiple instances of Traefik at the same time to achieve HA, as is a common pattern in the kubernetes ecosystem. When using a single instance of Traefik Proxy with Let’s Encrypt, you should encounter no issues. However, this could be a single point of failure. Unfortunately, it is not possible to run multiple instances of Traefik 2.0 with Let’s Encrypt enabled, because there is no way to ensure that the correct instance of Traefik receives the challenge request, and subsequent responses. Previous versions of Traefik used a KV store to attempt to achieve this, but due to sub-optimal performance that feature was dropped in 2.0. If you need Let’s Encrypt with high availability in a Kubernetes environment, we recommend using Traefik Enterprise which includes distributed Let’s Encrypt as a supported feature. If you want to keep using Traefik Proxy, LetsEncrypt HA can be achieved by using a Certificate Controller such as Cert-Manager. When using Cert-Manager to manage certificates, it creates secrets in your namespaces that can be referenced as TLS secrets in your ingress objects.

Since we are working in K8s, almost everything you can think of can be abstracted. For this instance, the TLS configuration is abstracted from the helm custom values yaml file and instead can be pegged to an Ingress resource when using Cert-manager.

To summarize in a few lines:

• Install Traefik so it can act your Ingress controller.
• Install Cert-manager so it can monitor your Ingress resources, and based on the TLS configuration and with some annotations, it will automatically initiate the DNS challenge, create the TLS certificates sand save them as a secret in the application namespace.
• Create Ingress resources to access your backend servcies while at the same time terminate HTTPS traffic (with Traefik).

Do note that while Traefik is an ingress controller and comes with its own IngressRoute CRD, it cannot be used together with Cert-manager at the moment. In this case we have to use the default Kubernetes Ingress resource.

In this post I will show how I configured Traefik and Cert-manager (patterned to the Traefik blog post!) to be used with Cloudflare. I will also show how you can have another layer of security by having multiple entry points and explicitly defining one during the ingress configuration. Though before proceeding, do note that I assume you already have an idea how to configure dynamic DNS and port-forwarding on Cloudflare and on your router.

Traefik

Recently I’ve started using ArgoCD for managing all my applications including those that are installed via helm charts. For now we will install Traefik using helm.

helm repo add traefik https://traefik.github.io/charts # Add the Traefik repository
helm repo update # Update the repositories
helm show values traefik/traefik > values.yaml # Download the custom values yaml

I would recommend you to browse through the custom values yaml file to get familiarized with the available configuration and other features of Traefik. If you want to get started quickly then you can follow the below yaml configuration:

values.yaml

deployment:
  enabled: true
  kind: Deployment
  replicas: 1
  terminationGracePeriodSeconds: 60
  minReadySeconds: 0

ingressRoute:
  dashboard:
    enabled: true
    matchRule: Host(`traefik.yownowndomain.com`) && PathPrefix(`/dashboard`) || Host(`traefik.yownowndomain.com`) && PathPrefix(`/api`)
    entryPoints: ["traefik","websecure"]

additionalArguments:
  - "--serversTransport.insecureSkipVerify=true"
  - "--providers.kubernetesingress.ingressendpoint.publishedservice=traefik/traefik"

ports:
  traefik:
    port: 9000
    expose: true
    exposedPort: 9000
    protocol: TCP
  web:
    port: 80
    expose: true
    exposedPort: 80
    protocol: TCP
    redirectTo: websecure
  webext:
    port: 8080
    expose: true
    exposedPort: 8080
    protocol: TCP
    redirectTo: websecureexternal
  websecure:
    port: 443
    expose: true
    exposedPort: 443
    protocol: TCP
    http3:
      enabled: false
    tls:
      enabled: true
      options: ""
      certResolver: ""
    middlewares:
      - default-default@kubernetescrd
  websecureext:
    port: 8443
    expose: true
    exposedPort: 8443
    protocol: TCP
    http3:
      enabled: false
    tls:
      enabled: true
      options: ""
      certResolver: ""
    middlewares:
      - default-default@kubernetescrd
  metrics:
    port: 9100
    expose: false
    exposedPort: 9100
    protocol: TCP
  dnsovertls:
    port: 8853
    expose: true
    exposedPort: 853
    protocol: TCP
    tls:
      enabled: false
    middlewares:
      - default-default@kubernetescrd
tlsOptions:
  default:
    minVersion: VersionTLS12
    cipherSuites:
      - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
      - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
      - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

service:
  enabled: true
  single: true
  type: LoadBalancer
  annotations:
    io.cilium/lb-ipam-ips: 10.80.0.2
  labels:
    exposedExternal: "yes"

In my custom yaml file notice that I have four different entryPoints defined. I configured it like this to have separate incoming streams between my internal-only applications against the internet-exposed ones. Having a common entryPoint for both internal and externally exposed applications is a security flaw since hackers might still be able to access your internal-only apps if they guessed the sub-domain and locally configured CNAME forwarding (they don’t need access to your DNS configuration).

Once your custom values yaml file is ready. Hit up the helm install command and proceed to the next step to create a Middleware resource.

default-middleware.yaml

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: traefik
spec:
  headers:
    customResponseHeaders:
      X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
      X-Forwarded-Proto: "https"
      server: ""
    customRequestHeaders:
      X-Forwarded-Proto: "https"
    sslProxyHeaders:
      X-Forwarded-Proto: "https"
    referrerPolicy: "same-origin"
    hostsProxyHeaders:
      - "X-Forwarded-Host"
    contentTypeNosniff: true
    browserXssFilter: true
    forceSTSHeader: true
    stsIncludeSubdomains: true
    stsSeconds: 63072000
    stsPreload: true

helm install traefik/traefik -f values.yaml

kubectl apply -f default-middleware.yaml

Now that you have Traefik up and running, expose the webext and websecureext entryPoints on your router. Simply port-forward 80 to 8080, and 443 to 8443.

Cloudflare

Going to Cloudflare, you have to configure an access token to be used later on when configuring Cert-manager. Login to your Cloudflare account. Navigate to My Profile > API Tokens > Create Token.

Under Permissions, set Zone - DNS - Edit

Under Zone Resources, set Include - Specific zone - “yourowndomain.com”

E.g.

Create and save your token somewhere safe.

Cert-manager

Cert-manager can be installed using helm. Quoting from the official documentation, Cert-Manager together with the CRDs can be installed with the following commands:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.13.2 \
  --set installCRDs=true

After installation, it’s time to configure cert-manager for use with Cloudflare. You simply just have to create a secret containing your cloudflare email and API token:

cloudflare-api-token-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token
  namespace: cert-manager
type: Opaque
stringData:
  email: example@domain.com
  apiToken: AbcD321eFg456_

Next is to create a ClusterIssuer resource. This is one of the CRDs that will come with cert-manager. Here you have to specify your email to be used with Let’s Encrypt as well as the secret you created in the previous step.

clusterissuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cluster-issuer
  namespace: cert-manager
spec:
  acme:
    email: example@domain.com
    # Prod
    server: https://acme-v02.api.letsencrypt.org/directory
    # We use the staging server here for testing to avoid hitting
#    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # if not existing, it will register a new account and stores it
      name: letsencrypt-acc-key
    solvers:
      - dns01:
      # The ingressClass used to create the necessary ingress routes
          cloudflare:
            email: example@domain.com
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: apiToken
        selector:
          dnsZones:
            - 'yourowndomain.com'

If you are doing this the first time, I strongly suggest you first use the staging server of Let’s Encrypt and ensure certificate issuance works fine. To switch to the porduction server all you have to do is just re-create the resource and point to the new URL. Do note in the example above, the resource is pointing to the production server.

Exposing your application

To test this we create a simple nginx deployment that is exposed within the cluster and an ingress resource.

nginx.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
    - name: "http"
      port: 80
      targetPort: 80
  type: ClusterIP

When creating the ingress manifest, you have to ensure that you have the annotation: cert-manager.io/cluster-issuer: "letsencrypt-cluster-issuer".

Apart from this if you want your application to be accessed from a specific traefik entryPoint, then you need to explicitly mention this as another annotation. By default the ingress will be allowed from any entryPoint if you don’t specify this. In the below example manifest, I am keeping this annotation commented out so I can access my Nginx app from both internal and external entryPoints.

nginx-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-cluster-issuer"
    # traefik.ingress.kubernetes.io/router.entrypoints: websecureext
spec:
  tls:
    - hosts:
        - nginx.yownowndomain.com
      secretName: tls-nginx-ingress
  rules:
    - host: nginx.yownowndomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nginx
                port:
                  number: 80

Once you apply these manifests, simply do a kubectl get certificate. This will show that a new certificate and secret resource got automatically created.

kubectl get certificate
NAME                READY   SECRET              AGE
tls-nginx-ingress   False   tls-nginx-ingress   61s

Wait for a few minutes. You should see that cerficicate ready status should be set to True.

You can also describe this certificate to see the events.

Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    91s   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  90s   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "tls-nginx-ingress-2mvlp"
  Normal  Requested  90s   cert-manager-certificates-request-manager  Created new CertificateRequest resource "tls-nginx-ingress-1"
  Normal  Issuing    11s   cert-manager-certificates-issuing          The certificate has been successfully issued

To expose this on the internet, you will have to go back to Cloudflare and configure a CNAME forwarding to the sub-domain that points to your router’s external IP, given that you have already configured dynamic DNS.

You can also test this by configuring a CNAME entry on your local network’s DNS.

Once you ge this working you should be able to see a valid certificate when you access your nginx page.

The certificate will have a validity of 3 months but you don’t have to worry since Cert-manager will renew this automatically.

The network just reached the fourth month mark and it’s already about to undergo a somewhat major re-design. But just before going to the next chapter of the journey, I want to take this time to log the current state of the existing infrastructure and services. Don’t get me wrong — I don’t run a full-fledged data center, nor do I even have a server rack setup. I only have photos and videos as critical files and I only have two users at home including myself (lol). It’s called a homelab for a reason. Though it would still be fun to document the progress be it big or small and know how far I’ve come along the way.

Homelab Networking and Hardware

On high level, below is the current diagram of existing services, subnets, VLANs, and devices.

Orange boxes signify the servers running a hypervisor — one Proxmox, and the other, Unraid. The Proxmox box is mainly running my network stack which includes OPNsense, a LXC container running Pihole as my DNS, and another one running tailscale for a machine-to-machine VPN with the remote peer acting as an exit node. While it might be possible to run tailscale from within OPNsense, I decided to configure it separately since I only have a simple use case of tunneling torrent traffic to a small thin client back home in Manila.

On the other hand, the box running Unraid primarily runs our NAS and a couple of docker containers and VMs. It’s been very stable, running without any hiccup except for the time when all containers and VMs together with all the other devices were on a single subnet. This is what pushed me to segregate the traffic with VLANs. Speaking of, this is easily made possible by a cheap “smart” (you can say it’s managed) switch from TP-Link. I can’t stress enough how much I love this brand! I’ve been using their routers for more than a decade now and it has never failed me.

Back to the Unraid server, it had also undergone some few upgrades (the initial specs can be found in one of my first few posts here). First is the addition of a 500GB SSD as an un-mirrored cache drive to avoid wear and tear of the NVMe drives. This is also mainly used for non-critical data such as when storing movies, tv shows, and ISO files, etc.

Second is the installation of a third party heat sink for the NVMe drives which brought down temperature levels by about 5-10C.

Third, the installation of a 330W UPS from APC, allowing graceful shutdown in case of power outages and protection from potential power surges.

Fourth, the installation of a Noctua NH-L12S cooler to pull down the temperature levels of the CPU.

Lastly, a RAM upgrade to 64GB after experiencing memory shortage when working with GNS3. 32GB wasn’t enough when running multiple containers and spawning multiple VMs on top of GNS3.

Services

Talking about the services, I am running just a handful. For the monitoring stack I have Telegraf, InfluxDB and Grafana to monitor resource utilization on the Proxmox box. I also have UptimeKuma providing a neat and simple dashboard for monitoring service uptime.

For resource monitoring of my Unraid box, I just use the built-in dashboard which already gives me all the information I need such as CPU and disk temps, CPU, disk, and memory utilization, and power consumption.

For productivity and utilities, I have Guacamole for VM and command line remote access, Krusader for file management within Unraid, Visual Studio Code for modifying configuration files and blogging, Firefox for VM-less browser access, Wiki.js for saving important information and notes taking of anything related to the Homelab and the projects I am working on, Jdownloader for HTTP downloads (mainly ISO files), Heimdall as my homelab dashboard, and Traefik as my reverse proxy!

For media services, I have Jellyfin for transcoding and serving media both at home and when on-the-go. To populate my fiance and I’s media inventory with videos from our favorite youtube channels, I use ytdl-sub. This automatically downloads the videos of any defined youtube channel, manages the specified retention period, and adds the necessary metadata for a nice and smooth feel when browsing in Jellyfin. And since the videos are played back locally, this means we also get ad-free content when watching. For legal downloading of media, I have deluge. Since I don’t use this often, I still transfer files manually to the Jellyfin directory.

I am aware I am still lacking in the automated media management department by making use of the *ARR services, but I can’t find time to set them up especially now that this homelab environment keeps me branching out to different projects all at the same time. At the moment I just have bigger fish to fry.

VMs

As for the VMs, I only have quite a few. My main workstation is running PopOS based on Ubuntu. I do use a Macbook at home but still login with RDP to this machine almost all the time since it allows me anytime-anywhere remote access and I can easily pick up where I left off when I am working on something.

Though 99% in shutdown state, I also keep a Windows 10 VM in case I need to use software which can’t run on Linux. I also keep a staging environment running on Debian for the remote tailscale exit node installed back home in Manila. Another one is an Ubuntu VM running GNS3 server for other side projects on kubernetes and networking. My VM disks are created in the NVMe drives allowing me to play around with VMs efficiently in terms of very quick load times. The NVMe drives are also mirrored in RAID1 so I don’t have to worry about losing data when one of the two drives crashes.

The last but not the least is my router! Running as a VM on Proxmox, it just works wonderfully. I did experience random reboots though, but this got fixed after upgrading the kernel from 5.4 to the new 6.1 together with the intel micro-code package. My longest uptime with it was around 40 days before I had to reboot for maintenance work. Inside OPNsense I am running Unbound caching DNS resolutions for faster downstream queries. The browsing experience became significanly better after enabling this. Websites started to load faster also after enabling DNS over HTTPS. I was trying to find an explanation to this but I will just leave it out in the cold for now. I also have my Cloudflare Dynamic DNS agent running here. For VPN, I have Wireguard allowing me unfiltered access to the entire network. I also use this on daily basis — tunneling at the very least my DNS traffic to Pihole for internet ads blocking.

I think that’s all of it for the network’s status quo. Over the next few weeks I will have to wrap my head around Terraform, Ansible, and Kubernetes. This is because I’ll be migrating most, if not all, of these services to a “production” k8s cluster with the help of automation. I’ve been sitting on the fence about this for quite some time now, but I finally decided go with it to have hands-on experience with Infrastructure-as-Code. Is it necessary? Not really, but it sure is interesting.

Install Gravity Sync

Keep in mind that the following IP addresses were used for the setup:

I already had one pihole instance running on 10.0.0.80. I just had to shut this down to test my new HA-configured pihole.

To keep the configuration of both piholes in sync, we can install gravity-sync. I faced permission issues when I ran this as root. Make sure you have PermitRootLogin yes set in the /etc/ssh/sshd_config file. Then only execute below installation script.

curl -sSL https://raw.githubusercontent.com/vmstan/gs-install/main/gs-install.sh | bash

The installation procedure is straightforward. Just input the required information and wait for it to complete. You can then conduct a quick test of pulling or pushing configuration.

Create a new domain in pihole1 and ensure it’s not reflected on pihole2.

You can then execute gravity-sync compare to see if there is any configuration mismatch.

Then you can execute gravity-sync pull from the pihole2 and check if the delta configuration is reflected.

Automate

Execute gravity-sync auto on both pihole instances.

By default pihole should sync after 5 mins. For my setup I will set it hourly.

gravity-sync auto hour

Install keepalived

I found this tutorial on reddit link here shared by Panja0 on how to make use of keepalived by monitoring the pihole-FTL service with a simple bash script. I wanted to try and do it from scratch to add to my knowledge but I have decided to park this for now and test out this tutorial. For those interested, the keepalived documentation can be found here.

Install libipset13 and keepalived.

yum install -y libipset13 keepalived

Create a new scripts directory in /etc.

mkdir /etc/scripts

Create a new file and copy-paste the content from https://pastebin.com/npw6tcuk.

Add permision to execute the file:

chmod u+x /etc/scripts/chk_ftl

You need to edit keepalived.conf on both pihole instances. This file is located in /etc/keepalived/. The config files can be downloaded in the following links: keealived.conf for master keealived.conf for master

Modify the following fields according to your IP configuration. The auth_pass field is a password that should be match between both configuration files. unicast_src_ip unicast_peer auth_pass virtual_ipaddress

Restart the keepalived service and now the IP should be assigned to your primary pihole!

sudo systemctl restart keepalived

You should now be able to see DNS queries coming in.

Testing for auto fail-over

Shutdown the other pihole. Now pihole2b started accepting traffic

Let’s see if it will preempt. Poweron other pihole and check the keepalived. You can also check 10.0.0.80/admin (virtual ip) and see if it’s routed to the primary .

Before going further, first, I’d like to thank the author of the tutorial, Infinity Don over at Dev.to, for sharing this cool idea together with the docker images. Ever since my interest in virtualization and containerization started, I have been looking for a weekend past-time for getting my hands dirty with, and luckily I stumbled upon his article here.

The article details on how to get everything running but I somehow still faced few minor issues. I’d still recommend to refer to the tutorial if you are someone who takes troubleshooting as a fun challenge.

I like recording everything I execute so if you have the intention of creating your own network and don’t want to be spoiled, then I’d suggest not to continue and just go read the tutorial above instead.

If you are just interested in how I got things working, then feel free to continue.

The objective

In bullet points:

• Create a cluster for running EPC components as simple as possible.
• Use k3s as the kubernetes platform to have a smaller resource footprint.
• Use Calico as the CNI instead of flannel (comes with k3s by default).
• Use GNS3 for network simulation so I can start getting familiar with it for future projects.
• Learn packet core!

VMs and Networking

The k3s cluster is running on Ubuntu 22.04 VMs. Openairinterface5G as the RAN and UE simulator as in the tutorial, though both are bifurcated into two VMs (the tutorial makes use of a single VM with the help of a second loopback address). For the latter, Ubuntu 18.04 was used to avoid any incompatibility issue since the documentation stated only this version. All Ubuntu VMs are assigned 4GB RAM. Ubuntu 22 VMs are assigned 2 vCPUs, and the two Ubuntu 18 VMs, with 4 vCPUs to speed up the build time of the simulator. For the router, the same Vyos router is used with separate connections for the management interface and the internet. I could have made use of the management interface as my internet gateway but I faced slow throughput when downloading packages on the VMs. The suggestion I got in the GNS3 forum is to use a NAT interface instead.

Preparing the environment

I am running a remote GNS3 server inside another Ubuntu VM. For this I had to enable nested virtualization (Intel VT-x in my case) in the BIOS. I also had to passthrough the CPU instruction set extensions to my GNS3 host, and from the host to the Ubuntu 18.04 VMs, to clear some errors I faced in the initial attempt of building the eNB and UE simulator. To make it easier I just configured CPU passthrough on both my hypervisor and the GNS3 server.

You can check if your VMs are having the CPU extended instruction set flags using the command below:

cat /proc/cpuinfo | grep -i avx

VM images were created with already updated packages so I don’t have to keep on updating everytime I have to re-instantiate a VM. For updating manually:

sudo apt-get update -y && sudo apt-get upgrade -y 

As for the kubernetes platform, I used k3s. It comes with flannel as the CNI but it can be installed with Calico. The steps I followed for this are logged in another post.

Vyos configuration

Configuring VyOS is fairly simple though prior CLI experience with routers such as cisco and the likes will surely help. The documentation can be found here for installing Vyos on GNS3, and the configuration guide in this link.

To provide internet connectivity to my VMs I had to masquerade internal VM IPs towards the NAT GW. A static default route is set towards this GW, and the rest of routes are mainly for management access from the different external VLANs I have in my local network.

As for the BGP configuration, the command in the tutorial seems to be a little outdated. The following commands were used to enable BGP:

set protocols bgp neighbor 10.200.0.12 remote-as 63400
set protocols bgp system-as 63400
set protocols bgp neighbor 10.200.0.12 address-family ipv4-unicast 

I faced some minor timing issue when merging pcap files. To fix, all I had to do was to ensure the same NTP server is used across all nodes.

Snippet of my Vyos configuration:

interfaces {
    ethernet eth0 {
        address 10.200.0.1/24
        hw-id 0c:0a:6b:f7:00:00
    }
    ethernet eth1 {
        address 172.16.0.1/24
        hw-id 0c:0a:6b:f7:00:01
    }
    ethernet eth2 {
        hw-id 0c:0a:6b:f7:00:02
    }
    ethernet eth3 {
        address 10.20.0.20/24
        hw-id 0c:0a:6b:f7:00:03
    }
    ethernet eth4 {
        address dhcp
        hw-id 0c:0a:6b:f7:00:04
    }
    loopback lo {
    }
}
nat {
    source {
        rule 100 {
            outbound-interface eth4
            source {
                address 10.200.0.0/24
            }
            translation {
                address masquerade
            }
        }
        rule 101 {
            outbound-interface eth4
            source {
                address 172.16.0.0/24
            }
            translation {
                address masquerade
            }
        }
    }
}
protocols {
    bgp {
        neighbor 10.200.0.12 {
            address-family {
                ipv4-unicast {
                }
            }
            remote-as 63400
        }
        system-as 63400
    }
    static {
        route 0.0.0.0/0 {
            next-hop 192.168.122.1 {
            }
        }
        route 10.0.0.0/24 {
            next-hop 10.20.0.1 {
            }
        }
        route 10.10.0.0/24 {
            next-hop 10.20.0.1 {
            }
        }
        route 10.30.0.0/24 {
            next-hop 10.20.0.1 {
            }
        }
        route 10.120.0.0/24 {
            next-hop 10.20.0.1 {
            }
        }
        route 10.130.0.0/24 {
            next-hop 10.20.0.1 {
            }
        }
        route 10.140.0.0/24 {
            next-hop 10.20.0.1 {
            }
        }
    }
}
service {
    ntp {
        allow-client {
            address 0.0.0.0/0
            address ::/0
        }
        server ntp.ubuntu.com {
            prefer
        }
    }
    ssh {
        port 22
    }
}

Installing Open5gs

Once the environment is ready, then it’s time to install Open5gs. Clone the repository to your local directory from Infinity Don’s repository.

git clone https://bitbucket.org/infinitydon/virtual-4g-simulator/src/master/open5gs/

The yaml files are based on older versions of kubernetes, so to make this work with the latest one at the time of this writing, that is v1.26.3, I had to modify hss-database/mongo-statefulset.yaml. Just update the apiVersion to rbac.authorization.k8s.io/v1 (remove beta1).

I had to specify the mongodb version as well in hss-database/monogo-statefulset.yaml to avoid mongodb going to crashloopback state (reference here).

      containers:
        - name: mongo
          image: mongo:4.2.10

Port 2123 on the SMF had to be opened to allow GTP-C connections with SGW. Update smf/smf-deploy.yaml and add the following.

smf-deploy.yaml

  - name: gtpc
    port: 2123
    protocol: UDP

If BGP is to be used then this next step is optional. This how I established the connection of the eNB to the MME before the realizing I would need BGP eventually. Here I just modified the mme-deploy.yaml to expose the MME IP for the s1ap connectivity.

mme-deploy.conf

apiVersion: v1
kind: Service
metadata:
  name: open5gs-mme-svc-pool
  namespace: open5gs
  labels:
    epc-mode: mme
spec:
  ports:
#  - name: s1ap
#    port: 36412
#    protocol: SCTP
#    nodePort: 36412
  - name: s6a
    port: 3868
    protocol: TCP    
    nodePort: 30001
  type: NodePort
  selector:
    epc-mode: mme

The mme-configmap.yaml was also updated to set TAC to 1.

For the BGP configuration, the same yaml files from the tutorial were used. These yaml files are not present in the repository. Created a new directory and two yaml files with following configuration:

calico-bgpconfig.yaml

apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  asNumber: 63400

calico-bgppeer.yaml

apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
    name: bgppeer-vyos
spec:
    asNumber: 63400
    peerIP: 10.200.0.1

Time to spin up the core! I tried to execute in small steps but then encountered pods getting stuck in crashloopback state. This is because each pod was trying to bring up the peer connectivity with other pods.

First create the namespace:

kubectl create ns open5gs

Then execute all at once!

kubectl apply -f hss-database/
kubectl apply -f web-ui/
kubectl apply -f hss/
kubectl apply -f mme/
kubectl apply -f upf
kubectl apply -f sgw-u/
kubectl apply -f sgw-c/
kubectl apply -f smf/
kubectl apply -f pcrf/
kubectl apply -f nrf/
kubectl apply -f calico/

Expect some pods to be restarting and wait until you see all in running state.

Check the WebUI if you are able to access with default credentials admin / 1423.

Install OAISIM5G

Next is to install the simulator. I exhausted a lot of time trying to make it work with the latest available version only to realize that the article I was referring to is no longer valid after a certain development tag.

Download and extract the latest package suggested in the article:

wget https://gitlab.eurecom.fr/oai/openairinterface5g/-/archive/2021.w51_c/openairinterface5g-2021.w51_c.tar.gz
tar -zxvf openairinterface5g-2021.w51_c.tar.gz

enodeB configuration file

ci-scripts/conf_files/rcc.band7.tm1.nfapi.conf

    ////////// MME parameters:
    mme_ip_address      = ( { ipv4       = "10.200.0.11";
                              ipv6       = "192:168:30::17";
                              port       = 30002 ;
                              active     = "yes";
                              preference = "ipv4";
                            }
                          );
...                          
...
    NETWORK_INTERFACES :
    {
        ENB_INTERFACE_NAME_FOR_S1_MME            = "ens4";
        ENB_IPV4_ADDRESS_FOR_S1_MME              = "172.16.0.30";
        ENB_INTERFACE_NAME_FOR_S1U               = "ens4";
        ENB_IPV4_ADDRESS_FOR_S1U                 = "172.16.0.30";
        ENB_PORT_FOR_S1U                         = 2152; # Spec 2152
        ENB_IPV4_ADDRESS_FOR_X2C                 = "172.16.0.30";
        ENB_PORT_FOR_X2C                         = 36422; # Spec 36422

    };
...
...
MACRLCs = (
        {
        num_cc = 1;
        local_s_if_name  = "ens4";
        remote_s_address = "172.16.0.40";
        local_s_address  = "172.16.0.30";
        local_s_portc    = 50001;
        remote_s_portc   = 50000;
        local_s_portd    = 50011;
        remote_s_portd   = 50010;
        tr_s_preference = "nfapi";
        tr_n_preference = "local_RRC";
        }

Optional aliases for running OAISIM eNB and checking the logs:

alias oaisimenb='cd ~/openairinterface5g-2021.w51_c/cmake_targets/ran_build/build; sudo -E ./lte-softmodem -O ../../../ci-scripts/conf_files/rcc.band7.tm1.nfapi.conf > enb.log 2>&1'
alias oaisimenblogs='tail -f ~/openairinterface5g-2021.w51_c/cmake_targets/ran_build/build/enb.log'

UE configuration

ci-scripts/conf_files/ue.nfapi.conf with your preferred editor

...
...
L1s = (
        {
        num_cc = 1;
        tr_n_preference = "nfapi";
        local_n_if_name  = "ens4";
        remote_n_address = "172.16.0.30";
        local_n_address  = "172.16.0.40";
        local_n_portc    = 50000;
        remote_n_portc   = 50001;
        local_n_portd    = 50010;
        remote_n_portd   = 50011;
        }
);
...
...

Optional aliases for running OAISIM UE and checking the logs:

alias oaisimue='cd ~/openairinterface5g-2021.w51_c/cmake_targets/ran_build/build; nohup sudo -E ./lte-uesoftmodem -O ../../../ci-scripts/conf_files/ue.nfapi.conf --L2-emul 3 --num-ues 1 --nums_ue_thread 1 > ue.log 2>&1'
alias oaisimuelogs='tail -f ~/openairinterface5g-2021.w51_c/cmake_targets/ran_build/build/ue.log'

Building the simulators

When building the simulators the first time, it’s important to execute with the -I flag so it will also install the package dependencies. If you are building without -I after a fresh Ubuntu install, you will face some errors: e.g. for eNB:

./build_oai -c -I --eNB

e.g. for UE:

./build_oai -c -I --UE

Define the HSS profile and run the simulator

The UE information can be taken from openair3/NAS/TOOLS/ue_eurecom_test_sfr.conf. This should be defined in the HSS via the WebUI.

Once that is done, fire up the eNB and the UE. I lose ssh access to the VM whenever I start the UE simulator . I have to resort to VNC access to initiate some traffic like ping or iperf. The VM should have the PGW provided IP assigned to oaitun_ue1.

Below is a sample trace on the Vyos router for an UE ping request to the Google DNS. The first IP in each packet is the eNB IP and the second one of the UE’s.

Below some throughput testing to a public IPerf server. First one is from the VM’s physical interface, second one over GTP.

My Sentiments:

I originally planned of avoiding BGP and save that for a later learning. But it was realized along the way that the eNB needs to be able to reach the pod IP address of the SGW. I initially configured a NAT on the Vyos router for this – DNAT-ing UDP traffic for the uplink from the SGW Pod IP (3GPP TS 36.413) to an exposed NodePort. While this worked with the eNB accepting the traffic, the response packets are coming out from a different SGW/worker source port. This shouldn’t be the case. The response destination port must be copied from the request source port according to 3GPP TS 29.281. I tried to configure another NAT for the response but the Vyos router just dropped the packets. Enabling the NAT logs didn’t help either because no logs were populating.

The solution was to resort to configuring BGP – exposing the Pod IP to the Vyos router so the eNB can reach SGW without any translation.

Talking about if this kind of setup will be helpful for learning EPC, I can’t really say yet. I published this post before I forget the logistics and everything, though I’m sure there’s a lot more to explore and it does feel like I can do a lot more with it. I also do not have much experience on the packet core side of things so it’s a bit hard to gauge but doing this project alone made me refer to 3GPP specs out of minimal curiousity. If playing around with these existing docker images will be tough, the option to install open5gs with VMs is always there. Another idea is to build docker images with the latest version of open5gs. In the end it all really depends on your objective. But whether what you plan to do is something simple or something complex, I am sure there will always be learning along the way.

Other errors faced

I was able to capture some of the errors (not all) faced during the deployment. One of the first ones that showed up is the AVX warning error in the first attempt of building the OAISIM5G simulator. I was clueless at first since google wasn’t showing anything until I found a PDF stating that OAISIM5G needs a CPU that supports AVX. Realized that this must be listed under the extended instruction sets. So when I checked and found it there, I just had to make sure those flags are propagated all the way to my Ubuntu VMs.

If you try to build the simulator without the -I flag, you will also face the following error. While you can also get away with it by manually installing the listed packages, it’s just better to do it by the official documentation to avoid potential problems later on.

At one point I could see tunneled interface getting created on the UE simulator but with no IP getting assigned. When I checked the SGW logs it was showing no response is getting received for create session request. This is where I had to open up port 2123 for the connection to go through.

Below is also a snapshot showing the UDP response for GTP of SGW-U coming out of a different source IP due to the worker node performing a NAT on the source Pod IP. While you may be able to find a work around with it by configuring another NAT, I’ve had no luck trying to make it work. I retired from doing so since anyway a Pod IP (at least in this kind of setup) isn’t really best paired with a NAT.

Looking more into Traefik

I took reference from two sources to understand how Traefik works and how it can be used with my docker containers on Unraid. The yaml template I used is also sourced from the Ibracorp link.

• Traefik documentation
• Ibracorp tutorial

In summary, Traefik can query docker via API with the help of container labels. Rules depend on those labels and so as long as you have the correct yaml configuration, you can treat it as set and forget. For those containers that you will be running in the times ahead, you just need to follow the same label structure.

The tutorial in the link includes configuring MFA by setting up a middleware. But I have a basic requirement of just providing external access to my services. So I will be saving this part for later in case I still require it in the future.

I had the following goals in mind:

1. Be able to access containers named <container_name> externally from the internet via <container_name>.su-root.net.
2. Be able to access <container_name> externally from the internet via <container_name>.su-root.net.
3. Configure #1 and #2 with the least manual configuration required (but still secure).

Given this, I will only go through the necessary configuration in as little footprint as possible.

Starting with traefik

There are two different configuration files, a static one read at startup, and a dynamic one used during runtime.

Static configuration (traefik.yml) -> Define the entry points, providers, certificate provider.

Dynamic coniguration (fileConfig.yml) -> Define the routing and middleware.

There are multiple providers supported by Traefik but I will only focus on docker that is natively used by Unraid.

I can simply make use of the labels if I want to route traffic to specific containers that have multiple ports or those having a different container name against the cname I want to use.

traefik.yml configuration

The first part is to define the entry points where incoming traffic coming is expected.

• Define http entry point and redirect to https
• Define https entry point to use a certificate generated by Let’s Encrypt.
• Define the middleware so we enforce the addition of STS header.

Next is to define the providers:

• fileConfig.yml file that contains the dynamic routing configuration
• Docker by quering using API via dockerproxy.

Enable the UI and print info level logs.

And the last but not the least, the settings for Let’s Encrypt, the certificate provider.

global:
  checkNewVersion: true
  sendAnonymousUsage: false

serversTransport:
  insecureSkipVerify: true

entryPoints:
  http:
    address: ':80'
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https

  https:
    address: ':443'
    http:
      tls:
        certResolver: letsencrypt
        domains:
          - main: mydomain.com
            sans:
              - '*.mydomain.com'
      middlewares:
        - 'securityHeaders@file'

providers:
  providersThrottleDuration: 2s

  file:
    filename: /etc/traefik/fileConfig.yml
    watch: true

  docker:
    watch: true
    # Default host rule to containername.domain.example
    defaultRule: "Host(`{{ lower (trimPrefix `/` .Name )}}.mydomain.com`)"    # Replace with your domain
    exposedByDefault: false
    endpoint: "tcp://dockersocket:2375" # Uncomment if you are using docker socket proxy

# Enable traefik ui 
api:
  dashboard: true
  insecure: true

log:
  level: INFO

# Use letsencrypt to generate ssl serficiates
certificatesResolvers:
  letsencrypt:
    acme:
      email: mail@mydomain.com
      storage: /etc/traefik/acme.json
      dnsChallenge:
        provider: cloudflare
        # Used to make sure the dns challenge is propagated to the rights dns servers
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

Before I go to the other next yaml file, it’s important to note that I use IPvlan networking for all my docker containers. This provides a good level of network isolation against the host and I just find it way easier to manage my applications with unique IP addresses. The need to mention is due to the fact that I cannot get Traefik to detect the exposed port on the external IP address unless I explicitly define it with a label. I cannot find any mention of this in the documentation and I haven’t done enough experiments to say that this works without a label when using a different network type such a bridge or a custom docker networks, and if it makes sense, even with an untagged ipvlan network.

Sample error I get without the loadbalancer label:

“level=error msg=“service "kuma" error: port is missing” providerName=docker container=kuma-cf32b….”

fileConfig.yml configuration

Moving over to the dynamic configuration, if you want to route to non-docker services OR if for any reason you require specific routing for a container, you can define them here.

First, you have to specify the entry point for the router. Next is the rule that needs to fulfilled for this router to be activated. And third, the mapped service containing the destination and port.

The middleware you defined in the first yaml file will also be referenced from here. TLS options can also be in this file but it is not a requirement. I haven’t gone through each of header but since I can see this enables STS, then I will just leave it as it is.

http:
  routers:
    opnsense:
      entryPoints:
        - https
      rule: 'Host(`opnsense.mydomain.com`)'
      service: opnsense

    proxmox:
      entryPoints:
        - https
      rule: 'Host(`proxmox.mydomain.com`)'
      service: proxmox

  services:
    opnsense:
      loadBalancer:
        servers:
          - url: https://10.100.0.1/

    proxmox:
      loadBalancer:
        servers:
          - url: https://10.100.0.2:8006/

  middlewares:
    securityHeaders:
      headers:
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
          X-Forwarded-Proto: "https"
          server: ""
        customRequestHeaders:
          X-Forwarded-Proto: "https"
        sslProxyHeaders:
          X-Forwarded-Proto: "https"
        referrerPolicy: "same-origin"
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        contentTypeNosniff: true
        browserXssFilter: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsSeconds: 63072000
        stsPreload: true

# Only use secure ciphers - https://ssl-config.mozilla.org/#server=traefik&version=2.6.0&config=intermediate&guideline=5.6              
tls:
  options:
    default:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

Exposing containers

Traefik will automatically and implicitly create a router and a service for each of the exposed container and the only thing you need is a set of labels:

1. For traefik to detect the container traefik.enable=true

2. To define the allowed entry point traefik.http.routers.APP_NAME.entryPoints=https

3. So traefik knows to which port it has to route the packets traefik.http.services.APP_NAME.loadbalancer.server.port=APP_PORT

4. To tell traefik to use some another router name. (optional) traefik.http.routers.APP_NAME.rule=Host(\APP_NAME.mydomain.com`)`

The last one in the list is required only if you want traefik to route a sub-domain that is different from the container name. e.g. you want traefik to route kuma.mydomain.net to a container named uptime-kuma.

Below is an example of a container having three labels:

Make sure you have the respective cnames on your DNS pointing to Traefik, and once that is all set, you can start accessing your services over https. I would also suggest to test the redirection at least for the first service you are exposing.

When you have multiple services exposed, you might want to check out the Traefik UI to get a good visibility of your configuration. Do note that you will not be able to do any action from here and it’s solely for checking the active routes.

Understanding Traefik might be a bit confusing at first but the more you read about the documentation, the more it gets interesting. Perhaps I’ll be able to use more of its other functions in future projects!

When the remote server was all ready and configured together with VNC, I then created an LXC container, installed Deluge, and configured the tailscale client to use the remote server as an exit node.

sudo tailscale up --exit-node=<tailscale_ip> --exit-node-allow-lan-access=true

Do note that setting “–exit-node-allow-lan-access” to true is required or else you wouldn’t be able to access the Deluge WebUI. This parameter will only recognize the local configured subnet so if your container is sitting inside a different VLAN like mine, then you will also need to configure an outbound NAT rule on your firewall or router. You should configure it in a way that any private local IP (outside the container subnet) that tries to reach your container, should have the Source IP translated to the Vlan interface IP of your router.

I am sure there are other better ways to implement this but if you simply need a torrent client with tunneled traffic, then I believe this is the best and simplest it can be done.

Proxmox already supports LXC containers by default and in this case, running Pihole on LXC provides some advantage in terms of flexibility and ease of configuration. This is especially for those who have less experience working with docker but only with Linux in general. I will share one good use case for this later on but for now let’s start setting up Pihole on Proxmox.

The hardware requirements according to the pihole documentation: Minimum of 2GB disk (4GB recommended) 512MB RAM

First you need to download an LXC template. I like using debian as the base image since it’s very lightweight and it’s where Ubuntu is based from. If you don’t have it yet, you can download it by going to Datacenter > pve > storage > CT templates > Templates and search for Debian 11 Bullseye.

Click on Create CT and input a container ID, hostname, and the root password. Check Unprivileged container and nesting (optional). Note that nesting is not really required but in my case the proxmox terminal will keep on printing permission errors if I leave this unchecked.

Select debian as the template.

For disk, I assigned 6GB. For RAM, double the recommended, since I have enough.

Assign a static IP to the container. Gateway should also be defined.

For DNS I will assign my unbound IP which is the same as my OPNsense gateway IP. (Unbound is a DNS caching tool built-into OPNsense). This will basically be my upstream DNS for my Pihole. If you don’t have Unbound running then you can input any public DNS like Google (8.8.8.8, 8.8.4.4) or Cloudflare (1.1.1.1).

Confirm

Enable start on boot flag.

Login as root

Update and upgrade

apt-get update -y && apt-get upgrade -y

Install curl:

apt-get install curl -y

Install Pihole:

curl -sSL https://install.pi-hole.net | bash

Install custom upstream and point to unbound IP

Install the default blacklist.

Install admin interface and lighthttpd.

Query logging or any other option is fine.

Reset the pihole password.

sudo pihole -a -p

Now go to your container IP and append /admin (e.g. http://10.0.0.88/admin)

If you are running it on proxmox like me you’d probably get the same error as in the snap below. You can ignore this especially if you have multi-core host. If you want to be sure you can check your CPU utilization with the top command.

After changing your DNS to pihole, check with nslookup if your device is able to send and receive to and from the Pihole IP.

As for the specs, it’s a fanless appliance running a quad-core Intel 12th-gen J6412 (2 GHz base, 2.6 GHz boost) with 5x Intel i226 2.5Gbe NICs. I chose the bare minimum option so I can avoid the crappy RAM and storage that might come with it. For the RAM I was able to source second-hand 2x8GB Samsung DDR4 SODIMMs and for the storage I didn’t any more bother and just ordered a 240GB SSD from Amazon since these come cheap these days.

Installed Proxmox and spawned a VM running a router software based on freeBSD called OPNsense. I can’t believe I’ve been missing on this piece of software for the longest time. It feels like I’ve rekindled my love for networking ever since I switched to a different role in my career graduating from configuring routers and switches.

The entire setup process was very straightforward. I just got confused a tad bit in the beginning when I was trying to figure out which interface to assign for management access. On my first attempt, I assigned the first interface in the list, expecting that it would be mapped to the first physical port. But when I tried to plug an ethernet cable from my laptop, for some reason I couldn’t ping the management IP I set. Long story short I re-installed Proxmox again only to know that the first installation was already correct. Maybe some lose cable?

Anyway the router has been running stable for the past week. This is with virtio running as a driver for the Intel i226 NICs. On average I’m getting about 500Mbps for internet (I used to get about 700MBps with a TP-Link A10 router and yes I can deal with that 200Mbps difference for now!). I’ll explore physically passing through the NICs later on once I get the time. But as far as I can tell, I’m just loving it. On top of that Proxmox has also been remarkably gaining my interest now that I have a pi-hole running in an LXC container.

K3s makes use of containerd as the default container platform but I opted to go with docker since I want to become more familiar with it. Note that at this time docker is still considered experimental with k3s as written in the official documentation but I still went on with it anyway. In the following steps I also included how you can execute the kubectl commands without the need of sudo and how they can be executed from a remote machine.

Installation on master node

I have the following setup at the moment where my cluster has a node subnet of 20.0.0.0/24.

Before anything else, modify the hosts on file on each node to reflect the IP and hostnames of the your entire cluster:

127.0.0.1 localhost
20.0.0.10 k8s-master m
20.0.0.11 k8s-worker-1 w1
20.0.0.12 k8s-worker-2 w2

Update and upgrade ubuntu packages:

sudo apt update
sudo apt upgrade
sudo reboot

Install required tools and docker on both master and worker nodes. First add the docker repository.

sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"

Install docker:

sudo apt update && sudo apt install -y docker-ce

Run docker and enable persistence in case of server reboot:

sudo systemctl start docker
sudo systemctl enable docker

Check if docker is running:

sudo systemctl status docker

Install k3s on the master node:

curl -sfL https://get.k3s.io | sh -s - --docker

Check status of k3s:

sudo systemctl status k3s

Check if node is running:

sudo kubectl get node -o wide

As an optional step, unblock ports used by kubernetes. By default firewall will be inactive but in case you have enabled it previously, then you need to perform the following:

sudo ufw allow 6443/tcp
sudo ufw allow 443/tcp

You can also check the status of your firewall with:

ufw status

Disable with:

ufw disable

Extract the token to be used for joining worker nodes:

sudo cat /var/lib/rancher/k3s/server/node-token

Installation on worker node

Next is to install k3s on the worker nodes. Here you will need the master node IP address as well as the token from the previous step.

curl -sfL http://get.k3s.io | K3S_URL=https://<master_IP>:6443 K3S_TOKEN=<join_token> sh -s - --docker

Verify all nodes are joined to the cluster

sudo kubectl get nodes -o wide

Now you can try to deploy a test nginx pod

sudo kubectl run --image nginx nginx-test
sudo kubectl get pod nginx-test -o wide

Permit non-root user to execute kubectl commands

Now if you try to execute kubectl commands without sudo, you would probably face the error below.

This is because kubernetes will be installed with root credentials and any other user to execute kubectl will have to be permitted. The easy way is to copy the k3s.yaml file in the /etc/rancher/k3s directory to your user’s kube config file. First create the directory (with your non-root user) if you don’t have it yet. Note of the user and group variables:

mkdir ~/.kube
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
chown <user>:<group> ~/.kube/config

Then you have to export the kube config every time so to do this automatically upon login modify your .bashrc and add the line:

export KUBECONFIG=~/.kube/config

For you to be able to execute kubectl commands with just typing ‘k’ you can also add the following:

source <(kubectl completion bash)
alias k=kubectl
complete -o default -F __start_kubectl k

Lastly if you want to be able to execute kubectl commands from a remote machine outside your cluster just ensure you have the same kube config file in the previous step and change the server IP from localhost to the actual one.

The port forwarding function can be found in Advanced Settings > NAT forwarding > Virtual Server. Do note the name and location of this function can vary between brands and even models of the same brand.

Here I just need to make a new entry to point the external port 443 to my Unraid internal IP with port 1443.

To put it simply how this works, anyone who tries to access your public IP address via https or port 443 will be redirected to whatever local IP and port you specify. Now you can try to access your external IP with http or whatever and your connection should be forwarded to the correct service. In the example below nc.su-root.net is pointing to my public IP and internally getting forwarded to Swag.

For us who don’t want to pay more for a static IP, luckily there is a service called dynamic DNS. This is, for most of the time, free and should be supported by most modern home routers.

What is Dynamic DNS (DDNS)?

Let me explain this in layman’s terms. First, DNS stands for domain name service. A domain is the name you type when you want to access a website such as www.google.com or www.su-root.net. Domains will always be mapped to a public IP address as long as it’s registered.

Normally, a website will have a static public IP throughout its lifetime unless it is moved to a different server. For websites hosted by hosting companies such as AWS or Bluehost, the mapping of the IP address to a domain name is most of the time managed by them. This is as long as the domain provider is configured to use their nameservers.

For private hosts running behind a normal internet service provider, this is where DDNS comes into picture. If your ISP keeps changing your IP address (e.g. daily or weekly), then DDNS keeps track of this and updates their records automatically (as long as your router supports this feature). This is the best way for people who don’t want to acquire a static IP that incur additional costs on their monthly bill.

How do you configure Dynamic DNS?

Now let me walk you through how to configure it with the help of my home router. Before anything else you need to create an account from a DDNS provider such as no-ip.

From the Dashboard, go to > Dynamic DNS > No-IP Hostnames then click Create Hostname.

Input your preferred hostname and select any of the free domains. Keep Record Type as DNS Host (A), input your current public IP address, and proceed to create the hostname. You can check your IP here if you don’t know it yet.

Next is to configure the DDNS on the router.

What I have is a TP-Link Archer A10 from 3 years ago and it still works perfectly fine without any hiccup. Like I said, most modern routers should support DDNS, but if yours don’t and you plan to walk the path of self-hosting, then you can get a new or even a used one for just a couple of bucks.

Login to your router’s web GUI. This is, for 99% of the time, the same as your default gateway IP. Mine happens to be “http://10.0.0.1”. You can also check your current WAN IP from here.

Next, go to the dynamic DNS configuration. In my case it’s under Advanced Settings > Network. Select NO-IP, input your credentials and the newly created domain. Save.

From here on your router will keep updating your DDNS (No-IP) every time your IP address changes. To verify if your configuration is working, you can do a DNS look up and check if it’s correctly mapped to your IP.

This is the first step to the path of self-hosting and once you have the services running on your server, then you can expose them later on by port forwarding which I will be covering in a different post.