The network just reached the fourth month mark and it’s already about to undergo a somewhat major re-design. But just before going to the next chapter of the journey, I want to take this time to log the current state of the existing infrastructure and services. Don’t get me wrong — I don’t run a full-fledged data center, nor do I even have a server rack setup. I only have photos and videos as critical files and I only have two users at home including myself (lol). It’s called a homelab for a reason. Though it would still be fun to document the progress be it big or small and know how far I’ve come along the way.

Homelab Networking and Hardware

On high level, below is the current diagram of existing services, subnets, VLANs, and devices.

Orange boxes signify the servers running a hypervisor — one Proxmox, and the other, Unraid. The Proxmox box is mainly running my network stack which includes OPNsense, a LXC container running Pihole as my DNS, and another one running tailscale for a machine-to-machine VPN with the remote peer acting as an exit node. While it might be possible to run tailscale from within OPNsense, I decided to configure it separately since I only have a simple use case of tunneling torrent traffic to a small thin client back home in Manila.

On the other hand, the box running Unraid primarily runs our NAS and a couple of docker containers and VMs. It’s been very stable, running without any hiccup except for the time when all containers and VMs together with all the other devices were on a single subnet. This is what pushed me to segregate the traffic with VLANs. Speaking of, this is easily made possible by a cheap “smart” (you can say it’s managed) switch from TP-Link. I can’t stress enough how much I love this brand! I’ve been using their routers for more than a decade now and it has never failed me.

Back to the Unraid server, it had also undergone some few upgrades (the initial specs can be found in one of my first few posts here). First is the addition of a 500GB SSD as an un-mirrored cache drive to avoid wear and tear of the NVMe drives. This is also mainly used for non-critical data such as when storing movies, tv shows, and ISO files, etc.

Second is the installation of a third party heat sink for the NVMe drives which brought down temperature levels by about 5-10C.

Third, the installation of a 330W UPS from APC, allowing graceful shutdown in case of power outages and protection from potential power surges.

Fourth, the installation of a Noctua NH-L12S cooler to pull down the temperature levels of the CPU.

Lastly, a RAM upgrade to 64GB after experiencing memory shortage when working with GNS3. 32GB wasn’t enough when running multiple containers and spawning multiple VMs on top of GNS3.

Services

Talking about the services, I am running just a handful. For the monitoring stack I have Telegraf, InfluxDB and Grafana to monitor resource utilization on the Proxmox box. I also have UptimeKuma providing a neat and simple dashboard for monitoring service uptime.

For resource monitoring of my Unraid box, I just use the built-in dashboard which already gives me all the information I need such as CPU and disk temps, CPU, disk, and memory utilization, and power consumption.

For productivity and utilities, I have Guacamole for VM and command line remote access, Krusader for file management within Unraid, Visual Studio Code for modifying configuration files and blogging, Firefox for VM-less browser access, Wiki.js for saving important information and notes taking of anything related to the Homelab and the projects I am working on, Jdownloader for HTTP downloads (mainly ISO files), Heimdall as my homelab dashboard, and Traefik as my reverse proxy!

For media services, I have Jellyfin for transcoding and serving media both at home and when on-the-go. To populate my fiance and I’s media inventory with videos from our favorite youtube channels, I use ytdl-sub. This automatically downloads the videos of any defined youtube channel, manages the specified retention period, and adds the necessary metadata for a nice and smooth feel when browsing in Jellyfin. And since the videos are played back locally, this means we also get ad-free content when watching. For legal downloading of media, I have deluge. Since I don’t use this often, I still transfer files manually to the Jellyfin directory.

I am aware I am still lacking in the automated media management department by making use of the *ARR services, but I can’t find time to set them up especially now that this homelab environment keeps me branching out to different projects all at the same time. At the moment I just have bigger fish to fry.

VMs

As for the VMs, I only have quite a few. My main workstation is running PopOS based on Ubuntu. I do use a Macbook at home but still login with RDP to this machine almost all the time since it allows me anytime-anywhere remote access and I can easily pick up where I left off when I am working on something.

Though 99% in shutdown state, I also keep a Windows 10 VM in case I need to use software which can’t run on Linux. I also keep a staging environment running on Debian for the remote tailscale exit node installed back home in Manila. Another one is an Ubuntu VM running GNS3 server for other side projects on kubernetes and networking. My VM disks are created in the NVMe drives allowing me to play around with VMs efficiently in terms of very quick load times. The NVMe drives are also mirrored in RAID1 so I don’t have to worry about losing data when one of the two drives crashes.

The last but not the least is my router! Running as a VM on Proxmox, it just works wonderfully. I did experience random reboots though, but this got fixed after upgrading the kernel from 5.4 to the new 6.1 together with the intel micro-code package. My longest uptime with it was around 40 days before I had to reboot for maintenance work. Inside OPNsense I am running Unbound caching DNS resolutions for faster downstream queries. The browsing experience became significanly better after enabling this. Websites started to load faster also after enabling DNS over HTTPS. I was trying to find an explanation to this but I will just leave it out in the cold for now. I also have my Cloudflare Dynamic DNS agent running here. For VPN, I have Wireguard allowing me unfiltered access to the entire network. I also use this on daily basis — tunneling at the very least my DNS traffic to Pihole for internet ads blocking.

I think that’s all of it for the network’s status quo. Over the next few weeks I will have to wrap my head around Terraform, Ansible, and Kubernetes. This is because I’ll be migrating most, if not all, of these services to a “production” k8s cluster with the help of automation. I’ve been sitting on the fence about this for quite some time now, but I finally decided go with it to have hands-on experience with Infrastructure-as-Code. Is it necessary? Not really, but it sure is interesting.

Uninstall k3s To uninstall execute the following as root:

/usr/local/bin/k3s-uninstall.sh
rm -rf /var/lib/rancher

Reinstall k3s

Execute:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy --cluster-cidr=192.168.0.0/16" sh -s - --docker

Copy the k3s yaml file to your home directory to allow remote access. Ensure correct ownership:

mkdir /home/luis/.kube/ #(Only required if you don't have the .kube directory in your home folder yet)
cp  /etc/rancher/k3s/k3s.yaml /home/luis/.kube/config
chown -R luis:luis /home/luis/.kube/

Check nodes:

kubectl get node

Get the token of your master node:

cat /var/lib/rancher/k3s/server/node-token

Use this token and the IP of your master node in the installation command to be executed in your worker nodes. Execute as root:

curl -sfL https://get.k3s.io | K3S_URL=https://serverip:6443 K3S_TOKEN=mytoken sh -s - --docker

Execute this in the worker node as root.

You will get the error:

The connection to the server localhost:8080 was refused – did you specify the right host or port?

This is because flannel was disabled and there is no CNI running.

And if you try to check pods from the master node the status will only be in ContainerCreating:

Install Calico

From here you will need to install calico. To do so execute:

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/tigera-operator.yaml

Then install the required custom resources:

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/custom-resources.yaml

Now check the pods

kubectl get pod -o wide --all-namespaces

You will notice that some traefik containers are in error or CrashLoopBackOff state. I am not sure why but I don’t really need it. To disable Traefik:

kubectl -n kube-system delete helmcharts.helm.cattle.io traefik traefik-crd

Stop k3s services.

sudo systemctl stop k3s
sudo systemctl status k3s

Modify below the k3s service configuration file:

sudo vi /etc/systemd/system/k3s.service

And add the following line:

'--disable=traefik' \

Reload the service and delete the Traefik yaml file:

sudo systemctl daemon-reload
sudo rm /var/lib/rancher/k3s/server/manifests/traefik.yaml

Start k3s:

sudo systemctl start k3s
sudo systemctl status k3s

Check nodes

kubectl get node -o wide

Check the pods regularly.

You might notice that some calico pods are in crashloopback state. Check again after a few minutes and they should be in running state

K3s with Calico should be running fine now at this point!

Ubuntu Server Installation

You can download the Ubuntu iso image from this page. It’s either you download to your local machine or using Unraid with the help of the CLI or some docker app.

While waiting for the download to complete. Go to Settings > VM Manager and set ‘Enable VMs’ to yes. You can also toggle the advanced view to have a more detailed look on the settings. Here you will see the directory of the iso files and the storage so be sure to have your shares properly configured.

Just to give you an idea my iso directory is saved to a cache that doesn’t have redundancy. Image files that are not intended to be used any sooner can either be deleted or transferred to an archive directory on your main array.

It is important to transfer the iso file to the iso share or else you won’t be able to select this later on.

Go to VMs > Add VM > Ubuntu. Select the CPU cores and memory as well as the ISO file.

Assign the disk size.

For network model ensure that br0 is selected. For VM console port Unraid starts with the default port of VNC but just to avoid any conflict later on when you run appliances on GNS3 just choose a different port range from 5900. Uncheck ‘Start VM after creation’. Everything else can be kept in default. Create the VM.

Note network-model should be set to virtio-net to avoid any networking issues with the docker containers.

Go back and edit this VM and toggle from form to XML view on the upper right hand side. To enable nested virtualization insert the following line in the part.

<feature policy='require' name='vmx'/>

Update, start the VM, and open the console. Install the OS how you normally do it.

As for me I always choose Ubuntu Server (not minimized) everytime to avoid any package dependencies later on.

Be sure the VM is assigned a DHCP IP when you get to the network settings or assign a static one if you wish. I always prefer the latter.

Modify disk to utilize all. There really is no use case to use LVM if you are just running GNS3 server.

Also ensure OpenSSH to be installed!

Reboot the server after installation and check your IP within the VNC console. Try to login via ssh.

Perform basic checks especially the disk space! The root directory should be assigned the biggest chunk of the disk size.

Install GNS3 Server

I am sharing the link to the official documentation here for reference. It says to execute the following commands as root:

sudo -i
cd /tmp
curl https://raw.githubusercontent.com/GNS3/gns3-server/master/scripts/remote-install.sh > gns3-remote-install.sh
bash gns3-remote-install.sh --with-openvpn --with-iou --with-i386-repository

Now if we check the status of the gns3 services. It would always show it’s activating

If you check the log file mentioned there you would see the following

CRITICAL web_server.py:88 Could not start the server: [Errno 99] error while attempting to bind on address (‘172.16.253.1’, 3080): cannot assign requested address

Let’s check the configuration file getting loaded and see if we can see anything there.

The controller config file can be found in the highlighted path shown above. At the same time we can see GNS3 is trying to listen on 172.16.253.1.

This seems to be the default IP configured in the installation. Change this to the local IP of your VM.

Restart GNS3 and check the status again. It should be running now.

At this point you should be able to connect from a remote GNS3 GUI to this server.

Lastly to start GNS3 server automatically every reboot, just enable the service:

systemctl enable gns3

In bullet points:

• All disks and shares can be found in /mnt/.
• /mnt/disk1/ is location of disk1.

• /mnt/user0 contains all of the shares but only those of which are in the array and not the cache.
• Shares are spread across the disks.
• All shares on unRAID can be found in /mnt/user. Try to click on any of the share and it will show you the path.
• If you mount an external drive from the Unassigned Devices, the path will be in /mnt/disks/
• If you try to mount a remote share (e.g. a SMB share from another unraid system), the path will be in /mnt/remotes/.
• There are different tools which can be used for data transfer e.g. Krusader, rsync, remote shares, or by using the cli.

How to use Krusader

Install binhex krusader.

Delete /media

We will create three different folders. One to access local shares, one for unassigned disks, and another one for remote shares.

On the bottom part click ‘Add another path, port, variable, label or device.’

Follow as below:

Add another one as below:

And another one as below:

Click Apply and wait for installation to complete.

Go to Docker > Krusader > WebUI

Go through the wizard by clicking on OK.

On both the left and right side panel go to the / directory. And from here add a profile called ‘startup’.

Go to Settings > Configure Krusader > Startup profile > choose startup. Apply.

Now you can navigate to your directories and copy or move files across the panels. One good thing to note is even if you close the browser, the copying or moving of files will continue as long as the docker application is running. For attaching external drives via USB, you will have to mount them on the ‘Main’ tab before it appears in /mnt/disks/ directory.

• Motherboard: Asrock H570M-ITX/AC
• CPU: Intel Core i7-11700
• RAM: Kingston Fury DDR4 3200MT/s 16GB x 2
• Drives: 2x 8 TB WD Red Plus, 2x 1TB Samsung 980 NVMe, 1x 500GB Samsung 860 EVO
• PSU: Fractal Design ION SFX-L 500W 80PLUS Gold
• Case: Jonsbo N1

Below you can find the steps I followed to install unRAID v6.11.5 with the help of SpaceInvaderOne’s video. Of course only follow at your own risk. I would even advise you to go to SpaceInvaderOne’s youtube channel and watch his videos instead.

Preparation

1. Create a bootable USB with the Unraid USB creator and allow UEFI boot since there is no external GPU.
2. Change boot order to have USB as first priority.
3. Enable VD-T and all other virtualization settings in BIOS.
4. Disable built-in RAID function in BIOS.
5. Plug server to router and access using kamata.local (name-of-server.local)
6. Change to a static IP address by going under Settings > Network Settings.

7. Go to plugins tab and install community plugins manager. You can get this from Unraid forum > Plugin support > [Plug-In] Community Applications. Acquire the URL under the INSTALLATION section.

8. Go back to Installed Plugins and click on the icon of Community Applications

9. Go to Plugins > Preclear Disk icon. Preclear only the mechanical disks and not NVMe or SSDs. Below the first one has already been started. You can leave everything else in default or you can just skip pre-read to save on time. My 8TB drives took about 13 hours to complete the zeroing process and another 13 hours to complete post-read.

10. Once preclear is done, you can create the array.

Creating the array

It’s really surprising how easy it is to create the array in just a few clicks.

1. In the Main tab, assign one of the mechanical disks as parity but do note the parity drive should always be equal or bigger than any of the other drives in your array. Assign another drive as disk 1.

2. Start the array and confirm the warning message prompt.

3. You will see that the disks are unmountable. This is because they haven’t been formatted yet. Go to the bottom part and formal all the disks. Ensure those are only disks part of the new array.

4. Once formatting is complete, the parity-sync will start automatically. This will take time depending on the drive capacity. As for my case it took around another 13 hours. Do note that you don’t have to wait for parity to complete before you can start using the array. You can already upload files at this point but you will not have redundancy not unless parity is complete.

5. When parity is completed you will see a green dot on the left side of device name. You can also scroll down and you should see a text showing parity is valid. At this point anything you upload into the disk will have redundancy. Just remember that parity needs to be calculated every time you modify something in the disk.

Creating a cache pool

Now that you have the array running with parity-sync complete, you can start configuration of the cache pool. Here you will see that I still have my NVMe drives under unassigned devices. I plan have them running on RAID1 as well – same as the array.

1. Stop the array so you can add a cache pool. I named it cache_protected to remind me this cache has redundancy (any name will do). Click on Add Pool and assign 2 slots. Select the one drive for each of the slot.

2. You can click on the name of the cache pool to see the settings. I kept mine on default. Start the array. It will show the the disks again are unmountable so you just have to format them the same way it was done for array disks.

3. Cache pool creation will not take so long. When that’s done the second device under Pool Devices will have a text saying that it is part of a pool.

This is the final step to running your array with a cache pool. If you want to move files into any of the shares you can use a docker application called Krusader. I just tried this today but haven’t had more time to explore so I am parking this for now.