This might come in handy to those who are thinking of setting up their own VPS to connect to their K8s cluster, whether it be hosted at home or in the cloud, or maybe to those who just simply want to connect two mutually remote devices running on two different networks with a firewall.

To start off, first, keep in mind that my NAS in the backend is exported via NFS by an LXC container hosted on Proxmox, and the remote service (Deluge in this case) that would be mounting the share is running as a docker container.

Tailscale pre-requisites for LXC containers on PVE

On the NFS server end, the container needs access to /dev/net/tun on the host. To allow this we need two lines in the container configuration (reference: tailscale kb article).

lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
lxc.cgroup2.devices.allow: c 10:200 rwm

Stop the container and hop on to your PVE host’s terminal. Modify /etc/pve/lxc/1201.conf and the above two lines.

Tailscale installation on the LXC container and remote server

Next up is installation of tailscale. The installation is fairly easy. An installation command can be generated together with a pre-approved authentication. Go to your Tailscale account > Machines > Add device > Generate install script. Ensure to create a reusable key. There is no need to add tags if the device is intended for a single user. Execute on both the LXC container and remote server

e.g.

curl -fsSL https://tailscale.com/install.sh | sh && sudo tailscale up --auth-key=tskey-auth-kHD2rsVNiv11CNTRL-188WY38WSG2iqTHTZs1oF2e3LMSmMYuXF

You should now be able to see your device under the Machines tab. The tailscale IP will also be visible from there. If you want to check the status of tailscale: sudo systemctl status tailscaled

To avoid manual intervention in the future, you can disable the key expiry for each device in the Machines tab.

Configure NFS

Allow the tailscale IP in your /etc/exports file:

/mnt/nfs-share 100.22.112.36/32(rw,sync,no_subtree_check,all_squash,anonuid=1000,anongid=1000,insecure)

On the remote server, ensure the containers are stopped and modify your compose file to mount the nfs share to the /downloads directory.

version: "2.1"
services:
  deluge:
    image: lscr.io/linuxserver/deluge:latest
    container_name: deluge
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Asia/Manila
      - DELUGE_LOGLEVEL=error
    volumes:
      - .:/config
      - nfsdir:/downloads
    ports:
      - 8112:8112
      - 6881:6881
      - 6881:6881/udp
    restart: unless-stopped
volumes:
  nfsdir:
    driver: local
    driver_opts:
      type: nfs
      o: addr=100.22.112.36,rw,vers=4.1
      device: ":/mnt/nfs-share"

Start the container and try to create a file in the NFS share.

Install tailscale-operator on k8s

It will be easier for you to refer to the official documentation so I am sharing the actual KB article here.

In summary, two items are needed to be configured from the Tailscale Management page — tags and OAuth client credentials. Complete these steps and carry on with the helm installation. Helm installation steps are also shared below.

Installation with helm

Replace the clientId and clientSecret variables:

helm repo add tailscale https://pkgs.tailscale.com/helmcharts
helm repo update
helm upgrade \
  --install \
  tailscale-operator \
  tailscale/tailscale-operator \
  --namespace=tailscale \
  --create-namespace \
  --set-string oauth.clientId="<OAauth client ID>" \
  --set-string oauth.clientSecret="<OAuth client secret>" \
  --wait

Create a ProxyGroup of type egress.

apiVersion: tailscale.com/v1alpha1
kind: ProxyGroup
metadata:
  name: ts-proxies
spec:
  type: egress
  replicas: 3

Installation with ArgoCD

If you are tidy, maintain your infrastructure as code, and use ArgoCD, then go ahead and download the values.yaml file and update the clientId and clientSecret fields.

helm show values tailscale/tailscale-operator > values.yaml

...
oauth:
  clientId: "kq7LLBiMeL11CNTRL"
  clientSecret: "tskey-client-kq7LLBiMeL11CNTRL-rPFdRtSCj74REgir443J84CYp12Lo8Vce"
...

Create the manifest for your ArgoCD application (replace yourgituser and yourrepo).

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: tailscale
  namespace: argocd
spec:
  destination:
    namespace: tailscale
    server: https://kubernetes.default.svc
  project: default
  sources:
    - repoURL: 'https://pkgs.tailscale.com/helmcharts'
      chart: tailscale-operator
      targetRevision: 1.78.3 # This is the latest version as of the time of this post.
      helm:
        releaseName: tailscale-operator
        valueFiles:
          - $values/tailscale/values.yaml
    - repoURL: git@github.com:yourgithubuser/yourrepo.git
      targetRevision: HEAD
      ref: values
    - repoURL: git@github.com:yourgithubuser/yourrepo.git
      path: tailscale/manifests
      targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
    syncOptions:
      - CreateNamespace=true

Don’t forget to create the egress ProxyGroup manifest!

Testing connectivity from a pod to the remote server

Login to any pod and perform a curl to the other end. If you don’t have a test pod, then you can deploy an alpine image.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: alpine
  name: alpine
  namespace: alpine
spec:
  replicas: 1
  selector:
    matchLabels:
      run: alpine
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        run: alpine
    spec:
      containers:
      - image: alpine:latest
        command:
          - /bin/sh
          - "-c"
          - "apk --update add curl && sleep 1440m"
        imagePullPolicy: IfNotPresent
        name: alpine

kubectl -n alpine exec -it alpine-9c7cb4b95-nx7b4 -- curl http://10.43.59.234:8112

Pod to tailscale communication

Now that I have a secondary network attachd to my cluster, I can utilize my ingress controller, that is Traefik, to provide HTTPS access to devices on that network.

To do so traefik must be allowed to reach ExternalNames. Set providers.allowExternalNameServices to true in the values.yaml file:

...
providers:
  kubernetesIngress:
    enabled: true
    allowExternalNameServices: true
...

Create the ingress resource as you would normally do, and also create a service of type ExternalName. You can either provide the IP or the tailscale FQDN.

In case of the latter:

apiVersion: v1
kind: Service
metadata:
  annotations:
    tailscale.com/tailnet-fqdn: "voyager.taile9a23.ts.net"
    tailscale.com/proxy-group: "ts-proxies"
  name: ts-egress-voyager
spec:
  externalName: voyager-deluge # any value - will be overwritten by the operator
  type: ExternalName
  ports:
  - port: 8112
    protocol: TCP
    name: web

Note that you have to specify the ProxyGroup that was created in the previous step. Also an additional clusterIP service will be created on top of this:

NAME                         TYPE           CLUSTER-IP      EXTERNAL-IP                                      PORT(S)    AGE
ts-egress-voyager            ExternalName   <none>          ts-ts-egress-voyager-mgmz2.tailscale.svc.local   8112/TCP   2s
ts-ts-egress-voyager-mgmz2   ClusterIP      10.43.231.143   <none>                                           8112/TCP   2s

This is the final step and you should now be able to access Deluge on the remote server via HTTPS.

Tailscale Lovin'

What I love about tailscale is how easy it is to setup and the how the folks over there continue to offer a free plan for personal use. I still remember how difficult it was to setup a VPN with friends 15-20 years ago just to be able to play games like Counter-Strike or Starcraft (maybe I was not just as experienced yet? I don’t know!) But with tailscale today, it’s totally a game changer. I also want to mention here that tailscale offer Mullvad VPN access for $5/month per 5 devices. Although I am not sure if this is supported by the tailscale-operator, at the least all your other devices can utilize this (you can even install tailescale on the AppleTV!). I probably will consider subscribing to this in the not so distant future if I don’t find any other use case for remote Deluge server.