For those who want to use Calico with k3s instead of Flannel, I am sharing here the steps I followed. For the project I am working on, I had to re-install k3s with Calico upon learning that flannel works on layer-2, whereas I need layer-3 routing for BGP. I am not sure if there’s a way to advertise BGP with flannel as the CNI but it looks like Calico already runs it natively. That should reduce any additional configuration that might be required going further if your project also requires BGP routing.
Uninstall k3s To uninstall execute the following as root:
/usr/local/bin/k3s-uninstall.sh
rm -rf /var/lib/rancher
Reinstall k3s
Execute:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy --cluster-cidr=192.168.0.0/16" sh -s - --docker
Copy the k3s yaml file to your home directory to allow remote access. Ensure correct ownership:
mkdir /home/luis/.kube/ #(Only required if you don't have the .kube directory in your home folder yet)
cp /etc/rancher/k3s/k3s.yaml /home/luis/.kube/config
chown -R luis:luis /home/luis/.kube/
Check nodes:
kubectl get node
Get the token of your master node:
cat /var/lib/rancher/k3s/server/node-token
Use this token and the IP of your master node in the installation command to be executed in your worker nodes. Execute as root:
curl -sfL https://get.k3s.io | K3S_URL=https://serverip:6443 K3S_TOKEN=mytoken sh -s - --docker
Execute this in the worker node as root.
You will get the error:
The connection to the server localhost:8080 was refused – did you specify the right host or port?
This is because flannel was disabled and there is no CNI running.
And if you try to check pods from the master node the status will only be in ContainerCreating:
Install Calico
From here you will need to install calico. To do so execute:
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/tigera-operator.yaml
Then install the required custom resources:
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/custom-resources.yaml
Now check the pods
kubectl get pod -o wide --all-namespaces
You will notice that some traefik containers are in error or CrashLoopBackOff state. I am not sure why but I don’t really need it. To disable Traefik:
kubectl -n kube-system delete helmcharts.helm.cattle.io traefik traefik-crd
Stop k3s services.
sudo systemctl stop k3s
sudo systemctl status k3s
Modify below the k3s service configuration file:
sudo vi /etc/systemd/system/k3s.service
And add the following line:
'--disable=traefik' \
Reload the service and delete the Traefik yaml file:
sudo systemctl daemon-reload
sudo rm /var/lib/rancher/k3s/server/manifests/traefik.yaml
Start k3s:
sudo systemctl start k3s
sudo systemctl status k3s
Check nodes
kubectl get node -o wide
Check the pods regularly.
You might notice that some calico pods are in crashloopback state. Check again after a few minutes and they should be in running state
K3s with Calico should be running fine now at this point!