I got everything working eventually after multiple iterations of trial and error, and upgrading my traefik helm release countless times. And after finding this Traefik blog post where they explain how it all actually works and can be configured. At this point I’ve migrated all of my docker applications into my k8s cluster, and reading through the Traefik documentation again, it just makes a lot more sense now.

I guess the most important bit of the documentation is the part where they mentioned the requirement of using Cert-manager when using Traefik Proxy in Kubernetes. Quoting from the documentation:

LetsEncrypt Support with the Ingress Provider By design, Traefik is a stateless application, meaning that it only derives its configuration from the environment it runs in, without additional configuration. For this reason, users can run multiple instances of Traefik at the same time to achieve HA, as is a common pattern in the kubernetes ecosystem. When using a single instance of Traefik Proxy with Let’s Encrypt, you should encounter no issues. However, this could be a single point of failure. Unfortunately, it is not possible to run multiple instances of Traefik 2.0 with Let’s Encrypt enabled, because there is no way to ensure that the correct instance of Traefik receives the challenge request, and subsequent responses. Previous versions of Traefik used a KV store to attempt to achieve this, but due to sub-optimal performance that feature was dropped in 2.0. If you need Let’s Encrypt with high availability in a Kubernetes environment, we recommend using Traefik Enterprise which includes distributed Let’s Encrypt as a supported feature. If you want to keep using Traefik Proxy, LetsEncrypt HA can be achieved by using a Certificate Controller such as Cert-Manager. When using Cert-Manager to manage certificates, it creates secrets in your namespaces that can be referenced as TLS secrets in your ingress objects.

Since we are working in K8s, almost everything you can think of can be abstracted. For this instance, the TLS configuration is abstracted from the helm custom values yaml file and instead can be pegged to an Ingress resource when using Cert-manager.

To summarize in a few lines:

• Install Traefik so it can act your Ingress controller.
• Install Cert-manager so it can monitor your Ingress resources, and based on the TLS configuration and with some annotations, it will automatically initiate the DNS challenge, create the TLS certificates sand save them as a secret in the application namespace.
• Create Ingress resources to access your backend servcies while at the same time terminate HTTPS traffic (with Traefik).

Do note that while Traefik is an ingress controller and comes with its own IngressRoute CRD, it cannot be used together with Cert-manager at the moment. In this case we have to use the default Kubernetes Ingress resource.

In this post I will show how I configured Traefik and Cert-manager (patterned to the Traefik blog post!) to be used with Cloudflare. I will also show how you can have another layer of security by having multiple entry points and explicitly defining one during the ingress configuration. Though before proceeding, do note that I assume you already have an idea how to configure dynamic DNS and port-forwarding on Cloudflare and on your router.

Traefik

Recently I’ve started using ArgoCD for managing all my applications including those that are installed via helm charts. For now we will install Traefik using helm.

helm repo add traefik https://traefik.github.io/charts # Add the Traefik repository
helm repo update # Update the repositories
helm show values traefik/traefik > values.yaml # Download the custom values yaml

I would recommend you to browse through the custom values yaml file to get familiarized with the available configuration and other features of Traefik. If you want to get started quickly then you can follow the below yaml configuration:

values.yaml

deployment:
  enabled: true
  kind: Deployment
  replicas: 1
  terminationGracePeriodSeconds: 60
  minReadySeconds: 0

ingressRoute:
  dashboard:
    enabled: true
    matchRule: Host(`traefik.yownowndomain.com`) && PathPrefix(`/dashboard`) || Host(`traefik.yownowndomain.com`) && PathPrefix(`/api`)
    entryPoints: ["traefik","websecure"]

additionalArguments:
  - "--serversTransport.insecureSkipVerify=true"
  - "--providers.kubernetesingress.ingressendpoint.publishedservice=traefik/traefik"

ports:
  traefik:
    port: 9000
    expose: true
    exposedPort: 9000
    protocol: TCP
  web:
    port: 80
    expose: true
    exposedPort: 80
    protocol: TCP
    redirectTo: websecure
  webext:
    port: 8080
    expose: true
    exposedPort: 8080
    protocol: TCP
    redirectTo: websecureexternal
  websecure:
    port: 443
    expose: true
    exposedPort: 443
    protocol: TCP
    http3:
      enabled: false
    tls:
      enabled: true
      options: ""
      certResolver: ""
    middlewares:
      - default-default@kubernetescrd
  websecureext:
    port: 8443
    expose: true
    exposedPort: 8443
    protocol: TCP
    http3:
      enabled: false
    tls:
      enabled: true
      options: ""
      certResolver: ""
    middlewares:
      - default-default@kubernetescrd
  metrics:
    port: 9100
    expose: false
    exposedPort: 9100
    protocol: TCP
  dnsovertls:
    port: 8853
    expose: true
    exposedPort: 853
    protocol: TCP
    tls:
      enabled: false
    middlewares:
      - default-default@kubernetescrd
tlsOptions:
  default:
    minVersion: VersionTLS12
    cipherSuites:
      - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
      - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
      - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

service:
  enabled: true
  single: true
  type: LoadBalancer
  annotations:
    io.cilium/lb-ipam-ips: 10.80.0.2
  labels:
    exposedExternal: "yes"

In my custom yaml file notice that I have four different entryPoints defined. I configured it like this to have separate incoming streams between my internal-only applications against the internet-exposed ones. Having a common entryPoint for both internal and externally exposed applications is a security flaw since hackers might still be able to access your internal-only apps if they guessed the sub-domain and locally configured CNAME forwarding (they don’t need access to your DNS configuration).

Once your custom values yaml file is ready. Hit up the helm install command and proceed to the next step to create a Middleware resource.

default-middleware.yaml

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: traefik
spec:
  headers:
    customResponseHeaders:
      X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex"
      X-Forwarded-Proto: "https"
      server: ""
    customRequestHeaders:
      X-Forwarded-Proto: "https"
    sslProxyHeaders:
      X-Forwarded-Proto: "https"
    referrerPolicy: "same-origin"
    hostsProxyHeaders:
      - "X-Forwarded-Host"
    contentTypeNosniff: true
    browserXssFilter: true
    forceSTSHeader: true
    stsIncludeSubdomains: true
    stsSeconds: 63072000
    stsPreload: true

helm install traefik/traefik -f values.yaml

kubectl apply -f default-middleware.yaml

Now that you have Traefik up and running, expose the webext and websecureext entryPoints on your router. Simply port-forward 80 to 8080, and 443 to 8443.

Cloudflare

Going to Cloudflare, you have to configure an access token to be used later on when configuring Cert-manager. Login to your Cloudflare account. Navigate to My Profile > API Tokens > Create Token.

Under Permissions, set Zone - DNS - Edit

Under Zone Resources, set Include - Specific zone - “yourowndomain.com”

E.g.

Create and save your token somewhere safe.

Cert-manager

Cert-manager can be installed using helm. Quoting from the official documentation, Cert-Manager together with the CRDs can be installed with the following commands:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.13.2 \
  --set installCRDs=true

After installation, it’s time to configure cert-manager for use with Cloudflare. You simply just have to create a secret containing your cloudflare email and API token:

cloudflare-api-token-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token
  namespace: cert-manager
type: Opaque
stringData:
  email: example@domain.com
  apiToken: AbcD321eFg456_

Next is to create a ClusterIssuer resource. This is one of the CRDs that will come with cert-manager. Here you have to specify your email to be used with Let’s Encrypt as well as the secret you created in the previous step.

clusterissuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cluster-issuer
  namespace: cert-manager
spec:
  acme:
    email: example@domain.com
    # Prod
    server: https://acme-v02.api.letsencrypt.org/directory
    # We use the staging server here for testing to avoid hitting
#    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # if not existing, it will register a new account and stores it
      name: letsencrypt-acc-key
    solvers:
      - dns01:
      # The ingressClass used to create the necessary ingress routes
          cloudflare:
            email: example@domain.com
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: apiToken
        selector:
          dnsZones:
            - 'yourowndomain.com'

If you are doing this the first time, I strongly suggest you first use the staging server of Let’s Encrypt and ensure certificate issuance works fine. To switch to the porduction server all you have to do is just re-create the resource and point to the new URL. Do note in the example above, the resource is pointing to the production server.

Exposing your application

To test this we create a simple nginx deployment that is exposed within the cluster and an ingress resource.

nginx.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
    - name: "http"
      port: 80
      targetPort: 80
  type: ClusterIP

When creating the ingress manifest, you have to ensure that you have the annotation: cert-manager.io/cluster-issuer: "letsencrypt-cluster-issuer".

Apart from this if you want your application to be accessed from a specific traefik entryPoint, then you need to explicitly mention this as another annotation. By default the ingress will be allowed from any entryPoint if you don’t specify this. In the below example manifest, I am keeping this annotation commented out so I can access my Nginx app from both internal and external entryPoints.

nginx-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-cluster-issuer"
    # traefik.ingress.kubernetes.io/router.entrypoints: websecureext
spec:
  tls:
    - hosts:
        - nginx.yownowndomain.com
      secretName: tls-nginx-ingress
  rules:
    - host: nginx.yownowndomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nginx
                port:
                  number: 80

Once you apply these manifests, simply do a kubectl get certificate. This will show that a new certificate and secret resource got automatically created.

kubectl get certificate
NAME                READY   SECRET              AGE
tls-nginx-ingress   False   tls-nginx-ingress   61s

Wait for a few minutes. You should see that cerficicate ready status should be set to True.

You can also describe this certificate to see the events.

Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    91s   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  90s   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "tls-nginx-ingress-2mvlp"
  Normal  Requested  90s   cert-manager-certificates-request-manager  Created new CertificateRequest resource "tls-nginx-ingress-1"
  Normal  Issuing    11s   cert-manager-certificates-issuing          The certificate has been successfully issued

To expose this on the internet, you will have to go back to Cloudflare and configure a CNAME forwarding to the sub-domain that points to your router’s external IP, given that you have already configured dynamic DNS.

You can also test this by configuring a CNAME entry on your local network’s DNS.

Once you ge this working you should be able to see a valid certificate when you access your nginx page.

The certificate will have a validity of 3 months but you don’t have to worry since Cert-manager will renew this automatically.