K3s, Raspberry Pi, Ghost and IPv6

K3s, Raspberry Pi, Ghost and IPv6

Since a long time I wanted to create my own blog to brain dump all my stuff I am working on. After some rearrangements I found two Raspberry PIs which were not in use. Therefore I decided to create a mini Kubernetes cluster on them and create this blog. Due to the fact IPv4 is not available on non business internet contracts I need to choose IPv6.

Hardware Setup

To run this Blog I am using two Raspberry PI 4 devices. One with 2GB RAM and the other device with 4GB of RAM.

Software Setup

The used Software for this blog is the Operating System Ubuntu 22.04 LTS which has been installed on SD Cards with Raspberry Pi Imager

Kubernetes will be installed with K3S.

The software Ghost will be installed via a Helm chart provided by bitnami.

Installing the Operating System

The installation can be customised before flashing the Image to the disk. Additional attributes like hostname, username, wifi password or ssh access key can be added

Installing Kubernetes pre-requisites

To get started and install K3s we need to install docker to the hosts. This can be done with the following commands

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

# Install latest docker version
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Install specific version
apt-cache madison docker-ce | awk '{ print $3 }'

5:24.0.0-1~ubuntu.22.04~jammy
5:23.0.6-1~ubuntu.22.04~jammy
...


VERSION_STRING=5:24.0.0-1~ubuntu.22.04~jammy
sudo apt-get install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin

Installing K3s with IPv6 support

K3s can be simply installed via a single line command. But in our case since we are behind a CGNAT (no native IPv4) we want to expose deployed services via IPv6

# Define custom K3S parameter to enable IPv6 support and priotize IPv6 traffic

INSTALL_K3S_EXEC=server --kubelet-arg=node-ip=:: --cluster-cidr=10.42.0.0/16,fd69::/48 --service-cidr=10.43.0.0/16,fd69::/112 --node-ip=<raspberry-primary-ipv4>,<raspberry-routable-ipv6>

# Invoke the K3S installation
curl -sfL https://get.k3s.io | sh -

# Get Agent installation token for the worker node
cat /var/lib/rancher/k3s/server/agent-token
K10877.....::server:....f12323

This step needs to be done only on the primary Kubernetes nodes which will be the only master node in the cluster. For all other nodes we need to change some values to register as a worker node.

# Define K3S installation parameter for the agent node
export INSTALL_K3S_EXEC="--kubelet-arg=node-ip=::"

# Invoke the K3S Agent installation
curl -sfL https://get.k3s.io | K3S_URL=https://[fd00::e65f:1ff:fe6c:a64b]:6443 K3S_TOKEN=K10877.....::server:....f12323 sh -


After this step a K3S Kubernetes cluster should be created and contain two nodes

# Execute on the master node
k3s kubectl get nodes
NAME             STATUS   ROLES                  AGE     VERSION
rpi-k8s-node00   Ready    control-plane,master   5h30m   v1.28.4+k3s2
rpi-k8s-node01   Ready    <none>                 5h20m   v1.28.4+k3s2

# Display running pods
k3s kubectl --all-namespaces -o wide
NAMESPACE      NAME                                                   READY   STATUS      RESTARTS   AGE     IP               NODE             NOMINATED NODE   READINESS GATES
kube-system    coredns-6799fbcd5-qsxz4                                1/1     Running     0          5h30m   fd69:0:0:1::4    rpi-k8s-node00   <none>           <none>
kube-system    local-path-provisioner-84db5d44d9-mmt2p                1/1     Running     0          5h30m   fd69:0:0:1::3    rpi-k8s-node00   <none>           <none>
kube-system    metrics-server-67c658944b-zzhm2                        1/1     Running     0          5h30m   fd69:0:0:1::5    rpi-k8s-node00   <none>           <none>
kube-system    helm-install-traefik-crd-v86xk                         0/1     Completed   0          5h30m   fd69:0:0:1::6    rpi-k8s-node00   <none>           <none>
kube-system    helm-install-traefik-l8jfj                             0/1     Completed   2          5h30m   fd69:0:0:1::2    rpi-k8s-node00   <none>           <none>
kube-system    svclb-traefik-f44b60c8-tjwdz                           2/2     Running     0          5h29m   fd69:0:0:1::7    rpi-k8s-node00   <none>           <none>
kube-system    traefik-f4564c4f4-r6hjz                                1/1     Running     0          5h29m   fd69:0:0:1::8    rpi-k8s-node00   <none>           <none>
kube-system    svclb-traefik-f44b60c8-5lgjm                           2/2     Running     0          5h21m   fd69:0:0:2::2    rpi-k8s-node01   <none>           <none>

Installing Cert Manager with DNS challenge

To obtain valid Let's Encrypt certificates we need to add cert-manager. The installation will be done with helm and their predefined chart

# Add cert-manager repository
helm repo add jetstack https://charts.jetstack.io

# Update the repository
helm repo update

# Install cert-manager
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set installCRDs=true --set 'extraArgs={--dns01-recursive-nameservers-only,--dns01-recursive-nameservers=8.8.8.8:53,1.1.1.1:53}'

Now we need to add all needed information to obtain certificates via DNS challenge from Cloudflare

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-token: BKW_XXXXXXXXXXXXXXXXXXXXX
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: le-global-issuer
spec:
  acme:
    email: me@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-key
    solvers:
    - dns01:
        cloudflare:
          email: me@example.com
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token
---

Kubernetes persistent storage: NFS

Since we do not want to save all data onto the SDCards on the raspberry we add an additional storage class. This will be done with a external NFS server and a provisioner.

# Create a file called nfs and paste the following manifest
nano nfs

# Install NFS provisioner via manifest with all needed information
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: nfs
  namespace: default
spec:
  chart: nfs-subdir-external-provisioner
  repo: https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner
  targetNamespace: default
  set:
    nfs.server: nfs-server-ip6
    nfs.path: /volume2/swarm-nfs/rpik3s
    storageClass.name: nfs
---

# Apply the helm chart
k3s kubectl apply nfs

Installing Ghost Blog

After all pre-requisites are done we can finally deploy our Ghost instance to the Raspberry Pi cluster. The software will be deployed via Bitnami Helm Chart. Some additional configurations needs to be done.

# create a new file: custom.yaml
global:
  storageClass: "nfs"

image:
  registry: docker.io
  repository: bitnami/ghost
  tag: 5.75.2
  pullPolicy: IfNotPresent

ghostUsername: ghost-admin
ghostPassword: "beep_boop_top_secret"
ghostEmail: me@example.com
ghostBlogTitle: Cawfee & Tech

ghostEnableHttps: false
#smtpHost: ""
#smtpPort: ""
#smtpUser: ""
#smtpPassword: ""
#smtpService: ""
#smtpProtocol: ""

ghostSkipInstall: false

livenessProbe:
  enabled: false
readinessProbe:
  enabled: false

replicaCount: 1
updateStrategy:
  type: Recreate

containerPorts:
  http: 2368
  https: 2368

service:
  type: LoadBalancer
  ports:
    http: 80
    https: 443

persistence:
  enabled: true
  storageClass: "nfs"
  accessModes:
    - ReadWriteOnce
  size: 8Gi

volumePermissions:
  enabled: true

ingress:
  enabled: true
  hostname: blog.cawfee.site
  path: /
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-hostname: "blog.cawfee.site"
    cert-manager.io/cluster-issuer: "le-cs-issuer"
  ingressClassName: "traefik"
  extraTls:
   - hosts:
       - blog.cawfee.site
     secretName: ghost.cawfee-site

mysql:
  enabled: true
  architecture: standalone
  auth:
    rootPassword: "some_secret_root_pw"
    database: bitnami_ghost
    username: bn_ghost
    password: "beep_boop_mysql_user_password"
  primary:
    persistence:
      enabled: true
      storageClass: "nfs"
      accessModes:
        - ReadWriteOnce
      size: 8Gi
---
      
# apply the helm chart with the custom values
helm upgrade --install --namespace blog --create-namespace blog -f custom.yaml oci://registry-1.docker.io/bitnamicharts/ghost

The end? nope!

Do not forget to open the ports to your Kubernetes nodes or the website will not be reachable from outside!