Microk8s registry with custom images

This is a post in the saga about running Kubernetes on a cluster of Ubuntu powered Raspberry Pi 4.

I see that I built that RPI4 cluster five years ago but it still works well and works both as a playground and fullfilling some actual needs. I have updated to Ubuntu 24.04 since then but otherwise it is pretty much the same.

Sooner or later one needs a custom image and I did write about that earlier. Yesterday I noticed that solution was not 100% correct. So for my own records and other microk8s users I sum up the steps here.

On the backplane host

Install the microk8s registry plugin: microk8s enable registry

Install the buildx plugin: docker-buildx-plugin

Create a file buildkitd.toml with this content (replace compute with your backplane host):

[registry."compute:32000"]
  http = true

Create a builder for multi-arch builds:
docker buildx create --use --name mybuilder --driver-opt network=host --buildkitd-flags '--allow-insecure-entitlement network.host' --buildkitd-flags '--allow-insecure-entitlement network.host' --config buildkitd.toml

Create your image description, for example:
FROM debian:stable
RUN <<EOF
apt-get update && apt-get install -y ffmpeg mediainfo wget ca-certificates python3 python3-pip procps elinks curl python3-bs4

Build and push:
docker buildx build -f podcast-stable-image -t compute:32000/mydebianstable:registry-20250811 --platform linux/amd64,linux/arm64 . --builder mybuilder --push

Now you can refer to this image from your deployment.yaml (where compute is replaced with your backplane host): image: compute:32000/mydebianstable:registry-20250811

On all nodes that will pull from the registry

Microk8s will assume https when trying to pull the image from the registry but the registry plugin only supports http. We tell containerd to use http by editing /var/snap/microk8s/current/args/containerd-template.toml and replacing the last section with:

[plugins."io.containerd.grpc.v1.cri".registry]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."compute:32000"]
    endpoint = ["http://compute:32000"]
    [plugins."io.containerd.grpc.v1.cri".registry.configs."compute:32000".tls]
      insecure_skip_verify = true

Then restart with: sudo snap stop microk8s && sudo snap start microk8s

That should be it, you can schedule a job on a specific node (“rpi4-6” in the example below) where you have done the changes to see that it works. For example by modifying the deployment with yq (sudo snap install yq) before applying:
kubectl create job --from=cronjob/cronjobnamehere makeupapodnamehere -o yaml | yq e '.spec.template.spec.nodeName = "rpi4-6"' - | kubectl apply -f -