# Minikube on M1 pro without Docker desktop

## Down the M1 rabbit hole.

This year I started my own company: 0to1. To get up and running quickly I decided to stick with technology that I knew before, hardware I have used for the past 8 years without breaking down has been Apple MacBook Pro’s. The new Macbook M1 Pro had raving reviews, super performance, and long battery life.

Knowing that the processor architecture changed from Intel (x86) to ARM (M1), after a year things should certainly have been ironed out… right?

After checking https://isapplesiliconready.com/for/m1 I found out that some tools didn’t work, like VirtualBox but there are often others that do work without issues.

After installing Rancher desktop on the m1 I ran into an issue that /usr/local/bin was not created and ownership was not set to my user, this was easy to resolve by the solution below.

Rancher Desktop was working fine, but I needed an etcd node integrated with Kubernetes for a little programming project I have going on. Looking back I should have just used Rancher Desktop to run an etcd cluster locally, but I remembered Minikube had etcd installed automatically.

I wanted to see if Minikube on the M1 could be easily set up. So decided to try out Minikube.

## Minikube it is!

brew install minikube

The brew install was easy, but starting minikube gave an error that Virtualbox was not found, this is the default driver that is used by minikube. As I mentioned above, VirtualBox isn’t supported on the M1 and likely never will. So I needed another driver.

😄 minikube v1.24.0 on Darwin 12.1 (arm64)

✨ Automatically selected the virtualbox driver

❌ Exiting due to DRV_UNSUPPORTED_OS: The driver ‘virtualbox’ is not supported on darwin/arm64

A list of all the available supported drivers for Apple can be found here:

https://minikube.sigs.k8s.io/docs/drivers/

Docker Desktop recently changed its policy to a paid plan for certain users, this gave me a new challenge:

Can I run Docker without docker-desktop on the new M1?

All the solutions online gave the Intel processor solution: Install minikube. This quickly became a chicken or egg situation. Using Docker as the driver without using docker-desktop is not an option at the moment, the only solution mentioned time and time again seems to be the docker-desktop on the M1 Pro.

Surely there are other drivers that I could use?

## Other drivers

Since Rancher desktop is using Qemu on the M1 Pro, I surely could use it… Right? Since KVM uses Qemu I decided to give the kvm2 driver a try:

Rogiers-MBP:0to1 rogierdikkes$minikube start –driver=kvm2 😄 minikube v1.24.0 on Darwin 12.1 (arm64) ❌ Exiting due to DRV_UNSUPPORTED_OS: The driver ‘kvm2’ is not supported on darwin/arm64 No support in Minikube, so I tried hyperkit: brew install hyperkit Error: hyperkit: no bottle available! You can try to install from source with: brew install –build-from-source hyperkit Slowly options started running out, building from source? This is while the package is easy to install on the Intel MacBooks, so I didn’t want to go this route. Time to give good ol’ Redhat a try. ## Podman to the rescue? With podman the installation went very smoothly, unfortunately, I didn’t make the finish line with podman. With Podman I ran into bugs on the M1. I decided to write down what I did in case the bug I encountered gets fixed. brew install podman After installing Podman you have to init the machine. Minikube requires 2 CPUs, so provide the podman VM with 2 CPUs at init. podman machine init --cpus 2 Downloading VM image: fedora-coreos-35.20220103.2.0-qemu.aarch64.qcow2.xz: done Extracting compressed file Then start the machine: podman machine start INFO[0000] waiting for clients… INFO[0000] listening tcp://127.0.0.1:7777 INFO[0000] new connection from to /var/folders/s1/cythsgh50hd9tl7_38qknb2r0000gn/T/podman/qemu_podman-machine-default.sock Waiting for VM … Machine “podman-machine-default” started successfully As you can see podman downloads a qcow2 file to use with Qemu. A similar approach Rancher Desktop takes. After this is completed you can start minikube with the podman driver: minikube start --driver=podman 😄 minikube v1.24.0 on Darwin 12.1 (arm64) ✨ Using the podman (experimental) driver based on user configuration 👍 Starting control plane node minikube in cluster minikube 🚜 Pulling base image … E0107 13:11:31.539966 90940 cache.go:201] Error downloading kic artifacts: not yet implemented, see issue #8426 🔥 Creating podman container (CPUs=2, Memory=1953MB) …| ERRO[0123] accept tcp [::]:38413: use of closed network connection ERRO[0123] accept tcp [::]:41795: use of closed network connection ERRO[0123] accept tcp [::]:36729: use of closed network connection ERRO[0123] accept tcp [::]:44515: use of closed network connection ERRO[0123] accept tcp [::]:35925: use of closed network connection ✋ Stopping node “minikube” … 🔥 Deleting “minikube” in podman … 🤦 StartHost failed, but will try again: creating host: create: creating: create kic node: create container: podman run -d -t –privileged –device /dev/fuse –security-opt seccomp=unconfined –tmpfs /tmp –tmpfs /run -v /lib/modules:/lib/modules:ro –hostname minikube –name minikube –label created_by.minikube.sigs.k8s.io=true –label name.minikube.sigs.k8s.io=minikube –label role.minikube.sigs.k8s.io= –label mode.minikube.sigs.k8s.io=minikube –network minikube –ip 192.168.49.2 –volume minikube:/var:exec –memory-swap=1953mb –memory=1953mb –cpus=2 -e container=podman –expose 8443 –publish=127.0.0.1::8443 –publish=127.0.0.1::22 –publish=127.0.0.1::2376 –publish=127.0.0.1::5000 –publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.28: exit status 126 stdout: stderr: Error: the requested cgroup controller cpu is not available: OCI runtime error 🔥 Creating podman container (CPUs=2, Memory=1953MB) … 😿 Failed to start podman container. Running “minikube delete” may fix it: creating host: create: creating: setting up container node: creating volume for minikube container: podman volume create minikube –label name.minikube.sigs.k8s.io=minikube –label created_by.minikube.sigs.k8s.io=true: exit status 125 stdout: stderr: Error: volume with name minikube already exists: volume already exists ❌ Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: creating: setting up container node: creating volume for minikube container: podman volume create minikube –label name.minikube.sigs.k8s.io=minikube –label created_by.minikube.sigs.k8s.io=true: exit status 125 stdout: stderr: Error: volume with name minikube already exists: volume already exists What seems to be happening is that the group CPU is unavailable in the first deployment. Currently, there is an issue open for this, so podman was not the way forward. ## next! VFTOOL Since podman did not work and the Minikube setup on the M1 was broken I had the hope that the binaries for the Ubuntu setup would work. Again I just should have used Rancher Desktop, but now I was curious if there was any way to get Minikube working on the M1. The idea was to get Ubuntu running on the M1, since VirtualBox was no longer an option, my approach was to use vftool to run a virtual machine. Inside that virtual machine, you run minikube. I had the hope that the Ubuntu binaries would not have issues and Minikube would just get installed. First, let’s create a folder so we can isolate our mess and allows us to start setting up vftool to build a virtual machine. mkdir -p ~/VM/Ubuntu cd ~/VM/ git clone git@github.com:evansm7/vftool.git cd ./vftool make file ./build/vftool cp ./build/vftool /usr/local/bin/ Above copy command to /usr/local/bin is optional, it saved me a couple of seconds to find the binary when you want to use vftool after the build. The file the command should give the following output: Mach-O 64-bit executable arm64 After this is complete you can move onward to setting up the VM vftool will be using. First, download the iso for Ubuntu. We also need 7zip to unpack the ISO to get access to the kernel and initrd. We need to gunzip the kernel and copy the initrd to our VM folder. cd ~/VM/Ubuntu curl --silent -L https://cdimage.ubuntu.com/focal/daily-live/current/focal-desktop-arm64.iso -o ~/VM/Ubuntu/focal.iso brew install 7zip 7zz x -tiso -y ~/VM/Ubuntu/focal.iso -o~/VM/Ubuntu/focal-unpacked cp ~/VM/Ubuntu/focal-unpacked/casper/initrd ~/VM/Ubuntu/ cp ~/VM/Ubuntu/focal-unpacked/casper/vmlinuz ~/VM/Ubuntu/vmlinuz.gz gunzip ~/VM/Ubuntu/vmlinuz.gz After this is done you have the 2 files you need, ISO and the focal-unpacked folder is no longer necessary. The next step is to create a block device to use for Ubuntu, I have tried to use the ISO downloaded however vftool kept giving the error that it was an invalid storage device. So we create an empty block device to use as the disk for the virtual machine: dd if=/dev/zero of=~/VM/Ubuntu/focal-server-cloudimg-arm64.img seek=60000000 obs=1024 count=0 This creates an empty raw device for you to use with the size of 60G, the reason is that minikube uses around 33G for its installation alone. We will be using this volume for the docker images and minikube images etc. If we don’t use this image we will run out of disk space when starting minikube. Since this virtual machine is entirely in memory we need to add a lot of memory to make minikube run without getting OOM. Now let’s start the virtual machine! vftool -k ~/VM/Ubuntu/vmlinuz -i ~/VM/Ubuntu/initrd -d ~/Ubuntu/VM/focal-server-cloudimg-arm64.img -m 8192 -p 2 -a "console=tty0 console=hvc0 root=/dev/vda1 ds=nocloud" The following message returns: 2022-01-07 18:11:43.308 vftool[1565:1811117] vftool (v0.3 10/12/2020) starting 2022-01-07 18:11:43.309 vftool[1565:1811117] +++ kernel at /Users/rogierdikkes/VM/Ubuntu/vmlinuz, initrd at /Users/rogierdikkes/VM/Ubuntu/initrd, cmdline ‘console=tty0 console=hvc0 root=/dev/vda1 ds=nocloud’, 2 cpus, 4096MB memory 2022-01-07 18:11:43.310 vftool[1565:1811117] +++ fd 3 connected to /dev/ttys007 2022-01-07 18:11:43.310 vftool[1565:1811117] +++ Waiting for connection to: /dev/ttys007 2022-01-07 18:11:47.523 vftool[1565:1811117] +++ Attaching disc /Users/rogierdikkes/VM/Ubuntu/focal-server-cloudimg-arm64.img 2022-01-07 18:11:47.523 vftool[1565:1811117] +++ Configuration validated. 2022-01-07 18:11:47.523 vftool[1565:1811117] +++ canStart = 1, vm state 0 2022-01-07 18:11:47.602 vftool[1565:1811140] +++ VM started With screen we will now connect to this virtual machine, you need to search for the tty which you can attach to. In my case it was /dev/ttys007 screen /dev/ttys007 This should open a session with Ubuntu. Now you can finish the Ubuntu install on the virtual machine. While installing you can keep everything default, except for 1 field: URL. The URL the field is incorrect in the installation, the defaults point towards images that do not exist: http://cdimage.ubuntu.com/releases/focal/release/ubuntu-20.04-live-server-arm64.iso http://cdimage.ubuntu.com/releases/focal/release/ubuntu-20.04-desktop-arm64.iso Replace these 2 URLs with the ISO we downloaded earlier: https://cdimage.ubuntu.com/focal/daily-live/current/focal-desktop-arm64.iso After this, you see the download and you get the login prompt. You can use the username ubuntu without a password. Now this is done we need to mount the raw disk to /var/lib/docker so all the images end up here: sudo mkdir -p /var/lib/docker sudo mkfs.ext4 /dev/vda sudo mount /dev/vda /var/lib/docker Once this is done we will create a symlink to the /var/lib/docker directory to place the minikube files at the same location as the docker files. It’s a bit messy, but I kept getting out of disk space errors and was done with this… sudo ln -s /var/lib/docker /var/lib/minikube After this is done let’s install the Docker engine on Ubuntu, you can use this guide. https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository For the completion guide I list the steps here in a oneliner: sudo apt update -y && \ sudo apt install -y ca-certificates curl gnupg lsb-release && \ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg && \ echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && \ sudo apt update -y && \ sudo apt install -y docker-ce docker-ce-cli containerd.io conntrack OverlayFS cannot mount on top of the liveCD, this will give the following error: docker: Error response from daemon: error creating aufs mount to /var/lib/docker/aufs/mnt/8017e1251b9eab3500fcd0b677b96c9e5f9430dd5249354b120feb0019aa09e8-init: mount target=/var/lib/docker/aufs/mnt/8017e1251b9eab3500fcd0b677b96c9e5f9430dd5249354b120feb0019aa09e8-init data=br:/var/lib/docker/aufs/diff/8017e1251b9eab3500fcd0b677b96c9e5f9430dd5249354b120feb0019aa09e8-init=rw:/var/lib/docker/aufs/diff/077e60cb2a82a857e71c73b6ec7cd8c667d0f1b15fdc1f5a228eac2e8f852872=ro+wh,dio,xino=/dev/shm/aufs.xino: invalid argument. In dmesg you can see the following error: [ 681.897119] overlayfs: missing ’lowerdir’ You can solve this by using a heredoc to configure the vfs storage-driver in the Docker daemon.json file, this has some negative side effects but it was the only storage driver I got working since aufs was deprecated and overlayfs doesn’t work in the liveCD. sudo bash -c "tee <<EOF > /etc/docker/daemon.json { \"storage-driver\": \"vfs\" } EOF" Because we just used the vfs storage-driver we cannot run the minikube command anymore with driver type none, minikube will fail in its preflight check: error execution phase preflight: [preflight] Some fatal errors occurred: $ERROR SystemVerification$: unsupported graph driver: vfs After adding the storage driver vfs you need to restart the docker process: sudo systemctl restart docker Check if the hello-world runs: sudo docker run hello-world After this is completed correctly you can download minikube: curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-arm64 chmod +x ./minikube sudo cp ./minikube /usr/local/bin Adjust the following parameter: sudo sysctl fs.protected_regular=0 Set a password, else you cannot add yourself to the docker group: passwd After setting the password add the ubuntu user to the docker group: sudo usermod -aG docker$USER && newgrp docker

After this is complete you can run the minikube command, running it with sudo will make it fail:

minikube start --driver=docker

Since minikube starts just like your OS in memory this is the moment where OOM events happen. Now let’s check if minikube works:

minikube kubectl -- get pods -A

NAMESPACE NAME READY STATUS RESTARTS AGE

kube-system coredns-78fcd69978-dt8cw 1/1 Running 0 12s

kube-system etcd-minikube 1/1 Running 0 32s

kube-system kube-apiserver-minikube 1/1 Running 0 28s

kube-system kube-controller-manager-minikube 1/1 Running 0 25s

kube-system kube-proxy-qc7sm 1/1 Running 0 12s

kube-system kube-scheduler-minikube 1/1 Running 0 28s

kube-system storage-provisioner 1/1 Running 0 20s

## Done?

Halfway through the vftool the setup I started to realize that connecting from my MacBook to the Minikube setup would take more and more time. I started to think about alternatives. I had the Rancher Desktop running before on Intel and had the idea that it was an alternative. There is another post on how to set up Rancher Desktop. Having the ability to correct directly from my M1 terminal and my vscode setup was not possible in the vftool setup above.

The vftool setup is not 100% stable, I had dozens of kernel panics during the setup and trying to figure out how to set up minikube inside vftool. I even had kernel oops during the creation of the symlink.