Using Podman instead of Docker¶

Podman has an Docker API compatibility layer. k3d uses the Docker API and is compatible with Podman v4 and higher.

Podman support is experimental

k3d is not guaranteed to work with Podman. If you find a bug, do help by filing an issue

Tested with podman version:

Client:       Podman Engine
Version:      4.3.1
API Version:  4.3.1

Using Podman¶

Ensure the Podman system socket is available:

sudo systemctl enable --now podman.socket
# or to start the socket daemonless
# sudo podman system service --time=0 &

Disable timeout for podman service:
See the podman-system-service (1) man page for more information.

mkdir -p /etc/containers/containers.conf.d
echo 'service_timeout=0' > /etc/containers/containers.conf.d/timeout.conf

To point k3d at the right Docker socket, create a symbolic link:

sudo ln -s /run/podman/podman.sock /var/run/docker.sock
# or install your system podman-docker if available
sudo k3d cluster create

Alternatively, set DOCKER_HOST when running k3d:

export DOCKER_HOST=unix:///run/podman/podman.sock
export DOCKER_SOCK=/run/podman/podman.sock
sudo --preserve-env=DOCKER_HOST --preserve-env=DOCKER_SOCK k3d cluster create

Using rootless Podman¶

Ensure the Podman user socket is available:

systemctl --user enable --now podman.socket
# or podman system service --time=0 &

Set DOCKER_HOST when running k3d:

XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
export DOCKER_SOCK=$XDG_RUNTIME_DIR/podman/podman.sock
k3d cluster create

Using cgroup (v2)¶

By default, a non-root user can only get memory controller and pids controller to be delegated.

To run properly we need to enable CPU, CPUSET, and I/O delegation

Make sure you’re running cgroup v2

If /sys/fs/cgroup/cgroup.controllers is present on your system, you are using v2, otherwise you are using v1.

mkdir -p /etc/systemd/system/user@.service.d
cat > /etc/systemd/system/user@.service.d/delegate.conf <<EOF
[Service]
Delegate=cpu cpuset io memory pids
EOF
systemctl daemon-reload

Reference: https://rootlesscontaine.rs/getting-started/common/cgroup2/#enabling-cpu-cpuset-and-io-delegation

Using remote Podman¶

Start Podman on the remote host, and then set DOCKER_HOST when running k3d:

export DOCKER_HOST=ssh://username@hostname
export DOCKER_SOCK=/run/user/1000/podman/podman.sock
k3d cluster create

macOS¶

Initialize a podman machine if not done already

podman machine init

Or start an already existing podman machine

podman machine start

Grab connection details

podman system connection ls
Name                         URI                                                         Identity                                      Default
podman-machine-default       ssh://core@localhost:53685/run/user/501/podman/podman.sock  /Users/myusername/.ssh/podman-machine-default  true
podman-machine-default-root  ssh://root@localhost:53685/run/podman/podman.sock           /Users/myusername/.ssh/podman-machine-default  false

Edit your OpenSSH config file to specify the IdentityFile

vim ~/.ssh/config

Host localhost
    IdentityFile /Users/myusername/.ssh/podman-machine-default

Rootless mode¶

Delegate the cpuset cgroup controller to the user’s systemd slice, export the docker environment variables referenced above for the non-root connection, and create the cluster:

podman machine ssh bash -e <<EOF
  printf '[Service]\nDelegate=cpuset\n' | sudo tee /etc/systemd/system/user@.service.d/k3d.conf
  sudo systemctl daemon-reload
  sudo systemctl restart "user@\${UID}"
EOF

export DOCKER_HOST=ssh://core@localhost:53685
export DOCKER_SOCKET=/run/user/501/podman/podman.sock
k3d cluster create --k3s-arg '--kubelet-arg=feature-gates=KubeletInUserNamespace=true@server:*'

Rootful mode¶

Export the docker environment variables referenced above for the root connection and create the cluster:

export DOCKER_HOST=ssh://root@localhost:53685
export DOCKER_SOCK=/run/podman/podman.sock
k3d cluster create

Podman network¶

The default podman network has dns disabled. To allow k3d cluster nodes to communicate with dns a new network must be created.

podman network create k3d
podman network inspect k3d -f '{{ .DNSEnabled }}'
true

Creating local registries¶

Because Podman does not have a default “bridge” network, you have to specify a network using the --default-network flag when creating a local registry:

k3d registry create --default-network podman mycluster-registry

To use this registry with a cluster, pass the --registry-use flag:

k3d cluster create --registry-use mycluster-registry mycluster

Incompatibility with --registry-create

Because --registry-create assumes the default network to be “bridge”, avoid --registry-create when using Podman. Instead, always create a registry before creating a cluster.

Missing cpuset cgroup controller

If you experince an error regarding missing cpuset cgroup controller, ensure the user unit xdg-document-portal.service is disabled by running systemctl --user stop xdg-document-portal.service. See this issue