Choosing a Local Dev Cluster
How do you run Kubernetes locally?
There are lots of Kubernetes dev solutions out there. The choices can be overwhelming. We’re here to help you figure out the right one for you.
Docker for Desktop
Docker for Desktop is what we recommend most often for MacOS users.
In the Docker For Mac preferences, click Enable Kubernetes
- Widely used and supported.
- Nothing else to install.
- Built images are immediately available in-cluster. No pushing and pulling from image registries.
- If Kubernetes breaks, it’s easier to reset the whole thing than debug it.
- Different defaults than a prod cluster and difficult to customize.
- Not available on Linux.
Microk8s is what we recommend most often for Linux users.
sudo snap install microk8s --classic && \ sudo microk8s.enable dns && \ sudo microk8s.enable registry
Make microk8s your local Kubernetes cluster:
sudo microk8s.kubectl config view --flatten > ~/.kube/microk8s-config KUBECONFIG=~/.kube/microk8s-config:~/.kube/config kubectl config view --flatten > ~/.kube/temp-config mv ~/.kube/temp-config ~/.kube/config kubectl config use-context microk8s
- No virtual machine overhead
- Ships with plugins that make common configs as easy as
- The in-cluster registry makes image updates much faster. When you enable the registry, Tilt will use it automatically.
- Resetting the cluster is slow and error-prone
Minikube is what we recommend when you’re willing to pay some overhead for a more high-fidelity cluster.
Minikube creates a Kubernetes cluster in a VM, and has tons of options for customizing the cluster.
- The most full-featured local Kubernetes solution
- Can easily run different Kubernetes versions, container runtimes, and controllers
- You can build images in-cluster with
minikube docker-env. When you use Minikube, Tilt will automatically use Minikube’s Docker, so that you don’t need to push to a remote registry.
- The VM makes everything much slower, both at start-time and run-time
- We often see engineers struggle to set it up the first time, getting lost in a maze of VM drivers that they’re unfamiliar with
- You usually want to shutdown minikube when you’re finished
KIND runs Kubernetes inside a Docker container.
The Kuberetes team uses KIND to test Kubernetes itself. But its fast startup time also makes it a good solution for local dev. Run it with:
GO111MODULE="on" go get email@example.com kind create cluster export KUBECONFIG="$(kind get kubeconfig-path)"
- Fast to startup (less than 45 seconds on most machines)
- Creating and deleting ephemeral clusters feels great
- Can run in most CI environments (TravisCI, CircleCI, etc)
- Pushing images into the cluster is slow.
- KIND can copy images into the cluster with
kind load. This is slower than an in-cluster registry because it copies the whole image even if a single layer has changed. But it’s still faster than a remote registry. Tilt will use
kind loadif it detects KIND.
- We have examples on how to run an in-cluster registry, but it’s brittle.
K3D runs K3s, a lightweight Kubernetes distro, inside a Docker container.
K3s is fully compliant with “full” Kubernetes, but has a lot of optional and legacy features removed.
curl -sfL https://get.k3s.io | sh - k3d create cluster export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"
- Extremely fast to start up (less than 5 seconds on most machines)
- Pushing images into the cluster is slow.
- Tilt does not yet natively support
k3d import-images, which has all the same problems as
- The least widely used. That’s not necessarily bad. Just be aware that there’s less documentation on its pitfalls. Tools (including the Tilt team!) tend to be slower to add support for it.
(EKS, AKS, GKE, and custom clusters)
By default, Tilt will not let you develop against a remote cluster.
If you start Tilt while you have
kubectl configured to talk to a remote
cluster, you will get an error. You have to explicitly whitelist the cluster with:
We only recommend remote clusters for large engineering teams where a dedicated dev infrastructure team can maintain your dev cluster.
Or if you need to debug something that only reproduces in a complete cluster.
- Can customize to your heart’s desire
- Share common services (e.g., a dev database) across developers
- Use a cheap laptop and the most expensive cloud instance you can buy for development
- Need to use a remote image registry. Make sure you have Tilt’s live_update set up!
- Need to set up namespaces and access control so that each dev has their own sandbox
- If the cluster needs to be reset, we hope you’re good friends with your DevOps team