Choosing a Local Dev Cluster
How do you run Kubernetes locally?
There are lots of Kubernetes dev solutions out there. The choices can be overwhelming. We’re here to help you figure out the right one for you.
Kind runs Kubernetes inside a Docker container.
The Kubernetes team uses Kind to test Kubernetes itself. But its fast startup time also makes it a good solution for local dev. Follow these instructions to set up Kind for use with Tilt:
- Creating a new cluster is fast (~20 seconds). Deleting a cluster is even faster.
- Much more robust than Docker for Mac. Uses containerd instead of docker-shim. Short-lived clusters tend to be more reliable.
- Supports a local image registry (with our custom setup instructions). Pushing images is fast. No fiddling with image registry auth credentials.
- Can run in most CI environments (TravisCI, CircleCI, etc.)
- The local registry setup is still new, and changing rapidly. You need to be using Tilt v0.12.0+
- If Tilt can’t find the registry, it will use the much slower
kind loadto load images. (This con is mitigated if you use Kind with a local registry, as described in the instructions linked above.)
Docker for Desktop
Docker for Desktop is the easiest to get started with if you’re on MacOS.
In the Docker For Mac preferences, click Enable Kubernetes
- Widely used and supported.
- Nothing else to install.
- Built images are immediately available in-cluster. No pushing and pulling from image registries.
- If Kubernetes breaks, it’s easier to reset the whole thing than debug it.
- Much more resource-intensive because it uses docker-shim as the container runtime.
- Different defaults than a prod cluster and difficult to customize.
- Not available on Linux.
Microk8s is what we recommend most often for Linux users.
sudo snap install microk8s --classic && \ sudo microk8s.enable dns && \ sudo microk8s.enable registry
Make microk8s your local Kubernetes cluster:
sudo microk8s.kubectl config view --flatten > ~/.kube/microk8s-config && \ KUBECONFIG=~/.kube/microk8s-config:~/.kube/config kubectl config view --flatten > ~/.kube/temp-config && \ mv ~/.kube/temp-config ~/.kube/config && \ kubectl config use-context microk8s
- No virtual machine overhead on Linux
- Ships with plugins that make common configs as easy as
- Supports a local image registry with
microk8s.enable registry. Pushing images is fast. No fiddling with image registry auth credentials.
- Resetting the cluster is slow and error-prone.
- Optimized for Linux. You can use it on MacOS and Windows with Multipass.
Minikube is what we recommend when you’re willing to pay some overhead for a more high-fidelity cluster.
Minikube creates a Kubernetes cluster in a VM, and has tons of options for customizing the cluster.
- The most full-featured local Kubernetes solution
- Can easily run different Kubernetes versions, container runtimes, and controllers
- You can build images in-cluster with
minikube docker-env. When you use Minikube, Tilt will automatically use Minikube’s Docker, so that you don’t need to push to a remote registry.
- The VM makes everything much slower, both at start-time and run-time
- We often see engineers struggle to set it up the first time, getting lost in a maze of VM drivers that they’re unfamiliar with
- You usually want to shutdown minikube when you’re finished
k3s is fully compliant with “full” Kubernetes, but has a lot of optional and legacy features removed. Follow these instructions to set up k3d for use with Tilt:
- Extremely fast to start up (less than 5 seconds on most machines)
- It’s easy to run k3d with a local registry that Tilt will auto-detect, which means less finicky setup, and fast pushing/pulling of images
- Tilt does not yet natively support
k3d import-images, so for a smooth local dev experience with Tilt, you have to use a local image registry (which you get for free if you set up k3d using the instructions linked above)
- The least widely used. That’s not necessarily bad. Just be aware that there’s less documentation on its pitfalls. Tools (including the Tilt team!) tend to be slower to add support for it.
(EKS, AKS, GKE, and custom clusters)
By default, Tilt will not let you develop against a remote cluster.
If you start Tilt while you have
kubectl configured to talk to a remote
cluster, you will get an error. You have to explicitly whitelist the cluster with:
We only recommend remote clusters for large engineering teams where a dedicated dev infrastructure team can maintain your dev cluster.
Or if you need to debug something that only reproduces in a complete cluster.
- Can customize to your heart’s desire
- Share common services (e.g., a dev database) across developers
- Use a cheap laptop and the most expensive cloud instance you can buy for development
- Need to use a remote image registry. Make sure you have Tilt’s live_update set up!
- Need to set up namespaces and access control so that each dev has their own sandbox
- If the cluster needs to be reset, we hope you’re good friends with your DevOps team