Once you’ve bought into container-based development, you’ll want to set up CI to ensure your servers don’t break.
The Tilt team has this problem too! For every change, we create a real single-use cluster and verify that all our services and example projects still work. So we have a lot of experience with best practices.
There are two tools you’ll need to configure:
1) How to use
tilt ci to build and deploy servers.
2) How to run a Kubernetes cluster in CI that
tilt ci can deploy to.
Let’s dig into examples of each one.
tilt ci command
tilt ci command is made for CI jobs.
How it Works
Under the hood,
1) Executes your Tiltfile.
2) Runs all
3) Builds all images.
4) Deploys all Kubernetes resources.
5) Waits until all servers and other resources are healthy.
tilt ci defaults to log-streaming mode. The web UI is still accessible,
but the terminal will be a simple log stream.
If any step fails, Tilt will exit immediately with an error code.
If any resource crashes or looks like it will never succeed (e.g., a pod that takes too long to schedule), Tilt will also exit with an error code.
Once all services are healthy, it will exit with status code 0.
Any portforwards will not be active after
tilt ci has exited.
Most of our example projects use CircleCI to run
Each example invokes
ctlptl to set up a single-use
cluster. But the
ctlptl tool has many options for setting up clusters,
depending on what you need.
tilt ci will be waiting on a server to come up. But it can be hard to tell
what it’s waiting on. Tilt has tools to help debug this!
tilt ci still runs the normal web dashboard at
http://localhost:10350/. That should
be familiar, so use that first.
But maybe the dashboard doesn’t help. The next step is to inspect the
Session object is the Tilt API that drives
For human-readable output, run:
tilt describe session
For machine-readable output, run:
tilt get session -o yaml tilt get session -o json
The status of the session reflects:
- The current Tilt PID.
- The time Tilt started at.
- Each target that Tilt is waiting on, and its current state (waiting, running, or terminated).
We also pair with teams on how to integrate this status reporting with their own in-house tools, like in the tilt-status VSCode extension.
For more info, here’s the complete API reference of the Session object.
Single-use Kubernetes clusters
Running a Kubernetes cluster in CI can be harder than running one locally.
Here are some options, with the pros and cons of each:
Easiest: Cluster on VM-based CI
kind is currently the gold standard for running
Kubernetes in CI. The Kubernetes project itself uses it for testing.
create clusters inside Docker.
kind comes with the ability to run a local registry, so you can push images to
the registry on
localhost:5000 and pull them from inside
Set up a CI pipeline that:
Creates a VM.
Installs all our dependencies, including Docker.
kindcluster with a local registry at
localhost:5000, using their script.
This is the approach we use to test
ctlptl with both
kind. Here’s the CI
Tilt will auto-detect that the registry is running on
localhost:5000 and push
images there instead of your prod image registry.
The downside is that most teams are more comfortable managing container images than managing VMs. VMs are slower. Upgrading dependencies is more heavyweight.
Recommended: Cluster in Remote Docker-based CI
Many CI environments offer a remote Docker environment outside the container. You can run test code in a container that talks to Docker, without the pitfalls of running Docker inside Docker.
Set up a CI pipeline that:
Create a container with your code.
Create a remote Docker environment.
kindcluster with a local registry inside the remote Docker environment.
socatto expose the remote registry and Kubernetes cluster inside the local container.
socat element makes this a bit tricky.
ctlptl, a CLI for declaratively
setting up local Kubernetes clusters. If you’re using
ctlptl, it will try to
detect when you have a remote docker environment and set up the
If you want to wire it up yourself, check out this Bash script.
Not Recommended: Remote Registry
You may already have a container image registry that you prefer for development, like Quay.io or Google Cloud Registry.
If you want to use this registry in CI, you need to set up permissions so that your CI job can write to this registry.
Create a dedicated image registry for CI.
Create a service account or access token with a secret in your CI build.
kindto create the cluster.
This is an approach a lot of teams try. We usually don’t recommend it. The downsides:
Managing secrets and permissions for the remote registry can be a pain. You’ll want to set it up so that anyone who can send a pull request to your repository can also write to the remote registry. This may look different depending on your org structure.
It’s hard to guarantee that images aren’t leaking between tests. For example, if image pushing failed, you’ll want to be sure we weren’t picking up a cached image from a previous test. One solution is to reset the whole registry at regular intervals.
But it might be the best option if you’re not able to easily modify where your tools are pushing images to and pulling images from.