Custom Image Builders
docker build is the common way to build container images, but there are many
This guide will show you how to use them!
The Easiest Way: Use a Tilt Extension
The Tilt community has contributed many extensions that let you define your container images with many widely-used image builders.
tilt-example-builders repo demonstrates how to use them.
Each subdirectory of the repo uses a different image builder.
Each builder contains a test that spins up a real, ephemeral Kubernetes cluster on CircleCI, builds the image, deploys it to the cluster, and verifies that the image behaves as expected.
Also check out the complete index of Tilt extensions. There are many more image builder extensions that don’t have official example projects yet.
The Next Way: Running Your Own Image Builder
If there’s no extension yet for your image builder, that’s OK.
All image-builder extensions use the
function, a more complete API for running your image builds as subprocesses of Tilt.
Let’s take a look at how to use it.
custom_build calls require:
A name of the image to build (as a ref, e.g.
A command to run (e.g.
bazel build //frontend:imageor
Files to watch (e.g.
['frontend', 'util', 'data.txt']). When a dependency changes, Tilt starts an update to build the image then apply the YAML.
There are a couple different image-building patterns.
Custom Docker Builds
Suppose you have a script that wraps
docker build, but adds some application-specific abstractions.
Here’s a simple example that invokes
docker build to build an image named
frontend from the directory
custom_build( 'frontend', 'docker build -t $EXPECTED_REF frontend', ['./frontend'], )
Tilt will run this command to build the image, verify that the image is in the Docker image store, then push the image to the appropriate image registry.
You can also use this pattern to use
docker flags that the
function doesn’t support.
Jib, Bazel, or any other builder that interoperates with Docker
Many tools can create Docker images, then write them to the local Docker image store.
For example, Jib has plugins that integrate with your existing Java tooling and create Java-based images.
The tilt-example-java repo has an example
custom_build to generate images with Gradle and Jib:
custom_build( 'example-java-image', './gradlew jibDockerBuild --image $EXPECTED_REF', deps=['src'])
Bazel, the general-purpose build system, also takes this approach. Bazel’s rules_docker extension assembles Docker images and writes them to the local Docker image store.
See the tutorial on how to use
custom_build to build images with
Buildah (or any image builder indepdendent of Docker)
Buildah is an independent Docker image builder.
Buildah has its own API and own image store. A
custom_build() call needs to
both build and push the image.
custom_build( 'frontend', 'buildah bud -t $EXPECTED_REF frontend && buildah push $EXPECTED_REF $EXPECTED_REF', ['./frontend'], skips_local_docker=True, )
skips_local_docker parameter indicates that we don’t expect the image to
ever show up in the local Docker image store. Tilt shouldn’t try to verify the
There are a couple of caveats you should be aware of with
buildah (and similar builders):
They often require privileged access. You may need to run Tilt with
sudoor inside an appropriate sandbox
If you’re using Tilt to push to an insecure registry, you will need to configure your builder for that registry. For example, to use
buildahwith Microk8s, you need to add
How to Write Your Own
We’ve looked at a couple simple recipes for how to write a custom build script.
To write more complex ones, we need to understand in more detail how they work.
All the commands above contain
$EXPECTED_REF. What is that?
Tilt always pushes a content-based, immutable tag, not a bare ref. (Instead of
gcr.io/company-name/frontend, Tilt injects
gcr.io/company-name/frontend:tilt-ffd9c2013b5bf5d4, where the
ffd9c2013b5bf5d4 part is based on the contents of your image). Before explaining why (see below), let’s describe what this means for your Tiltfile and build script.
There are two ways for Tilt and your build script to coordinate image builds.
The Good Way
Most tools take a destination of the image as an argument (e.g.
Before running your build script, Tilt sets the environment variable
$EXPECTED_REFwith a randomized tag (e.g.
The custom build script builds the image and tags it with
After the build script exits, Tilt reads the new image at
$EXPECTED_REF, re-tags it with a content-based tag, and pushes it to the image registry.
The Hacky Way
Other tools have an image ref hard-coded in configuration. They’ll build to the same tag each time.
Instead of writing a wrapper script around your tool, tell Tilt what tag the
build image will have with
After Tilt runs your build command, it will find this image and retag and push it with a content-based tag.
This method is generally less robust, because the script is building to a mutable tag instead of an immutable tag.
An Improvement on the Hacky Way
If you’re willing to invest more into your custom build script, you should use content-based tags!
custom_build(outputs_image_ref_to='ref.txt') will tell Tilt that your custom
build script intends to write a tagged image reference to the file
Tilt will then inject that image into your deployments.
If Tilt has detected a local registry, it will populate the environment variable
REGISTRY_HOST=localhost:5000) before calling the build script.
Determining the Content-based Tag
In rare cases, another script in your build system may need to know what tag Tilt is going to deploy. This typically only comes up if your team has written their own artisanal image build system that’s closely coupled with Kubernetes.
Tilt has a special command to help with this. After you build the image, run:
tilt dump image-deploy-ref $EXPECTED_REF
Tilt will read the image, determine the hash of the context, and print out the full name and content-based tag.
NOTE: This is not a common use-case. Usually, when teams ask about this, they’re writing a workflow engine that creates its own pods (like Airflow), and need a way to get the deploy tag at runtime. So they hack custom_build to grab the deploy tag at build-time, and plumb it through to their runtime pods. There’s a better way to do this. Use this guide instead.
Live Update and Other Features
docker_build supports other options. The most impactful is
live_update, which lets you update code in Kubernetes without doing a full image build.
custom_build supports this as well, using the same syntax.
custom_build supports most other options of
docker_build, and a few specific to non-Docker container builders.
Adjust File Watching with
While most of the points in our Debugging File Changes guide hold true for
ignore parameter (which adjusts the set of files watched for a given build) works a bit differently, and is worth discussing briefly.
ignore parameter takes a pattern or list of patterns (following
.dockerignore syntax; files matching any of these patterns will not trigger a build.
Of note, these patterns are evaluated relative to each
dep. E.g. given the following call:
custom_build( 'image-foo', 'docker build -t $EXPECTED_REF .', deps=['dep1', 'dep2'], ignore=['baz'] )
Tilt will ignore
Why Tilt uses Immutable Tags
Immutable tags have a long history in the Kubernetes community.
The Knative team has this presentation that gives a good overview:
Why we resolve tags in Knative
Mutable tags have good usability and security properties. For example, a
registry:v2 image that has the latest, most secure minor version of the v2
Immutable tags have good reliability and caching properties. For example, if
you’re rolling out 3 pods of
registry:v2, you want to be sure all pods have
the exact same version. Deploying with a mutable reference creates a race
condition. Pods created at different times from the same definition may end up
running different code as the reference is overwritten.
Tilt only deploys immutable tags. Instead of pushing to
gcr.io/company-name/frontend, Tilt re-tags the image as
gcr.io/company-name/frontend:tilt-ffd9c2013b5bf5d4. The unique bit is a
Nonce or a digest of the
contents. (Technically the tag isn’t write-protected in any way, but the
improbability of collisions means we can pretend it’s immutable.)
Tilt then injects the new tag into the container spec. This makes the Tilt experience faster and more reliable, because we can instruct Kubernetes to cache the tag aggressively as if it’s immutable.
Knative uses a similar strategy, but the immutability is enforced by a Kuberentes operator, instead of by client-side tooling.
When You’re Done
If you have a more complex build script that you’re not sure how to integrate
with Tilt, we’d love to hear about it. Come find us in the
#tilt channel in
Kubernetes Slack or
file an issue on GitHub.
We’ll love you even more if you share it with other Tilt users as an extension!