Custom Resource Definitions
Kubernetes defines well-thought-out primitives like Deployments, Jobs, and StatefulSets for running your containers.
Tilt knows how to inject images, monitor status, and fetch logs for these built-in resources.
But your team may define your own custom resources! Or you may use a project like KubeDB that uses custom resources to operate a database.
Tilt needs your help to recognize them.
Custom Resource Functions
For any resource, Tilt needs to know 3 things:
Is this an independent resource?
By default, Tilt doesn’t categorize custom resources. They all get grouped together in one “uncategorized” bucket of resources.
When a resource is independent, Tilt deploys it separately, and gives it its own log pane.
You can watch its status and logs separate from other resources.
To make a resource independent, use the k8s_resource
function in your
Titlfile:
k8s_resource(new_name='my-postgres-server',
objects=['postgres-name'])
Does it contain images?
Tilt can build images and deploy them with your resource – if it knows how to find them!
To identify the location of the resource, use the k8s_kind
function in your Tiltfile:
k8s_kind('UselessMachine', image_json_path='{.spec.image}')
See the k8s_kind API docs for more detail, or look at this example.
Does it create pods?
Tilt can sometimes follow the Kubernetes metadata to figure out which resources have pods, and which pods belong to which resource. But other times it needs help.
To tell Tilt that a resource has pods, and it should wait for them to become healthy:
k8s_kind('UselessMachine', image_json_path='{.spec.image}', pod_readiness='wait')
To tell Tilt that a resource does NOT have pods, and it should consider a resource healthy if no pods come up:
k8s_kind('UselessMachine', image_json_path='{.spec.image}', pod_readiness='ignore')
Most custom resources set owner references, which Tilt can follow to determine
which pods belong to your resource. If your resource does not, you can use the
k8s_resource
function for each resource to specify a custom label selector for
that resource.
k8s_resource(new_name='postgres',
extra_pod_selectors=[{'kubedb.com/name': 'quick-postgres'}])
See the k8s_resource API docs for more detail, or look at this example that selects the Postgres pods created by the KubeDB Postgres operator.
Congratulations!
Once you’ve specified a resource name, an image location, and a pod selector for your custom resource, you can treat it like any other Kubernetes built-in.
Tilt will automatically fetch logs, and create any port-forwards you specify.
Installing Custom Resource Operators
To use a custom resource in a cluster, you’ll need to install the Kubernetes operator that reads the new object type.
Most frameworks have an installation guide to help you set it up.
Here are some options for standardizing that installation in a Tilt environment.
Using local()
or local_resource()
The local
Tiltfile function can run arbitrary shell
scripts. This is the easiest way to get started.
But local
is a blunt instrument - it will run everytime the Tiltfile reloads,
and it will block startup while it waits for the install to finish.
local('./install-my-crd-operator.sh')
If you want to parallelize it, you can use local_resource()
:
local_resource(
name='my-crd-operator',
cmd='./install-my-crd-operator.sh',
allow_parallel=True)
Then, use resource_deps
on your other resources to make
sure they wait on your local_resource
to finish.
k8s_resource(
name='custom-resource',
resource_deps=['my-crd-operator'])
Using k8s_custom_deploy()
The k8s_custom_deploy
Tiltfile function
uses a shell script to apply changes to a cluster.
The install shell script must print the result YAML to stdout
so that Tilt can track which objects the custom deployer has created.
k8s_custom_deploy(
name='my-crd-operator',
apply_cmd='./install-my-crd-operator.sh',
delete_cmd='./teardown-my-crd-operator.sh')
Use k8s_custom_deploy
if you want to monitor the health of your operator and
view its logs from the interface. See the
knative
extension for a working example.
Advanced Pod Creation
Many Kubernetes-based frameworks create pods that themselves create other pods!
How do you build images for these second-order pods?
We see two common techniques:
Custom Resource Injection
If the framework uses a custom resource, you can use the k8s_kind
function
described above to inject an image anywhere in the resource! It doesn’t have to
be a “normal” image field.
k8s_kind('UselessMachine', image_json_path='{.spec.imageToDeploy}')
Env Variable Injection
Another common strategy is to use environment variables.
In your Kuberentes container spec, add an env variable like:
env:
- name: IMAGE_TO_DEPLOY
value: my-image
In your Tiltfile, build the image, and tell Tilt to look in env variables for the image name.
docker_build('my-image', '.', match_in_env_vars=True)
Tilt will replace my-image:latest
with the content-based image reference that
Tilt built. (The custom_build guide has more
details on this.) Your
framework can then read the image reference and create pods with it.
The Airflow example uses this strategy with Airflow’s KubernetesPodOperator.
Custom Resource Examples
If you have an example of a custom resource with Tilt you’d like to share, feel free to add it to this page!
- KubeDB Postgres - Expose a Postgres server running on Kubernetes.
- ElasticSearch and Kibana - Expose a ElasticSearch cluster with a Kibana frontend running on Kubernetes.
- Airflow - Deploy an Airflow cluster and iterate on tasks. This doesn’t use custom resources but it demonstrate alternative ways of using and injecting images.
- Prometheus - Deploy the Prometheus operator and an instance of Prometheus. Demonstrates how to use resource dependencies and pod selectors to install the CRDs in the right order, and monitor the pods created by those CRDs.
- Knative -
Knative gives you CRDs to describe scale-to-zero services with DNS set up for
you. The Knative extension has Tiltfile functions to help set it up. Install
the Knative operator with
knative_install
, and inject images into Knative Services withknative_yaml
.