Custom images

This document will help you get a CAPZ Kubernetes cluster up and running with your custom image.

Reference images

An image defines the operating system and Kubernetes components that will populate the disk of each node in your cluster.

By default, images offered by "capi" in the Azure Marketplace are used.

You can list these reference images with this command:

az vm image list --publisher cncf-upstream --offer capi --all -o table

It is recommended to use the latest patch release of Kubernetes for a supported minor release.

Building a custom image

Cluster API uses the Kubernetes Image Builder tools. You should use the Azure images from that project as a starting point for your custom image.

The Image Builder Book explains how to build the images defined in that repository, with instructions for Azure CAPI Images in particular.

Operating system requirements

For your custom image to work with Cluster API, it must meet the operating system requirements of the bootstrap provider. For example, the default kubeadm bootstrap provider has a set of preflight checks that a VM is expected to pass before it can join the cluster.

Kubernetes version requirements

The reference images are each built to support a specific version of Kubernetes. When using your custom images based on them, take care to match the image to the version: field of the KubeadmControlPlane and MachineDeployment in the YAML template for your workload cluster.

To upgrade to a new Kubernetes release with custom images requires this preparation:

  • create a new custom image which supports the Kubernetes release version
  • copy the existing AzureMachineTemplate and change its image: section to reference the new custom image
  • create the new AzureMachineTemplate on the management cluster
  • modify the existing KubeadmControlPlane and MachineDeployment to reference the new AzureMachineTemplate and update the version: field to match

See Upgrading clusters for more details.

Creating a cluster from a custom image

To use a custom image, it needs to be referenced in an image: section of your AzureMachineTemplate. See below for more specific examples.

To use an image from the Azure Compute Gallery, previously known as Shared Image Gallery (SIG), fill in the resourceGroup, name, subscriptionID, gallery, and version fields:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureMachineTemplate
metadata:
  name: capz-compute-gallery-example
spec:
  template:
    spec:
      image:
        computeGallery:
          resourceGroup: "cluster-api-images"
          name: "capi-1234567890"
          subscriptionID: "01234567-89ab-cdef-0123-4567890abcde"
          gallery: "ClusterAPI"
          version: "0.3.1234567890"

If you build Azure CAPI images with the make targets in Image Builder, these required values are printed after a successful build. For example:

$ make -C images/capi/ build-azure-sig-ubuntu-1804
# many minutes later...
==> sig-ubuntu-1804:
Build 'sig-ubuntu-1804' finished.

==> Builds finished. The artifacts of successful builds are:
--> sig-ubuntu-1804: Azure.ResourceManagement.VMImage:

OSType: Linux
ManagedImageResourceGroupName: cluster-api-images
ManagedImageName: capi-1234567890
ManagedImageId: /subscriptions/01234567-89ab-cdef-0123-4567890abcde/resourceGroups/cluster-api-images/providers/Microsoft.Compute/images/capi-1234567890
ManagedImageLocation: southcentralus
ManagedImageSharedImageGalleryId: /subscriptions/01234567-89ab-cdef-0123-4567890abcde/resourceGroups/cluster-api-images/providers/Microsoft.Compute/galleries/ClusterAPI/images/capi-ubuntu-1804/versions/0.3.1234567890

Please also see the replication recommendations for the Azure Compute Gallery.

If the image you want to use is based on an image released by a third party publisher such as for example Flatcar Linux by Kinvolk, then you need to specify the publisher, offer, and sku fields as well:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureMachineTemplate
metadata:
  name: capz-compute-gallery-example
spec:
  template:
    spec:
      image:
        computeGallery:
          resourceGroup: "cluster-api-images"
          name: "capi-1234567890"
          subscriptionID: "01234567-89ab-cdef-0123-4567890abcde"
          gallery: "ClusterAPI"
          version: "0.3.1234567890"
          plan:
            publisher: "kinvolk"
            offer: "flatcar-container-linux-free"
            sku: "stable"

This will make API calls to create Virtual Machines or Virtual Machine Scale Sets to have the Plan correctly set.

Using image ID

To use a managed image resource by ID, only the id field must be set:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureMachineTemplate
metadata:
  name: capz-image-id-example
spec:
  template:
    spec:
      image:
        id: "/subscriptions/01234567-89ab-cdef-0123-4567890abcde/resourceGroups/myResourceGroup/providers/Microsoft.Compute/images/myImage"

A managed image resource can be created from a Virtual Machine. Please refer to Azure documentation on creating a managed image for more detail.

Managed images support only 20 simultaneous deployments, so for most use cases Azure Compute Gallery is recommended.

Using Azure Marketplace

To use an image from Azure Marketplace, populate the publisher, offer, sku, and version fields and, if this image is published by a third party publisher, set the thirdPartyImage flag to true so an image Plan can be generated for it. In the case of a third party image, you must accept the license terms with the Azure CLI before consuming it.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureMachineTemplate
metadata:
  name: capz-marketplace-example
spec:
  template:
    spec:
      image:
        marketplace:
          publisher: "example-publisher"
          offer: "example-offer"
          sku: "k8s-1dot18dot8-ubuntu-1804"
          version: "2020-07-25"
          thirdPartyImage: true

To use an image from Azure Community Gallery, set name field to gallery's public name and don't set subscriptionID and resourceGroup fields:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureMachineTemplate
metadata:
  name: capz-community-gallery-example
spec:
  template:
    spec:
      image:
        computeGallery:
          gallery: testGallery-3282f15c-906a-4c4b-b206-eb3c51adb5be
          name: capi-flatcar-stable-3139.2.0
          version: 0.3.1651499183

If the image you want to use is based on an image released by a third party publisher such as for example Flatcar Linux by Kinvolk, then you need to specify the publisher, offer, and sku fields as well:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureMachineTemplate
metadata:
  name: capz-community-gallery-example
spec:
  template:
    spec:
      image:
        computeGallery:
          gallery: testGallery-3282f15c-906a-4c4b-b206-eb3c51adb5be
          name: capi-flatcar-stable-3139.2.0
          version: 0.3.1651499183
          plan:
            publisher: kinvolk
            offer: flatcar-container-linux-free
            sku: stable

This will make API calls to create Virtual Machines or Virtual Machine Scale Sets to have the Plan correctly set.

In the case of a third party image, you must accept the license terms with the Azure CLI before consuming it.

Example: CAPZ with Mariner Linux

To clarify how to use a custom image, let's look at an example of using Mariner Linux with CAPZ.

Mariner is a minimal, open source Linux distribution, optimized for Azure. The image-builder project has support for building Mariner images.

Build Mariner with image-builder

Populate an az-creds.env file with your Azure credentials:

AZURE_SUBSCRIPTION_ID=xxxxxxx
AZURE_TENANT_ID=xxxxxxx
AZURE_CLIENT_ID=xxxxxxxx
AZURE_CLIENT_SECRET=xxxxxx

Then run image-builder, referencing those credentials as an environment file:

docker run -it --rm --env-file azure-creds.env registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.17 build-azure-sig-mariner-2

The entrypoint to this docker image is make. (You can clone the image-builder repository and run make -C images/capi build-azure-sig-mariner-2 locally if you prefer.)

This makefile target creates an Azure resource group called "cluster-api-images" in southcentralus by default. When it finishes, it will contain an Azure Compute Gallery with a Mariner image.

# skipping output to show just the end of the build...
==> azure-arm.sig-mariner-2: Resource group has been deleted.
==> azure-arm.sig-mariner-2: Running post-processor: manifest
Build 'azure-arm.sig-mariner-2' finished after 18 minutes 2 seconds.

==> Wait completed after 18 minutes 2 seconds

==> Builds finished. The artifacts of successful builds are:
--> azure-arm.sig-mariner-2: Azure.ResourceManagement.VMImage:

OSType: Linux
ManagedImageResourceGroupName: cluster-api-images
ManagedImageName: capi-mariner-2-1689801407
ManagedImageId: /subscriptions/xxxxxxx-xxxx-xxx-xxx/resourceGroups/cluster-api-images/providers/Microsoft.Compute/images/capi-mariner-2-1689801407
ManagedImageLocation: southcentralus
ManagedImageSharedImageGalleryId: /subscriptions/xxxxxxx-xxxx-xxx-xxx/resourceGroups/cluster-api-images/providers/Microsoft.Compute/galleries/ClusterAPI1689801353abcd/images/capi-mariner-2/versions/0.3.1689801407
SharedImageGalleryResourceGroup: cluster-api-images
SharedImageGalleryName: ClusterAPI1689801353abcd
SharedImageGalleryImageName: capi-mariner-2
SharedImageGalleryImageVersion: 0.3.1689801407
SharedImageGalleryReplicatedRegions: southcentralus

Add the Mariner image to a CAPZ cluster template

Edit your cluster template to add image fields to any AzureMachineTemplates:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureMachineTemplate
metadata:
  name: ${CLUSTER_NAME}-control-plane
  namespace: default
spec:
  template:
    spec:
      image:
        computeGallery:
          resourceGroup: cluster-api-images
          name: capi-mariner-2
          subscriptionID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
          gallery: ClusterAPI1689801353abcd
          version: "0.3.1689801407"

The last four fields are the SharedImageGalleryImageName, your Azure subscription ID, the SharedImageGalleryName, and the SharedImageGalleryImageVersion from the final output of the image-builder command above. Make sure to add this image section to both the control plane and worker node AzureMachineTemplates.

Deploy a Mariner cluster

Since our Compute Gallery image lives in southcentralus, our cluster should too. Set AZURE_LOCATION=southcentralus in your environment or in your template.

Now you can deploy your CAPZ Mariner cluster as usual with kubectl apply -f or other means.

Mariner stores CA certificates in an uncommon location, so we need to tell cloud-provider-azure's Helm chart where. Add this argument to the helm command you use to install cloud-provider-azure:

--set-string cloudControllerManager.caCertDir=/etc/pki/tls

That's it! You should now have a CAPZ cluster running Mariner Linux.