ASO Managed Clusters (AKS)

  • Feature status: alpha, not experimental, fully supported
  • Feature gate: MachinePool=true

New in CAPZ v1.15.0 is a new flavor of APIs that addresses the following limitations of the existing CAPZ APIs for advanced use cases for provisioning AKS clusters:

  • A limited set of Azure resource types can be represented.
  • A limited set of Azure resource topologies can be expressed. e.g. Only a single Virtual Network resource can be reconciled for each CAPZ-managed AKS cluster.
  • For each Azure resource type supported by CAPZ, CAPZ generally only uses a single Azure API version to define resources of that type.
  • For each Azure API version known by CAPZ, only a subset of fields defined in that version by the Azure API spec are exposed by the CAPZ API.

This new API defines new AzureASOManagedCluster, AzureASOManagedControlPlane, and AzureASOManagedMachinePool resources. An AzureASOManagedCluster might look like this:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: AzureASOManagedCluster
metadata:
  name: my-cluster
  namespace: default
spec:
  resources:
  - apiVersion: resources.azure.com/v1api20200601
    kind: ResourceGroup
    metadata:
      name: my-resource-group
    spec:
      location: eastus

See here for a full AKS example using all the new resources.

The main element of the new API is spec.resources in each new resource, which defines arbitrary, literal ASO resources inline to be managed by CAPZ. These inline ASO resource definitions take the place of almost all other configuration currently defined by CAPZ. e.g. Instead of a CAPZ-specific spec.location field on the existing AzureManagedControlPlane, the same value would be expected to be set on an ASO ManagedCluster resource defined in an AzureASOManagedControlPlane's spec.resources. This pattern allows users to define, in full, any ASO-supported version of a resource type in any of these new CAPZ resources.

The obvious tradeoff with this new style of API is that CAPZ resource definitions can become more verbose for basic use cases. To address this, CAPZ still offers flavor templates that use this API with all of the boilerplate predefined to serve as a starting point for customization.

The overall theme of this API is to leverage ASO as much as possible for representing Azure resources in the Kubernetes API, thereby making CAPZ the thinnest possible translation layer between ASO and Cluster API.

This experiment will help inform CAPZ whether this pattern may be a candidate for a potential v2 API. This functionality is enabled by default and can be disabled with the ASOAPI feature flag (set by the EXP_ASO_API environment variable). Please try it out and offer any feedback!

Disable Local Accounts

When local accounts are disabled, like for AKS Automatic clusters, the kubeconfig generated by AKS assumes clients have access to the kubelogin utility locally to authenticate with Entra. This is not the case for clients like the Cluster API controllers which need to access Nodes in the workload cluster. To allow those controllers access, CAPZ will augment the kubeconfig from AKS to remove the exec plugin and add a token which is an Entra ID access token that clients can handle natively by passing as an Authorization: Bearer ... token. CAPZ authenticates with Entra using the same ASO credentials used to create the ManagedCluster resource, which might be any of the options described in ASO's documentation and must be assigned the Azure Kubernetes Service RBAC Cluster Admin Role.

When defining the embedded ManagedCluster in an AzureASOManagedControlPlane, ASO will fail to retrieve adminCredentials when local accounts are disabled, so userCredentials must be specified instead. In order to leave room for CAPZ to manage the canonical ${CLUSTER_NAME}-kubeconfig secret well-known to Cluster API, another name must be specified for this Secret to avoid CAPZ and ASO overwriting each other:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: AzureASOManagedControlPlane
metadata:
  name: ${CLUSTER_NAME}
spec:
  resources:
  - apiVersion: containerservice.azure.com/v1api20231001
    kind: ManagedCluster
    metadata:
      name: ${CLUSTER_NAME}
    spec:
      operatorSpec:
        secrets:
          userCredentials:
            name: ${CLUSTER_NAME}-user-kubeconfig # NOT ${CLUSTER_NAME}-kubeconfig
            key: value

Migrating existing Clusters to AzureASOManagedControlPlane

Existing CAPI Clusters using the AzureManagedControlPlane and associated APIs can be migrated to use the new AzureASOManagedControlPlane and its associated APIs. This process relies on CAPZ's ability to adopt existing clusters that may not have been created by CAPZ, which comes with some caveats that should be reviewed first.

To migrate one cluster to the ASO-based APIs:

  1. Pause the cluster by setting the Cluster's spec.paused to true.
  2. Wait for the cluster to be paused by waiting for the absence of the clusterctl.cluster.x-k8s.io/block-move annotation on the AzureManagedControlPlane and its AzureManagedMachinePools. This should be fairly instantaneous.
  3. Create a new namespace to contain the new resources to avoid conflicting ASO definitions.
  4. Adopt the underlying AKS resources from the new namespace, which creates the new CAPI and CAPZ resources.
  5. Forcefully delete the old Cluster. This is more complicated than normal because CAPI controllers do not reconcile paused resources at all, even when they are deleted. The underlying Azure resources will not be affected.
    • Delete the cluster: kubectl delete cluster <name> --wait=false
    • Delete the cluster infrastructure object: kubectl delete azuremanagedcluster <name> --wait=false
    • Delete the cluster control plane object: kubectl delete azuremanagedcontrolplane <name> --wait=false
    • Delete the machine pools: kubectl delete machinepool <names...> --wait=false
    • Delete the machine pool infrastructure resources: kubectl delete azuremanagedmachinepool <names...> --wait=false
    • Remove finalizers from the machine pool infrastructure resources: kubectl patch azuremanagedmachinepool <names...> --type merge -p '{"metadata": {"finalizers": null}}'
    • Remove finalizers from the machine pools: kubectl patch machinepool <names...> --type merge -p '{"metadata": {"finalizers": null}}'
    • Remove finalizers from the cluster control plane object: kubectl patch azuremanagedcontrolplane <name> --type merge -p '{"metadata": {"finalizers": null}}'
    • Note: the cluster infrastructure object should not have any finalizers and should already be deleted
    • Remove finalizers from the cluster: kubectl patch cluster <name> --type merge -p '{"metadata": {"finalizers": null}}'
    • Verify the old ASO resources like ResourceGroup and ManagedCluster managed by the old Cluster are deleted.