Configuring the Kubernetes Cloud Provider for Azure

The Azure cloud provider has a number of configuration options driven by a file on cluster nodes. This file canonically lives on a node at /etc/kubernetes/azure.json. The Azure cloud provider documentation details the configuration options exposed by this file.

CAPZ automatically generates this file based on user-provided values in AzureMachineTemplate and AzureMachine. All AzureMachines in the same MachineDeployment or control plane will all share a single cloud provider secret, while AzureMachines created inidividually will have their own secret.

For AzureMachineTemplate and standalone AzureMachines, the generated secret will have the name “${RESOURCE}-azure-json”, where “${RESOURCE}” is the name of either the AzureMachineTemplate or AzureMachine. The secret will have two data fields: control-plane-azure.json and worker-node-azure.json, with the raw content for that file containing the control plane and worker node data respectively. When the secret ${RESOURCE}-azure-json already exists in the same namespace as an AzureCluster and does not have the label "${CLUSTER_NAME}": "owned", CAPZ will not generate the default described above. Instead it will directly use whatever the user provides in that secret.

Overriding Cloud Provider Config

While many of the cloud provider config values are inferred from the capz infrastructure spec, there are other configuration parameters that cannot be inferred, and hence default to the values set by the azure cloud provider. In order to provider custom values to such configuration options through capz, you must use the spec.cloudProviderConfigOverrides in AzureCluster. The following example overrides the load balancer rate limit configuration:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureCluster
metadata:
  name: ${CLUSTER_NAME}
  namespace: default
spec:
  location: eastus
  networkSpec:
    vnet:
      name: ${CLUSTER_NAME}-vnet
  resourceGroup: cherry
  subscriptionID: ${AZURE_SUBSCRIPTION_ID}
  cloudProviderConfigOverrides:
    rateLimits:
      - name: "defaultRateLimit"
        config:
          cloudProviderRateLimit: true
          cloudProviderRateLimitBucket: 1
          cloudProviderRateLimitBucketWrite: 1
          cloudProviderRateLimitQPS: 1,
          cloudProviderRateLimitQPSWrite: 1,
      - name: "loadBalancerRateLimit"
        config:
          cloudProviderRateLimit: true
          cloudProviderRateLimitBucket: 2,
          CloudProviderRateLimitBucketWrite: 2,
          cloudProviderRateLimitQPS: 0,
          CloudProviderRateLimitQPSWrite: 0

External Cloud Provider

To deploy a cluster using external cloud provider, create a cluster configuration with the external cloud provider template.

After the cluster has provisioned, install the cloud-provider-azure components using the official helm chart:

helm install --repo https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/helm/repo cloud-provider-azure --generate-name --set infra.clusterName=${CLUSTER_NAME}

The Helm chart will pick the right version of cloud-controller-manager and cloud-node-manager to work with the version of Kubernetes your cluster is running.

After running helm install, you should eventually see a set of pods like these in a Running state:

kube-system   cloud-controller-manager                                            1/1     Running   0          41s
kube-system   cloud-node-manager-5pklx                                            1/1     Running   0          26s
kube-system   cloud-node-manager-hbbqt                                            1/1     Running   0          30s
kube-system   cloud-node-manager-mfsdg                                            1/1     Running   0          39s
kube-system   cloud-node-manager-qrz74                                            1/1     Running   0          24s

For more information see the official cloud-provider-azure helm chart documentation.

If you’re not familiar with using Helm to manage Kubernetes applications as packages, there’s lots of good Helm documentation on the official website.

The Cluster API Provider Azure Book

Configuring the Kubernetes Cloud Provider for Azure

Overriding Cloud Provider Config

External Cloud Provider

Storage Drivers

Azure File CSI Driver

Azure Disk CSI Driver