The Cluster API Provider Azure Book

Managed - Whether to enable managed AAD.

AdminGroupObjectIDs - AAD group object IDs that will have admin role of the cluster.

Tier - Tier of an AKS cluster.

AuthorizedIPRanges - Authorized IP Ranges to kubernetes API server.

EnablePrivateCluster - Whether to create the cluster as a private cluster or not.

PrivateDNSZone - Private dns zone mode for private cluster.

EnablePrivateClusterPublicFQDN - Whether to create additional public FQDN for private cluster or not.

UltraSSDEnabled enables or disables Azure UltraSSD capability for the virtual machine. Defaults to true if Ultra SSD data disks are specified, otherwise it doesn’t set the capability on the VM.

Name - The name of the managed cluster add-on.

Config - Key-value pairs for configuring the add-on.

Enabled - Whether the add-on is enabled or not.

A nil or empty list indicates that AzureCluster cannot use the identity from any namespace.

Selector is a selector of namespaces that AzureCluster can use this Identity from. This is a standard Kubernetes LabelSelector, a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed.

A nil or empty selector indicates that AzureCluster cannot use this AzureClusterIdentity from any namespace.

BalanceSimilarNodeGroups - Valid values are ‘true’ and ‘false’. The default is false.

Expander - If not specified, the default is ‘random’. See expanders for more information.

MaxEmptyBulkDelete - The default is 10.

MaxGracefulTerminationSec - The default is 600.

MaxNodeProvisionTime - The default is ‘15m’. Values must be an integer followed by an ’m’. No unit of time other than minutes (m) is supported.

MaxTotalUnreadyPercentage - The default is 45. The maximum is 100 and the minimum is 0.

NewPodScaleUpDelay - For scenarios like burst/batch scale where you don’t want CA to act before the kubernetes scheduler could schedule all the pods, you can tell CA to ignore unscheduled pods before they’re a certain age. The default is ‘0s’. Values must be an integer followed by a unit (’s’ for seconds, ’m’ for minutes, ‘h’ for hours, etc).

OkTotalUnreadyCount - This must be an integer. The default is 3.

ScanInterval - How often cluster is reevaluated for scale up or down. The default is ‘10s’.

ScaleDownDelayAfterAdd - The default is ‘10m’. Values must be an integer followed by an ’m’. No unit of time other than minutes (m) is supported.

ScaleDownDelayAfterDelete - The default is the scan-interval. Values must be an integer followed by an ’s’. No unit of time other than seconds (s) is supported.

ScaleDownDelayAfterFailure - The default is ‘3m’. Values must be an integer followed by an ’m’. No unit of time other than minutes (m) is supported.

ScaleDownUnneededTime - The default is ‘10m’. Values must be an integer followed by an ’m’. No unit of time other than minutes (m) is supported.

ScaleDownUnreadyTime - The default is ‘20m’. Values must be an integer followed by an ’m’. No unit of time other than minutes (m) is supported.

ScaleDownUtilizationThreshold - The default is ‘0.5’.

SkipNodesWithLocalStorage - The default is false.

SkipNodesWithSystemPods - The default is true.

BastionHostSkuName configures the tier of the Azure Bastion Host. Can be either Basic or Standard. Defaults to Basic.

EnableTunneling enables the native client support feature for the Azure Bastion Host. Defaults to false.

ExtendedLocation is an optional set of ExtendedLocation properties for clusters on Azure public MEC.

AdditionalTags is an optional set of tags to add to Azure resources managed by the Azure provider, in addition to the ones added by default.

IdentityRef is a reference to an AzureIdentity to be used when reconciling this cluster

AzureEnvironment is the name of the AzureCloud to be used. The default value that would be used by most users is “AzurePublicCloud”, other values are: - ChinaCloud: “AzureChinaCloud” - GermanCloud: “AzureGermanCloud” - PublicCloud: “AzurePublicCloud” - USGovernmentCloud: “AzureUSGovernmentCloud”

CloudProviderConfigOverrides is an optional set of configuration values that can be overridden in azure cloud provider config. This is only a subset of options that are available in azure cloud provider config. Some values for the cloud provider config are inferred from other parts of cluster api provider azure spec, and may not be available for overrides. See: https://cloud-provider-azure.sigs.k8s.io/install/configs Note: All cloud provider config values can be customized by creating the secret beforehand. CloudProviderConfigOverrides is only used when the secret is managed by the Azure Provider.

Type is the type of Azure Identity used. ServicePrincipal, ServicePrincipalCertificate, UserAssignedMSI, ManualServicePrincipal or WorkloadIdentity.

ResourceID is the Azure resource ID for the User Assigned MSI resource. Only applicable when type is UserAssignedMSI.

ClientID is the service principal client ID. Both User Assigned MSI and SP can use this field.

ClientSecret is a secret reference which should contain either a Service Principal password or certificate secret.

TenantID is the service principal primary tenant id.

AllowedNamespaces is used to identify the namespaces the clusters are allowed to use the identity from. Namespaces can be selected either using an array of namespaces or with label selector. An empty allowedNamespaces object indicates that AzureClusters can use this identity from any namespace. If this object is nil, no namespaces will be allowed (default behaviour, if this field is not provided) A namespace should be either in the NamespaceList or match with Selector to use the identity.

Conditions defines current service state of the AzureClusterIdentity.

(Members of AzureClusterClassSpec are embedded into this type.)

NetworkSpec encapsulates all things related to Azure network.

BastionSpec encapsulates all things related to the Bastions in the cluster.

ControlPlaneEndpoint represents the endpoint used to communicate with the control plane. It is not recommended to set this when creating an AzureCluster as CAPZ will set this for you. However, if it is set, CAPZ will not change it.

FailureDomains specifies the list of unique failure domains for the location/region of the cluster. A FailureDomain maps to Availability Zone with an Azure Region (if the region support them). An Availability Zone is a separate data center within a region and they can be used to ensure the cluster is more resilient to failure. See: https://learn.microsoft.com/azure/reliability/availability-zones-overview This list will be used by Cluster API to try and spread the machines across the failure domains.

Ready is true when the provider resource is ready.

Conditions defines current service state of the AzureCluster.

LongRunningOperationStates saves the states for Azure long-running operations so they can be continued on the next reconciliation loop.

(Members of AzureClusterClassSpec are embedded into this type.)

NetworkSpec encapsulates all things related to Azure network.

BastionSpec encapsulates all things related to the Bastions in the cluster.

Gallery specifies the name of the compute image gallery that contains the image

Name is the name of the image

Version specifies the version of the marketplace image. The allowed formats are Major.Minor.Build or ‘latest’. Major, Minor, and Build are decimal numbers. Specify ‘latest’ to use the latest version of an image available at deploy time. Even if you use ‘latest’, the VM image will not automatically update after deploy time even if a new version becomes available.

SubscriptionID is the identifier of the subscription that contains the private compute gallery.

ResourceGroup specifies the resource group containing the private compute gallery.

Plan contains plan information.

ProviderID is the unique identifier as specified by the cloud provider.

FailureDomain is the failure domain unique identifier this Machine should be attached to, as defined in Cluster API. This relates to an Azure Availability Zone

Image is used to provide details of an image to use during VM creation. If image details are omitted the image will default the Azure Marketplace “capi” offer, which is based on Ubuntu.

Identity is the type of identity used for the virtual machine. The type ‘SystemAssigned’ is an implicitly created identity. The generated identity will be assigned a Subscription contributor role. The type ‘UserAssigned’ is a standalone Azure resource provided by the user and assigned to the VM

UserAssignedIdentities is a list of standalone Azure identities provided by the user The lifecycle of a user-assigned identity is managed separately from the lifecycle of the AzureMachine. See https://learn.microsoft.com/azure/active-directory/managed-identities-azure-resources/how-to-manage-ua-identity-cli

SystemAssignedIdentityRole defines the role and scope to assign to the system-assigned identity.

Deprecated: RoleAssignmentName should be set in the systemAssignedIdentityRole field.

OSDisk specifies the parameters for the operating system disk of the machine

DataDisk specifies the parameters that are used to add one or more data disks to the machine

SSHPublicKey is the SSH public key string, base64-encoded to add to a Virtual Machine. Linux only. Refer to documentation on how to set up SSH access on Windows instances.

AdditionalTags is an optional set of tags to add to an instance, in addition to the ones added by default by the Azure provider. If both the AzureCluster and the AzureMachine specify the same tag name with different values, the AzureMachine’s value takes precedence.

AdditionalCapabilities specifies additional capabilities enabled or disabled on the virtual machine.

AllocatePublicIP allows the ability to create dynamic public ips for machines where this value is true.

EnableIPForwarding enables IP Forwarding in Azure which is required for some CNI’s to send traffic from a pods on one machine to another. This is required for IpV6 with Calico in combination with User Defined Routes (set by the Azure Cloud Controller manager). Default is false for disabled.

Deprecated: AcceleratedNetworking should be set in the networkInterfaces field.

Diagnostics specifies the diagnostics settings for a virtual machine. If not specified then Boot diagnostics (Managed) will be enabled.

SpotVMOptions allows the ability to specify the Machine should use a Spot VM

SecurityProfile specifies the Security profile settings for a virtual machine.

Deprecated: SubnetName should be set in the networkInterfaces field.

DNSServers adds a list of DNS Server IP addresses to the VM NICs.

VMExtensions specifies a list of extensions to be added to the virtual machine.

NetworkInterfaces specifies a list of network interface configurations. If left unspecified, the VM will get a single network interface with a single IPConfig in the subnet specified in the cluster’s node subnet field. The primary interface will be the first networkInterface specified (index 0) in the list.

Ready is true when the provider resource is ready.

Addresses contains the Azure instance associated addresses.

VMState is the provisioning state of the Azure virtual machine.

ErrorReason will be set in the event that there is a terminal problem reconciling the Machine and will contain a succinct value suitable for machine interpretation.

This field should not be set for transitive errors that a controller faces that are expected to be fixed automatically over time (like service outages), but instead indicate that something is fundamentally wrong with the Machine’s spec or the configuration of the controller, and that manual intervention is required. Examples of terminal errors would be invalid combinations of settings in the spec, values that are unsupported by the controller, or the responsible controller itself being critically misconfigured.

Any transient errors that occur during the reconciliation of Machines can be added as events to the Machine object and/or logged in the controller’s output.

ErrorMessage will be set in the event that there is a terminal problem reconciling the Machine and will contain a more verbose string suitable for logging and human consumption.

This field should not be set for transitive errors that a controller faces that are expected to be fixed automatically over time (like service outages), but instead indicate that something is fundamentally wrong with the Machine’s spec or the configuration of the controller, and that manual intervention is required. Examples of terminal errors would be invalid combinations of settings in the spec, values that are unsupported by the controller, or the responsible controller itself being critically misconfigured.

Any transient errors that occur during the reconciliation of Machines can be added as events to the Machine object and/or logged in the controller’s output.

Conditions defines current service state of the AzureMachine.

LongRunningOperationStates saves the states for Azure long-running operations so they can be continued on the next reconciliation loop.

Spec is the specification of the desired behavior of the machine.

ControlPlaneEndpoint represents the endpoint used to communicate with the control plane. Immutable, populated by the AKS API at create.

Ready is true when the provider resource is ready.

"Free"

FreeManagedControlPlaneTier is the free tier of AKS without corresponding SLAs.

"Paid"

PaidManagedControlPlaneTier is the paid tier of AKS with corresponding SLAs. Deprecated. It has been replaced with StandardManagedControlPlaneTier.

"Standard"

StandardManagedControlPlaneTier is the standard tier of AKS with corresponding SLAs.

Version defines the desired Kubernetes version.

ResourceGroupName is the name of the Azure resource group for this AKS Cluster. Immutable.

NodeResourceGroupName is the name of the resource group containing cluster IaaS resources. Will be populated to default in webhook. Immutable.

VirtualNetwork describes the vnet for the AKS cluster. Will be created if it does not exist. Immutable except for subnet.

SubscriptionID is the GUID of the Azure subscription to hold this cluster. Immutable.

Location is a string matching one of the canonical Azure region names. Examples: “westus2”, “eastus”. Immutable.

ControlPlaneEndpoint represents the endpoint used to communicate with the control plane. Immutable, populated by the AKS API at create.

AdditionalTags is an optional set of tags to add to Azure resources managed by the Azure provider, in addition to the ones added by default.

NetworkPlugin used for building Kubernetes network. Allowed values are “azure”, “kubenet”. Immutable.

NetworkPluginMode is the mode the network plugin should use. Allowed value is “overlay”.

NetworkPolicy used for building Kubernetes network. Allowed values are “azure”, “calico”. Immutable.

Outbound configuration used by Nodes. Immutable.

SSHPublicKey is a string literal containing an ssh public key base64 encoded. Use empty string to autogenerate new key. Use null value to not set key. Immutable.

DNSServiceIP is an IP address assigned to the Kubernetes DNS service. It must be within the Kubernetes service address range specified in serviceCidr. Immutable.

LoadBalancerSKU is the SKU of the loadBalancer to be provisioned. Immutable.

IdentityRef is a reference to a AzureClusterIdentity to be used when reconciling this cluster

AadProfile is Azure Active Directory configuration to integrate with AKS for aad authentication.

AddonProfiles are the profiles of managed cluster add-on.

SKU is the SKU of the AKS to be provisioned.

LoadBalancerProfile is the profile of the cluster load balancer.

APIServerAccessProfile is the access profile for AKS API server. Immutable except for authorizedIPRanges.

AutoscalerProfile is the parameters to be applied to the cluster-autoscaler when enabled

AzureEnvironment is the name of the AzureCloud to be used. The default value that would be used by most users is “AzurePublicCloud”, other values are: - ChinaCloud: “AzureChinaCloud” - PublicCloud: “AzurePublicCloud” - USGovernmentCloud: “AzureUSGovernmentCloud”

Identity configuration used by the AKS control plane.

KubeletUserAssignedIdentity is the user-assigned identity for kubelet. For authentication with Azure Container Registry.

HTTPProxyConfig is the HTTP proxy configuration for the cluster. Immutable.

OIDCIssuerProfile is the OIDC issuer profile of the Managed Cluster.

Ready is true when the provider resource is ready.

Initialized is true when the control plane is available for initial contact. This may occur before the control plane is fully ready. In the AzureManagedControlPlane implementation, these are identical.

Conditions defines current service state of the AzureManagedControlPlane.

LongRunningOperationStates saves the states for Azure long-running operations so they can be continued on the next reconciliation loop.

OIDCIssuerProfile is the OIDC issuer profile of the Managed Cluster.

AdditionalTags is an optional set of tags to add to Azure resources managed by the Azure provider, in addition to the ones added by default.

Name - name of the agent pool. If not specified, CAPZ uses the name of the CR as the agent pool name. Immutable.

Mode - represents mode of an agent pool. Possible values include: System, User.

SKU is the size of the VMs in the node pool. Immutable.

OSDiskSizeGB is the disk size for every machine in this agent pool. If you specify 0, it will apply the default osDisk size according to the vmSize specified. Immutable.

AvailabilityZones - Availability zones for nodes. Must use VirtualMachineScaleSets AgentPoolType. Immutable.

Node labels - labels for all of the nodes present in node pool. See also AKS doc.

Taints specifies the taints for nodes present in this agent pool. See also AKS doc.

ProviderIDList is the unique identifier as specified by the cloud provider.

Scaling specifies the autoscaling parameters for the node pool.

MaxPods specifies the kubelet --max-pods configuration for the node pool. Immutable. See also AKS doc, K8s doc.

OsDiskType specifies the OS disk type for each node in the pool. Allowed values are ‘Ephemeral’ and ‘Managed’ (default). Immutable. See also AKS doc.

EnableUltraSSD enables the storage type UltraSSD_LRS for the agent pool. Immutable.

OSType specifies the virtual machine operating system. Default to Linux. Possible values include: ‘Linux’, ‘Windows’. ‘Windows’ requires the AzureManagedControlPlane’s spec.networkPlugin to be azure. Immutable. See also AKS doc.

EnableNodePublicIP controls whether or not nodes in the pool each have a public IP address. Immutable.

NodePublicIPPrefixID specifies the public IP prefix resource ID which VM nodes should use IPs from. Immutable.

ScaleSetPriority specifies the ScaleSetPriority value. Default to Regular. Possible values include: ‘Regular’, ‘Spot’ Immutable.

ScaleDownMode affects the cluster autoscaler behavior. Default to Delete. Possible values include: ‘Deallocate’, ‘Delete’

SpotMaxPrice defines max price to pay for spot instance. Possible values are any decimal value greater than zero or -1. If you set the max price to be -1, the VM won’t be evicted based on price. The price for the VM will be the current price for spot or the price for a standard VM, which ever is less, as long as there’s capacity and quota available.

KubeletConfig specifies the kubelet configurations for nodes. Immutable.

KubeletDiskType specifies the kubelet disk type. Default to OS. Possible values include: ‘OS’, ‘Temporary’. Requires Microsoft.ContainerService/KubeletDisk preview feature to be set. Immutable. See also AKS doc.

LinuxOSConfig specifies the custom Linux OS settings and configurations. Immutable.

SubnetName specifies the Subnet where the MachinePool will be placed Immutable.

EnableFIPS indicates whether FIPS is enabled on the node pool. Immutable.

Ready is true when the provider resource is ready.

Replicas is the most recently observed number of replicas.

Any transient errors that occur during the reconciliation of Machines can be added as events to the Machine object and/or logged in the controller’s output.

Any transient errors that occur during the reconciliation of Machines can be added as events to the Machine object and/or logged in the controller’s output.

Conditions defines current service state of the AzureManagedControlPlane.

LongRunningOperationStates saves the states for Azure long-running operations so they can be continued on the next reconciliation loop.

(Members of ImagePlan are embedded into this type.)

Version specifies the version of an image sku. The allowed formats are Major.Minor.Build or ‘latest’. Major, Minor, and Build are decimal numbers. Specify ‘latest’ to use the latest version of an image available at deploy time. Even if you use ‘latest’, the VM image will not automatically update after deploy time even if a new version becomes available.

ThirdPartyImage indicates the image is published by a third party publisher and a Plan will be generated for it.

SubscriptionID is the identifier of the subscription that contains the shared image gallery

ResourceGroup specifies the resource group containing the shared image gallery

Gallery specifies the name of the shared image gallery that contains the image

Name is the name of the image

Version specifies the version of the marketplace image. The allowed formats are Major.Minor.Build or ‘latest’. Major, Minor, and Build are decimal numbers. Specify ‘latest’ to use the latest version of an image available at deploy time. Even if you use ‘latest’, the VM image will not automatically update after deploy time even if a new version becomes available.

Publisher is the name of the organization that created the image. This value will be used to add a Plan in the API request when creating the VM/VMSS resource. This is needed when the source image from which this SIG image was built requires the Plan to be used.

Offer specifies the name of a group of related images created by the publisher. For example, UbuntuServer, WindowsServer This value will be used to add a Plan in the API request when creating the VM/VMSS resource. This is needed when the source image from which this SIG image was built requires the Plan to be used.

SKU specifies an instance of an offer, such as a major release of a distribution. For example, 18.04-LTS, 2019-Datacenter This value will be used to add a Plan in the API request when creating the VM/VMSS resource. This is needed when the source image from which this SIG image was built requires the Plan to be used.

Name specifies the name of backend pool for the load balancer. If not specified, the default name will be set, depending on the load balancer role.

"false"

BalanceSimilarNodeGroupsFalse …

"true"

BalanceSimilarNodeGroupsTrue …

"Basic"

BasicBastionHostSku SKU for the Azure Bastion Host.

"Standard"

StandardBastionHostSku SKU for the Azure Bastion Host.

StorageAccountType determines if the storage account for storing the diagnostics data should be disabled (Disabled), provisioned by Azure (Managed) or by the user (UserManaged).

UserManaged provides a reference to the user-managed storage account.

"Disabled"

DisabledDiagnosticsStorage is used to determine that the diagnostics storage account should be disabled.

"Managed"

ManagedDiagnosticsStorage is used to determine that the diagnostics storage account should be provisioned by Azure.

"UserManaged"

UserManagedDiagnosticsStorage is used to determine that the diagnostics storage account should be provisioned by the User.

Lifecycle determines the resource lifecycle.

ClusterName is the cluster associated with the resource.

ResourceID is the unique identifier of the resource to be tagged.

Name is the name of the resource, it’s applied as the tag “Name” on Azure.

Role is the role associated to the resource.

Any additional tags to be added to the resource.

"none"

CPUManagerPolicyNone …

"static"

CPUManagerPolicyStatic …

NameSuffix is the suffix to be appended to the machine name to generate the disk name. Each disk name will be in format _.

DiskSizeGB is the size in GB to assign to the data disk.

ManagedDisk specifies the Managed Disk parameters for the data disk.

Lun Specifies the logical unit number of the data disk. This value is used to identify data disks within the VM and therefore must be unique for each data disk attached to a VM. The value must be between 0 and 63.

CachingType specifies the caching requirements.

Boot configures the boot diagnostics settings for the virtual machine. This allows to configure capturing serial output from the virtual machine on boot. This is useful for debugging software based launch issues. If not specified then Boot diagnostics (Managed) will be enabled.

Option enables ephemeral OS when set to “Local” See https://learn.microsoft.com/azure/virtual-machines/ephemeral-os-disks for full details

ID defines resourceID for diskEncryptionSet resource. It must be in the same subscription

"least-waste"

ExpanderLeastWaste …

"most-pods"

ExpanderMostPods …

"priority"

ExpanderPriority …

"random"

ExpanderRandom …

Name defines the name for the extended location.

Type defines the type for the extended location.

(Members of FrontendIPClass are embedded into this type.)

Type describes the type of future, such as update, create, delete, etc.

ResourceGroup is the Azure resource group for the resource.

ServiceName is the name of the Azure service. Together with the name of the resource, this forms the unique identifier for the future.

Name is the name of the Azure resource. Together with the service name, this forms the unique identifier for the future.

Data is the base64 url encoded json Azure AutoRest Future.

HTTPProxy is the HTTP proxy server endpoint to use.

HTTPSProxy is the HTTPS proxy server endpoint to use.

NoProxy indicates the endpoints that should not go through proxy.

TrustedCA is the alternative CA cert to use for connecting to proxy servers.

Type specifies the IP tag type. Example: FirstPartyUsage.

Tag specifies the value of the IP tag associated with the public IP. Example: SQL.

Type - The Identity type to use.

UserAssignedIdentityResourceID - Identity ARM resource ID when using user-assigned identity.

"ManualServicePrincipal"

ManualServicePrincipal represents a manual service principal.

"ServicePrincipal"

ServicePrincipal represents a service principal using a client password as secret.

"ServicePrincipalCertificate"

ServicePrincipalCertificate represents a service principal using a certificate as secret.

"UserAssignedMSI"

UserAssignedMSI represents a user-assigned managed identity.

"WorkloadIdentity"

WorkloadIdentity represents a WorkloadIdentity.

ID specifies an image to use by ID

SharedGallery specifies an image to use from an Azure Shared Image Gallery Deprecated: use ComputeGallery instead.

Marketplace specifies an image to use from the Azure Marketplace

ComputeGallery specifies an image to use from the Azure Compute Gallery

Publisher is the name of the organization that created the image

Offer specifies the name of a group of related images created by the publisher. For example, UbuntuServer, WindowsServer

SKU specifies an instance of an offer, such as a major release of a distribution. For example, 18.04-LTS, 2019-Datacenter

CPUManagerPolicy - CPU Manager policy to use.

CPUCfsQuota - Enable CPU CFS quota enforcement for containers that specify CPU limits.

CPUCfsQuotaPeriod - Sets CPU CFS quota period value. Must end in “ms”, e.g. “100ms”

ImageGcHighThreshold - The percent of disk usage after which image garbage collection is always run. Valid values are 0-100 (inclusive).

ImageGcLowThreshold - The percent of disk usage before which image garbage collection is never run. Valid values are 0-100 (inclusive) and must be less than imageGcHighThreshold.

TopologyManagerPolicy - Topology Manager policy to use.

AllowedUnsafeSysctls - Allowlist of unsafe sysctls or unsafe sysctl patterns (ending in *). Valid values match kernel.shm*, kernel.msg*, kernel.sem, fs.mqueue.*, or net.*.

FailSwapOn - If set to true it will make the Kubelet fail to start if swap is enabled on the node.

ContainerLogMaxSizeMB - The maximum size in MB of a container log file before it is rotated.

ContainerLogMaxFiles - The maximum number of container log files that can be present for a container. The number must be ≥ 2.

PodMaxPids - The maximum number of processes per pod. Must not exceed kernel PID limit. -1 disables the limit.

"OS"

KubeletDiskTypeOS …

"Temporary"

KubeletDiskTypeTemporary …

"Internal"

Internal is the value for the Azure load balancer internal type.

"Public"

Public is the value for the Azure load balancer public type.

SwapFileSizeMB specifies size in MB of a swap file will be created on the agent nodes from this node pool. Max value of SwapFileSizeMB should be the size of temporary disk(/dev/sdb). Must be at least 1. See also AKS doc.

Sysctl specifies the settings for Linux agent nodes.

TransparentHugePageDefrag specifies whether the kernel should make aggressive use of memory compaction to make more hugepages available. See also [Linux doc].

[Linux doc]: https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html#admin-guide-transhuge for more details.

TransparentHugePageEnabled specifies various modes of Transparent Hugepages. See also [Linux doc].

[Linux doc]: https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html#admin-guide-transhuge for more details.

IdleTimeoutInMinutes specifies the timeout for the TCP idle connection.

ManagedOutboundIPs - Desired managed outbound IPs for the cluster load balancer.

OutboundIPPrefixes - Desired outbound IP Prefix resources for the cluster load balancer.

OutboundIPs - Desired outbound IP resources for the cluster load balancer.

AllocatedOutboundPorts - Desired number of allocated SNAT ports per VM. Allowed values must be in the range of 0 to 64000 (inclusive). The default value is 0 which results in Azure dynamically allocating ports.

IdleTimeoutInMinutes - Desired outbound flow idle timeout in minutes. Allowed values must be in the range of 4 to 120 (inclusive). The default value is 30 minutes.

ID is the Azure resource ID of the load balancer. READ-ONLY

FrontendIPsCount specifies the number of frontend IP addresses for the load balancer.

BackendPool describes the backend pool of the load balancer.

(Members of LoadBalancerClassSpec are embedded into this type.)

"SystemAssigned"

ManagedControlPlaneIdentityTypeSystemAssigned Use an implicitly created system-assigned managed identity to manage cluster resources. Components in the control plane such as kube-controller-manager will use the system-assigned managed identity to manipulate Azure resources.

"UserAssigned"

ManagedControlPlaneIdentityTypeUserAssigned Use a user-assigned identity to manage cluster resources. Components in the control plane such as kube-controller-manager will use the specified user-assigned managed identity to manipulate Azure resources.

"loadBalancer"

ManagedControlPlaneOutboundTypeLoadBalancer …

"managedNATGateway"

ManagedControlPlaneOutboundTypeManagedNATGateway …

"userAssignedNATGateway"

ManagedControlPlaneOutboundTypeUserAssignedNATGateway …

"userDefinedRouting"

ManagedControlPlaneOutboundTypeUserDefinedRouting …

ServiceEndpoints is a slice of Virtual Network service endpoints to enable for the subnets.

PrivateEndpoints is a slice of Virtual Network private endpoints to create for the subnets.

Immutable except for serviceEndpoints.

ResourceGroup is the name of the Azure resource group for the VNet and Subnet.

DiskEncryptionSet specifies the customer-managed disk encryption set resource id for the managed disk.

SecurityProfile specifies the security profile for the managed disk.

MinSize is the minimum number of nodes for auto-scaling.

MaxSize is the maximum number of nodes for auto-scaling.

ID is the Azure resource ID of the NAT gateway. READ-ONLY

(Members of NatGatewayClassSpec are embedded into this type.)

PrivateDNSZoneName defines the zone name for the Azure Private DNS.

SubnetName specifies the subnet in which the new network interface will be placed.

PrivateIPConfigs specifies the number of private IP addresses to attach to the interface. Defaults to 1 if not specified.

AcceleratedNetworking enables or disables Azure accelerated networking. If omitted, it will be set based on whether the requested VMSize supports accelerated networking. If AcceleratedNetworking is set to true with a VMSize that does not support it, Azure will return an error.

"overlay"

NetworkPluginModeOverlay is used with networkPlugin=azure, pods are given IPs from the PodCIDR address space but use Azure Routing Domains rather than Kubenet’s method of route tables. See also AKS doc.

Vnet is the configuration for the Azure virtual network.

Subnets is the configuration for the control-plane subnet and the node subnet.

APIServerLB is the configuration for the control-plane load balancer.

NodeOutboundLB is the configuration for the node outbound load balancer.

ControlPlaneOutboundLB is the configuration for the control-plane outbound load balancer. This is different from APIServerLB, and is used only in private clusters (optionally) for enabling outbound traffic.

(Members of NetworkClassSpec are embedded into this type.)

(Members of NetworkClassSpec are embedded into this type.)

Vnet is the configuration for the Azure virtual network.

Subnets is the configuration for the control-plane subnet and the node subnet.

APIServerLB is the configuration for the control-plane load balancer.

NodeOutboundLB is the configuration for the node outbound load balancer.

ControlPlaneOutboundLB is the configuration for the control-plane outbound load balancer. This is different from APIServerLB, and is used only in private clusters (optionally) for enabling outbound traffic.

"System"

NodePoolModeSystem represents mode system for azuremachinepool.

"User"

NodePoolModeUser represents mode user for azuremachinepool.

Enabled is whether the OIDC issuer is enabled.

IssuerURL is the OIDC issuer url of the Managed Cluster.

DiskSizeGB is the size in GB to assign to the OS disk. Will have a default of 30GB if not provided

ManagedDisk specifies the Managed Disk parameters for the OS disk.

CachingType specifies the caching requirements.

"Flexible"

FlexibleOrchestrationMode treats VMs as individual resources accessible by standard VM APIs.

"Uniform"

UniformOrchestrationMode treats VMs as identical instances accessible by the VMSS VM API.

Name specifies the name of the private endpoint.

Location specifies the region to create the private endpoint.

PrivateLinkServiceConnections specifies Private Link Service Connections of the private endpoint.

CustomNetworkInterfaceName specifies the network interface name associated with the private endpoint.

PrivateIPAddresses specifies the IP addresses for the network interface associated with the private endpoint. They have to be part of the subnet where the private endpoint is linked.

ApplicationSecurityGroups specifies the Application security group in which the private endpoint IP configuration is included.

ManualApproval specifies if the connection approval needs to be done manually or not. Set it true when the network admin does not have access to approve connections to the remote resource. Defaults to false.

Name specifies the name of the private link service.

PrivateLinkServiceID specifies the resource ID of the private link service.

GroupIDs specifies the ID(s) of the group(s) obtained from the remote resource that this private endpoint should connect to.

RequestMessage specifies a message passed to the owner of the remote resource with the private endpoint connection request.

"Canceled"

Canceled represents an action which was initiated but terminated by the user before completion.

"Creating"

Creating …

"Deleted"

Deleted represents a deleted VM NOTE: This state is specific to capz, and does not have corresponding mapping in Azure API (https://learn.microsoft.com/azure/virtual-machines/states-billing#provisioning-states)

"Deleting"

Deleting …

"Failed"

Failed …

"Migrating"

Migrating …

"Succeeded"

Succeeded …

"Updating"

Updating …

Name is the name of the rate limit spec.

"owned"

ResourceLifecycleOwned is the value we use when tagging resources to indicate that the resource is considered owned and managed by the cluster, and in particular that the lifecycle is tied to the lifecycle of the cluster.

"shared"

ResourceLifecycleShared is the value we use when tagging resources to indicate that the resource is shared between multiple clusters, and should not be destroyed if the cluster is destroyed.

ID is the Azure resource ID of the route table. READ-ONLY

"Standard"

SKUStandard is the value for the Azure load balancer Standard SKU.

"DiskWithVMGuestState"

SecurityEncryptionTypeDiskWithVMGuestState OS disk confidential encryption with a platform-managed key (PMK) or a customer-managed key (CMK).

"VMGuestStateOnly"

SecurityEncryptionTypeVMGuestStateOnly disables OS disk confidential encryption.

ID is the Azure resource ID of the security group. READ-ONLY

(Members of SecurityGroupClass are embedded into this type.)

"*"

SecurityGroupProtocolAll is a wildcard for all IP protocols.

"Icmp"

SecurityGroupProtocolICMP represents the ICMP protocol.

"Tcp"

SecurityGroupProtocolTCP represents the TCP protocol.

"Udp"

SecurityGroupProtocolUDP represents the UDP protocol.

This field indicates whether Host Encryption should be enabled or disabled for a virtual machine or virtual machine scale set. This should be disabled when SecurityEncryptionType is set to DiskWithVMGuestState. Default is disabled.

SecurityType specifies the SecurityType of the virtual machine. It has to be set to any specified value to enable UefiSettings. The default behavior is: UefiSettings will not be enabled unless this property is set.

UefiSettings specifies the security settings like secure boot and vTPM used while creating the virtual machine.

Name is a unique name within the network security group.

A description for this rule. Restricted to 140 chars.

Protocol specifies the protocol type. “Tcp”, “Udp”, “Icmp”, or “*”.

Direction indicates whether the rule applies to inbound, or outbound traffic. “Inbound” or “Outbound”.

Priority is a number between 100 and 4096. Each rule should have a unique value for priority. Rules are processed in priority order, with lower numbers processed before higher numbers. Once traffic matches a rule, processing stops.

SourcePorts specifies source port or range. Integer or range between 0 and 65535. Asterix ‘*’ can also be used to match all ports.

DestinationPorts specifies the destination port or range. Integer or range between 0 and 65535. Asterix ‘*’ can also be used to match all ports.

Source specifies the CIDR or source IP range. Asterix ‘*’ can also be used to match all source IPs. Default tags such as ‘VirtualNetwork’, ‘AzureLoadBalancer’ and ‘Internet’ can also be used. If this is an ingress rule, specifies where network traffic originates from.

Destination is the destination address prefix. CIDR or destination IP range. Asterix ‘*’ can also be used to match all source IPs. Default tags such as ‘VirtualNetwork’, ‘AzureLoadBalancer’ and ‘Internet’ can also be used.

Action specifies whether network traffic is allowed or denied. Can either be “Allow” or “Deny”. Defaults to “Allow”.

"Allow"

SecurityRuleActionAllow allows traffic defined in the rule.

"Deny"

SecurityRuleActionDeny denies traffic defined in the rule.

"Inbound"

SecurityRuleDirectionInbound defines an ingress security rule.

"Outbound"

SecurityRuleDirectionOutbound defines an egress security rule.

"ConfidentialVM"

SecurityTypesConfidentialVM defines the SecurityType of the virtual machine as a Confidential VM.

"TrustedLaunch"

SecurityTypesTrustedLaunch defines the SecurityType of the virtual machine as a Trusted Launch VM.

"false"

SkipNodesWithLocalStorageFalse …

"true"

SkipNodesWithLocalStorageTrue …

"false"

SkipNodesWithSystemPodsFalse …

"true"

SkipNodesWithSystemPodsTrue …

"Deallocate"

SpotEvictionPolicyDeallocate is the default eviction policy and will deallocate the VM when the node is marked for eviction.

"Delete"

SpotEvictionPolicyDelete will delete the VM when the node is marked for eviction.

MaxPrice defines the maximum price the user is willing to pay for Spot VM instances

EvictionPolicy defines the behavior of the virtual machine when it is evicted. It can be either Delete or Deallocate.

Name defines a name for the subnet resource.

Role defines the subnet role (eg. Node, ControlPlane)

CIDRBlocks defines the subnet’s address space, specified as one or more address prefixes in CIDR notation.

ServiceEndpoints is a slice of Virtual Network service endpoints to enable for the subnets.

PrivateEndpoints defines a list of private endpoints that should be attached to this subnet.

"bastion"

DefaultAzureBastionSubnetRole is the default Subnet role for AzureBastion.

"bastion"

SubnetBastion defines a Bastion subnet role.

"control-plane"

SubnetControlPlane defines a Kubernetes control plane node role.

"node"

SubnetNode defines a Kubernetes workload node role.

ID is the Azure resource ID of the subnet. READ-ONLY

SecurityGroup defines the NSG (network security group) that should be attached to this subnet.

RouteTable defines the route table that should be attached to this subnet.

NatGateway associated with this subnet.

(Members of SubnetClassSpec are embedded into this type.)

(Members of SubnetClassSpec are embedded into this type.)

SecurityGroup defines the NSG (network security group) that should be attached to this subnet.

NatGateway associated with this subnet.

FsAioMaxNr specifies the maximum number of system-wide asynchronous io requests. Valid values are 65536-6553500 (inclusive). Maps to fs.aio-max-nr.

FsFileMax specifies the max number of file-handles that the Linux kernel will allocate, by increasing increases the maximum number of open files permitted. Valid values are 8192-12000500 (inclusive). Maps to fs.file-max.

FsInotifyMaxUserWatches specifies the number of file watches allowed by the system. Each watch is roughly 90 bytes on a 32-bit kernel, and roughly 160 bytes on a 64-bit kernel. Valid values are 781250-2097152 (inclusive). Maps to fs.inotify.max_user_watches.

FsNrOpen specifies the maximum number of file-handles a process can allocate. Valid values are 8192-20000500 (inclusive). Maps to fs.nr_open.

KernelThreadsMax specifies the maximum number of all threads that can be created. Valid values are 20-513785 (inclusive). Maps to kernel.threads-max.

NetCoreNetdevMaxBacklog specifies maximum number of packets, queued on the INPUT side, when the interface receives packets faster than kernel can process them. Valid values are 1000-3240000 (inclusive). Maps to net.core.netdev_max_backlog.

NetCoreOptmemMax specifies the maximum ancillary buffer size (option memory buffer) allowed per socket. Socket option memory is used in a few cases to store extra structures relating to usage of the socket. Valid values are 20480-4194304 (inclusive). Maps to net.core.optmem_max.

NetCoreRmemDefault specifies the default receive socket buffer size in bytes. Valid values are 212992-134217728 (inclusive). Maps to net.core.rmem_default.

NetCoreRmemMax specifies the maximum receive socket buffer size in bytes. Valid values are 212992-134217728 (inclusive). Maps to net.core.rmem_max.

NetCoreSomaxconn specifies maximum number of connection requests that can be queued for any given listening socket. An upper limit for the value of the backlog parameter passed to the listen(2)(https://man7.org/linux/man-pages/man2/listen.2.html) function. If the backlog argument is greater than the somaxconn, then it’s silently truncated to this limit. Valid values are 4096-3240000 (inclusive). Maps to net.core.somaxconn.

NetCoreWmemDefault specifies the default send socket buffer size in bytes. Valid values are 212992-134217728 (inclusive). Maps to net.core.wmem_default.

NetCoreWmemMax specifies the maximum send socket buffer size in bytes. Valid values are 212992-134217728 (inclusive). Maps to net.core.wmem_max.

NetIpv4IPLocalPortRange is used by TCP and UDP traffic to choose the local port on the agent node. PortRange should be specified in the format “first last”. First, being an integer, must be between [1024 - 60999]. Last, being an integer, must be between [32768 - 65000]. Maps to net.ipv4.ip_local_port_range.

NetIpv4NeighDefaultGcThresh1 specifies the minimum number of entries that may be in the ARP cache. Garbage collection won’t be triggered if the number of entries is below this setting. Valid values are 128-80000 (inclusive). Maps to net.ipv4.neigh.default.gc_thresh1.

NetIpv4NeighDefaultGcThresh2 specifies soft maximum number of entries that may be in the ARP cache. ARP garbage collection will be triggered about 5 seconds after reaching this soft maximum. Valid values are 512-90000 (inclusive). Maps to net.ipv4.neigh.default.gc_thresh2.

NetIpv4NeighDefaultGcThresh3 specified hard maximum number of entries in the ARP cache. Valid values are 1024-100000 (inclusive). Maps to net.ipv4.neigh.default.gc_thresh3.

NetIpv4TCPFinTimeout specifies the length of time an orphaned connection will remain in the FIN_WAIT_2 state before it’s aborted at the local end. Valid values are 5-120 (inclusive). Maps to net.ipv4.tcp_fin_timeout.

NetIpv4TCPKeepaliveProbes specifies the number of keepalive probes TCP sends out, until it decides the connection is broken. Valid values are 1-15 (inclusive). Maps to net.ipv4.tcp_keepalive_probes.

NetIpv4TCPKeepaliveTime specifies the rate at which TCP sends out a keepalive message when keepalive is enabled. Valid values are 30-432000 (inclusive). Maps to net.ipv4.tcp_keepalive_time.

NetIpv4TCPMaxSynBacklog specifies the maximum number of queued connection requests that have still not received an acknowledgment from the connecting client. If this number is exceeded, the kernel will begin dropping requests. Valid values are 128-3240000 (inclusive). Maps to net.ipv4.tcp_max_syn_backlog.

NetIpv4TCPMaxTwBuckets specifies maximal number of timewait sockets held by system simultaneously. If this number is exceeded, time-wait socket is immediately destroyed and warning is printed. Valid values are 8000-1440000 (inclusive). Maps to net.ipv4.tcp_max_tw_buckets.

NetIpv4TCPTwReuse is used to allow to reuse TIME-WAIT sockets for new connections when it’s safe from protocol viewpoint. Maps to net.ipv4.tcp_tw_reuse.

NetIpv4TCPkeepaliveIntvl specifies the frequency of the probes sent out. Multiplied by tcpKeepaliveprobes, it makes up the time to kill a connection that isn’t responding, after probes started. Valid values are 1-75 (inclusive). Maps to net.ipv4.tcp_keepalive_intvl.

NetNetfilterNfConntrackBuckets specifies the size of hash table used by nf_conntrack module to record the established connection record of the TCP protocol. Valid values are 65536-147456 (inclusive). Maps to net.netfilter.nf_conntrack_buckets.

NetNetfilterNfConntrackMax specifies the maximum number of connections supported by the nf_conntrack module or the size of connection tracking table. Valid values are 131072-1048576 (inclusive). Maps to net.netfilter.nf_conntrack_max.

VMMaxMapCount specifies the maximum number of memory map areas a process may have. Maps to vm.max_map_count. Valid values are 65530-262144 (inclusive).

VMSwappiness specifies aggressiveness of the kernel in swapping memory pages. Higher values will increase aggressiveness, lower values decrease the amount of swap. Valid values are 0-100 (inclusive). Maps to vm.swappiness.

VMVfsCachePressure specifies the percentage value that controls tendency of the kernel to reclaim the memory, which is used for caching of directory and inode objects. Valid values are 1-500 (inclusive). Maps to vm.vfs_cache_pressure.

Name is the name of the role assignment to create for a system assigned identity. It can be any valid UUID. If not specified, a random UUID will be generated.

DefinitionID is the ID of the role definition to create for a system assigned identity. It can be an Azure built-in role or a custom role. Refer to built-in roles: https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles

Scope is the scope that the role assignment or definition applies to. The scope can be any REST resource instance. If not specified, the scope will be the subscription.

Effect specifies the effect for the taint

Key is the key of the taint

Value is the value of the taint

"best-effort"

TopologyManagerPolicyBestEffort …

"none"

TopologyManagerPolicyNone …

"restricted"

TopologyManagerPolicyRestricted …

"single-numa-node"

TopologyManagerPolicySingleNumaNode …

"always"

TransparentHugePageOptionAlways …

"defer"

TransparentHugePageOptionDefer …

"defer+madvise"

TransparentHugePageOptionDeferMadvise …

"madvise"

TransparentHugePageOptionMadvise …

"never"

TransparentHugePageOptionNever …

SecureBootEnabled specifies whether secure boot should be enabled on the virtual machine. Secure Boot verifies the digital signature of all boot components and halts the boot process if signature verification fails. If omitted, the platform chooses a default, which is subject to change over time, currently that default is false.

VTpmEnabled specifies whether vTPM should be enabled on the virtual machine. When true it enables the virtualized trusted platform module measurements to create a known good boot integrity policy baseline. The integrity policy baseline is used for comparison with measurements from subsequent VM boots to determine if anything has changed. This is required to be set to Enabled if SecurityEncryptionType is defined. If omitted, the platform chooses a default, which is subject to change over time, currently that default is false.

ProviderID is the identification ID of the user-assigned Identity, the format of an identity is: ‘azure:///subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identityName}’

StorageAccountURI is the URI of the user-managed storage account. The URI typically will be https://<mystorageaccountname>.blob.core.windows.net/ but may differ if you are using Azure DNS zone endpoints. You can find the correct endpoint by looking for the Blob Primary Endpoint in the endpoints tab in the Azure console or with the CLI by issuing az storage account list --query='[].{name: name, "resource group": resourceGroup, "blob endpoint": primaryEndpoints.blob}'.

DiskEncryptionSet specifies the customer-managed disk encryption set resource id for the managed disk that is used for Customer Managed Key encrypted ConfidentialVM OS Disk and VMGuest blob.

SecurityEncryptionType specifies the encryption type of the managed disk. It is set to DiskWithVMGuestState to encrypt the managed disk along with the VMGuestState blob, and to VMGuestStateOnly to encrypt the VMGuestState blob only. When set to VMGuestStateOnly, VirtualizedTrustedPlatformModule should be set to Enabled. When set to DiskWithVMGuestState, EncryptionAtHost should be disabled, SecureBoot and VirtualizedTrustedPlatformModule should be set to Enabled. It can be set only for Confidential VMs.

Name is the name of the extension.

Publisher is the name of the extension handler publisher.

Version specifies the version of the script handler.

Settings is a JSON formatted public settings for the extension.

ProtectedSettings is a JSON formatted protected settings for the extension.

"None"

VMIdentityNone …

"SystemAssigned"

VMIdentitySystemAssigned …

"UserAssigned"

VMIdentityUserAssigned …

CIDRBlocks defines the virtual network’s address space, specified as one or more address prefixes in CIDR notation.

Tags is a collection of tags describing the resource.

ResourceGroup is the resource group name of the remote virtual network.

RemoteVnetName defines name of the remote virtual network.

ForwardPeeringProperties specifies VnetPeeringProperties for peering from the cluster’s virtual network to the remote virtual network.

ReversePeeringProperties specifies VnetPeeringProperties for peering from the remote virtual network to the cluster’s virtual network.

AllowForwardedTraffic specifies whether the forwarded traffic from the VMs in the local virtual network will be allowed/disallowed in remote virtual network.

AllowGatewayTransit specifies if gateway links can be used in remote virtual networking to link to this virtual network.

AllowVirtualNetworkAccess specifies whether the VMs in the local virtual network space would be able to access the VMs in remote virtual network space.

UseRemoteGateways specifies if remote gateways can be used on this virtual network. If the flag is set to true, and allowGatewayTransit on remote peering is also set to true, the virtual network will use the gateways of the remote virtual network for transit. Only one peering can have this flag set to true. This flag cannot be set if virtual network already has a gateway.

(Members of VnetPeeringClassSpec are embedded into this type.)

ResourceGroup is the name of the resource group of the existing virtual network or the resource group where a managed virtual network should be created.

ID is the Azure resource ID of the virtual network. READ-ONLY

Name defines a name for the virtual network resource.

Peerings defines a list of peerings of the newly created virtual network with existing virtual networks.

(Members of VnetClassSpec are embedded into this type.)

(Members of VnetClassSpec are embedded into this type.)

Peerings defines a list of peerings of the newly created virtual network with existing virtual networks.