Cloud & infrastructure

Save Money on AKS by Stopping Your Dev Clusters

A complete, anonymized Terraform and PowerShell example for scheduled AKS shutdowns, with managed identity, weekday schedules, and optional Crossplane webhook cleanup.

BY CHRIS HOUSENovember 04, 202610 min read

On this page

My personal Kubernetes lab doesn't need to run while I'm asleep. Azure will happily keep charging for the compute anyway.

In ARMServicePortal, I configured schedules to stop my AKS clusters when I don't need them and start them again later. The setup is Terraform, Azure Automation, and PowerShell. The interesting part was a Crossplane webhook that got in the way of shutting down.

I previously wrote about starting and stopping clusters through my AI assistant. This is the scheduled version, with the code cleaned up so you can adapt it to your own clusters.

What the schedule saves

These are the weekday schedules in my mgmt cluster configuration and app cluster configuration:

Cluster	Start	Stop	Days
mgmt cluster	5 p.m.	10 p.m.	Monday–Friday
app cluster	7:30 a.m.	11:30 p.m.	Monday–Friday

Both use America/New_York, so those are Eastern times. The mgmt cluster's window is 25 hours a week instead of 168: 143 hours scheduled off, roughly 85% of the week. That's scheduled running time, not a measured 85% reduction in the total bill.

AKS can stop the nodes and control plane while retaining cluster state. Scaling application pods to zero still leaves nodes running; scaling user node pools to zero leaves the system pool. Microsoft's stop and start documentation covers the supported cluster configurations.

Use the code in your own subscription

Copy the four code blocks below into a new folder, using these filenames:

Set-AksPowerState.ps1 — the start and stop runbook.
variables.tf — the Terraform inputs.
main.tf — the account, permissions, modules, and schedules.
terraform.tfvars — your cluster names and hours.

Keep all four files in the same folder. The code blocks have copy buttons, and the settings use placeholder names you can replace with your own.

This version uses one Automation account for both existing clusters, with Contributor access scoped to each cluster. All cluster and resource group names in the example are placeholders. It adds explicit module versions, subscription selection, status checks, and failure handling to the original implementation. Scheduled execution and webhook deletion both start disabled.

You need Terraform 1.5+, Azure CLI, existing AKS dev clusters in one subscription, and permission to create the scheduler resources and assign roles on those clusters. The sample uses a new resource group for its Automation account. It does not create or take Terraform ownership of your AKS clusters.

The PowerShell runbook

Save this as Set-AksPowerState.ps1. One runbook handles both actions. It authenticates through the Automation account's managed identity, selects the supplied subscription, and checks the current state before changing anything.

The optional Crossplane step removes only ValidatingWebhookConfiguration/crossplane-no-usages. It does not remove Crossplane or pause its reconciler. A failed or indeterminate cleanup result stops the script before it requests a cluster shutdown.

[CmdletBinding()]
param(
    [Parameter(Mandatory = $true)]
    [ValidatePattern('^[0-9a-fA-F-]{36}$')]
    [string] $SubscriptionId,

    [Parameter(Mandatory = $true)]
    [ValidateNotNullOrEmpty()]
    [string] $ResourceGroupName,

    [Parameter(Mandatory = $true)]
    [ValidateNotNullOrEmpty()]
    [string] $ClusterName,

    [Parameter(Mandatory = $true)]
    [ValidateSet('Start', 'Stop')]
    [string] $Action,

    # String because Azure Automation schedule parameters are strings.
    [ValidateSet('true', 'false')]
    [string] $RemoveCrossplaneWebhook = 'false'
)

$ErrorActionPreference = 'Stop'
Import-Module Az.Accounts -RequiredVersion 5.5.0
Import-Module Az.Aks -RequiredVersion 7.2.0

# Avoid inheriting another job's saved Azure context.
Disable-AzContextAutosave -Scope Process | Out-Null
$context = (Connect-AzAccount -Identity).Context
$context = Set-AzContext -SubscriptionId $SubscriptionId -DefaultProfile $context

$clusterArgs = @{
    ResourceGroupName = $ResourceGroupName
    Name             = $ClusterName
    DefaultProfile   = $context
}
$cluster = Get-AzAksCluster @clusterArgs
$desiredState = if ($Action -eq 'Start') { 'Running' } else { 'Stopped' }

if ($cluster.ProvisioningState -ne 'Succeeded') {
    throw "Cluster operation in progress or failed: $($cluster.ProvisioningState). Check it before retrying."
}
if ($cluster.PowerState.Code -eq $desiredState) {
    Write-Output "$ClusterName is already $desiredState; no changes needed."
    return
}
if ($cluster.PowerState.Code -notin @('Running', 'Stopped')) {
    throw "Unexpected power state: $($cluster.PowerState.Code)."
}

if ($Action -eq 'Stop' -and $RemoveCrossplaneWebhook -eq 'true') {
    # Only opt in after diagnosing this specific webhook as the blocker.
    # This removes an admission check; it does not pause the reconciler.
    $result = Invoke-AzAksRunCommand @clusterArgs `
        -Command 'kubectl delete validatingwebhookconfiguration crossplane-no-usages --ignore-not-found=true' `
        -Force

    if ($null -eq $result -or $null -eq $result.ExitCode -or
        $result.ExitCode -ne 0 -or $result.ProvisioningState -ne 'Succeeded') {
        throw 'Webhook cleanup failed or returned no successful result. Cluster stop was not requested.'
    }
    Write-Output 'Crossplane webhook cleanup completed.'
}

# These cmdlets require SubscriptionId; their DefaultProfile is not functional.
# Omit NoWait so the job waits for the Azure operation to complete.
$operationArgs = @{
    ResourceGroupName = $ResourceGroupName
    Name             = $ClusterName
    SubscriptionId   = $SubscriptionId
    ErrorAction      = 'Stop'
}
Write-Output "$Action requested for $ClusterName in $ResourceGroupName."
if ($Action -eq 'Start') {
    Start-AzAksCluster @operationArgs | Out-Null
} else {
    Stop-AzAksCluster @operationArgs | Out-Null
}

$after = Get-AzAksCluster @clusterArgs
if ($after.PowerState.Code -ne $desiredState -or
    $after.ProvisioningState -ne 'Succeeded') {
    throw "Expected $desiredState/Succeeded; got $($after.PowerState.Code)/$($after.ProvisioningState). Inspect the cluster before retrying."
}
Write-Output "$ClusterName verified: $desiredState."

This script runs inside Azure Automation, where Connect-AzAccount -Identity can use the account's identity. An ordinary local terminal doesn't have that identity. The manual CLI alternative is below.

The Terraform configuration

Keep these files beside the PowerShell script. The module imports use Windows PowerShell 5.1, matching the runbook's PowerShell type. Az.Aks 7.2.0 requires Az.Accounts 5.5.0 or newer; this example pins that pair and imports Accounts first.

Save the inputs as variables.tf:

variable "subscription_id" {
  description = "Subscription containing the existing AKS clusters and new Automation account."
  type        = string
}

variable "location" {
  type    = string
  default = "eastus"
}

variable "automation_resource_group_name" {
  type    = string
  default = "rg-aks-schedules"
}

variable "automation_account_name" {
  type    = string
  default = "aa-aks-schedules"
}

variable "timezone" {
  description = "Schedule time zone; first-run timestamp offsets must match it for their dates."
  type        = string
  default     = "America/New_York"
}

variable "schedules_enabled" {
  description = "Attach schedules only after a successful manual stop/start test."
  type        = bool
  default     = false
}

variable "clusters" {
  description = "Existing clusters, keyed by a short label. First-run times must be future weekdays."
  type = map(object({
    name                      = string
    resource_group_name       = string
    first_start               = string
    first_stop                = string
    remove_crossplane_webhook = optional(bool, false)
  }))

  validation {
    condition = length(var.clusters) > 0 && alltrue([
      for cluster in values(var.clusters) :
      can(formatdate("YYYY-MM-DD", cluster.first_start)) &&
      can(formatdate("YYYY-MM-DD", cluster.first_stop))
    ])
    error_message = "Supply at least one cluster and valid RFC3339 first_start/first_stop timestamps."
  }
}

Scheduler resources

Save the resources as main.tf. Azure Automation lives outside the clusters, so it can start them while Kubernetes is offline. Contributor is the role my original setup uses; it allows more than start/stop, but its scope here is each named cluster.

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
  }
}

provider "azurerm" {
  features {}
  subscription_id = var.subscription_id
}

locals {
  cluster_ids = {
    for key, cluster in var.clusters : key =>
    "/subscriptions/${var.subscription_id}/resourceGroups/${cluster.resource_group_name}/providers/Microsoft.ContainerService/managedClusters/${cluster.name}"
  }
  jobs = merge([
    for key, cluster in var.clusters : {
      "${key}-start" = {
        cluster_key = key
        action      = "Start"
        first_run   = cluster.first_start
      }
      "${key}-stop" = {
        cluster_key = key
        action      = "Stop"
        first_run   = cluster.first_stop
      }
    }
  ]...)
}

# This example manages scheduler resources, not the existing AKS clusters.
resource "azurerm_resource_group" "scheduler" {
  name     = var.automation_resource_group_name
  location = var.location
}

resource "azurerm_automation_account" "scheduler" {
  name                = var.automation_account_name
  resource_group_name = azurerm_resource_group.scheduler.name
  location            = azurerm_resource_group.scheduler.location
  sku_name            = "Basic"

  identity {
    type = "SystemAssigned"
  }
}

# Same role as my original setup, scoped to each cluster, not the subscription.
resource "azurerm_role_assignment" "cluster" {
  for_each             = local.cluster_ids
  scope                = each.value
  role_definition_name = "Contributor"
  principal_id         = azurerm_automation_account.scheduler.identity[0].principal_id
  principal_type       = "ServicePrincipal"
}

# Import dependencies before the runbook can be used.
# These imports and the runbook all use Windows PowerShell 5.1.
resource "azurerm_automation_module" "accounts" {
  name                    = "Az.Accounts"
  resource_group_name     = azurerm_resource_group.scheduler.name
  automation_account_name = azurerm_automation_account.scheduler.name

  module_link {
    uri = "https://www.powershellgallery.com/api/v2/package/Az.Accounts/5.5.0"
  }
}

resource "azurerm_automation_module" "aks" {
  name                    = "Az.Aks"
  resource_group_name     = azurerm_resource_group.scheduler.name
  automation_account_name = azurerm_automation_account.scheduler.name

  module_link {
    uri = "https://www.powershellgallery.com/api/v2/package/Az.Aks/7.2.0"
  }
  depends_on = [azurerm_automation_module.accounts]
}

resource "azurerm_automation_runbook" "power" {
  name                    = "Set-AksPowerState"
  resource_group_name     = azurerm_resource_group.scheduler.name
  location                = azurerm_resource_group.scheduler.location
  automation_account_name = azurerm_automation_account.scheduler.name
  runbook_type            = "PowerShell"
  log_verbose             = false
  log_progress            = false
  content                 = file("${path.module}/Set-AksPowerState.ps1")

  depends_on = [azurerm_automation_module.aks]
}

resource "azurerm_automation_schedule" "cluster" {
  for_each                = local.jobs
  name                    = each.key
  resource_group_name     = azurerm_resource_group.scheduler.name
  automation_account_name = azurerm_automation_account.scheduler.name
  frequency               = "Week"
  interval                = 1
  timezone                = var.timezone
  start_time              = each.value.first_run
  week_days               = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"]
}

# With schedules_enabled=false there are no bindings, so no scheduled jobs run.
resource "azurerm_automation_job_schedule" "cluster" {
  for_each                = var.schedules_enabled ? local.jobs : {}
  resource_group_name     = azurerm_resource_group.scheduler.name
  automation_account_name = azurerm_automation_account.scheduler.name
  schedule_name           = azurerm_automation_schedule.cluster[each.key].name
  runbook_name            = azurerm_automation_runbook.power.name

  # Azure Automation requires lowercase parameter keys.
  parameters = {
    subscriptionid          = var.subscription_id
    resourcegroupname       = var.clusters[each.value.cluster_key].resource_group_name
    clustername             = var.clusters[each.value.cluster_key].name
    action                  = each.value.action
    removecrossplanewebhook = tostring(var.clusters[each.value.cluster_key].remove_crossplane_webhook)
  }

  depends_on = [azurerm_role_assignment.cluster]
}

output "automation_account_name" {
  value = azurerm_automation_account.scheduler.name
}

output "automation_resource_group_name" {
  value = azurerm_resource_group.scheduler.name
}

output "runbook_name" {
  value = azurerm_automation_runbook.power.name
}

The job bindings pass lowercase parameter names because Azure Automation requires them. The resources and module imports can finish before you attach any scheduled jobs.

Set your clusters and times

Save this block as terraform.tfvars, then replace the subscription ID, names, resource groups, location, and first-run dates. Delete the app entry if you only need one cluster.

The sample dates must be changed when they are no longer in the future. Choose future weekdays and allow enough time for the module imports. Azure requires the first run to be at least five minutes after schedule creation. The timestamp's offset must match your time zone on that date: New York uses -05:00 on these November dates and -04:00 in summer. The named zone handles subsequent daylight-saving transitions. Schedule reference.

subscription_id = "00000000-0000-0000-0000-000000000000"
location        = "eastus"
timezone        = "America/New_York"

# Leave false until you have tested Stop and Start manually.
schedules_enabled = false

clusters = {
  mgmt = {
    name                = "aks-example-mgmt"
    resource_group_name = "rg-example-mgmt"
    # Replace these dates with FUTURE weekdays before your first apply.
    # -05:00 is correct for New York on these November dates; summer is -04:00.
    first_start = "2026-11-09T17:00:00-05:00"
    first_stop  = "2026-11-09T22:00:00-05:00"
    # Only enable after diagnosing this specific Crossplane webhook blocker.
    remove_crossplane_webhook = false
  }
  app = {
    name                      = "aks-example-app"
    resource_group_name        = "rg-example-app"
    first_start               = "2026-11-09T07:30:00-05:00"
    first_stop                = "2026-11-09T23:30:00-05:00"
    remove_crossplane_webhook = false
  }
}

Using explicit dates keeps the schedule from shifting whenever Terraform runs. Change the inputs deliberately when you want different hours.

Deploy and test one cluster

From the folder containing those four files, authenticate and inspect the Terraform plan:

az login
az account set --subscription YOUR_SUBSCRIPTION_ID

terraform init
terraform fmt
terraform validate
terraform plan
terraform apply

The first apply creates the scheduler and imports its dependencies. With schedules_enabled = false, it creates no job bindings, so it won't start shutting down clusters on a timer. Keep Terraform state private and retain the generated provider lock file.

In the Azure portal, open the Automation account, choose Runbooks → Set-AksPowerState → Start, and provide:

Parameter	Value for the first test
SubscriptionId	Your subscription ID
ResourceGroupName	The existing cluster's resource group
ClusterName	The existing cluster's name
Action	`Stop`
RemoveCrossplaneWebhook	`false`

Wait for the job to complete, then check the cluster:

az aks show --resource-group YOUR_RESOURCE_GROUP --name YOUR_CLUSTER \
  --query "{power:powerState.code,provisioning:provisioningState}" -o table

Expect Stopped and Succeeded. AKS advises waiting 15–30 minutes after stopping before restarting. Run the same runbook with Action=Start, verify Running, and check node readiness and your applications. A successful Azure operation isn't an application health check. If the first job gets an authorization error, allow time for the new role assignment to propagate and inspect the job logs.

Once that cycle works, set schedules_enabled = true in terraform.tfvars, then run terraform plan and terraform apply again. Move expired first-run dates forward before activation. Check the next scheduled jobs' outcomes instead of assuming a timer means the cluster stopped.

To disable future scheduled runs, change that flag back to false and apply. That doesn't cancel an active job or start an already stopped cluster. Avoid overlapping manual runs, deployments, and scheduled shutdowns.

For a manual stop/start from your own authenticated terminal, use these commands separately, allowing shutdown to finish before starting again:

# Stop when you are finished using the cluster.
az aks stop --resource-group YOUR_RESOURCE_GROUP --name YOUR_CLUSTER

# Start later, when you need it again.
az aks start --resource-group YOUR_RESOURCE_GROUP --name YOUR_CLUSTER

If Crossplane blocks the stop

My original runbook deletes a specific webhook before calling Stop-AzAksCluster. This example makes that an opt-in setting because most clusters don't need it. If you have diagnosed that exact webhook as the blocker, change remove_crossplane_webhook = true for that cluster and apply. For a manual runbook test, pass RemoveCrossplaneWebhook=true explicitly.

Removing a webhook removes its admission checks. This runbook does not restore them. Verify or restore the expected configuration through your normal chart or GitOps setup after startup. Prefer correcting overbroad webhook rules where possible; AKS documents those alternatives.

The workaround uses AKS Run Command, which must be enabled and able to schedule its helper pod. Microsoft doesn't recommend Run Command for programmatic access. It is included to show the original workaround; for a permanent cleanup workflow, use your established cluster access or fix the webhook rules.

What still costs money

Stopping compute doesn't delete disks or networking resources. Those can continue to incur charges. Check the whole resource group in Cost Management when comparing bills, including the scheduler's usage.

This fits a lab or dev environment that can tolerate downtime. Leave time for startup and application readiness; restart capacity isn't guaranteed. Standalone pods aren't preserved, so run workloads under controllers such as Deployments or StatefulSets.

FILED UNDERazure aks kubernetes terraform crossplane cost-optimization

Found this useful?

What the schedule saves

Use the code in your own subscription

The PowerShell runbook

The Terraform configuration

Scheduler resources

Set your clusters and times

Deploy and test one cluster

If Crossplane blocks the stop

What still costs money

Related Posts

Two Claude Code Mods: A Spend Meter and a Kube Context Picker

Platform AI Part 3: Building a Multi-Agent System for DevOps

Platform AI Part 2: Adding Agentic Tool Use for AKS Management

Comments