Diagnosing and Automating Azure VM Boot Failures with PowerShell and Azure CLI

Introduction

If you manage Azure infrastructure, you’ve likely encountered virtual machines that just won’t boot properly. These failures can be caused by anything from corrupted disk snapshots to misconfigured extensions. Today, we’ll walk through a common real-world scenario where a VM fails to start, how to diagnose it using PowerShell and Azure CLI, and finally automate the recovery process for repeatable use.

---

Scenario

One of your critical Azure VMs isn’t booting after a recent OS update. Attempts to connect via RDP time out, and the Azure portal shows the VM in a “Starting” or “Failed” state but never fully running.

---

Step 1: Initial Diagnostics with PowerShell and Azure CLI

Start by gathering VM status and error details:

# Log in to Azure
Connect-AzAccount

# Define variables
$resourceGroup = "Prod-RG"
$vmName = "webapp-vm01"

# Get VM instance view
$vmStatus = Get-AzVM -ResourceGroupName $resourceGroup -Name $vmName -Status

# Check VM statuses
$vmStatus.Statuses | Where-Object { $_.Code -like "PowerState/*" }

# Use Azure CLI for detailed boot diagnostics
az vm boot-diagnostics get-boot-log --resource-group $resourceGroup --name $vmName

The Get-AzVM -Status cmdlet provides the power state and provisioning state, while boot diagnostics can reveal errors logged during startup.

---

Step 2: Identify Common Causes

Typical root causes include:

• Disk corruption or detaching issues
• Extension deployment failures
• Network Security Group (NSG) blocking RDP/SSH
• Insufficient resources preventing VM allocation

Check VM extensions status:

Get-AzVMExtension -ResourceGroupName $resourceGroup -VMName $vmName | Select-Object Name, ProvisioningState, Type

If an extension is stuck or failed, it can prevent VM from reaching a running state.

---

Step 3: Automated Remediation Script

Assuming the problem is a stuck extension or corrupted boot diagnostics data, here’s a PowerShell script that:

• Stops the VM if running
• Removes problematic extensions
• Clears boot diagnostics logs
• Starts the VM again

# Stop the VM if running
if (($vmStatus.Statuses | Where-Object { $_.Code -like "PowerState/running" })) {
    Stop-AzVM -ResourceGroupName $resourceGroup -Name $vmName -Force -ErrorAction Stop
    Write-Host "Stopped VM $vmName"
}

# Remove all extensions (customize as needed)
$extensions = Get-AzVMExtension -ResourceGroupName $resourceGroup -VMName $vmName
foreach ($ext in $extensions) {
    Remove-AzVMExtension -ResourceGroupName $resourceGroup -VMName $vmName -Name $ext.Name -Force
    Write-Host "Removed extension $($ext.Name)"
}

# Clear boot diagnostics (requires knowing storage account)
$vm = Get-AzVM -ResourceGroupName $resourceGroup -Name $vmName
$storageUri = $vm.DiagnosticsProfile.BootDiagnostics.StorageUri
if ($storageUri) {
    $storageAccountName = ($storageUri -split '\.')[0] -replace 'https://', ''
    $context = (Get-AzStorageAccount -ResourceGroupName $resourceGroup -Name $storageAccountName).Context

    # Remove boot diagnostics VHD logs
    $container = Get-AzStorageContainer -Name 'bootdiagnostics-logs' -Context $context -ErrorAction SilentlyContinue
    if ($container) {
        $blobs = Get-AzStorageBlob -Container $container.Name -Context $context
        foreach ($blob in $blobs) {
            Remove-AzStorageBlob -Blob $blob.Name -Container $container.Name -Context $context
            Write-Host "Removed boot diagnostics blob $($blob.Name)"
        }
    }
}

# Start the VM again
Start-AzVM -ResourceGroupName $resourceGroup -Name $vmName
Write-Host "Started VM $vmName"

---

Step 4: Scheduling Automation and Alerting

To avoid manual intervention, schedule this script to run as an Azure Automation Runbook or a scheduled task on your management server. Combine it with alerting rules in Azure Monitor that trigger when VM boot failures are detected.

---

Bonus: Using Azure CLI and Bash for Multi-Cloud Engineers

If you prefer Azure CLI or work in mixed environments, here is a comparable snippet to remove extensions and restart VMs:

RESOURCE_GROUP="Prod-RG"
VM_NAME="webapp-vm01"

# Check VM power state
az vm get-instance-view --resource-group $RESOURCE_GROUP --name $VM_NAME --query "instanceView.statuses[?starts_with(code, 'PowerState/')].code" -o tsv

# Stop VM
az vm stop --resource-group $RESOURCE_GROUP --name $VM_NAME

# List extensions
az vm extension list --resource-group $RESOURCE_GROUP --vm-name $VM_NAME --query "[].name" -o tsv | while read ext; do
    az vm extension delete --resource-group $RESOURCE_GROUP --vm-name $VM_NAME --name $ext
    echo "Deleted extension $ext"
done

# Start VM
az vm start --resource-group $RESOURCE_GROUP --name $VM_NAME

---

Conclusion

Diagnosing Azure VM boot failures can be tricky, but with a systematic approach and some PowerShell automation, you can reduce downtime and manual troubleshooting. The key is to gather diagnostic info, identify stuck extensions or corrupted logs, and automate cleanups and restarts. This approach scales well in enterprise environments where you manage dozens or hundreds of VMs.

Stay tuned for upcoming posts where we’ll tackle automating AWS EC2 recoveries using Python and Terraform!

---

Blog Post 2: Automating AWS EC2 Instance Health Checks and Remediation Using PowerShell and AWS CLI

Introduction

AWS EC2 instances sometimes enter impaired states due to failed system status checks or application crashes. Continuous monitoring combined with scripted remediation can dramatically improve availability. In this post, we’ll build a PowerShell script that queries EC2 instance health, diagnoses common issues, and automatically reboots or replaces unhealthy instances.

---

Step 1: Prerequisites and Setup

• AWS Tools for PowerShell installed (Install-Module -Name AWSPowerShell)
• AWS CLI configured with appropriate permissions
• Target EC2 instance IDs and tags to filter

---

Step 2: Check EC2 Instance Health Status

PowerShell snippet to fetch status checks:

Import-Module AWSPowerShell

$region = "us-east-1"
$instanceId = "i-0abcdef1234567890"

# Get instance status
$instanceStatus = Get-EC2InstanceStatus -InstanceId $instanceId -Region $region

$sysStatus = $instanceStatus.InstanceStatuses[0].SystemStatus.Status
$instStatus = $instanceStatus.InstanceStatuses[0].InstanceStatus.Status

Write-Host "System Status: $sysStatus"
Write-Host "Instance Status: $instStatus"

You want both to be ok for a healthy instance.

---

Step 3: Automated Remediation Logic

If the system or instance status is impaired, perform a reboot. If repeated failures occur, consider terminating and recreating the instance using an Auto Scaling Group or Terraform.

# Define remediation function
function Remediate-EC2Instance {
    param(
        [string]$InstanceId,
        [string]$Region
    )
    $status = Get-EC2InstanceStatus -InstanceId $InstanceId -Region $Region
    $sysStatus = $status.InstanceStatuses[0].SystemStatus.Status
    $instStatus = $status.InstanceStatuses[0].InstanceStatus.Status

    if ($sysStatus -eq "impaired" -or $instStatus -eq "impaired") {
        Write-Host "Instance $InstanceId is impaired. Rebooting..."
        Restart-EC2Instance -InstanceId $InstanceId -Region $Region
        Start-Sleep -Seconds 60

        # Re-check status after reboot
        $newStatus = Get-EC2InstanceStatus -InstanceId $InstanceId -Region $Region
        if ($newStatus.InstanceStatuses[0].SystemStatus.Status -eq "impaired" -or $newStatus.InstanceStatuses[0].InstanceStatus.Status -eq "impaired") {
            Write-Warning "Instance $InstanceId remains impaired after reboot. Consider replacing."
            # Optional: Terminate and trigger ASG or Terraform to recreate
            # Remove-EC2Instance -InstanceId $InstanceId -Region $Region -Force
        }
        else {
            Write-Host "Instance $InstanceId recovered after reboot."
        }
    }
    else {
        Write-Host "Instance $InstanceId is healthy."
    }
}

# Example usage
Remediate-EC2Instance -InstanceId $instanceId -Region $region

---

Step 4: Scheduling and Scaling

Run this script as a scheduled task or integrate into event-driven Lambda (using AWS CLI commands) for automated health remediation. For fleets, loop over multiple instances filtered by tag:

“`powershell
$instances = Get-EC2Instance -Region $region -Filter @{ Name =