How to Setup NVIDIA Driver on NV-Series Azure VM

I recently had the opportunity to assist on a project where a partner was using N-Series Azure VMs.  My part of this effort was developing a script to automate the setup of the VMs. To perform the VM setup and configuration, an ARM template was used.  The ARM template approach was used because doing so provided consistency with several other ARM templates being used for other parts of the project.

Setting up Azure VMs using ARM templates is common. There are many articles, blog posts, and sample templates available to help get started.  That isn’t itself especially interesting. The interesting part, at least for me, was the N-Series aspect. N-Series VMs require a separate step to install the NVIDIA driver to take advantage of the GPU capabilities of the VM.  There are instructions on how to install the driver, but those instructions assume you like to remote into the VM each time you create a VM, and then run an installation program. That’s tolerable if doing it only a few times. Any more than that, and it’s time for automation.

The v370.12 driver (which is the current version linked via the Azure documentation page) uses a self-extracting file to first extract the setup components to a directory, and then executes the setup program.  By scouring a few other blogs on performing a silent install of NVIDIA drivers, I could piece together the necessary switches to provide to the installation program to perform a silent install.

> 370.12_grid_win8_win7_server2012R2_server2008R2_64bit_international.exe -s -noreboot -clean

This tells the installation program to install silently, to not perform a reboot after the installation is complete, and to perform a clean install (restores all NVIDIA settings to the default values).

Now I need to work that it into a PowerShell script to execute via a custom script extension. By doing so, I can let ARM do its thing by provisioning the VM and related resources (NIC, Virtual Network, IP address, etc.), and then invoke a PowerShell script to install the NVIDIA driver.

The custom script extension will execute a few different steps:

  1. Download the NVIDIA driver setup file from Azure Blob storage. I put the setup file in blob storage to make sure that this specific one is the one to be used.
  2. Download a PowerShell script which will execute the NVIDIA driver setup program with parameters to do so silently.
  3. Wait for the installation program to finish
  4. Force a reboot of the VM

It should be noted that the driver installation and GPU detection can take a couple of minutes.

As you can see in the following snippets, the custom script extension and related PowerShell script are fairly trivial.

ARM Template Custom Script Extension

{
      "type": "extensions",
      "name": "CustomScriptExtension",
      "apiVersion": "2015-06-15",
      "location": "[resourceGroup().location]",
      "dependsOn": [
      	"[variables('vmName')]"
],
"properties": {
      	"publisher": "Microsoft.Compute",
      "type": "CustomScriptExtension",
      	"typeHandlerVersion": "1.8",
      "autoUpgradeMinorVersion": true,
      	"settings": {
            	"fileUris": [
                  	"[concat(variables('assetStorageUrl'), variables('scriptFileName'))]",
                        "[concat(variables('assetStorageUrl'), variables('nvidiaDriverSetupName'))]"
]
},
            "protectedSettings": {
            	"commandToExecute": "[concat('powershell -ExecutionPolicy Unrestricted -File ', variables('scriptFileName'), ' ', variables('scriptParameters'))]",
                  "storageAccountName": "[parameters('assetStorageAccountName')]",
                  "storageAccountKey": "[listKeys(concat('Microsoft.Storage/storageAccounts/', parameters('assetStorageAccountName')), '2015-06-15').key1]"
      	}
}
}

PowerShell script executed by the Custom Script Extension

<# Custom Script for Windows to install a file from Azure Storage #>
param(
    [string] $nvidiaDriverSetupPath
)

# ----- Silent install of NVidia driver -----
& ".\$nvidiaDriverSetupPath" -s -noreboot -clean

# ----- Sleep to allow the setup program to finish. -----
Start-Sleep -Seconds 120

# ----- NVidia driver installation requires a reboot. -----
Restart-Computer -Force

In this scenario, I also need to get the assets used by the custom script extension – the NVIDIA driver setup file and PowerShell script (which will execute the NVIDIA driver setup file) – uploaded to Azure Blob storage.  That can easily be accomplished with the same PowerShell script used to deploy the ARM template.  That script will perform the following tasks:

  1. Create a new resource group
  2. Create a new storage account and container
  3. Upload the NVIDIA driver setup file and related PowerShell script to the newly created storage account
  4. Execute the ARM template

You can find the full ARM template, custom script, and deployment script on my GitHub project which accompanies this post.

In order to verify it all worked, I can RDP into the VM and verify the driver installation.

This slideshow requires JavaScript.

What about unsigned drivers?

An earlier version of the NVIDIA driver, v369.95, was not digitally signed.  It was also provided as a ZIP file instead of an EXE (like v370.12).   To use this version of the NVIDIA driver, a few changes to the setup script are necessary. First, the file contents need to be extracted/unzipped.  That’s doable via some PowerShell in the script executed via the custom script extension.  Getting around the lack of a digitally signed driver is a bit more . . . interesting. If you were to install the driver manually, you would receive a prompt from Windows asking you to confirm that installing the driver is REALLY what is desired.

NVIDIA-security-prompt.png

Completing the manual installation will result in a certificate installed to the VM’s Trusted Publisher certificate store.  The certificate can then be exported and saved to Azure Blob storage.

cert-manager.png

I can use that certificate as part of the automated install process. By using the certutil.exe program it is possible to install the certificate into the Trusted Publisher store on a new VM.  This step can be included in the PowerShell script executed via the custom script extension.

An example of this approach can be found at https://github.com/mcollier/setup-nvidia-drivers-azure-vm/tree/driver-369.95.

Alternative Approach

An alternative approach is to create a custom VM image with the necessary NVIDIA driver already installed.  The advantage with this approach is you don’t have to go through the custom script step. However, any new VM deployed from such an image will still need to go through a reboot after GPU detection following the first startup. You can also add additional software or configuration as needed.  The disadvantage is you’re then accepting responsibility for keeping the VM patched on a regular basis. If you use an image provided by Microsoft, those images are patched on a regular (often at least once per month) basis.

Resources

Here are some resources which helped me in coming up with the solution presented above.

Advertisements

Copy Managed Images

Introduction

Azure Managed Disks were made generally available (GA) in February 2017. Managed Disks greatly simplify working with Azure Virtual Machines (VM) and Virtual Machine Scale Sets (VMSS). They effectively eliminate the need for you to have to worry about Azure Storage accounts and related VHD constraints/limits. When using managed disks for VMs or VMSS, you select the type of disk storage (SSD or HDD) and the size of disk needed. The Azure platform takes care of the rest. Besides the simplified management aspect, managed disks bring several additional benefits, but I’ll not reiterate those here, as there is a lot of good info already available (here, here and here).

While managed disks simplify management of Azure VMs, they also simplify working with VM images. Prior to managed disks, an image would need to be copied to the Storage account where the derived VM would be created. Doable, but not exactly convenient. With the introduction of managed disks, since the concept of using Storage accounts for disks and images has gone away, there is no need to copy the image. You can now create managed images as the ARM resources. You can easily create a VM by referencing the managed image, so long as the VM and image are in the same region and the same Azure subscription. You can consult the following two articles for detailed documentation on this topic:

However, what if you need to use the managed image in another Azure subscription (to which you have access)? Or, what if you need to use the managed image in another region? These capabilities are not yet available as part of the platform. However, there are workarounds you can use, with the currently available capability, to facilitate these needs.

In this post, we’ll explore the following two common scenarios:

  1. Copy a managed image to another Azure subscription
  2. Copy a managed image to another region

High Level Steps

To get a managed image in one Azure subscription to be available for use in another Azure subscription, there are a series of steps that currently need to be followed. In the near future, I expect this process to greatly be simplified by enhancements to the Azure platform’s managed image functionality. Until then, the high-level steps are as follows:

  1. Deploy a VM
  2. Configure the VM
  3. Generalize (using Sysprep) the VM
  4. Create an managed image in the source subscription
  5. Create a managed snapshot of the OS disk from the generalized VM
  6. Copy the managed snapshot to the target Azure subscription
    1. Alternative 1 – different region, same subscription
    2. Alternative 2 – different region, different subscription
  7. In the target subscription, create an managed image from the copied snapshot
  8. Optional: from the new managed image in the target subscription, create a new temporary VM
  9. Delete the snapshot in both the source and target Azure subscription
  10. Delete the temporary VM created in step #8

Getting Started

For the purposes of this post, I’m going to assume you have already created a VM using managed disks, configured it to your liking (e.g. installing some software, making configuration changes, etc.), and generalized the VM.

Create an Image

Assuming you have a generalized (deallocated) VM, the next step is to create a managed image.  It is worth pointing out that, at this time, creating the image is largely irrelevant when trying to copy the image to another region and/or subscription. As you’ll soon see, the artifact that is copied is the snapshot of disk(s) of the source (generalized) VM. The ability copy the image is not yet supported . . . hence this blog post to describe a workaround.

If you already have the Image, you can obviously skip this step. The steps are as follows:

PowerShell

<# -- Create a Managed Disk Image if necessary -- #>
$vm = Get-AzureRmVM -ResourceGroupName $resourceGroupName -Name $vmName
$image = New-AzureRmImageConfig -Location $region -SourceVirtualMachineId $vm.Id
New-AzureRmImage -Image $image -ImageName $imageName -ResourceGroupName $resourceGroupName

Azure CLI 2.0

# ------ Create an image ------
# Get the ID for the VM.
vmid=$(az vm show -g $ResourceGroupName -n vm --query "id" -o tsv)

# Create the image.
az image create -g $ResourceGroupName \
	--name $imageName \
	--location $location \
	--os-type Windows \
	--source $vmid

Create a snapshot

Now that you have an image, the next step is to create a snapshot of the OS disk of the source VM. If your image needs data disks, you’ll want to create a snapshot of the data disks as well (not shown below).

PowerShell

<# -- Create a snapshot of the OS (and optionally data disks) from the generalized VM -- #>
$vm = Get-AzureRmVM -ResourceGroupName $resourceGroupName -Name $vmName
$disk = Get-AzureRmDisk -ResourceGroupName $resourceGroupName -DiskName $vm.StorageProfile.OsDisk.Name
$snapshot = New-AzureRmSnapshotConfig -SourceUri $disk.Id -CreateOption Copy -Location $region

$snapshotName = $imageName + "-" + $region + "-snap"

New-AzureRmSnapshot -ResourceGroupName $resourceGroupName -Snapshot $snapshot -SnapshotName $snapshotName

Azure CLI 2.0

diskName=$(az vm show -g $ResourceGroupName -n vm --query "storageProfile.osDisk.name" -o tsv)
az snapshot create -g $ResourceGroupName -n $snapshotName --location $location –source $diskName

Copy the snapshot

The next step is to copy the snapshot to the target Azure subscription. In the following example, the first thing to do is grab the snapshot’s Resource ID. That ID is used to specific the source snapshot when creating the new snapshot.

PowerShell

<#-- copy the snapshot to another subscription, same region --#>
$snap = Get-AzureRmSnapshot -ResourceGroupName $resourceGroupName -SnapshotName $snapshotName

<#-- change to the target subscription #>
Select-AzureRmSubscription -SubscriptionId $targetSubscriptionId
$snapshotConfig = New-AzureRmSnapshotConfig -OsType Windows `
                                            -Location $region `
                                            -CreateOption Copy `
                                            -SourceResourceId $snap.Id

$snap = New-AzureRmSnapshot -ResourceGroupName $resourceGroupName `
                            -SnapshotName $snapshotName `
                            -Snapshot $snapshotConfig

Azure CLI 2.0

# ------ Copy the snapshot to another Azure subscription ------
# set the source subscription (to be sure)
az account set --subscription $SubscriptionID
snapshotId=$(az snapshot show -g $ResourceGroupName -n $snapshotName --query "id" -o tsv )

# change to the target subscription
az account set --subscription $TargetSubscriptionID
az snapshot create -g $ResourceGroupName -n $snapshotName --source $snapshotId

Alternative: Copy the snapshot to a different region for the same subscription

The previous examples showed how to copy the snapshot to a different subscription, with the restriction being the region for the source and target must be the same. There may be times when you need to get the snapshot to another region. The follow example shows how to copy the snapshot to another region, yet under the context of the same Azure subscription. The big difference here is the need to get at the blob which is the basis for the snapshot. That can be accomplished by getting a Shared Access Signature (SAS) for the snapshot.

PowerShell

# Create the name of the snapshot, using the current region in the name.
$snapshotName = $imageName + "-" + $region + "-snap"

# Get the source snapshot
$snap = Get-AzureRmSnapshot -ResourceGroupName $resourceGroupName -SnapshotName $snapshotName

# Create a Shared Access Signature (SAS) for the source snapshot
$snapSasUrl = Grant-AzureRmSnapshotAccess -ResourceGroupName $resourceGroupName -SnapshotName $snapshotName -DurationInSecond 3600 -Access Read

# Set up the target storage account in the other region
$targetStorageContext = (Get-AzureRmStorageAccount -ResourceGroupName $resourceGroupName -Name $storageAccountName).Context
New-AzureStorageContainer -Name $imageContainerName -Context $targetStorageContext -Permission Container

# Use the SAS URL to copy the blob to the target storage account (and thus region)
Start-AzureStorageBlobCopy -AbsoluteUri $snapSasUrl.AccessSAS -DestContainer $imageContainerName -DestContext $targetStorageContext -DestBlob $imageBlobName
Get-AzureStorageBlobCopyState -Container $imageContainerName -Blob $imageBlobName -Context $targetStorageContext -WaitForComplete

# Get the full URI to the blob
$osDiskVhdUri = ($targetStorageContext.BlobEndPoint + $imageContainerName + "/" + $imageBlobName)

# Build up the snapshot configuration, using the target storage account's resource ID
$snapshotConfig = New-AzureRmSnapshotConfig -AccountType StandardLRS `
                                            -OsType Windows `
                                            -Location $targetRegionName `
                                            -CreateOption Import `
                                            -SourceUri $osDiskVhdUri `
                                            -StorageAccountId "/subscriptions/${sourceSubscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.Storage/storageAccounts/${storageAccountName}"

# Create the new snapshot in the target region
$snapshotName = $imageName + "-" + $targetRegionName + "-snap"
$snap2 = New-AzureRmSnapshot -ResourceGroupName $resourceGroupName -SnapshotName $snapshotName -Snapshot $snapshotConfig

Azure CLI 2.0

az account set --subscription $SubscriptionID
snapshotId=$(az snapshot show -g $ResourceGroupName -n $snapshotName --query "id" -o tsv )

# Get the SAS for the snapshotId
snapshotSasUrl=$(az snapshot grant-access -g $ResourceGroupName -n $snapshotName --duration-in-seconds 3600 -o tsv)

# Setup the target storage account in another region
targetStorageAccountKey=$(az storage account keys list -g $ResourceGroupName --account-name $targetStorageAccountName --query "[:1].value" -o tsv)

storageSasToken=$(az storage account generate-sas --expiry 2017-05-02'T'12:00'Z' --permissions aclrpuw --resource-types sco --services b --https-only --account-name $targetStorageAccountName --account-key $targetStorageAccountKey -o tsv)
az storage container create -n $imageStorageContainerName --account-name $targetStorageAccountName --sas-token $storageSasToken

# Copy the snapshot to the target region using the SAS URL
imageBlobName = "$imageName-osdisk.vhd"
copyId=$(az storage blob copy start --source-uri $snapshotSasUrl --destination-blob $imageBlobName --destination-container $imageStorageContainerName --sas-token $storageSasToken --account-name $targetStorageAccountName)

# Figure out when the copy is destination-container
# TODO: Put this in a loop until status is 'success'
az storage blob show --container-name $imageStorageContainerName -n $imageBlobName --account-name $targetStorageAccountName --sas-token $storageSasToken --query "properties.copy.status"

# Get the URI to the blob

blobEndpoint=$(az storage account show -g $ResourceGroupName -n $targetStorageAccountName --query "primaryEndpoints.blob" -o tsv)
osDiskVhdUri="$blobEndpoint$imageStorageContainerName/$imageBlobName"

# Create the snapshot in the target region
snapshotName="$imageName-$targetLocation-snap"
az snapshot create -g $ResourceGroupName -n $snapshotName -l $targetLocation --source $osDiskVhdUri

Alternative: Copy the snapshot to a different region for a different subscription

The previous example showed how to copy the snapshot to a different region, yet associated with the same subscription. In the following example, we’ll tweak the example script a bit to show copying the snapshot to a different region and a different subscription.

These three examples should cover the scenarios needed to get the snapshot wherever it needs to be. From there, the steps to create the image should be the same, since they all start with the snapshot.

PowerShell

# Create the name of the snapshot, using the current region in the name.
$snapshotName = $imageName + "-" + $region + "-snap"

# Get the source snapshot
$snap = Get-AzureRmSnapshot -ResourceGroupName $resourceGroupName -SnapshotName $snapshotName

# Create a Shared Access Signature (SAS) for the source snapshot
$snapSasUrl = Grant-AzureRmSnapshotAccess -ResourceGroupName $resourceGroupName -SnapshotName $snapshotName -DurationInSecond 3600 -Access Read

# Set up the target storage account in the other region and subscription
Select-AzureRmSubscription -SubscriptionId $targetSubscriptionId

$targetStorageContext = (Get-AzureRmStorageAccount -ResourceGroupName $targetResourceGroupName -Name $targetStorageAccountName).Context
New-AzureStorageContainer -Name $imageContainerName -Context $targetStorageContext -Permission Container

# Use the SAS URL to copy the blob to the target storage account (and thus region)
Start-AzureStorageBlobCopy -AbsoluteUri $snapSasUrl.AccessSAS -DestContainer $imageContainerName -DestContext $targetStorageContext -DestBlob $imageBlobName
Get-AzureStorageBlobCopyState -Container $imageContainerName -Blob $imageBlobName -Context $targetStorageContext -WaitForComplete

# Get the full URI to the blob
$osDiskVhdUri = ($targetStorageContext.BlobEndPoint + $imageContainerName + "/" + $imageBlobName)

# Build up the snapshot configuration, using the target storage account's resource ID
$snapshotConfig = New-AzureRmSnapshotConfig -AccountType StandardLRS `
                                            -OsType Windows `
                                            -Location $targetRegionName `
                                            -CreateOption Import `
                                            -SourceUri $osDiskVhdUri `
                                            -StorageAccountId "/subscriptions/${targetSubscriptionId}/resourceGroups/${targetResourceGroupName}/providers/Microsoft.Storage/storageAccounts/${targetStorageAccountName}"

# Create the new snapshot in the target region
$snapshotName = $imageName + "-" + $targetRegionName + "-snap"
$snap2 = New-AzureRmSnapshot -ResourceGroupName $resourceGroupName -SnapshotName $snapshotName -Snapshot $snapshotConfig

Azure CLI 2.0

az account set --subscription $SubscriptionID
snapshotId=$(az snapshot show -g $ResourceGroupName -n $snapshotName --query "id" -o tsv )

# Get the SAS for the snapshotId
snapshotSasUrl=$(az snapshot grant-access -g $ResourceGroupName -n $snapshotName --duration-in-seconds 3600 -o tsv)

# Switch to the DIFFERENT subscription
az account set --subscription $TargetSubscriptionID

# Setup the target storage account in another region
targetStorageAccountKey=$(az storage account keys list -g $ResourceGroupName --account-name $targetStorageAccountName --query "[:1].value" -o tsv)

storageSasToken=$(az storage account generate-sas --expiry 2017-05-02'T'12:00'Z' --permissions aclrpuw --resource-types sco --services b --https-only --account-name $targetStorageAccountName --account-key $targetStorageAccountKey -o tsv)

az storage container create -n $imageStorageContainerName --account-name $targetStorageAccountName --sas-token $storageSasToken

# Copy the snapshot to the target region using the SAS URL
imageBlobName = "$imageName-osdisk.vhd"
copyId=$(az storage blob copy start --source-uri $snapshotSasUrl --destination-blob $imageBlobName --destination-container $imageStorageContainerName --sas-token $storageSasToken --account-name $targetStorageAccountName)

# Figure out when the copy is destination-container
# TODO: Put this in a loop until status is 'success'
az storage blob show --container-name $imageStorageContainerName -n $imageBlobName --account-name $targetStorageAccountName --sas-token $storageSasToken --query "properties.copy.status"

# Get the URI to the blob
blobEndpoint=$(az storage account show -g $ResourceGroupName -n $targetStorageAccountName --query "primaryEndpoints.blob" -o tsv)
osDiskVhdUri="$blobEndpoint$imageStorageContainerName/$imageBlobName"

# Create the snapshot in the target region
snapshotName="$imageName-$targetLocation-snap"
az snapshot create -g $ResourceGroupName -n $snapshotName -l $targetLocation --source $osDiskVhdUri

Create an Image (in target subscription)

Once the snapshot has been copied to the target Azure subscription, the next step is to use the snapshot as a basis for creating a new managed image. Be sure to proceed to the next step (Create a temporary VM from the Image) don’t stop here!

PowerShell

<# -- In the second subscription, create a new Image from the copied snapshot --#>
Select-AzureRmSubscription -SubscriptionId $targetSubscriptionId

$snap = Get-AzureRmSnapshot -ResourceGroupName $resourceGroupName -SnapshotName $snapshotName

$imageConfig = New-AzureRmImageConfig -Location $destinationRegion

Set-AzureRmImageOsDisk -Image $imageConfig `
                        -OsType Windows `
                        -OsState Generalized `
                        -SnapshotId $snap.Id

New-AzureRmImage -ResourceGroupName $resourceGroupName `
                 -ImageName $imageName `
                 -Image $imageConfig

Azure CLI 2.0

az account set --subscription $TargetSubscriptionID
snapshotId=$(az snapshot show -g $ResourceGroupName -n $snapshotName --query "id" -o tsv )
az image create -g $ResourceGroupName -n $imageName -l $location --os-type Windows --source $snapshotId

Optional: create a temporary VM from the Image (in target subscription)

Earlier, when you created the snapshot and copied it to the target Azure subscription, you may have noticed the process went relatively quick. One reason for this is how Azure copies the data – it uses a copy-on-read process. Meaning, the full dataset isn’t copied until it is needed. To trigger the data to be fully copied, a VM can be created. The VM created in this step is used to trigger the data transfer, and then we can safely delete the VM and snapshots. This step can be considered optional – as the first time a VM is created from the snapshot, doing so will ensure the data is fully copied.

In the example below, I’m using an ARM template to provision the new (temporary) VM. This template is very similar to this one, expect that I’ve modified the one I’m using to allow for the use of a managed image.

PowerShell

<# -- In the second subscription, create a new VM from the new Image. -- #>
$currentDate = Get-Date -Format yyyyMMdd.HHmmss
$deploymentLabel = "vmimage-$currentDate"

$image = Get-AzureRmImage -ResourceGroupName $resourceGroupName -ImageName $imageName

<# -- Get a random series of letters to help with making a somewhat unique DNS suffix. -- #>
$dnsPrefix = "myvm-" + -join ((97..122) | Get-Random -Count 7 | ForEach-Object {[char]$_})

$creds = Get-Credential -Message "Enter username and password for new VM."

$templateParams = @{
    vmName = $vmName;
    adminUserName = $creds.UserName;
    adminPassword = $creds.Password;
    dnsLabelPrefix = $dnsPrefix
    managedImageResourceId = $image.Id
}

# Put the dummy VM in a separate resource group as it makes it super easy to clean up all the extra stuff that goes with a VM (NIC, IP, VNet, etc.)
$rgNameTemp = $resourceGroupName + "-temp"
New-AzureRmResourceGroup -Location $region `
                         -Name $rgNameTemp

New-AzureRmResourceGroupDeployment  -Name $deploymentLabel `
                                    -ResourceGroupName $rgNameTemp `
                                    -TemplateParameterObject $templateParams `
                                    -TemplateUri 'https://raw.githubusercontent.com/mcollier/copy-azure-managed-disk-images/master/azuredeploy.json' `
                                    -Verbose

Azure CLI 2.0

az group create -l $location -n $resourceGroupTempName
imageId=$(az image show -g mcollier-managed-image -n image2 --query "id")
az group deployment create -g resourceGroupTempName --template-uri https://raw.githubusercontent.com/mcollier/copy-azure-managed-disk-images/master/azuredeploy.json --parameters "{\"vmName\":{\"value\": \"$vmName\"}, \"adminUsername\":{\"value\": \"$user\"}, \"adminPassword\":{\"value\": \"$pwd\"}, \"dnsLabelPrefix\":{\"value\": \"$dnsPrefix\"}, \"managedImageResourceId\":{\"value\": \"$imageId\"}}"

Delete the snapshots

Since the new image and temporary VM have been created in the target subscription, there is no longer a need for the snapshot. You would want to delete the snapshot in both the source and target Azure subscription.

PowerShell

<# -- Delete the snapshot in the second subscription -- #>
Remove-AzureRmSnapshot -ResourceGroupName $resourceGroupName -SnapshotName $snapshotName -Force

Azure CLI 2.0

az snapshot delete -g $resourceGroupName -n $snapshotName

Delete the temporary VM

Earlier there was a step to create a temporary VM. The VM was created to trigger the data copy process. It serves no other purpose at this point, thus it is safe to delete. If you followed the earlier steps to create the temporary VM, it was created in its own resource group. Thus, simply delete the resource group.
PowerShell

Remove-AzureRmResourceGroup -Name $rgNameTemp -Force

Azure CLI 2.0

az group delete -n $resourceGroupName

Summary

As you can see, there are several steps necessary to copy a Managed Disk Image from one Azure subscription to another. The steps aren’t difficult, just a bit unintuitive at this point. This post should help make the process a bit easier to understand. When the ability to copy images across subscriptions and regions is available as a first-class feature in Azure (hopefully later this year), this post will be effectively obsolete. I’m OK with that.  🙂

Resources / More Information

 

I would like to thank Chetan Agarwal and Neil Mackenzie for their assistance in reviewing this post.

Creating a Custom SQL Server VM Image in Azure

Recently I had the opportunity to work on a project were I needed to create a custom SQL Server image for use with Azure VMs.  The process was a little more challenging than I initially anticipated.  I think this is mostly because I was not familiar with the process of preparing a SQL Server image.  Perhaps this isn’t much of a challenge for an experienced SQL Server DBA or IT Pro.  For me, it was a great learning experience.

Why a Custom SQL Server Image?

The Azure VM image gallery already contains a SQL Server image.  It’s very easy to create a new SQL Server VM using this image.  However, doing so has a few important trade-offs to consider:

  • Unable to fully customize the base install of SQL Server.  This is a template/image after all – you get a VM configured the way the image was configured.
  • Unable to use your own SQL Server license.  If your company has an Enterprise Agreement (EA) with Microsoft, it’s likely there is already some SQL Server licenses built into that agreement.  Depending on the details, it may be significantly cheaper to use the licenses from the EA instead of paying the SQL Server VM image upcharge from Azure.

The Basic Steps

There are 6 basic steps to creating a custom SQL Server VM image for use in Azure.

  1. Provision a new base Windows Server VM
  2. Download the SQL Server installation media
  3. Run SQL Server setup to prepare an image
  4. Configure Windows to complete the installation of SQL Server
  5. Capture the image and add it to the Azure VM image gallery
  6. Create a new VM instance using the custom SQL Server image

The basic idea here is to create a base VM, customize it with a SQL Server image, capture the VM to create an image, and then provision new VMs using that captured VM image.

Create_SQL_VM_Image_Azure 2

Let’s dive into each of these in a little more detail.

Note: the terminology here can be a little confusing. When referring to the VM used to create the template/image, I’ll use the term “base VM”. When referring to the VM created from the base VM, I’ll use the term “VM instance”.

1. Provision a new base Windows Server VM

There are multiple ways to create a Windows Server VM in Azure.  Creating a VM via the Azure management portal and PowerShell are probably the two most popular options.  Be sure to check out this tutorial to learn how to do so via the portal. For the purposes of this post, I’ll do so via PowerShell.

$img = Get-AzureVMImage `
	| where { ( $_.PublisherName -ilike "Microsoft*" -and $_.ImageFamily -ilike "Windows Server 2012 Datacenter" ) } `
	| Sort-Object -Unique -Descending -Property ImageFamily `
	| sort -Descending -Property PublishDate `
	| select -First(1)

$vmConfig = New-AzureVMConfig -Name "sql-1" -InstanceSize Small -ImageName $img.ImageName |
    Add-AzureProvisioningConfig -Windows -AdminUsername "[admin-username-here]" -Password "[admin-password-here]" 

New-AzureVM -ServiceName "SQLServerVMTemplate" -VMs $vmConfig -Location "East US" -WaitForBoot

 2. Download the SQL Server installation media

With the base Windows Server 2012 VM created, we can now get ready to prepare (sysprep) the SQL Server installation.  To do that, we need to get the SQL Server installation media onto the machine.  The easiest way I found to do this was to leverage Azure blob storage.

  1. Upload the SQL Server ISO file to Azure blob storage
  2. Remote Desktop (RDP) into the base VM
  3. From the VM, download the SQL Server ISO file to the local disk
  4. Mount the SQL Server ISO file to the VM
  5. Copy the ISO contents (not the ISO file itself) to the VM’s C:\ drive.  For example, use C:\sql

The SQL Server installation media files need to be copied to the local C: drive so it can be used later to complete the SQL Server installation (when provisioning the actual SQL Server VM instance).

3. Run SQL Server setup to prepare an image

In order to prepare the (sysprep’d) SQL Server VM image (which we can use as a template for future VMs), we need to run the SQL Server installation and instruct it to prepare an image – not run the full installation.  An easy way to do this is with a SQL Server configuration file, an example of which I’ve included below.

ConfigurationFile.ini

;SQL Server 2012 Configuration File
[OPTIONS]
; Specifies a Setup workflow, like INSTALL, UNINSTALL, or UPGRADE. This is a required parameter.
ACTION="PrepareImage"
; Detailed help for command line argument ENU has not been defined yet.
ENU="True"
; Parameter that controls the user interface behavior. Valid values are Normal for the full UI, AutoAdvance for a simplified UI, and EnableUIOnServerCore for bypassing Server Core setup GUI block.
;UIMODE="Normal"
; Specifies setup not display any user interface.
;QUIET="False"
; Specifies setup to display progress only, without any user interaction.
QUIETSIMPLE="True"
; Specifies whether SQL Server Setup should discover and include product updates. The valid values are True and False or 1 and 0. By default SQL Server Setup will include updates that are found.
UpdateEnabled="True"
; Specifies features to install, uninstall, or upgrade. The list of top-level features include SQL, AS, RS, IS, MDS, and Tools. The SQL feature will install the Database Engine, Replication, Full-Text, and Data Quality Services (DQS) server. The Tools feature will install Management Tools, Books online components, SQL Server Data Tools, and other shared components.
FEATURES=SQLENGINE
; Specifies the location where SQL Server Setup will obtain product updates. The valid values are "MU" to search Microsoft Update, a valid folder path, a relative path such as .\MyUpdates or a UNC share. By default SQL Server Setup will search Microsoft Update or a Windows Update service through the Window Server Update Services.
UpdateSource="MU"
; Displays the command line parameters usage
HELP="False"
; Specifies that the detailed Setup log should be piped to the console.
INDICATEPROGRESS="False"
; Specifies that Setup should install into WOW64. This command line argument is not supported on an IA64 or a 32-bit system.
X86="False"
; Specifies the root installation directory for shared components.  This directory remains unchanged after shared components are already installed.
INSTALLSHAREDDIR="C:\Program Files\Microsoft SQL Server"
; Specifies the root installation directory for the WOW64 shared components.  This directory remains unchanged after WOW64 shared components are already installed.
INSTALLSHAREDWOWDIR="C:\Program Files (x86)\Microsoft SQL Server"
; Specifies the Instance ID for the SQL Server features you have specified. SQL Server directory structure, registry structure, and service names will incorporate the instance ID of the SQL Server instance.
INSTANCEID="MSSQLSERVER"
; Specifies the installation directory.
INSTANCEDIR="C:\Program Files\Microsoft SQL Server"

There are two steps in this process:

  1. Copy the ConfigurationFile.ini file (from your local PC) to the same location as the SQL Server installation media (i.e. c:\sql) on the base VM.
  2. Run SQL Server setup to prepare an image.  From a command prompt (on the base VM), navigate to the C:\sql folder and then execute the following command:
Setup.exe /ConfigurationFile=ConfigurationFile.ini /IAcceptSQLServerLicenseTerms=true

 4. Configure Windows to complete the installation of SQL Server

At this point the base VM should have an “installation” of SQL Server that is not fully completed. The SQL Server bits are in place, but they’re not configured for a full server install . . . at least not yet. The final configuration of SQL Server will take place when the VM instance (of which this template/image is the base) is provisioned and boots up for the first time. This is accomplished by using a CMD file with the following content:

@ECHO OFF && SETLOCAL && SETLOCAL ENABLEDELAYEDEXPANSION && SETLOCAL ENABLEEXTENSIONS
REM All commands will be executed during first Virtual Machine boot
"C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\SQLServer2012\setup.exe" /QS /ACTION=CompleteImage /INSTANCEID=MSSQLSERVER /INSTANCENAME=MSSQLSERVER /IACCEPTSQLSERVERLICENSETERMS=1 /SQLSYSADMINACCOUNTS=%COMPUTERNAME%\Administrators /BROWSERSVCSTARTUPTYPE=AUTOMATIC /INDICATEPROGRESS /TCPENABLED=1 /PID="[YOUR-SQL-SERVER-PRODUCT-ID-HERE]"
  1. On your local PC, save the file as SetupComplete2.cmd
  2. RDP / log into the base VM
  3. Copy the SetupComplete2.cmd from your local PC file to the c:\Windows\OEM folder on the base VM
  4. Change the value for the SQLSYSADMINACCOUNTS value to be that of the administrative account created on the VM (or better yet – the local Administrators group account)
  5. If needed, supply the SQL Server product ID (PID) value.

When Windows starts on the new VM instance for the first time, the SetupComplete2.cmd file should automatically run.  It is invoked by the SetupComplete.cmd file already on the machine.

vm-image-setupcomplete

5. Capture the image and add it to the Azure VM image gallery

At this point a base SQL Server VM has been created and the groundwork laid to complete the install. Now it is time to create the VM image from the base VM, and do to that you sysprep and capture the base VM.  Please follow the guide on How to Capture a Windows Virtual Machine to Use as a Template.

6. Create a new VM using the custom SQL Server image

With a new custom VM image template available in the VM image gallery, you can provision a new VM instance using that custom template.  Upon first boot, the newly provisioned VM should complete the full SQL Server installation as laid out in your SetupComplete2.cmd file.  Please follow the guide on How to Create a Custom Virtual Machine for more information on creating the VM from the template.

 

Closing Thoughts

One of the quirks I noticed when preparing the base SQL Server image is that it was not possible to prepare the image with SQL Server Management Studio (SSMS).  I would have to do the install after the newly provisioned VM instance is created. Not hard, but time consuming (an annoying if doing this on multiple VM instances).  I later learned that SQL Server 2012 Cumulative Update 1 does allow for preparing a SQL Server image with SSMS installed.  I’ve included a link below that describes the process for creating a SQL Server image with CU1.

In the end, this process really is not all that hard.  Time consuming?  Yes!  The worst part (at least for me) was really just understanding how the SQL Server installation and sysprep process works.  Once I wrapped my head around that, the process was a lot smoother.

 

Helpful Resources

While I was learning how to create a custom SQL Server VM image, the following resources were very helpful:

 

I would like to thank Scott Klein for his assistance in verifying these steps.  His help was extremely valuable to ensure I was doing this the right way.

Windows Server 2012 available for Windows Azure!

Did you know that Windows Server 2012 is available for Windows Azure Virtual Machines?  It is!  This is a great way to take the new OS for a quick test spin and prove out some ideas or new applications.

Be sure to check out https://www.windowsazure.com/en-us/manage/windows/ to learn more about Windows Azure Virtual Machines.

image

Sign up for a FREE 90-day Windows Azure trial at http://bit.ly/MikeAzureTrial.