Azure, PowerShell Kieran Jacobsen Azure, PowerShell Kieran Jacobsen

Azure Automation State Configuration – Installing Common DSC Modules

Azure Automation State Configuration (previously Azure Automation DSC), is a service provided by Azure that allows you to write, manage and compile PowerShell Desired State Configuration (DSC) configurations and assign these configurations to target virtual machines (or any server or workstation to be honest). On its own, State Configuration provides some basic configuration examples, but its true power comes form the ability for you to define your own configurations.

Azure Automation State Configuration

Azure Automation State Configuration (previously Azure Automation DSC), is a service provided by Azure that allows you to write, manage and compile PowerShell Desired State Configuration (DSC) configurations and assign these configurations to target virtual machines (or any server or workstation to be honest). On its own, State Configuration provides some basic configuration examples, but its true power comes form the ability for you to define your own configurations.

User created configurations need to be imported and compiled before the configuration is applied to a virtual machine. It is common when creating DSC configurations to rely on a variety of DSC Resources. The modules containing these resources need to be imported into Azure Automation for it to be able to compile any configuration. So how do we import and update these modules?

The first mechanism to manage the PowerShell modules is through the Azure Portal, by going to your Automation Account > Shared Resources > Modules. Here you can add modules from a zip file, update the built in Azure modules, or add modules from the PowerShell Gallery. This provides a simple mechanism particularly if you are starting; unfortunately, most production configurations will need multiple modules, making the Azure Portal difficult to use.

We can also manage modules via ARM templates. Define the template is relatively straight forward but requires a couple of tricks to get started. I will put together a separate blog post on how you can define your own ARM template.

Accelerating Adoption

I wanted to help those starting out with State configuration. My goal was to help accelerate new State Configuration deployments, by creating a “starter” ARM template that would install the most common PowerShell DSC modules that are in the PowerShell Gallery and the most common in my production DSC configurations. This ARM template could also be used to ensure that the modules are also updated on a regular basis.

So what modules are included? I selected 32 modules in the end. These cover the configuration of core Windows, Windows Server roles and features, security hardening, package management and Chocolatey. Most of the modules are maintained by Microsoft, however four modules, cChoco, cSpeculationControlFixes, UpdateServicesDSC and xSystemSecurity that are maintain by members of the community.

Module Name Author Description
ActiveDirectoryCSDsc Microsoft Corporation This DSC Resource module can be used to install or uninstall Certificate Services components in Windows Server.
AuditPolicyDsc Microsoft Corporation The AuditPolicyDsc module allows you to configure and manage the advanced audit policy on all currently supported versions of Windows.
cChoco Chocolatey Software Lawrence Gripper Javy de Koning Chocolatey DSC Resources for use with internal packages and the community package repository. Learn more at http://chocolatey.org/
CertificateDsc Microsoft Corporation This module includes DSC resources that simplify administration of certificates on a Windows Server
ComputerManagementDsc Microsoft Corporation The ComputerManagementDsc module is originally part of the Windows PowerShell Desired State Configuration (DSC) Resource Kit. This version has been modified for use in Azure. This module contains the xComputer and xDisk resources. These DSC Resources allow you to perform computer management tasks, like joining a domain or initializing disks.
cSpeculationControlFixes Kieran Jacobsen PowerShell DSC for enabling Speculation Control fixes on Windows Server
DFSDsc Microsoft Corporation DSC resources for configuring Distributed File System Replication and Namespaces.
GPRegistryPolicy Microsoft Corporation Module with cmdlets to work with GP Registry Policy .pol files
GPRegistryPolicyParser Microsoft Corporation Module with parser cmdlets to work with GP Registry Policy .pol files
NetworkingDsc Microsoft Corporation Module with DSC Resources for Networking area
PackageManagementProviderResource Microsoft Corporation Module with DSC resources for the package management.
PSDscResources Microsoft Corporation This module contains the standard DSC resources. Because PSDscResources overwrites in-box resources, it is only available for WMF 5.1. Many of the resource updates provided here are also included in the xPSDesiredStateConfiguration module which is still compatible with WMF 4 and WMF 5 (though that module is not supported and may be removed in the future).
SecurityPolicyDsc Microsoft Corporation This module is a wrapper around secedit.exe which provides the ability to configure user rights assignments
SqlServerDsc Microsoft Corporation Module with DSC Resources for deployment and configuration of Microsoft SQL Server.
StorageDsc Microsoft Corporation This module contains all resources related to the PowerShell Storage module, or pertaining to disk management.
UpdateServicesDsc Michael Greene Module with DSC Resources for deployment and configuration of Windows Server Update Services.
WindowsDefender Microsoft Corporation Windows Defender module allows you to configure Windows Defender settings.
xActiveDirectory Microsoft Corporation The xActiveDirectory module is originally part of the Windows PowerShell Desired State Configuration (DSC) Resource Kit. This version has been modified for use in Azure. This module contains the xADDomain, xADDomainController, xADUser, and xWaitForDomain resources. These DSC Resources allow you to configure and manage Active Directory.
xDhcpServer Microsoft Corporation Module with DSC Resources for DHCP Server area
xDismFeature Microsoft Corporation Module with DSC Resources for Deployment Image Servicing and Management features.
xDnsServer Microsoft Corporation Module with DSC Resources for DNS Server area
xFailOverCluster Microsoft Corporation Module containing DSC resources used to configure FailOver Clusters.
xInternetExplorerHomePage Microsoft Corporation This DSC Resources can easily set an URL for the home page of Internet Explorer
xPendingReboot Microsoft Corporation This module identifies pending reboots in Windows Server and acts on them.
xPSDesiredStateConfiguration Microsoft Corporation The xPSDesiredStateConfiguration module is a part of the Windows PowerShell Desired State Configuration (DSC) Resource Kit, which is a collection of DSC Resources produced by the PowerShell Team. This module contains the xDscWebService, xWindowsProcess, xService, xPackage, xArchive, xRemoteFile, xPSEndpoint and xWindowsOptionalFeature resources.
xRemoteDesktopAdmin Microsoft Corporation Module with DSC Resources for enabling administrative Remote Desktop Connections
xSmbShare Microsoft Corporation Module with DSC Resources for SmbShare area
xSystemSecurity Arun Chandrasekhar Handles Windows related security settings like UAC and IE ESC. xUAC enables or disables the User Account Control prompt, while xIEEsc enables or disables IE Enhanced Security Configuration.
xTimeZone Microsoft Corporation This DSC Resources can easily set the System Time Zone.
xWebAdministration Microsoft Corporation Module with DSC Resources for Web Administration
xWindowsEventForwarding Microsoft Corporation This module can be used to manage configuration of a Windows Event Forwarding server in a Collector role.
xWindowsUpdate Microsoft Corporation Module with DSC Resources for Windows Update
xWinEventLog Microsoft Corporation Configure Windows Event Logs

Want to get started and use the template?

You will need an Azure Automation account to start.

You can then use the Deploy to Azure button on the Git Repository or download the latest release and deploy using Azure CLI or PowerShell.

What to contribute?

The project is up on GitHub and I welcome everyone to make suggestions and recommendations. If you need a had with your first PR, check out this guide from egghead.io. I have connected this project to Azure DevOps build pipelines so all PRs will be validated for any issues.

Read More
Azure, PowerShell Kieran Jacobsen Azure, PowerShell Kieran Jacobsen

Securing PowerShell DSC within Azure ARM Templates

There is no doubt that Azure ARM templates are now the best way to deploy and manage resources within Azure. Recently, I found myself creating an ARM template that deployed some Windows Server 2016 virtual machines with PowerShell DSC extension to configure them. This is typically very simple, define the virtual machine include the extension, however, this time I also needed to include sensitive pieces of information; API keys and credentials, and then I wondered, well, what is the best way to protect these?

If you are familiar with DSC, you can either leave these assets in plaintext or encrypt them with a certificate. Now most documentation and posts found online, will direct for the plaintext approach when using the DSC extension, but there is a significantly better approach, which allows for sensitive information to be encrypted throughout the deployment process. Even better, it is really simple to implement.

Whilst I was working through setting this up, I found there are some quirks in how you need to put the ARM template together, and thought it would be a good topic for a post. I am going to talk about some of the challenges that I found, as well as provide some working examples.

Understand the apiVersion and typeHandlerVersion

When you define the PowerShell DSC extension for an Azure virtual machine, there are two fields that control the version and functionality that the extension will provide, apiVersion and typeHandlerVersion.

In terms of ARM Templates, the apiVersion specifies how the deployment process talks to Azure, that is the layout and content of calls that are made to the underlying API, as such, it controls what we can place into out template files. Microsoft uses the apiVersion to provide backward compatibility. When breaking changes are introduced, typically as part of introducing new features, Microsoft will define a new version. This allows for older scripts and templates to continue to function long after they have been written. For the PowerShell DSC extension, the apiVersion will impact what fields are valid in your template definition.

The second, typeHandlerVersion, is extremely critical, it defines what version of the extension will be deployed to the virtual machine and then executed. Whilst apiVersion controls talking to API, the typeHandlerVersion controls what is happening on the virtual machine itself. This extension regularly receives new features and bug fixes, from support for new operating systems, privacy settings, to WMF versions, and a tonne of issues fixed in between.

Now for the catch, where the devil enters the details. Depending upon what versions you specify, your template syntax may need to change. I found out about these quirks the hard way, this tiny detail can cost you quite a bit of time troubleshooting syntax issues. A mismatch between the apiVersion¸ typeHandlerVersion and your template’s syntax could leave to errors during template validation, DSC compilation or DSC application.

So, what happens if you use older version or mix versions and syntax? Well, nothing bad might happen, or you might end up with either template validation errors, DSC complication errors, or even errors during DSC processing (that is, as the DSC is applied to the server).

But what about autoUpgradeMinorVersion?

According to the documentation, the extensions have a Boolean attribute, autoUpgradeMinorVersion, that allows a user to pick simply the major version, and have Azure install the latest version to their virtual machine during provisioning time. It should be highlighted that this only happens during the provisioning of the extension, extensions will not be upgraded unless the user explicitly removes and then re-provisions the extension. Hot fixes (that is, those of the format of 2.9.x.x), are automatically selected, you don’t get any control. The problem is, this only upgrades the extension, it isn’t going to impact your ARM template syntax.

Be Wary of Visual Studio Created Templates

If you are developing your ARM templates within Visual Studio, you are probably using the “Add New Resource” window to include the PowerShell DSC Extension into your virtual machine. It so happens, that, at least in Visual Studio 2017, it will default to an old typeHandlerVersion of 2.9. For most, this isn’t a problem, especially if autoUpgradeMinorVersion is set to true.

I do want to point out that versions 2.4 up to 2.13 were retired in August 2016.

If you do need to handle sensitive data, then work with the release history for the PowerShell DSC extension, and the documentation on the syntax to ensure that you are working against the latest versions.

How to correctly handle sensitive data?

Note/Warning: At the time of writing, this all worked correctly for versions 2.24 to 2.26. It may not be correct for later versions.

So how do we correctly pass sensitive data between our template through to our virtual machine and DSC?

To begin with, you need to understand that within the properties for the DSC extension, we have two attributes that can be used to specify settings, the first, settings, which you are probably familiar with. There is another, protectedSettings, you might already be familiar with this one, typically you will see configurationUrlSasToken, but you can also specify sensitive data to be passed to the DSC configuration here as well.

So, I have a super simple DSC configuration, it will create a user account using the specified details. I have opted to provide the accounts username and password using a PSCredential parameter, whilst I am also going to set a description for the account, specified as a string parameter.

Configuration Main
{
    Param
    (
        [Parameter(Mandatory=$true)]
        [ValidateNotNullorEmpty()]
        [PSCredential]
        $Credential,

        [Parameter(Mandatory=$true)]
        [ValidateNotNullorEmpty()]
        [String]
        $AccountDescription
    )

    Node Localhost
    {
        User NewUser
        {
            UserName             = $Credential.UserName
            Description          = $AccountDescription
            Disabled             = $false
            Ensure               = 'Present'
            Password             = $Credential.Password
            PasswordNeverExpires = $true
        }
    }
}

Obviously, I want to ensure that credential is kept securely, I don’t want it left in plaintext as part of the DSC compilation.

The ARM template looks like this:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "VirtualMachineName": {
            "type": "string",
            "metadata": {
                "description": "Name of the virtual machine"
            }
        },
        "Username": {
            "type": "string",
            "metadata": {
                "description": "Account Username"
            }
        },
        "Password": {
            "type": "securestring",
            "metadata": {
                "description": "Account Password"
            }
        },
        "DSCPackagePath": {
            "type": "string",
            "metadata": {
                "description": "Path to DSC Package"
            }
        },
        "DSCPackageSasToken": {
            "type": "securestring",
            "metadata": {
                "description": "Sas Token"
            }
        }
    },
    "variables": {
        "AccountDescription": "Account Created as part of DSC"
    },
    "resources": [
        {
            "name": "[concat(parameters('VirtualMachineName'),'/Microsoft.Powershell.DSC')]",
            "type": "Microsoft.Compute/virtualMachines/extensions",
            "location": "[resourceGroup().location]",
            "apiVersion": "2016-03-30",
            "dependsOn": [
                "[concat('Microsoft.Compute/virtualMachines/', parameters('VirtualMachineName'))]"
            ],
            "properties": {
                "publisher": "Microsoft.Powershell",
                "type": "DSC",
                "typeHandlerVersion": "2.24",
                "autoUpgradeMinorVersion": true,
                "protectedSettings": {
                    "configurationUrlSasToken": "[parameters('DSCPackageSasToken')]",
                    "configurationArguments": {
                        "Credential": {
                            "Username": "[parameters('Username')]",
                            "Password": "[parameters('Password')]"
                        }
                    }
                },
                "settings": {
                    "configuration": {
                        "url": "[parameters('DSCPackagePath')]",
                        "script": "MyDsc.ps1",
                        "function": "Main"
                    },
                    "configurationArguments": {
                        "AccountDescription": "[variables('AccountDescription')]"
                    }
                }
            }
        }
    ]
}

See the cool shortcut I have done in the protectedSettings? See how the Credential is built within the template, I provide the username and password and the Azure takes care of the rest and DSC just gets a PSCredential. Pretty neat!

It is worth highlighting that my non-sensitive data, like the AccountDescription, are still found within the settings attribute, I have only moved my sensitive data to protectedSettings. Of course, this decision is entirely up to you.

Now, it might just have been my paranoia, but I found that things when much more smoothly if I placed the protectedSettings, before settings. It shouldn’t matter, but things just seemed happier, and I know that doesn’t sound very logical.

If you have organisational privacy concerns

It is worth noting that if you have strict organisational privacy or data sharing controls, you should disable data collection, 

"settings": {
    "privacy": {
        "DataCollection": "Disable"
    }
}

In Summary

The two examples show here are up on GitHub, I have also included a more detailed example as a Visual Studio 2017 project.

Kieran Jacobsen

PS. I am still looking for talented individuals to join my team at Readify. I am after people with a passion for infrastructure, Azure, operations, and everything in between. Does that sound like you? Hit me up on Twitter or LinkedIn!

Read More
PowerShell Kieran Jacobsen PowerShell Kieran Jacobsen

How an issue with PowerShell DSC in WMF 5 cost us $5526.95

Short Version

  • PowerShell DSC in WMF 5.0 can lead to a significant increase in IO operations.
  • Operations will increase over time.
  • Fix is included in WMF 5.1, and a workaround is available.

With the move to the cloud, we often don’t consider how simple application bugs could lead to large infrastructure bills, however in this case, a minor issue with WMF 5.0 lead to a IO usage bill of $5526.95 over just under 6 months.

Background

In early December, as part of a regular review of Readify’s Azure usage, we noticed a spike in the usage for one of our Azure Subscriptions. The subscription accounts for a significant portion of our usage, and is used internally.

Looking closely at our usage break down, we started to see something that was very unusual, Data Management Geo Redundant Standard IO - Page Blob Write, had taken 2nd place, overtaking a D13v2 virtual machine that runs 24/7. These write operations are billed in units of 100000000 operations leading to the realisation that we had a significant IO consumer within one of our Azure Storage accounts. We quickly narrowed this down to two storage accounts housing the majority of our IaaS deployments.

Digging Deeper

In terms of IaaS, we don’t have anything deployed that would be considered particularly IO intensive, well, not to this level. We just have some AD DCs, ADFS servers, VPN, VSTS build and some Azure Automation Hybrid workers. Nothing was jumping out as an obvious culprit.

Looking at some metrics for the storage account, two stood out as being unusually high; Total Requests and Total Egress.

I started to look at the usage reports produced by the Azure EA portal. It appeared at first that the usage increased at around September. At first I thought it could have been something to do with some work around Azure Recovery Services, either some replication or backups, but after a few days, it was obvious this wasn't the cause of our issues.

Calling Support

At this point, I raised a support case with the Azure team.

I just want to say a massive thank you to the Azure Support team, they did an amazing job in resolving the issue. They started off just as perplexed as I was. We went through the workloads, and nothing presented as an obvious cause of our issues.

We went back through the usage reports in the EA portal. Looking back further, we determine the usage increased in July. Had we implemented anything in July? At this point, my mind drew a complete blank.

The engineer offered to see if the back-end engineering teams might be able to help us narrow down the usage, hopefully leading us to the VHD(s), and thus the virtual machine that was driving this usage.

The next day, they sent through an excel spreadsheet. The main culprits were our DCs and ADFS servers. These had been around for over a year, and had very little changes. I thanked support and promised I would spend some time studying the machines and get back to them if I found something.

Staring at Resource Monitor

I fired up Resource Monitor on one of the machines, and spent the next hour watching the disk usage. After almost going cross-eyed, I saw it, it was hard to believe, but I saw something that wasn’t quite right. I saw huge write cycles from the WMI Host process to C:\Windows\System32\config\systemprofile\AppData\Local\Microsoft\Windows\PowerShell\CommandAnalysis\PowerShellAnalysisCacheEntry*. Could that be the cause?

I quickly looked on other virtual machines, and saw similar patterns. Things were pointing to these files, but what were they? What was the WMI Host doing?

The cause is found!

I decided the first thing to do is find out what those files were, so I opened my favourite search engine, Duck Duck Go, and entered _wmi host PowerShellAnalysisCacheEntry, and hit search. The first two results, get-psdrive & abnormal I/O and Topic: Dsc v5 get-psdrive abnormal I/O leapt out.

That is when it hit me. We implemented Azure Automation DSC in July. Crap!

I started with the second result, Topic: Dsc v5 get-psdrive abnormal I/O, on PowerShell.org. In the post, from May, the user Brik Brac talks about seeing a major I/O performance issue using DSC v5.

Brik Brac also created the post on the Windows Server User voice, get-psdrive & abnormal I/O, in this post, Zachary Alexander a member of the PowerShell team acknowledged the issue, and posted up a workaround. He also posted that this issue would be fixed in WMF 5.1. WMF 5.1 will be released sometime in January 2017.

Simply put, the issue lies in how PowerShell discovers modules to import when performing certain PowerShell PS Drive Operations, and this heavily impacts DSC. Each time DSC runs, be it monitoring or applying configuration, the number of write operations on these cache files increases, left unchecked, the growth is dramatic. The files don't grow much in size, just the number of write operations to them. You can read more on auto discovery here, How Module Command Discovery Works in PSv3.

xRegistry is one of the resources impacted the most by this issue and as part of our DSC, we use xRegistry resources to harden the SSL configuration on our servers. In fact, this was one of the first reasons we moved to DSC.

I quickly emailed the support engineer I was working with, after a quick phone call, we felt confident that we had found the source of our issues. Now we just needed to prove it.

Resolving the issue

There were two ways we could resolve the issue. Firstly, I opted to install the WMF 5.1 preview onto systems where possible; secondly; we implemented the work around, much like outlined by Zachary in the User Voice post that clears the cache files.

Thankfully, implementing these steps was quite simple, but then we had a nervous wait to see if we had fixed the issue.

I started to see the Total Requests drop over the course of the day whilst I installed WMF 5.1 and ensured DSC was appropriately applied using Azure Automation DSC. By the end of the following day, the Total Requests had completely bottomed out, from over 35M to several thousand.

Ensuring this does not repeat

There are two ways to ensure that something like this doesn’t happen again.

First, implement threshold alerts for your Storage accounts. We didn’t have any defined, and since have. I strongly recommend that everyone ensures that they have alerts for the Total Requests metric.

Next, check your server monitoring. We had monitoring for CPU, memory usage, and free disk space, but I didn’t have any monitoring of disk IO. You can do this in the Azure Portal, or via OMS. If I had this in place, I would have picked up this issue much earlier. The other thing I will be doing is ensuring that our server and storage account deployment templates include basic performance alerting.

More Information

Read More
PowerShell Kieran Jacobsen PowerShell Kieran Jacobsen

PowerShell 5, DSC, SABnzbd and Sonarr (Part 1)

I have been trying to do a lot more with PowerShell DSC of late, including writing my own DSC resources.

Last week, I decided to write some resources for SABnzbd and Sonarr. Why? Well firstly for practice, but also because no one else has looked at building up packages for these apps (there wasn’t even Chocolatey packages).

I am not going to go into detail on implementing classes in PowerShell, instead checkout:

SABnzbd

The module cSABnzbd contains a single resource: cSABnzbdInstall. This resource specifying:

  • Ensure: <String> { Ensure | Absent }
  • ServiceCredential: <PSCredential>

If you opt for Ensure to be “Present”, then the resource will install and update SABnzbd. That is right, as new versions are released, it will be automatically updated.

The ServiceCredential allows for us to control what account the SABnzbd services run as. If not specified, LocalHost will be used.

GetLatestVersion()

SABnzbd hosts both its source code and its binary releases in GitHub. This is great! GitHub provides a simple REST API that allows us to very easily and dynamically get the latest release for a project.

If we want to get information on the latest release for a particular repository, we simply perform a get request on https://api.github.com/repos/{owner}/{repo}/releases/latest. The response will contain things like the version name, the release date and the assets in the release. The GitHub API doesn’t need any authentication credentials or tokens; these requests can be performed anonymously.

Performing the API request for sabnzbd the URL for us would be https://api.github.com/repos/sabnzbd/sabnzbd/releases/latest, and the returned information looks like:

The function is pretty simple:

Get()

The Get() function is quite easy to implement. We simply need to call Get-Package, and sets the result against the ensure parameter. We then return the object as required. The result is:

Test()

The Test() is also pretty simple. We use Get-Package again to see if the package is installed or not.

If Ensure = ‘Present’, then we need to check:

  1. If no package was returned, then return false
  2. If it is installed, compare the installed version to the version number returned using GetLatestVersion()

If Ensure = ‘Absent’, then we simply compare $Package to $null.

Set()

Now the Set() function is more complex.

If Ensure = ‘Present’, then we need to:

  1. Find the latest .exe in the release information using GetLatestVersion()
  2. Download the file, this is listed under the assets (look for a .exe).
  3. Silently run setup using the /S switch
  4. Install and start the services (optionally running under the user account specified with ServiceCredential).

If we don’t want SABnzbd installed, then we simply call the uninstall.exe.

Using and Configuring SABnzbd

You should be able to access SABnzbd via http://localhost:8080. When you connect the first time, you will be stepped through the initial setup wizard. I haven’t included any resources for specifying any settings like servers, scheduling or custom folders, you can do this as you would normally do. For more help, try the User Manual.

Getting the module

The module can be found on:

You will need PowerShell 5 or greater installed due to the use of Classes. 

If you encounter any issues, please raise an issue on the GitHub page.

In the next post we will take a look at Sonarr.

Kieran Jacobsen

Read More