Azure Arc is Microsoft’s solution to allow customers to manage on-premises resources (servers and the likes) using Azure Resource Manager, i.e. the Azure Portal, CLI and the Azure PowerShell module. It basically enables on-premise server management in a way that it feels like you’re administrating an Azure Virtual Machine instead of an on-premises server. Note that Azure Arc supports Windows and Linux, but this blog series will only cover aspects of Windows machines.

This post is part of a blog series covering the security implications of an Azure Arc deployment. The first part is a technical deep-dive into the possibilities of an attacker who gained access to the machine itself, and which implications that has for the Cloud environment. This second part of the series deals with technical aspects of what permissions an attacker would require to move laterally the on-premises Azure Arc machine and what traces such actions would leave. The third post sums up all the uncovered aspects and gives general, non-technical advice on how to integrate Azure Arc in existing security concepts and what to be careful about.

TL;DR; Altogether 15 built-in roles were identified to be able to execute code on Azure Arc machines, including inconspicuous roles such as “Guest Configuration Resource Contributor”, “Log Analytics Contributor” or “Azure Extension for SQL Server Deployment”. To prevent any code execution, Azure Arc has a (well hidden) functionality to disable any features that could allow it.

Azure Arc Extensions

One of the selling points of Azure Arc is that it enables Cloud admins to install extensions, which provide easy “[…] post-deployment configuration and automation”.

Features such as “Updates”, “SQL Server Configuration” or “Run command” are also implemented as extensions in the background.

Responsible for the installation and management of extensions on the machine is the “Guest Configuration Extension Service”. The service executable is gc_extension_service.exe and it runs as “NT Authority\SYSTEM”. Unlike the service’s description (“The service installs the requested extensions”) specifies, it is not only used to install extensions, it is also used to interact with them through Azure. Whenever an extension is changed or actions through an extension are triggered, it is actually this service that initiates the action on the machine.

Run Command

The intended way to execute scripts on Azure Arc machines is the “Run command” feature, which is still a preview feature at the time of writing. It is also only implemented through Azure CLI or PowerShell, the GUI just references the available actions.

The implementation allows any authorized admin to execute PowerShell scripts on the machine as “NT Authority\SYSTEM”. The bare minimum parameters to execute a command is as follows. This executes “whoami” on the selected Azure Arc machine, as defined in the SourceScript parameter.

This cmdlet requires both read and write permissions on the corresponding machine, or rather on the runCommands feature. Note that even though the runCommands feature is internally treated as an extension, Microsoft.HybridCompute/machines/extensions/write is not required to use runCommands.

While read and write permissions are required for the cmdlet, the write permission is actually sufficient to execute commands if you use the Azure REST API directly. You will still need to get the parameters (machine name, resource group name, etc.) from somewhere though. Also, you need to provide a storage account URI with a SAS, or you will not be able to obtain the command’s output with only write permissions. That’s not mandatory though.

There are more features on runCommand that are interesting, but the most intriguing for an attacker is SourceScriptUri: passing the parameter with an URI will download the PowerShell script referenced through that URI and execute it. No source restrictions apply, except of course the Azure Arc machine needs to be able to reach it.

Commands executed through runCommand will take several minutes (usually 5-10) to complete and (using the cmdlet) hang until it is completed (if you didn’t specify the flag AsyncExecution). The cmdlet will then provide the commands output or errors as a result.

When watching what’s happening on the machine in the meantime, it can be observed that the “Guest Configuration Extension Service” is responsible for executing the commands, as mentioned in the previous section. The service will launch cmd.exe to call RunCommandHandler.exe, which will handle the actual execution. It will download the source script, either directly from Azure or from the provided URI, store it on disk and execute it in a PowerShell child process. Note that cmd.exe doesn’t have the service as its parent process, I assume because it is executed via WMI.

This executable is located at C:\Packages\Plugins\microsoft.cplat.core.runcommandhandlerwindows\2.0.9\bin\RunCommandHandler.exe and is written in C#, so if you’re interested in its inner workings grab dnSpy and take a look. I didn’t see anything unexpected there though.

CustomScriptExtension

Another means of code execution is through extensions. The most commonly known one is the CustomScriptExtension, though I wouldn’t put it beyond Microsoft that “Azure Extension for SQL Server” also allows it – using a combination of xp_cmdshell and a certain potato, but I haven’t tested that yet. As the permissions to deploy a CustomScriptExtension and “Azure Extension for SQL Server” are identical, this doesn’t matter for this blog post anyway.

The permissions required to install an extension to Azure Arc are Microsoft.HybridCompute/machines/extensions/write using the REST API, and an additional .../read if you are using the Azure PowerShell module (New-AzConnectedMachineExtension, Set-AzConnectedMachineExtension or Update-AzConnectedMachineExtension).

To execute code through a custom script extension using the Azure Arc Virtual Machine Rest API, the following commands can be used:

And to retrieve the output:

Note that while this does not require the /read permission, it requires knowledge of the resource group name and the machine name. The extension name can be chosen arbitrarily, if no CustomScriptExtension extension exists. If there is already one, the exact name of that extension needs to be provided to update it, or the execution will fail. Note that even if another user created the CustomScriptExtension before, you will be able to overwrite it, given the /write permission.

Additionally, this includes the commandToExecute in the protectedSettings, which is intended to store secret values. The result is that I could find no way to view the executed command in any way through Azure (PowerShell and Portal). The forceUpdateTag property is also non-default and is set to allow updating the extension in a way that actually triggers its execution after an update.

The execution on the host will be initiated through the gc_extension_service.exe that will execute the C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\<version>\enable.cmd batch script, which will launch bin\CustomScriptHandler.exe

Machine Configuration

One final method to execute commands on Azure Arc machines is the use of a Machine Configuration. I explained this in another one of my blog posts. Please read this post for any details on execution, permissions, preventions and detections. As a quick summary: Machine Configurations can contain a Desired State Configuration package, which can execute PowerShell code. For Azure Arc machine configurations can be deployed out of the box, as long as you have this permission: Microsoft.GuestConfiguration/guestConfigurationAssignments/write.

Default Roles with Permissions to Execute Code

As detailed in the corresponding section, runCommand requires Microsoft.HybridCompute/machines/runcommands/write to execute code on the Azure Arc machine. Built-in roles with those permissions, at the time of writing, are the following:

In accordance, CustomScriptExtension requires Microsoft.HybridCompute/machines/extensions/write, which is a more common permission than the specific runCommands/write permission. Built-in roles with those permissions, at the time of writing, are the following:

Here are actually a few unexpected roles, like “Log Analytics Contributor” or “Azure Extension for SQL Server Deployment”. This stems from the fact that, as explained in a previous section, multiple features are actually extensions in the background. As the RBAC role model is not suited to tailor permissions for this properly, the unintended side effect is that roles such as “Log Analystics Contributor” can execute code on Azure Arc machines.

Lastly, Machine Configuration require the Microsoft.GuestConfiguration/guestConfigurationAssignments/write permission. Built-in roles with those permissions, at the time of writing, are the following:

To sum it up, the following built-in roles are capable of executing code on Azure Arc machines:

• Owner
• Contributor
• Azure Arc VMware VM Contributor
• Azure Arc VMware Administrator role
• Azure Connected Machine Resource Administrator
• Azure Connected Machine Resource Manager
• Azure Arc ScVmm VM Contributor
• Azure Arc ScVmm Administrator role
• Azure Stack HCI Administrator
• Azure Stack HCI VM Contributor
• Hybrid Server Resource Administrator
• Windows Admin Center Administrator Login
• Guest Configuration Resource Contributor
• Log Analytics Contributor
• Azure Extension for SQL Server Deployment

Mitigation and Hardening

Luckily, Microsoft thought ahead and implemented ways to prevent all forms (known to me) of execution through Azure on Azure Arc machines. Using the command line utility “azcmagent.exe” that is installed on each machine during the Azure Arc onboarding, extensions can be completely disabled, or blocked through an allow or block list. Additionally, Machine Configurations can be completely disabled, too. Note that these configurations can only be made or changed directly on the machine with local administrator permissions. There is no way to do this through Azure. Note that by default none of these hardening measures are applied.

To only allow the “Update” Portal feature:

No extensions can be allowed like this:

Alternatively, disable the extension manager entirely:

Block all guest configurations:

It is then no longer possible to run commands:

Detection

Regarding detection, there are several approaches. Azure tracks an installed extension in the Activity Log as Install or Update an Azure Arc extensions, the assignment of a machine configuration as Write GuestConfigurationAssignments and the execution through Run Command as Install or Update an Azure Arc runcommands. However, there are not much information regarding the actual execution included in these events. Nevertheless, these events are critical and should be monitored tightly for Azure Arc machines.

For all three execution techniques there are IOCs in Azure after execution, as a CustomScriptExtension, the Run Command and the Machine Configuration are all created in Azure and will exist until deleted. However, as the required permission to create them is /write and this permission includes both creation and deletion rights, attackers that can execute commands are always able to remove these IOCs afterward.

Looking at the machine itself, a lot more viable sources of information are available. First things first, detecting an execution through Run Command or the CustomScriptExtension is easily logged through the launch of RunCommandHandler.exe and CustomScriptHandler.exe. Both will only and always be launched in their particular use case, and hence are very strong indicators. It’s a bit harder for Machine Configurations, as gc_worker.exe is also used to deploy other Azure Policies, and there are a few enabled by default for all Azure Arc machines. Instead, you could look for new folders created in C:\ProgramData\GuestConfig\Configuration\ or a new registry key HKLM\SOFTWARE\Microsoft\Windows Azure\RunCommandHandler\<RunCommandId>, as new machine configurations will be deployed there. It becomes an even stronger IOC if a folder is created and shortly after removed again, as this indicates the execution of a configuration and a consecutive deletion. The default policies will be created there, too, but it’s easy enough to filter them out. The ones I’ve seen are the following, though that is likely subject to change:

• AuditSecureProtocol
• AzureWindowsBaseline
• WindowsDefenderExploitGuard
• WindowsLogAnalyticsAgentInstalled

Here are a few folders and logs I’ve come across that contain useful information for any form of incidence response for execution through Run Command:

• C:\Packages\Plugins\microsoft.cplat.core.runcommandhandlerwindows\2.0.9\Downloads\ – contains executed run commands as Script_<RunCommandName>_<sequential_id>.ps1 (deleted if the corresponding Run Command is deleted)
• C:\ProgramData\GuestConfig\extension_reports\RunCommandHandler_report.txt
• C:\ProgramData\GuestConfig\extension_logs\microsoft.cplat.core.runcommandhandlerwindows\RunCommandHandler.log
• C:\ProgramData\GuestConfig\ext_mgr_logs\gc_ext.log
• HKLM\SOFTWARE\Microsoft\Windows Azure\RunCommandHandler\<RunCommandId> (deleted if the corresponding Run Command is deleted)
• C:\Packages\Plugins\microsoft.cplat.core.runcommandhandlerwindows\2.0.9\status\<RunCommandName>.<sequential_id>.status
• C:\ProgramData\GuestConfig\extension_logs\microsoft.cplat.core.runcommandhandlerwindows\*_stdout.txt and *_stderr.txt

A particular tough cookie is the CustomScriptExtension when the protectedSettings are used to hide the executed command. While this can be an IOC on its own, there are valid use cases for hiding your commands from logs in regular use, too, so I wouldn’t rely too much on that. In general, the encrypted commands are stored in C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\<version>\RuntimeSettings\<sequential-id>.settings. If the command is stored in the regular settings property, it will be in there too, but unencrypted. Protected settings, however, are encrypted using a private key stored on the Azure Arc machine in the certificate trust store. While the key is marked as non-exportable, it can still be used locally. This little piece of code can aid in decrypting them if executed on the host where the certificate and the private key are. It is based on how CustomScriptHandler.exe decrypts these settings (thanks again, dnSpy!).

Here are a few folders and logs I’ve come across that contain useful information for any form of incidence response for execution through CustomScriptExtension:

• C:\ProgramData\GuestConfig\extension_logs\Microsoft.Compute.CustomScriptExtension\CustomScriptHandler.log
• C:\ProgramData\GuestConfig\ext_mgr_logs\gc_ext.log
• C:\ProgramData\GuestConfig\extension_reports\<extension_name>_report.txt
• Files that are deleted if the corresponding extension is deleted:
  • C:\ProgramData\GuestConfig\downloads\
  • C:\ProgramData\GuestConfig\extension_logs\Microsoft.Compute.CustomScriptExtension\state.json
  • C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\<version>\RuntimeSettings\<sequential-id>.settings