Azure Arc is Microsoft’s solution to allow customers to manage on-premises resources (servers and the likes) using Azure Resource Manager, i.e. the Azure Portal, CLI and the Azure PowerShell module. It basically enables on-premise server management in a way that it feels like you’re administrating an Azure Virtual Machine instead of an on-premises server. Note that Azure Arc supports Windows and Linux, but this blog series will only cover aspects of Windows machines.

This post is part of a blog series covering the security implications of an Azure Arc deployment. This first part is a technical deep-dive into the possibilities of an attacker who gained access to the machine itself, and which implications that has for the Cloud environment. The second part of the series deals with technical aspects of what permissions an attacker would require to move laterally to the on-premises Azure Arc machine and what traces such actions would leave. The third post sums up all the uncovered aspects and gives general, non-technical advice on how to integrate Azure Arc in existing security concepts and what to be careful about.

TL;DR; In contrast to regular Azure VMs, on Azure Arc connected machines only Administrators, services running as NT Authority\Network Service or members of the local group “Hybrid Agent Extension Applications” are allowed to obtain tokens for the managed identity. This will then, however, provide such accounts with all access to Azure that the managed identity has.

Azure Arc Setup

When enrolling a Windows system in Azure Arc, the default way will be a PowerShell script that the Azure Portal will present to you, which you ought to execute with administrative permissions on the server that you wish to join.

This script will download another PowerShell script from https://aka.ms/azcmagent-windows, store it at $env:TEMP\install_windows_azcmagent.ps1 and run it. This script will then perform some checks, download the Azure Hybrid Agent’s MSI from https://aka.ms/AzureConnectedMachineAgent and install it. An installation log will be created in $env:TEMP\installationlog.txt, if you run into trouble. As a sidenote: the script will check and abort if the machine is an Azure VM. However, if you ever want to deploy an Azure VM to Azure Arc, just set the MSFT_ARC_TEST environment variable before executing the PowerShell script. I would only recommend that for tests though, as this will likely have unintended side effects.

Azure Hybrid Agent Command Line Utility

After the Azure Hybrid Agent is installed, a service called “Azure Hybrid Instance Metadata Services” (“himds”) will be up and running as NT Service\himds. The setup script will interact with this service using the “azcmagent” command line utility to connect the machine to Azure (the variables are set in the script):

This utility can also be used to interact with the Hybrid Agent from within the machine itself.

The “check” command can help with debugging connection issues in Azure Arc, for example when you want to set up a private link instead of using public Azure Arc endpoints. From an attacker point of view, however, the most interesting command (that does not even require elevated access) is the “show” command, as it will display detailed information on the Azure resource of the machine, like the tenant and subscription ID, as well as the name of the resource group it is in. Note that the name of the Azure Arc resource will be the hostname of the machine.

Note that usage of the agent is logged intensively in $env:ProgramW6432\AzureConnectedMachineAgent\Log\azcmagent.log:

The command line utility most likely relies on two named pipes to interact with the himds service, one for the admin and one for the non-admin commands. I haven’t looked into this any further, but from the number of bugs in named pipes in the past this will definitely be an interesting thing to look into ;-)

Azure Hybrid Instance Metadata Service (himds)

An attacker who wants to use their foothold on an Azure Arc machine to escalate to the Azure Cloud will definitely take a look at the identity endpoint. When a machine is connected to Azure Arc, it will automatically get a managed identity enrolled. While by default it will not have any permissions, it is the intended way to grant an Azure Arc machine permissions within Azure. For example, when Azure Arc is used to manage Hybrid Workers for Automation accounts, it’s not unlikely that the managed identity will have permissions required for those automation tasks.

As managed identities are not a novel topic in Azure, I will not go much into detail here. What is unique to Azure Arc, however, is that obtaining a token actually requires administrative access, which is not the case for regular Azure VMs. With an elevated shell, connection to the managed identity is easy as ever:

Without elevation, however, only an error is raised:

This can also be found here and here in the official Microsoft documentation:

• On Windows, you must be a member of the local Administrators group or the Hybrid Agent Extension Applications group.
• On Linux, you must be a member of the himds group.

The referenced links do describe how to obtain tokens as administrators though. Following this, more light can be shed on why elevation is required. The first step is identical to other services with managed identities: looking at the environment variables reveals HTTP endpoints hosting REST APIs, including the token or identity endpoint:

While on regular Azure services the endpoints are actually reachable at “http://169.254.169.254”, Azure Arc himds service binds it to localhost:

Nevertheless, the metadata endpoint can be used the same as on other Azure services:

What is different though, is the token endpoint: a regular token acquisition request is rejected with “Missing Basic Authorization header”:

In the same response the basic auth header contains a path to a file. Note that the name of the key file changes for every request.

The key contains a base64 blob. The file permissions, however, reveal how Microsoft limited the use of the token endpoint to only Administrators or members of the “Hybrid agent extension applications” group: access to the key file is restricted on a file system level. Note that this is not entirely true, as any service running as NT Authority\Network Service also has the permission to acquire a token:

Decoding the base64 blob reveals an ID and another base64 blob, while the latter appears to be only random data.

Setting the content of the key file as a Basic Auth header for the same request will then result in the expected result, and an access token for the managed identity can be obtained.

Note that during my tests such a key file was automatically deleted after approximately 10 minutes and the key could no longer be used (“Invalid authentication info”).

On a side note: So far all communication was with localhost, or rather a local service running – why can’t we just talk with Azure directly to obtain our tokens, like on an Azure VM? Because the managed identity is not authenticated via the machine’s IP or something, but with a certificate installed on the Azure Arc machine. This is, by the way, the reason why the authentication of the managed identity will show up as “client assertion” in the Sign-In log. The local token endpoint simply removes that complexity and lowers the bar from “having access to a machine’s certificate’s private key” to “have access to a key file”. Though I haven’t tried it, you might be able to extract the certificate and use it elsewhere to authenticate as the managed identity of the Azure Arc machine.

Monitoring

Observing the log-in on Azure side is not at all different from other Azure services and can be achieved through sign-in logs, where managed identity log-ins are clearly marked as such.

On the machine itself, there is the file C:\ProgramData\AzureConnectedMachineAgent\Log\himds.log that contains logs on every obtained token, which key was used and for which resource access was requested to: