Microsoft has released details on how its Hotpatching feature applies security fixes to Windows Server without requiring a restart – but although the company has said it is working on wider availability, it remains uniquely Azure.
Credited to “Andrea Allievi & Hotpatch Team”, Allievi being a Senior Operating System Engineer at Microsoft, the Publish explains both the rationale and the technology behind the functionality. It’s not just about convenience.
Often times, users and system administrators will delay installing a patch due to the restart that is frequently required at the end of the installation. This delay in updating, while seemingly convenient, is in fact a security problem, ”explains the message published on November 19. , referencing a report showing that 42% of exploited vulnerabilities occur after a patch is released.
Microsoft focused on the problem in the context of Azure host machines. “The instances of Windows Server that power the Azure fleet need to be highly available. However, we also demand that these operating system instances be secure, ”the post added. Therefore, Hotpatch has been “used in Azure Host OS for some time”, making the technique “battle tested”.
The method of restarting the fix is easy to understand: the system shuts down, cleanly terminating all processes, then the binaries that implement the Windows NT kernel are updated and the processes in the restarted system call run in updated files.
Hotpatching is different in nature. According to the team, this “works at the function level, meaning that functions are patched individually and not individual files or components.” The way it works is to redirect calls to the uncorrected function to “a patched function belonging to a hotpatch image”. It works with x64, ARM64 (new in Windows Server 2022), and 32-bit code.
The implementation of this solution requires a Hotpatch engine, “mainly in NT and Secure Kernel”, explain the engineers, the Secure Kernel being part of the operating system that runs in a more secure and isolated environment called VTL1 (Virtual Trust Level 1). The Hotpatch engine identifies the patch images, verifies that they match the unpatched base image, and then maps the patch image to the same address space as the base image.
The engine is smart enough to update references to global variables in corrected functions to point to global variables in the base image. Then it executes the patch, so that “the functions of the original base image pass to the corresponding functions in the patch image”. This bouncing of code paths is described as “the trampoline”.
Patching a system in this way indefinitely would lead to increasing convolution. Therefore, there is a periodic refresh with a new base image set, implemented as a traditional cumulative update and requiring a restart. The stream Documentation does this every three months. There is a hint that even better patching techniques may come. “Hotpatching is one of the first techniques designed to provide users with a future of no-reboot security update,” the team said.
Windows Server 2022 introduces not only the ARM64 support mentioned above, but also compatibility with Retpoline, a return trampoline introduced to overcome the side channel attacks of Specter v2.
The catch with these features can be found in the last paragraph of the article. “Patch-based security updates are available for customers running Windows Server 2019 and Windows Server 2022 Azure Edition images in the Azure cloud as part of automatic management,” the team says. This is little comfort for the countless other Windows Server users.
“The patch functionality for Azure is great, but on-premises servers are long overdue for a replacement or new patch method that is not WSUS” commented a customer in July in response, also observing that “every month the server chokes for hours trying to synchronize the WSUS database after the updates are released.” WSUS is Windows Server Update Services, deployed on corporate networks to deploy fixes internally.
At the time, Senior Program Director Ned Pyle said “we have an answer on this soon, but I can’t say more at this time.” Now the hotpatch teams say “we are working to deliver hotpatch-based security updates to more Windows customers.” Note that Hotpatch for Azure VMs is still in preview. The documentation described it as “a new way to install updates on a supported Windows Server Azure Edition virtual machine.”
The impact of Hotpatch could be significant as it is faster and less disruptive than the existing patch and restart cycle, and can be automated without introducing downtime. But Microsoft has yet to explain why it’s all about Azure. If this is to test functionality in a controlled environment before more general availability anywhere Windows Server may run, that’s understandable. If this is a way to give Microsoft’s cloud an artificial advantage over both local and other public clouds and hosting companies, it would be unwelcome for customers. ®