What is CPU oversubscription?
CPU oversubscription occurs when more virtual processors (VPs) are allocated to virtual machines (VMs) than there are logical processors (LPs) available on the host machine. This technique is commonly used in virtualized environments to maximize resource utilization, improve cost efficiency, reduce data centers energy consumption and environmental impact. While this approach maximizes resource efficiency, it also introduces challenges related to balancing of compute resources and ensuring consistent performance for critical workloads which can be understood and mitigated through robust monitoring strategies.
Introducing new CPU jitter counters in Windows Server 2025
To address the challenges posed by CPU oversubscription, Windows Server 2025 introduces new CPU jitter counters. These counters are designed to monitor and quantify the variability in CPU processing times, providing valuable insights into the performance of virtualized environments.
CPU jitter refers to the variation in the time it takes for a CPU to process a given workload. This variability can be caused by several factors, including CPU contention, interrupt handling, and other system activities. The new CPU jitter counters in Windows Server 2025 measure this variability, offering a detailed view of performance fluctuations.
\Hyper-V Hypervisor Virtual Processor(*)\CPU Wake Up Time Per Dispatch
\Hyper-V Hypervisor Root Virtual Processor(*)\CPU Wake Up Time Per Dispatch
Wakeup delay: The average amount of time a given VP had to wait for an idle LP to wake up and start running that VP when there is no contention for that LP. This counter estimated the amount of delay incurred by a VP when the system is underutilized.
\Hyper-V Hypervisor Virtual Processor(*)\CPU Contention Time Per Dispatch
\Hyper-V Hypervisor Root Virtual Processor(*)\CPU Contention Time Per Dispatch
Contention delay: The average amount of time a given VP had to spend waiting for an LP to run on that is attributed to VP contention, e.g. having to wait for other VPs to finish running first. This counter estimated the amount of delay incurred by a VP when the system is fully utilized.
Benefits of using CPU jitter counters
The introduction of CPU jitter counters in Windows Server 2025 offers several key benefits:
- Improved performance management: By identifying the sources of CPU jitter, administrators can optimize resource allocation and reduce contention, leading to more stable and predictable performance.
- Enhanced troubleshooting: Detailed insights into CPU processing times enable quicker identification and resolution of performance bottlenecks and anomalies.
- Proactive monitoring: Visibility to CPU jitter allows for proactive management of resources, ensuring that potential issues are addressed before they affect service quality.
Implementing CPU jitter counters in Windows Server 2025 is a straightforward process.
- Launch Performance Monitor on Windows Server 2025.
- Select from broad sets of available counters.
- Add selected counters (total, per VP, LP, root VM, etc.) so that they appear under Added counters.
- Start data monitoring. There are also options of creating User Defined Data Collection Sets.
Best practices for managing CPU oversubscription
To effectively manage CPU oversubscription and leverage the benefits of CPU jitter counters, consider the following best practices:
- Establish baselines: Before diving into optimization, establish performance baselines under normal operating conditions. This will help in identifying deviations and potential oversubscription issues.
- Use multiple counters: Relying on a single counter can provide a skewed view of the system's performance. Use a combination of the aforementioned counters to get a comprehensive understanding of CPU utilization and potential oversubscription.
- Monitor trends over time: Short-term spikes in CPU usage might not necessarily indicate performance issues. Monitor trends over an extended period to identify persistent issues that require intervention.
- Regularly review and optimize resource allocation: Periodically review the resource allocation for VMs and adjust as necessary based on workload demands to minimize contention and improve performance.
Suggested primary CPU oversubscription monitoring counters
New Windows Server 2025 counters:
- \Hyper-V Hypervisor Virtual Processor(*)\CPU Wake Up Time Per Dispatch
- \Hyper-V Hypervisor Virtual Processor(*)\CPU Contention Time Per Dispatch
CPU jitter performance monitoring counters:
- \Hyper-V Hypervisor Virtual Processor(*)\CPU Wait Time Per Dispatch
The average combined amount of time a give VP had to spent waiting to run on an LP without specifying the reason for the wait.
- \Hyper-V Hypervisor Virtual Processor(*)\Logical Processor Dispatches/sec
The number of times this VP was scheduled to run on any LP
Overall CPU oversubscription load monitoring counters:
- \Hyper-V Hypervisor Virtual Processor(*)\% Total Run Time
The percentage of time spent by the virtual processor in guest and hypervisor mode combined.
- \Hyper-V Hypervisor Virtual Processor(*)\% Guest Run Time
The percentage of time spent by the virtual processor in guest mode specifically.
- \Hyper-V Hypervisor Logical Processor(*)\% Total Run Time
The percentage of time spent by the physical processor in guest and hypervisor mode combined.
- \Hyper-V Hypervisor Logical Processor(*)\Scheduler Local Run List Size
The number of virtual processors that are scheduled to run on a given physical processor.
The newly introduced CPU Contention Time Per Dispatch counter provides great insight on impact of system resource contention to jitter. It measures exactly the time a VP had to spend waiting for an LP to start processing the load. It is an enhancement of the previous CPU Wait Time Per Dispatch which captures multiple factors contributing to the wait time. In combination with the Logical Processor Dispatches/sec we can exactly calculate the system delay due to contention.
Contention delay time = (CPU Contention Time Per Dispatch) x (Logical Processor Dispatches/sec) [ns]
Contention delay time should be strictly monitored and used as a factor to further balance workloads. Knowing the details of the system’s process delays will allow us to achieve higher VM density without compromising system performance and end user experience.
Through Windows Server 2025 performance monitor we can calculate the created contention delay time per VP. Please note that usingCPU Contention Time Per Dispatch as a direct comparison may cause misleading results. It must be always monitored together with Logical Processor Dispatches/sec to account for workloads’ specifics.
In a test environment a host with 4CPUs (8LPs) with 2 VMs with 4 VP each, we run a load of 2 threads per VM.
2VMs of 4VPs each; VM1 VP0CPU Contention Time Per Dispatch (red) and Logical Processor Dispatches/sec (green)
It is recommended that each VP to be monitored independently or use the total counter values. In case of total counter values CPU Contention Time Per Dispatch is provided as average of all VPs, and Logical Processor Dispatches/sec is a sum of all dispatches per VPs (it must be divided to number of VPs on the system to align with single metric for a VP).
From the example shown on Fig.1 we have VM1 VP0 average values of:
CPU Contention Time Per Dispatch = 30,617 [ns]
Logical Processor Dispatches/sec = 577 [dispatches/s]
Contention delay time = 30,617 x 577 = 17.67 [ms]
We can clearly increase the load on the system and improve efficiency by doubling the VM density and reaching 100% CPU oversubscription adding 2 additional VMs with 4 VPs each. If we run the same load across all VMs as on the first 2 VMs (each of 2 threads) we can observe the increase of the contention delay.
4VMs of 4VPs each 100% CPU oversubscription; VM1 VP0CPU Contention Time Per Dispatch (red) and Logical Processor Dispatches/sec (green)
From the example shown on Fig.2 we have VM1 VP0 average values of:
CPU Contention Time Per Dispatch = 586,700 [ns]
Logical Processor Dispatches/sec = 430 [dispatches/s]
Contention delay time = 586,700 x 430 = 252.28 [ms]
This may be acceptable for some workloads, or we need to place VMs with diverse workload to reduce contention delays.
If system loads increase per VM from 2 threads load to 4 threads load on all 4 VMs (total of 16 threads, all VMs 100% loaded, 100%CPU oversubscription). Contention delay significantly increases.
4VMs of 4VPs each 100% CPU oversubscription 100% utilization; VM1 VP0CPU Contention Time Per Dispatch (red) and Logical Processor Dispatches/sec (green)
From the example shown on Fig.3 we have VM1 VP0 average values of:
CPU Contention Time Per Dispatch = 938,641 [ns]
Logical Processor Dispatches/sec = 794 [dispatches/s]
Contention delay time = 938,641 x 794 = 745.28 [ms]
Contention delay reaching high values could be an indication that VM(s) needs to be migrated to a different host to address the contention scenario.
Virtual Machine Resource Control
Hyper-V provides advanced tools to control resource allocation and jitter through per-VM CPU resource controls (including a CPU cap, weight, and reserve), isolating VM groups to specific host processors, and minroot.
Using CPU weights allows for more granular control over how CPU resources are allocated among VMs. This ensures maintaining consistent performance levels for critical workloads.
Assigning specific LPs to a VM group ensures that the VMs in that group have exclusive access to these compute resources, eliminating competition with other groups. High-priority VMs can achieve optimal performance as they are not de-scheduled by other VMs. This setup is particularly useful for applications requiring real-time processing or low-latency operations.
The minroot configuration allows Hyper-V administrators to dedicate a subset of processors exclusively to the root partition, effectively segregating the work done in the Hyper-V host from the workloads running in guest virtual machines. This segregation helps in better resource management and enhances system stability.
For more information on implementation and utilization of the functions listed above, seeOptimizing your Hyper-V hosts.
Advancing performance management
The introduction of new CPU jitter counters in Windows Server 2025 marks a significant advancement in performance management for virtualized environments. By providing detailed insights into CPU variability, these counters empower administrators to optimize resource allocation, increase VM density, improve performance stability, and enhance the overall reliability of their systems. As we continue to embrace virtualization and cloud computing, tools like CPU jitter counters will play a crucial role in ensuring that our digital infrastructure remains robust and efficient.
