Rokroot Debug Training: home page
User Info
You are not logged in.
 » Login  «
 » Register  «
 
 Kernel
 User
 Managed
 User/Kernel
 User/Managed
 All


Search the Catalog


Your cart is empty.     Total: $0.00


Mission Statement

History

302_Ongoing Investigations - The Interrupt That Wouldn't Let Go


Ongoing investigation: seventeen 0x133 bugchecks in six hours on a Hyper-V host, traced in the kernel dumps to an interrupt livelock on a shared, level-triggered IRQ 22 in HP's iLO 2 channel driver, apparently set off by Kubernetes VMs cold-starting. What the dumps show, what is suspected, what is ruled out and the tests still to run, with every piece of evidence. Updated as the investigation goes on.

Recommended prerequisite training modules: 006, 055, 180, 221, 222
Reading time: 28 min
Price:  $0.00  

Seventeen bugchecks in six hours on a Hyper-V host: an interrupt livelock on IRQ 22 in HP's iLO channel driver, apparently set off by Kubernetes VMs cold-starting. Investigation in progress.

Investigated 2026-11-18 to 2026-11-20 at Rokroot. Status: in progress. Last updated 2026-11-20.

This investigation isn't finished. The mechanism is shown in the kernel dumps. What sets it off, and what keeps the interrupt line asserted, are still suspicions. Section 10 lists the tests that would settle them, and this page is updated as they're run.

Every claim is marked shown (read in a dump, log or configuration), likely (the best-supported reading, not proven) or unknown.


Summary

On 18 November 2026 a Hyper-V host named SERVER2 bugchecked 17 times between 05:11 and 11:20, every time with 0x133 DPC_WATCHDOG_VIOLATION.

  • It carries the network's only domain controller, so every crash took DNS and DHCP down with it.
  • Each reboot auto-started every VM, and 12 to 41 minutes later the host went down again.

The five dumps that survived blame four different things: the TCP timer, a TCP lock, the Hyper-V message handler and an IP timeout routine. None of them did it.

  • What the dumps show (shown, 4 of 5 dumps):
    • For 94–136 seconds before each crash, CPU 1 was taking about 64,000 interrupts a second, while CPUs 0, 2 and 3 sat idle.
    • They all came on one shared, level-triggered line: IRQ 22, vector 0x91.
  • Who claimed them (shown): HP's iLO 2 channel-interface driver, cpqcidrv.sys 1.17 from 2012.
    • In the full dump its interrupt service routine claimed 5,949,471 interrupts, while its DPC ran 101 times.
    • That DPC is what services the device, so the line never dropped.
  • What holds the line (likely, not proven): a second device on the same line, probably the iLO's own virtual USB controller. It never gets serviced, because the HP driver claims its interrupts first.
  • What sets it off (likely): Kubernetes VMs cold-starting.
    • Every crash in the loop came in a boot that started them, and the host has stayed up since the first boot that didn't.
    • But the same signature hit on 10-07 with no Kubernetes VM at all.
Mechanism (shown)An interrupt livelock on shared, level-triggered IRQ 22 on CPU 1. cpqcidrv's ISR claims every interrupt, its DPC is starved, and the DPC watchdog fires.
What keeps the line asserted (likely / unknown)The iLO's virtual USB controller, behind cpqcidrv in the chain and never serviced; or the iLO itself not honouring the driver's mask after a reset the day before.
Trigger (likely)The boot-time load of Generation 1 Kubernetes VMs on a 2010 four-core CPU running at about half speed.
Not the cause (shown)tcpip, NETIO and the Hyper-V message handler (bystanders); the November update; the RAID cache; a traffic flood; a hardware fault.
NowUp since 11-18 11:27 with the Kubernetes VMs off, and no storm since. Monitored every 15 minutes. The driver is still 1.17.

This page walks through:

  • the outage, and the first reading, which was wrong;
  • how the dumps were made to name the real culprit: per-interrupt-object counters, the DPC watchdog's own profile buffer and the driver's code;
  • the suspected mechanism and the suspected trigger;
  • what has been ruled out, an attack included;
  • where things stand, and the tests still to run.

1. The setting

MachineRole
SERVER2HP ProLiant ML330 G6 (2010). Xeon E5606: 4 cores, no hyper-threading, 2.13 GHz nominal. 72 GB of RAM, a Smart Array P410 with a RAID 1 and a RAID 6 volume, Windows Server 2019 build 17763.9487. It is the Hyper-V host for the domain controller (DC-VM), a mail server, a log server, an old test machine and, until 11-18, a six-VM Kubernetes lab. It also runs three Windows containers.
VMHOSTAnother Hyper-V host. The investigating agent ran there and reached SERVER2 remotely.

The management controller. The G6 has an iLO 2 running firmware 2.33 (2018), the last release. Windows sees it as several PCI functions of one device:

  • the channel interface (PCI\VEN_0E11&DEV_B204), driven by cpqcidrv.sys 1.17.0.0 (2012-11-06). HP's management agents use it to talk to the iLO.
  • the management controller itself, driven by hpqilo2.sys 1.15 (2012).
  • a virtual USB controller (PCI\VEN_103C&DEV_3300), which carries the remote console's keyboard and mouse.

No HP driver release supports Server 2019 on a G6; 2012 R2 was the last supported OS.

The Kubernetes lab (shown, from Hyper-V):

  • Six VMs: three control planes, a load balancer and two workers, 15 virtual CPUs in all.
  • How they're built: all are Generation 1 VMs (version 9.0) that boot from emulated IDE, and five run on checkpoint (differencing) disks.
  • The software: Ubuntu 26.04.1 with kernel 7.0, Kubernetes 1.37 and Calico 3.33.
  • The load: with the other VMs, a cold start meant 24 virtual CPUs on 4 cores. See vms.csv.

The investigators. Two AI coding agents (Claude Code) worked on this host that week: one maintained the Kubernetes lab, the other investigated the outage. The first appears in section 7 as a suspect. The second appears in section 4 with a debugging lesson.


2. What happened (2026-11-18, local time)

  • 05:11:48: the first crash, in steady state: the host had been up since 11-14.
    • It came 3½ minutes after a nightly container job restarted its container.
    • The job had run the same way on the three nights before.
  • Then a loop. Every reboot auto-started all ten VMs, and every boot crashed again: 05:34, 05:56, 06:23, 06:45, 07:05, 07:31, 07:49, 08:11, 08:34, 08:53, 09:13, 09:34, 09:52, 10:11, 10:58 and 11:20. That's seventeen crashes, all 0x133.
  • Each crash took the domain controller with it, so DNS and DHCP went down for everyone. The owner noticed at about 09:15.
  • The 10:17 boot: the owner shut down the three control planes and the load balancer between 10:35 and 10:50. The crash still came, at 10:58.
  • The 11:03 boot: only the two worker VMs started, with no control plane (13 virtual CPUs in all). It crashed at 11:20.
  • The 11:27 boot: the first with no Kubernetes VM, because the owner set them not to start. It hasn't crashed since.
BootKubernetes VMs startedvCPUs at the crashCrash
11-14 16:01 (steady state)five, earlier2411-18 05:11
05:18 to 09:58 (14 boots)all six2412 to 22 minutes after each boot
10:17all six; four shut down 10:35–10:501310:58
11:03the two workers only1311:20
11:27none8none since

What a crash looked like from outside:

  • The machine simply rebooted. Windows logged an "unexpected shutdown" (6008) whose last-alive time could be up to 3 minutes before the real crash. Kernel-Power 41 carried no bugcheck code.
  • The iLO's Integrated Management Log (IML) recorded all 17, including the 09:34 crash, for which Windows never filed a bugcheck report.
  • The first timeline missed two. It was built from Windows' logs and counted 15.

3. The first reading, and why it was wrong

That afternoon's conclusion was overload. The evidence for it:

  • The load: 24 virtual CPUs on 4 cores.
  • The CPU speed: the cores measured at about 1.1 GHz instead of 2.13, a firmware limit logged every day since August.
  • The network: all VM traffic went through one NIC with neither VMQ nor RSS.
  • The slowness: the guests booted many times slower than usual during the crash cycles. The worker VMs took up to 8½ minutes to load their network driver, against 18–48 seconds the day before, and a 7-second host task took 13 minutes.

The story fit, and turning the Kubernetes VMs off "fixed" it.

Earlier that morning, other suspects were named without kernel evidence: the new RAID cache module, the November cumulative update, the guest network, a USB Wi-Fi adapter on another host, the modem and an SSD. Each was plausible, and each was dropped once it was checked.

Why overload didn't survive the dumps:

  • Starvation means a CPU that doesn't get to run. The full dump shows CPU 1 running flat out instead.
    • It took a clock interrupt in every tick period.
    • It spent 50.9 s of the last 94 inside interrupt service routines.
  • The other three CPUs were idle, with nothing ready to run.

Load may be the trigger, but it isn't the mechanism. Section 4 shows how the dumps say so.


4. What the dumps show

Five dumps survived: minidumps of the 09:13:10, 09:52:27, 10:11:45, 10:58:06 and 11:20:59 crashes, and a full kernel dump of the last one. Windows keeps only the last five minidumps, so the others were overwritten.

A note on how they were read.

  • How kd ran: on SERVER2 itself, with a command file and its standard input redirected from an empty file.
  • What it did afterwards: after each run kd stayed at a full core for 1½ to 5 minutes and couldn't be killed.
  • Why (likely): that's the non-paged pool leak of case study 301, which is set off by exactly that kind of standard input. SERVER2 had enough memory to survive it.
  • The rule since: never run kd on a VM host.

4.1 Five dumps, four different culprits

CrashCPUWhere the watchdog firedWhat it really was
09:13:101winhvr!WinHvLookupPortId (cumulative)the Hyper-V message ISR, itself interrupted
09:52:273a tcpip receive DPC spinning on a TCP lock (KxWaitForLockOwnerShip)likely waiting for a lock held by the stalled CPU 1
10:11:451a tcpip timer DPC in nt!KeTestSpinLocka function-entry fast path, not a lock wait
10:58:061winhvr!WinHvOnInterrupt (cumulative)the Hyper-V message ISR, itself interrupted
11:20:591the TCP timer-wheel scan, NETIO!RtlGetNextExpiredTimerWheelEntry: 20 s in one DPCa timer wheel with one entry in it

Microsoft's own crash analysis (Windows Error Reporting) bucketed the rest of the day's crashes just as variously; bugchecks.csv has the full list:

  • b57nd60a!shoot_nbls_up, the NIC driver, twice;
  • USBPORT!USBPORT_InterruptService twice;
  • cpqcidrv!unknown_function once;
  • nt!KiExitDispatcher and nt!KiGenericCallDpcWorker.

A DPC watchdog names whatever was running when it fired: the victim, not the culprit. When the victims keep changing, look underneath them.

A trap in !analyze: it prints CPU_MHZ: 850. That's hexadecimal: 0x850 is 2,128 MHz, not a throttled CPU.

4.2 CPU 1 is drowning in interrupts

!prcb and dt nt!_KPRCB on each processor of the full dump, 17 minutes after boot:

CPUInterrupts since bootInterrupt timeDPC timeInterruptRate
02,945,0936.6 s6.0 s0
18,021,40479.2 s22.8 s0x3f2 (1,010)
21,918,6844.6 s2.8 s0
31,982,6514.9 s7.3 s2

How to read InterruptRate: it's a running average of interrupts per clock tick. KeAccumulateTicks updates it at every tick to (old + 15 × new) / 16.

  • CPU 1: 1,010 per 15.6 ms tick is about 64,600 interrupts a second.
  • CPUs 0, 2 and 3: 0 to 2 per tick. They were in their idle loops, with no DPC queued and no thread ready to run.

Was it VM I/O? No (shown).

  • Hypervisor, VMBus and IPI interrupts on CPU 1 were normal. They have no interrupt objects, so they're counted as the PRCB's total minus every object's count: 495,344 on CPU 1, against 402,799 to 748,874 on the others.
  • The storage and NIC interrupts were ordinary on every CPU.
  • CPU 1 ran fewer DPCs than CPUs 0 and 3.

What was queued behind the stall: CPU 1 had 21 DPCs waiting, unserviced for 94 seconds:

  • cpqcidrv's own DPC, first in the queue;
  • PpmPerfAction and ten nt!PopExecuteProcessorCallback;
  • USBPORT_IsrDpc;
  • vmswitch!VmsVrssDpc, six Vid!VidpMessageSlotEventSignalDpc and vmbusr!ParentRingInterruptDpc.

The Hyper-V DPCs were victims of the stalled queue.

4.3 Which interrupt: walk the interrupt objects

!intstats /d fails with this debugger and these symbols ("Couldn't read INTERRUPT_TRACKING_ROOT offsets"). But the counters live in the interrupt objects themselves, and one kd loop reads them all (Appendix A):

  • Each KINTERRUPT has an ISRDPCSTATS block: IsrCount, DpcCount, the time spent in the ISR, and when the DPC last ran.
  • The PRCB holds a 256-entry array, InterruptObject[], one entry per vector.
  • Objects sharing a vector are chained from that entry.

On CPU 1, vector 0x91, in chain order:

#ISRInterrupts claimedTime inside itNotes
1USBPORT!USBPORT_InterruptService38.2 scalled on every storm interrupt; declined
2USBPORT!USBPORT_InterruptService317.9 scalled on every storm interrupt; declined
3USBPORT!USBPORT_InterruptService9587.8 scalled on every storm interrupt; declined
4cpqcidrv+0x327c5,949,47116.8 sits DPC ran 101 times
5USBPORT!USBPORT_InterruptService26,5670.2 slast called 94.2 s before the crash

Together that's 108 billion cycles, 50.9 s at 2,128 MHz, inside one vector's ISRs during a 94-second episode. The rest is interrupt entry and exit.

What cpqcidrv's interrupt object says: Vector : 0x91, ShareVector : 0x1, Mode : 0 ( LevelSensitive ), polarity active-low. The 09:13:10 minidump shows the same thing: 7,143,152 claims against 84 DPCs.

On the live machine, five devices share IRQ 22:

  • CpqCiDrv;
  • the iLO's virtual USB controller ("Standard Universal PCI to USB Host Controller", PCI\VEN_103C&DEV_3300);
  • two Intel ICH10 USB UHCI controllers (3A39 and 3A36);
  • an ICH10 USB EHCI controller (3A3C).

That's five devices for five interrupt objects. hpqilo2 has IRQ 21 to itself.

How a shared, level-triggered line works:

  • The device holds the line asserted until it's serviced.
  • On each interrupt, Windows calls the ISRs in the chain in order (KiChainedDispatch, KiScanInterruptObjectList). For a level-triggered line it stops at the first ISR that returns TRUE, "mine". If the line is still asserted afterwards, the interrupt simply comes back for the others.
  • If an ISR claims an interrupt that isn't its own, the real owner is never called and the line never drops. The CPU is interrupted again at once, and again.

That's a livelock: the CPU is fully busy and nothing makes progress.

4.4 When it started

ISRDPCSTATS.DpcTimeStart holds the processor's timestamp counter (TSC) at the moment the object's DPC last ran. For cpqcidrv:

CrashLast cpqcidrv DPC before the crashThe episode by the DPC watchdog's count
11:20:59200.45 billion cycles = 94.2 s6,027 ticks = 94.17 s
09:13:10255.35 billion cycles = 120.0 s7,680 ticks = 120.0 s

The storm began when cpqcidrv's DPC last ran, in both dumps where that counter can be read. That DPC is the routine that re-enables the device's interrupt. The USB object behind cpqcidrv in the chain ran its last ISR and DPC 18.8 ms before it.

Onset times: 09:11:10, 10:09:47, about 10:56 and 11:19:25. That's 11 to 16 minutes after boot, and once about 39.

4.5 The DPC watchdog's own profile buffer

When a DPC runs long, Windows captures a stack at every clock tick into the PRCB (DpcWatchdogProfile, up to DpcWatchdogProfileCurrentEmptyCapture). The buffer is in the minidumps too. Reading it with dps shows what CPU 1 was doing, tick by tick:

  • 11:20:59, 108 captures. The interrupted DPC's instruction pointer reads NETIO!RtlGetNextExpiredTimerWheelEntry+0x60 58 times, +0x66 41 times, +0x6b 6 times and +0x6e 3 times. Four instructions in 108 ticks.
  • 10:11:45, 499 captures:
    • 179 are inside the USB controllers' ISRs;
    • 84 are in cpqcidrv's ISR;
    • 98 are in the kernel's chain dispatch;
    • 138 are on the DPC, which moved 7 instructions in 7.8 seconds.
    • 423 of the 499 contain KiChainedDispatch.
  • 10:58:06 and 09:13:10, 641 captures each: the same picture. About 220 are in the USB ISRs, 95–102 in cpqcidrv's and about 135 in chain dispatch, while the code underneath goes nowhere.

That settles the culprits of 4.1: they weren't looping, they were standing still. The CPU never got back to them.

4.6 The bystanders, checked

  • The TCP timer wheel (11:20:59):
    • Almost empty: the partition being processed held one entry, due 58.9 s later. CPU 1's eight partitions held three entries in all.
    • Bounded: the scan visits at most 129 slots per call.
    • So the "20-second DPC" was almost no work, stretched over 66 seconds of wall time.
  • winhvr!WinHvLookupPortId is an artefact of export symbols.
    • The real code: with symbols it's winhvr!WinHvOnInterrupt, the hypervisor message ISR, caught at its entry and its exit.
    • No lock and no loop, and it was itself interrupted by vector 0x91.
  • The CPU 3 crash (09:52:27) is a real 20-second spin on a TCP partition lock in the receive path.
    • The owner is unreadable: the minidump holds only CPU 3.
    • Likely CPU 1: the full dump of the last crash shows CPU 1 holding exactly that kind of lock for 66 s while livelocked.
  • Who was waiting for CPU 1 (11:20:59):
    • ten threads in PoGetIdleTimes, waiting up to 94 s for an answer from CPU 1 (mmc, sqlservr, snmp and the HP management agent among them);
    • a crash-reporting thread that hadn't run since the storm began.
  • !qlocks: no queued spinlock was held or awaited on any CPU.

4.7 The driver's code

The ISR, u cpqcidrv+0x327c, and the end of its DPC, from the full dump (kd run 7). The comments are added; kdq-m1-7-output.txt has the raw listing.

cpqcidrv+0x327c:                        ; the ISR
  movzx eax, word ptr [rdx+0B0h]        ; read the device's status register
  test  al, 1                           ; status bit 0 set?
  je    ...                             ;   no: return FALSE, not mine
  mov   cl, byte ptr [rdx+0B2h]         ; read the enable register
  and   ecx, 0FFFFFFFEh                 ; clear enable bit 0: mask the interrupt
  mov   byte ptr [rdx+0B2h], cl
  lea   rcx, [r8+2B0h]
  call  [cpqcidrv+0x6138]               ; queue the DPC
  mov   al, 1                           ; return TRUE: mine

cpqcidrv+0x32dc:                        ; run at the end of the DPC
  mov   cl, [rdx+0B2h]
  or    ecx, 1                          ; set enable bit 0 again
  mov   [rdx+0B2h], cl

The ISR checks the status bit, but never its own enable bit.

  • The window: once it has masked its device and queued its DPC, the status bit can stay set until the DPC runs.
  • What happens in it: any interrupt on the shared line, from any device, finds bit 0 set and gets claimed. The real source is never called.
  • What would fix it: the DPC. On CPU 1 it never ran again.

The driver files date from November 2012: cpqcidrv.sys Nov 7, hpqilo2.sys Nov 1.


5. The suspected mechanism

IRQ 22 / vector 0x91: shared, level-triggered, active low, delivered to CPU 1

  interrupt --> KiChainedDispatch: call the ISRs in order, stop at the first "mine"
                  1  USB (ICH10 UHCI)            not mine
                  2  USB (ICH10 UHCI)            not mine
                  3  USB (ICH10, keyboard)       not mine
                  4  cpqcidrv (iLO channel)      "mine"   <-- its status bit is still set
                  5  USB (iLO virtual)           never asked, never serviced
  the line is still asserted --> interrupt again at once (about 64,000 a second)
  cpqcidrv's DPC, which would end it, waits behind the storm --> DPC watchdog --> 0x133

Candidate A (likely): a collision between two functions of the same iLO.

  • The 5th object is likely the iLO's own virtual USB controller. The dump's allocation order of the interrupt objects matches the live machine's PCI arrival order one for one. It ends with the channel interface (bus 1, device 4, function 2), then the virtual USB controller (bus 1, device 4, function 4). One kd command would confirm it.
  • That controller was 30 times busier in the crash boot: 26,567 interrupts in 943 s, about 28 a second. In the stable boot since, it takes about 0.9 a second. What made the difference is unknown.
  • If it interrupts while cpqcidrv's DPC is pending, cpqcidrv claims the interrupt and the virtual USB controller is never serviced.
  • The window needs a delay. On a quiet host, cpqcidrv's ISR and its DPC are microseconds apart, so for the collision to happen often, something has to delay the DPC. That fits a rare crash on a quiet host (10-07) and frequent crashes under boot load.

Candidate B (unknown): the iLO stops honouring the mask.

  • For: the iLO was reconfigured and reset on 11-17 at 02:32, 27 hours before the first crash (section 7). If it kept status bit 0 set, or ignored the mask write, cpqcidrv would claim forever on its own. And the onset coinciding with cpqcidrv's last DPC fits this reading directly.
  • Against: the iLO driver logged no event on 11-18, and this reading doesn't explain why the storms followed VM boots.

What the dumps can't show: the device's registers. A kernel dump doesn't contain them. Only HP diagnostics or a live kernel debugger could show whether the iLO ignores the mask.

Side note: a physical keyboard and mouse were plugged into the server from 11-14 until 12:42 on 11-18, on ICH10 UHCI 3A36, which is also on IRQ 22.

  • Probably not the starved source: by the same object ordering it's the 3rd object (958 interrupts), and it was serviced normally.
  • The alternative if the ordering argument is wrong.

6. The suspected trigger: Kubernetes cold starts

For (shown):

  • Every crash of the loop came in a boot that cold-started Kubernetes VMs, and the storm began 11 to 16 minutes after boot (once about 39).
  • The 11-14 crash came 9.5 minutes after a boot in which five Kubernetes VMs cold-started. The RAID array was rebuilding at the same time.
  • The first boot without them has stayed up: 51 hours by 11-20 15:00.
  • Removing the control planes didn't stop it: 13 vCPUs were enough. So it isn't one VM, and it isn't the cluster's control traffic.
  • In 9 of the 16 boot cycles the worker VMs hadn't loaded their network driver when the host went down. So the early, emulated boot phase is a better candidate than running containers or Kubernetes network traffic.

Why boot load on this host (likely):

  • Generation 1 VMs boot through emulated IDE. Every boot-loader, kernel and initrd read goes through device emulation on the host until the guest's storage driver loads.
  • The Xeon E5606 (Westmere) has no APICv. Without hardware interrupt virtualization, the guests' timer programming, IPIs and EOIs cost VM exits handled in software.
  • The cores ran at about 1.1 GHz (measured).
  • So the cost lands on the host's CPUs. Both costs grow with the number of VMs starting at once, and they land on the host CPUs, CPU 1 included, where cpqcidrv's DPC has to run. In candidate A, a few milliseconds of DPC delay are all it takes.

What changed on 11-17 (shown; the effect is likely): the Kubernetes lab was upgraded that day:

  • The guest OS: Ubuntu 23.10 → 24.04 → 26.04 on every VM, and kernel 6.5 → 7.0. Bigger kernels and initrds are read through emulated IDE at every start.
  • The disks: checkpoints put five VMs on differencing disks on the RAID 6 array.
  • The stack: Calico replaced Weave; containerd 2.4; Kubernetes 1.31 → 1.37.
  • The size: a third control plane, with dynamic memory, took the total from 21 to 24 vCPUs.

Against, or not explained (shown):

  • The first crash, at 05:11, wasn't in a boot.
    • The host had been up since 11-14, with all 24 vCPUs running for the last 7 hours.
    • It came 3½ minutes after a nightly container restart, which ran the same way on the three nights before.
  • 10-07: the same 0x133 parameters with no Kubernetes VM at all. Only four VMs ran, on an older Windows build. The iLO's watchdog reset the server, and no dump survives.
  • The IML shows 8 hang resets (ASR) between 2022 and 2026.
  • On 11-12 a boot with five Kubernetes VMs cold-starting survived. They then ran 43 hours without a crash, after 79 hours on the build before.

So the weakness is older than Kubernetes on this host. The likely reading is that boot load turns a rare race into a frequent one. Whether removing Kubernetes fixed it or only made it rare is unknown.


7. Ruled out, and why

SuspectWhy it looked plausibleVerdict
A tcpip, NETIO or Hyper-V regression in the November updateMost dumps name tcpip, NETIO or winhvr, and the update replaced tcpip and NETIO six days earlierRefuted as the cause. The named code was standing still (4.5). The same 0x133 hit on 10-07 on an older build. winhvr.sys wasn't changed by the update. Another host with the same files has no 0x133, although it runs no Kubernetes (a weak control).
Plain overload24 vCPUs on 4 slow cores; guests 5 to 20 times slowerRefuted as the mechanism: CPU 1 was running flat out while the others idled. Still the likely trigger.
VM I/O interruptsHyper-V DPCs queued on CPU 1Refuted. Hypervisor and VMBus interrupt counts on CPU 1 were normal. The queued Hyper-V DPCs were victims.
The new RAID cache module (11-14)New hardware four days beforeNot supported. The array is healthy, with no storage errors on 11-17 or 11-18, and storage interrupt counts are ordinary. 10-07 predates the module.
A hardware faultOne WHEA record during the second crashNot supported. The record is an NMI created by Windows itself, with no machine-check, memory or PCIe section: likely its own freeze NMI to a CPU that didn't answer. Temperatures, fans, power and memory logs are clean.
The nightly container jobThe first crash came 3½ minutes after itNot supported. It ran the same way on three earlier nights, one of them with the Kubernetes VMs up, and the 16 later crashes had no such job.
The iLO re-addressed on 11-17The agent working on the Kubernetes lab changed the iLO's address with hponcfg at 02:32, because it clashed with a control plane's address, and the iLO resetNot necessary: the 10-07 and 11-14 crashes came before it. Nothing still sends traffic in a way that fits. What it did to the iLO's internal state is unknown (candidate B).
The third control plane (11-17 22:10)Added the day beforeNot necessary: the 11:03 boot had no control plane and still crashed. A load multiplier at most.
The firmware CPU limitCores at about 1.1 GHzChronic since August: a factor in the load, not what changed.

8. Could it be an attack?

Both iLO functions on IRQ 22 belong to a management processor with its own network connection, so the question had to be asked.

  • No known pattern: no HPE advisory, driver note, Microsoft article or community report ties cpqcidrv or the iLO 2 channel interface to interrupt storms or 0x133.
  • The one real-world iLO implant, iLOBleed, was found only on iLO 4 (Gen8 and Gen9).
  • The iLO's own logs show no logins or failed logins since 2024 and no iLO firmware flash ever; the only flash on record is the system ROM in 2020. The one configuration change is the 11-17 re-addressing, explained above.
  • The network: the firewall publishes nothing to the iLO, and nothing came from it through the firewall.
  • The timing: the storms followed the owner's VM boot settings, which an outsider doesn't control, and the signature predates the configuration change.

Verdict (likely): a bug, not an attack. An implant can't be excluded with read-only checks, because a version string can be faked. So re-flashing the iLO firmware from HPE's package is on the list.


9. Where it stands (2026-11-20)

  • Up since 11-18 11:27, about 51 hours, with the Kubernetes VMs set not to start. There has been no crash, bugcheck, ASR reset or WHEA event since.
  • Monitored every 15 minutes since 11-19 (hosthealth-2026-11-19-to-20.csv, 103 samples).
    • The busiest CPU peaks at about 3,000 interrupts a second (CPU 0, which takes the clock), against 64,000 on CPU 1 in the storm.
    • DPC plus interrupt time stays under 6 %.
  • The driver is still cpqcidrv 1.17. On 11-19 the owner decided to install 1.18.0.0 (2013), HP's last build, which is already on the server.
    • HP's only note for 1.18 is support for 2012 R2 on some G6 servers. It isn't documented as a fix, and installing it won't prove anything on its own.
  • The Kubernetes lab is moving off this host (the owner's decision, 11-18). A new Hyper-V host is being specced for it. The domain controller stays.
  • A quiet month proves little. At the old rate, crashes on 3 days in 90, a crash-free 30 days happens about 37 % of the time even if nothing were fixed. And with Kubernetes gone, the suspected trigger is gone too.

10. The tests that would settle it

Each is the owner's decision, and some can take the domain controller down.

  1. Name the fifth device on vector 0x91. One kd command on the full dump shows whether the object behind cpqcidrv is the iLO's virtual USB controller. Copy the dump to an expendable machine first: kd isn't run on VM hosts any more.
  2. Read the iLO's own event log around the onset times (09:11, 09:52, 10:09, 10:56, 11:19) for resets, faults or floods.
  3. Log the counters before any test, at 1-second intervals: CPU 1's interrupts per second and % interrupt time, each USB controller's interrupts per second, and the root virtual processors' run time. A jump on CPU 1 from about 1,000 to about 64,000 interrupts a second, 1½ to 2 minutes before the bugcheck, is the storm.
  4. The A/B test. Cold-start only the two worker VMs, as in the 11:03 boot: once with the CpqCiDrv device disabled in Windows (no reboot, easy to undo), and once as now.
    • If only the second storms: the HP driver and iLO pair is the cause, and VM load is the trigger.
    • If both storm: this page is wrong.
  5. A 60-second kernel trace of ISRs and DPCs on the quiet host, to measure the delay between cpqcidrv's ISR and its DPC. That's the window candidate A depends on.
  6. The strongest test: set "Lights-Out Functionality: Disabled" in the iLO's setup at boot. The iLO's PCI functions, the virtual USB controller included, should disappear from Windows. It costs the remote console and power control, and possibly the hang watchdog.
  7. Re-flash iLO 2 firmware 2.33 from HPE's package, to rule out an implant completely.
  8. For the cluster's new home:
    • Generation 2 VMs, so no emulated IDE;
    • staggered starts;
    • merged checkpoints;
    • no dynamic memory on the control planes.

What would prove this page wrong:

  • a future 0x133 dump with a normal InterruptRate on every CPU, and cpqcidrv's IsrCount close to its DpcCount;
  • or a storm with the CpqCiDrv device disabled.

11. Open questions

  • What keeps IRQ 22 asserted: the iLO's virtual USB controller behind cpqcidrv, or the iLO itself?
  • What made that controller 30 times busier in the crash boots?
  • Why did the first storm hit at 05:11, in steady state?
  • Were the 10-07 and 11-14 crashes the same livelock? No dumps survive.
  • Do the guests use Hyper-V's synthetic timers and enlightened IPIs? This has never been checked, and the VMs must be running to check it.
  • Did removing Kubernetes fix it, or only make it rare?

12. Lessons so far

  1. A DPC watchdog names the victim. When five dumps blame four different functions, the cause is underneath all of them.
  2. Check InterruptRate in every CPU's PRCB first. About 1,000 per tick on one CPU, with the others idle, is an interrupt storm, not a slow DPC.
  3. The interrupt objects keep their own books. ISRDPCSTATS gives each ISR's claim count, its DPC count and when its DPC last ran, even when !intstats fails. Walking PRCB.InterruptObject[] takes one kd loop (Appendix A).
  4. Read the DPC watchdog's profile buffer. It's in the minidump, and it shows whether the "stuck" code is looping or standing still.
  5. On a shared, level-triggered line, an ISR that claims too eagerly starves everything behind it. cpqcidrv's ISR checks its status bit, but not its own enable bit.
  6. Count crashes from the management controller's log. The first timeline, built from Windows' logs, had 15. The iLO's log had 17.
  7. Overload is a trigger, not a mechanism. "It stopped when we turned the VMs off" tells you what sets it off, not what breaks.
  8. CPU_MHZ: 850 is hexadecimal.
  9. Keep the dumps. Twelve of the seventeen crashes, and both earlier ones, left nothing to analyse. The full dump of the last crash carried this investigation.

Appendix A: The kd commands

Every interrupt object on the current CPU. One line per object: vector, address, ISR, IRQL, share flag, mode, claim count, ISR cycles, the last ISR's timestamp, DPC count and the next object in the chain. The offsets are for Windows Server 2019 build 17763; check them with dt nt!_KINTERRUPT and dt nt!_KPRCB on any other build.

.for (r $t2 = 0; @$t2 < 0x100; r $t2 = @$t2 + 1) { r $t3 = poi(@$prcb+0x2e80+@$t2*8); r $t4 = @$t3; r $t6 = 0; .while ((@$t6 < 6) & ($vvalid(@$t4, 0x100) != 0) & (@$t4 != 0)) { .printf "KINT V %x K %p isr %y irql %x shr %x mode %x cnt %p cyc %p last %p act %x dcnt %p nx %p\n", @$t2, @$t4, poi(@$t4+18), by(@$t4+5c), by(@$t4+64), dwo(@$t4+6c), poi(@$t4+c0), poi(@$t4+b0), poi(@$t4+b8), by(@$t4+e0), poi(@$t4+d8), poi(@$t4+8); r $t6 = @$t6 + 1; r $t4 = poi(@$t4+8) - 8; .if (@$t4 == @$t3) { r $t6 = 0n99 } } }

One object in full:

dt nt!_KINTERRUPT <address> ServiceRoutine Vector Irql ShareVector Mode
dt nt!_ISRDPCSTATS <address>+b0

The PRCB fields that show a storm:

dt nt!_KPRCB @$prcb InterruptCount InterruptRate InterruptLastCount InterruptTime DpcTime DpcWatchdogCount

The DPC watchdog's profile buffer. On this build, DpcWatchdogProfile is at PRCB+0x61a8 and DpcWatchdogProfileCurrentEmptyCapture at +0x61b0.

.printf "profile %p .. %p\n", poi(@$prcb+61a8), poi(@$prcb+61b0)
dps poi(@$prcb+61a8) poi(@$prcb+61b0)

All of these are in the kd folder of the evidence, with their output.


Appendix B: Evidence files

Warning: The kd command files are input for kd.exe. If you rerun them, do what case study 301 teaches: give kd a standard input that stays open, set a time limit, and never run it on a machine that carries VMs you can't lose.

Download everything (evidence.zip, 453.8 KB)

Or open any file here; each opens as text in a new tab.

Raw data behind this case study. Times are local (UTC−8) unless a column says UTC.

In the published copy, the user profile in paths is shown as <user>, and machines go by stand-in names (tools\casestudy.py, PUBLISH.json).

timeline

Compiled from SERVER2's System log (6008, 41, 1001), the Hyper-V Worker log (VM starts and stops), the Application log (Windows Error Reporting) and the iLO's Integrated Management Log (IML), read on 2026-11-18.

  • bugchecks.csv: the 17 bugchecks of 2026-11-18, plus the earlier ones on 11-14 and 10-07. Windows' last-alive time (6008), the IML minute, the debugger's time where a dump exists, the CPU, and Microsoft's WER bucket.
  • boot-cycles-2026-11-18.csv: each boot of the loop. When it started, when its VMs started, how many virtual CPUs were running at the crash, and how long after the last VM start it died.
  • vms.csv: the VMs on SERVER2 and how the Kubernetes VMs were built.

dumps

kd output for the five minidumps that survived (minidump-HHMM-analyze.txt, !analyze -v and more) and for the full kernel dump of the 11:20:59 crash (full-dump-1120-*.txt, five passes).

FileCrashCPU
minidump-0913-analyze.txt09:13:101
minidump-0952-analyze.txt09:52:273
minidump-1011-analyze.txt10:11:451
minidump-1058-analyze.txt10:58:061
minidump-1120-analyze.txt11:20:591
full-dump-1120-*.txt11:20:59 (MEMORY.DMP)1

kd

The targeted runs that found the storm. Each kdq-m1-N-commands.txt is the command file kd read, and kdq-m1-N-output.txt is what it printed.

RunDumpWhat it looks at
1fullvertarget, every CPU's PRCB (interrupt counts and times, DPC data, idle thread, power state), !cpuinfo
2fullthe DPC watchdog's profile buffer, every interrupt object on every CPU with its ISR and DPC counts, the TCP timer wheels, the threads waiting on CPU 1
309:13 and 09:52 minidumpsstacks, PRCB, interrupt objects, pending DPCs (only the first dump loaded; see the log)
410:11 minidumpstack, PRCB, interrupt objects, !dpcs, profile buffer
510:58 minidumpthe same
609:52 minidumpthe same, plus the lock the CPU 3 DPC spins on
7fullcpqcidrv: module dates, its ISR and DPC disassembled, its interrupt object and ISR/DPC statistics, the objects sharing its vector
809:13 minidumpPRCB, the interrupt chain's ISR/DPC statistics, profile buffer

The walk over PRCB.InterruptObject[] in runs 2 to 6 uses offsets for this build of Windows Server 2019 (17763). Check them with dt nt!_KINTERRUPT before using it on another build.

health

  • hosthealth-2026-11-19-to-20.csv: SERVER2's 15-minute health record since monitoring started on 11-19. It shows uptime, crashes, bugchecks, ASR and WHEA events, the busiest CPU's interrupts per second and DPC-plus-interrupt time, the VMs running, and the HP driver versions.

Timeline

Crash dump analyses

Targeted kd runs: commands and output

Since the crash loop

Top level






Watch Promo Video Now!

Latest Modules: 166 114 165 164 082 005 004 002 001 055 197 056 037 036 033 032 031 030 008 111 175


Suggested Training Tracks

Intro Theory: 001 002  004 005 006 008 009

Blue Screen: 036

Debugger Setup: 032 033 037 055 197

Debugging Concepts: 008 056

Debugger Commands: 082

Debugs By Type: 181

Debugging Managed Apps: 089

Things Pertaining To Dump Files: 036

Heap: 111 114 166

High CPU: 165 193

Architectural Theory: 008

Deadlock Strategies:  198

Memory Leak Strategies: 111 115 164 192

Advanced Kernel Debugging Techniques: 175 192 256

Labs: 164 165 166 181 256

Case Studies: 301

Ongoing Investigations: 302
Background photo: Overheating marks at a disassembled bearing shell, Cologne, Germany by CEphoto, Uwe Aranas, licensed CC BY-SA 3.0. Mirrored and resized by this site; the changed photo is shared under the same licence.