After reinstalling Windows 11 completely several times, I noticed there is an erroneous "Microsoft Basic Display Adapter" in Device Manager, which turns out to be the built-in graphics on my (workstation) motherboard. The GPU vendor is ASPEED and they are commonly seen on workstation or server motherboards to provide basic display output.
Apparently before the first reboot, Windows would also install the driver for the ASPEED GPU so upon reboot Windows would begin booting on whichever GPU I had connected to the display, but then once it reached a certain point in the boot process, the display would switch over to the "primary" display, which in some cases would be the other NVIDIA GPU, but in other cases would be the ASPEED GPU.
The solution is to disable the ASPEED GPU in device manager (named "Microsoft Basic Display Adapter") PRIOR to rebooting the computer. Once Windows decides the ASPEED GPU is the primary display, I'm not aware of a method to fix it other than attaching a display to the GPU (not possible for me as the GPU connector is via a motherboard header rather than a display connector on the motherboard rear I/O), or if your motherboard settings allow for disabling the integrated graphics (mine does not appear to have that option).
Fun stuff!