Troubleshooting – Known issues, bugs and common quirks of KVM QEMU virtual machines with gpu passthrough

Hardware

Error “Unknown PCI header type ‘127’ “

Attention!

The ASUS Prime x370/x470/x570 pro BIOS versions for AMD RYZEN 3000-series support (version 4602 – version 5220), will break a PCI passthrough setup. Error “Unknown PCI header type ‘127’“.

BIOS versions up to (and including) 4406, 2019/03/11 are working.

BIOS versions from (and including) 5406, 2019/11/25 are working.

Host OS

Kernel versions

• 4.15 (18.04 LTS) – is good for VFIO use cases. One should avoid kernel below that, at least in combination with an AMD Ryzen processor as 4.15 has fixed the SMT bug.
• 4.16 – 5.0 are good for VFIO.
• 5.1 – 5.3 should be avoided for VFIO
• 5.4 (20.04 LTS) is good for VFIO. Starting with 5.4 the former Kernel module VFIO is now part of the kernel itself. This changes the activation syntax and break setups from former versions.
• 5.6 – 5.8 are good for VFIO.
• 5.9 – 5.10 should be avoided for VFIO. Has issues with Nvidia GPUs and libvirt.
• 5.11 is good for VFIO (especially nested-virtualization)

Guest OS 

Windows 10

Attention!

Windows releases >=1803 require the Host to have kvm ignore_msrs=1 enabled, otherwise BSOD occur.

This can be done by adding a file to modprobe.d, like so (the name doesn’t matter kvm.conf was chosen for maintainability):

sudo nano /etc/modprobe.d/kvm.conf

Insert:

options kvm ignore_msrs=1

Windows 10 1903

Attention!

Do not use 6ch/9ch audio devices in the virtual machine. It creates awful stuttering and performance loss. Use ac97 audio devices instead.

Error code 43 for passed through Nvidia GPUs

The Nvidia driver throws Error Code 43, if it recognizes the GPU is being passed through to a virtual machine. I wrote an separate article on that matter.

Sound errors

After some sleepless nights I wrote an separate article on that matter.

KVM / QEMU

QEMU 4.0 with Q35

Attention!

Starting with QEMU version 4.0, virtual machines with q35 chip will require <ioapic driver='kvm'/> under <features> tag in the libvirt configuration (respectively kernel_irqchip=on as QEMU-machine arg).

QEMU 5.0

QEMU version 5.0 – 5.0.0-6 should be avoided. 5.0.0-6.1 is the first usable version.