Virtual machines with PCI passthrough on Ubuntu 20.04, straightforward guide for gaming on a virtual machine

IOMMU groups on ASUS Prime x370-pro

Preamble

The direct way to a PCI passthrough virtual machines on Ubuntu 20.04 LTS. I try limit changes of the host operating system to a minimum, but provide enough details, that even Linux rookies are able to participate.

The final system will run Xubuntu 20.04 as host operating system(OS), and Windows 10 2004 as guest OS. Gaming is the main use-case of the guest system.

Unfortunately, the setup process can be pretty complex. It consists of fixed base settings, some variable settings and several optional (mostly performance) settings. In order to sustain readability of this post, and because I aim to use the virtual machine for gaming only, I minimized the variable parts for latency optimization. The variable topics itself are linked in articles – I hope this makes sense. 🙂

About this guide

This guide targets Ubuntu 20.04 and is based on my former guides for Ubuntu 18.04 and 16.04 host systems.

However, this guide should be also applicable to Pop!_OS 19.04 and newer. If you wish to proceed with Pop!_OS as host system you can do so, just look out for my colorful Pop!_OS labels.

Breaking changes for older passthrough setups in Ubuntu 20.04

Kernel version

Starting with kernel version 5.4, the “vfio-pci” driver is no longer a kernel module, but build-in into the kernel. We have to find a different way instead of using initramfs-tools/modules config files, as I recommended in e.g. the 18.04 guide.

QEMU version

Ubuntu 20.04 ships with QEMU version 4.2. It introduces audio fixes. This needs attention if you want to use a virtual machine definition from an older version.

If you want to use a newer version of QEMU, you can build it on your own.

Attention!

QEMU version 5.0.0 – 5.0.0-6 should not be used for a passthrough setup due to stability issues.

Bootmanager

This is no particular “problem” with Ubuntu 20.04, but at least for one popular distro which is based on Ubuntu – Pop!_OS 20.04.

I think, starting with version 19.04, Pop!_OS uses systemd as boot manager, instead of grub. This means triggering kernel commands works differently in Pop!_OS.

Introduction to VFIO, PCI passthrough and IOMMU

Virtual Function I/O (or VFIO) allows a virtual machine (VM) direct access to a PCI hardware resource, such as a graphics processing unit (GPU). Virtual machines with set up GPU passthrough can gain close to bare metal performance, which makes running games in a Windows virtual machine possible.

Let me make the following simplifications, in order to fulfill my claim of beginner friendliness for this guide:

PCI devices are organized in so called IOMMU groups. In order to pass a device over to the virtual machine, we have to pass all the devices of the same IOMMU group as well. In a prefect world each device has its own IOMMU group — unfortunately that’s not the case.

Passed through devices are isolated, and thus no longer available to the host system. Furthermore, it is only possible to isolate all devices of one IOMMU group at the same time.

This means, even when not used in the VM, a device can no longer be used on the host, when it is an “IOMMU group sibling” of a passed through device.

Requirements

Hardware

In order to successfully follow this guide, it is mandatory that the used hardware supports virtualization and has properly separated IOMMU groups.

The used hardware

  • AMD Ryzen7 1800x (CPU)
  • Asus Prime-x370 pro (Mainboard)
  • 2x 16 GB DDR4-3200 running at 2400MHz (RAM)
  • Nvidia GeForce 1050 GTX (Host GPU; PCIE slot 1)
  • Nvidia GeForce 1060 GTX (Guest GPU; PCIE slot 2)
  • 500 GB NVME SSD (Guest OS; M.2 slot)
  • 500 GB SSD (Host OS; SATA)
  • 750W PSU

When composing the systems’ hardware, I was eager to avoid the necessity of kernel patching. Thus, the ACS override patch is not required for said combination of mainboard and CPU.

If your mainboard has no proper IOMMU separation you can try to solve this by using a patched kernel, or patch the kernel on your own.

BIOS settings

Enable the following flags in your BIOS:

  • Advanced \ CPU config - SVM Module -> enable
  • Advanced \ AMD CBS - IOMMU -> enable

Attention!

The ASUS Prime x370/x470/x570 pro BIOS versions for AMD RYZEN 3000-series support (v. 4602 – 5220), will break a PCI passthrough setup.

Error: “Unknown PCI header type ‘127’“.

BIOS version up to (and including) 4406, 2019/03/11 are working.

BIOS version from (and including) 5406, 2019/11/25 are working.

I used Version: 4207 (8th Dec 2018)

Host operating system settings

I have installed Xubuntu 20.04 x64 (UEFI) from here.

Ubuntu 20.04 LTS ships with kernel version 5.4 which works good for VFIO purposes – check via: uname -r

Attention!

Any kernel, starting from version 4.15, works for a Ryzen passthrough setup.

Except kernel versions 5.1, 5.2 and 5.3 including all of their subversion.

Before continuing make sure that your kernel plays nice in a VFIO environment.

Install the required software

Install QEMU, Libvirt, the virtualization manager and related software via:

sudo apt install qemu-kvm qemu-utils libvirt-daemon-system libvirt-clients bridge-utils virt-manager ovmf

Setting up the PCI passthrough

We are going to passthrough the following devices to the VM:

  • 1x GPU: Nvidia GeForce 1060 GTX
  • 1x USB host controller
  • 1x SSD: 500 GB NVME M.2

Enabling IOMMU feature

Enable the IOMMU feature via your grub config. On a system with AMD Ryzen CPU run:

sudo nano /etc/default/grub

Edit the line which starts with GRUB_CMDLINE_LINUX_DEFAULT to match:

GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt"

In case you are using an Intel CPU the line should read:

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"

Once you’re done editing, save the changes and exit the editor (CTRL+x CTRL+y).

Afterwards run:

sudo update-grub

→ Reboot the system when the command has finished.

Verify if IOMMU is enabled by running after a reboot:

dmesg |grep AMD-Vi

dmesg output

[ 0.792691] AMD-Vi: IOMMU performance counters supported 
[ 0.794428] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 
[ 0.794429] AMD-Vi: Extended features (0xf77ef22294ada): 
[ 0.794434] AMD-Vi: Interrupt remapping enabled 
[ 0.794436] AMD-Vi: virtual APIC enabled 
[ 0.794688] AMD-Vi: Lazy IO/TLB flushing enabled

[collapse]

For systemd boot manager as used in Pop!_OS

One can use the kernelstub module, on systemd booting operating systems, in order to provide boot parameters. Use it like so:

sudo kernelstub -o "amd_iommu=on amd_iommu=pt"

Identification of the guest GPU

Attention!

After the upcoming steps, the guest GPU will be ignored by the host OS. You have to have a second GPU for the host OS now!

In this chapter we want to identify and isolate the devices before we pass them over to the virtual machine. We are looking for a GPU and a USB controller in suitable IOMMU groups. This means, either both devices have their own group, or they share one group.

The game plan is to apply the vfio-pci driver to the to be passed-through GPU, before the regular graphic card driver can take control of it.

This is the most crucial step in the process. In case your mainboard does not support proper IOMMU grouping, you can still try patching your kernel with the ACS override patch.

One can use a bash script like this in order to determine devices and their grouping:

#!/bin/bash
# change the 999 if needed
shopt -s nullglob
for d in /sys/kernel/iommu_groups/{0..999}/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;

source: wiki.archlinux.org + added sorting for the first 999 IOMMU groups

script output


IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 2 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 6 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 7 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 9 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 10 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 11 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 12 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 12 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU Group 13 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU Group 13 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU Group 13 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU Group 13 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU Group 13 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU Group 13 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
IOMMU Group 13 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
IOMMU Group 14 01:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology P1 NVMe PCIe SSD [c0a9:2263] (rev 03)
IOMMU Group 15 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset USB 3.1 xHCI Controller [1022:43b9] (rev 02)
IOMMU Group 15 02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset SATA Controller [1022:43b5] (rev 02)
IOMMU Group 15 02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset PCIe Upstream Port [1022:43b0] (rev 02)
IOMMU Group 15 03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 15 03:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 15 03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 15 03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 15 03:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 15 03:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 15 07:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1143 USB 3.1 Host Controller [1b21:1343]
IOMMU Group 15 08:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 15 09:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 04)
IOMMU Group 15 0a:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788]
IOMMU Group 16 0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] [10de:1c82] (rev a1)
IOMMU Group 16 0b:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
IOMMU Group 17 0c:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1060 6GB] [10de:1b83] (rev a1)
IOMMU Group 17 0c:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU Group 18 0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
IOMMU Group 19 0d:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
IOMMU Group 20 0d:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1022:145c]
IOMMU Group 21 0e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
IOMMU Group 22 0e:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 23 0e:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]

[collapse]

The syntax of the resulting output reads like this:

iommu script output description
Figure1: IOMMU script output description

The interesting bits are the PCI bus id (marked dark red in figure1) and the device identification (marked orange in figure1).

These are the devices of interest:

selected devices for isolation


NVME M.2 
========
IOMMU group 14
        01:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology P1 NVMe PCIe SSD [c0a9:2263] (rev 03)
                Driver: nvme

Guest GPU - GTX1060 
=================
IOMMU group 17
        0c:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1060 6GB] [10de:1b83] (rev a1)
                Driver: nvidia
        0c:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
                Driver: snd_hda_intel

USB host
=======
IOMMU group 20
        0d:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1022:145c]
                Driver: xhci_hcd

[collapse]
IOMMU groups on ASUS Prime x370-pro
Figure2: IOMMU groups for virtual machine passthrough, on ASUS Prime x370-pro

We will isolate the GTX 1060 (PCI-bus 0c:00.0 and 0c:00.1; device id 10de:1b83, 10de:10f0).

The USB-controller (PCI-bus 0d:00.3; device id 1022:145c) is used later on.

The NVME SSD can be passed through without identification numbers. It is crucial though that it has its own group.

Isolation of the guest GPU

In order to isolate the GPU we have two options. Select the devices by PCI bus address or by device ID. Both options have pros and cons.

Apply VFIO-pci driver by device id (via bootmanager)

This option should only be used, in case the graphics cards in the system are not exactly the same model.

Update the grub command again, and add the PCI device ids with the vfio-pci.ids parameter.

Run sudo nano /etc/default/grub and update the GRUB_CMDLINE_LINUX_DEFAULT line again:

GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt kvm.ignore_msrs=1 vfio-pci.ids=10de:1b83,10de:10f0"

Remark! The command “ignore_msrs” is only necessary for Windows 10 versions higher 1803 (otherwise BSOD).

Save and close the file. Afterwards run:

sudo update-grub

Attention!

After the following reboot the isolated GPU will be ignored by the host OS. You have to use the other GPU for the host OS NOW!

→ Reboot the system.

Retrospective, for a systemd boot manager system like Pop!_OS 19.04 and newer, you can use:

sudo kernelstub --add-options "vfio-pci.ids=10de:1b80,10de:10f0,8086:1533"

Apply VFIO-pci driver by PCI bus id (via script)

This method works even if you want to isolate one of two identical cards. Attention though, in case PCI-hardware is added or removed from the system the PCI bus ids will change (also sometimes after BIOS updates).

Create another file via sudo nano /etc/initramfs-tools/scripts/init-top/vfio.sh and add the following lines:

#!/bin/sh

PREREQ=""

prereqs()
{
   echo "$PREREQ"
}

case $1 in
prereqs)
   prereqs
   exit 0
   ;;
esac

for dev in 0000:0c:00.0 0000:0c:00.1 
do 
 echo "vfio-pci" > /sys/bus/pci/devices/$dev/driver_override 
 echo "$dev" > /sys/bus/pci/drivers/vfio-pci/bind 
done

exit 0

Thanks to /u/nazar-pc on reddit. Make sure the line

for dev in 0000:0c:00.0 0000:0c:00.1

Has the correct PCI bus ids for the GPU you want to isolate. Now save and close the file.

Make the script executable via:

sudo chmod +x /etc/initramfs-tools/scripts/init-top/vfio.sh

Create another file via sudo nano /etc/initramfs-tools/modules

And add the following lines:

options kvm ignore_msrs=1

Save and close the file.

When all is done run:

sudo update-initramfs -u -k all

Attention!

After the following reboot the isolated GPU will be ignored by the host OS. You have to use the other GPU for the host OS NOW!

→ Reboot the system.

Verify the isolation

In order to verify a proper isolation of the device, run:

lspci -nnv

find the line "Kernel driver in use" for the GPU and its audio part. It should state vfio-pci.

output

0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1060 6GB] [10de:1b83] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: ASUSTeK Computer Inc. GP104 [1043:8655]
	Flags: fast devsel, IRQ 44
	Memory at f4000000 (32-bit, non-prefetchable) [size=16M]
	Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Memory at d0000000 (64-bit, prefetchable) [size=32M]
	I/O ports at d000 [size=128]
	Expansion ROM at f5000000 [disabled] [size=512K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

[collapse]

Congratulations, the hardest part is done! 🙂

Creating the Windows virtual machine

The virtualization is done via an open source machine emulator and virtualizer called QEMU. One can either run QEMU directly, or use a GUI called virt-manager in order setup, and run a virtual machine.

I prefer the GUI. Unfortunately not every setting is supported in virt-manager. Thus, I define the basic settings in the UI, do a quick VM start and “force stop” it right after I see the GPU is passed-over correctly. Afterwards one can edit the missing bits into the VM config via virsh edit.

Make sure you have your windows ISO file, as well as the virtio windows drivers downloaded and ready for the installation.

Pre-configuration steps

As I said, lots of variable parts can add complexity to a passthrough guide. Before we can continue we have to make a decision about the storage type of the virtual machine.

Creating image container if needed.

In this guide I pass my NVME M.2 SSD to the virtual machine. Another viable solution is to use a raw image container, see the storage post for further information.

Creating an Ethernet Bridge

We will use a bridged connection for the virtual machine. This requires a wired connection to the computer.

I simply followed the great guide from heiko here.

see the ethernet setups post for further information.

Create a new virtual machine

As said before we use the virtual machine manager GUI to create the virtual machine with basic settings.

In order to do so start up the manager and click the “Create a new virtual machine” button.

Step 1

Select “Local install media” and proceed forward (see figure 3).

Virt-manager select installation type for operating system
Virt-manager select installation type for operating system

Step2

Now we have to select the windows ISO file we want to use for the installation (see figure3). Also check the automatic system detection. Hint:Use the button “browse local” (one of the buttons on the right side) to browse to the ISO location.

select the windows iso file
Figure4: Create a virtual machine step 2 – select the windows ISO file.

Step 3

Put in the amount of RAM and CPU cores you want to passthrough and continue with the wizard. I want to use 8 Cores (16 is maximum, the screenshot shows 12 by mistake!) and 16384 MiB of RAM in my VM.

Figure4: Memory and CPU settings.
Figure5: Create a virtual machine step 3 – Memory and CPU settings

Step 4

In case you use a storage file, select your previously created storage file and continue. I uncheck the “Enable storage for this virtual machine” check-box and add my device later on.

Figure5: Create a virtual machine step 4 - Select the previous created storage.
Figure6: Create a virtual machine step 4 – Select the previous created storage.

Step 5

On the last steps are slightly more clicks required.

Put in a meaningful name for the virtual machine. This becomes the name of the xml config file, I guess I would not use anything with spaces in it. It might work without a problem, but I wasn’t brave enough to do so in the past.

Furthermore make sure you check “Customize configuration before install”.

For the “network selection” pick “Specify shared device name” and type in the name of the network bridge we had created previously. You can use ifconfig in a terminal to show your Ethernet devices. In my case that is “bridge0”.

Figure6: Create a virtual machine step 5 - Before installation
Figure7: Create a virtual machine step 5 – Before installation.

Remark! The CPU count in Figure7 is wrong. It should say 8 if you followed the guide.

First configuration

Once you have pressed “finish” the virtual machine configuration window opens. The left column displays all hardware devices which this VM uses. By left clicking on them, you see the options for the device on the right side. You can remove hardware via right click. You can add more hardware via the button below. Make sure to hit apply after every change.

The following screenshots may vary slightly from your GUI (as I have added and removed some hardware devices).

Overview

On the Overview entry in the list make sure that for “Firmware” UEFIx86_64 [...] OVMF [...] is selected. “Chipset” should be Q35 see figure8.

Figure7: Overview configuration
Figure8: Virtual machine configuration – Overview configuration

CPUs

For the “Model:” click in to the drop-down, as if it is a text field, and type in

host-passthrough

This will pass all CPU information to the guest. You can read the CPU Model Information chapter in the performance guide for further information.

For “Topology” check “Manually set CPU topology” with the following values:

  • Sockets: 1
  • Cores: 4
  • Threads: 2
Figure8: CPU configuration
Figure9: Virtual machine configuration – CPU configuration *outdated screenshot

Disk 1, the Guests hard drive

Depending on your storage pre-configuration step you have to choose the adequate storage type for your disk (raw image container or passed-through real hardware). See the storage article for further options.

Setup with raw image container

When you first enter this section it will say “IDE Disk 1”. We have to change the “Disk bus:” value to VirtIO.

Figure9: Disk configuration
Figure10: Virtual machine configuration – Disk configuration
Setup with passed-through drive

Find out the correct device-id of the drive via lsblk to get the disk name.

When editing the VM configuration for the first time add the following block in the <devices> section (one line after </emulator> should be fitting).

<disk type='block' device='disk'>
  <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
  <source dev='/dev/nvme0n1'/>
  <target dev='sdb' bus='sata'/>
  <boot order='1'/>
  <address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>

Edit the line

<source dev='/dev/nvme0n1'/>

To match your drive name.

VirtIO Driver

Next we have to add the virtIO driver ISO, so it is used during the Windows installation. Otherwise, the installer can not recognize the storage volume we have just changed from IDE to virtIO.

In order to add the driver press “Add Hardware”, select “Storage” select the downloaded image file.

For “Device type:” select CD-ROM device. For “Bus type:” select IDE otherwise windows will also not find the CD-ROM either 😛 (see Figure10).

Figure10: Adding virtIO driver CDROM.
Figure11: Virtual machine configuration – Adding virtIO driver CD-ROM.

The GPU passthrough

Finally! In order to fulfill the GPU passthrough, we have to add our guest GPU and the USB controller to the virtual machine. Click “Add Hardware” select “PCI Host Device” and find the device by its ID. Do this three times:

  • 0000:0c:00.0 for GeForce GTX 1060
  • 0000:0c:00.1 for GeForce GTX 1060 Audio
  • 0000:0d:00.3 for the USB controller
Figure11: Adding PCI devices.
Figure12: Virtual machine configuration – Adding PCI devices (screenshot is still with old hardware).

Remark: In case you later add further hardware (e.g. another PCIE device), these IDs might/will change – just keep in mind if you change the hardware just redo this step with updated Ids.

This should be it. Plugin a second mouse and keyboard in the USB ports of the passed-through controller (see figure2).

Hit “Begin installation”, a Tiano core logo should appear on the monitor connected to the GTX 1060. If a funny white and yellow shell pops-up you can use exit in order to leave it.

When nothing happens, make sure you have both CD-ROM device (one for each ISO windows10 and virtIO driver) in your list. Also check the “boot options” entry.

Once you see the Windows installation, use “force off” from virt-manager to stop the VM.

Final configuration and optional steps

In order edit the virtual machines configuration use: virsh edit your-windows-vm-name

Once your done editing, you can use CTRL+x CTRL+y to exit the editor and save the changes.

I have added the following changes to my configuration:

AMD Ryzen CPU optimizations

I moved this section in a separate article – see the CPU pinning part of the performance optimization article.

Hugepages for better RAM performance

This step is optionaland requires previous setup: See the Hugepages post for details.

find the line which ends with </currentMemory> and add the following block behind it:

  <memoryBacking>   
    <hugepages/> 
  </memoryBacking>

Remark: Make sure <memoryBacking> and <currentMemory> have the same indent.

Performance tuning

This acrticle describes performance optimizations for gaming on a virtual machine (VM) with GPU passthrough.

Troubleshooting

Removing Error 43 for Nvidia cards

This guide uses an Nvidia card as guest GPU. Unfortunately, the Nvidia driver throws Error 43 , if it recognizes the GPU is being passed through to a virtual machine.

I rewrote this section and moved it into a separate article.

Getting audio to work

After some sleepless nights I wrote a separate article about that – see chapter Pulse Audio with QEMU 4.2 (and above).

Removing stutter on Guest

There are quite a few software- and hardware-version combinations, which might result in weak Guest performance. I have created a separate article on known issues and common errors.

My final virtual machine libvirt XML configuration

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
  <name>win10-q35</name>
  <uuid>b89553e7-78d3-4713-8b34-26f2267fef2c</uuid>
  <title>Windows 10 20.04</title>
  <description>Windows 10 18.03 updated to 20.04 running on /dev/nvme0n1 (500 GB)</description>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">16777216</memory>
  <currentMemory unit="KiB">16777216</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement="static">8</vcpu>
  <iothreads>2</iothreads>
  <cputune>
    <vcpupin vcpu="0" cpuset="8"/>
    <vcpupin vcpu="1" cpuset="9"/>
    <vcpupin vcpu="2" cpuset="10"/>
    <vcpupin vcpu="3" cpuset="11"/>
    <vcpupin vcpu="4" cpuset="12"/>
    <vcpupin vcpu="5" cpuset="13"/>
    <vcpupin vcpu="6" cpuset="14"/>
    <vcpupin vcpu="7" cpuset="15"/>
    <emulatorpin cpuset="0-1"/>
    <iothreadpin iothread="1" cpuset="0-1"/>
    <iothreadpin iothread="2" cpuset="2-3"/>
  </cputune>
  <os>
    <type arch="x86_64" machine="pc-q35-4.2">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/OVMF/OVMF_CODE.ms.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10-q35_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vpindex state="on"/>
      <synic state="on"/>
      <stimer state="on"/>
      <reset state="on"/>
      <vendor_id state="on" value="1234567890ab"/>
      <frequencies state="on"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <ioapic driver="kvm"/>
  </features>
  <cpu mode="host-passthrough" check="none">
    <topology sockets="1" cores="4" threads="2"/>
    <cache mode="passthrough"/>
    <feature policy="require" name="topoext"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="block" device="disk">
      <driver name="qemu" type="raw" cache="none" io="native" discard="unmap"/>
      <source dev="/dev/nvme0n1"/>
      <target dev="sdb" bus="sata"/>
      <boot order="1"/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/mrn/Downloads/virtio-win-0.1.185.iso"/>
      <target dev="sdc" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="2"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x17"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x8"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </controller>
    <controller type="pci" index="10" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </controller>
    <interface type="bridge">
      <mac address="52:54:00:fe:9f:a5"/>
      <source bridge="bridge0"/>
      <model type="virtio-net-pci"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0c" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0c" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0d" slot="0x00" function="0x3"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </hostdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value="-device"/>
    <qemu:arg value="ich9-intel-hda,bus=pcie.0,addr=0x1b"/>
    <qemu:arg value="-device"/>
    <qemu:arg value="hda-micro,audiodev=hda"/>
    <qemu:arg value="-audiodev"/>
    <qemu:arg value="pa,id=hda,server=unix:/run/user/1000/pulse/native"/>
  </qemu:commandline>
</domain>

to be continued…


Sources

The glorious Arch wiki

heiko-sieger.info: Really comprehensive guide

Great post by user “MichealS” on level1techs.com forum

Wendels draft post on Level1techs.com

Updates

  • 2020-06-18 …. initial creation of the 20.04 guide
  • 2020-09-09 …. added further Pop!_OS remarks and clarified storage usage

24 Comment

  1. BunnieofDoom says: Reply

    Good day Mathias,

    Do you know if there is a way to fix the ACPI tables? A game I am currently playing seems to use that check to flag if your playing from inside a VM and disconnect you.

    1. Mathias Hueber says: Reply

      Sorry, no idea. Which game is it though?

  2. Using this configuration, does the video output of the guest OS appear on the HDMI port of the guest GPU? And is the second keyboard/mouse required all the time or only during installation?

    I am planning to attach the HDMI output to a spare port on my monitor, but I don’t want a separate keyboard and mouse.

    1. Mathias Hueber says: Reply

      I use a kvm USB switch for mouse and keyboard. Depending on your use case (latency) there are other options too.

      The video output for the guest comes always directly from guest gpu.
      My monitor uses one hdmi for the host and display port for the guest input. I change input channels on the monitor directly to switch between host and guest. Depending on your monitor and gpu outputs you can choose what signals you use.

  3. Patrick says: Reply

    Given the lack of space on my desk, I’d also like to share my keyboard and mouse between host and guest. Since I’m a total beginner with all this, which solution would you recommend?

    I’ve found https://rokups.github.io/#!pages/kvm-hid.md and it looks like a good start, even though it’s already 2 years old. Not sure if that’s still the recommended way to do a single-keyboard setup?

    Also when you say latency, do you mean between each key press and the host/guest noticing the event or do you mean latency only when switching devices from host to guest? I wouldn’t mind the latter as I don’t switch so often but if the mouse was off by a second in the guest constantly, it’s unusable for gaming.

    Oh, and thanks for writing all this down, this is a great starting point for a vm-noob like me!

    1. Mathias Hueber says: Reply

      Yes, it’s the latency between each key press. Thus, software kvm solutions are usually not a good idea for fast paced games.

      I share my mouse and keyboard as well, but I use a kvm USB switch. It works pretty well. This eliminates the latency debt, at least for the hardware connection part.

  4. sonata says: Reply

    hi, it seems lost libvirt XML configuration file in the end of article. could you please update and append it ? thank you.

    1. Mathias Hueber says: Reply

      Thank you for the hint!

  5. Stephan says: Reply

    “In this guide I pass my NVME M.2 SSD to the virtual machine.”
    But how did you do that? Could you tell me?

    1. Stephan says: Reply

      Ah it was late at night. So add device id to grub file, change apparmor and add device in virtmanager. Thank you for this guide. I think it is was of the best guides for passthrough on the internet.

      You say “In order to isolate the GPU we have two options. Select the devices by PCI bus address or by device ID. Both options have pros and cons. ”
      Do you maybe have more information about this, a link maybe?

      1. Mathias Hueber says: Reply

        > So add device id to grub file, change apparmor and add device in virtmanager.

        The NVME SSD can be passed through without identification numbers. It is crucial though that it has its own group. I also didn’t add the apparmor stuff. The storage article was written in combination with the 18.04 version – I guess I have to clean this up a bit.

        > Do you maybe have more information about this, a link maybe?

        If you select the device by device ID you can not distinguish between cards if you are using the same card, as both cards will have the same device ID
        If you select the device by bus address you are basically selecting via the “possition” on the board. This lets you select one card even if both are identically (brand/model). The downside is that bus adresses change as soon as you add or remove hardware.

        I hope this helps

  6. Maxime says: Reply

    Thanks for the guide, very insightful !

    A couple questions though :

    – Can you achieve 240 FPS/Herz ?
    – What is the input lag like ? (assuming the guest Keyboard & Mouse are plugged directly)
    – Does it work with a dedicated Nvidia card for the guest and the onboard graphic chip available on the Intel CPUs ?

  7. Eli says: Reply

    I have an issue when I try to Begin installation and it gives this error https://pastebin.com/n162y61b (group 14 is not viable). When I do lspci -nnv it shows
    Kernel driver in use: vfio-pci
    Kernel modules: radeon
    like its supposed to so I don’t fully understand what is wrong. I am new to linux and would like help. Thanks for reading.

    1. Mathias Hueber says: Reply

      The error message of lspci -nnv says: Please ensure all devices within the iommu_group are bound to their vfio bus driver.
      Which devices are part of the IOMMU group? You have to unbind all devices which are part of the same IOMMU. In case your mainboard does not support propper IOMMU groupings you can tinker around with the ACS patch.

      1. Eli says: Reply

        After using
        dmesg |egrep group |awk ‘{print $NF” “$0}’ |sort -n
        in the terminal,(https://pastebin.com/xNZVQiZ7) it looks like I failed to isolate the gpu from the group. Do you have a link or more information on how to isolate/unbind all devices? I’m sorry this is very new to me and I appreciate the help so far.

        (In case it helps, I’m using a b450 aorus pro wifi with r7 2700x, Host 1050ti and Guest Radeon HD 4870)

        1. Mathias Hueber says: Reply

          Have you done the step “Identification of the guest GPU” what is the output of the script?

          1. Eli says:

            https://pastebin.com/wti8i8aQ
            Using this output, I followed the other steps and I copied what you did just with my identification number instead. But after looking at the group it is in, I think I did this incorrectly in some way. I did look through the other steps and double checked that I wrote the right id’s.

          2. Mathias Hueber says:

            Ok. Your plan is to pass the radeon card to the guest, correct?
            Currently, your radeon card is in iommu group 14. As you can see in the output of the script, there are additional devices (12 to be precise) in this group. If you want to pass one or more device(s) from group 14, you have to pass all 12 to the guest. This is not feasible as for example your network adapter is part of this group.
            A better group to pass to the guest is for example iommu group 15. Your Nvidia GPU is the only device in it – thus it is easy to pass. If you want to pass the radeon card you could try to switch the gpus on the mainboard.
            One more thing, the USB controller from iommu group 18 looks good for passing it to the guest.

          3. Eli says:

            Sorry it has taken me so long to respond. I have booted up the vm, but I do not see a gpu there it shows up like a normal vm without a gpu. https://imgur.com/a/lUIIyZx
            https://imgur.com/a/DfrrQnn I read the post by Amygdala but I don’t really understand the wiki that was linked. Thank you so much for the help.

  8. Amygdala says: Reply

    I’ve arrived at the point where the VM boots, but it displays the video output in the Virt-Manager window rather than on it’s own dedicated monitor. Specs:
    Ryzen 9 3900
    32GB RAM
    Host GPU: Nvidia Geforce GTX 1070
    VM GPU: Nvidia Geforce RTX 2070 Super
    VM Storage: Physical disk passed as block device.

    The 2070 has 4 devices in its IOMMU group, and I’ve added all of those device ids to grub. On boot the VM GPU does appear to be grabbed by the vfio-pci kernel module. When I boot the VM, the VM GPU screen goes blank, but nothing else. The VM display output instead appears in the virt-manager window. I tried removing the Display Spice device but that didn’t seem to help. Any suggestions would be most welcome

  9. Amygdala says: Reply

    Progress! Using info found in the Almighty Arch Wiki this gave me the display output on the VM monitor:
    https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#“BAR_3:_cannot_reserve_[mem]”_error_in_dmesg_after_starting_VM

    Now I assume I will need to run this every time I start the VM, so some sort of libvirt hook needs to be rolled. I had to execute these commands from an interactive root login, not from sudo. So that might be interesting..

  10. Sam says: Reply

    Hello Mathias, thank you for the detailed guide. Sorry if this is a foolish question. For background, I have one GPU and one sound card installed in the 2 PCIe x16 slots on my motherboard, i.e. no space to install another GPU.

    When you say “In order to pass a device over to the virtual machine, we have to pass all the devices of the same IOMMU group as well.”, would it be possible to set it up such that the GPU and sound card used by the host, then when the VM launches are passed through to the guest system (i.e. can no longer be used by the host), becoming available to the host again when the VM quits?

  11. Adam says: Reply

    Hi, thanks for the detailed guide, I’m trying it out on an Intel NUC with Nvidia eGPU. It has VT-d support, and the IOMMU groups look correct. I’m able to set the vfio-pci ids with grub, and after reboot the correct vfio driver is loaded (checking over SSH), but after the splash screen the ubuntu desktop login does not boot, just a black screen, it seems intel’s integrated graphics is not being used for the host (to be clear the monitor is plugged into the NUC not the eGPU). Any ideas? Thanks!

    1. Adam Suban-Loewen says: Reply

      Update: Still haven’t resolved the boot issue, ignoring the intel integrated graphics. But found a workaround: boot with the eGPU off, login to desktop environment displayed with intel integrated graphics, then turn on the eGPU and confirm it’s using vfio-pci as the driver. Everything else worked as expected, benchmarking surprisingly well. Thanks for such a great guide! If anyone has any insight to what’s going on with boot, please let me know.

Leave a Reply