Hardware passthrough using VFIO

Virtual Function I/O is a feature of Linux that allows assigning real hardware devices to a virtual machine so the guest OS has full control of it and can use it as on a real machine. This is useful to avoid overhead of emulating devices (e.g. assigning a network card) or to allow using devices such as GPUs that are not emulated by QEMU.

The guest OS will see passed through cards as if these were plugged into it as on a real machine, thus it needs appropriate guest drivers for these cards. This also means that passthrough cards have to be removed from the host driver and assigned to the vfio-pci driver instead, which means the host cannot use them any more and the guest will take full control of those cards. That also means you cannot pass through the video card or network card that the host uses for its output or network but may need a second card you don't use on the host to pass through or live with loosing graphics output from the host while the guest is running if you only have one GPU for example (that makes it more difficult to set up, so probably start with a second card first before trying more advanced setups).

There are several guides online for setting up GPU passthrough, mostly for Windows guest for gaming and using KVM. They also often suggest using high level tools such as libvirt and virt-manager which is supposed to make it simple but in reality just makes it more complicated than it needs to be. Reading the original presentation on VFIO one can see that this is actually quite simple. What makes it complex is all the differences between motherboards, CPUs, Linux distros and cards to be passed through. Additionally, GPU passthrough is more difficult because these have peculiarities that need to be considered and also the situation with GPU support on AmigaNG machines and different drivers on Amiga like OSes adds another level of complexity on top of that.

This page attempts to give a simple overview and demystify setup of passthrough of GPUs and other cards to Amiga like OSes running on QEMU, but it is not a "for dummies" or "quickly and easily" guide. You will still need to know what you are doing to achieve this, but hopefully will have more understanding after reading this (and linked) pages.

Prerequisites

Linux running on the host machine. As VFIO is a Linux feature this does not work on other host OSes.
A machine and CPU that supports IOMMU (may be called Intel VT-d or AMD-Vi). Recent hardware should work, here is an incomplete list of older ones.
Appropriate settings in BIOS or UEFI setup to enable IOMMU (or a firmware update may be needed). This is different for every vendor and may be found under one of the names above or called virtualisation or SVM on AMD. Depending on the card to be passed through other settings such as resizable BARs may need to be disabled.
Linux kernel boot option to enable IOMMU. The way to set it is different for each distro and boot loader. Add this to kernel command line:
```
intel_iommu=on iommu=pt
```
This should work for both Intel and AMD. Intel needs explicit option to enable IOMMU. For AMD it is enabled by default and ignores the Intel option, hence no option for that. The iommu=pt part may or may not help, it should also work without that.

Setup

Identify the card to be passed through and its IOMMU group. You can get a detailed list of devices with lspci -vvnn and an overview of how they are connected with
```
lspci -tvnn; ls /sys/kernel/iommu_groups/*/devices
```
If you see no IOMMU groups listed, go back to prerequisites.
The numbers identifying devices are domain:bus:device.function. Find the card you want to pass through and all the other devices that are in the same IOMMU group. All these devices in the group (except root PCI bridges) have to be assigned to vfio-pci. GPUs usually have another function for HDMI audio so you'll see at least two entries (plus maybe the root bridge). If there are more devices in the group you may have to move the card to a different slot or enable ACS option in firmware setup if available. Grouping is dependent on motherboard type so find a combination where the devices you want to pass through are in their own group. Note the [vendor:device] PCI IDs (shown after the name) of the devices in the IOMMU group for passthrough.
Assign devices to vfio-pci. Create a file /etc/modprobe.d/vfio.conf with the contents
```
options vfio-pci ids=1002:6779,1002:aa98
softdep drm pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci
```
where the vfio-pci ids are the vendor:device values for all devices (except the root PCI bridge) in the IOMMU group. The softdep lines ensure that vfio-pci is loaded before other drivers that may bind to these devices. If passing through something else than a GPU, you may need to add similar line for the driver used by that card.
Update initrd with a distro specific way (e.g. update-initramfs -u) if needed and reboot. Then check with lspci -vvnn that the driver in use for all cards in the IOMMU group to be passed through is vfio-pci (except root ports that use driver pcieport or similar). Ensure this is working before continuing.

Usage

To pass through the devices to the guest, add appropriate options to QEMU command. For example a graphics card may need something like this:

-vga none
-device vfio-pci,host=01:00.0,multifunction=on,x-vga=on
-device vfio-pci,host=01:00.1

where the host values are the bus:device.function address of the device. The graphics card has two functions so you need multifunction=on for the first function to let it know that there are other functions and it is a VGA card with legacy VGA resources that are to be passed through with x-vga=on to replace the emulated VGA which is disabled with -vga none otherwise it had conflicting VGA resources. You also need to run QEMU as root to be able to access VFIO devices. After this the card should appear in the guest virtual machine as a PCI device and the guest OS can use it as other PCI devices.

In principle that is all there is to it and this also works most of the time for simpler devices such as network or SATA cards. It may be a good idea to get such a simpler card working first just to see how it works generally before trying to get GPU passthrough work.

Complications

If your goal is to get better graphics by passing through a GPU to Amiga like OSes this is where it gets more complicated. I will assume you are familiar with what cards are supported on real machines. If not, check here for AmigaOS or here for MorphOS. Also consider that QEMU before version 10 only supported VFIO passthrough for RadeonHD and newer cards. Older cards may work now with QEMU v10.0.0 but are still untested for now. So first you need to find a card within those constraints that you can also plug in the host.

Some knowledge of the theory of operation of graphics cards on real machines is also useful. I try to give a quick summary here. Most graphics cards are made for x86 compatible PCs. These have an option ROM that is run by BIOS on a PC to initialise the card when the machine is switched on. To use such PC card on an AmigaNG machine, the machine's firmware has an x86 and BIOS emulator that runs this ROM. This emulator is not perfect and often crashes on newer cards (or the QEMU default video bios of the emulated cards) but despite of that it may run enough to get the card working, but then you will get no output until the OS driver loads. The OS driver may need the firmware to run the card's ROM first so if you boot with -kernel bboot and not with -bios machine.rom it may not work. But some newer Radeon cards have AtomBIOS which some drivers may know how to run so with some luck those cards may work even without the BIOS emulator in firmware run first. In short, you may need to use the firmware ROM for the machine if passing through GPUs to correctly initialise the card, so this may be more difficult than just getting AmigaOS boot on QEMU with BBoot. The sam460ex has the firmware included in QEMU. For amigaone and pegasos2 there is info here on how to get firmware ROM.

Other considerations specific for machines emulated by QEMU:

amigaone
Install CD does not come with basic RadeonHD driver so need to install first with -device sm501 and add driver from Enhancer Software before passthrough. Has oldest firmware so may have problems with newer cards. Least tested but was improved with QEMU v10.0.0 so now it may be the easiest to use.
pegasos2
AmigaOS kernel has problems on this machine so until an AmigaOS update is released it needs BBoot (possibly from within firmware ROM) to work around some of them. Some cards may need to use bus=pci.0 which did not work before QEMU v9.2.0.
sam460ex
PCIe does not work yet and its U-Boot cannot handle RadeonHD and newer on PCI and older cards were not supported before QEMU v10.0.0 so maybe now it can work with older cards or very old PCI cards but it is untested.