Not able to passthrough driver to qemu VM Ubuntu 20 and Ubuntu 22

[aryehassbian]

Hi,

I have installed the card on a KVM and made sure that all needed setup for the card to be pass through to a VM was set as it should.
Added in grub:
GRUB_CMDLINE_LINUX_DEFAULT="iommu=1 intel_iommu=on iommu=pt systemd.unified_cgroup_hierarchy=0 kvm.ignore_msrs=1 vfio-pci.ids=8086:56a1"
also added the following settings:
cat /lib/modules/$(uname -r)/modules.builtin | grep vfio returns:

kernel/drivers/vfio/vfio.ko
kernel/drivers/vfio/vfio_virqfd.ko
kernel/drivers/vfio/vfio_iommu_type1.ko
kernel/drivers/vfio/pci/vfio-pci-core.ko
kernel/drivers/vfio/pci/vfio-pci.ko

inside: /etc/modprobe.d/vfio.conf I have

blacklist nouveau
blacklist snd_hda_intel
options vfio-pci ids=8086:56a1

Just added the last line to what I already had.
inside : /etc/modules-load.d/vfio.conf I have vfio-pci

Card is being passthrough to the VM and I am able to see it using lspci -v |grep -A8 VGA from within the VM I get

09:00.0 VGA compatible controller: Intel Corporation Device 56a1 (rev 08) (prog-if 00 [VGA controller])
        Subsystem: Intel Corporation Device 1021
        Physical Slot: 0-5
        Flags: bus master, fast devsel, latency 0, IRQ 38
        Memory at f8000000 (64-bit, non-prefetchable) [size=16M]
        Memory at d0000000 (64-bit, prefetchable) [size=256M]
        Expansion ROM at f9000000 [disabled] [size=2M]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Endpoint, MSI 00

Installed all packages as in the installation guide provided for BOTH Ubuntu 20.04 and Ubuntu 22.04 and I am not able to get the igpu passthrough correctly. when I run stat -c "%G" /dev/dri/render* I get
stat: cannot stat '/dev/dri/render*': No such file or directory as far as I have learned that indicates that the card is not setup correctly.

The xml settings, of the vm used, is setup for the regular machine I am using with the following:

at this point reffering to Ubuntu 20.04
<type arch='x86_64' machine='pc-q35-focal-hpb'>hvm</type> using the pc-q35-focal-hpb and
<cpu mode='host-passthrough' check='none' migratable='on'/>

After installing the dkms I do get LMEM bar size issue that I also need to get addressed as well.
Just to make it clear - I did all the installation ONLY insdie the VM and the card had no installation for the driver inside the KVM hypervisor - I thought that since I passthrough the card there is no need for it get the driver and tools installed inside the KVM itself.

If there is something I a missing that can make the gpu passthrough as it should with the ability to get rendering on the VM as expected I would be happy to get assistance on it and that I will be able to get the card working inside the VM and get OpenCL as well.

This is a major show stopper and I really appreciate you assistance on getting this issue resolved and make the card work as expected inside a VM.

Thanks
Arye

[jketreno]

when I run stat -c "%G" /dev/dri/render* I get
stat: cannot stat '/dev/dri/render*': No such file or directory as far as I have learned that indicates that the card is not setup correctly.

Does dmesg give any errors related to i915 loading?

If you run the following within the VM, does it list that your device is bound to the i915 kernel module?

lspci -nnk | grep VGA -A 3 | grep -E "VGA|driver"

We recently published some instructions for performing PCI passthrough of the discrete GPU in a hybrid (integrated+discrete) system, which I'm using for headless VM. I haven't tried running an accelerated desktop in the VM though:

https://dgpu-docs.intel.com/driver/client/virtualized.html