Random GPU passthru notes

Random GPU passthru notes

Setting up rlimits

if qemu says this:

2020-10-20T03:55:36.078813Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x7,multifunction=on,x-vga=on: VFIO_MAP_DMA failed: Cannot allocate memory

dmesg probably says something like this:

[  147.460769] vfio_pin_pages_remote: RLIMIT_MEMLOCK (65536) exceeded

Add this to /etc/security/limits.conf (at least on gentoo):

dave         hard memlock    20000000

dave         soft memlock    20000000

Giving myself permission to use vfio

I've never found a better way to do this, on boot I execute this to give myself permission to use the correct vfio group that my second video card is in:

    chown dave:dave /dev/vfio/31

Keys getting stuck, mouse jumping randomly 

Only seems to happen when the kbd and mouse are used at the same time. Easiest way to test was to move the mouse while typing in nodepad.  The mouse would jump around and sometimes (not often) a key would get stuck. 

Started happening after I did a full emerge -vup world upgrade (the kernel from 4.13.? to 5.9.1, it had been a while).  Even tried using my old kernel, and old qemu (I saved them before upgrading), still broken.

thanks to: https://www.reddit.com/r/VFIO/comments/824cdi/evdev_passthrough_problem_in_games_keys_get_stuck/

My startup script only had the -object input-linux lines for keyboard and mouse, so I guess it was defaulting to PS/2 devices?  Has been working fine for 3 years.  Installed virtio drivers and added these devices and it was all fine again.

-device virtio-keyboard

-device virtio-mouse

Sound needs to be modules

My monitors use the snd-hda sound driver, so does my computer.  That's 3 snd-hda devices.  When I compile the hda intel/realtek driver in the kernel it claims all three devices.  vfio requires all devices in an iommu group be unbound (so to pass the video card to qemu, the sound card must also be passed, not possible when the kernel has claimed the sound device on the monitor).  echoing something to the /unbind thing just waits forever and never returns.

The way around this is to make a modprobe file to claim the gpu and onboard sound for vfio before the kernel driver can get hold of it.

/etc/modprobe.d/vfio.conf:

options vfio-pci ids=10de:1b81,10de:10f0

softdep snd-hda-intel pre: vfio-pci

My Full win10 startup script with pci passthru:

qemu-system-x86_64 -name guest=Win107  \

-machine pc-i440fx-2.9,accel=kvm,usb=off,vmport=off \

-cpu Haswell-noTSX,kvm=off \

-m 16384 \

-overcommit mem-lock=off \

-smp sockets=1,cores=6,threads=1,maxcpus=6 \

-uuid 12345567-1234-1234-1234-123455667780 \

-no-user-config -nodefaults \

-no-hpet -no-shutdown  \

-global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1  \

-boot order=dc  \

-bios /usr/share/seabios/bios.bin \

-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7  \

-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5  \

-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1  \

-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2  \

-drive file=/mnt/images/win10.qcow2,format=qcow2,if=none,id=drive-ide0-0-0  \

-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2  \

-drive file=/dev/disk/by-id/wwn-0x50014ee206f54395,format=raw,if=none,id=drive-ide0-0-1  \

-device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1,bootindex=3  \

-drive file="/home/dave/images/virtio-win-0.1.189.iso",format=raw,if=none,id=drive-ide0-1-0,readonly=on  \

-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1  \

-chardev pty,id=charserial0 \

-device isa-serial,chardev=charserial0,id=serial0 \

-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \

-device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01 \

-netdev user,id=net0 \

-device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x7,multifunction=on,x-vga=on \

-device vfio-pci,host=02:00.1,id=hostdev1,bus=pci.0,addr=0x8 \

-device usb-host,vendorid=0x046d,productid=0xc215 \

-msg timestamp=on \

-vga none \

-audiodev pa,id=1 -device intel-hda -device hda-output,audiodev=1 \

-object input-linux,id=kbd0,evdev=/dev/input/by-id/usb-046d_G15_Gaming_Keyboard-event-kbd,grab_all=on,repeat=on \

-object input-linux,id=mouse0,evdev=/dev/input/by-id/usb-Logitech_USB-PS_2_Optical_Mouse-event-mouse \

-device virtio-keyboard \

-device virtio-mouse