KVM, PCI passthrough, Looking Glass and all-all
After a successful transition to Linux software developers, it gained a moment when there was little work, and also changed the main OS. Fears were caused by non-platform software to support existing projects. Part of the software earned through wine. However, it turned out that a certain software refuses to work under wine. It was decided to run the software on the QEMU + KVM virtual machine. The software began to run, but working in it was quite inconvenient. Software virtual video cards do not differ in performance, and support for 3D graphics is very modest. I had to uncover a tambourine and look for a way out.
It did not take long to find a way out, but the idea of ​​beating a tambourine with a searchlight turned out to be very strange. On the subject of forwarding video cards to a virtual machine, the Internet is replete with instructions from different times and for different hardware. What is a huge article on the Arch Linux site [0] . I will give an abbreviated version of the instruction for forwarding a video card.
For example, here [1] .
Do not forget
')
We are looking for the right devices and see which drivers use them.
Add the VFIO modules so that they load when loading.
We configure the VFIO module so that it intercepts devices, preventing the main drivers from loading. If necessary, add to the blacklist, the main driver.
IOMMU turned on.
Composite devices fell into one group.
KVM and VFIO drivers are loaded.
Video card for guest OS captured VFIO.
We start the virtual machine without forwarding the video card to install the guest OS. Since the sight is on Looking Glass, then the guest should choose Windows 10. Windows 8 / 8.1 according to the latest data is also supported.
To begin, boot with two video cards. We look that the forbidden card appeared in the system, put drivers on it and make sure that they have earned.
After that, as the forwarded video card has earned, and in the device manager it says “The device is working properly”, we start the virtual machine only with the forwarded video card.
We connect the monitor to it and admire the desktop image of the guest OS.
And here begins the most interesting. For someone, everything is fine, the picture went and then everything is simple. My experience stumbled twice in the absence of an image. The first time was the probros of the integrated Intel 6700T HD 530 processor, the video outputs were empty and the failure was attributed to the fact that the inlays didn’t fit well. The second time, the external Nvidia GF210 was being forwarded, which was already specially purchased for experiments. The result was even more interesting. In non EFI mode, the video card was successfully forwarded and even showed a picture, but shutting down the guest OS misfired .
Subsequent probros could just hang the host. A couple of hours of light googling leads to the fact that the problem of video card hangup is quite common. This results in both an incorrect shutdown of a virtual machine, and with some chance even a correct shutdown. As an exit, it is recommended to forward in EFI mode. But VBIOS Nvidia GF210 does not support EFI ...
Do not sew. QEMU supports substitution of VBIOS when forwarding a video card. But VBIOS still have to learn how to support EFI mode. Usually, it is recommended to check before starting a video card, for example, here [2] . But we have to deal with what we have, and we did not want to look for a fresh video card with EFI support. So you need to patch VBIOS. All operations performed with VBIOS are done at your own risk. I used the software package and instructions for it from here [3] . After reading the VBIOS, we get the file
We start the virtual machine and look.
The outputs of the video card shone with darkness. Once again, morality was tested by failure. The first thing that comes to mind is that the guest OS crashes at the start. Logs, I need her logs. To do this, run
Connect and see the working machine, only the monitor has a strange Generic Non-PnP Monitor (Universal Monitor is not PnP). There is a picture, so you can try to run Looking Glass.
Although this technology uses OpenGL, no space is needed after gl. But read the instructions [5] on the project site is a must. For the guest OS download the screen capture application looking-glass-host.exe [6] , download and install Microsoft Visual C ++ 2015 Redistributable [7] , download the driver for the IVSHMEM device [8] . For the host set dependencies, download and build the client application.
We start the virtual machine with an IVSHMEM device. 32Mb memory size is selected for 1920x1080 resolution.
We connect via VNC, install the driver on the IVSHMEM device, it is possible that a standard driver will be supplied to it, located in “System devices”. We start looking-glass-host.exe . On the host, run
This is where the NVidia GF210 system worked, and then the Intel HD530 was launched on the same route. With the screen resolution there was a small problem, for changing to a rare resolution, for example 2048x1152, I had to use the Custom Resolution Utility [9] .
Another caveat, when adding a looking-glass-host.exe application to autoload, you need to configure the user to automatically log in, for security reasons the guest OS does not allow to capture the login screen.
If you do not set the task of obtaining images on the physical video output, then this result will be enough to get a working virtual machine with a physical video card and responsive management. Control is exercised from the host in a separate window or in full screen. However, there are nuances.
Performance . Overhead resources for virtualization and not the most agile guest OS will not allow you to comfortably work on weak and medium-weak hardware. You need a powerful processor of at least 6-8 cores, a good graphics card for the guest OS, 16GB + RAM, at least 8GB for each OS. And dancing with a tambourine to squeeze the maximum out of iron.
Patience . If it does not work right away, then you will have to spend your time and energy decently. Search, read, try. Search again, read and try again. Leave a few more links that I came across, maybe there will be some more useful information. [10] [11] [12]
Links
Caution, links open in this window.
0. https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
1. https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware
2. https://www.techpowerup.com/vgabios/
3. https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html
4. https://www.tightvnc.com/download.php
5. https://looking-glass.hostfission.com/quickstart
6. https://github.com/gnif/LookingGlass/releases
7. https://www.microsoft.com/en-us/download/details.aspx?id=48145
8. https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/upstream-virtio/
9. https://www.monitortests.com/forum/Thread-Custom-Resolution-Utility-CRU
10. https://heiko-sieger.info/running-windows-10-on-linux-using-kvm-with-vga-passthrough
11. https://ycnrg.org/vga-passthrough-with-ovmf-vfio/
12. https://www.reddit.com/r/VFIO/comments/8h352p/guide_running_windows_via_qemukvm_and_intel_gvtg/
Select a separate video card for the guest system!
It did not take long to find a way out, but the idea of ​​beating a tambourine with a searchlight turned out to be very strange. On the subject of forwarding video cards to a virtual machine, the Internet is replete with instructions from different times and for different hardware. What is a huge article on the Arch Linux site [0] . I will give an abbreviated version of the instruction for forwarding a video card.
0. Check that the hardware supports IOMMU
For example, here [1] .
1. Turn on IOMMU support in the kernel.
cat / etc / default / grub
GRUB_CMDLINE_LINUX_DEFAULT = "quiet splash amd_iommu = on"
or
GRUB_CMDLINE_LINUX_DEFAULT = "quiet splash intel_iommu = on"
Do not forget
sudo update-grub .')
2. We select the video card from the driver
We are looking for the right devices and see which drivers use them.
lspci -nnk
04: 00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [GeForce 210] [ 10de: 0a65 ] (rev a2)
Kernel driver in use: nouveau
Kernel modules: nvidiafb, nouveau
04: 00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [ 10de: 0be3 ] (rev a1)
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
Add the VFIO modules so that they load when loading.
cat / etc / modules | grep vfio
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
We configure the VFIO module so that it intercepts devices, preventing the main drivers from loading. If necessary, add to the blacklist, the main driver.
cat /etc/modprobe.d/vfio.conf
options vfio-pci ids = 10de: 0a65,10de: 0be3
blacklist nouveau
3. Reboot and check that everything worked out.
IOMMU turned on.
dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
DMAR: Intel® Virtualization Technology for Directed I / O
or
AMD-Vi: Found IOMMU at 0000: 00: 00.2 cap 0x40
AMD-Vi: Interrupt remapping enabled
AMD-Vi: Lazy IO / TLB flushing enabled
Composite devices fell into one group.
for a in / sys / kernel / iommu_groups / *; do find $ a -type l; done | sort - version-sort
/sys/kernel/iommu_groups/15/devices/0000:01:00.0
/sys/kernel/iommu_groups/15/devices/0000:01:00.1
/sys/kernel/iommu_groups/16/devices/0000:02:00.0
/sys/kernel/iommu_groups/17/devices/0000:03:00.0
/sys/kernel/iommu_groups/18/devices/0000:04:00.0
/sys/kernel/iommu_groups/18/devices/0000:04:00.1
KVM and VFIO drivers are loaded.
lsmod | grep -e kvm -e vfio
kvm_amd 94208 0 ccp 90112 1 kvm_amd kvm 622592 1 kvm_amd vfio_pci 45056 0 vfio_virqfd 16384 1 vfio_pci irqbypass 16384 2 vfio_pci, kvm vfio_iommu_type1 24576 0 vfio 28672 2 vfio_iommu_type1, vfio_pci
Video card for guest OS captured VFIO.
lspci -nnk
04: 00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [GeForce 210] [ 10de: 0a65 ] (rev a2)
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau
04: 00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [ 10de: 0be3 ] (rev a1)
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
4. Configure QEMU and run the guest OS
Install if not already installed
sudo apt install qemu-kvm qemu-utils seabios ovmf virt-viewer
Create a disk where the guest OS will be installed
qemu-img create -f raw -o preallocation = full guest.img 50G
or
fallocate -l 50G guest.img
We start the virtual machine without forwarding the video card to install the guest OS. Since the sight is on Looking Glass, then the guest should choose Windows 10. Windows 8 / 8.1 according to the latest data is also supported.
vga_qxl.sh
#! / bin / bash
remote-viewer spice: //127.0.0.1: 5900 &
sudo qemu-system-x86_64 \
-machine q35, accel = kvm \
-enable-kvm \
-cpu host, kvm = off, check \
-smp cpus = 2, sockets = 1, cores = 2, threads = 1 \
-m 6G \
-rtc base = localtime, clock = host \
-device piix3-usb-uhci \
-device usb-tablet \
-drive if = pflash, format = raw, readonly, file = / usr / share / OVMF / OVMF_CODE.fd \
-drive file = 'w10.iso', if = ide, format = raw, index = 2, media = cdrom, cache = none \
-drive file = 'virtio-win-0.1.141_st.iso', if = ide, format = raw, index = 3, media = cdrom, cache = none \
-drive file = 'guest.img', if = ide, format = raw, index = 4, media = disk, cache = writeback \
-vga qxl \
-spice port = 5900, addr = 127.0.0.1, disable-ticketing \
-monitor stdio \
-netdev user, id = n1, ipv6 = off, smb = "/ media / user / data" \
-device e1000, netdev = n1, mac = 67: 77: 78: 88: 89: 99 \
"$ @"
5. We forward the video card to the guest OS
To begin, boot with two video cards. We look that the forbidden card appeared in the system, put drivers on it and make sure that they have earned.
vga_qxl_pass.sh
#! / bin / bash
remote-viewer spice: //127.0.0.1: 5900 &
sudo qemu-system-x86_64 \
-machine q35, accel = kvm \
-enable-kvm \
-cpu host, kvm = off, check \
-smp cpus = 2, sockets = 1, cores = 2, threads = 1 \
-m 6G \
-rtc base = localtime, clock = host \
-device piix3-usb-uhci \
-device usb-tablet \
-drive if = pflash, format = raw, readonly, file = / usr / share / OVMF / OVMF_CODE.fd \
-drive file = 'virtio-win-0.1.141_st.iso', if = ide, format = raw, index = 3, media = cdrom, cache = none \
-drive file = 'guest.img', if = ide, format = raw, index = 4, media = disk, cache = writeback \
-vga qxl \
-spice port = 5900, addr = 127.0.0.1, disable-ticketing \
-device ioh3420, bus = pcie.0, addr = 1c.0, multifunction = on, port = 1, chassis = 1, id = root \
-device vfio-pci, host = 04: 00.0, bus = root, addr = 00.0, multifunction = on \
-device vfio-pci, host = 04: 00.1, bus = root, addr = 00.1 \
-monitor stdio \
-netdev user, id = n1, ipv6 = off, smb = "/ media / user / data" \
-device e1000, netdev = n1, mac = 67: 77: 78: 88: 89: 99 \
"$ @"
After that, as the forwarded video card has earned, and in the device manager it says “The device is working properly”, we start the virtual machine only with the forwarded video card.
vga_pass.sh
#! / bin / bash
sudo qemu-system-x86_64 \
-machine q35, accel = kvm \
-enable-kvm \
-cpu host, kvm = off, check \
-smp cpus = 2, sockets = 1, cores = 2, threads = 1 \
-m 6G \
-rtc base = localtime, clock = host \
-device piix3-usb-uhci \
-device usb-tablet \
-drive if = pflash, format = raw, readonly, file = / usr / share / OVMF / OVMF_CODE.fd \
-drive file = 'virtio-win-0.1.141_st.iso', if = ide, format = raw, index = 3, media = cdrom, cache = none \
-drive file = 'guest.img', if = ide, format = raw, index = 4, media = disk, cache = writeback \
-vga none \
-device ioh3420, bus = pcie.0, addr = 1c.0, multifunction = on, port = 1, chassis = 1, id = root \
-device vfio-pci, host = 04: 00.0, bus = root, addr = 00.0, multifunction = on \
-device vfio-pci, host = 04: 00.1, bus = root, addr = 00.1 \
-monitor stdio \
-netdev user, id = n1, ipv6 = off, smb = "/ media / user / data" \
-device e1000, netdev = n1, mac = 67: 77: 78: 88: 89: 99 \
"$ @"
We connect the monitor to it and admire the desktop image of the guest OS.
The place where simple solutions end
And here begins the most interesting. For someone, everything is fine, the picture went and then everything is simple. My experience stumbled twice in the absence of an image. The first time was the probros of the integrated Intel 6700T HD 530 processor, the video outputs were empty and the failure was attributed to the fact that the inlays didn’t fit well. The second time, the external Nvidia GF210 was being forwarded, which was already specially purchased for experiments. The result was even more interesting. In non EFI mode, the video card was successfully forwarded and even showed a picture, but shutting down the guest OS misfired .
Subsequent probros could just hang the host. A couple of hours of light googling leads to the fact that the problem of video card hangup is quite common. This results in both an incorrect shutdown of a virtual machine, and with some chance even a correct shutdown. As an exit, it is recommended to forward in EFI mode. But VBIOS Nvidia GF210 does not support EFI ...
Sew or not sew, that is the question
Do not sew. QEMU supports substitution of VBIOS when forwarding a video card. But VBIOS still have to learn how to support EFI mode. Usually, it is recommended to check before starting a video card, for example, here [2] . But we have to deal with what we have, and we did not want to look for a fresh video card with EFI support. So you need to patch VBIOS. All operations performed with VBIOS are done at your own risk. I used the software package and instructions for it from here [3] . After reading the VBIOS, we get the file
gt210.rom , patch and at the output we have gt210_uefi.rom . This is what you need to slip the video card when booting the virtual machine.vga_pass_rom.sh
#! / bin / bash
sudo qemu-system-x86_64 \
-machine q35, accel = kvm \
-enable-kvm \
-cpu host, kvm = off, check \
-smp cpus = 2, sockets = 1, cores = 2, threads = 1 \
-m 6G \
-rtc base = localtime, clock = host \
-device piix3-usb-uhci \
-device usb-tablet \
-drive if = pflash, format = raw, readonly, file = / usr / share / OVMF / OVMF_CODE.fd \
-drive file = 'virtio-win-0.1.141_st.iso', if = ide, format = raw, index = 3, media = cdrom, cache = none \
-drive file = 'guest.img', if = ide, format = raw, index = 4, media = disk, cache = writeback \
-vga none \
-device ioh3420, bus = pcie.0, addr = 1c.0, multifunction = on, port = 1, chassis = 1, id = root \
-device vfio-pci, host = 04: 00.0, bus = root, addr = 00.0, multifunction = on, romfile = gt210_uefi.rom \
-device vfio-pci, host = 04: 00.1, bus = root, addr = 00.1 \
-monitor stdio \
-netdev user, id = n1, ipv6 = off, smb = "/ media / user / data" \
-device e1000, netdev = n1, mac = 67: 77: 78: 88: 89: 99 \
"$ @"
We start the virtual machine and look.
Darkness
The outputs of the video card shone with darkness. Once again, morality was tested by failure. The first thing that comes to mind is that the guest OS crashes at the start. Logs, I need her logs. To do this, run
vga_qxl.sh . We look at the previous launch. And there everything is fine, except that the food dramatically pulled. It turns out that it works, although it does not work. The first idea was to connect via RDP and see what was happening there, but still it is better to use VNC for this, for example tightvnc [4] . Install VNC, configure port 5600 and push this port for access from the host.vga_vnc_pass_rom.sh
#! / bin / bash
sudo qemu-system-x86_64 \
-machine q35, accel = kvm \
-enable-kvm \
-cpu host, kvm = off, check \
-smp cpus = 2, sockets = 1, cores = 2, threads = 1 \
-m 6G \
-rtc base = localtime, clock = host \
-device piix3-usb-uhci \
-device usb-tablet \
-drive if = pflash, format = raw, readonly, file = / usr / share / OVMF / OVMF_CODE.fd \
-drive file = 'virtio-win-0.1.141_st.iso', if = ide, format = raw, index = 3, media = cdrom, cache = none \
-drive file = 'guest.img', if = ide, format = raw, index = 4, media = disk, cache = writeback \
-vga none \
-device ioh3420, bus = pcie.0, addr = 1c.0, multifunction = on, port = 1, chassis = 1, id = root \
-device vfio-pci, host = 04: 00.0, bus = root, addr = 00.0, multifunction = on, romfile = gt210_uefi.rom \
-device vfio-pci, host = 04: 00.1, bus = root, addr = 00.1 \
-monitor stdio \
-netdev user, id = n1, hostfwd = tcp: 127.0.0.1: 5600-: 5600, ipv6 = off, smb = "/ media / user / data" \
-device e1000, netdev = n1, mac = 67: 77: 78: 88: 89: 99 \
"$ @"
Connect and see the working machine, only the monitor has a strange Generic Non-PnP Monitor (Universal Monitor is not PnP). There is a picture, so you can try to run Looking Glass.
Looking glass
Although this technology uses OpenGL, no space is needed after gl. But read the instructions [5] on the project site is a must. For the guest OS download the screen capture application looking-glass-host.exe [6] , download and install Microsoft Visual C ++ 2015 Redistributable [7] , download the driver for the IVSHMEM device [8] . For the host set dependencies, download and build the client application.
build_looking_glass_a12.sh
#! / bin / bash
sudo apt-get install cmake libsdl2-dev libsdl2-ttf-dev nettle-dev libspice-protocol-dev libfontconfig1-dev libx11-dev fonts-freefont-ttf libconfig-dev
wget github.com/gnif/LookingGlass/archive/a12.tar.gz
tar -xf a12.tar.gz
cd lookingglass-a12
mkdir client / build
cd client / build
cmake ../
make
We start the virtual machine with an IVSHMEM device. 32Mb memory size is selected for 1920x1080 resolution.
vga_vnc_lg_pass_rom.sh
#! / bin / bash
if [! -f / dev / shm / looking-glass]; then
touch / dev / shm / looking-glass
chown `whoami`: kvm / dev / shm / looking-glass
chmod 660 / dev / shm / looking-glass
fi
sudo qemu-system-x86_64 \
-machine q35, accel = kvm \
-enable-kvm \
-cpu host, kvm = off, check \
-smp cpus = 2, sockets = 1, cores = 2, threads = 1 \
-m 6G \
-rtc base = localtime, clock = host \
-device piix3-usb-uhci \
-device usb-tablet \
-drive if = pflash, format = raw, readonly, file = / usr / share / OVMF / OVMF_CODE.fd \
-drive file = 'virtio-win-0.1.141_st.iso', if = ide, format = raw, index = 3, media = cdrom, cache = none \
-drive file = 'guest.img', if = ide, format = raw, index = 4, media = disk, cache = writeback \
-vga none \
-device ioh3420, bus = pcie.0, addr = 1c.0, multifunction = on, port = 1, chassis = 1, id = root \
-device vfio-pci, host = 04: 00.0, bus = root, addr = 00.0, multifunction = on, romfile = gt210_uefi.rom \
-device vfio-pci, host = 04: 00.1, bus = root, addr = 00.1 \
-device ivshmem-plain, memdev = ivshmem, bus = pcie.0 \
-object memory-backend-file, id = ivshmem, share = on, mem-path = / dev / shm / looking-glass, size = 32M \
-monitor stdio \
-netdev user, id = n1, hostfwd = tcp: 127.0.0.1: 5600-: 5600, ipv6 = off, smb = "/ media / user / data" \
-device e1000, netdev = n1, mac = 67: 77: 78: 88: 89: 99 \
"$ @"
We connect via VNC, install the driver on the IVSHMEM device, it is possible that a standard driver will be supplied to it, located in “System devices”. We start looking-glass-host.exe . On the host, run
./LookingGlass-a12/client/build/looking-glass-client .This is where the NVidia GF210 system worked, and then the Intel HD530 was launched on the same route. With the screen resolution there was a small problem, for changing to a rare resolution, for example 2048x1152, I had to use the Custom Resolution Utility [9] .
Another caveat, when adding a looking-glass-host.exe application to autoload, you need to configure the user to automatically log in, for security reasons the guest OS does not allow to capture the login screen.
Afterword
If you do not set the task of obtaining images on the physical video output, then this result will be enough to get a working virtual machine with a physical video card and responsive management. Control is exercised from the host in a separate window or in full screen. However, there are nuances.
Performance . Overhead resources for virtualization and not the most agile guest OS will not allow you to comfortably work on weak and medium-weak hardware. You need a powerful processor of at least 6-8 cores, a good graphics card for the guest OS, 16GB + RAM, at least 8GB for each OS. And dancing with a tambourine to squeeze the maximum out of iron.
Patience . If it does not work right away, then you will have to spend your time and energy decently. Search, read, try. Search again, read and try again. Leave a few more links that I came across, maybe there will be some more useful information. [10] [11] [12]
Links
Caution, links open in this window.
0. https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
1. https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardware
2. https://www.techpowerup.com/vgabios/
3. https://www.win-raid.com/t892f16-AMD-and-Nvidia-GOP-update-No-requests-DIY.html
4. https://www.tightvnc.com/download.php
5. https://looking-glass.hostfission.com/quickstart
6. https://github.com/gnif/LookingGlass/releases
7. https://www.microsoft.com/en-us/download/details.aspx?id=48145
8. https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/upstream-virtio/
9. https://www.monitortests.com/forum/Thread-Custom-Resolution-Utility-CRU
10. https://heiko-sieger.info/running-windows-10-on-linux-using-kvm-with-vga-passthrough
11. https://ycnrg.org/vga-passthrough-with-ovmf-vfio/
12. https://www.reddit.com/r/VFIO/comments/8h352p/guide_running_windows_via_qemukvm_and_intel_gvtg/
Source: https://habr.com/ru/post/433878/
All Articles