vfio进行网卡透传

VFIO is a new method of doing PCI device assignment ("PCI passthrough"aka "<hostdev>") available in newish kernels (3.6?; it's in Fedora 18 atany rate) and via the "vfio-pci" device in qemu-1.4+. In contrast to thetraditional KVM PCI device assignment (available via the "pci-assign"device in qemu), VFIO works properly on systems using UEFI "SecureBoot"; it also offers other advantages, such as grouping of relateddevices that must all be assigned to the same guest (or not at all).Here's some useful reading on the subject.  http://lwn.net/Articles/474088/  http://lwn.net/Articles/509153/Short description (from Alex Williamson's KVM Forum Presentation)1) Assume this is the device you want to assign:01:10.0 Ethernet controller: Intel Corporation 82576Virtual Function (rev 01)2) Find the vfio group of this device:# readlink /sys/bus/pci/devices/0000:01:10.0/iommu_group../../../../kernel/iommu_groups/15==IOMMU Group = 153) Check the devices in the group:# ls /sys/bus/pci/devices/0000:01:10.0/iommu_group/devices/0000:01:10.0(so this group has only 1 device)4) Unbind from device driver# echo 0000:01:10.0 >/sys/bus/pci/devices/0000:01:10.0/driver/unbind5) Find vendor & device ID$ lspci -n -s 01:10.001:10.0 0200: 8086:10ca (rev 01)6) Bind to vfio-pci$ echo 8086 10ca /sys/bus/pci/drivers/vfio-pci/new_id(this will result in a new device node "/dev/vfio/15",  which is what qemu will use to setup the device for passthrough)7) chown the device node so it is accessible by qemu user:# chown qemu /dev/vfio/15; chgrp qemu /dev/vfio/15(note that /dev/vfio/vfio, which is installed as 0600 root:root, must also be made mode 0666, still owned by root - this is supposedly not dangerous)I'll look into this, the intention has always been that /dev/vfio/vfiois a safe interface that's only empowered when connected toa /dev/vfio/$GROUP, which implies some privileges.8) set the limit for locked memory equal to all of guest memory size + [some amount large enough to encompass all of io space]# ulimit -l 2621440   # ((2048 + 512) * 1024)9) pass to qemu using -device vfio-pci: sudo qemu qemu-system-x86_64 -m 2048 -hda rhel6vm \              -vga std -vnc :0 -net none \              -enable-kvm \              -device vfio-pci,host=01:10.0,id=net0(qemu will then use something like step (2) to figure out which device node it needs to use)Why the "ulimit -l"?--------------------Any qemu guest that is using the old pci-assign must have *all* guestmemory and IO space locked in memory. Normally the maximum amount oflocked memory allowed for a process is controlled by "ulimit -l", butin the case of pc-assign, the kvm kernel module has always justignored the -l limit and locked it all anyway.With vfio-pci, all guest memory and IO space must still be locked inmemory, but the vfio module *doesn't* ignore the process limits, solibvirt will need to set ulimit -l for any guest that wants to dovfio-based pci passthrough. Since (due to the possibility of hotplug)we don't know at the time the qemu process is started whether or notit might need to do a pci passthrough, we will need to use prlimit(2)to modify the limit of the already-running qemu.Proposed XML Changes--------------------To support vfio pci device assignment in libvirt, I'm thinking somethinglike this (note that the <driversubelement is already used for<interfaceand <diskto choose which backend to use for a particulardevice):   <hostdev managed='yes'>     <driver name='vfio'/>     ...   </hostdev>   <interface type='hostdev' managed='yes'>     <driver name='vfio'/>vfio is the overall userspace driver framework while vfio-pci is thespecific qemu driver we're using here.  Does it make more sense to callthis 'vfio-pci'?  It's possible that we could later have a device treeqemu driver which would need to be involved with -device vfio-dt (orsomething) and have different options.     ...   </hostdev>(this new use of <driverinside <interfacewouldn't conflict withthe existing <driver name='qemu|vhost'>, since neither of those couldever possibly be a valid choice for <interface type='hostdev'>. Theone possible problem would be if someone had an <interfacetype='network'which might possibly point to a hostdev or standardbridged network, and wanted to make sure that in the case of a bridgednetwork, that <driver name='qemu' was used. I suppose in this case,the driver name in the network definition would override any drivername in the interface?)Sepaking of <network>, here's how vfio would be specified in a hostdev <networkdefinition:   <network>     <name>vfio-net</name>     <forward mode='hostdev' managed='yes'>       <driver name='vfio'/>       <pf dev='eth3'/<!-- or a list of VFs -->     </forward>     ...   </network>Another possibility for the <networkxml would be to add a"driver='vfio'" to each individual <interfaceline, in case someonewanted some devices in a pool to be asigned using vfio and some usingthe old style, but that seems highly unlikely (and could createproblems in the future if we ever needed to add a 2nd attribute to the<driverelement).Actually, at one point I considered that vfio should be turned onglobally in libvirtd.conf (or qemu.conf), but that would makeswitchover a tedious process, as all existing guests using PCIpassthrough would need to be shutdown prior to the change. As long asthere are no technical problems with allowing both types on the samehost, it's more flexible to choose on a device-by-device basis.>Now some questions:1) Is there any reason that we shouldn't/can't allow both pci-assignand vfio-pci at the same time on the same host (and even guest).vfio-pci and pci-assign can be mixed, but don't intermix devices withina group.  Sometimes this will work (if the grouping is isolationreasons), but sometimes it won't (when the grouping is for visibility).Best to just avoid that scenario.2) Does it make any sense to support a "managed='no'" mode for vfio,which skipped steps 2-6 above? (this would be parallel to the existingpci-assign managed='no'(where no unbinding/binding of the device tothe host's pci-stub driver is done, but the device name is simplypassed to qemu assuming that all that work was already done)) Orshould <driver name='vfio'/automatically mean that allunbinding/binding be done for each device.I don't think it hurts to have it, but I can't think of a use case.Even with pci-assign, I can only think of cases where customers haveused it to try to work around things they shouldn't be doing with it.3) Is it at all bothersome that qemu must be the one opening thedevice node, and that there is apparently no way to have libvirt openit and send the fd to qemu?I have the same question.  The architecture of vfio is that the userwill open /dev/vfio/vfio (vfiofd) and add a group to it (groupfd).Multiple groupfds can be added to a single vfiofd, allowing groups toshare IOMMU domains.  However, it's not guaranteed that the IOMMU driverwill allow this (the domains may be incompatible).  Qemu will thereforeattempt to add any new group to an existing vfiofd before re-opening anew one.  There's also the problem that a group has multiple devices, soif device A from group X gets added with vfiofd and groupXfd and libvirtthen passes a new vfiofd' and groupXfd' for attaching device B, alsofrom group X... what's qemu to do?So in order to pass file descriptors libvirt has to either know exactlyhow things are working or just always pass a vfiofd and groupfd, whichqemu will discard if it doesn't need.  The latter implies that fds couldlive on and be required past the point where the device that added themhas been removed (in the example above, add A and qemu uses vfiofd andgroupXfd, hot add B and qemu discards vfiofd' and groupXfd', remove Aand qemu continues to use vfiofd and groupXfd for B). 

*********************************************************************

-device pci-assign 已经不使用了，会报错invalid argument

最新的内核里，建议废除KVM_ASSIGN机制，只支持vfio，如果还是使用老的 KVM ASSIGN的话，那么需要手动修改.config文件 “KVM_DEVICE_ASSIGNMENT=y”，才能使用kvm assgin。 注意，要vim手动修改，make menuconfig里面已经没有了

看了一下代码，assigned-dev.c 是kvm_assgin的实现，只有选择CONFIG_KVM_DEVICE_ASSIGNMENT才会对其进行编译
arch/x86/kvm/Makefile：
kvm-$(CONFIG_KVM_DEVICE_ASSIGNMENT) += assigned-dev.o iommu.o

这里是一篇关于如何使用kvm-pci-assign机制的文章
http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

参考链接
http://www.spinics.net/lists/kvm/msg120779.html
http://nanxiao.me/en/why-does-qemu-complain-no-iommu-found/

同样使用kvm-assgin的话，使用最新的QEMU同样存在问题
“qemu-system-x86_64: pci_get_msi_message: unknown interrupt type”
这同样是VFIO的问题
如果想使用kvm-pci-assgin，那么就使用2.6.0以前的QEMU吧
参考链接
http://qemu.11.n7.nabble.com/PATCH-v9-00-25-IOMMU-Enable-interrupt-remapping-for-Intel-IOMMU-td412217.html


另外有个地方可以下载到kvm很多有用的脚本
https://github.com/smilejay/kvm-book.git