Continuing blog post series around Xen and IOMMU enabling in coreboot we are reaching a point in which some features seem to work correctly on top of recent patch series in firmware.

What we can do at this point is PCI passthrough to guest VMs. Previously trying that on Xen caused problems:

  • random hangs
  • firmware cause Linux kernel booting issues (hang during boot)
  • IOMMU disabled – unable to use PCI passthrough

Now we can see something like that in dom0:

Of course, after above operation, we can’t access enp2s0 in dom0. Having the ability to set pass through we can think about creating pfSense HVM and having isolation between various roles on our PC Engines apu2 router.

What are the pros of that solution:

  • price – this is DIY solution where you just pay price of apu2 and spent some time with setup, of course, you can also pay for that to companies like 3mdeb, what should be still cheaper than other commercial solutions – this makes it attractive to SOHO
  • scalability – you can decide how much resources of your router you want to give to the firewall; the remaining pool can be used for other purposes this saves you a couple of cents on the energy bill
  • security – even if attacker get access to pfSense (very unlikely), escaping VM and gaining full control and persistence on hardware is not possible without serious Xen bug, on the other hand, bugs in on the other VMs (e.g. network storage, web application, 3rd party software) cannot be leveraged to gain control over the router
  • virtual machine – VMs by itself have a bunch of advantages, somewhere mentioned above, but other are easier migration, lower cost to introduce in existing network

Required components

  • PC Engines apu2c4
  • pxe-server – or other means of booting Debian based Dom0 with Xen 4.8 and Linux 4.14.59 (or any other modern kernel which has correct support enabled as in this kernel config)
  • 2 connected Ethernet ports
  • some storage (min 10GB)

Prepare Xen

I’m using apic=verbose,debug iommu=verbose,debug for better visibility of Xen configuration. More to that we need some preparation in Dom0:

Storage

PCI passthrough

After above commands 02:00.0 should be listed in xl pci-assignable-list output:

xl allows assigning devices even if IOMMU is not present, but it will issue an error during VM creation.

Xen pfsense.cfg

First let’s create

Then you can create VM:

Install pfSense

After running VM you can attach to console:

You should see pfSense installer boot log:

Unfortunately, pfSense had problem getting DHCP offer and didn’t configure IP address – we tried to figure out what is wrong but my BSD-fu is low. We also checked static IP configuration, but there is no result either. This leads us to ask on the forum.

Xen debian.cfg

Of course, you have to replace apu2_ip_addr with correct IP. After xl create debian.cfg you can run VNC (tightvnc worked for me) and proceed with the installation.

PCI passthrough in Debian

Below screenshot show device 02:00.0, which is apu2 middle NIC, passed-through to VM.

Debian lspci

PCI passthrough on Debian worked without any issue DHCP offer was received correctly and I could proceed with performance checks.

Speedtest

Simplest possible test is comparison of throughput between eth0 and eth1. The first is connected directly to our company switch and the second connects pfSense HVM using PCI passthrough.

I used speedtest-cli v2.0.2.

Results for apu2 Dom0:

Results for Debian HVM with NIC PCI passthrough:

Debian HVM speedtest-cli

iperf

Below results are for very simple LAN connection apu3 -> switch -> apu2:

Results for Debian HVM with NIC PCI passthrough:

Debian HVM iperf

As you can see there is no difference between results, based on that we can conclude that PCI passthrough works and there is no overhead when using IOMMU.

Below log show results from Debian PV and prove how virtualized drivers lead to performance overhead.

Possible problems

xen-pciback not loaded

Solution:

PCI device not assignable

Assign PCI device using xl pci-assignable-add.

No IOMMU

This error means you don’t have IOMMU correctly enabled. For AMD platforms xl dmesg contain:

Lack of block backend

xen-blkback should be loaded or compiled in otherwise blow error pop-up.

Crash after couple tries

After a couple tries of creating pfSense VM I faced below error:

Solution: recompile kernel with BLK_DEV_LOOP

Read-only not supported

Solution: change pfsense.cfg by adding rw to img file.

References

Summary

I hope this post was useful for you. Please feel free to share your opinion and if you think there is value, then share with friends.

We plan to present above results during OSFC 2018 feel free to catch us there and ask questions.

We believe there are still many devices with VT-d or AMD-Vi advertised in specs, but not enabled because of buggy or not-fully-featured firmware. We are always open to support vendors who want to boot hardware by extending and improving their firmware. If you are user or vendor struggling with hardware which cannot be fully utilized because of firmware, feel free to contact us contact@3mdeb.com.