This guide covers how to install the Substance applications on Ubuntu 16.04 or similar distros (such as Debian or Mint) manually, without using alien (check out this forum post on how to do this via alien). We basically just need to extract the RPM contents, copy it to /opt, then run the postinstall scripts to setup icons, links, and file associations.

At the end, we'll also look at setting up the Python API. If you have a Substance subscription plan, you can download the Automation Toolkit from the Licenses page after logging into Allegorithmic's website.

Installation

Install Prerequisites

sudo apt-get update  
sudo apt-get install rpm rpm2cpio  

Install Substance Packages

First, create a new directory called sub_extract inside of your home directory:

mkdir ~/sub_extract  
cd ~/sub_extract  

Then download (or copy) your Substance RPMs to this new directory. You should see something like this when listing the directory (version and actual packages will obviously be different, depending on what you're trying to install):

jacob@jotunn:~/sub_extract$ ls  
Substance_Automation_Toolkit-2017.2.4-132-linux-x64-standard-indie.tar.gz  
Substance_Painter-2018.1.0-2128-linux-x64-standard-full.rpm  
Substance_Designer-2018.1.0-1039-linux-x64-standard-full.rpm  
Substance_Player-2018.1.0-1039-linux-x64-standard-full.rpm  

Next, extract the RPM package contents for each RPM:

for thisrpm in *.rpm; do rpm2cpio $thisrpm | cpio -idmv ; done  

If you downloaded the Substance Automation Toolkit, also extract that into the same directory:

tar -xvf Substance_Automation_Toolkit-*.tar.gz -C opt/Allegorithmic  

Now — copy the files into their final installation location. The applications expect to be installed in /opt/Allegorithmic, so that's where we'll put them:

sudo cp -Rvp ~/sub_extract/opt/Allegorithmic /opt/  

Last, we need to extract the installation scripts from the RPMs. This one-liner will dump the scripts for each RPM, then extract the postinstall script:

for thisrpm in *.rpm; do outscript=$(echo "$thisrpm" | sed -e 's/^\([a-zA-Z0-9_]*\)-.*/\1/').sh ; rpm -qp --scripts $thisrpm | sed -n '/postinstall/,/preuninstall/p' | grep -v ':$' > $outscript ; chmod +x $outscript ; done  

You should now have matching scripts in your directory that corresponds to each RPM:

jacob@jotunn:~/sub_extract$ ls -lAh *.sh  
-rwxrwxr-x 1 jacob jacob 1022 Mar 31 01:02 Substance_Designer.sh
-rwxrwxr-x 1 jacob jacob 1016 Mar 31 01:02 Substance_Painter.sh
-rwxrwxr-x 1 jacob jacob 1016 Mar 31 01:02 Substance_Player.sh

After you have reviewed each script (they should just fix the permissions, then set up icons and file associations), run them:

sudo bash Substance_Designer.sh 1  
sudo bash Substance_Painter.sh 1  
sudo bash Substance_Player.sh 1  

(Note: The 1 parameter is required as it triggers the postinstall actions in the RPM script)

If no errors were returned, then they were successful. The application launchers should now show up in your Applications Menu! You can also launch the main applications via substancedesigner, substancepainter, or substanceplayer from a terminal window or run dialog.

Tweaks/Fixes

Panning Not Working

If you're using the Compiz compositor for your desktop, more than likely the Alt+Middle Mouse (Middle Mouse = Button 2) combo will be assigned to resizing windows. Since this is also the default configuration for panning the camera in Substance applications, this poses a problem. To fix it, open up the CompizConfig Settings Manager:

sudo ccsm  

Then find the "Resize Window" plugin and click on it. On the first tab, click the key binding for "Initialize Window Resize", then either change it to something else, or disable it. This key is not required to be bound for window resizing to work. Once set, click "Back", then close CCSM to save the settings.

Fix missing UI asset

When running Painter or Designer, you may see an error message such as the following:

[Script] file:///home/jacob/Documents/Allegorithmic/Substance Painter/plugins/substance-source/MainHeaderBar.qml:161:7: QML Image: Cannot open: file:///home/jacob/Documents/Allegorithmic/Substance Painter/plugins/substance-source/exit.svg

In this case, the file does exist, but it's named Exit.svg. Since most filesystems on Linux/UNIX are case-sensitive, but NTFS and HFS+ are not (HFS+ can be, but is usually configured to be case-insensitive), the application is unable to locate the file.

To solve this, simply add a symlink from exit.svg to Exit.svg:

ln -s ~/Documents/Allegorithmic/Substance\ Painter/plugins/substance-source/{E,e}xit.svg  

I imagine this will be fixed by Allegorithmic in a future release. :D

Screenshots

Below are a couple of screenshots of Substance Painter and Substance Designer running on Ubuntu 16.04— working great! My machine has an nVidia GTX 960, running the proprietary NVIDIA 381.22 drivers. These applications can use a ton of VRAM (dependent upon your output texture size), so the more graphics memory you have, the better!

Python API Setup

The toolkit should already be installed in /opt/Allegorithmic/Substance_Automation_Toolkit if you followed the previous installation procedure.

The pysbs module (the Python module that houses all of the magic) is contained within a zip archive that can be installed via pip:

sudo pip install /opt/Allegorithmic/Substance_Automation_Toolkit/Python\ API/Pysbs*.zip  

If you have multiple Python versions, you can also install the module for those, too. For example, to also install for Python 3:

sudo pip3 install /opt/Allegorithmic/Substance_Automation_Toolkit/Python\ API/Pysbs*.zip  

If everything went smoothly, you should be able to run one of the included sample scripts, located in /opt/Allegorithmic/Substance_Automation_Toolkit/samples. Let's test out the variations.py script, which takes an example stone texture, creates different variations, then bakes out all of the textures as PNG images.

cd /opt/Allegorithmic/Substance_Automation_Toolkit/samples  
python variations.py  

After some time, it should finish, and we can check the results (xdg-open should open the image in your default image viewer, like eog)

cd /opt/Allegorithmic/Substance_Automation_Toolkit/samples/variations/_output  
ls  
xdg-open basecolor_opus_0.png  

That's it! Now you can start writing scripts that generate, manipulate, or bake out substances! Try using ipython to interactively test and explore the API.

• Substance Automation Toolkit documentation

I recently picked up a Lattice MachXO3 starter kit from Mouser-- it's a pretty cheap and convenient board for experimenting with an FPGA, without having a bazillion peripherals attached to it (and it's under $30).

Lattice's design & synthesis software, Lattice Diamond, is available for both Windows and Linux, which is great. However, they only provide an RPM package, with official support for RHEL. This is a pain, but isn't a big issue. Other guides I saw online tried to use alien to repackage the RPM as a deb package, but that seems like a pain, so we can just extract the files and run the post install script manually.

Prerequisites

We will need some RPM tools to work with the provided package, as well as libusb1 for Python 2.7. Ensure those are installed as follows:

sudo apt-get install rpm rpm2cpio  
sudo pip install libusb1  

Step 1: Acquire the RPM

First, you'll need to sign up for an account on Lattice's website, then you'll be able to download the software here.

Create a directory called diamond, then download the RPM to this directory.

mkdir diamond  

Step 2: Extract the RPM, Install Files

In the diamond directory, run the following command to extract the file contents:

rpm2cpio *.rpm | cpio -idmv  

Next, we will need the post-install scriptlet from the RPM.

rpm -qp --scripts *.rpm  

This command will print all of the scriptlets. Highlight and copy the postinstall section, then open a text editor and paste the contents into a file named postin.sh. Once that's done, make the file executable, and run it:

chmod +x postin.sh  
RPM_INSTALL_PREFIX=$PWD/usr/local bash postin.sh  

Now, copy the files to the correct location:

sudo cp -Rva --no-preserve=ownership ./usr/local/diamond /usr/local/  

The diamond directory we created for the intermediate steps can be removed once installation is complete (optional):

cd ../  
rm -Rf diamond  

Step 3: Setup udev rules

If you will be using a dev board or programming cable with Diamond, you will need to set up some udev rules to ensure the kernel's ftdi_sio driver doesn't bind to the device. We will also need to ensure correct permissions on the devices (NOTE: If Diamond or Programmer start with your device plugged in, but are unable to access the device due to permissions issues, they will segfault! Yay...).

Create a file called /etc/udev/rules.d/10-lattice.rules with the following contents (adjust as necessary). You'll need to be root, or use sudo to create this file:

ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6010", MODE="0666", SYMLINK+="ftdi-%n", RUN+="/bin/sh -c 'basename $(dirname $(realpath /sys%p/device)) > /sys/bus/usb/drivers/ftdi_sio/unbind'",RUN+="/root/ftdi_fixer.py"  

The vendor and product ID can be determined by running lsusb with your device plugged in to your machine.

The above entry runs a couple of commands whenever your device is plugged in. The first is to unbind the device from the ftdi_sio kernel driver. The second is a Python script (introduced shortly), which will properly fix the device entry permissions, since udev fails to do this correctly (it is likely I am doing something wrong, but at least this works).

The script /root/ftdi_fixer.py can be viewed here. This is a short script I wrote (which utilizes libusb1 we installed earlier) to fix the device entry permissions.

sudo curl https://ycc.io/scripts/ftdi_fixer.py -o/root/ftdi_fixer.py  
sudo chmod +x /root/ftdi_fixer.py  

Now that everything is in place, be sure to unplug your cable (if it's plugged in), then reload the udev rules:

sudo udevadm control --reload  

Now plugging in your cable, you should see entries like the following in syslog or dmesg:

[3117880.476085] ftdi_sio ttyUSB1: FTDI USB Serial Device converter now disconnected from ttyUSB1
[3117880.476109] ftdi_sio 3-14:1.1: device disconnected
[3117881.483165] ftdi_sio ttyUSB0: FTDI USB Serial Device converter now disconnected from ttyUSB0
[3117881.483193] ftdi_sio 3-14:1.0: device disconnected

This means that the device was disconnected from the driver (which is what we want). Be sure to check /var/log/syslog for execution errors for the ftdi_fixer.py script if you encounter problems.

Conclusion

Now that the device is connected and ready-to-go, you should be able to access the adapter from within Lattice Diamond or Lattice Programmer (formerly called ispVM).

Notes

• Cable/adapter detection, JTAG chain scanning, and Flash programming are super slow on Linux. While troubleshooting problems initially, I noticed that the Lattice software re-enumerates and checks every single USB device prior to starting any of these operations.
• The MachXO3 Starter Kit, as well as many other Lattice dev & starter boards, feature an FTDI FT2232H. This is common on many recent programming cables, as it allows using the MPSSE interface for banging out JTAG-- and FTDI even have an example that features JTAG (see previous link). The Lattice tools use the ftd2xx-mpsse driver on Linux. However, I was not able to get urJTAG to communicate with the board using its generic FT2232 cable type and matching settings.
• Using an FTDI MPSSE cable (Mouser, Amazon, Digikey) might work as an in-circuit JTAG programmer without any modification (as long as the VID/PID are 0x0403/0x6010, I believe this should work without any issues). Need to buy one and actually test this.

After many months of research and reading through various articles detailing VGA passthrough builds (such as here, here, and here), I finally decided to upgrade my machine— with PCI passthrough being a primary objective of the new build. I have never liked dual-booting, and using Windows as my primary OS is not really an option, as far as I'm concerned. My major issue now is that I need a bigger desk to stick more monitors on ;)

This write-up details my experience in setting up a Windows 10 guest to run on an Ubuntu 16.04 host. I have used libvirt to manage things, and instructions for using virt-manager to perform most tasks have been provided (some configuration, such as CPU pinning or using raw block storage, is not possible via virt-manager). There are a lot of places where things can go wrong, and I'll try to point those out. This should not be considered an exhaustive guide— I've provided links to additional resources at the end of this document in case you get stuck, or have needs that differed from mine.

Requirements

• CPU must support virtualization extensions (VT-x for Intel)
• CPU must support Directed I/O (VT-d for Intel, generically known as IOMMU)
• Motherboard must support VT-x and VT-d (or AMD equivalents). When buying new hardware, check the motherboard's User Manual, which can typically be found on the product page on the manufacturer's website. Example from the PDF manual for my motherboard, which I checked before purchasing.

Setup recommendations

• Kernel should be 4.1 or newer; otherwise you may need to apply various workarounds. vfio-pci is natively supported in 4.1.
• Your host operating system should be installed in UEFI mode, and your machine set to boot via UEFI
• The graphics card you plan to passthrough should have a UEFI or Hybrid BIOS
• Your CPU should fully support ACS if you don't want to worry about IOMMU groupings. If you have a CPU that does not fully support ACS and your IOMMU groupings are less than ideal, workarounds can be done. Intel CPUs with full ACS support. With my CPU and X99A mainboard combo, every single PCI device is inside of its own IOMMU group, without the need for quirks/patches. Many people have setup VGA passthrough without full ACS support-- but having it means one less thing to worry about.
• Two sets of keyboard/mice, or a KVM switch. Once your OS is installed, you'll need to passthrough a mouse and keyboard, which will then be unusable by the guest. After initial setup, you can use something like Synergy if you plan to use both the guest and host simultaneously. Otherwise a KVM switch (or a USB switch) might be a good/simple option.
• The graphics card you're passing through should NOT be the card initialized during boot. In my setup, slot PCIE_1 is the primary 16x/16x slot-- my host's graphics card is connected to this slot, which is used during boot. The guest's card is connected to PCIE_5 (16x/8x)
• Should have two or more discrete PCIe graphics cards. Although using Intel IGD for your host is possible, it is more difficult and error-prone to get working.

My Setup

Software

• Xubuntu 16.04 (64-bit UEFI)
• Linux kernel 4.4.0-59
• QEMU 2.5.0
• libvirt 1.3.1
• virt-manager 1.3.2

Hardware

• CPU: Intel Core i7-6800K Broadwell-E (6 cores/12 threads, 3.4 GHz, LGA 2011-v3, 140W TDP) [ARK]
• Motherboard: MSI X99A XPOWER GAMING TITANIUM (LGA2011-3, Intel X99A Chipset) [Mfg link]
• Memory: Corsair Vengeance LPX 32GB kit (4x8GB DDR4 2133/3200) [Mfg link]
• GPU 1 (host/boot, slot PCIE_1): NVIDIA GeForce GTX 770 (4GB GDDR4, GK104, Rev a1) [Mfg link]
• GPU 2 (guest, slot PCIE_5): XFX Radeon GTR RX 480 (1338MHz, 8GB GDDR5, "Hardswap Fan Black Edition") [Mfg link]

New hardware for my VGA passthrough build

Initial Host Setup

UEFI Check

To ensure that your host has booted via UEFI, check dmesg for EFI-related messages. It is also possible to confirm by checking that /sys/firmware/efi/efivars is populated.

dmesg | grep -i efi  

Install required packages

First, we need to install KVM, libvirt, and OVMF

sudo apt-get update  
sudo apt-get install qemu-kvm qemu-utils qemu-efi ovmf libvirt-bin libvirt-dev libvirt0 virt-manager  

Update modules list

Open up /etc/modules and append the following:

pci_stub  
vfio  
vfio_iommu_type1  
vfio_pci  
vfio_virqfd  
kvm  
kvm_intel  

Enable IOMMU

Now, we need to enable IOMMU support in the kernel at boot-time. To do this with GRUB, edit /etc/default/grub and append intel_iommu=on to the GRUB_CMDLINE_LINUX_DEFAULT option. On a stock install of Ubuntu 16.04, this then becomes:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on"  

Now update GRUB:

sudo update-grub  

Reboot & Check-up

Reboot your machine. This will allow the kernel to boot with IOMMU enabled, and will also load our vfio and pci-stub modules we defined previously.

To ensure IOMMU has been enabled, check for the string Directed I/O, which will be prefixed with either DMAR or PCI-DMA.

~$ dmesg | grep -i 'Directed I/O'
[    0.750152] DMAR: Intel(R) Virtualization Technology for Directed I/O

Determine PCI IDs

If you haven't done so already, you need to determine the PCI device IDs and bus location of the device(s) you want to passthrough. For modern video cards with HDMI audio, you'll also want to passthrough the audio device, which typically has the same location, but different function (BUS:SLOT.FUNC is the location format-- 03.00.1 is bus 3, slot 0, function 1).

To find video cards and their HDMI audio buddies:

lspci -nn | grep -A1 VGA  

On my machine, I get:

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:67df] (rev c7)  
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf0]  
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1)  
04:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)  

Since I want to passthrough the AMD card, I will make note of the PCI VID/PIDs: 1002:67df and 1002:aaf0 for the VGA and Audio device, respectively (VID:PID is the format; in this case 1002:67df has a vendor ID of 0x1002 and product ID of 0x67df). We also need to remember the location, which we will use to determine IOMMU groupings. In the example above: 03:00.0 and 03:00.1 are my locations.

Check IOMMU Groupings

Next up is to determine if the device(s) you want to passthrough are in isolated groups (eg. the groups do not overlap with other devices that you want to leave delegated to the host).

Listing the contents of all IOMMU groups:

find /sys/kernel/iommu_groups/*/devices/*  

You can also check this ugly one-liner I wrote that will inject the IOMMU group number into the lspci output:

for dp in $(find /sys/kernel/iommu_groups/*/devices/*); do ploc=$(basename $dp | sed 's/0000://'); igrp=$(echo $dp | awk -F/ '{print $5}'); dinfo=$(lspci -nn | grep -E "^$ploc"); echo "[IOMMU $igrp] $dinfo" ; done  

In my example, I have determined that my AMD card (and its audio device) are both in group 31. Furthermore, these are the only devices in group 31. This means we should have no problems passing it through with vfio-pci.

[IOMMU 31] 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:67df] (rev c7)
[IOMMU 31] 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf0]

Stubbing with pci-stub

Now that we've confirmed the IOMMU grouping and have the required info, we can set up pci-stub to claim these devices at boot. This prevents the host from assigning a kernel driver to them.

Open up /etc/initramfs-tools/modules, then add the ID(s) of your devices that should be reserved by pci-stub:

pci_stub ids=VID:PID,VID:PID,...  

On my machine, this has the PCI VID/PIDs for the AMD VGA and Audio devices:

pci_stub ids=1002:67df,1002:aaf0  

Important note: If you have other PCI devices that share the same VID/PID (eg. two identical graphics cards), and you plan to delegate one to the host, and the other to the guest-- then this method won't work. Check Alex's vfio-pci-override-vga.sh script at http://vfio.blogspot.co.uk/2015/05/vfio-gpu-how-to-series-part-3-host.html or use xen-pciback instead, which uses bus/location IDs rather than vendor IDs.

Now we need to rebuild the initrd image:

update-initramfs -u  

Once this has completed, reboot.

Once the machine has restarted, check dmesg to ensure that pci-stub has claimed the devices correctly:

~$ dmesg | grep pci-stub
[    2.151798] pci-stub: add 1002:67DF sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.151815] pci-stub 0000:03:00.0: claimed by stub
[    2.151819] pci-stub: add 1002:AAF0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.151827] pci-stub 0000:03:00.1: claimed by stub

Above, we can see that both 03:00.0 (AMD VGA) and 03:00.1 (AMD Audio) were successfully reserved by pci-stub for our future guest VM.

Network Prep

If you don't plan on passing through a network device to your virtual machine, then configuring a suitable bridge on the host is required to achieve a networking configuration where the guest and host can communicate. Although macvtap is probably the simplest, no-setup option, it has a major disadvantage-- your guest will be unable to communicate with the host. Instead, we'll create a bridge on the host and use the virtio network interface for best performance.

Add the following to the bottom of your /etc/sysctl.conf file:

# Enable IPv4 forwarding
net.ipv4.ip_forward=1  
net.ipv4.conf.all.rp_filter=1  
net.ipv4.icmp_echo_ignore_broadcasts=1  
net.ipv4.conf.default.proxy_arp=1

# Enable IPv6 forwarding & IPv6 Autoconfiguration (optional)
net.ipv6.conf.all.autoconf = 0  
net.ipv6.conf.all.accept_ra = 0  
net.ipv6.conf.all.forwarding=1  
net.ipv6.conf.all.proxy_ndp=1

If you're using NetworkManager (enabled by default in Ubuntu Desktop installations), you'll want to ensure the following setting is present in /etc/NetworkManager/NetworkManager.conf. This instructs NetworkManager to ignore any interface that we have explicitly configured in /etc/network/interfaces, but still allows us to use NetworkManager for WiFi, Bluetooth, and VPN configuration.

[ifupdown]
managed=false  

Below is a sample network configuration file (/etc/network/interfaces), where the primary interface (in this example, eno1 -- change this to match your chosen interface) is a part of the vbr0 bridge.

auto lo  
iface lo inet loopback

auto  eno1  
iface eno1 inet manual  
iface eno1 inet6 manual

auto vbr0  
iface vbr0 inet static  
  address YOUR_IPV4_ADDRESS
  netmask YOUR_IPV4_NETMASK
  gateway YOUR_IPV4_GATEWAY

  bridge_ports eno1
  bridge_stp off
  bridge_waitport 0
  bridge_fd 0

iface vbr0 inet6 static  
  address YOUR_IPV6_ADDRESS
  netmask 64
  gateway YOUR_IPV6_GATEWAY

  bridge_ports eno1
  bridge_stp off
  bridge_waitport 0
  bridge_fd 0

Finally, create the new bridge and add the primary interface to it. Once you do this, you will lose network connectivity until you reboot your machine or restart networking. Remember to change eno1 to match your interface.

brctl addbr vbr0  
brctl addif vbr0 eno1  

Once complete, reboot.

Guest VM Setup

VM Creation & Initial Configuration

During OS installation, there is no need to passthrough the GPU. Instead, we'll use Spice (or VNC) for ease of use.

This part can be done in virt-manager-- create a new VM, choose your OS (in my case, I installed Windows 10 Pro), select the installation ISO, then set up your storage. In my case, since I wanted to passthrough an entire physical SSD to my VM, this wasn't possible to setup via the virt-manager GUI (although, it's possible I just didn't have the patience to figure it out). If you plan to do this (or some other non-pool storage scenario), choose No Storage during the creation wizard. We can edit the XML later.

As mentioned above in the Network Prep section, it is recommended that you configure and use a bridge on the host machine if you want to be able to communicate with the host from the guest. In the setup wizard, choose the bridge you created (in my case, I named the bridge jobr0). Also be sure to tick the box "Customize configuration before install".

To complete our VM configuration:

• In the Overview section, make sure to change Firmware to UEFI x86_64: /usr/share/OVMF/OVMF_CODE.fd -- the path to the OVMF ROM may differ slightly depending on your installation. (This part is important! If you don't have this option in virt-manager, then you'll need to manually edit the XML prior to install)
• In the CPUs section, set the model to host-passthrough (you will need to enter it manually). More info on host-passthrough and CPU model configuration. Adjust the number of cores you want to allocate to the guest, if you haven't done so already.
• In the Boot Options section, tick the box for IDE CDROM1 to set it as the primary boot device.
• If you're using any VirtIO devices, click Add Hardware and add another IDE CDROM drive, then attach the VirtIO iso to it. This is required to be able to install Windows on a VirtIO disk! See the note below this list for a link to download the iso.
• In the NIC section, change the Device model to virtio. If you have a specific need, you can set the MAC address here as well. Otherwise you can leave it as the default.
• If you haven't already added a storage device, click Add Hardware to do so now. I would recommend using VirtIO as the bus type for best performance. You may also wish to change the cache mode to writethrough, as your guest is likely going to be doing its own caching.

Check here for an example libvirt XML configuration for OS installation phase.

The Windows libvirt drivers are supplied by RedHat and can be found over at the Fedora wiki: Windows Virtio Drivers. Unless you run into problems, you'll most likely want to choose the stable drivers.

Now we are ready to start the installation— click the Begin Installation button at the top-left. If you need to modify your XML manually (for example, to use a raw block device for storage), then you will still click the Begin Installation button, then immediately stop the VM. This will allow the VM Creation Wizard to save your VM configuration.

If installing Windows and you miss the 'Press any key to boot from CD/DVD...' prompt, then you'll be dumped into OVMF's UEFI shell. The easiest thing to do is just reboot again, or you can navigate to the UEFI binary from the shell.

If you're installing Windows and using a VirtIO disk, you'll be greeted with an empty list of disks to choose from. This is where the VirtIO iso from RedHat/Fedora comes into play.

Click the Load Driver button. Setup will complain that it couldn't find any compatible driver, but that's OK. Click Browse in the lower-left, then navigate to the appropriate directory for your OS. For Windows 10, this was E:\viostor\w10\amd64.

Final Passthrough Setup

After completing OS installation using Spice or VNC, power down the VM so that the final adjustments can be made.

Remove Unnecessary Devices

First, remove any of the following devices from your VM configuration, if they exist:

• Display Spice
• Video Spice/QXL/VNC
• Channel spice
• Tablet
• Mouse
• USB Redirector (Type: SpiceVMC)

The IDE CDROM device created for the Windows ISO can be removed, however, leave the device associated with the RedHad VirtIO drivers connected, as we'll use it later to install additional drivers.

Add Guest Video Card

If using virt-manager, click the Add Hardware button, then choose PCI Host Device from the list. This will present you with a list of all of the devices on your PCI bus, similar to what you'd see from lspci. Most modern video cards will have a function for video (typically .0), and a function for audio (typically .1). Be sure to add both devices.

In the screenshot, the Video and Audio functions associated with the AMD RX480 can be seen on my machine.

Configure Guest USB

You will also need a way to interact with your VM. While sharing your existing mouse and keyboard is possible, it is not recommended, since you won't be able to shut your machine down locally or troubleshoot if something goes wrong.

On my machine, I have a KVM switch that connects to the guest and host (obviously, both USB ports are going to be physically on the same machine-- this might make more sense later). This setup allows me to pass input directly to the guest for initial setup, troubleshooting, as well as gaming. For everything else, a program such as Synergy can be used (you'll still need a keyboard/mouse connected before you get that working, though).

Option 1: (Best option, in my opinion) Use PCI passthrough to give the guest an entire USB bus. This has a few advantages: 1) Allows hotplugging devices while the guest is running, 2) No need to map devices by USB IDs or ports (which can change), 3) Lowest latency. The main disadvantage is that you'll need to determine which physical USB ports map to which USB bus. On my motherboard, even though it has 22 USB ports, most of those ports are on the same USB bus. Thankfully, it does have 4 ports (all grouped together nicely) on the back, which are on their own bus. This may or may not be a viable option, depending on your motherboard and the USB connectivity needs of the host. See the Additional Tweaks: USB Mapping section for more information.

Once you've determined the PCI device associated with the USB bus you'd like to passthrough, add a new PCI Host Device in virt-manager. In the example screenshot below, I will be passing through the VIA VL805 controller, which is mapped to 4 ports on the back of my motherboard, as I mentioned earlier.

Note that we don't need to stub-out the USB bus in question. In fact, this would be quite difficult, since the USB driver is typically a module statically built into the kernel, and is one of the very first modules initialized (would require a custom kernel build, and in the end, wouldn't be very useful). When your host first starts, all devices will associate with the host. When the guest starts, all devices will "disconnect" from the host, and PCI device will be available for the guest. On shutdown of the guest, the devices will reconnect to the host. If this is a problem for your setup, udev rules can be used to 'blacklist' single devices, or you can blacklist entire modules-- just be sure they aren't being used by anything the host needs!

Option 2: Use USB passthrough to provide the guest access to individual USB devices. The main advantage of this option is that it's usually super easy to set up. One drawback is that the KVM switch option I mentioned earlier will not work with this method (since the USB device needs to exist at the time the VM is started in order to pass it through). Additionally, this incurs additional overhead on the host, and increases latency (not really an issue for HID devices like keyboards/mice, but is definitely an issue for anything that requires timely isochronous transfers, like USB audio).

In virt-manager, click Add Hardware, then select USB Host Device. This will provide a list of devices, similar to the output of lsusb. You may notice in the screenshot that there are two identical Logitech devices. When this happens, virt-manager will automatically include the bus location in the configuration. However, this location can (and usually does) change between host reboots.

Booting up!

Once everything is configured to your liking, it's time to boot the VM! Make sure you have a usable input device as discussed above, then start it up. The video below is a quick demo of starting the VM on my machine. Towards the end of the video, you can hear the Windows "USB disconnect" sound as I hit my KVM's foot-pedal to switch input back to the host.

Sample libvirt domain XML: For reference, the libvirt configuration I used can be found here. This also demonstrates how to passthrough entire SSDs or single partitions.

Guest Driver Installation

Now that the guest OS has booted up, it's time to setup the VirtIO drivers. If you're using Linux, this is done automatically. On Windows, we'll use the ISO that was mounted previously. This is required if you've decided to use the VirtIO Network Adapter (less host overhead, better performance) rather than an emulated adapter.

Here's a recap in case you skipped over this before, or no longer have the ISO mounted. Create a new Storage device, change the device type to CDROM, then select the virtio-win ISO.

Ethernet Driver

After Windows has booted up, open up the Device Manager, and you should spot an unknown Ethernet Controller in the Other devices section. Right click on the Ethernet Controller, then choose Update Driver Software. Then when prompted, select Browse my computer for driver software. You can then click Browse... to navigate to the correct directory for your version of Windows. The VirtIO Ethernet driver is under the NetKVM directory. Once expanded, select the directory corresponding to your guest OS, then choose the architecture (amd64 for 64-bit, x86 for 32-bit).

Once completed, the new network adapter should pop into your Network Adapters section, and can be configured. The current version of the driver should provide you with a 10Gbps connection.

Balloon Driver

After installing the Ethernet driver, you can also install the VirtIO Balloon driver. While this is optional, you might as well do it since you've already got the ISO mounted and you're in the Device Manager! This will allow the Windows guest to dynamically expand and relinquish portions of memory as needed.

The balloon interface will show up as an Unknown device under Other devices. Right click, select Update driver software.. and follow the same steps that were done for the Ethernet driver. This time, you'll expand the balloon folder, then choose the correct OS and Arch.

If you added any other custom VirtIO devices, you can install them via the same method (such as serial ports).

Storage Driver

The VirtIO SCSI driver should already be installed if you followed the Load Driver process during installation.

Video & Misc Drivers

Now that your network interface is (hopefully) working, you can take the time to install the latest drivers for your passthrough VGA card. Other than that, you shouldn't really need to install any other drivers, unless you've attached or passed through obscure hardware.

• AMD
• Nvidia

Additional Tweaks

Hopefully your guest OS is now fully up & running— maybe with a few kinks to work out (for me, audio was a pain in the ass to get working, while video worked flawlessly the first time). I've outlined below a few additional tweaks, as well as some alternative methods that can be used.

USB Controller Passthrough

As explained in the setup section, passing through an entire USB bus is (in my opinion) the best option for USB connectivity. It allows hotplugging to be handled by the guest OS, there's no need to worry about device mapping when booting your guest, and it also helps reduce host overhead and latency.

The downside: Which ports map to which bus? For my motherboard, I created this Google Docs spreadsheet to help me map out the ports on my board (my motherboard has 22 USB ports!).

To make things a bit easier, I wrote a simple (nasty) Bash one-liner that will enumerate all of a machine's USB devices, showing which USB bus they are connected to, and the PCI bus location of each USB controller, as well as the corresponding VID:PID combos for both buses.

echo -en "\n USB BUS [PCI BUS] -- DESCRIPTION [PCI VID:PID]\n\tDEVICE NUM: USB VID:PID DESCRIPTION\n"; for upath in /sys/bus/pci/devices/0000:*/usb*; do arx=($(echo "$upath" | perl -ne '/^.+0000:(.+)\/usb([0-9]+)$/ && print "$1 $2"')); loc=${arx[0]}; bus=${arx[1]}; hname=$(lspci -nn | grep "^$loc" | awk -F: '{print $3 ":" $4}'); echo "** Bus ${bus} [${loc}] -- ${hname}"; lsusb -s "${bus}:" | sed 's/^Bus ..../\t/' | sort -n ; done  

For example, here is (abbreviated) output from my machine:

 USB BUS [PCI BUS] -- DESCRIPTION [PCI VID:PID]
    DEVICE NUM: USB VID:PID DESCRIPTION
** Bus 3 [00:14.0] --  Intel Corporation C610/X99 series chipset USB xHCI Host Controller [8086:8d31] (rev 05)
    Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Device 002: ID 0d8c:0008 C-Media Electronics, Inc. 
    Device 005: ID 0644:0200 TEAC Corp. All-In-One Multi-Card Reader CA200/B/S
    Device 006: ID 8087:0a2b Intel Corp. 
    Device 007: ID 046d:c52b Logitech, Inc. Unifying Receiver
    Device 024: ID 046d:0a29 Logitech, Inc. H600 [Wireless Headset]
    Device 039: ID 0566:3055 Monterey International Corp. 
    Device 040: ID 046d:c52b Logitech, Inc. Unifying Receiver
** Bus 4 [00:14.0] --  Intel Corporation C610/X99 series chipset USB xHCI Host Controller [8086:8d31] (rev 05)
    Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
** Bus 1 [00:1a.0] --  Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 [8086:8d2d] (rev 05)
    Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Device 002: ID 8087:800a Intel Corp. 

If you have a lot of devices plugged in, this may give you a pretty good idea of how things might be laid out. If not, you can use something like a phone/tablet or USB thumb drive, and plug it into each port of your computer— making note which bus that port is connected to each time you plug the device in.

Once you've mapped out which ports belong to which bus, you can then check the output from the one-liner above (or lspci -nn) and use the corresponding PCI VID/PID when adding a new PCI Host Device to your virtual machine.

As noted before, there's no need to stub-out the USB controller, since all devices will be gracefully disconnected prior to handoff from the host to guest VM. As a consequence of this, devices attached at startup will first be initialzed by the host until the time when the guest machine starts and takes control of the USB controller.

Raw Block Devices with VirtIO

Rather than using QCOW2 (default) or LVM2 as a backing store for your new VM, an entire SSD (or partition from an SSD can be used instead). The primary advantage of doing this is to allow the disk to be directly accessed from other operating systems if you ever needed to dual-boot or place the drive in another machine. It also bypasses the OS's filesystem layer by directly accessing the device, although the performance gains from doing this are likely minimal.

This will need to be done by manually editing the domain XML definition. Examples of two possible scenarios are shown below from my setup.

It's best to use the /dev/disk/by-id path rather than /dev/sdX, as this can change when your machine is rebooted for various reasons. The cache value should be set to 'none', as the guest operating system will provide its own caching.

Passing through an entire disk:

<disk type='block' device='disk'>  
  <driver name='qemu' type='raw' cache='none'/>
  <source dev='/dev/disk/by-id/ata-M4-CT128M4SSD2_00000000113303180B16'/>
  <backingStore/>
  <target dev='vda' bus='virtio'/>
  <boot order='1'/>
</disk>  

Passing through a single partition:

<disk type='block' device='disk'>  
  <driver name='qemu' type='raw' cache='none'/>
  <source dev='/dev/disk/by-id/nvme-Samsung_SSD_960_EVO_500GB_S3EUNX0HB09470T-part3'/>
  <backingStore/>
  <target dev='vdb' bus='virtio'/>
</disk>  

More information:
- Adding Hard Drives and Other Block Devices to a Guest

Benchmarks

3DMark 2016: Time Spy demo, with AMD Radeon beta driver

3DMark 2016: Fire Strike demo, with AMD Radeon release (official) driver

SteamVR Readiness Test: with AMD Radeon release (official) driver

CrystalDiskMark results: VirtIO SCSI storage - Crucial M4 128GB SATA3 SSD (raw block, device)

CrystalDiskMark results: VirtIO SCSI storage - Samsung 960 EVO NVMe SSD (raw block, partition)

References & Additional Resources

• VFIO Tips & Tricks, Alex Williamson
• PCI passthrough via OVMF, ArchLinux Wiki
• Hyper-V Enhancements for Windows 10 in KVM, Cole Robinson
• GPU Passthrough with QEMU on Arch Linux, DominicM
• GPU Passthrough via vfio-pci with KVM on Ubuntu 15.04, Rachel Chen
• Using GPUs in KVM Virtual Machines, ArrayFire
• KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9, ArchLinux BBS
• KVM VGA Passthrough Database (Google Docs spreadsheet)

A few months ago, I picked up a starter license for HipChat Server out of curiosity. As I use a few other Atlassian products, such as JIRA and Bitbucket Server (aka Stash), I figured it would be distributed similar to their other applications (a self-extracting script). However, HipChat is distributed as an OVA package (Open Virtual Appliance). The file itself is just a tarball containing a few vmdk (VMware) disk images, a checksum file, and an XML description of the contents.

If you're using VirtualBox or VMware, the OVA package can be opened without a problem, and the XML description provides all of the necessary information to create a pretty interface and construct a compliant virtual machine. However, if you're using Xen or KVM/QEMU, you'll need to do a bit more work to make this work properly.

Prerequisites

This guide assumes you have a working knowledge of how to create virtual machines with libvirt, currently have a working host with an LVM2 or QEMU storage pool, and a bridged network configuration.

This guide was written using a Debian 8 host, but can be easily adapted as needed. Most of the commands herein should be run as root, or via sudo.

First, ensure required tools are installed via your package manager (example for Debian-based distros):

apt-get install qemu-utils jq  

Retrieve current OVA archive, then extract it with tar:

wget https://hipchat-server-stable.s3.amazonaws.com/HipChat.ova  
tar -xvf HipChat.ova  

Depending on the type of storage pools you have configured (if any), only one of the following two sections should be performed. If you don't have any storage pools configured, use the QCOW2 section.

LVM2 Storage

If using LVM2 as a backing store, create a new LVM logical volume for each disk image (change the name and volume as necessary).

lvcreate -L$(qemu-img info --output=json system.vmdk | jq '.["virtual-size"]')b -n hipchat-system vg0  
lvcreate -L$(qemu-img info --output=json file_store.vmdk | jq '.["virtual-size"]')b -n hipchat-store vg1  
lvcreate -L$(qemu-img info --output=json chat_history.vmdk | jq '.["virtual-size"]')b -n hipchat-history vg1  

Next, extract the contents from the vmdk images to the new LVM volumes (be sure to update the paths to match your own):

qemu-img convert -np -O raw system.vmdk /dev/vg0/hipchat-system  
qemu-img convert -np -O raw file_store.vmdk /dev/vg1/hipchat-store  
qemu-img convert -np -O raw chat_history.vmdk /dev/vg1/hipchat-history  

QCOW2 Storage

If you are using QCOW2 images instead of LVM2 as your virtual machines' backing store, then a simple conversion should suffice:

qemu-img convert -O qcow2 system.vmdk hipchat-system.qcow2  
qemu-img convert -O qcow2 file_store.vmdk hipchat-store.qcow2  
qemu-img convert -O qcow2 chat_history.vmdk hipchat-history.qcow2  

Create VM Config

Below is an example of the disk configuration for LVM2. Click here for a full example, including Spice console access.

It should be noted that it doesn't really matter which target devices are associated with the disks, as all partitions on the system image are mounted via UUID, and the other volumes are part of an LVM2 pool that has been pre-configured on the system image.

    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/vg0/hipchat-system'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <alias name='system'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/vg1/hipchat-store'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='file_store'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/vg1/hipchat-history'/>
      <backingStore/>
      <target dev='vdc' bus='virtio'/>
      <alias name='chat_history'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>

Network Config

If your network does not use DHCP, then you will want to ensure that the VM is configured with a Spice or VNC console.

On initial startup, you will need to login to the VM with the username admin and password hipchat. Once logged in, you will be able to set a static IP address, and configure the hostname. Once this has been done, you can complete the setup by accessing the web interface via its IP address or hostname you have assigned it (assuming the DNS entry has been added on your DNS server).

Static Routes

Unfortunately, my network configuration requires adding an additional route to allow traffic to be passed to the gateway (eg. with OVH). See this article on how to accomplish this so that your changes are not obliterated the next time Chef runs: How to force network configurations and routes.

Create a file at /home/admin/startup_scripts/static_routes (file below shows example for a server at OVH):

#!/bin/bash
/usr/sbin/sudo /bin/dont-blame-hipchat -c "/sbin/route add 149.56.21.254 dev eth0"
/usr/sbin/sudo /bin/dont-blame-hipchat -c "/sbin/route add default gw 149.56.21.254"

Then ensure it's executable:

chmod +x /home/admin/startup_scripts/static_routes  

You can then run the script to immediately add the routes, or reboot the VM if you want to be extra-sure it works.

Complete Setup

Once networking has been configured and your machine is reachable from the Internet, complete the setup by visiting the IP or hostname of your new HipChat server in a web browser.

Rather than running Cobbler with the default Apache2 installation, below details how to serve Cobbler's WSGI Python services via NGINX's uwsgi_pass. This details the setup on CentOS 7.

Install Cobbler

The latest version of Cobbler can be installed from the EPEL repository on CentOS 7. First, be sure EPEL repos have been enabled, then install Cobbler and its friends. This guide is only focused on getting Cobbler WebUI working behind NGINX— full configuration of Cobbler, dnsmasq, and so on, is outside the scope of the article.

yum -y install epel-release  
yum -y install cobbler cobbler-web dnsmasq pykickstart  

Ensure NGINX is installed

NGINX can be installed from source, or via a package. I typically compile the latest version from git, but it can also be installed from the EPEL repos.

yum -y install nginx  

This setup assumes that NGINX is configured to run as user www-data. If this is not the case, then be sure to modify the uWSGI configuration and Cobbler file ownerships as necessary. Failure to do this will result in the WebUI being unusable. The default Cobbler package in EPEL for CentOS 7 ships with a default owner of apache (all of my other servers use www-data, so I have chosen to change it to match that convention).

Create NGINX config

server {  
    listen 80 default;
    listen [::]:80 default;
    server_name cobbler.example.com _;

    location ~ ^/cblr(?!/svc/)(.*)?$ {
        alias /var/www/cobbler/$1;
    }

    location ~ ^/cobbler_track/(.*)?$ {
        alias /var/www/cobbler/$1;
    }

    location /cobbler {
        alias /var/www/cobbler;
    }

    location /cblr/svc/ {
        include uwsgi_params;
        uwsgi_pass unix:/run/cobbler_svc.sock;
    }

    location /cobbler_api {
        rewrite ^/cobbler_api/?(.*) /$1 break;
        proxy_pass http://127.0.0.1:25151;
    }

    # only force-redirect the web ui
    rewrite ^/$ https://cobbler.example.com/cobbler_web permanent;
    rewrite ^/cobbler_web https://cobbler.example.com$request_uri? permanent;
}

server {  
    # NOTE: remove 'http2' if using nginx < 1.9
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    ssl on;
    ssl_certificate /path/to/your/cert.pem;
    ssl_certificate_key /path/to/your/private.key;

    ssl_prefer_server_ciphers on; 
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_buffer_size 8k; 

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';

    # NOTE: remove or adjust this line as needed,
    # if you're using custom DH params
    # (more info: https://weakdh.org/sysadmin.html)
    ssl_dhparam /etc/ssl/private/dhparams.pem;

    server_name cobbler.example.com;

    access_log /var/log/nginx/cobbler.access.log;
    error_log  /var/log/nginx/cobbler.error.log;

    location /cobbler_webui_content {
        alias /var/www/cobbler_webui_content;
    }

    location ~ ^/cblr(?!/svc/)(.*)?$ {
        alias /var/www/cobbler/$1;
    }

    location /cblr/svc/ {
        include uwsgi_params;
        uwsgi_pass unix:/run/cobbler_svc.sock;
    }

    location /cobbler {
        alias /var/www/cobbler;
    }

    location /cobbler_web {
        rewrite ^/cobbler_web/?(.*) /$1 break;
        include uwsgi_params;
        uwsgi_pass unix:/run/cobbler_web.sock;        
    }

    # redirect requests for / to the Web UI
    rewrite ^/$ https://cobbler.example.com/cobbler_web permanent;
}

Once NGINX has been configured, test the configuration to be sure there are no issues:

nginx -t  

If all is well, reload (or restart) NGINX to pull in the new config.

systemctl reload nginx  

uWSGI Install & Config

Install latest uWSGI via pip, then create a directory for our config files.

yum -y install python-devel python-pip  
pip install uwsgi  
mkdir /etc/uwsgi  

Create the first config file at /etc/uwsgi/cobbler_web.ini

[uwsgi]
wsgi-file = /usr/share/cobbler/web/cobbler.wsgi

master = true  
processes = 2  
max-requests = 5000

socket = /run/cobbler_web.sock  
chmod-socket = 660  
chown-socket = www-data:www-data  
uid = www-data  
gid = www-data  
vacuum = true

die-on-term = true  

And the second config file at /etc/uwsgi/cobbler_svc.ini

[uwsgi]
wsgi-file = /var/www/cobbler/svc/services.py

master = true  
processes = 2  
max-requests = 5000

socket = /run/cobbler_svc.sock  
chmod-socket = 666  
chown-socket = www-data:www-data  
uid = www-data  
gid = www-data  
vacuum = true

die-on-term = true  

Fault 1: ... 'login failed'

If Nginx is running as www-data, you will likely receive the following error message:

Fault 1: "<class 'cobbler.cexceptions.CX'>:'login failed'"  

Fix ownership of the web.ss auth file and webui_sessions. Failure to do this will result in a 500 error with the above error message.

chown www-data.www-data /var/lib/cobbler/web.ss  
chown www-data.www-data /var/lib/cobbler/webui_sessions  

After this is done, you need to fix the issue permanently with one of the following fixes:

• Open /usr/lib/systemd/system/cobblerd.service and add the following line:

ExecStartPost=/bin/bash -c "/bin/sleep 5 ; /bin/chown www-data.www-data /var/lib/cobbler/web.ss"  

Then reload systemd configuration: systemctl daemon-reload

• Alternatively, open /usr/lib/python2.7/site-packages/cobbler/cobblerd.py, find the line reading http_user = "apache" (line 65 in my version of cobblerd.py), and change it to http_user = "www-data". This may not be a good solution, as your changes may get overwritten during updates.

Service Setup

Next, we need to create systemd manifests.

Create /lib/systemd/system/cobbler-web.service and add the following:

[Unit]
Description=uWSGI instance for Cobbler WebUI  
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
ExecStart=/usr/bin/uwsgi /etc/uwsgi/cobbler_web.ini  
ExecStopPost=/usr/bin/rm /run/cobbler_web.sock

[Install]
WantedBy=multi-user.target  

Create /lib/systemd/system/cobbler-svc.service and add the following:

[Unit]
Description=uWSGI instance for Cobbler Service  
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
ExecStart=/usr/bin/uwsgi /etc/uwsgi/cobbler_svc.ini  
ExecStopPost=/usr/bin/rm /run/cobbler_svc.sock

[Install]
WantedBy=multi-user.target  

Then, perform a daemon-reload and enable & start the services

systemctl daemon-reload  
systemctl enable cobbler-web  
systemctl enable cobbler-svc  
systemctl start cobbler-web  
systemctl start cobbler-svc  

After starting the services, visit https://cobbler.example.com/cobbler_web and enter your credentials to login (or cobbler/Cobbler if you haven't configured authentication yet).

Good luck ^_^

Converting Xen 4.4 guest domains to run under KVM

I've been using Xen 4.4 with the XL (XenLite) toolchain for the past 18 months or so, and it has worked very reliably during that time. However, my biggest issue with the XL toolchain is that the support from third-party applications for monitoring, provisioning, and deployment is very slim. I wrote my own crude automated provisioning tool, xlctl, for this reason. I suppose this lack of support is why most folks choose to use XAPI/XCP (XenServer), as this toolchain is widely supported by various third-party tools, and allows you to use Citrix's GUI management tool, and other nice tools, like the web-based Xen Orchestra, or even Microsoft's Virtual Machine Manager (SCVMM).

Conversion to KVM

Luckily, I already have libvirtd installed and running, and am using it to manage my Xen domains (unfortunately for me, many libvirt applications will only work with the KVM or QEMU backends). Because I've already converted my native Xen configuration files over to XML (to use with libvirt), most of my configuration is already done, save for a bit of tweaking to define additional values needed by KVM.

For storage, I am using LVM2 volumes, distributed among 2 primary storage pools (vg0 for SSD-based storage, vg1 for spinny-disk storage), as well as additional img and iso pools for VM images and installation ISOs.

root@mirai ~ # virsh pool-list --all  
 Name                 State      Autostart 
------------------------------------------- 
 img                  active     yes       
 iso                  active     yes       
 vg0                  active     yes       
 vg1                  active     yes       

Domain conversion

Conversion of LVs to partitioned volumes

When using xenpv (paravirtualization), one could simply use PyGrub to boot directly from a supplied kernel on a filesystem. This bypasses all of the normal bootstrapping shenanigans that are typically required. However, with KVM, this is not possible. For this reason, we need to ensure that our volumes have a valid partition label (eg. MSDOS or GPT). If you partitioned your Xen domain logical volumes, then you can skip this step (lucky for you).

• Rename your existing LV with a -old suffix

lvrename /dev/vg1/xr1-disk xr1-disk-old  

• Create a new LV of the same size

lvcreate -L30G -n xr1-disk vg1  

• Use fdisk, gdisk, parted or whatever to partition the new LV. Assuming there will be only a single filesystem, creating one primary partition should be fine. Make sure to set the boot flag! You may need to run partprobe afterwards to force the kernel to load the new partition table.

• Ensure that the device mapper creates a mapping for your new LV partitions by using kpartx

kpartx -al /dev/vg1/xr1-disk  

• At this point, you can either format the filesystem and copy the files with rsync, or shrink the old filesystem by a couple megabytes and copy the entire filesystem with dd. The rsync option is likely going to be less error-prone and complete much faster, but the dd method will provide you with an exact copy of the previous filesystem state.

Option A: mkfs + rsync

• To create a new filesystem (eg. ext4), use the following, making sure to double-check how your new partitions were mapped

mkfs.ext4 /dev/mapper/vg1-xr1--disk1  

• Ensure that both the source and destination filesytems are mounted. I like to create a /mnt2 for this purpose, which will contain our source (original) filesystem, and /mnt will contain our target (new) filesystem.

mkdir /mnt2  
mount /dev/vg1/xr1-disk-old /mnt2  
mount /dev/mapper/vg1-xr1--disk1 /mnt  

• Perform the rsync. The chosen options will preserve all permissions, attributes, xattribs, ownership, times, and so on. --stats and --progress are optional, but provide some feedback about what's happening. Make sure to retain the trailing forward-slash (/) on both the source and destination.

rsync --stats --progress -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} /mnt2/ /mnt/  

• Ensure that everything was copied over correctly, then we can unmount the old filesystem. Once you've successfully booted the domain (later), you may then want to remove it via lvremove

umount /mnt2  

We will keep /mnt mounted (the new filesystem), since it will be needed later.

Option B: resize2fs + dd

This method assumes an Extended-type filesystem (ext2/3/4)

• First, determine the extents of the new partition. We do this by running fdisk against the new LV

fdisk -l /dev/vg1/xr1-disk  

In the output, take note of the number of sectors. Example:

Disk /dev/vg1/xr1-disk: 30 GiB, 32212254720 bytes, 62914560 sectors  
Units: sectors of 1 * 512 = 512 bytes  
Sector size (logical/physical): 512 bytes / 512 bytes  
I/O size (minimum/optimal): 512 bytes / 512 bytes  
Disklabel type: dos  
Disk identifier: 0xa49e92bb

Device             Boot Start      End  Sectors Size Id Type  
/dev/vg1/xr1-disk1       2048 62914559 62912512  30G 83 Linux

In this example: 62912512 sectors, with a sector size of 512 bytes = 32211206144 bytes

• Use this information to resize your existing filesystem via resize2fs. Be sure to take a backup BEFORE doing this if the filesystem contains any crucial data. That being said, I've never had an issue where resize2fs has borked my filesystem when shrinking. You must ensure the filesystem is not mounted before proceeding.

umount /dev/vg1/xr1-disk-old  
e2fsck -f /dev/vg1/xr1-disk-old  
resize2fs -p /dev/vg1/xr1-disk-old 62912512s  

The s unit denotes 512-byte sectors.

• If all went well, your filesystem should now be slightly smaller (note that the LV will remain the same size). Now take the block size output by resize2fs, and this will be used as our count value in dd. resize2fs will specify blocks in terms of 4KiB chunks, so if your version of resize2fs does not output this data, then multiply your sectors by 512 (bytes per sector) and divide by 4096 (number of bytes per block) to arrive at a block count (62912512 * 512 / 4096 = 7864064 4KiB blocks).

dd count=7864064 bs=4K if=/dev/vg1/xr1-disk-old of=/dev/mapper/vg1-xr1--disk1  

4K is always going to be the smallest size you'll need to ensure an exact transfer (4KiB = 1 block). However, if your filesystem happens to be evenly-divisible by a higher value, then you can use that instead to speed things up (just be sure to adjust the count accordingly).

• And to finish things up, run a fsck to ensure everything is intact

e2fsck -f /dev/mapper/vg1-xr1--disk1  

Mount guest disk(s)

First, we need to enter each domain via SSH or console when running -- or if your guest is stopped or the host machine is already running without a Xen hypervisor, you can mount the disk for each domain and perform the conversion in a chrooted environment.

Example of setting up a chrooted environment for one of the guest domains (this domain's disk is an LVM2 logical volume):

mount /dev/vg1/xr1-disk /mnt  
mount --bind /dev /mnt/dev  
mount --bind /proc /mnt/proc  
chroot /mnt  

Update fstab

Be sure to change any reference from /dev/xvda1 to /dev/vda, so that VirtIO can be utilized for best performance. Example:

/dev/vda1      /              ext4   noatime,nodiratime,errors=remount-ro   0   1

Install/Update Kernel

On each guest domain, we need to perform the following setups to configure a kernel and bootloader.

• Install a suitable kernel

For Debian/Ubuntu:

apt-get update  
apt-get -y install linux-image-virtual  

For CentOS/RedHat:

yum -y install kernel  

For ArchLinux:

pacman -S linux  

• Ensure any Xen modules added to the initrd are removed. Edit /etc/mkinitcpio.conf and remove any Xen modules from the MODULES= line. Once this is done, run:

mkinitcpio --kernel=$(file /boot/vmlinuz-linux | perl -pe 's/.*version ([^ ]+) .*/$1/')  

• Ensure biosdevname=0 net.ifnames=0 is added to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub to use the classic network interface naming scheme (eth0 rather than ens3, for example)

All - Ensure VirtIO support in your guest's new kernel:

find /lib/modules/ -name virtio*  

If you see various virtio_*.ko kernel modules for the installed kernel version, then the kernel should be good to go.

GRUB2 Setup & Install

• Install GRUB2

apt-get install grub2  

GRUB installer may freak out if you're running it chrooted, or doesn't detect a normal configuration, just choose 'Yes' to continue with the installation if prompted.

Serial console

Edit /etc/default/grub with the following to enable serial console for GRUB and kernel messages:

GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0"  
GRUB_TERMINAL="serial console"  

(When running grub-mkconfig you may receive a warning about default serial parameters for GRUB_SERIAL_COMMAND-- that's OK)

Note: Some kernels may not output systemd startup messages on the video console if these boot options are used (such as the kernels built for Arch).

Bootloader configuration

First, create a file located at /boot/grub/devices.map (or /mnt/boot/devices.map on the host). This should contain a reference to the host's device (or a loopback) that is used by guest domain. You will have needed to have run kpartx -al on this device so that the partitions are accessible via /dev/mapper (see previous section on LV conversion).

Contents of devices.map:

(hd0) /dev/mapper/vg1-xr1--disk

Once created, we are ready to build the GRUB configuration (still running in the domain itself, or via a chrooted environment). For Ubuntu/Debian:

grub-mkconfig -o /boot/grub/grub.cfg  

For CentOS/RedHat:

grub2-mkconfig -o /boot/grub2/grub.cfg  

Once complete, be sure to unmount any bind mounts, then unmount the guest domain's disk. This can be done all at once with -R:

umount -R /mnt  

Bootloader installation

Unfortunately, I could not determine an easy solution for this, since grub-install is a fucking pain to work with, as far as non-physical disks are concerned. There is probably some better way, but I didn't have time to fuck about any longer.

To finish the bootloader installation, boot up into a rescue disk for Ubuntu 16.04 (will work with Ubuntu, Debian, ArchLinux, and CentOS 7 -- these are the ones I've tested). You can skip network configuration, just be sure to select /dev/vda1 as the mounted filesystem, then enter /dev/vda as the Device for boot loader installation. Once installed, change the boot order such that hd is the primary boot device, then restart (might require destroying, then starting the domain after the reboot for boot order change to take effect).

Excerpt from domain's libvirt XML configuration for the boot disk and VNC configuration. This can (and probably should) be disabled once the bootloader is installed.

    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/opt/repo/iso/ubuntu-16.04.1-server-amd64.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
    </disk>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'/>

If all went well, you should be able to boot from hd and receive a GRUB menu (both on ttyS0 serial console and VNC, if you left it enabled).

This page exists to document the process I went through when setting up a tile server for OpenStreetMap data. During the process, I found the Manually building a tile server page over at switch2osm.org very helpful, and it serves as a basis for this guide.

There are a few reasons you may want to set up your own tileserver: first, you may want to create or tweak map styles to create your own unique tileset; or, you may have an application that makes heavy use of mapping features and does not qualify for free use of the OSM tile servers.

It should be noted that the process described below will not allow you to perform geocoding with the imported data. Geocoding is the process of converting an address or place name to coordinates on the globe (or vice-versa in the case of "reverse" geocoding). For this, you'll want to check out the Nominatim installation instructions over at the OSM wiki. It also relies on the osm2pgsql tool, but the data is indexed and stored differently. With both the Nominatim and Mapnik datasets, you'll be able to perform searching, geocoding, and tile rendering without relying on external sources.

Recommended hardware

• 8GB of RAM (recommended 64GB+)
• 4+ CPU threads is recommended
• 600GB+ of free local storage for full planet import (SSD or other high-speed storage recommended)
• 64-bit architecture (x86_64/amd64)

A full planet import on minimal hardware can take 8 to 14 days to complete. For a dedicated server with SSDs and a lot of RAM, this can be reduced to less than a day with optimal settings. Check out Hetzner or OVH to pick up well-spec'd a server for cheap.

Prerequisites

We're going to need a whole bunch of stuff to get started. Thankfully, the packages in the xenial repos are recent, so we can install Mapnik 3 and friends via the package manager.

We'll also need PostgreSQL with PostGIS support. The commands below will set up Postgres 9.5 and PostGIS 2, as well as various supporting libraries that we may need for compilation of mod_tile and renderd.

apt-get update  
apt-get install libboost-all-dev screen subversion git unzip wget bzip2 build-essential autoconf libtool libxml2-dev libgeos-dev libgeos++-dev libpq-dev libbz2-dev libproj-dev libprotobuf-c0-dev protobuf-c-compiler libfreetype6-dev libpng12-dev libtiff4-dev libicu-dev libgdal-dev libcairo-dev libcairomm-1.0-dev apache2 apache2-dev libagg-dev liblua5.2-dev ttf-unifont lua5.1 liblua5.1-dev libgeotiff-epsg postgresql-9.5 postgresql-9.5-postgis-2.2 postgresql-9.5-postgis-scripts libpq-dev libmapnik3.0 libmapnik-dev mapnik-utils mapnik-reference mapnik-doc python-mapnik python3-mapnik node-carto osm2pgsql  

mod_tile + renderd

mod_tile is a DSO module for Apache 2. It allows the user to define location(s) to serve tiles from (such as a /tiles/ URI in one of your VirtualHosts). It works in tandem with renderd, which utilizes the Mapnik library to do the actual rendering. Rendered tiles are then cached (either to disk, memcached, or Ceph) and used for subsequent requests until they expire.

Installation

We'll need to compile mod_tile & renderd ourselves, but it's an easy build. Below are the instructions for cloning the repo into /opt/src, but another directory can be used. The steps below assume you are running as root.

mkdir -p /opt/src  
cd /opt/src  
git clone https://github.com/openstreetmap/mod_tile.git  
cd mod_tile  
./autogen.sh
./configure --prefix=/usr
make -j`nproc` && make install && make install-mod_tile  
cp /opt/src/mod_tile/debian/renderd.init /etc/init.d/renderd  
chmod +x /etc/init.d/renderd  

Configuration

First, we need to point renderd to our config file.

echo 'DAEMON_ARGS="-c /usr/etc/renderd.conf"' > /etc/default/renderd  

Then open the configuration file (/usr/etc/renderd.conf) and make any necessary changes for your setup. Below are the changes I made:

[renderd]
num_threads=8  
tile_dir=/var/lib/mod_tile  
...

[default]
URI=/tiles/  
TILEDIR=/var/lib/mod_tile  
XML=/opt/maps/style/OSMBright/OSMBright.xml  
HOST=localhost  

Finally, to enable renderd to start when your machine boots, run the following:

systemctl enable renderd  

Postgres

Configuration

Open the main configuration file. For PostgreSQL 9.5 on Ubuntu 16.04, this is located at /etc/postgresql/9.5/main/postgresql.conf. Below are settings that I have adjusted for my machine with 256GB of RAM and 40 cores. These settings have provided the best performance for me, and between 24GB to 32GB of cache seems to be sweet spot, if the benchmarks on the OSM wiki are anything to go by. These settings should be adjusted depending on the amount of RAM available and number of CPU threads in your machine.

max_connections = 200  
shared_buffers = 128MB  
maintenance_work_mem = 4GB  
max_worker_processes = 16  
effective_cache_size = 24GB  
autovacuum = off  

The above configuration is for import only. Once you've successfully completed an import of OSM data into Postgres, the above parameters can be reverted to their defaults. At the very least, autovacuum should be turned back on, as it's responsible for reclaiming deleted objects, and helps to ensure your tables are optimal (unless you're doing manual VACUUM'ing).

For more examples, as well as other parameters you may want to tweak, check out the osm2pgsql/Benchmarks page on the OSM wiki.

User setup

First, open a Postgres shell as the superuser (postgres)

sudo -u postgres psql  

Next, create our osm user that will be used by renderd.

CREATE ROLE osm WITH login PASSWORD 'supersecret';  

If you'll be running the import as the root user, you can use the following to grant root with superuser privs, just like the postgres user:

CREATE ROLE root WITH login superuser;  

Creating the OSM database

With the root role created as a superuser, you can authenticate with peer authentication (Postgres authenticates you via the UID the process is running as) and connect via local rather than via tcp, which will help speed up the import. Using peer authentication also means you don't need to provide a password or username.

At the Postgres prompt, run the following to create a new osm database with UTF-8 encoding, owned by the osm user, and enable the PostGIS and hstore extensions.

The hstore extension is available in PostgreSQL 9.x, and is optional. It allows storing attributes as a hash/dictionary for fields that do not have corresponding dedicated columns. More info: https://www.postgresql.org/docs/9.0/static/hstore.html

CREATE DATABASE osm WITH OWNER osm ENCODING 'UTF-8' TEMPLATE template0;  
\c osm
CREATE EXTENSION hstore;  
CREATE EXTENSION postgis;  
ALTER TABLE geometry_columns OWNER TO osm;  
ALTER TABLE spatial_ref_sys OWNER TO osm;  

If the import aborts or fails, I would recommend DROP'ing your existing database before trying again. In my experience, it seemed like osm2pgsql was not removing the existing data, but this may also be due to autovacuum being disabled.

To drop your existing database:

DROP DATABASE osm;  

Then the creation commands above can be re-run to recreate your database.

osm2pgsql

Now it's time to import some data. Depending on your hardware, this can take anywhere between 8 hours and a couple weeks.

The --slim import process is divided into two phases: phase one is reading all of the nodes, ways, and relations, then caching & indexing them in temporary tables in Postgres; phase two is assembling the actual GIS tables that are used by Mapnik to render map tiles.

During the first phase, speed is primarily dependent upon your disk speed and amount of RAM available, as well as the type of file being imported (XML/bz2 or PBF). PBF is considerably faster, and there is no initial delay before node processing starts. Using a flat node cache also considerably speeds up the node processing if you have an SSD (creates a ~33GB file).

The second phase is more CPU-intensive, and depends upon the number of processes that are assigned via --number-processes.

Important note: If --number-processes is increased beyond 10, you will need to increase the max_connections setting in Postgres to 8x the number of processes. (eg. if --number-processes=16 then increase max_connections in postgresql.conf to 128, plus some buffer room). Don't forget to restart Postgres after any config changes. Failure to make this change will cause Postgres to reach max connections, and osm2pgsql will fail after the first phase. You don't get those hours/days back...

Obtain OSM data

• Full planet: http://wiki.openstreetmap.org/wiki/Planet.osm
• Regional extracts: http://download.geofabrik.de/
• Metro extracts: https://mapzen.com/data/metro-extracts/

Click one of the links above to download your preferred data source, then choose a mirror close to you to obtain a download link for a planet-latest file, or a regional extract. With the link copied, wget it to your server. Unless you have good reason to do otherwise, you should choose a .pbf file, as the import is considerably faster.

Import OSM data

Assuming you've downloaded your planet file, or a local/regional extract, we can proceed to the import. You'll almost certainly want to run this in a screen or with tmux. There is no resume support, so if osm2pgsql crashes, or you lose connection, or you accidentally hit Ctrl+C... then you'll have to start the process all over again.

First, create a screen (or tmux session):

screen -S osm_import  

Change to the directory that you want to run the import from, then adjust the parameters in the command below to suit your needs.

• --hstore - Enable hstore usage
• --slim - Enable slim mode
• -r pbf - Use PBF parser; do not use for bz2/xml imports
• -C 32000 - Cache size in MiB (approx 32GB in this case). If you have sufficient RAM, use between 20GB and 32GB. Otherwise, use 60% of your available memory
• --flat-nodes node.cache - Flat node cache; this is a ~33GB file on disk; using this option drastically speeds up node processing phase; should only be used with SSDs
• --number-processes 16 - Number of helper processes to spawn during the second phase
• -d osm - Use the osm Postgres database
• -U root - Use the root Postgres user
• planet-latest.osm.pbf - Source file

If you're using md5 authentication with Postgres, you can also specify the -H option for Postgres hostname, and then set the PGPASS env var with your Postgres user's password (eg. export PGPASS="supersecret"). In this example, we're using peer authentication with a local (UNIX socket) connection, so that's not required.

osm2pgsql --hstore --slim -r pbf -C 32000 --flat-nodes node.cache --number-processes 16 -d osm -U root planet-latest.osm.pbf  

If using screen, you can use Ctrl+A then d to detach from your session, then screen -r osm_import to re-attach.

Run-time results

Machine:

• 2x Xeon E5-2670v2 2.50GHz Ivy Bridge (total 20 cores/40 threads)
• 256GB DDR3 RAM (16 x 16GB ECC Reg. Buffered)
• 2x Intel DC 480GB SSDs (SSDSC2BB480G6), using LVM+ext4
• Linux kernel 4.4.0-31-generic
• Using osm2pgsql & Postgres settings described in this doc

Full planet import: 16.7 hours
Virginia, US State extract import: 3 minutes, 12 seconds

I've been looking for an easy way to migrate Xen domains between my two Xen hypervisors. I made an (unsuccessful) attempt at setting up an iSCSI target for shared storage between the two servers. However, they are in different datacenters, so this is less than ideal.

Both servers share a similar setup-- storage for each domain's disk is held on a local LVM group, vg0. For instance, the DomU xr11 would be stored on /dev/vg0/xr11-disk on onodera, one of my servers. Therefore, to use the xl migrate command to perform a "live" migration, this storage would simply need to be mirrored over to the other server. I use "live" loosely, since you wouldn't really want to be writing to the disk on your DomU while doing this. In fact, it's probably a good idea to issue a sync command on the guest, then use lvchange -pr to mark the volume as read-only until the migration is complete. But meh, we can't really be bothered with that.

Anyway, this can also be used to create backups of any LV to another server, or to assist in moving part of a volume group to another server (rather than the entire VG). While looking for some official LVM lvdump tool or similar, I found many folks just use dd and ssh to move volumes around their machines.

This script expands upon that idea-- provide it with the path to your LV (such as /dev/vg0/xr11-disk), and the server you would like to migrate it to, and it will create the LV on the new server with the exact same size, then copy the contents across with dd and ssh. I also decided to use gzip with a fast compression setting, since most disks are quite sparse, and there's no sense in wasting bandwidth sending a bunch of zeros or other unimportant data. As you can see from the screenshots below, I was able to migrate a 30GiB volume in about 4 minutes (actual filesystem utilization is about 2GiB).

Once the transfer completes, the script also runs an fsck -fp against the newly-copied volume to ensure filesystem consistency (in case you decided to copy the filesystem while the DomU was running).

The core of the script:

dd if=$LVPATH bs=1M status=none | pv --size=$LVSIZE | gzip -2 | ssh $NEWHOST -- "gunzip | dd of=$LVPATH bs=1M status=none"  

After the migration completes, if you're moving a disk volume for a Xen guest, you can then run xl migrate to do a live migration of the DomU to its new home (memory and state information should be retained). The screenshot below shows me testing this out, and it worked without a problem!

View lvmigrate.sh on Bitbucket or grab the script via wget, as shown below.

Also, be sure to install pv, as this is not included with most distros by default. The line below should install pv on Debian-based and RedHat-based distros, as well as Arch.

apt-get -y install pv || yum -y install pv || pacman -S pv  

Grab the script:

wget https://ycc.io/util/lvmigrate.sh  
chmod +x lvmigrate.sh  

Today I will be going through the process of shrinking a logical volume to free up space on a particular volume group (vg0) in Linux's Logical Volume Manager (LVM). This same/similar process can also be used for expanding volumes, albeit much less dangerous.

We will be shrinking /dev/vg0/onodera-web— that is, the logical volume onodera-web, which is a part of the volume group vg0. Its current size is 300.0GiB, however, I need space for other things, so it needs to be trimmed a bit.

Currently, we only have about ~66GiB free on vg0. After the resize, we should have well over 100GiB to play with:

root@onodera ~ # vgs  
  VG   #PV #LV #SN Attr   VSize   VFree 
  vg0    2   1   0 wz--n- 366.63g 66.63g

Unmount and initial check

First, make sure any running programs or processes that utilize data from this mountpoint have been stopped— in my case, I needed to stop nginx and pm2 (Node.js). Then unmount the volume, and run an fsck against it. You can use -f to force a check, even if the volume is marked clean. The -C0 option outputs status information to fd 0 (stdout).

umount /dev/vg0/onodera-web
fsck -C0 -f /dev/vg0/onodera-web

If you are running a website or other service from your LV, and would like to minimize downtime for large volumes while scanning, you can remount the filesystem in read-only mode prior to running fsck via mount -o remount,ro /dev/vg0/onodera-web — just be sure to unmount once the fsck finished successfully.

If the filesystem was modified, I would recommend re-running fsck once again. Once you've confirmed all is clean, time to move on.

I would also recommend saving information about the LV prior to resizing (such as the exact size of the volume), in case you should need to revert your changes, or the post-resize fsck fails:

lvdisplay --units=b /dev/vg0/onodera-web > preshrink.log

Filesystem resize (resize2fs)

We will now perform the actual resizing of the filesystem-- in this case, the filesystem type is ext4 (resize2fs can be used for any of the Extended filesystems: ext2/ext3/ext4).

resize2fs -p /dev/vg0/onodera-web 250G

If all went well, you should see a message such as: The filesystem on /dev/vg0/onodera-web is now 65536000 (4k) blocks long. (Note: 65536000 blocks * 4096 = 268435456000 bytes = exactly 250.0 GiB)

Logical volume resize (lvreduce)

Now to resize the logical volume to match the filesize:

lvreduce -L 250G /dev/vg0/onodera-web

Final check & remount

fsck -C0 -f /dev/vg0/onodera-web

If the check returned happily, you should now be able to safely remount the volume (note that you may need to adjust the next command if this volume is not in your fstab)

mount /dev/vg0/onodera-web

Now we can check our newly liberated free space:

root@onodera ~ # vgs  
  VG   #PV #LV #SN Attr   VSize   VFree  
  vg0    2   1   0 wz--n- 366.63g 116.63g

And that's it! Good luck ^_~

Taking a look at the settings and steps required to successfully set up and configure networking on a Windows 7 or Windows Server HVM domain on Xen 4.4 hypervisor running Debian

Getting networking to work in Windows under Xen is not a straightforward task, it seems. Here are my notes while attempting to get a Windows 7 domain set up on a server with Hetzner.

The host machine has a subnet statically routed to it by Hetzner, so we can't use straight bridging, because Hetzner's switches would just drop the packets from an unknown MAC address. So here, we take a look at setting up a bridge and using NAT in iptables to bridge the traffic from an "internal" 10.0.9.101 address to the machine's "external" IP address, 176.9.22.223 -- this IP is dedicated to this Windows virtual machine. Note that this type of NAT is basic or one-to-one NAT, since one address directly translates to another-- there's no port forwarding or any other nonsense.

Preface

The xenbr1 is a virtual interface and bridge with an IP of 10.0.9.1. Configuration from /etc/network/interfaces shown below. We will be adding our Windows domain to this bridge later on.

auto xenbr1  
iface xenbr1 inet static  
  address   10.0.9.1
  broadcast 10.0.9.255
  netmask   255.255.255.0
  pre-up brctl addbr xenbr1

Set up on the client domain

Note that I could not get this to work at all with the Intel e1000 ioemu device on Windows 7 x64, but ymmv. I needed to install the GPLPV drivers to even be able to ping the dom0 host.

If you need to install the GPLPV drivers on a host machine without network access, try the following steps to create a new LVM volume that's formatted NTFS.

• Install ntfs-3g utils if they are not already installed.

apt-get install ntfs-3g  

• Create a new 8GB logical volume 'winstrap' on group vg0. Format it as NTFS.

lvcreate -L8G -nwinstrap vg0  
mkfs.ntfs /dev/vg0/winstrap  

• Mount the volume so that we can put the drivers and things on it

mkdir /mnt/winstrap  
mount /dev/vg0/winstrap /mnt/winstrap  

• Download the .NET 4.5 Redist package (required for the Xen Shutdown driver to work) and the latest GPLPV drivers. Here we are downloading the GPLPV drivers for 64-bit Windows; adjust as needed.

cd /mnt/winstrap  
wget http://download.microsoft.com/download/1/6/7/167F0D79-9317-48AE-AEDB-17120579F8E2/NDP451-KB2858728-x86-x64-AllOS-ENU.exe  
wget http://apt.univention.de/download/addons/gplpv-drivers/gplpv_Vista2008x64_signed_0.11.0.373.msi  

Check this page for signed up-to-date GPLPV drivers from Univention, as well as downloads for older versions of Windows and 32-bit OSes

Now that everything is ready, unmount our little bootstrapper volume, and add it to your domain's config as an additional disk.

• Unmount the winstrap volume

umount /mnt/winstrap  

• Edit the disk line in the domain's XM config to include the winstrap volume (in addition to the primary disk, of course)

'phy:/dev/vg0/winstrap,xvdc,rw'  

• Now boot your domain, Windows should have mounted the raw NTFS volume as drive (D:) or some such (thankfully it doesn't care that there isn't a partition table). First install the .NET framework. Do not reboot. Then install the Xen GPLPV drivers, and reboot when prompted. After the machine reboots, it may ask to reboot again, so go ahead and shutdown, but this time destroy the machine so that we can edit its config.
• At this point, remove any references to ioemu devices in the domain's XM config. Your vif line should look similar to the following:

vif = [ 'mac=00:16:3E:29:7B:51, bridge=xenbr1' ]  

Windows network settings

Once the Xen Network Adapter is available, configure the following settings for IPv4. Substitute with your own settings where applicable.

• Address: 10.0.9.101 (should be on the same subnet as xenbr1)
• Netmask: 255.255.255.0
• Gateway: Same gateway as the Dom0 host
• DNS resolvers: Use your ISP's DNS resolvers or Google DNS (8.8.8.8, 8.8.4.4)

Host network settings

Bridge interfaces

Add the domain's interface, named vifXX.0 (where XX is the domain ID, run xl list to check, or better yet xl network-list DOMNAME). xenbr1 is a secondary interface with an IP address of 10.0.9.1 in this case. This should probably be implemented as a vif script or something, so that it doesn't need to be done or undone everytime the machine is created or destroyed.

brctl addif xenbr1 vifXX.0  

Once you have destroyed the domain, its interface will be removed, and will in turn be removed from the bridge configuration. Keep this in mind when recreating the domain again.

Set up forwarding rules in iptables

Forwards incoming traffic on 176.9.22.223 to 10.0.9.103 on the xenbr1 bridge. Outbound traffic from the Windows domain (10.0.9.103) will have its source address rewritten as 176.9.22.223 so that the remote host can actually send a response and make its way back here. Intra-machine traffic to other domains (routed or bridge) still works as expected.

iptables -t nat -A PREROUTING -d 176.9.22.223 -j DNAT --to 10.0.9.103  
iptables -t nat -A POSTROUTING -s 10.0.9.103 -d 176.9.22.223 -j MASQUERADE  
iptables -t nat -A POSTROUTING -s 10.0.9.103 -j SNAT --to-source 176.9.22.223  

Conclusion

After this, you should be able access the domain just as if it were directly connected to your main interface, eth0 or similar, with both ingress and egress traffic being directly forwarded. There is probably a more elegant solution for this, but this seems to work, at least for my set up.

This article covers creating a DomU on Xen 4.4 by installation of an OS from its native install media (typically an ISO image), first as an HVM domain during installation, then converting to a PV domain after installation is complete. To demonstrate, we setup a paravirtualized CentOS 7 (64-bit) domain.

For many candidate DomU distributions, installation via the xen-tools xen-create-image (eg. with debootstrap, rinse, etc.) is not easy to accomplish. This works great for Debian, Ubuntu, and CentOS 5 — but other OSes takes a bit of work, or may not be easily possible without installation from its original media.

Here, we will look at installing an operating system on to a Xen domain by first creating an HVM-based domain which will run during our install of the OS. Then, once installation is completed, we can convert the domain to boot via PyGrub so that it can benefit from the increased speed that comes with paravirtualization (PV) over full hardware virtualization (HVM).

This article looks at installing CentOS 7 (64-bit), but the process can be used for other OSes as well.

Here, I am using Xen 4.4 with the xl toolstack on a Debain 8 Dom0, but can be adjusted as needed for older versions of Xen, or if using the xm toolstack.

Creating an HVM DomU

Create logical volume for disk

• -L60G create 60GiB disk
• -nxr1-disk use volume label xr1-disk
• vg0 create volume in LVM group vg0

lvcreate -L60G -nxr1-disk vg0  

Verify all is good with lvdisplay /dev/vg0/xr1-disk

Fetch installation media

• Select a nearby mirror from http://isoredirect.centos.org/centos/7/isos/x86_64/

mkdir -p /opt/iso  
cd /opt/iso  
wget http://mirror.solarvps.com/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-1503-01.iso  

Create XL/XM config

Manually create config in /etc/xen/config.d/xr1.cfg (xr1 is our DomU hostname)

# Use HVM instead of PV
builder = "hvm"

# Set memory and vcpus as needed
memory = 4096  
vcpus = 2

# Host/Domain name
name = "xr1"

# Setup bridged interface with Intel e1000
vif = [ 'type=ioemu, model=e1000, mac=00:16:3E:29:QQ:QQ, bridge=xenbr1' ]

# Disks - our LVM we just created & the installer ISO image
disk = [  
        'phy:/dev/vg0/xr1-disk,xvda,rw',
        'file:/opt/iso/CentOS-7-x86_64-Minimal-1503-01.iso,xvdb:cdrom,r'
       ]

# Set boot order (d = CDROM, c = HDD)
boot = "dc"

# Use VESA-compliant display with more VRAM
vga = "stdvga"  
videoram = 64

# Use VNC for display
vnc = 1  
vnclisten  = "176.9.0.X"  
vncdisplay = 0  
vncpasswd  = "supersecret"

Boot the domain and begin OS installation

Start the DomU

xl create /etc/xen/config.d/xr1.cfg  

Now connect via a VNC client to the vnclisten IP address on port 5900+vncdisplay (in this case, port 5900)

Now complete the installation steps. Go through the localization portions as needed, important bits outlined below:

• CentOS should auto-detect the installation media and display the installation source as Local media. If not, check to ensure the CDROM device is configured properly in the XM config file
• Choose Installation Destination icon and set up the partitions. Your disk should be displayed here, if it's not, destroy the domain and check the XM config to ensure your disk settings are correct. I created the following partitions...
  • Create first partition, mounted on /boot with a size of 512MiB. Ensure partition type is set to ext3
  • Create second partition, mounted on / -- leave the size field empty to consume the remaining space. Set partition type to ext4
  • Note that no swap space was created, as I do not want to use swap for this domain. It can be added later, though, by expanding the LV this DomU is using, then using fdisk to create a swap partition on the extra space (I think this should be possible, but haven't actually tried it)
• Under Network & Host name, set your DomU's fully-qualified hostname and network settings. Note that in my configuration, this domain will ultimately be routed using the xenbr0:1 interface, but that won't work just yet. It can still be set though, so that when we reboot the DomU in paravirtualized mode, it should connect to the network without issues (and without having to use xl console hopefully)

Make sure to set a root password while the installation is proceeding. Then, once the install is complete, choose the Reboot option. This will probably cause the DomU to hang, but that's OK. Hop back over to Dom0 and kill it:

xl destroy xr1  

(xr1 is the name of our DomU, as specified as name in the config file)

Booting in HVM mode

After the domain has been destroyed, edit the XM config file again, and comment-out or remove the installation media disk. The new disk and boot lines are shown below.

disk = [ 'phy:/dev/vg0/xr1-disk,xvda,rw' ]  
boot = "c"  

Now re-create the domain, and it should boot into our newly-installed OS.

xl create /etc/xen/config.d/xr1.cfg  

Once again, reconnect via VNC, and once booting is complete, you should be greeted by the login prompt! Go ahead and login as root using the password created during installation.

We then want to check the network settings to ensure that the domain will be properly configured when booting in PV mode, since Xen's console kinda sucks, and we won't have VNC access at that point. Also check to ensure the kernel supports paravirtualization, and also detects that it is being virtualized. To do this, run dmesg | grep paravirtual -- this should something like Booting paravirtualized kernel on Xen HVM. If so, we should be good to go. Go ahead and shutdown -h now. Then xl destroy xr1 from the hypervisor.

Booting in PV mode

Now, we will be converting the DomU from an HVM domain to a PV domain. This is to obtain much better I/O performance, as CentOS 7 has a paravirtualized Xen-aware kernel. First, move the old configuration out of the way:

mv /etc/xen/config.d/xr1.cfg /etc/xen/config.d/xr1-hvm.cfg  

Then create a new config for booting into PV mode

# We will be using PyGrub as the bootloader
bootloader = "/usr/lib/xen-4.4/bin/pygrub"

# Set hostname, memory, vpcus, etc.
name = "xr1"  
memory = 4096  
vcpus = 2

# Use the same disk as used previously, with the same device name
disk = [ 'phy:/dev/vg0/xr1-disk,xvda,rw' ]

# Set up a proper routed network connection
vif = [ 'ip=176.9.XXX.XXX, mac=00:16:3E:29:QQ:QQ, gatewaydev=xenbr0:1' ]  

Save the new configuration as /etc/xen/config.d/xr1.cfg, then start up the domain

xl create /etc/xen/config.d/xr1.cfg  

If no errors were shown, it should have booted, as long as you didn't choose any exotic boot or partition type options during installation. We can now try ssh'ing to the newly created DomU with ssh xr1 or ssh 1.2.3.4.

Cloning & Provisioning

Now that you have set up one virtual machine, you can use this first image as a master to create others. This can be done by cloning the LVM volume (or image file), or by mounting the new DomU's disk locally (eg. to /mnt/xr1) so that you can create a tarball that can later be deployed with xen-create-image, which would also allow automatic setup of network parameters and copying of the skeleton directory (/etc/xen-tools/skel).

With the huge amount of new TLDs being released over the past few years, the built-in whois client in Linux can't seem to keep its builtin WHOIS server database up-to-date. Here, we'll look at an easy way to determine the registry's WHOIS server, and query it from the shell. Then generate our own whois.conf file to make future lookups seamless.

If you're short on time, jump to the end of the article for a link to the updated whois.conf file containing up-to-date WHOIS servers for the new gTLDs

Determining the correct server

Likely, you've come across a few domains that you've tried to query due to a DNS problem, or to check the registration info, and you came across a message similar to this:

jacob@minorin:~$ whois mori.moe  
No whois server is known for this kind of object.  

This is because the GNU whois program on most Linux distros comes with the WHOIS server list compiled into the binary. The good news is that you can create a /etc/whois.conf file, which contains a list of regular expressions that correspond to WHOIS servers. However, if you don't want to muck around with this, we can use the whois-servers.net DNS entries to determine the correct WHOIS server.

jacob@minorin:~$ dig +short CNAME moe.whois-servers.net  
whois.nic.moe.  

Yay! Now that we have the WHOIS server, we can use the -h flag in whois to query that server directly. Also, before we move on, it's important to note that "official" source for WHOIS server information is whois.iana.org. For most TLDs, IANA will return an attribute titled whois. Example:

jacob@minorin:~$ whois org -h whois.iana.org | grep whois  
whois:        whois.pir.org  

Moving on to the query:

jacob@minorin:~$ whois mori.moe -h whois.nic.moe  
Domain Name:                                 MORI.MOE  
Domain ID:                                   D389770-MOE  
Sponsoring Registrar:                        BR domain Inc.  
Sponsoring Registrar IANA ID:                1898  
Registrar URL (registration services):       http://www.brdomain.jp  
Domain Status:                               clientTransferProhibited  

You can combine the two if you're lazy and don't want to re-type or copy/paste the server name, like so:

jacob@minorin:~$ whois mori.moe -h `dig +short CNAME moe.whois-servers.net`  

For 3rd level registrations, such as amazon.co.uk, you would query the WHOIS server for the top-level domain (such as uk in this example) in most cases -- however, there are certain 2nd level domains that have their own WHOIS server. If you can't find the domain in the TLD registry's database, do a WHOIS query to IANA for the 2nd level domain to determine if it has its own WHOIS server (eg. whois co.il -h whois.iana.org)

Creating a whois.conf file

For a more elegant solution, you can add the entries for each of the new TLD WHOIS servers to the /etc/whois.conf file. By default, this file doesn't exist, so you'll probably have to create it.

To automate this process, I wrote a simple PHP script which fetches the official list from IANA to enumerate all (public) TLDs, then iterates through the list to determine WHOIS server information from whois-servers.net and whois.iana.org for each of the TLDs. It then uses this information to generate a shiny new whois.conf file and a JSON file that can be used for other purposes.

yc_whoisgen (PHP source, plain text)

As of this writing (15 Mar 2015), there are over 800 top-level domains. You can simply add new entries to your whois.conf as you encounter them, or utilize the script above to generate a new list for yourself. Linked below is updated whois.conf generated by yc_whoisgen

whois.conf (plain text, 07 Mar 2017) — copy to /etc/whois.conf as root

An excerpt from an example whois.conf file showing the regular expression to match a domain on the left, and the WHOIS server where the lookup should be routed on the right side.

\.blue$ whois.afilias.net
\.moe$ whois.nic.moe
\.wtf$ whois.donuts.co
\.ninja$ whois.unitedtld.com

To get started looking up new gTLD domains, you can go ahead and grab the updated config that I've generated above and copy it to /etc/whois.conf. You can also check out the IANA Root Zone Database, if you're still curious about TLDs, or the IANA Official list of TLDs.

Checking out the new whois.conf in vim

DEPRECATED: This patch has been superseded by the inclusion of this feature within Deluge 1.3.14 and later (finally!). The info and patches below still apply to 1.3.x prior to revision 14, but it is best to upgrade if possible.

Stingy WebUI

The current release of Deluge, 1.3.11, as well as the development trunk do not contain a way to force the WebUI to bind only to a single interface or IP address. deluged does include both --interface and --ui-interface options. However, these both specify the IP address of the interface to listen on for BitTorrent connections and JSON-RPC connections respectively.

The result of this being that the Deluge WebUI will instead bind to 0.0.0.0, or all available interfaces on your machine.

jacob@ikumi:~$ sudo netstat -tlpn | grep deluged  
tcp  0  0 0.0.0.0:8112      0.0.0.0:*     LISTEN      1144/deluged.pid  
tcp  0  0 17.9.23.38:4433   0.0.0.0:*     LISTEN      1144/deluged.pid  
tcp  0  0 17.9.23.38:58846  0.0.0.0:*     LISTEN      1144/deluged.pid  
tcp  0  0 17.9.23.38:63617  0.0.0.0:*     LISTEN      1144/deluged.pid  

The first line of the above output of netstat shows the undesirable results of being unable to specify the WebUI interface IP address. While not a problem in most circumstances, if you have a server with many IP addresses, you generally don't want an application such as Deluge occupying the same port on an entire /29 net block. You may have already noticed this issue if you have attempted to run multiple instances of Deluge, both using the default port 8112 for the WebUI, as you would have received an error when trying to bind to the address, as it's already occupied. Of course, you could just change each instance to use its own unique port number.

Making the changes

A git-diff patch to apply the necessary changes automagically to the source is probably the best/easiest solution. Patch and application instructions are provided in the next section

Before starting

Ensure you have a copy of the Deluge source code, preferably cloned from the git repo. Full instructions on fetching, building and installing Deluge from source can be found here.

git clone git://deluge-torrent.org/deluge.git  

Be sure that you have checked out the current stable branch of Deluge.

git checkout -b 1.3-stable origin/1.3-stable  

Digging in

Searching through the source, we can see the file where the listener is bound is located in deluge/ui/web/server.py

self.socket = reactor.listenTCP(self.port, self.site)  

Since Deluge uses the Twisted library to handle communication, the documentation is readily available for reference. Checking the (rather terse) documentation page for listenTCP, we can see that there are two additional arguments that can be passed to the Twisted listener object. We can modify our listenTCP call to include the interface we want to use, instead of using the default listen-to-all-the-things mode.

self.socket = reactor.listenTCP(self.port, self.site, 50, self.iface)  

The third argument is backlog-- we don't really care about this, so we'll use the default value of 50 (see the linked documentation). self.iface is our new property we'll be using for the interface. Note that the same change can be made for the SSL listener:

self.socket = reactor.listenSSL(self.port, self.site, ServerContextFactory(), 50, self.iface)  

listenSSL has an additional ServerContextFactory() argument which was a part of the original call.

Adding our new config value

We also need to initialize the iface attribute to the value specified in the webui.conf file parsed by deluged. To do so, find the __init__ method, and you will notice the section where the other attributes are being initialized-- add our new option to the list:

self.iface = self.config["iface"]  

It's also probably a good idea to set a default value, in case one isn't specified in the config file, otherwise Twisted might freak out. Find the CONFIG_DEFAULTS object and add a default value for iface:

"iface": "0.0.0.0",

Patching

Rather than editing the source (for fun or out of curiosity), the easiest way would be to just apply the patch.

Patch (diff): https://ycc.io/patch/deluge_1313_webui_ip.diff

Updated for v1.3.13

To apply the patch, you should have a clean, working copy/clone of the source. Change the the directory where the source is located and run the following:

wget https://ycc.io/patch/deluge_1313_webui_ip.diff  
git apply deluge_1313_webui_ip.diff  

Building & Installing

Full build and installation instructions can be found on Deluge's website, so here is the tl;dr version:

python setup.py build  
sudo python setup.py install  

Building plugins:

cd deluge/plugins  
for i in */setup.py; do python $i bdist_egg ; done ; rm -Rf *.egg-info ; rm -Rf build ; mv dist/*.egg ./  

You can then either symlink, or copy the plugins to ~/.config/deluge/plugins for the user deluge will be running under.

If you receive errors about setuptools, then you will need to install them first, then rebuild/reinstall.

• For Debian/Ubuntu/Mint: sudo apt-get install python-setuptools
• For RedHat/CentOS/Fedora: (as root) yum -y install python-setuptools

Conclusion

After the build and installation are complete, ensure that you stop and restart deluged completely. I would also recommend enabling debug mode initially in case any problems arise.

Where do I change the IP?

Note that since our new option isn't integrated into the GUI, you will have to manually edit the config file to change it. On most systems, this is located in ~/.config/deluge/web.conf. This is a JSON file, so you can simply add a value like the following:

   "iface": "1.2.3.4",

The config parser in Deluge will pick this up and use it for the WebUI IP address! You can still retain the old functionality by using "0.0.0.0" to bind to all IP addresses, if you'd like.