Setting up XCP-NG in the Cloud

In the previous post we went over getting started with XCP-NG. While it runs great on most hardware not everyone has extra desktops or servers laying around. Proxmox is available by default on many dedicated server vendors such as OVH, PhoenixNAP, and Hetzer which makes it easy to install compared to XCP-NG.

When looking at dedicated server providers make sure to look for those that provide you access to the server's Intelligent Platform Management Interface (IPMI) or another Out of Band Management (OOBM) platform with the ability to present virtual media. In our case we went with OVH as we have had previous servers hosted through them and had access to IPMI.

If you prefer a video format it can be found here on YouTube.

OVH has made some changes in their line up recently with a new ECO line, now composed of Kimsufi, So you Start, and Rise. Previously Kimsufi and So you Start had their own websites and interfaces though were stilled owned by OVH. They have now been integrated into OVH's mainline offerings. If you are looking for something more modern or with additional features such as private bandwidth, Service Level Agreements (SLAs) on uptime, etc. there are also Advance, Scale, and High Grade tiers to look at, but the price increases accordingly. For our goals So you Start was the perfect middle ground, though you risk not getting a server with IPMI, so keep that in mind. You may still be able to get around that with some smart hands time. If you want to guarantee access to IPMI go with at least the Rise range.

Here are some quick links to both So You Start and Rise server availability to see what OVH has available as well as a couple of examples for both tiers:

Browse through the offerings and order the server that meets your needs best. Pay particular attention to availability when ordering as OVH operates several datacenters around the world. Most of these servers are available in Canada, France, England, or the United States so make sure to select a geographic area that makes sense to where you or your customers are located.

Once you begin the checkout process you will pick the datacenter to provision the new server in

You will also be presented with an order summary

As well as some engagement options

When purchasing a service from most cloud providers you can choose a longer engagement term or sometimes called a reservation to reduce your overall cost. OVH offers month-to-month, 3 month, 6 month, and 12 month engagements for their dedicated servers. Sometimes going with a longer engagement option than monthly will remove the server setup fees which are typically the cost of 1 month of service.

Given that this will be a single server without additional physical infrastructure to connect to I like to install OPNsense as a VM to handle firewall and NAT duties for any VMs that will run on this host.

To do this at least a single additional IP address needs to be purchased which can be done from the OVH Bare Metal Cloud dashboard under IP, then Order IPs. Select the service you want to allocate the additional addresses to and then chose from either a single IP address or a block of IP addresses. I did a single IP address as OPNsense is the only VM that will need a direct WAN IP address, but if you have other VMs such as a PBX that you may want exposed directly to the internet then a block would be more cost effective. Options range from a /30 (2 hosts) to a /24 (254 hosts). If purchasing a single IP address the gateway will be the .1 address of the network, so if the IP is 189.64.27.168 the gateway would be 189.64.27.1 in that example.

Once ordered go back to the IP area of the dashboard and click on the 3 dots to view options for thew new additional IP address. Select Add a Virtual MAC and use the OVH type. Note down the MAC address as we will need to use that for the WAN interface of the OPNsense VM later.

Once the order is processed, which can take some time depending on the load OVH is under, you can access the OVH Bare Metal Cloud portal to view the server under Dedicated servers.

Before we go messing around with the server make sure to disable monitoring under Service status so OVH doesn't attempt to intervene during this process as the server appears to go offline.

Once on your server there are several options in the navigation bar, click on the IPMI option.

Supermicro offers both HTML5 and Java options for accessing their IPMI, sometimes HTML5 will not offer the ability to mount virtual media so you may have to try both. If you do need to use the Java option Iced Tea Web with Adoptium's JDK 8 have been the best in my experience and can be found on their websites. Make sure to set the JVM used for Iced Tea Web to the JDK 8 that was downloaded and installed.

Once in IPMI boot into the BIOS and ensure that you're booting in UEFI mode. Then click on Virtual Media, then Virtual Storage to open the Virtual Storage window where you can change Device 1 to ISO File once the XCP-NG installer has been downloaded in the next step.

Before getting started you can download the latest XCP-NG LTS from here https://xcp-ng.org/#easy-to-install or you can grab the 8.3 Alpha here https://xcp-ng.org/blog/2022/11/18/xcp-ng-8-3-alpha/

Once the ISO is downloaded, you can burn it to a DVD, write it to a flash drive with RUFUS, drop it on a flash drive with VENTOY installed, or mount over IPMI. Then boot into the installer, you'll be greeted with a welcome screen:

Proceed by pressing Enter on the ok button. XCP-NG will check for existing installs. You will then be prompted to accept the EULA , arrow over to Accept EULA and press Enter if you agree. If the installer found a previously installed version on the disks you will be prompted to upgrade or clean install:

Select which option is best for you, I will use clean install because I want to start fresh. The next step is picking the primary disk where the OS will be installed to. This should be at least 64GB in size and not a flash drive.

I'll be using my SATADOM, if you have multiple you can select Software RAID to install XCP-NG to a mirror of 2 drives. Select the disk and press Enter to proceed. Next we will tell the installer which disks we would like to use for Virtual Machine Storage. Xen refers to these as storage repositories.

In my case I'll be using the 4 SSDs in the system with ZFS so I will leave this unconfigured and proceed by selecting Ok and pressing Enter. I will receive a warning for not selecting any drives for Virtual Machines but can proceed.

Next you'll be prompted to pick your install source, select Local media and proceed. You'll then be prompted to verify the install source, I would recommend verifying. This can take a while depending on the type of media you're using for the installation.

If there are no issues the next prompt is for the root password:

Enter a strong password you remember and proceed. Next we need to select the management interface from a list of connected network adaptors:

Select the NIC you want to use then proceed to IP configuration

You can pick elect to get your IP address from DHCP or set it statically. You can also enter a VLAN ID if you're using tagged VLAN interfaces on your switch. Enter the configuration how you would like and then proceed to DNS information

The installer will generate a random hostname starting with "xcp-ng-" which can be changed. You will also want to set your DNS servers here, up to 3. Then proceed to time zone selection.

You can press a single character to jump to a geographic area, then arrows to select the proper one. Find the correct area for you and then proceed to the specific time zones. You can press a single character to jump to a time zone, then arrows to select the proper one again. Find the correct time zone then proceed to time settings.

You have 2 options for setting time:

• Use NTP
• Manually

I would recommend using NTP.

If you have internal time servers you can enter them here in a tiered list. I used pool.ntp.org for my configuration. Proceed to the final step to confirm the install or return to a previous step. Like verifying the install source this can take a while depending on the type of media you're using for the installation.

Part way through you'll be prompted to install supplemental packs, you can ignore this for now and continue the install. Once it's finished you will be greeted with this to confirm it:

Remove the install media and press enter to reboot into XCP-NG.

Now that XCP-NG is installed log in and confirm you have access to the internet. If that checks out we recommend using SSH for the remainder of the configuration to enable copy and pasting commands. We'll be using variable to make keeping track of things a little easier here due to there being so many UUIDs, you don't have to.

First order of business is to create the ISO folder and storage repository

# Create ISO folder
mkdir /media/isos

# Download and unzip the OPNsense ISO
wget https://mirror.ams1.nl.leaseweb.net/opnsense/releases/23.1/OPNsense-23.1-OpenSSL-dvd-amd64.iso.bz2
bzip2 -d OPNsense-23.1-OpenSSL-dvd-amd64.iso.bz2

# Create ISO SR
xe sr-create name-label=LocalISO type=iso device-config:location=/media/isos content-type=iso device-config:legacy_mode=true

Next the LAN network needs to be created and we can clean up the naming a bit to make things friendlier

# Get Network UUIDs and note the WAN UUID for the WANnet variable
xe network-list
WANnet=eeadc847-5be3-d183-f96d-188e3f9a5dfc

# Set the WAN network name to something friendly
xe network-param-set uuid=$WANnet name-label="OVH WAN"

# Create the network for the internal LAN behind OPNsense and note the LAN UUID for the LANnet variable
xe network-create name-label="Internal LAN"
LANnet=25c73bb9-6a20-ebf4-19ca-a4ba27d797f8

Now we can create the VM

# Get the template UUID for Other install media
xe template-list

# Set the name variable for the VM
NAME=maw-sys-prod-fw01

# Create a new VM for OPNsense
xe vm-install template-uuid=552bce37-51b2-445d-84f2-5f33fa112d7e new-name-label=$NAME

# Get the VM's UUID
xe vm-list
VMUUID=7d84d0ae-5ce1-f890-7602-12ba04158f08

Next let's do some prep work to get the ISO UUID, SR we want to install to UUID and set the MAC variable for the WAN interface

# Get the OPNsense ISO's UUID and set the ISO variable
xe cd-list
ISO=OPNsense-23.1-OpenSSL-dvd-amd64.iso

# Get SR UUID and set the SR variable
xe sr-list
SR=ea3fd586-ee4e-da19-bd2f-60f1d5a47c5b

# Set the MAC variable to the virtual MAC created earlier
MAC="02:00:00:6c:77:7e"

Before we go any further we can verify the variables we have declared so far are correctly set

echo $VMUUID
echo $NAME
echo $ISO
echo $MAC
echo $SR
echo $WANnet
echo $LANnet

Now it's time to create the resources for the VM

# Create OS boot disk
xe vm-disk-add sr-uuid=$SR vm="$NAME" disk-size=64GiB device=0

# Set RAM to 4GB
xe vm-memory-set vm=$VMUUID memory=4GiB

# Set CPUs to 2
xe vm-param-set uuid=$VMUUID VCPUs-max=2
xe vm-param-set uuid=$VMUUID VCPUs-at-startup=2

# Attach ISO to CD Drive
xe vm-cd-add uuid=$VMUUID cd-name=$ISO device=1

# Set to boot from CD
xe vm-param-set HVM-boot-policy="BIOS order" uuid=$VMUUID

# Create the WAN VIF
xe vif-create vm-uuid=$VMUUID network-uuid=$WANnet mac=$MAC device=0

# Create the LAN VIF
xe vif-create vm-uuid=$VMUUID network-uuid=$LANnet device=1

We can make things a bit nicer for ourselves later by naming the newly created VIFs

# Get VM VIFs and note down the VIF UUIDs for WAN and LAN
xe vm-vif-list
WANvif=9a55836a-b0c0-deb0-502b-966f88a39022
LANvif=c7ca2370-ad29-129f-03b4-c589aeb1152d

# Set VIF names for ease of use
xe network-param-set uuid=$WANvif name-label="OVH WAN"
xe network-param-set uuid=$LANvif name-label="LAN"

Also ensure to run these commands to disable TX checksum on the VIFs per the XCP-NG documentation for virtualized firewalls

# Disable TX checksum for OPNsense per documentation
xe vif-param-set uuid=$WANvif other-config:ethtool-tx="off"
xe vif-param-set uuid=$LANvif other-config:ethtool-tx="off"

With the VM now created you can connect to XCP-NG over your preferred method. if you installed 8.3 connect over XO Lite to configure OPNsense or if you used 8.2 another option would be XCP-NG Center.

This will be a small section as there isn't much differentiating this from another install that isn't in the cloud. Take note of the MAC addresses for the WAN and LAN VIFs to make identifying the interfaces easier in OPNsense.

To begin the install start the VM up and boot into the installer. Log in with installer / opnsense to begin. I used ZFS in my configuration to leverage snapshots, but you can also use UFS. Set a strong password and continue through the installer. Once done reboot the VM to load into OPNsense.

Once booted use option 1 to assign interfaces according to the notes on which VIF is LAN and which is LAN by MAC. After interfaces are assigned use option 2 to set the IP address for the WAN interface to the one purchased from OVH. Recall that if you purchased a single IP address that the gateway will be the .1 address of the network, so if the IP is 189.64.27.168 the gateway would be 189.64.27.1 in that example.

You can now use option 8 to ping 1.1.1.1 to verify that the VM can connect to the internet. From here you can configure OPNsense from a VM attached to the LAN network on XCP-NG.

This process was fairly lengthy so we plan to create some additional content around securing the XCP-NG install as well as setting up OPNsense