General Installation Guide

General Recommendations for Installing Proxmox VE

This article contains pointers to good articles and general recommendations, tips and tricks when installing Proxmxox VE on a Production Environment.

Hostname

When installing Proxmox VE the hostname/node name should contain the domain, if no domain is available then simply use .local or .localdomain

APT Repositories

In both of the following cases you’ll want to update the node after correcting the repositories.

#!/bin/bash

apt update
apt dist-upgrade --fix-broken --fix-missing
apt autoclean
apt autoremove

Without a Support Subscription

As soon as you install the node(s) in your cluster:

  • Enable the PVE No-Subscription Repository
  • Enable the CEPH No-Subscription Repository
  • Disable the PVE Enterprise Repository
  • Disable the CEPH Enterprise Repository

With a Support Subscription

Enable your License Key and enjoy those sweet Enterprise Packages!

Useful Packages and Dependencies

Firstly, run an apt-get update -y to get the latest APT Lists if you haven’t already updated the node.

The following Packages/Dependencies are usually very good to have:

  • dnsutils → Includes dig command.
  • net-tools → Includes many useful things like ifconfig.
  • ethtool → For networking debugging and disabling off-loading (Is pre-installed in newer PVE Versions).
  • ifupdown2 → Allows hot-reloading bridges (Is pre-installed in newer PVE Versions).
  • sudo → Not installed by default.
  • lvm2
  • xfsprogs
  • aptitude
  • htop
  • iotop
  • locate
  • lshw
  • git
  • cifs-utils
  • bmon
  • ethtool
  • rename
  • iptraf-ng
  • nmap
  • arping
  • sysstat
apt-get install \
    sudo \
    net-tools \
    lvm2 \
    xfsprogs \
    aptitude \
    htop \
    iotop \
    locate \
    lshw \
    git \
    cifs-utils \
    bmon \
    ethtool \
    rename \
    iptraf-ng \
    nmap \
    arping \
    sysstat \
    -y

Microcode (CPU Support)

Do an apt update and install the AMD or Intel microcode depending on your HW. For this you may need the contrib, non-free or non-free-firmware repositories.

Get your current release with:

source /etc/os-release && echo $VERSION_CODENAME

Assuming $release is your Debian release:

Trixie (deb822 syntax)

Types: deb deb-src
URIs: http://deb.debian.org/debian
Suites: ${release} ${release}-updates
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Types: deb deb-src
URIs: https://security.debian.org/debian-security
Suites: ${release}-security
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Bookworm or before

# Debian Repository Sources
deb https://deb.debian.org/debian ${release} main non-free-firmware
deb-src https://deb.debian.org/debian ${release} main non-free-firmware

deb https://deb.debian.org/debian ${release}-updates main non-free-firmware
deb-src https://deb.debian.org/debian ${release}-updates main non-free-firmware

deb https://security.debian.org/debian-security ${release}-security main non-free-firmware
deb-src https://security.debian.org/debian-security ${release}-security main non-free-firmware

Network Recommendations

Packages

As mentioned before, the following dependencies are good to have if not installed:

apt-get update -y
apt-get install ifupdown2 ethtool net-tools

ifupdown2 and ethtool are installed by default since PVE version 7.4

Off-loading

It’s recommendable to Disable NIC Offloading for performance reasons.

Example on /etc/network/interfaces file.

# eno1 is an example Physical Interface Name
auto eno1
iface eno1 inet manual
        post-up /sbin/ethtool -offload eno1 tx off rx off; /sbin/ethtool -K eno1 gso off; /sbin/ethtool -K eno1 tso off;
#INTERFACE NAME

auto vmbr0
iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
#BRIDGE NAME

Network Interface Card UDEV Pinning

It’s also recommended to PIN the network interfaces so that their names do not change on kernel updates.

See: Renaming a Network Interface Card with SystemD UDEV Rules.

This is especially important on production environments.

Corosync Ring Redundancy

It’s very important to have Corosync Ring Redundancy and, if possible, have it separated from the LAN and (where applicable) CEPH traffic networks through Layer 1 or Layer 2 separation (different switches, VLANs…).

To setup a redundant Corosync Ring you may refer to the official Proxmox VE documentation.

See:

The TLDR —too long didn’t read— is:

  • Make a backup of /etc/pve/corosync.conf
    • cp /etc/pve/corosync.conf{,.bkp}
  • Make a copy to edit the new file
    • cp /etc/pve/corosync.conf{,.new}
  • Edit the new file (nano /etc/pve/corosync.conf.new) adding the following:
    • New ring address to each node
    • In the totem block add an interface block (containing linknumber and bindnetaddr).
  • In the totem block increase the config_version count.
  • Replace the file (mv /etc/pve/corosync.conf{.new,})

Migration and Replication Networks

Migration and Replication Networks can be assigned only when there’s also a Corosync Ring running through that bind network.

Once you’ve assigned a ring to the desired network (e.g. an IPv4 subnet running on a 10GbE switch) you must configure the Migration/Replication Networks to use that ring in:

DatacenterOptionsMigration Network / Replication Network

Storage

If you have an even number of pairs of disks, and prefer Local I/O performance over fast(er) migration and distributed storage, use ZFS in RAID10 without any underlying HW RAID controllers (or just set them to JBOD/Passthrough).

Otherwise feel free to test out CEPH, its pretty well integrated with Proxmox.

We personally prefer the ZFS solution due to its excellent performance, but it requires a bit more micromanagement in terms of migrating and replicating VMs/CTs.

CEPH also takes more space but is easier to scale and handle with larger clusters.

ZFS Recommendations

For ZFS related recommendations and tips, refer to the Proxmox VE ZFS Volume Setups documentation page.

CEPH Recommendations

With CEPH you usually want to:

  • Calculate the correct PG groups with the formula or PGCalc from CEPH.
  • Disable PG Autoscaling.
  • If you’re using HDDs (Enterprise & Consumer grade) or Consumer-grade SSDs (They don’t have PLP), disabling write cache with CEPH is recommended.
  • Enable jumbo frames.

SSD Wear & Logging

Unlike some other posts that have recommended logging to RAM in Proxmox VE with Log2RAM for extending SSD life, we do not agree with this take.

Our general consensus on this is as follows:

  • Always try to have the rpool in ZFS RAID1
  • Don’t Log to RAM in production environments, ever.

If you ever have a problem to debug, fix, or diagnose, you will need those logs, and they will not be there for you if you log them to RAM.

Our personal experience is that our rpool SSDs should have no more than 35% wear after 5-7 years, and that’s if they’re consumer-grade 480GB drives.

If you’re using enterprise, pro-sumer grade drives, or larger and higher quality consumer-grade SSDs, you should not worry about this “issue” whatsoever.