General Installation Guide
General Recommendations for Installing Proxmox VE
This article contains pointers to good articles and general recommendations, tips and tricks when installing Proxmxox VE on a Production Environment.
Hostname
When installing Proxmox VE the hostname/node name should contain the
domain, if no domain is available then simply use .local
or .localdomain
APT Repositories
In both of the following cases you’ll want to update the node after correcting the repositories.
#!/bin/bash
apt update
apt dist-upgrade --fix-broken --fix-missing
apt autoclean
apt autoremove
Without a Support Subscription
As soon as you install the node(s) in your cluster:
- Enable the PVE No-Subscription Repository
- Enable the CEPH No-Subscription Repository
- Disable the PVE Enterprise Repository
- Disable the CEPH Enterprise Repository
With a Support Subscription
Enable your License Key and enjoy those sweet Enterprise Packages!
Useful Packages and Dependencies
Firstly, run an apt-get update -y
to get the latest APT Lists if you haven’t
already updated the node.
The following Packages/Dependencies are usually very good to have:
- dnsutils → Includes
dig
command. - net-tools → Includes many useful things like
ifconfig
. - ethtool → For networking debugging and disabling off-loading (Is pre-installed in newer PVE Versions).
- ifupdown2 → Allows hot-reloading bridges (Is pre-installed in newer PVE Versions).
- sudo → Not installed by default.
- lvm2
- xfsprogs
- aptitude
- htop
- iotop
- locate
- lshw
- git
- cifs-utils
- bmon
- ethtool
- rename
- iptraf-ng
- nmap
- arping
- sysstat
apt-get install \
sudo \
net-tools \
lvm2 \
xfsprogs \
aptitude \
htop \
iotop \
locate \
lshw \
git \
cifs-utils \
bmon \
ethtool \
rename \
iptraf-ng \
nmap \
arping \
sysstat \
-y
Microcode (CPU Support)
Do an apt update and install the AMD or Intel microcode depending on your HW.
For this you may need the contrib
, non-free
or non-free-firmware
repositories.
Get your current release with:
source /etc/os-release && echo $VERSION_CODENAME
Assuming $release
is your Debian release:
Trixie (deb822 syntax)
Types: deb deb-src
URIs: http://deb.debian.org/debian
Suites: ${release} ${release}-updates
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Types: deb deb-src
URIs: https://security.debian.org/debian-security
Suites: ${release}-security
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Bookworm or before
# Debian Repository Sources
deb https://deb.debian.org/debian ${release} main non-free-firmware
deb-src https://deb.debian.org/debian ${release} main non-free-firmware
deb https://deb.debian.org/debian ${release}-updates main non-free-firmware
deb-src https://deb.debian.org/debian ${release}-updates main non-free-firmware
deb https://security.debian.org/debian-security ${release}-security main non-free-firmware
deb-src https://security.debian.org/debian-security ${release}-security main non-free-firmware
Network Recommendations
Packages
As mentioned before, the following dependencies are good to have if not installed:
apt-get update -y
apt-get install ifupdown2 ethtool net-tools
ifupdown2
andethtool
are installed by default since PVE version 7.4
Off-loading
It’s recommendable to Disable NIC Offloading for performance reasons.
Example on /etc/network/interfaces
file.
# eno1 is an example Physical Interface Name
auto eno1
iface eno1 inet manual
post-up /sbin/ethtool -offload eno1 tx off rx off; /sbin/ethtool -K eno1 gso off; /sbin/ethtool -K eno1 tso off;
#INTERFACE NAME
auto vmbr0
iface vmbr0 inet manual
bridge-ports eno1
bridge-stp off
bridge-fd 0
#BRIDGE NAME
Network Interface Card UDEV Pinning
It’s also recommended to PIN the network interfaces so that their names do not change on kernel updates.
See: Renaming a Network Interface Card with SystemD UDEV Rules.
This is especially important on production environments.
Corosync Ring Redundancy
It’s very important to have Corosync Ring Redundancy and, if possible, have it separated from the LAN and (where applicable) CEPH traffic networks through Layer 1 or Layer 2 separation (different switches, VLANs…).
To setup a redundant Corosync Ring you may refer to the official Proxmox VE documentation.
See:
- Proxmox VE Docs - Adding Redundant Links to an Existing Cluster
- Proxmox VE Wiki - Separate Cluster Network
The TLDR —too long didn’t read— is:
- Make a backup of
/etc/pve/corosync.conf
cp /etc/pve/corosync.conf{,.bkp}
- Make a copy to edit the new file
cp /etc/pve/corosync.conf{,.new}
- Edit the new file (
nano /etc/pve/corosync.conf.new
) adding the following:- New ring address to each node
- In the
totem
block add aninterface
block (containinglinknumber
andbindnetaddr
).
- In the
totem
block increase theconfig_version
count. - Replace the file (
mv /etc/pve/corosync.conf{.new,}
)
Migration and Replication Networks
Migration and Replication Networks can be assigned only when there’s also a Corosync Ring running through that bind network.
Once you’ve assigned a ring to the desired network (e.g. an IPv4 subnet running on a 10GbE switch) you must configure the Migration/Replication Networks to use that ring in:
Datacenter
→Options
→Migration Network
/Replication Network
Storage
If you have an even number of pairs of disks, and prefer Local I/O performance over fast(er) migration and distributed storage, use ZFS in RAID10 without any underlying HW RAID controllers (or just set them to JBOD/Passthrough).
Otherwise feel free to test out CEPH, its pretty well integrated with Proxmox.
We personally prefer the ZFS solution due to its excellent performance, but it requires a bit more micromanagement in terms of migrating and replicating VMs/CTs.
CEPH also takes more space but is easier to scale and handle with larger clusters.
ZFS Recommendations
For ZFS related recommendations and tips, refer to the Proxmox VE ZFS Volume Setups documentation page.
CEPH Recommendations
With CEPH you usually want to:
- Calculate the correct PG groups with the formula or PGCalc from CEPH.
- Disable PG Autoscaling.
- If you’re using HDDs (Enterprise & Consumer grade) or Consumer-grade SSDs (They don’t have PLP), disabling write cache with CEPH is recommended.
- Enable jumbo frames.
SSD Wear & Logging
Unlike some other posts that have recommended logging to RAM in Proxmox VE
with Log2RAM
for extending SSD life, we do not agree with this take.
Our general consensus on this is as follows:
- Always try to have the
rpool
in ZFS RAID1 - Don’t Log to RAM in production environments, ever.
If you ever have a problem to debug, fix, or diagnose, you will need those logs, and they will not be there for you if you log them to RAM.
Our personal experience is that our rpool
SSDs should have no more than 35%
wear after 5-7 years, and that’s if they’re consumer-grade 480GB drives.
If you’re using enterprise, pro-sumer grade drives, or larger and higher quality consumer-grade SSDs, you should not worry about this “issue” whatsoever.