Insights from practice

Three HPE rack servers as a Proxmox cluster: planning usable reserve

Three HPE rack servers as a Proxmox cluster: how much reserve remains after a host failure? A planning example covering quorum, Ceph, networking, power, backup and costs.

DAXS logo above a cluster plan with three HPE ProLiant hosts, separate network and power paths, and an external Proxmox Backup Server.

Three HPE rack servers alone do not make a resilient Proxmox cluster. The objective becomes testable only when sufficient computing power, memory, accessible VM data, networking and power paths remain after a node failure. This planning example shows the decisions needed for a Proxmox VE cluster using three HPE ProLiant DL360 Gen11 servers.

This article describes a hypothetical sizing profile, not a ready-made HPE order number. Each node is assumed to have a 24-core CPU, 256 GiB of ECC RAM, mirrored boot media, four enterprise SAS SSDs of 3.84 TB each for VM data, two redundant power supplies and suitable 25-Gbit network connections. HPE offers the DL360 Gen11 as a rack server with CPU, DIMM, drive, network and power supply options. Whether this precise combination of CPU, backplane, HBA/JBOD operation, SSD type, fans, NIC, cables and firmware is approved must be established through a bill-of-materials and compatibility check confirmed by the HPE partner. Ceph OSD drives must not be hidden behind an unsuitable hardware RAID array.

Is there enough reserve after a host failure?

Three nodes provide 72 physical CPU cores and 768 GiB of gross RAM. If one fails, 48 cores and 512 GiB remain. In this example, each surviving host reserves four cores and 32 GiB for Proxmox, Ceph and operational peaks. The calculated capacity left for VMs is therefore 40 core equivalents and 448 GiB of RAM. A measured peak of 28 core equivalents and 300 GiB of VM RAM would leave 12 core equivalents and 148 GiB of headroom in this model. A vCPU count alone is not a measurement: CPU steal, NUMA, I/O, memory use and the placement of individual large VMs must be measured in the pilot. Acceptance also covers recovery time and application priorities.

For reliable quorum in HA operation, Proxmox needs at least three votes. With three equally weighted nodes, losing one leaves two votes and therefore a majority. That does not yet guarantee a VM restart: data must be accessible, a destination host must have resources and HA rules must fit. The Corosync network needs stable low latency and an independent second path. A shared failed switch or common PDU defeats the intended host redundancy.

Local SSDs: which data remains available after a failure?

In this example, the twelve local data SSDs form a Ceph pool with three copies and the host as the failure domain. Raw capacity is 3 × 4 × 3.84 = 46.08 TB . Three copies reduce theoretical usable capacity before metadata and reserves to 15.36 TB. With an upper limit set solely for planning of 70%, the working figure is around 10.75 TB. This is not guaranteed Ceph capacity: OSD fill levels, distribution, recovery, snapshots and actual data patterns must be measured.

If one of the three hosts fails, the two remaining copies can continue serving I/O under a suitably configured min_size rule; the pool is then degraded. A third copy on a third host failure domain can be restored only once the node returns or is replaced. That is a limit of a design with exactly three nodes. Ceph replication also does not replace a separate backup. A Proxmox Backup Server belongs outside these three hosts in its own failure domain, with a tested restore and an additional off-site copy. The DAXS article Proxmox Backup Server: verification, retention and restore explores restore testing in more detail.

Networking and power are part of the architecture

Bandwidth and low latency matter for Ceph. For a new cluster, we plan separate, redundantly connected 25-Gbit paths as a testable starting assumption. The required rate follows from write load, replication traffic and the recovery time target, rather than the port count in a data sheet. Corosync should run over independent physical paths; management and VM traffic are separated or limited appropriately. The exact VLAN, switch and NIC assignments must be documented in a port plan.

Two server power supplies help only when connected to separate power paths. DAXS would arrange for each host’s power supply A to connect to PDU/UPS A and power supply B to PDU/UPS B, check consumption under load and test failure of one path. iLO management, spare SSDs, firmware maintenance and maintenance windows are further operating items. Three servers in the same rack do not protect against failure of the entire room.

An open three- and five-year calculation

Let a currently confirmed HPE quote for a fully equipped node be h euros, redundant networking including optics be n, and a separate backup server including media be k. HPE service and spare parts cost, in this model, s euros annually, while power and housing cost e euros annually. These quoted figures are not invented. Published Proxmox list prices are €550 annually per occupied VE socket and €2,240 for a PBS Standard server: here, 3 × 550 + 2,240 = €3,890 per year. Hypothetical one-off implementation, testing and handover take 80 hours at €90 (€7,200); annual operations take 50 hours at €90 (€4,500).

Hypothetical euro calculation excluding VAT; outstanding HPE and site quotes remain variables
Item 3 years 5 years
Nodes, networking, backup hardware 3h + n + k 3h + n + k
One-off implementation 7,200 € 7,200 €
VE/PBS + operations per year 3 × (3,890 + 4,500 + s + e) 5 × (3,890 + 4,500 + s + e)
Total 3h + n + k + 32,370 + 3s + 3e € 3h + n + k + 49,150 + 5s + 5e €

The sensitivity is directly visible: €1,000 more per node increases both horizons by €3,000; €1,000 more in annual HPE service adds €3,000 or €5,000 respectively. If each host needs a second occupied CPU, subscriptions alone increase by €1,650 annually at the same Proxmox Standard level, plus hardware, RAM and energy. The calculation excludes VMware migration costs, a second backup site and price increases; these must be added for the actual decision.

DAXS first checks peak loads and failure objectives, then the approved HPE bill of materials, port and power plan, Ceph and backup restore, and complete quotes. To plan such a cluster for your environment, use Contact to get started.

Sources: HPE ProLiant DL360 Gen11 QuickSpecs · Proxmox VE Administration Guide: cluster, HA and Ceph · Ceph: number of copies and min_size · Proxmox VE: subscription prices · Proxmox Backup Server: prices.