Ceph in a Proxmox cluster: high availability without a separate SAN

Ceph in a Proxmox cluster can distribute VM data across local SSDs on several hosts, replacing a separate SAN. What matters operationally is what happens to the copies after a host fails: do quorum, free capacity and networking suffice to restore the required redundancy in time? The following calculation separates data reachability, automatic VM restarts and actual recovery time.

This article focuses on Ceph’s operating mechanics. The specific HPE bill of materials needs its own hardware design; choosing between Ceph, ZFS replication and external storage requires a separate application comparison. The baseline here uses local enterprise SSDs as individual Ceph OSDs, a replicated VM pool and the host as failure domain. Boot devices and external backup are outside this pool.

How do local SSDs provide reachable VM data?

Ceph distributes data objects through placement groups onto OSDs. The CRUSH rule decides which hosts hold copies. This example explicitly sets size=3 and min_size=2: three copies including the primary, with I/O requiring at least two reachable copies. Placing a copy on three OSDs of the same host would not provide threefold protection against host failure. Monitor quorum, OSD state and VM data reachability need separate checks. The Proxmox cluster needs at least three votes for reliable HA quorum; Ceph monitor quorum and VM HA rules are additional requirements.

After one of exactly three hosts fails, correctly distributed replication leaves two host copies. Under the stated conditions, the pool can serve degraded I/O, but a third copy in a third host failure domain exists only once a host returns or is replaced. With four hosts, Ceph can rebuild it on the fourth host if capacity and performance are free. This is the practical difference between “data stays reachable” and “redundancy is restored during the outage”. Even with Ceph, an affected VM can stop and only restart on another host; zero interruption does not follow.

Calculate capacity with failure headroom

Assume each host has four local data SSDs of 1.92 TB decimal each. Three hosts provide 23.04 TB raw, four provide 30.72 TB. With three copies, theoretical logical data capacity before metadata, snapshots and operating reserves is 7.68 or 10.24 TB. A hypothetical 70-percent planning cap initially yields 5.38 or 7.17 TB. It is neither a Ceph default nor a committed usable volume.

To keep the four-host cluster below this chosen planning cap even after a host fails, once rebuilding has finished, the example therefore plans no more than around 70 percent of the remaining capacity: 5.38 TB logical data, rather than 7.17 TB. Also decisive are the fullest OSDs and Ceph’s thresholds for nearfull, backfillfull and full. A single full OSD can block writes or backfill even if the overall total looks healthy. Decimal TB are not binary TiB.

How long does rebuilding take?

With an example 5 TB of logical data and three copies, the four-host cluster holds 15 TB raw, ideally balanced at 3.75 TB per host. Losing one means transferring approximately 3.75 TB to the three remaining hosts for the third copy. At a measured effective rebuild rate of 250 MB/s, 3.75 TB ÷ 250 MB/s = around 4 hours 10 minutes is an arithmetic lower bound; at 125 MB/s, around 8 hours 20 minutes. Peering, distribution, competing VM I/O, recovery limits and new writes lengthen actual time. A 25-Gbit card’s nominal rate is not such a measurement.

DAXS would require a failure test under running application load: observe ceph health detail, ceph osd df and placement-group progression, record actual recovery bytes and duration, and measure latency and VM availability. In normal operation, PGs should return to active+clean . The test must also show whether a switch, link or power-path failure affects Corosync quorum or Ceph traffic simultaneously. Proxmox recommends at least 10 Gbit/s exclusively for Ceph, and more with fast SSDs; the actual port plan depends on OSD throughput, recovery objectives and physically separate paths.

What does the additional rebuild failure domain cost?

For the budget decision, let h be the quoted price of a fully populated host, n the redundant network already sized for four connections, and b an external backup server including media. Per host, assume p euros annually for service, power and colocation. For this model only, setup and acceptance are 100 hours × €90 = €9,000; ongoing operations are 60 hours × €90 = €5,400 annually. The same labor is assumed for three and four hosts and must be estimated again for a real quote. Official list prices for Proxmox VE Standard and one Proxmox Backup Server Standard are €550 per occupied CPU socket/year and €2,240 per backup server/year respectively. Each host has exactly one occupied socket in this assumption. All values are net; replace them with actual quotes and labor estimates.

The formula adds Proxmox sockets, PBS and €5,400 operations each year: €9,290 plus 3p for three hosts, €9,840 plus 4p for four. A €1,000 higher host unit price increases both four-host horizons by €4,000; €1,000 higher annual costs per host add 12,000 or €20,000. If the four-host option needs different switches, add the difference to n . Ceph copies still are not backup: a separate system, external copy and tested restore belong in acceptance. The DAXS article Proxmox Backup Server: verification, retention and restore describes this check.

Decide only after measuring latency, outage and rebuilding time and obtaining a sound hardware, network and operations quote. DAXS can develop this measurement and acceptance protocol with the business through Contact .

Sources: Proxmox VE Administration Guide: cluster and Ceph · Ceph: size and min_size · Ceph: health checks and fullness · Ceph: recovery and backfill · Proxmox VE: pricing · Proxmox Backup Server: pricing.