When all of a company’s virtual machines run on a single host, a failed power supply or motherboard means every service stops until the server is repaired. Proxmox VE includes clustering and High Availability out of the box, with no separate license. But HA is not just a checkbox in the settings; if its foundation is not designed properly, it won’t work on the day of a failure, or it may even create new problems.
Cluster and HA: two separate concepts
A cluster is a group of Proxmox hosts that form a single management unit: they are managed from one web console, share configuration, and let you move VMs between them. HA is a layer on top of the cluster: if a host becomes unavailable, the VMs defined in HA automatically start on another host.
Note that HA here means a restart, not uninterrupted operation. The VM boots on the new host, and a few minutes of downtime is normal. For planned maintenance, however, you can move VMs to another host with Live Migration and no downtime.
Quorum: the heart of the cluster
Cluster members constantly check each other’s status using the Corosync service. The cluster only makes decisions when a majority of members (quorum) is available. This rule prevents the worst-case scenario: two hosts that have lost contact with each other, both deciding they should start the same VM, which ends with that VM’s disk corrupted.
- At least three votes are required: A two-member cluster loses its majority when one member is lost. Either have three hosts, or add a QDevice for two hosts: a small separate system that only takes part in voting.
- Fencing: A host that loses contact with the majority restarts itself through a watchdog mechanism, ensuring a VM never runs in two places at once.
Take the cluster network seriously
Corosync is sensitive to low, stable latency, not to high bandwidth. If backup or storage traffic gets heavy on the same network card, members may lose track of each other and hosts may restart for no apparent reason.
- Dedicate a separate network, or at least a separate VLAN, to Corosync.
- Define a second path (a second link) for Corosync so a single switch failure doesn’t take down the cluster.
- Separate storage and migration traffic from the management and Corosync networks.
Storage: the VM must reach its disk on another host
For a VM to start on another host, its disk must be available there. There are three common approaches:
- Shared storage such as NFS or iSCSI; simple, but the storage itself must not be a single point of failure.
- Ceph on the hosts themselves; distributed and with no separate appliance, but it needs at least three hosts, a fast network and careful design.
- ZFS replication between hosts; cheaper, but synchronization is periodic, and changes since the last sync are lost when a failure occurs.
HA is not a replacement for backup. If a VM is infected or corrupted, HA simply starts the same broken VM on another host.
What to know before building a cluster
- Choose member hostnames and IP addresses carefully from the start; changing them after joining the cluster is troublesome.
- A host being added to the cluster should ideally have no VMs; check the exact requirements in the documentation for your version.
- Keep all members on the same Proxmox version and upgrade them one at a time, according to a plan.
- Reserve enough resources: if one host goes down, the others must be able to carry its VMs.
- Depending on the version, use HA groups or rules to specify which hosts each VM should preferably run on.
Checklist
- You have three quorum votes, or two hosts plus a QDevice.
- The Corosync network is separate and has a second path.
- Shared or replicated storage is ready for HA VMs.
- There is spare capacity to absorb the failure of one host.
- You have tested a host failure during off-peak hours.
- Independent VM backups are in place.
A well-designed cluster turns a hardware failure from a crisis into a few minutes of downtime. ITPlus724 engineers can design and deploy your Proxmox cluster or review your existing one.