Architecture¶

Overview¶

Layouts 1.3 and 2.0 use the HPE Morpheus Software Agent’s QuorumCheckService for high availability. This architecture replaces the Pacemaker-based resource management used by Legacy layouts, providing tighter integration with the HPE Morpheus Software platform and simplified operations.

Component Stack¶

Layouts 1.3 and 2.0 rely on three core components:

• Corosync: Cluster membership and messaging layer

• DLM (Distributed Lock Manager): Coordinates file locks for the HPE Clustered Datastore (GFS2) filesystem

• Morpheus Agent (QuorumCheckService): Quorum decisions, failure detection, fencing, and recovery orchestration

Quorum Algorithm¶

The HPE Morpheus Software agent on each host runs a QuorumCheckService that continuously monitors cluster health and makes quorum decisions.

Peer Communication¶

• Each host exposes a /quorum endpoint on port 7443

• Hosts ping each other every 20 seconds

• A node is considered unreachable after 60 seconds with no response

• On a two-node GFS2 cluster, both Hosts ping each other on TCP 7443 and must also reach the Distributed Worker URL. Do not apply same-site peer cross-verification to this topology.

• On a stretch cluster, reachability is confirmed by a majority of same-site peers. Site-level arbitration is documented in Stretch Clusters & Witness Nodes.

Majority Calculation¶

The quorum algorithm uses a simple majority formula:

neededForQuorum = (totalNodes / 2) + 1

totalNodes counts every Agent quorum member. On a two-node GFS2 cluster that includes a Distributed Worker witness, there are three members and majority is 2 of 3.

Agent quorum is the operational authority for layouts 1.3 and 2.0. Corosync Quorate does not decide Agent quorum and may report No on a healthy two-node or stretch cluster. See Building New HVM Clusters and Troubleshooting & Diagnostics.

Witness Topologies¶

A Distributed Worker can provide a quorum-only vote for two supported GFS2 topologies. HPE Morpheus Software classifies the witness from Site Group presence, not from Host count. Agent quorum is sent only for layout 1.3 or later clusters that have at least one HPE Shared File System (GFS2) datastore.

Quorum States¶

Each node operates in one of three states:

Designated Coordinator¶

The cluster elects a Designated Coordinator from among the reachable nodes. The coordinator is determined by the lowest hash of reachable host key hashes (sorted alphabetically). Only the coordinator is authorized to issue fencing operations.

Heartbeat Failover (MvmHeartbeatFailover)¶

The heartbeat failover mechanism monitors host health via shared storage and orchestrates VM recovery when a host fails.

Heartbeat Writes¶

• Each host writes hb.properties to shared storage every 20 seconds

• The heartbeat file contains hostname, memory statistics, and timestamp

• VM XML definitions are written for all running VMs alongside the heartbeat

Failure Detection¶

• A host is considered offline after 140 seconds (7 missed heartbeat intervals)

• The Designated Coordinator assigns the failed host’s VMs to surviving hosts based on available memory

Storage Structure¶

Heartbeat data is stored on the HPE Clustered Datastore (Shared LUN) at:

<datastore>/mvm-hb/<cluster_uuid>-2/<host_hash>/
  ├── hb.properties
  ├── *.xml          (VM definitions)
  └── recover/

APD (All Paths Down) Protection¶

APD protection prevents split-brain data corruption when a host loses access to shared storage.

• After 6 consecutive failed heartbeat writes (~2 minutes), all non-pinned VMs on shared storage are shut down

• This prevents VMs from running with stale data while another host may be serving the same storage

Timing Constants¶

Persistence Files¶

The HPE Morpheus Software agent persists cluster state to the following files on each host: