Adding and Removing Hosts¶

Adding a Host to an Existing Cluster¶

Hosts can be added to an existing HVM cluster to increase compute capacity or improve failure tolerance.

Prerequisites¶

Before adding a new host:

• The new host meets all requirements listed in Building New HVM Clusters (OS, CPU, memory, network, storage)

• The new host has network connectivity to all existing cluster hosts on port 7443

• Shared storage (iSCSI targets) is accessible from the new host

• Sufficient license capacity is available

Procedure¶

1. Navigate to Infrastructure > Clusters > [Cluster]

2. Click + Add Worker

3. Provide the new host’s SSH IP, hostname, and credentials

4. Click Complete

What Happens Automatically¶

When a host is added to a layout 1.3 or 2.0 cluster, HPE Morpheus Software performs the following automated operations:

1. Package installation and host preparation — The same provisioning phases run as during initial cluster creation (KVM, Corosync, DLM, layout-specific networking, firewall, libvirt)

2. Corosync authkey distribution — The cluster’s Corosync authentication key is retrieved from Cypher and installed on the new host

3. Corosync configuration update — The new host’s node block is added to corosync.conf on all existing hosts, the config_version is incremented, and corosync-cfgtool -R is run on each host to reload the configuration live (no restart required)

4. DLM join — The new host joins the DLM cluster and gains access to existing lockspaces

5. GFS2 journal addition — For each HPE Clustered Datastore (Shared LUN), a new journal is added using gfs2_jadd if the current journal count is less than the new total node count

6. iSCSI target discovery — All configured iSCSI targets are discovered and auto-logged-in on the new host

7. Quorum list update — HPE Morpheus Software sends updated quorumInfo to all agents in the cluster (including the new host). Each agent persists the updated member list to .quorum-nodes and begins peer-to-peer quorum pinging with the new host

8. Storage pool creation — libvirt storage pools are created on the new host for each existing datastore

Quorum Impact¶

Adding a host changes the quorum majority calculation:

Verification After Adding¶

After the new host is provisioned:

1. Navigate to Infrastructure > Clusters > [Cluster] > Summary > Quorum panel

2. Confirm the total node count has increased

3. Confirm the new host shows as ONLINE

4. On the new host, verify Corosync membership:

   corosync-quorumtool -l
   

5. Verify DLM sees all members:

   dlm_tool status -v
   

6. Compare physical-interface inventory with sudo hvmcli interfaces list --filter ethernet. If an interface was added to the host after enrollment, follow the Netplan and UI refresh procedure in Preparing HVM Hosts before assigning it to a Virtual Switch.

Removing a Host from an Existing Cluster¶

Hosts can be removed from an HVM cluster when decommissioning hardware, reducing cluster size, or replacing failed nodes.

Prerequisites¶

Before removing a host:

• Confirm the cluster layout and ensure the resulting cluster will still have quorum (at minimum 3 nodes for a single-site cluster)

• Confirm the Quorum panel reports ACHIEVED, all surviving hosts are reachable, lockspaces are healthy, and shared datastores are mounted with healthy paths

• Evacuate all VMs from the host using maintenance mode (see Host Maintenance & Evacuation)

• Verify no VMs are pinned to the host that cannot be moved

• Confirm no local or raw-device VM remains on the host and that remaining hosts have the required VM networks and storage access

Procedure¶

1. Place the host in maintenance mode to evacuate VMs (see Host Maintenance & Evacuation)

2. Navigate to Infrastructure > Clusters > [Cluster] > Hosts

3. Select the host to remove

4. Click Remove

5. Confirm the removal

6. Wait for the removal operation to finish; do not manually edit Corosync or Agent quorum files

What Happens Automatically¶

When a host is removed from a layout 1.3 or 2.0 cluster, HPE Morpheus Software performs the following:

1. GFS2 unmount — All HPE Clustered Datastores (GFS2 filesystems) are unmounted on the departing host

2. Cluster services stop — DLM and Corosync are stopped and disabled on the departing host

3. Corosync configuration update — The departing host’s node block is removed from corosync.conf on all remaining hosts. The configuration is archived (backed up to /etc/corosync/archive/), config_version is incremented, and corosync-cfgtool -R reloads the configuration live on each remaining host

4. Datastore cleanup — Each datastore’s location reference to the removed host is cleaned up

5. Quorum list update — Updated quorumInfo is sent to all remaining agents. The departing host is removed from .quorum-nodes on all cluster members

Quorum Impact When Removing¶

Handling Offline Host Removal¶

If the host to be removed is offline (powered off or unreachable):

• First confirm the host is powered off or otherwise cannot access shared storage. If isolation cannot be proven, do not remove it; contact HPE Support

• Use the same UI Remove action. On layout 1.3 and later, HPE Morpheus Software removes the node from the Corosync ring using surviving online hosts and updates Agent quorum membership

• If the departing node is still active in the Corosync ring, or the updated Corosync configuration cannot be written consistently to all survivors, HPE Morpheus Software aborts the reload and raises an alarm rather than risk partitioning the ring

• If no other hosts are online, cleanup cannot proceed and HPE Morpheus Software raises an Unable to remove host alarm

Do not publish or use direct host-shell removal commands as a substitute for this workflow. If the UI operation raises an alarm or cannot establish a safe ring departure, retain the host record and contact HPE Support.

Post-removal Verification¶

1. Confirm the host no longer appears in the cluster Hosts list

2. Confirm the Quorum panel reports ACHIEVED with the new member count and no removed host under down, fenced, or unclean members

3. Confirm Corosync and DLM list only the remaining members and lockspaces are healthy

4. Confirm every shared datastore is mounted and healthy on every remaining host

5. Confirm evacuated or failed-over VMs run once, on hosts with their required networks and storage