Pacemaker Resource Resurrection Guide

In case you're ever confronted with an active-passive high availability cluster managed by pacemaker (if the crm_mon command exists on your system, it probably is), it's 4am and you just need that damn service to work again, here's what you can try.

Find out if your component is running under pacemaker

Run crm_mon -nfr1 (if you want to run it permanently, leave out the 1). The result looks a bit like this example:

root@hive:~# crm_mon -nfr1
Last updated: Wed Jul  5 14:17:32 2017
Last change: Wed Jul  5 13:36:41 2017 via crm_attribute on hive
Stack: corosync
Current DC: erwin (1084757795) - partition with quorum
Version: 1.1.10-42f2063
2 Nodes configured
3 Resources configured

Node hive (1084757794):  online
Node erwin (1084757795): online
        ip_smbd         (ocf::heartbeat:IPaddr2):       Started
        fs_smbd         (ocf::heartbeat:Filesystem):    Started
        rc_smbd         (lsb::smbd):                    Started

Inactive resources:

Migration summary:
* Node hive:
* Node erwin:

There are three relevant sections:

Node summary

The node summary lists all nodes in the cluster, and which resources are currently running on them (if any):

Node hive (1084757794):  online
Node erwin (1084757795): online
        ip_smbd         (ocf::heartbeat:IPaddr2):       Started
        fs_smbd         (ocf::heartbeat:Filesystem):    Started
        rc_smbd         (lsb::smbd):                    Started

In this example, node hive is online but does not have any resources running on it. Node erwin is also online, and everything's running there.

Nodes can also be offline or in standby.

Inactive resources

This section lists resources that are currently not running anywhere (either because they're set to stopped, or the node that ran them has crashed).

In this example, that section is empty.

Migration summary

This section shows errors encountered during resource operations. Resource operations are starting, stopping and monitoring resources.

Check if pacemaker is running

If crm_mon complains about not being able to connect to the cluster, this is a good indication that pacemaker is not running.

Pacemaker consists of two services: corosync (the telephone used by the two pacemakers to talk to one another), and pacemaker itself. Both services have init scripts and need to be up and running for pacemaker to work correctly.

Always start the corosync service first (should be in autostart), and pacemaker second (probably not in autostart). You don't need to wait between the two commands though.

Pacemaker is usually not in autostart because it may be desirable to allow the admin (that is, you) to check configs / put nodes in standby / have another beer before starting it. Pacemaker can be somewhat unpredictable at times, and this way you have a chance to implement safety measures.

My resources won't start!

If your resource appears as "stopped" in the inactive resources section, it is not configured to start. Run crm resource start <resource name>, e.g. crm resource start rc_smbd, to start it.

If it appears as started but still does not start and one of your cluster nodes is down, try crm_resource -U migrate <resource name>. This may or may not work, and your best bet is to get the failed node up and running again (or start the component manually).

If the resource has failed operations in the Migration summary, try crm resource cleanup <resource name>. This also may or may not work.

Sometimes, Pacemaker runs into an awkward state and completely freezes. In that case, run the following commands on all nodes:

killall -9 corosync crmd lrmd pengine cib

Best run that multiple times, until it says:

corosync: no process found
crmd: no process found
lrmd: no process found
pengine: no process found
cib: no process found

Then start the services again:

service corosync start
service pacemaker start

and pray to the deity of your choice.

None of that worked!

Are you in an on-call schedule? Now is the time to escalate.