Hello all,
First, thanks in advance for any help anyone may provide. I've been battling
this problem off and on for months and it is driving me mad:
Once every week or two my cluster fails. For reasons unknown it seems to
initiate a failover and then the shorewall service (lsb) does not get started
(or is stopped). The majority of the time it happens just after 4am. Although
it has happened at other times, although much less frequently. Tonight I am
going to have to be up at 4am to poke around on the cluster and observe what is
happening, if anything.
One theory is some sort of resource starvation such as CPU but I've stress
tested it and run a backup and a big file copy through the firewall at the same
time and never get more than 1 core of cpu (almost all due to the backup) out
of 4 utilized and nothing interesting happening to pacemaker/resources.
My setup is a bit complicated in that I have 63 IPaddr2 resources plus the
shorewall resource. Plus order and colocation rules to make sure it all sticks
together the IPs come up before shorewall.
I am running the latest RHEL/CentOS RPMs in CentOS 6.5:
[root@new-fw1 shorewall]# rpm -qa |grep -i corosync
corosync-1.4.1-17.el6.x86_64
corosynclib-1.4.1-17.el6.x86_64
[root@new-fw1 shorewall]# rpm -qa |grep -i pacemaker
pacemaker-1.1.10-14.el6.x86_64
pacemaker-libs-1.1.10-14.el6.x86_64
pacemaker-cluster-libs-1.1.10-14.el6.x86_64
pacemaker-cli-1.1.10-14.el6.x86_64
I am a little concerned about how pacemaker manages the shorewall resource. It
usually fails to bring up shorewall after a failover event. Shorewall could
fail to start if the IP addresses shorewall is expecting to be on the
interfaces are not there yet. But I have dependencies to prevent this from ever
happening such as:
order shorewall-after-dmz-gw inf: dmz-gw shorewall
I also wonder if the shorewall init script is properly LSB compatible. It
wasn't out of the box and I had to make a minor change. But now it does seem to
be LSB compatible:
[root@new-fw2 ~]# /etc/init.d/shorewall status ; echo "result: $?"
Shorewall-4.5.0.1 Status at new-fw2.mydomain.com - Fri Dec 27 16:57:14 PST 2013
Shorewall is running
State:Started (Fri Dec 27 04:11:14 PST 2013) from /etc/shorewall/
result: 0
[root@new-fw2 ~]# /etc/init.d/shorewall stop ; echo "result: $?"
Shutting down shorewall: [ OK ]
result: 0
[root@new-fw2 ~]# /etc/init.d/shorewall status ; echo "result: $?"
Shorewall-4.5.0.1 Status at new-fw2.mydomain.com - Fri Dec 27 16:57:48 PST 2013
Shorewall is stopped
State:Stopped (Fri Dec 27 16:57:47 PST 2013)
result: 3
[root@new-fw2 ~]# /etc/init.d/shorewall start ; echo "result: $?"
Starting shorewall: Shorewall is already running
[ OK ]
result: 0
[root@new-fw2 ~]# /etc/init.d/shorewall status ; echo "result: $?"
Shorewall-4.5.0.1 Status at new-fw2.mydomain.com - Fri Dec 27 16:58:04 PST 2013
Shorewall is running
State:Started (Fri Dec 27 16:57:53 PST 2013) from /etc/shorewall/
result: 0
So it shouldn't be an LSB issue at this point...
I have a very hard time making heads or tails of the
/var/log/cluster/corosync.log log files. For example, I just had this appear in the log files:
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave spider2-eth0-40 (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave spider2-eth0-41 (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave corpsites (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave dbrw (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave mjhdev (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave datapass1-ssl-eth0-2 (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave datapass2-ssl-eth0-2 (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave datapass2-ssl-eth0-1 (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave datapass2-ssl-eth0 (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave rrdev2 (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave webmail-resumepromotion (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1553] new-fw1.mydomain.com crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Dec 27 19:56:31 [1553] new-fw1.mydomain.com crmd: info: do_te_invoke: Processing graph 599 (ref=pe_calc-dc-1388202991-1545) derived from /var/lib/pacemaker/pengine/pe-input-894.bz2
Dec 27 19:56:31 [1553] new-fw1.mydomain.com crmd: notice: run_graph: Transition 599 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-894.bz2): Complete
Dec 27 19:56:31 [1553] new-fw1.mydomain.com crmd: info: do_log: FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
Dec 27 19:56:31 [1553] new-fw1.mydomain.com crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: notice: process_pe_message: Calculated Transition 599: /var/lib/pacemaker/pengine/pe-input-894.bz2
Is this normal? What does "Leave" mean? This is on the inactive node. The
active node doesn't seem to be making any log entries at all.
My corosync.conf and full crm config are as follows. There has also been
concern expressed about the size of our crm config. Is this unreasonable? I am
going to attempt to attach the 120k bzipped logfile to this email but odds are
the list won't take attachments. If it doesn't make it and anyone wants to see
it I will figure out somewhere else to stash it.
The logs are so voluminous that it is hard to know which parts are necessary.
corosync.conf:
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {
version: 2
secauth: off
threads: 0
interface {
ringnumber: 0
bindnetaddr: 10.0.2.0
mcastaddr: 226.94.1.1
mcastport: 5000
member {
memberaddr: 10.0.2.83
}
member {
memberaddr: 10.0.2.84
}
}
transport udpu
}
logging {
fileline: off
to_stderr: no
to_logfile: yes
to_syslog: yes
logfile: /var/log/cluster/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
amf {
mode: disabled
}
crm configure show:
node new-fw1.mydomain.com \
attributes standby="off"
node new-fw2.mydomain.com \
attributes standby="off"
primitive actonomy ocf:heartbeat:IPaddr2 \
params ip="206.71.189.165" nic="eth1" cidr_netmask="25" \
op monitor interval="30s" \
meta is-managed="true"
primitive corpsites ocf:heartbeat:IPaddr2 \
params ip="206.71.189.150" nic="eth1" cidr_netmask="25" \
op monitor interval="30s" \
meta target-role="Started"
primitive datapass1-ssl-eth0-2 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.166" nic="eth1" cidr_netmask="25" \
op monitor interval="30s"
primitive datapass2-ssl-eth0 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.161" nic="eth1" cidr_netmask="25" \
op monitor interval="30s"
primitive datapass2-ssl-eth0-1 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.162" nic="eth1" cidr_netmask="25" \
op monitor interval="30s"
primitive datapass2-ssl-eth0-2 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.167" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive dbrw ocf:heartbeat:IPaddr2 \
params ip="206.71.189.250" nic="eth1" cidr_netmask="32" \
op monitor interval="30s" \
meta target-role="Started"
primitive dmz-gw ocf:heartbeat:IPaddr2 \
params ip="10.0.2.254" cidr_netmask="32" \
op monitor interval="30s"
primitive mjhdev ocf:heartbeat:IPaddr2 \
params ip="209.216.236.138" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive mjhwiki ocf:heartbeat:IPaddr2 \
params ip="206.71.189.163" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive mx1 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.65" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive pin1 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.147" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive pin2 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.146" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive pin3 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.145" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive rabbit ocf:heartbeat:IPaddr2 \
params ip="206.71.189.149" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive reza ocf:heartbeat:IPaddr2 \
params ip="216.240.176.66" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive rrdev ocf:heartbeat:IPaddr2 \
params ip="206.71.189.164" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive rrdev2 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.240" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive shorewall lsb:shorewall \
op monitor interval="60s" \
meta target-role="Started" is-managed="true"
primitive spider2-eth0-1 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.67" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-10 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.76" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-11 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.77" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-12 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.78" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-13 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.79" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-14 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.80" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-15 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.81" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-16 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.82" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-17 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.83" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-18 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.84" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-19 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.85" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-2 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.68" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-20 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.86" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-21 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.87" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-22 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.88" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-23 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.89" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-24 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.90" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-25 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.91" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-26 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.92" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-27 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.93" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-28 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.94" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-29 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.95" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-3 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.69" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-30 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.96" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-31 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.97" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-32 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.98" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-33 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.99" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-34 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.100" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-35 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.101" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-36 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.102" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-37 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.103" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-38 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.104" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-39 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.105" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-4 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.70" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-40 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.106" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-41 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.107" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-5 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.71" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-6 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.72" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-7 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.73" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-8 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.74" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-9 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.75" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive sugar ocf:heartbeat:IPaddr2 \
params ip="206.71.189.130" nic="eth1" cidr_netmask="25" \
op monitor interval="30s"
primitive webmail ocf:heartbeat:IPaddr2 \
params ip="206.71.189.254" nic="eth1" cidr_netmask="25" \
op monitor interval="30s"
primitive webmail-resumepromotion ocf:heartbeat:IPaddr2 \
params ip="206.71.189.242" nic="eth1" cidr_netmask="25" \
op monitor interval="30s"
primitive wiki ocf:heartbeat:IPaddr2 \
params ip="209.216.236.135" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
colocation dmz-gw-with-actonomy inf: dmz-gw actonomy
colocation dmz-gw-with-mjhwiki inf: dmz-gw mjhwiki
colocation dmz-gw-with-mx1 inf: dmz-gw mx1
colocation dmz-gw-with-pin1 inf: dmz-gw pin1
colocation dmz-gw-with-pin2 inf: dmz-gw pin2
colocation dmz-gw-with-pin3 inf: dmz-gw pin3
colocation dmz-gw-with-rabbit inf: dmz-gw rabbit
colocation dmz-gw-with-reza inf: dmz-gw reza
colocation dmz-gw-with-rrdev inf: dmz-gw rrdev
colocation dmz-gw-with-spider2-eth0-1 inf: dmz-gw spider2-eth0-1
colocation dmz-gw-with-spider2-eth0-10 inf: dmz-gw spider2-eth0-10
colocation dmz-gw-with-spider2-eth0-11 inf: dmz-gw spider2-eth0-11
colocation dmz-gw-with-spider2-eth0-12 inf: dmz-gw spider2-eth0-12
colocation dmz-gw-with-spider2-eth0-13 inf: dmz-gw spider2-eth0-13
colocation dmz-gw-with-spider2-eth0-14 inf: dmz-gw spider2-eth0-14
colocation dmz-gw-with-spider2-eth0-15 inf: dmz-gw spider2-eth0-15
colocation dmz-gw-with-spider2-eth0-16 inf: dmz-gw spider2-eth0-16
colocation dmz-gw-with-spider2-eth0-17 inf: dmz-gw spider2-eth0-17
colocation dmz-gw-with-spider2-eth0-18 inf: dmz-gw spider2-eth0-18
colocation dmz-gw-with-spider2-eth0-19 inf: dmz-gw spider2-eth0-19
colocation dmz-gw-with-spider2-eth0-2 inf: dmz-gw spider2-eth0-2
colocation dmz-gw-with-spider2-eth0-20 inf: dmz-gw spider2-eth0-20
colocation dmz-gw-with-spider2-eth0-21 inf: dmz-gw spider2-eth0-21
colocation dmz-gw-with-spider2-eth0-22 inf: dmz-gw spider2-eth0-22
colocation dmz-gw-with-spider2-eth0-23 inf: dmz-gw spider2-eth0-23
colocation dmz-gw-with-spider2-eth0-24 inf: dmz-gw spider2-eth0-24
colocation dmz-gw-with-spider2-eth0-25 inf: dmz-gw spider2-eth0-25
colocation dmz-gw-with-spider2-eth0-26 inf: dmz-gw spider2-eth0-26
colocation dmz-gw-with-spider2-eth0-27 inf: dmz-gw spider2-eth0-27
colocation dmz-gw-with-spider2-eth0-28 inf: dmz-gw spider2-eth0-28
colocation dmz-gw-with-spider2-eth0-29 inf: dmz-gw spider2-eth0-29
colocation dmz-gw-with-spider2-eth0-3 inf: dmz-gw spider2-eth0-3
colocation dmz-gw-with-spider2-eth0-30 inf: dmz-gw spider2-eth0-30
colocation dmz-gw-with-spider2-eth0-31 inf: dmz-gw spider2-eth0-31
colocation dmz-gw-with-spider2-eth0-32 inf: dmz-gw spider2-eth0-32
colocation dmz-gw-with-spider2-eth0-33 inf: dmz-gw spider2-eth0-33
colocation dmz-gw-with-spider2-eth0-34 inf: dmz-gw spider2-eth0-34
colocation dmz-gw-with-spider2-eth0-35 inf: dmz-gw spider2-eth0-35
colocation dmz-gw-with-spider2-eth0-36 inf: dmz-gw spider2-eth0-36
colocation dmz-gw-with-spider2-eth0-37 inf: dmz-gw spider2-eth0-37
colocation dmz-gw-with-spider2-eth0-38 inf: dmz-gw spider2-eth0-38
colocation dmz-gw-with-spider2-eth0-39 inf: dmz-gw spider2-eth0-39
colocation dmz-gw-with-spider2-eth0-4 inf: dmz-gw spider2-eth0-4
colocation dmz-gw-with-spider2-eth0-40 inf: dmz-gw spider2-eth0-40
colocation dmz-gw-with-spider2-eth0-41 inf: dmz-gw spider2-eth0-41
colocation dmz-gw-with-spider2-eth0-5 inf: dmz-gw spider2-eth0-5
colocation dmz-gw-with-spider2-eth0-6 inf: dmz-gw spider2-eth0-6
colocation dmz-gw-with-spider2-eth0-7 inf: dmz-gw spider2-eth0-7
colocation dmz-gw-with-spider2-eth0-8 inf: dmz-gw spider2-eth0-8
colocation dmz-gw-with-spider2-eth0-9 inf: dmz-gw spider2-eth0-9
colocation dmz-gw-with-sugar inf: dmz-gw sugar
colocation dmz-gw-with-webmail inf: dmz-gw webmail
colocation dmz-gw-with-wiki inf: dmz-gw wiki
colocation shorewall-with-actonomy inf: shorewall actonomy
colocation shorewall-with-corpsites inf: shorewall corpsites
colocation shorewall-with-datapass1-ssl-eth0-2 inf: shorewall datapass1-ssl-eth0-2
colocation shorewall-with-datapass2-ssl-eth0 inf: shorewall datapass2-ssl-eth0
colocation shorewall-with-datapass2-ssl-eth0-1 inf: shorewall datapass2-ssl-eth0-1
colocation shorewall-with-datapass2-ssl-eth0-2 inf: shorewall datapass2-ssl-eth0-2
colocation shorewall-with-dbrw inf: shorewall dbrw
colocation shorewall-with-dmz-gw inf: shorewall dmz-gw
colocation shorewall-with-mjhdev inf: shorewall mjhdev
colocation shorewall-with-mjhwiki inf: shorewall mjhwiki
colocation shorewall-with-mx1 inf: shorewall mx1
colocation shorewall-with-pin1 inf: shorewall pin1
colocation shorewall-with-pin2 inf: shorewall pin2
colocation shorewall-with-pin3 inf: shorewall pin3
colocation shorewall-with-rabbit inf: shorewall rabbit
colocation shorewall-with-reza inf: shorewall reza
colocation shorewall-with-rrdev inf: shorewall rrdev
colocation shorewall-with-rrdev2 inf: shorewall rrdev2
colocation shorewall-with-spider2-eth0-1 inf: shorewall spider2-eth0-1
colocation shorewall-with-spider2-eth0-10 inf: shorewall spider2-eth0-10
colocation shorewall-with-spider2-eth0-11 inf: shorewall spider2-eth0-11
colocation shorewall-with-spider2-eth0-12 inf: shorewall spider2-eth0-12
colocation shorewall-with-spider2-eth0-13 inf: shorewall spider2-eth0-13
colocation shorewall-with-spider2-eth0-14 inf: shorewall spider2-eth0-14
colocation shorewall-with-spider2-eth0-15 inf: shorewall spider2-eth0-15
colocation shorewall-with-spider2-eth0-16 inf: shorewall spider2-eth0-16
colocation shorewall-with-spider2-eth0-17 inf: shorewall spider2-eth0-17
colocation shorewall-with-spider2-eth0-18 inf: shorewall spider2-eth0-18
colocation shorewall-with-spider2-eth0-19 inf: shorewall spider2-eth0-19
colocation shorewall-with-spider2-eth0-2 inf: shorewall spider2-eth0-2
colocation shorewall-with-spider2-eth0-20 inf: shorewall spider2-eth0-20
colocation shorewall-with-spider2-eth0-21 inf: shorewall spider2-eth0-21
colocation shorewall-with-spider2-eth0-22 inf: shorewall spider2-eth0-22
colocation shorewall-with-spider2-eth0-23 inf: shorewall spider2-eth0-23
colocation shorewall-with-spider2-eth0-24 inf: shorewall spider2-eth0-24
colocation shorewall-with-spider2-eth0-25 inf: shorewall spider2-eth0-25
colocation shorewall-with-spider2-eth0-26 inf: shorewall spider2-eth0-26
colocation shorewall-with-spider2-eth0-27 inf: shorewall spider2-eth0-27
colocation shorewall-with-spider2-eth0-28 inf: shorewall spider2-eth0-28
colocation shorewall-with-spider2-eth0-29 inf: shorewall spider2-eth0-29
colocation shorewall-with-spider2-eth0-3 inf: shorewall spider2-eth0-3
colocation shorewall-with-spider2-eth0-30 inf: shorewall spider2-eth0-30
colocation shorewall-with-spider2-eth0-31 inf: shorewall spider2-eth0-31
colocation shorewall-with-spider2-eth0-32 inf: shorewall spider2-eth0-32
colocation shorewall-with-spider2-eth0-33 inf: shorewall spider2-eth0-33
colocation shorewall-with-spider2-eth0-34 inf: shorewall spider2-eth0-34
colocation shorewall-with-spider2-eth0-35 inf: shorewall spider2-eth0-35
colocation shorewall-with-spider2-eth0-36 inf: shorewall spider2-eth0-36
colocation shorewall-with-spider2-eth0-37 inf: shorewall spider2-eth0-37
colocation shorewall-with-spider2-eth0-38 inf: shorewall spider2-eth0-38
colocation shorewall-with-spider2-eth0-39 inf: shorewall spider2-eth0-39
colocation shorewall-with-spider2-eth0-4 inf: shorewall spider2-eth0-4
colocation shorewall-with-spider2-eth0-40 inf: shorewall spider2-eth0-40
colocation shorewall-with-spider2-eth0-41 inf: shorewall spider2-eth0-41
colocation shorewall-with-spider2-eth0-5 inf: shorewall spider2-eth0-5
colocation shorewall-with-spider2-eth0-6 inf: shorewall spider2-eth0-6
colocation shorewall-with-spider2-eth0-7 inf: shorewall spider2-eth0-7
colocation shorewall-with-spider2-eth0-8 inf: shorewall spider2-eth0-8
colocation shorewall-with-spider2-eth0-9 inf: shorewall spider2-eth0-9
colocation shorewall-with-sugar inf: shorewall sugar
colocation shorewall-with-webmail inf: shorewall webmail
colocation shorewall-with-webmail-resumepromotion inf: shorewall webmail-resumepromotion
colocation shorewall-with-wiki inf: shorewall wiki
order shorewall-after-actonomy inf: actonomy shorewall
order shorewall-after-corpsites inf: corpsites shorewall
order shorewall-after-datapass1-ssl-eth0-2 inf: datapass1-ssl-eth0-2 shorewall
order shorewall-after-datapass2-ssl-eth0 inf: datapass2-ssl-eth0 shorewall
order shorewall-after-datapass2-ssl-eth0-1 inf: datapass2-ssl-eth0-1 shorewall
order shorewall-after-datapass2-ssl-eth0-2 inf: datapass2-ssl-eth0-2 shorewall
order shorewall-after-dbrw inf: sugar dbrw
order shorewall-after-dmz-gw inf: dmz-gw shorewall
order shorewall-after-mjhdev inf: mjhdev shorewall
order shorewall-after-mjhwiki inf: mjhwiki shorewall
order shorewall-after-mx1 inf: mx1 shorewall
order shorewall-after-pin1 inf: pin1 shorewall
order shorewall-after-pin2 inf: pin2 shorewall
order shorewall-after-pin3 inf: pin3 shorewall
order shorewall-after-rabbit inf: rabbit shorewall
order shorewall-after-reza inf: reza shorewall
order shorewall-after-rrdev inf: rrdev shorewall
order shorewall-after-rrdev2 inf: rrdev2 shorewall
order shorewall-after-spider2-eth0-1 inf: spider2-eth0-1 shorewall
order shorewall-after-spider2-eth0-10 inf: spider2-eth0-10 shorewall
order shorewall-after-spider2-eth0-11 inf: spider2-eth0-11 shorewall
order shorewall-after-spider2-eth0-12 inf: spider2-eth0-12 shorewall
order shorewall-after-spider2-eth0-13 inf: spider2-eth0-13 shorewall
order shorewall-after-spider2-eth0-14 inf: spider2-eth0-14 shorewall
order shorewall-after-spider2-eth0-15 inf: spider2-eth0-15 shorewall
order shorewall-after-spider2-eth0-16 inf: spider2-eth0-16 shorewall
order shorewall-after-spider2-eth0-17 inf: spider2-eth0-17 shorewall
order shorewall-after-spider2-eth0-18 inf: spider2-eth0-18 shorewall
order shorewall-after-spider2-eth0-19 inf: spider2-eth0-19 shorewall
order shorewall-after-spider2-eth0-2 inf: spider2-eth0-2 shorewall
order shorewall-after-spider2-eth0-20 inf: spider2-eth0-20 shorewall
order shorewall-after-spider2-eth0-21 inf: spider2-eth0-21 shorewall
order shorewall-after-spider2-eth0-22 inf: spider2-eth0-22 shorewall
order shorewall-after-spider2-eth0-23 inf: spider2-eth0-23 shorewall
order shorewall-after-spider2-eth0-24 inf: spider2-eth0-24 shorewall
order shorewall-after-spider2-eth0-25 inf: spider2-eth0-25 shorewall
order shorewall-after-spider2-eth0-26 inf: spider2-eth0-26 shorewall
order shorewall-after-spider2-eth0-27 inf: spider2-eth0-27 shorewall
order shorewall-after-spider2-eth0-28 inf: spider2-eth0-28 shorewall
order shorewall-after-spider2-eth0-29 inf: spider2-eth0-29 shorewall
order shorewall-after-spider2-eth0-3 inf: spider2-eth0-3 shorewall
order shorewall-after-spider2-eth0-30 inf: spider2-eth0-30 shorewall
order shorewall-after-spider2-eth0-31 inf: spider2-eth0-31 shorewall
order shorewall-after-spider2-eth0-32 inf: spider2-eth0-32 shorewall
order shorewall-after-spider2-eth0-33 inf: spider2-eth0-33 shorewall
order shorewall-after-spider2-eth0-34 inf: spider2-eth0-34 shorewall
order shorewall-after-spider2-eth0-35 inf: spider2-eth0-35 shorewall
order shorewall-after-spider2-eth0-36 inf: spider2-eth0-36 shorewall
order shorewall-after-spider2-eth0-37 inf: spider2-eth0-37 shorewall
order shorewall-after-spider2-eth0-38 inf: spider2-eth0-38 shorewall
order shorewall-after-spider2-eth0-39 inf: spider2-eth0-39 shorewall
order shorewall-after-spider2-eth0-4 inf: spider2-eth0-4 shorewall
order shorewall-after-spider2-eth0-40 inf: spider2-eth0-40 shorewall
order shorewall-after-spider2-eth0-41 inf: spider2-eth0-41 shorewall
order shorewall-after-spider2-eth0-5 inf: spider2-eth0-5 shorewall
order shorewall-after-spider2-eth0-6 inf: spider2-eth0-6 shorewall
order shorewall-after-spider2-eth0-7 inf: spider2-eth0-7 shorewall
order shorewall-after-spider2-eth0-8 inf: spider2-eth0-8 shorewall
order shorewall-after-spider2-eth0-9 inf: spider2-eth0-9 shorewall
order shorewall-after-sugar inf: sugar shorewall
order shorewall-after-webmail inf: webmail shorewall
order shorewall-after-webmail-resumepromotion inf: webmail-resumepromotion shorewall
order shorewall-after-wiki inf: wiki shorewall
property $id="cib-bootstrap-options" \
dc-version="1.1.10-14.el6-368c726" \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes="2" \
stonith-enabled="false" \
last-lrm-refresh="1388199388" \
no-quorum-policy="ignore" \
maintenance-mode="false"
rsc_defaults $id="rsc-options" \
resource-stickiness="100"
--
Tracy Reed
First, thanks in advance for any help anyone may provide. I've been battling
this problem off and on for months and it is driving me mad:
Once every week or two my cluster fails. For reasons unknown it seems to
initiate a failover and then the shorewall service (lsb) does not get started
(or is stopped). The majority of the time it happens just after 4am. Although
it has happened at other times, although much less frequently. Tonight I am
going to have to be up at 4am to poke around on the cluster and observe what is
happening, if anything.
One theory is some sort of resource starvation such as CPU but I've stress
tested it and run a backup and a big file copy through the firewall at the same
time and never get more than 1 core of cpu (almost all due to the backup) out
of 4 utilized and nothing interesting happening to pacemaker/resources.
My setup is a bit complicated in that I have 63 IPaddr2 resources plus the
shorewall resource. Plus order and colocation rules to make sure it all sticks
together the IPs come up before shorewall.
I am running the latest RHEL/CentOS RPMs in CentOS 6.5:
[root@new-fw1 shorewall]# rpm -qa |grep -i corosync
corosync-1.4.1-17.el6.x86_64
corosynclib-1.4.1-17.el6.x86_64
[root@new-fw1 shorewall]# rpm -qa |grep -i pacemaker
pacemaker-1.1.10-14.el6.x86_64
pacemaker-libs-1.1.10-14.el6.x86_64
pacemaker-cluster-libs-1.1.10-14.el6.x86_64
pacemaker-cli-1.1.10-14.el6.x86_64
I am a little concerned about how pacemaker manages the shorewall resource. It
usually fails to bring up shorewall after a failover event. Shorewall could
fail to start if the IP addresses shorewall is expecting to be on the
interfaces are not there yet. But I have dependencies to prevent this from ever
happening such as:
order shorewall-after-dmz-gw inf: dmz-gw shorewall
I also wonder if the shorewall init script is properly LSB compatible. It
wasn't out of the box and I had to make a minor change. But now it does seem to
be LSB compatible:
[root@new-fw2 ~]# /etc/init.d/shorewall status ; echo "result: $?"
Shorewall-4.5.0.1 Status at new-fw2.mydomain.com - Fri Dec 27 16:57:14 PST 2013
Shorewall is running
State:Started (Fri Dec 27 04:11:14 PST 2013) from /etc/shorewall/
result: 0
[root@new-fw2 ~]# /etc/init.d/shorewall stop ; echo "result: $?"
Shutting down shorewall: [ OK ]
result: 0
[root@new-fw2 ~]# /etc/init.d/shorewall status ; echo "result: $?"
Shorewall-4.5.0.1 Status at new-fw2.mydomain.com - Fri Dec 27 16:57:48 PST 2013
Shorewall is stopped
State:Stopped (Fri Dec 27 16:57:47 PST 2013)
result: 3
[root@new-fw2 ~]# /etc/init.d/shorewall start ; echo "result: $?"
Starting shorewall: Shorewall is already running
[ OK ]
result: 0
[root@new-fw2 ~]# /etc/init.d/shorewall status ; echo "result: $?"
Shorewall-4.5.0.1 Status at new-fw2.mydomain.com - Fri Dec 27 16:58:04 PST 2013
Shorewall is running
State:Started (Fri Dec 27 16:57:53 PST 2013) from /etc/shorewall/
result: 0
So it shouldn't be an LSB issue at this point...
I have a very hard time making heads or tails of the
/var/log/cluster/corosync.log log files. For example, I just had this appear in the log files:
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave spider2-eth0-40 (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave spider2-eth0-41 (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave corpsites (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave dbrw (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave mjhdev (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave datapass1-ssl-eth0-2 (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave datapass2-ssl-eth0-2 (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave datapass2-ssl-eth0-1 (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave datapass2-ssl-eth0 (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave rrdev2 (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: info: LogActions: Leave webmail-resumepromotion (Started new-fw2.mydomain.com)
Dec 27 19:56:31 [1553] new-fw1.mydomain.com crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Dec 27 19:56:31 [1553] new-fw1.mydomain.com crmd: info: do_te_invoke: Processing graph 599 (ref=pe_calc-dc-1388202991-1545) derived from /var/lib/pacemaker/pengine/pe-input-894.bz2
Dec 27 19:56:31 [1553] new-fw1.mydomain.com crmd: notice: run_graph: Transition 599 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-894.bz2): Complete
Dec 27 19:56:31 [1553] new-fw1.mydomain.com crmd: info: do_log: FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
Dec 27 19:56:31 [1553] new-fw1.mydomain.com crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 27 19:56:31 [1551] new-fw1.mydomain.com pengine: notice: process_pe_message: Calculated Transition 599: /var/lib/pacemaker/pengine/pe-input-894.bz2
Is this normal? What does "Leave" mean? This is on the inactive node. The
active node doesn't seem to be making any log entries at all.
My corosync.conf and full crm config are as follows. There has also been
concern expressed about the size of our crm config. Is this unreasonable? I am
going to attempt to attach the 120k bzipped logfile to this email but odds are
the list won't take attachments. If it doesn't make it and anyone wants to see
it I will figure out somewhere else to stash it.
The logs are so voluminous that it is hard to know which parts are necessary.
corosync.conf:
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {
version: 2
secauth: off
threads: 0
interface {
ringnumber: 0
bindnetaddr: 10.0.2.0
mcastaddr: 226.94.1.1
mcastport: 5000
member {
memberaddr: 10.0.2.83
}
member {
memberaddr: 10.0.2.84
}
}
transport udpu
}
logging {
fileline: off
to_stderr: no
to_logfile: yes
to_syslog: yes
logfile: /var/log/cluster/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
amf {
mode: disabled
}
crm configure show:
node new-fw1.mydomain.com \
attributes standby="off"
node new-fw2.mydomain.com \
attributes standby="off"
primitive actonomy ocf:heartbeat:IPaddr2 \
params ip="206.71.189.165" nic="eth1" cidr_netmask="25" \
op monitor interval="30s" \
meta is-managed="true"
primitive corpsites ocf:heartbeat:IPaddr2 \
params ip="206.71.189.150" nic="eth1" cidr_netmask="25" \
op monitor interval="30s" \
meta target-role="Started"
primitive datapass1-ssl-eth0-2 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.166" nic="eth1" cidr_netmask="25" \
op monitor interval="30s"
primitive datapass2-ssl-eth0 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.161" nic="eth1" cidr_netmask="25" \
op monitor interval="30s"
primitive datapass2-ssl-eth0-1 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.162" nic="eth1" cidr_netmask="25" \
op monitor interval="30s"
primitive datapass2-ssl-eth0-2 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.167" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive dbrw ocf:heartbeat:IPaddr2 \
params ip="206.71.189.250" nic="eth1" cidr_netmask="32" \
op monitor interval="30s" \
meta target-role="Started"
primitive dmz-gw ocf:heartbeat:IPaddr2 \
params ip="10.0.2.254" cidr_netmask="32" \
op monitor interval="30s"
primitive mjhdev ocf:heartbeat:IPaddr2 \
params ip="209.216.236.138" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive mjhwiki ocf:heartbeat:IPaddr2 \
params ip="206.71.189.163" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive mx1 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.65" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive pin1 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.147" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive pin2 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.146" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive pin3 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.145" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive rabbit ocf:heartbeat:IPaddr2 \
params ip="206.71.189.149" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive reza ocf:heartbeat:IPaddr2 \
params ip="216.240.176.66" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive rrdev ocf:heartbeat:IPaddr2 \
params ip="206.71.189.164" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive rrdev2 ocf:heartbeat:IPaddr2 \
params ip="206.71.189.240" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
primitive shorewall lsb:shorewall \
op monitor interval="60s" \
meta target-role="Started" is-managed="true"
primitive spider2-eth0-1 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.67" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-10 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.76" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-11 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.77" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-12 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.78" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-13 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.79" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-14 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.80" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-15 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.81" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-16 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.82" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-17 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.83" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-18 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.84" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-19 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.85" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-2 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.68" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-20 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.86" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-21 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.87" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-22 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.88" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-23 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.89" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-24 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.90" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-25 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.91" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-26 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.92" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-27 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.93" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-28 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.94" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-29 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.95" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-3 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.69" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-30 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.96" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-31 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.97" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-32 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.98" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-33 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.99" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-34 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.100" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-35 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.101" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-36 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.102" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-37 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.103" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-38 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.104" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-39 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.105" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-4 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.70" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-40 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.106" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-41 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.107" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-5 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.71" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-6 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.72" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-7 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.73" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-8 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.74" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive spider2-eth0-9 ocf:heartbeat:IPaddr2 \
params ip="216.240.176.75" nic="eth2" cidr_netmask="32" \
op monitor interval="30s"
primitive sugar ocf:heartbeat:IPaddr2 \
params ip="206.71.189.130" nic="eth1" cidr_netmask="25" \
op monitor interval="30s"
primitive webmail ocf:heartbeat:IPaddr2 \
params ip="206.71.189.254" nic="eth1" cidr_netmask="25" \
op monitor interval="30s"
primitive webmail-resumepromotion ocf:heartbeat:IPaddr2 \
params ip="206.71.189.242" nic="eth1" cidr_netmask="25" \
op monitor interval="30s"
primitive wiki ocf:heartbeat:IPaddr2 \
params ip="209.216.236.135" nic="eth1" cidr_netmask="32" \
op monitor interval="30s"
colocation dmz-gw-with-actonomy inf: dmz-gw actonomy
colocation dmz-gw-with-mjhwiki inf: dmz-gw mjhwiki
colocation dmz-gw-with-mx1 inf: dmz-gw mx1
colocation dmz-gw-with-pin1 inf: dmz-gw pin1
colocation dmz-gw-with-pin2 inf: dmz-gw pin2
colocation dmz-gw-with-pin3 inf: dmz-gw pin3
colocation dmz-gw-with-rabbit inf: dmz-gw rabbit
colocation dmz-gw-with-reza inf: dmz-gw reza
colocation dmz-gw-with-rrdev inf: dmz-gw rrdev
colocation dmz-gw-with-spider2-eth0-1 inf: dmz-gw spider2-eth0-1
colocation dmz-gw-with-spider2-eth0-10 inf: dmz-gw spider2-eth0-10
colocation dmz-gw-with-spider2-eth0-11 inf: dmz-gw spider2-eth0-11
colocation dmz-gw-with-spider2-eth0-12 inf: dmz-gw spider2-eth0-12
colocation dmz-gw-with-spider2-eth0-13 inf: dmz-gw spider2-eth0-13
colocation dmz-gw-with-spider2-eth0-14 inf: dmz-gw spider2-eth0-14
colocation dmz-gw-with-spider2-eth0-15 inf: dmz-gw spider2-eth0-15
colocation dmz-gw-with-spider2-eth0-16 inf: dmz-gw spider2-eth0-16
colocation dmz-gw-with-spider2-eth0-17 inf: dmz-gw spider2-eth0-17
colocation dmz-gw-with-spider2-eth0-18 inf: dmz-gw spider2-eth0-18
colocation dmz-gw-with-spider2-eth0-19 inf: dmz-gw spider2-eth0-19
colocation dmz-gw-with-spider2-eth0-2 inf: dmz-gw spider2-eth0-2
colocation dmz-gw-with-spider2-eth0-20 inf: dmz-gw spider2-eth0-20
colocation dmz-gw-with-spider2-eth0-21 inf: dmz-gw spider2-eth0-21
colocation dmz-gw-with-spider2-eth0-22 inf: dmz-gw spider2-eth0-22
colocation dmz-gw-with-spider2-eth0-23 inf: dmz-gw spider2-eth0-23
colocation dmz-gw-with-spider2-eth0-24 inf: dmz-gw spider2-eth0-24
colocation dmz-gw-with-spider2-eth0-25 inf: dmz-gw spider2-eth0-25
colocation dmz-gw-with-spider2-eth0-26 inf: dmz-gw spider2-eth0-26
colocation dmz-gw-with-spider2-eth0-27 inf: dmz-gw spider2-eth0-27
colocation dmz-gw-with-spider2-eth0-28 inf: dmz-gw spider2-eth0-28
colocation dmz-gw-with-spider2-eth0-29 inf: dmz-gw spider2-eth0-29
colocation dmz-gw-with-spider2-eth0-3 inf: dmz-gw spider2-eth0-3
colocation dmz-gw-with-spider2-eth0-30 inf: dmz-gw spider2-eth0-30
colocation dmz-gw-with-spider2-eth0-31 inf: dmz-gw spider2-eth0-31
colocation dmz-gw-with-spider2-eth0-32 inf: dmz-gw spider2-eth0-32
colocation dmz-gw-with-spider2-eth0-33 inf: dmz-gw spider2-eth0-33
colocation dmz-gw-with-spider2-eth0-34 inf: dmz-gw spider2-eth0-34
colocation dmz-gw-with-spider2-eth0-35 inf: dmz-gw spider2-eth0-35
colocation dmz-gw-with-spider2-eth0-36 inf: dmz-gw spider2-eth0-36
colocation dmz-gw-with-spider2-eth0-37 inf: dmz-gw spider2-eth0-37
colocation dmz-gw-with-spider2-eth0-38 inf: dmz-gw spider2-eth0-38
colocation dmz-gw-with-spider2-eth0-39 inf: dmz-gw spider2-eth0-39
colocation dmz-gw-with-spider2-eth0-4 inf: dmz-gw spider2-eth0-4
colocation dmz-gw-with-spider2-eth0-40 inf: dmz-gw spider2-eth0-40
colocation dmz-gw-with-spider2-eth0-41 inf: dmz-gw spider2-eth0-41
colocation dmz-gw-with-spider2-eth0-5 inf: dmz-gw spider2-eth0-5
colocation dmz-gw-with-spider2-eth0-6 inf: dmz-gw spider2-eth0-6
colocation dmz-gw-with-spider2-eth0-7 inf: dmz-gw spider2-eth0-7
colocation dmz-gw-with-spider2-eth0-8 inf: dmz-gw spider2-eth0-8
colocation dmz-gw-with-spider2-eth0-9 inf: dmz-gw spider2-eth0-9
colocation dmz-gw-with-sugar inf: dmz-gw sugar
colocation dmz-gw-with-webmail inf: dmz-gw webmail
colocation dmz-gw-with-wiki inf: dmz-gw wiki
colocation shorewall-with-actonomy inf: shorewall actonomy
colocation shorewall-with-corpsites inf: shorewall corpsites
colocation shorewall-with-datapass1-ssl-eth0-2 inf: shorewall datapass1-ssl-eth0-2
colocation shorewall-with-datapass2-ssl-eth0 inf: shorewall datapass2-ssl-eth0
colocation shorewall-with-datapass2-ssl-eth0-1 inf: shorewall datapass2-ssl-eth0-1
colocation shorewall-with-datapass2-ssl-eth0-2 inf: shorewall datapass2-ssl-eth0-2
colocation shorewall-with-dbrw inf: shorewall dbrw
colocation shorewall-with-dmz-gw inf: shorewall dmz-gw
colocation shorewall-with-mjhdev inf: shorewall mjhdev
colocation shorewall-with-mjhwiki inf: shorewall mjhwiki
colocation shorewall-with-mx1 inf: shorewall mx1
colocation shorewall-with-pin1 inf: shorewall pin1
colocation shorewall-with-pin2 inf: shorewall pin2
colocation shorewall-with-pin3 inf: shorewall pin3
colocation shorewall-with-rabbit inf: shorewall rabbit
colocation shorewall-with-reza inf: shorewall reza
colocation shorewall-with-rrdev inf: shorewall rrdev
colocation shorewall-with-rrdev2 inf: shorewall rrdev2
colocation shorewall-with-spider2-eth0-1 inf: shorewall spider2-eth0-1
colocation shorewall-with-spider2-eth0-10 inf: shorewall spider2-eth0-10
colocation shorewall-with-spider2-eth0-11 inf: shorewall spider2-eth0-11
colocation shorewall-with-spider2-eth0-12 inf: shorewall spider2-eth0-12
colocation shorewall-with-spider2-eth0-13 inf: shorewall spider2-eth0-13
colocation shorewall-with-spider2-eth0-14 inf: shorewall spider2-eth0-14
colocation shorewall-with-spider2-eth0-15 inf: shorewall spider2-eth0-15
colocation shorewall-with-spider2-eth0-16 inf: shorewall spider2-eth0-16
colocation shorewall-with-spider2-eth0-17 inf: shorewall spider2-eth0-17
colocation shorewall-with-spider2-eth0-18 inf: shorewall spider2-eth0-18
colocation shorewall-with-spider2-eth0-19 inf: shorewall spider2-eth0-19
colocation shorewall-with-spider2-eth0-2 inf: shorewall spider2-eth0-2
colocation shorewall-with-spider2-eth0-20 inf: shorewall spider2-eth0-20
colocation shorewall-with-spider2-eth0-21 inf: shorewall spider2-eth0-21
colocation shorewall-with-spider2-eth0-22 inf: shorewall spider2-eth0-22
colocation shorewall-with-spider2-eth0-23 inf: shorewall spider2-eth0-23
colocation shorewall-with-spider2-eth0-24 inf: shorewall spider2-eth0-24
colocation shorewall-with-spider2-eth0-25 inf: shorewall spider2-eth0-25
colocation shorewall-with-spider2-eth0-26 inf: shorewall spider2-eth0-26
colocation shorewall-with-spider2-eth0-27 inf: shorewall spider2-eth0-27
colocation shorewall-with-spider2-eth0-28 inf: shorewall spider2-eth0-28
colocation shorewall-with-spider2-eth0-29 inf: shorewall spider2-eth0-29
colocation shorewall-with-spider2-eth0-3 inf: shorewall spider2-eth0-3
colocation shorewall-with-spider2-eth0-30 inf: shorewall spider2-eth0-30
colocation shorewall-with-spider2-eth0-31 inf: shorewall spider2-eth0-31
colocation shorewall-with-spider2-eth0-32 inf: shorewall spider2-eth0-32
colocation shorewall-with-spider2-eth0-33 inf: shorewall spider2-eth0-33
colocation shorewall-with-spider2-eth0-34 inf: shorewall spider2-eth0-34
colocation shorewall-with-spider2-eth0-35 inf: shorewall spider2-eth0-35
colocation shorewall-with-spider2-eth0-36 inf: shorewall spider2-eth0-36
colocation shorewall-with-spider2-eth0-37 inf: shorewall spider2-eth0-37
colocation shorewall-with-spider2-eth0-38 inf: shorewall spider2-eth0-38
colocation shorewall-with-spider2-eth0-39 inf: shorewall spider2-eth0-39
colocation shorewall-with-spider2-eth0-4 inf: shorewall spider2-eth0-4
colocation shorewall-with-spider2-eth0-40 inf: shorewall spider2-eth0-40
colocation shorewall-with-spider2-eth0-41 inf: shorewall spider2-eth0-41
colocation shorewall-with-spider2-eth0-5 inf: shorewall spider2-eth0-5
colocation shorewall-with-spider2-eth0-6 inf: shorewall spider2-eth0-6
colocation shorewall-with-spider2-eth0-7 inf: shorewall spider2-eth0-7
colocation shorewall-with-spider2-eth0-8 inf: shorewall spider2-eth0-8
colocation shorewall-with-spider2-eth0-9 inf: shorewall spider2-eth0-9
colocation shorewall-with-sugar inf: shorewall sugar
colocation shorewall-with-webmail inf: shorewall webmail
colocation shorewall-with-webmail-resumepromotion inf: shorewall webmail-resumepromotion
colocation shorewall-with-wiki inf: shorewall wiki
order shorewall-after-actonomy inf: actonomy shorewall
order shorewall-after-corpsites inf: corpsites shorewall
order shorewall-after-datapass1-ssl-eth0-2 inf: datapass1-ssl-eth0-2 shorewall
order shorewall-after-datapass2-ssl-eth0 inf: datapass2-ssl-eth0 shorewall
order shorewall-after-datapass2-ssl-eth0-1 inf: datapass2-ssl-eth0-1 shorewall
order shorewall-after-datapass2-ssl-eth0-2 inf: datapass2-ssl-eth0-2 shorewall
order shorewall-after-dbrw inf: sugar dbrw
order shorewall-after-dmz-gw inf: dmz-gw shorewall
order shorewall-after-mjhdev inf: mjhdev shorewall
order shorewall-after-mjhwiki inf: mjhwiki shorewall
order shorewall-after-mx1 inf: mx1 shorewall
order shorewall-after-pin1 inf: pin1 shorewall
order shorewall-after-pin2 inf: pin2 shorewall
order shorewall-after-pin3 inf: pin3 shorewall
order shorewall-after-rabbit inf: rabbit shorewall
order shorewall-after-reza inf: reza shorewall
order shorewall-after-rrdev inf: rrdev shorewall
order shorewall-after-rrdev2 inf: rrdev2 shorewall
order shorewall-after-spider2-eth0-1 inf: spider2-eth0-1 shorewall
order shorewall-after-spider2-eth0-10 inf: spider2-eth0-10 shorewall
order shorewall-after-spider2-eth0-11 inf: spider2-eth0-11 shorewall
order shorewall-after-spider2-eth0-12 inf: spider2-eth0-12 shorewall
order shorewall-after-spider2-eth0-13 inf: spider2-eth0-13 shorewall
order shorewall-after-spider2-eth0-14 inf: spider2-eth0-14 shorewall
order shorewall-after-spider2-eth0-15 inf: spider2-eth0-15 shorewall
order shorewall-after-spider2-eth0-16 inf: spider2-eth0-16 shorewall
order shorewall-after-spider2-eth0-17 inf: spider2-eth0-17 shorewall
order shorewall-after-spider2-eth0-18 inf: spider2-eth0-18 shorewall
order shorewall-after-spider2-eth0-19 inf: spider2-eth0-19 shorewall
order shorewall-after-spider2-eth0-2 inf: spider2-eth0-2 shorewall
order shorewall-after-spider2-eth0-20 inf: spider2-eth0-20 shorewall
order shorewall-after-spider2-eth0-21 inf: spider2-eth0-21 shorewall
order shorewall-after-spider2-eth0-22 inf: spider2-eth0-22 shorewall
order shorewall-after-spider2-eth0-23 inf: spider2-eth0-23 shorewall
order shorewall-after-spider2-eth0-24 inf: spider2-eth0-24 shorewall
order shorewall-after-spider2-eth0-25 inf: spider2-eth0-25 shorewall
order shorewall-after-spider2-eth0-26 inf: spider2-eth0-26 shorewall
order shorewall-after-spider2-eth0-27 inf: spider2-eth0-27 shorewall
order shorewall-after-spider2-eth0-28 inf: spider2-eth0-28 shorewall
order shorewall-after-spider2-eth0-29 inf: spider2-eth0-29 shorewall
order shorewall-after-spider2-eth0-3 inf: spider2-eth0-3 shorewall
order shorewall-after-spider2-eth0-30 inf: spider2-eth0-30 shorewall
order shorewall-after-spider2-eth0-31 inf: spider2-eth0-31 shorewall
order shorewall-after-spider2-eth0-32 inf: spider2-eth0-32 shorewall
order shorewall-after-spider2-eth0-33 inf: spider2-eth0-33 shorewall
order shorewall-after-spider2-eth0-34 inf: spider2-eth0-34 shorewall
order shorewall-after-spider2-eth0-35 inf: spider2-eth0-35 shorewall
order shorewall-after-spider2-eth0-36 inf: spider2-eth0-36 shorewall
order shorewall-after-spider2-eth0-37 inf: spider2-eth0-37 shorewall
order shorewall-after-spider2-eth0-38 inf: spider2-eth0-38 shorewall
order shorewall-after-spider2-eth0-39 inf: spider2-eth0-39 shorewall
order shorewall-after-spider2-eth0-4 inf: spider2-eth0-4 shorewall
order shorewall-after-spider2-eth0-40 inf: spider2-eth0-40 shorewall
order shorewall-after-spider2-eth0-41 inf: spider2-eth0-41 shorewall
order shorewall-after-spider2-eth0-5 inf: spider2-eth0-5 shorewall
order shorewall-after-spider2-eth0-6 inf: spider2-eth0-6 shorewall
order shorewall-after-spider2-eth0-7 inf: spider2-eth0-7 shorewall
order shorewall-after-spider2-eth0-8 inf: spider2-eth0-8 shorewall
order shorewall-after-spider2-eth0-9 inf: spider2-eth0-9 shorewall
order shorewall-after-sugar inf: sugar shorewall
order shorewall-after-webmail inf: webmail shorewall
order shorewall-after-webmail-resumepromotion inf: webmail-resumepromotion shorewall
order shorewall-after-wiki inf: wiki shorewall
property $id="cib-bootstrap-options" \
dc-version="1.1.10-14.el6-368c726" \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes="2" \
stonith-enabled="false" \
last-lrm-refresh="1388199388" \
no-quorum-policy="ignore" \
maintenance-mode="false"
rsc_defaults $id="rsc-options" \
resource-stickiness="100"
--
Tracy Reed