The Linux and Unix Menagerie: heartbeat

Hey There,

Today's post is going to steer away from the Linux and/or Unix Operating Systems just slightly, and look at a problem a lot of folks run into, but have problems diagnosing, when they first set up a Veritas cluster.

Our only assumptions for this post are that Veritas Cluster Server is installed correctly on a two-node farm, everything is set up to failover and switch correctly in the software and no useful information can be obtained via the standard Veritas status commands (or, in other words, the software thinks everything's fine, yet it's reporting that it's not working correctly ;)

Generally, with issues like this one (the software being unable to diagnose its own condition), the best place to start is at the lowest level. So, we'll add the fact that the physical network cabling and connections have been checked to our list of assumptions.

Our next step would be to take a look at the next layer up on the protocol stack, which would be the LLT (low latency transport protocol) layer (which, coincidentally, shares the same level as the MAC, so you may see it referred to, elsewhere, as MAC/LLT, or just MAC, when LLT is actually meant!) This is the base layer at which Veritas controls how it sends its heartbeat signals.

The layer-2 LLT protocol is most commonly associated with the DLPI (all these initials... man. These stand for the Data Link Provider Interface). Which brings us around to the point of this post ;)

Veritas Cluster Server comes with a utility called "dlpiping" that will specifically test device-to-device (basically NIC-to-NIC or MAC-to-MAC) communication at the LLT layer. Note that if you can't find the dlpiping command, it comes standard as a component in the VRTSllt package and is generally placed in /opt/VRTSllt/ by default. If you want to use it without having to type the entire command, you can just add that directory to your PATH environment variable by typing:

host # PATH=$PATH:/opt/VRTSllt;export PATH

In order to use dlpiping to troubleshoot this issue, you'll need to set up a dlpiping server on at least one node in the cluster. Since we only have two nodes in our imaginary cluster, having it on only one node should be perfect.

To set up the dlpiping server on either node, type the following at the command prompt (unless otherwise noted, all of these Veritas-specific commands are in /opt/VRTSllt and all system information returned, by way of example here, is intentionally bogus):

host # getmac /dev/ce:0 <--- This will give use the MAC address of the NIC we want to set the server up on (ce0, in this instance). For this command, even if your device is actually named ce0, eth0, etc, you need to specify it as "device:instance"
/dev/ce:0 00:00:00:FF:FF:FF

Next, you just have to start it up and configure it slightly, like so (Easy peasy; you're done :)

host # dlpiping -s /dev/ce:0

This command runs in the foreground by default. You can background it if you like, but once you start it running on whichever node you start it on, you're better off leaving that system alone so that anything else you do on it can't possibly affect the outcome of your tests. Since our pretend machine's cluster setup is completely down right now anyway, we'll just let it run in the foreground. You can stop the server, at any time, by simply typing a ctl-C:

^C
host #

Now, on every other server in the cluster, you'll need to run the dlpiping client. We only have one other server in our cluster, but you would, theoretically, repeat this process as many times as necessary; once for each client. Note, also, that for the dlpiping server and client setups, you should repeat the setup-and-test process for at least one NIC on every node in the cluster that forms a distinct heartbeat-chain. You can determine which NIC's these are by looking in the /etc/llttab file.

host # dlpiping -c /dev/ce:0 00:00:00:FF:FF:FF <--- This is the exact output from the getmac command we issued on the dlpiping server host.

If everything is okay with that connection, you'll see a response akin to a Solaris ping reply:

0:00:00:FF:FF:FF is alive

If something is wrong, the output is equally simple to decipher:

no response from 00:00:00:FF:FF:FF

Assuming everything is okay, and you still have problems, you should check out the support site for Veritas Cluster Server and see what they recommend you try next (most likely testing the IP layer functionality - ping! ;)

If things don't work out, and you get the error, that's great (assuming you're a glass-half-full kind of person ;) Getting an error at this layer of the stack greatly reduces the possible-root-cause pool and leaves you with only a few options that are worth looking into. And, since we've already verified physical cabling connectivity (no loose or poorly fitted ethernet cabling in any NIC) and traced the cable (so we know NICA-1 is going to NICB-1, as it should), you can be almost certain that the issue is with the quality or type of your ethernet cabling.

For instance, your cable may be physically damaged or improperly pinned-out (assuming you make your own cables and accidentally made a bad one - mass manufacturers make mistakes, too, though). Also, you may be using a standard ethernet cable, where a crossover (or, in some instances, rollover) cable is required. Of course, whenever you run into a seeming dead-end like this, double check your Veritas Cluster main.cf file to make sure that it's not in any way related to a slight error that you may have missed earlier on in the process.

In any event, you are now very close to your solution. You can opt to leave your dlpiping server running for as long as you want. To my knowledge it doesn't cause any latency issues that are noticeable (at least in clusters with a small number of nodes). Once you've done your testing, however, it's also completely useless unless you enjoy running that command a lot ;)

Cheers,

, Mike

Hey There,

Today, I've included a script I wrote to monitor a small SunCluster 3.1 environment (Running Oracle Parallel Server - OPS) we have set up at our shop. It basically runs through the output of various "scstat" command variations and reports on all combinations of errors it encounters. I built some functionality in it to be as specific as possible about the error states and to make sure that it returns an "All Clear" once the error condition no longer exists.

I also wrote this to run in cron. I run it every 5 minutes. Any time period is acceptable, depending upon what you need, and, of course, you could always put this script in a wrapper so that it runs constantly (although that generally necessitates writing another script to make sure that this script is running, restarting it as necessary and vice versa - For Veritas Cluster Server fans out there, this is a cheap and quick way to imitate the relationship between had and hashadow).

This was also written for a small environment (2 machines with 2 "live" network connections each). I didn't include monitoring of the heartbeats, since the script is meant to be run locally (with "exactly" the same values in the customizable section) on all nodes in a cluster, and scstat's indication of failure on any of these tests is, in and of itself, a guarantee that loss of a heartbeat connection is the very least of your problems ;)

Enjoy!

This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

###################################
# suncluster_mon - check cluster health
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
# simple parser for:
# 1. scstat -n
# 2. scstat -i
# 3. scstat -g
# 4. scstat -D
#
###################################
 
###################################
# CUSTOMIZE SETTINGS HERE
# $debug can be set with the -d
#  switch when invoking the script
###################################
$primary_node="";
$secondary_node="";
$primary_eth="";
$secondary_eth="";
$date=`/usr/bin/date "+%A, %B %d - %H:%M:%S"`;
$hostname=`/usr/bin/hostname`;
$debug=0;
###################################
# PLEASE DO NOT EDIT BELOW
# EXCEPT MAIL SETTINGS AT END
###################################

chomp($date);

if ( $#ARGV >= 0 ) {
 foreach $arg (@ARGV) {
  if ( $arg =~ /-d/ ) {
   $debug=1;
   print "Debug Output Set\n";
  } else {
   push(@nogo, $arg)
  }
 }
 if ( @nogo > 0 ) {
  print "Unrecognized options: @nogo\n";
  print "Ignoring and continuing\n";
 }
}

@scstatn=`/usr/cluster/bin/scstat -n|/usr/bin/egrep 'node:'`;
for $nodestat (@scstatn) {
 if ( $nodestat =~ /Cluster node:/ ) {
  $nodestat =~ s/Cluster node://;
  @nodestat = split(" ", $nodestat);
  if ( $nodestat[1] eq "Offline" && $nodestat[0] eq $secondary_node ) {
   if ( $nodefailure ) {
    $alert="\nCluster Alert:\n$primary_node and $secondary_node are both in Offline state.  Cluster Failed\n";
   } else {
    $alert="\nCluster Alert:\n$primary_node is in Offline state.  $secondary_node is Online - Cluster Crippled\n";
   }
   $alert="\nCluster Alert:\n$nodestat[0] has switched to $nodestat[1] Status - Cluster Crippled - Checking For Dual Failure\n";
   $nodefailure=1;
   push(@alert, $alert);
  } elsif ( $nodestat[1] eq "Offline" && $nodestat[0] eq $primary_node ) {
   if ( $nodefailure ) {
    $alert="\nCluster Alert:\n$primary_node and $secondary_node are both in Offline state.  Cluster Failed\n";
   } else {
    $alert="\nCluster Alert:\n$primary_node is in Offline state.  $secondary_node is Online - Cluster Crippled\n";
   }
   $alert="\nCluster Alert:\n$nodestat[0] has switched to $nodestat[1] Status - Cluster Crippled - Checking For Dual Failure\n";
   $nodefailure=1;
   push(@alert, $alert);
  } else {
   if ( $debug ) {
    $alert="Cluster Info: $nodestat[0] and $nodestat[1] OK\n";
    push(@debug, $alert);
   }
   $beautiful++;
  }
 } else {
  print "WTF ---- SCSTAT -D\n";
 }
}
if ( $beautiful == 2 ) {
  if ( $debug )   {
   $alert="Cluster Info: Both nodes OK\n";
   push(@debug, $alert);
  }
}

@scstati=`/usr/cluster/bin/scstat -i|/usr/bin/egrep 'Group:'`;
for $ipmpstat (@scstati) {
 if ( $ipmpstat =~ /IPMP Group:/ ) {
  $ipmpstat =~ s/IPMP Group://;
  @ipmpstat = split(" ", $ipmpstat);
  if ( $ipmpstat[0] == $primary_node ) {
   if ( $ipmpstat[3] eq $primary_eth && $ipmpstat[4] ne "Online" ) {
    $alert="\nIPMP Alert:\n$ipmpstat[3] is in $ipmpstat[4] mode on $ipmpstat[0] - Checking for Online of $ipmpstat[0] $secondary_eth\n";
    $ethprimaryfailure=1;
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Online" && $ethprimaryfailure == 1 ) {
    $alert="\nIPMP Alert:\n$ipmpstat[0] $secondary_eth is not in an Online state on $ipmpstat[0] - IPMP Group Down on both Interfaces!\n";
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Standby" && ! $ethprimaryfailure ) {
    $alert="\nIPMP Alert:\n$ipmpstat[3] is in $ipmpstat[4] mode on $ipmpstat[0] - Failover interface is not UP - IPMP Group Crippled on $ipmpstat[0]!\n";
    push(@alert, $alert);
   } else {
    if ( $debug )   {
     $alert="IPMP Info: IPMP Group $ipmpstat[0] $ipmpstat[3] OK\n";
     push(@debug, $alert);
    }
   }
  } elsif ( $ipmpstat[0] == $secondary_node ) {
   if ( $ipmpstat[3] eq $primary_eth && $ipmpstat[4] ne "Online" ) {
    $alert="\nIPMP Alert:\n$primary_eth is in $ipmpstat[4] mode on $ipmpstat[0] - Checking for Online of $ipmpstat[0] $secondary_eth\n";
    $ethprimaryfailure=1;
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Online" && $ethprimaryfailure == 1 ) {
    $alert="\nIPMP Alert:\n$ipmpstat[0] ipmpstat[3] not in an Online state on ipmpstat[0] - IPMP Group Down on both Interfaces!\n";
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Standby" && ! $ethprimaryfailure ) {
    $alert="\nIPMP Alert:\n$ipmpstat[3] is in $ipmpstat[4] mode on $ipmpstat[0] - Failover interface is not UP - IPMP Group Crippled on $ipmpstat[0]!\n";
    push(@alert, $alert);
   } else {
    if ( $debug )   {
     $alert="IPMP Info: IPMP Group $ipmpstat[0] $ipmpstat[3] OK\n";
     push(@debug, $alert);
    }
   }
  }
 } else {
  print "WTF ---- SCSTAT -I\n";
 }
}

@scstatg=`/usr/cluster/bin/scstat -g|/usr/bin/egrep 'Resource:|Resources:|Group:'`;
for $rgstat (@scstatg) {
 if ( $rgstat =~ /Resources:/ ) {
  $rgstat =~ s/Resources://;
  @rgstat = split(" ", $rgstat);
  if ( @rgstat != 5 ) {
   shift @rgstat;
   $alert="\nOracle Resource Alert:\nOracle resource group $rgstat[0] is not running all resources on the cluster\n--Only running -- @rgstat\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Oracle Resource Info: Oracle resource group $rgstat[0] OK -- @rgstat OK\n";
    push(@debug, $alert);
   }
  }
 } elsif ( $rgstat =~ /Group:/ ) {
  $rgstat =~ s/Group://;
  @rgstat = split(" ", $rgstat);
  if ( $rgstat[2] ne "Online" && $rgstat[1] eq $primary_node ) {
   $alert="\nOracle Resource Alert:\nResource Group $rgstat[0] is in $rgstat[2] state on primary node $rgstat[1] - Checking for failover\n";
   $rgprimaryfailure=1;
   push(@alert, $alert);
  } elsif ( $rgstat[2] ne "Online" && $rgstat[1] eq $secondary_node && $rgprimaryfailure == 1 ) {
   $alert="\nOracle Resource Alert:\nResource Group $rgstat[0] is in $rgstat[2] state on cluster - Resource Group Down on both Nodes!\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Oracle Resource Info: $rgstat[1] Resource Group $rgstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } elsif ( $rgstat =~ /Resource:/ ) {
  $rgstat =~ s/Resource://;
  @rgstat = split(" ", $rgstat);
  if ( $rgstat[2] ne "Online" && $rgstat[1] eq $primary_node ) {
   $alert="\nOracle Resource Alert:\nResource$rgstat[0] is in $rgstat[2] state on primary node $rgstat[1] - Checking for failover\n";
   $rgprimaryfailure=1;
   push(@alert, $alert);
  } elsif ( $rgstat[2] ne "Online" && $rgstat[1] eq $secondary_node && $rgprimaryfailure == 1 ) {
   $alert="\nOracle Resource Alert:\nResource $rgstat[0] is in $rgstat[2] state on cluster - Resource Group Down on both Nodes!\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Oracle Resource Info: $rgstat[1] Resource $rgstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } else {
  print "WTF ---- SCSTAT -G\n";
 }
}

@scstatD=`/usr/cluster/bin/scstat -D|/usr/bin/egrep 'servers|status'`;
for $diskstat (@scstatD) {
 if ( $diskstat =~ /servers/ ) {
  $diskstat =~ s/Device group servers://;
  @diskstat = split(" ", $diskstat);
  if ( $diskstat[1] ne $primary_node ) {
   $alert="\nDisk Resource Alert:\n$diskstat[0] has switched primary node to $diskstat[1] from $diskstat[2]\n";
   push(@alert, $alert);
  } elsif ( $diskstat[1] ne $primary_node && $diskstat[1] ne $secondary_node ) {
   $alert="\nDisk Resource Alert:\n$diskstat[0] has failed on all nodes!\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Disk Resource Info: $diskstat[1] primary - $diskstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } elsif ( $diskstat =~ /status/ ) {
  $diskstat =~ s/Device group status://;
  @diskstat = split(" ", $diskstat);
  if ( $diskstat[1] ne "Online" ) {
   $alert="\nDisk Resource Group Alert:\n$diskstat[0] has switched to $diskstat[1] state on the cluster\n";
   push(@alert, $alert);
  } elsif ( $diskstat[1] eq "Online" ) {
   if ( $debug) {
    $alert="Disk Resource Group Info: $diskstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } else {
  print "WTF ---- SCSTAT -D\n";
 }
}

###################################
# EDIT To: Reply-To: and From:
# if you want mail to go somewhere
# useful and-or helpful!
###################################

if ( @alert > 0 || @debug > 0 ) {
 open(CMAIL, "|/usr/lib/sendmail -t");
 print CMAIL "Subject: CLUSTER ALERT - $hostname - $date\n";
 print CMAIL "From: you\@yourdomain.com\n";
 print CMAIL "Reply-To: you\@yourdomain.com\n";
 print CMAIL "To: recipients\@yourdomain.com\n";
 print CMAIL "\n\n";
 foreach $warning (@alert) {
  print CMAIL $warning;
 }
 if ( $debug > 0 ) {
  foreach $message (@debug) {
   print CMAIL $message;
  }
 }
 close(CMAIL);
 system("touch /tmp/cfail_ihot_stat");
} elsif ( @alert == 0 && -f "/tmp/cfail_ihot_stat" ) {
 open(CMAIL, "|/usr/lib/sendmail -t");
 print CMAIL "Subject: CLUSTER RESTORED - $hostname - $date\n";
 print CMAIL "From: you\@yourdomain.com\n";
 print CMAIL "Reply-To: you\@yourdomain.com\n";
 print CMAIL "To: recipients\@yourdomain.com\n";
 print CMAIL "\n\n";
 print CMAIL "All Cluster Services Back To Good State\n";
 print CMAIL "All Cluster Nodes: OK\n";
 print CMAIL "All IPMP Groups: OK\n";
 print CMAIL "All Oracle Resources: OK\n";
 print CMAIL "All Storage Groups: OK\n";
 close(CMAIL);
 unlink("/tmp/cfail_ihot_stat");
}