SunCluster Quick Command Reference

Hey There,

Well, sitting back and reflecting today, I realized that we've done our fair share of posts on cheat-sheets and such (Our LVM command reference, for example). However, when it comes to clustering systems, we've probably been unfair in our bias toward Veritas Clustering as opposed to SunCluster, for which we've only put up, I believe, one post with our Simple SunCluster Monitoring Script (Thankfully linked back to from The SunCluster Wiki). What's a poor blog to do? ;)

With that in mind, today we're going to lay down a quick command reference for SunCluster. Hopefully you'll find it useful and enjoy it :) Of course, it assumes that you know, mostly, how to use SunCluster (and are familiar with basic clustering concepts), but it can be used as an attractive wallpaper, if nothing else ;)

All of this information, and much more, can be found at Sun's Online Documentation Center for SunCluster.

Enjoy,

Quorum Commands:

host # clquorum add device - Add a SCSI Quorum Device
host # clquorum add -t netapp_nas -p filer =nasdevicename,lun_id =IDnumdevice Nasdevice - Add a NAS Quorum Device
host # clquorum add -t quorumserver -p qshost =IPaddress, port =portnumber quorumservername - Add a Quorum Server
host # clquorum remove device - Remove a Quorum Device

Resource Type Commands:

host # clresourcetype register type - Register a Resource Type
host # clresourcetype unregister type - Remove a Resource Type

Resource Group Commands:

host # clresourcegroup create group - Create a Failover Resource Group
host # clresourcegroup create -S group - Create a Scalable Resource Group
host # clresourcegroup online + - Bring Online All Resource Groups
host # clresourcegroup delete group - Delete a Resource Group
host # clresourcegroup delete -F group - Delete a Resource Group and All of Its Resources
host # clresourcegroup switch -n nodename group - Switch the Current Primary Node of a Resource Group
host # clresourcegroup unmanage group - Move a Resource Group Into the UNMANAGED State
host # clresourcegroup suspend group - Suspend Automatic Recovery of a Resource Group
host # clresourcegroup resume group - Resume Automatic Recovery of a Resource Group
host # clresourcegroup set -p Failback=true + name=value - Change a Resource Group Property
host # clresourcegroup add-node -n nodename group - Add a Node To a Resource Group
host # clresourcegroup remove-node -n nodename group - Remove a Node From a Resource Group

Resource Administration Commands:

host # clreslogicalhostname create -g group lh-resource - Create a Logical Hostname Resource
host # clressharedaddress create -g group sa-resource - Create a Shared Address Resource
host # clresource create -g group -t type resource - Create a Resource
host # clresource delete resource - Remove a Resource
host # clresource disable resource - Disable a Resource
host # clresource set -t type -p name=value + - Change a Single-Value Resource Property
host # clresource set -p name+=value resource - Add a Value to a List of Property Values; Existing values in the list are unchanged.
host # clresource create -t HAStoragePlus -g group -p FileSystemMountPoints=mount-point-list -p Affinityon=true rs-hasp - Create an HAStorage Plus Resource
host # clresource clear -f STOP_FAILED resource - Clear the STOP_FAILED Error Flag on a Resource

Device Administration Commands:

host # cldevicegroup create -t vxvm -n node-list -p failback=true vxdevgrp - Add a VxVM Device Group
host # cldevicegroup delete devgrp - Remove a Device Group
host # cldevicegroup switch -n nodename devgrp - Switch a Device Group to a New Node
host # cldevicegroup offline devgrp - Bring Offline a Device Group
host # cldevice refresh diskname - Update Device IDs for the Cluster

Additional Administration and Monitoring Commands:

To add a Node to Cluster:

From the node to be added, which has access:

host # clnode add -c clustername -n nodename -e endpoint1, endpoint2 - Use this only if the node has access to the cluster

To remove a Node From the Cluster:

From the node to be removed, which is in noncluster
mode and has access:

host # clnode remove - Use this only if the node has access to the cluster

host # clnode evacuate nodename - Switch All Resource Groups and Device Groups Off of a Node
host # clinterconnect disable nodename:endpoint - Manage the Interconnect Interfaces
host # clinterconnect enable nodename:endpoint - To disable a cable so that maintenance can be performed, then enable the same cable afterward.
host # cluster status - Display the Status of All Cluster Components
host # command status - Display the Status of One Type of Cluster Component
host # cluster show - Display the Complete Cluster Configuration
host # command show Component - Display the Configuration of One Type of Cluster
host # command list - List One Type of Cluster Component
host # clnode show-rev -v - Display Sun Cluster Release and Version Information

To list the software versionson the current node.

host # clnode show | grep nodename - Map Node ID to Node Name
host # cltelemetryattribute enable -t disk rbyte.rate wbyte.rate read.rate write.rate - Enable Disk Attribute Monitoring on All Cluster Disks
host # cltelemetryattribute disable -t disk rbyte.rate wbyte.rate read.rate write.rate - Disable Disk Attribute Monitoring on All Cluster Disks

SHUTTING DOWN AND BOOTING A CLUSTER

To shut Down the Entire Cluster:

host # cluster shutdown

From one node:

host # clnode evacuate
host # shutdown - Shut Down a Single Node

To boot a Single Node from the OBP:

ok> boot

To reboot a Node Into Noncluster Mode from the OBP:

ok> boot -x

, Mike

Setting Up Basic IPMP On Solaris Unix

Hey there,

I thought I'd start this week out with a yawn ...I mean a bang ;) This is a topic we've never touched on, but one that is used very often in most large computer networks: IP multi-pathing. Simply put, allowing for the "outside world" to have more than one path to your networked server. The closest we've come, to date, would be our post on SunCluster monitoring. I suppose that could be chalked up to the fact that we're concentrating on a particular flavour of Unix, while most of my Linux postings are (or attempt to be) broad-based and satisfy as many distro's as possible.

In any event, here's a quick primer on getting IPMP set up on your Solaris host with the minimum amount of hardware and hassle. Quick and easy (I think ;)

1. Why should I set up IPMP? You don't have to. It's not a requirement of anything except SunCluster (assuming you want to get your setup officially certified - otherwise you can hack your way around that, too). The main benefit to you is that you'll have the comfort of knowing that users will still be able to connect to your host over the network even if one of your network cards goes to pot And, of course, that you won't have to lift a finger to make things go back to the way they were once the disaster is over. Simply put: You'll have network failover working for you in case of a network card failure. The user will never know you had an issue, as they will always be accessing your host via the same IP address.

2. What is required to use IPMP? Generally, you'll want to have as many failover points as possible (or reduce the number of network single-points-of-failure). This would mean having two separate network adapters. This can be done with one (with failover happening between virtual interfaces on the same NIC), but in order to attain your minimum two-points-of-failure, you should have two different NIC's with each of those residing on a different physical bus. It's good practice actually have those network cards hooked up to the network and the links verified before proceeding. Also, you should do most of this through a serial console or ALOM connection. Since you're dealing with networking failure, if you connect via a regular network connection to any IP on the host, there's a good chance you'll be dropped unexpectedly at some point during the process.

SPECIAL NOTE: IPMP, on its own, is not meant to protect you from an entire network segment going down. It will handle failover between NIC's, but they all need to be on the same network segment, or subnet, so if the network goes down (our example 10.10.10.0/Class C), you're still going to be offline.

3. Is it easy to setup IPMP on Solaris? Yeah :) Here's how:

4. At the PROM level, be sure to set the local-mac-address? variable to true

ok > setenv local-mac-address? true

you can also set this at the OS level, using the eeprom command:

host # eeprom local-mac-address?=true

If you choose to make this change using eeprom at the OS level, you should reboot your box before proceeding.

5. Install the required pkg files: This step is really easy, since the in.mpathd binary comes in the SUNWcsr (Core Solaris) pkg file, which your system won't run without. Hopefully it's already installed ;)

6. Alter the FAILURE_DETECTION_TIME value in the /etc/default/mpathd configuration file from 10000 milliseconds to 3 or 4000 milliseconds. This isn't necessary, but it will drop the failure detection time below 10 seconds, which might save you from having to answer any questions if you experience an unexplainable split-second network "burp" - of course, use this to your taste. Setting it too low may cause your virtual interfaces to flap back and forth constantly!

7. In your /etc/hosts file, include information for the "floating IP" (The one everyone else will use to connect to your system) and the other two physical IP's. This isn't absolutely necessary, but it can be helpful later on if you happen to forget what's what on any given machine.

Ex:
10.10.10.1 hostname-phys1
10.10.10.2 hostname-phys2
10.10.10.3 virtualhost

8. Modify the /etc/hostname.* files so that they include the proper information (this is where the meat of the configuration is done, in my opinion. If it's even up for a vote ;)

Ex:

/etc/hostname.hme0 (contents)

hostname-phys1 group ipmpgroup netmask + broadcast + deprecated -failover up
addif virtualhost netmask + broadcast + up

/etc/hostname.qfe0 (contents)

hostname-phys2 group ipmpgroup netmask + broadcast + deprecated -failover standby up

NOTE: You can, in our example, set the /etc/hostname.qfe0 file to be exactly the same as /etc/hostname.hme0 (with the only different being the "hostname-phys2" at the beginning of the entry) and it will work just as well. If you are using Veritas Cluster Server to do the failing over for you, it will not work unless you have one of your failover NIC's set to "standby" rather than "up"

9. Add both of your virtual IP's to an IP group (named ipmpgroup in this case - it can be anything you want) using ifconfig (You could also have done this before (added NIC's to groups before creating your /etc/hostname.* files), but, as long as we haven't started this baby up yet, the exact order doesn't matter):

host # ifconfig hme0 group ipmpgroup
host # ifconfig qfe0 group ipmpgroup

10. Activate: You can do this one of two basic ways (I'm sure there are more if you're creative about it ;) Reboot your machine or make the interfaces active manually. To activate manually, all you really need to do is copy and paste the contents of the two /etc/hostname.* files onto the command line, one after the other. If you have two lines of input in any, or either file, try to fit them all onto one line when executing from the command line.

11. Test: Run a continuous ping against your "floating IP" (10.10.10.3 in our case), and start pulling cables and resetting them. Do this one at a time, of course. If you remove both physical network connections at once, your machine will be off the net :)

You can also use the if_mpadm command to help with testing, if you prefer to "virtually" pull the plug on your physical interfaces. For instance "if_mpadm -d INTERFACE_NAME" will disable an interface, just as if you pulled the cable out. "if_mpadm -r INTERFACE_NAME" will reset the interface to it's "natural" state (however you have it set up; even if that's wrong ;) Check out the if_mpadm man page for more information on this command, although there's not much more to it.

If you're interested in jumping forward and getting into more advanced IPMP configurations, you can also check out the Sun Documentation Site and read the IP Network Multipathing Administration Guide at your leisure.

Cheers,

, Mike

Making Solaris 10 Zones Recognize Veritas Raw Devices

Hey there,

Here's something I learned today that I never knew before! ...Actually, when you think about it, everything you ever learn is something you never knew before. Otherwise, you'd already know it, so there'd be no learning involved... Where am I? ;)

Today's short-post is a little trick concerning Solaris 10 and the Veritas File System/Volume Manager. I've been using Veritas with Solaris 10 for quite a while, so this was pretty interesting to find out. In the usual case, I could always set up Solaris zones and hand them Veritas File System partitions with no incident. Usually, something like this:

host # zonecfg -z myNewZone
... <-- Here we skip past the create and "set zonepath" type stuff.  We have more information on dealing with Solaris zones in this old post
zonecfg:myNewZone> add fs
zonecfg:myNewZone:fs> set dir=/my/directory
zonecfg:myNewZone:fs> set special=/myNewZone/my/directory
zonecfg:myNewZone:fs> set type=lofs
zonecfg:myNewZone:fs> end
zonecfg:myNewZone> commit
zonecfg:myNewZone> exit

and life was beautiful ;)

Out of sheer luck (or horrible misfortune), I've always ended up at places that (while they may make heavy use of Veritas Volume Manager, File System, Cluster Server, etc) either went with some combination of SunCluster, Sun Solaris, HP and Oracle or Informix. Now, there's nothing wrong with any of the possible combinations you could end up with here. In fact, there's - more exactly - nothing wrong, since all of them (well, not HP-UX) can work with Solaris 10 zones using raw devices.

If you're not familiar with basic disk access concepts in Unix, at a very basic level, disk in Unix is generally accessible via two standard gateways: Block devices or Raw/Character devices. Most of the time, you probably deal with the block device interface, as this allows buffered sector reads of the device (generally a hard drive/disk) and is generally faster for most operations. To demonstrate what I mean, if you wanted to read a few bytes from HardDrive1, you could read those few bytes from your block device (almost to the byte) , but you'd have to read the entire device in order to get those few bytes when accessing the information via the raw device interface.

Raw devices do have their benefits, of course, or they probably wouldn't be around any more. Like their name suggests, they allow direct (raw) access to the disk. This can make it possible for you to recover information if your disk becomes corrupted (for instance) but is mostly used by software that likes to manage disk on its own and not have to rely on the Operating System. Most database software uses disk in this manner. Oracle, for instance, creates its own "file system" and (although it will work on UFS or any other filesystem) they say it works a lot more efficiently when it can be allocated raw disk and allowed to manage it on its own. It's probably true to some degree. Does anyone really know? ;)

Anyway (back from that two-paragraph aside ;), what I learned today is that Solaris 10 Zones can't (or won't) deal with raw Veritas devices (WT_?) I also found out that there's a way to get around that issue (hence, the post - I still owe you that one that goes on and on forever and leads nowhere ;).

You can get around this deficiency by doing the following (assuming you've already taken care of the basics of setting up your Veritas Volumes and Filesystem) and have created your zone with a base filesystem, like above, and want to add a raw device.

1. Check out your raw device file in the appropriate directory:

host # ls -l /dev/vx/rdsk/myRawDisk
crw------- 1 root root 289, 45000 Aug 24 11:13 myRawDisk

and make note of the Major and Minor numbers of the device (289 and 45000 respectively, above). You'll need these to feed the mknod command you're going to hammer out next (we'll assume you set your zonepath equal to /myNewZone - oh, yeah, and it's also important that you're not in your "Zone" when you do this):

host # cd /myNewZone/dev
host # mknod myRawDisk c 289 45000

2. And (it seems too simple!!! ;) then you just need to (well... you should ;) log into your zone and check it out, with a simple "dd" test or something:

host # zlogin -l root myNewZone
...
zone # ls -l /dev/myRawDisk
crw------- 1 root root 289, 45000 Aug 24 11:13 /dev/myRawDisk
zone # dd if=/dev/myRawDisk of=/dev/null
4096+0 records in
4096+0 records out

And, there you go. That problem you never knew you had is solved! ;)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Simple Sun Cluster Monitoring Script

Hey There,

Today, I've included a script I wrote to monitor a small SunCluster 3.1 environment (Running Oracle Parallel Server - OPS) we have set up at our shop. It basically runs through the output of various "scstat" command variations and reports on all combinations of errors it encounters. I built some functionality in it to be as specific as possible about the error states and to make sure that it returns an "All Clear" once the error condition no longer exists.

I also wrote this to run in cron. I run it every 5 minutes. Any time period is acceptable, depending upon what you need, and, of course, you could always put this script in a wrapper so that it runs constantly (although that generally necessitates writing another script to make sure that this script is running, restarting it as necessary and vice versa - For Veritas Cluster Server fans out there, this is a cheap and quick way to imitate the relationship between had and hashadow).

This was also written for a small environment (2 machines with 2 "live" network connections each). I didn't include monitoring of the heartbeats, since the script is meant to be run locally (with "exactly" the same values in the customizable section) on all nodes in a cluster, and scstat's indication of failure on any of these tests is, in and of itself, a guarantee that loss of a heartbeat connection is the very least of your problems ;)

Enjoy!





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

###################################
# suncluster_mon - check cluster health
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
# simple parser for:
# 1. scstat -n
# 2. scstat -i
# 3. scstat -g
# 4. scstat -D
#
###################################
 
###################################
# CUSTOMIZE SETTINGS HERE
# $debug can be set with the -d
#  switch when invoking the script
###################################
$primary_node="";
$secondary_node="";
$primary_eth="";
$secondary_eth="";
$date=`/usr/bin/date "+%A, %B %d - %H:%M:%S"`;
$hostname=`/usr/bin/hostname`;
$debug=0;
###################################
# PLEASE DO NOT EDIT BELOW
# EXCEPT MAIL SETTINGS AT END
###################################

chomp($date);

if ( $#ARGV >= 0 ) {
 foreach $arg (@ARGV) {
  if ( $arg =~ /-d/ ) {
   $debug=1;
   print "Debug Output Set\n";
  } else {
   push(@nogo, $arg)
  }
 }
 if ( @nogo > 0 ) {
  print "Unrecognized options: @nogo\n";
  print "Ignoring and continuing\n";
 }
}

@scstatn=`/usr/cluster/bin/scstat -n|/usr/bin/egrep 'node:'`;
for $nodestat (@scstatn) {
 if ( $nodestat =~ /Cluster node:/ ) {
  $nodestat =~ s/Cluster node://;
  @nodestat = split(" ", $nodestat);
  if ( $nodestat[1] eq "Offline" && $nodestat[0] eq $secondary_node ) {
   if ( $nodefailure ) {
    $alert="\nCluster Alert:\n$primary_node and $secondary_node are both in Offline state.  Cluster Failed\n";
   } else {
    $alert="\nCluster Alert:\n$primary_node is in Offline state.  $secondary_node is Online - Cluster Crippled\n";
   }
   $alert="\nCluster Alert:\n$nodestat[0] has switched to $nodestat[1] Status - Cluster Crippled - Checking For Dual Failure\n";
   $nodefailure=1;
   push(@alert, $alert);
  } elsif ( $nodestat[1] eq "Offline" && $nodestat[0] eq $primary_node ) {
   if ( $nodefailure ) {
    $alert="\nCluster Alert:\n$primary_node and $secondary_node are both in Offline state.  Cluster Failed\n";
   } else {
    $alert="\nCluster Alert:\n$primary_node is in Offline state.  $secondary_node is Online - Cluster Crippled\n";
   }
   $alert="\nCluster Alert:\n$nodestat[0] has switched to $nodestat[1] Status - Cluster Crippled - Checking For Dual Failure\n";
   $nodefailure=1;
   push(@alert, $alert);
  } else {
   if ( $debug ) {
    $alert="Cluster Info: $nodestat[0] and $nodestat[1] OK\n";
    push(@debug, $alert);
   }
   $beautiful++;
  }
 } else {
  print "WTF ---- SCSTAT -D\n";
 }
}
if ( $beautiful == 2 ) {
  if ( $debug )   {
   $alert="Cluster Info: Both nodes OK\n";
   push(@debug, $alert);
  }
}

@scstati=`/usr/cluster/bin/scstat -i|/usr/bin/egrep 'Group:'`;
for $ipmpstat (@scstati) {
 if ( $ipmpstat =~ /IPMP Group:/ ) {
  $ipmpstat =~ s/IPMP Group://;
  @ipmpstat = split(" ", $ipmpstat);
  if ( $ipmpstat[0] == $primary_node ) {
   if ( $ipmpstat[3] eq $primary_eth && $ipmpstat[4] ne "Online" ) {
    $alert="\nIPMP Alert:\n$ipmpstat[3] is in $ipmpstat[4] mode on $ipmpstat[0] - Checking for Online of $ipmpstat[0] $secondary_eth\n";
    $ethprimaryfailure=1;
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Online" && $ethprimaryfailure == 1 ) {
    $alert="\nIPMP Alert:\n$ipmpstat[0] $secondary_eth is not in an Online state on $ipmpstat[0] - IPMP Group Down on both Interfaces!\n";
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Standby" && ! $ethprimaryfailure ) {
    $alert="\nIPMP Alert:\n$ipmpstat[3] is in $ipmpstat[4] mode on $ipmpstat[0] - Failover interface is not UP - IPMP Group Crippled on $ipmpstat[0]!\n";
    push(@alert, $alert);
   } else {
    if ( $debug )   {
     $alert="IPMP Info: IPMP Group $ipmpstat[0] $ipmpstat[3] OK\n";
     push(@debug, $alert);
    }
   }
  } elsif ( $ipmpstat[0] == $secondary_node ) {
   if ( $ipmpstat[3] eq $primary_eth && $ipmpstat[4] ne "Online" ) {
    $alert="\nIPMP Alert:\n$primary_eth is in $ipmpstat[4] mode on $ipmpstat[0] - Checking for Online of $ipmpstat[0] $secondary_eth\n";
    $ethprimaryfailure=1;
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Online" && $ethprimaryfailure == 1 ) {
    $alert="\nIPMP Alert:\n$ipmpstat[0] ipmpstat[3] not in an Online state on ipmpstat[0] - IPMP Group Down on both Interfaces!\n";
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Standby" && ! $ethprimaryfailure ) {
    $alert="\nIPMP Alert:\n$ipmpstat[3] is in $ipmpstat[4] mode on $ipmpstat[0] - Failover interface is not UP - IPMP Group Crippled on $ipmpstat[0]!\n";
    push(@alert, $alert);
   } else {
    if ( $debug )   {
     $alert="IPMP Info: IPMP Group $ipmpstat[0] $ipmpstat[3] OK\n";
     push(@debug, $alert);
    }
   }
  }
 } else {
  print "WTF ---- SCSTAT -I\n";
 }
}

@scstatg=`/usr/cluster/bin/scstat -g|/usr/bin/egrep 'Resource:|Resources:|Group:'`;
for $rgstat (@scstatg) {
 if ( $rgstat =~ /Resources:/ ) {
  $rgstat =~ s/Resources://;
  @rgstat = split(" ", $rgstat);
  if ( @rgstat != 5 ) {
   shift @rgstat;
   $alert="\nOracle Resource Alert:\nOracle resource group $rgstat[0] is not running all resources on the cluster\n--Only running -- @rgstat\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Oracle Resource Info: Oracle resource group $rgstat[0] OK -- @rgstat OK\n";
    push(@debug, $alert);
   }
  }
 } elsif ( $rgstat =~ /Group:/ ) {
  $rgstat =~ s/Group://;
  @rgstat = split(" ", $rgstat);
  if ( $rgstat[2] ne "Online" && $rgstat[1] eq $primary_node ) {
   $alert="\nOracle Resource Alert:\nResource Group $rgstat[0] is in $rgstat[2] state on primary node $rgstat[1] - Checking for failover\n";
   $rgprimaryfailure=1;
   push(@alert, $alert);
  } elsif ( $rgstat[2] ne "Online" && $rgstat[1] eq $secondary_node && $rgprimaryfailure == 1 ) {
   $alert="\nOracle Resource Alert:\nResource Group $rgstat[0] is in $rgstat[2] state on cluster - Resource Group Down on both Nodes!\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Oracle Resource Info: $rgstat[1] Resource Group $rgstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } elsif ( $rgstat =~ /Resource:/ ) {
  $rgstat =~ s/Resource://;
  @rgstat = split(" ", $rgstat);
  if ( $rgstat[2] ne "Online" && $rgstat[1] eq $primary_node ) {
   $alert="\nOracle Resource Alert:\nResource$rgstat[0] is in $rgstat[2] state on primary node $rgstat[1] - Checking for failover\n";
   $rgprimaryfailure=1;
   push(@alert, $alert);
  } elsif ( $rgstat[2] ne "Online" && $rgstat[1] eq $secondary_node && $rgprimaryfailure == 1 ) {
   $alert="\nOracle Resource Alert:\nResource $rgstat[0] is in $rgstat[2] state on cluster - Resource Group Down on both Nodes!\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Oracle Resource Info: $rgstat[1] Resource $rgstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } else {
  print "WTF ---- SCSTAT -G\n";
 }
}

@scstatD=`/usr/cluster/bin/scstat -D|/usr/bin/egrep 'servers|status'`;
for $diskstat (@scstatD) {
 if ( $diskstat =~ /servers/ ) {
  $diskstat =~ s/Device group servers://;
  @diskstat = split(" ", $diskstat);
  if ( $diskstat[1] ne $primary_node ) {
   $alert="\nDisk Resource Alert:\n$diskstat[0] has switched primary node to $diskstat[1] from $diskstat[2]\n";
   push(@alert, $alert);
  } elsif ( $diskstat[1] ne $primary_node && $diskstat[1] ne $secondary_node ) {
   $alert="\nDisk Resource Alert:\n$diskstat[0] has failed on all nodes!\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Disk Resource Info: $diskstat[1] primary - $diskstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } elsif ( $diskstat =~ /status/ ) {
  $diskstat =~ s/Device group status://;
  @diskstat = split(" ", $diskstat);
  if ( $diskstat[1] ne "Online" ) {
   $alert="\nDisk Resource Group Alert:\n$diskstat[0] has switched to $diskstat[1] state on the cluster\n";
   push(@alert, $alert);
  } elsif ( $diskstat[1] eq "Online" ) {
   if ( $debug) {
    $alert="Disk Resource Group Info: $diskstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } else {
  print "WTF ---- SCSTAT -D\n";
 }
}

###################################
# EDIT To: Reply-To: and From:
# if you want mail to go somewhere
# useful and-or helpful!
###################################

if ( @alert > 0 || @debug > 0 ) {
 open(CMAIL, "|/usr/lib/sendmail -t");
 print CMAIL "Subject: CLUSTER ALERT - $hostname - $date\n";
 print CMAIL "From: you\@yourdomain.com\n";
 print CMAIL "Reply-To: you\@yourdomain.com\n";
 print CMAIL "To: recipients\@yourdomain.com\n";
 print CMAIL "\n\n";
 foreach $warning (@alert) {
  print CMAIL $warning;
 }
 if ( $debug > 0 ) {
  foreach $message (@debug) {
   print CMAIL $message;
  }
 }
 close(CMAIL);
 system("touch /tmp/cfail_ihot_stat");
} elsif ( @alert == 0 && -f "/tmp/cfail_ihot_stat" ) {
 open(CMAIL, "|/usr/lib/sendmail -t");
 print CMAIL "Subject: CLUSTER RESTORED - $hostname - $date\n";
 print CMAIL "From: you\@yourdomain.com\n";
 print CMAIL "Reply-To: you\@yourdomain.com\n";
 print CMAIL "To: recipients\@yourdomain.com\n";
 print CMAIL "\n\n";
 print CMAIL "All Cluster Services Back To Good State\n";
 print CMAIL "All Cluster Nodes: OK\n";
 print CMAIL "All IPMP Groups: OK\n";
 print CMAIL "All Oracle Resources: OK\n";
 print CMAIL "All Storage Groups: OK\n";
 close(CMAIL);
 unlink("/tmp/cfail_ihot_stat");
}

, Mike

linux unix internet technology

Configuring A Basic HACMP Cluster On AIX Unix

Hey there,

Today we're going to go back to the AIX well (which we haven't visited since our fairly-old post on working with AIX LVM. That post links back to a bunch of other posts in the series, but it "has" been about 3 months since we've touched on the AIX OS, which is grossly disproportionate to the amount of attention Solaris, Linux and Open Source Software get. Perhaps, someday, I'll work on it enough that I'll feel more comfortable digging into its guts. ...figuratively, of course ;)

This little how-to post isn't meant to cover all the angles of cluster setup and configuration that there are to cover. If it were, that would be (aside from slightly-more-than-presumptuous) a long long long post. Your children would have children while you tried to remain interested ;) This is a simple little "get started" guide and only meant for a simple 2 node cluster. If you want to get fancier, visit IBM's HACMP Knowledge Base where they've got more documentation than you could ever wish for. Check the bottom of the post for some other good resource sites.

First things first (before we get to step one), as with any other sort of cluster setup (VCS, SunCluster, etc) you'll need to determine what you want from your cluster. That's beyond the scope of this post (or the other ones we just referenced), but you should have a clear plan of attack. Know what you want and everything you can think of that it might take to get there (in this case 2 nodes for redundancy and uptime - On the flipside, what we're looking to do here, which is also very important, is to just set up the basics of our cluster and disk heartbeats. The network is up and running, cabled and doing well. This is one "huge" assumption, but it just means less for us to have to soak up in one sitting :) It's always a good idea to write a list. ...unless you're in the habit of losing your lists, in which case you should keep everything in your head where you'll retain most everything until you lose "it." (And you can take that either way ;)

NOTE: We're going to be running through this using an AIX CLI utility called "smit." Of course, it's more of a TUI (Terminal Based User Interface) than real CLI, but, while you're using it, you can make it show you the underlying commands it's running with judicious use of the F6 key. Copy and paste all that stuff into a script and amaze your friends :)

1. Choose one node from which to do your configuration. It can be either of the two we're going to use (host1 or host2). It's important that you stick with your choice, though.

2. Check for any possible HACMP configuration already on the host (just in case) by using smit (called as smitty):

NOTE: For this how-to, we'll represent menu choices (drill-down) using tabs for each level down, and set them in boldface type. Oh, yes, and step two will fail if there are no cluster components to remove. It'll make you feel all warm and cozy inside, though ;)

host1 # smitty hacmp
 Extended Configuration
  Extended Topology Configuration
   Configure an HACMP Cluster
    Remove an HACMP Cluster

3. Then, proceed to configure the topology of your cluster (For us, this simply means two nodes sharing a heartbeat address)

host1 # smitty hacmp
 Extended Configuration
  Extended Topology Configuration

4. Here you have a few different things to do (under Extended Topology Configuration):

   Configure an HACMP Cluster

<-- You'll need at least two hosts to create a cluster and you can't have more than one cluster per pair of hosts (Only 1 for 2).

   Configure HACMP Nodes

<-- Be sure to use your boot IP address in the "communication path" field. We haven't gotten to the virtual IP yet

5. Now, on each node of your cluster (host1 and host2), be sure to start the clcomdES service if it isn't running already (might be on host1 since we're configuring the cluster on it, but shouldn't be on host2)

To check for it, type:

host1 # lssrc -s clcomdES

To start it , type:

host1 # startsrc -s clcomdES

6. Now go back up one level (to Extended Configuration) and run through these options (You can just ignore the ones we don't mention):

  Configure HACMP Network

<-- Add your public, or virtual Ethernet network information here. Be sure to configure the subnet mask appropriately and set "Enable IP Address Takeover via IP Aliases" equal to "yes." Also, unset (leave or make blank) the "IP Address Offset for Heartbeating over IP Aliases" setting.

  Configure HACMP Interfaces and Devices

<-- if you find any old, or strange, node names in here, delete them. They're probably from a previous setup. You can keep them, but they might cause you headache later on. Note that you'll generally only see this type of behaviour if the machine has been cloned. You can also manually define interfaces here (if you choose), but you should have already done that under "Configure HACMP Network" above. If you need to add any interfaces here, just choose "Add Pre-defined interface" and go to town :)

  Configure HACMP Persistent Node IP Label/Address

<-- This should be the address associated with your host's DNS name (not the cluster's DNS name!).

7. Now add service addresses for all of the service types you want to host on your cluster (Services are generally disk, network, etc - resources you want to share within your cluster and keep highly available) Add Service addresses for each of the services you want to host - we're only dealing with disk and IP here.

host1 # smitty hacmp
 Extended Configuration
  Extended Resource
   Configuration HACMP Extended Resources Configuration
    Configure HACMP Service IP Labels/Addresses

Here, add a volume group solely dedicated to providing disk-based heartbeat! Name it whatever you want, like "host1-diskhb" and make sure it's type is set to "Enhanced-Capable Concurrent Volume Group." You won't ever actually need this volume group for anything (disk storage, etc, I mean), but making it this type of volume prevents it from being discovered. You "Do Not" want to share your host's disk heartbeat device!

8. Now, proceed to configure your disk heartbeat network (this part practically does the work for you. Just keep stabbing at those keys ;)

host1 # smitty hacmp
 Extended Configuration
  Extended Topology Configuration
   Configure HACMP Networks
    Add a Network
     Discovered
      host1-diskhb

9. Next, add devices to our disk heartbeat network:

host1 # smitty hacmp
 Extended Configuration
  Extended Topology Configuration
   Configure HACMP Communication Interfaces/Devices
    Add Communication Interfaces/Devices
     Add Pre-defined Communication Interfaces and Devices
      Communication Devices

Choose the (hopefully, at this point) only heartbeat network available :) Name the device whatever you want (again, we'll go with host1-diskhb), supply the logical path to the device (if necessary) and supply the name of one of your nodes when required to. Note that this will trigger an error the first time you run it (This is perfectly normal!). It will state the obvious in a slightly different manner, but your error message will basically say that you can't have only one device defined, since two are required). The simple fix: Run step 9 again exactly, except pick your other node's name when required to. Now you have two members in your disk heartbeat network.

10. Now, set up your basic networking:

host1 # smitty hacmp
 tcpip
  Further Configuration
   Static routes
    Add a Static Route

Make sure to set your "Destination Type" to "net," your "Destination Address" to "default" and set the "Default Gateway" to the IP address of your default router (Same as when you set up your persistent host address in step 4 - Note that it's not shown in the documentation here, but will be there if you follow the smitty-steps down to the correct menu level :)

11. And you're set. Just for caution's sake, you should do some simple testing by running:

host1 # smitty hacmp
 Extended Configuration
  Extended Verify and Synchronization

Accept the defaults here and your changes and setup will be verified and synchronized with every node in your cluster (In our case today, the "other one" ;)

And you're good to go :)

Remember, our little experiment here was a very quick and dirty how-to with very limited scope. If you want an equally expedient, but more comprehensive and wide-ranging, look at setting up HACMP fast, check out TriParadigm's HACMP Configuration Page. It's a lot better than this one, but may be too much if you're just getting started (?) :)

Cheers,

Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Distributing New SSH Keys Using Rsh On Linux And Unix

Happy Lazy Sunday,

Today we're going to grind through our "Lazy Sunday" post with a quick script to update SSH keys network-wide, by using rsh (the less secure of the two protocols). Once you've accomplished this (or have already accomplished this) and are happy with your network's SSH setup, I'd suggest disabling rsh altogether. Then you can move on to quickly setting up your SSH keys all over the network, focus on maintaining the integrity of your sessions, if you have issues with that, and even setting simple SCP routines to help keep your network easy to manage.

My feeling is that, no matter what the circumstances (unless they make it so you "have" to use it), rsh should always be disabled, no matter what version of Linux or Unix you're running. Certain software (like SunCluster) can experience very strange issues if you don't allow it. At least, up to version 3.1. I can only afford to keep myself in the almost-very-best, after all ;)

Enjoy your Sunday, and enjoy this little script :) A lot of how you use it depends on the way your network is set up. If it's already secure, you can modify it slightly to use to update host keys, but it was mostly written for folks with a hybrid rsh/SSH setup, who want to move to full SSH implementation, so that keys can be upgraded everywhere at once from the same base set.

The only major assumptions this script makes is that you've built openssl, openssh and zlib from source, want to install in /usr/local and are running this script from the top level directory of those builds. It should be run with a list of servers following the command name:

host # ./sshdist.sh servera serverb serverc...

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

#
# sshdist.sh - distribute a newly built openssl, openssh and zlib and set up keys
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

cd /usr/local
for x in $@
do
 echo seeding $x...
 echo " openssh... \c"
 tar cpf - openssh*|rsh $x "cd /usr/local;tar xpf -"
 echo "openssl... \c"
 tar cpf - openssl*|rsh $x "cd /usr/local;tar xpf -"
 echo "zlib... \c"
 tar cpf - zlib|rsh $x "cd /usr/local;tar xpf -"
 echo done!
done

echo
echo "Consistency Check..."
for x in $@
do
 echo "${x}: \c"
 rsh $x "cd /usr/local;du -sk openssl* openssh* zlib*|xargs echo"
done

echo
echo "Generating Host Keys..."
for x in $@
do
 echo "${x}...\c"
 echo "removing old host and dsa keys..."
 rsh $x "rm /usr/local/openssh/ssh/ssh_host*key*"
 echo "generating new host key..."
 rsh $x "/usr/local/openssh/bin/ssh-keygen -b 1024 -f /usr/local/openssh/ssh/ssh_host_key -N ''"
 echo "generating new dsa key..."
 rsh $x "/usr/local/openssh/bin/ssh-keygen -d -f /usr/local/openssh/ssh/ssh_host_dsa_key -N ''"
 echo "Done!"
done

echo
echo "Changing the active SSH installation..."

for x in $@
do
 echo "$x... \c"
 rsh $x "cd /usr/local;ls -d ssh* >>encap.exclude"
 echo "Excluding old ssh from reinstallation and deleting all links..."
 rsh $x "cd /usr/local/ssh-1.2.2*/bin;ls|xargs -I {} -t rm /usr/local/bin/{}"
 rsh $x "cd /usr/local/ssh-1.2.2*/man/man1;ls|xargs -I {} -t rm /usr/local/man/man1/{}"
 rsh $x "cd /usr/local/ssh-1.2.2*/man/man8;ls|xargs -I {} -t rm /usr/local/man/man8/{}"
 rsh $x "cd /usr/local/ssh-1.2.2*/sbin;ls|xargs -I {} -t rm /usr/local/sbin/{}"
 echo "Stopping old SSH daemon... \c"
 rsh $x "ps -ef|grep sshd|grep -v grep|awk '{print \$2}'|xargs kill"
 echo "Relinking... \c"
 rsh $x "/usr/local/bin.pl /usr/local /usr/local"
 echo "Starting new SSH daemon... \c"
 rsh $x "/usr/local/sbin/sshd"
 echo Done!
done

echo
echo "Sanity Check..."

for x in localhost $@
do
 echo "$x : \c"
 rsh $x "ps -ef|grep sshd|grep -v grep"
done

, Mike