Configuring A Basic HACMP Cluster On AIX Unix

Hey there,

Today we're going to go back to the AIX well (which we haven't visited since our fairly-old post on working with AIX LVM. That post links back to a bunch of other posts in the series, but it "has" been about 3 months since we've touched on the AIX OS, which is grossly disproportionate to the amount of attention Solaris, Linux and Open Source Software get. Perhaps, someday, I'll work on it enough that I'll feel more comfortable digging into its guts. ...figuratively, of course ;)

This little how-to post isn't meant to cover all the angles of cluster setup and configuration that there are to cover. If it were, that would be (aside from slightly-more-than-presumptuous) a long long long post. Your children would have children while you tried to remain interested ;) This is a simple little "get started" guide and only meant for a simple 2 node cluster. If you want to get fancier, visit IBM's HACMP Knowledge Base where they've got more documentation than you could ever wish for. Check the bottom of the post for some other good resource sites.

First things first (before we get to step one), as with any other sort of cluster setup (VCS, SunCluster, etc) you'll need to determine what you want from your cluster. That's beyond the scope of this post (or the other ones we just referenced), but you should have a clear plan of attack. Know what you want and everything you can think of that it might take to get there (in this case 2 nodes for redundancy and uptime - On the flipside, what we're looking to do here, which is also very important, is to just set up the basics of our cluster and disk heartbeats. The network is up and running, cabled and doing well. This is one "huge" assumption, but it just means less for us to have to soak up in one sitting :) It's always a good idea to write a list. ...unless you're in the habit of losing your lists, in which case you should keep everything in your head where you'll retain most everything until you lose "it." (And you can take that either way ;)

NOTE: We're going to be running through this using an AIX CLI utility called "smit." Of course, it's more of a TUI (Terminal Based User Interface) than real CLI, but, while you're using it, you can make it show you the underlying commands it's running with judicious use of the F6 key. Copy and paste all that stuff into a script and amaze your friends :)

1. Choose one node from which to do your configuration. It can be either of the two we're going to use (host1 or host2). It's important that you stick with your choice, though.

2. Check for any possible HACMP configuration already on the host (just in case) by using smit (called as smitty):

NOTE: For this how-to, we'll represent menu choices (drill-down) using tabs for each level down, and set them in boldface type. Oh, yes, and step two will fail if there are no cluster components to remove. It'll make you feel all warm and cozy inside, though ;)

host1 # smitty hacmp
 Extended Configuration
  Extended Topology Configuration
   Configure an HACMP Cluster
    Remove an HACMP Cluster

3. Then, proceed to configure the topology of your cluster (For us, this simply means two nodes sharing a heartbeat address)

host1 # smitty hacmp
 Extended Configuration
  Extended Topology Configuration

4. Here you have a few different things to do (under Extended Topology Configuration):

   Configure an HACMP Cluster

<-- You'll need at least two hosts to create a cluster and you can't have more than one cluster per pair of hosts (Only 1 for 2).

   Configure HACMP Nodes

<-- Be sure to use your boot IP address in the "communication path" field. We haven't gotten to the virtual IP yet

5. Now, on each node of your cluster (host1 and host2), be sure to start the clcomdES service if it isn't running already (might be on host1 since we're configuring the cluster on it, but shouldn't be on host2)

To check for it, type:

host1 # lssrc -s clcomdES

To start it , type:

host1 # startsrc -s clcomdES

6. Now go back up one level (to Extended Configuration) and run through these options (You can just ignore the ones we don't mention):

  Configure HACMP Network

<-- Add your public, or virtual Ethernet network information here. Be sure to configure the subnet mask appropriately and set "Enable IP Address Takeover via IP Aliases" equal to "yes." Also, unset (leave or make blank) the "IP Address Offset for Heartbeating over IP Aliases" setting.

  Configure HACMP Interfaces and Devices

<-- if you find any old, or strange, node names in here, delete them. They're probably from a previous setup. You can keep them, but they might cause you headache later on. Note that you'll generally only see this type of behaviour if the machine has been cloned. You can also manually define interfaces here (if you choose), but you should have already done that under "Configure HACMP Network" above. If you need to add any interfaces here, just choose "Add Pre-defined interface" and go to town :)

  Configure HACMP Persistent Node IP Label/Address

<-- This should be the address associated with your host's DNS name (not the cluster's DNS name!).

7. Now add service addresses for all of the service types you want to host on your cluster (Services are generally disk, network, etc - resources you want to share within your cluster and keep highly available) Add Service addresses for each of the services you want to host - we're only dealing with disk and IP here.

host1 # smitty hacmp
 Extended Configuration
  Extended Resource
   Configuration HACMP Extended Resources Configuration
    Configure HACMP Service IP Labels/Addresses

Here, add a volume group solely dedicated to providing disk-based heartbeat! Name it whatever you want, like "host1-diskhb" and make sure it's type is set to "Enhanced-Capable Concurrent Volume Group." You won't ever actually need this volume group for anything (disk storage, etc, I mean), but making it this type of volume prevents it from being discovered. You "Do Not" want to share your host's disk heartbeat device!

8. Now, proceed to configure your disk heartbeat network (this part practically does the work for you. Just keep stabbing at those keys ;)

host1 # smitty hacmp
 Extended Configuration
  Extended Topology Configuration
   Configure HACMP Networks
    Add a Network
     Discovered
      host1-diskhb

9. Next, add devices to our disk heartbeat network:

host1 # smitty hacmp
 Extended Configuration
  Extended Topology Configuration
   Configure HACMP Communication Interfaces/Devices
    Add Communication Interfaces/Devices
     Add Pre-defined Communication Interfaces and Devices
      Communication Devices

Choose the (hopefully, at this point) only heartbeat network available :) Name the device whatever you want (again, we'll go with host1-diskhb), supply the logical path to the device (if necessary) and supply the name of one of your nodes when required to. Note that this will trigger an error the first time you run it (This is perfectly normal!). It will state the obvious in a slightly different manner, but your error message will basically say that you can't have only one device defined, since two are required). The simple fix: Run step 9 again exactly, except pick your other node's name when required to. Now you have two members in your disk heartbeat network.

10. Now, set up your basic networking:

host1 # smitty hacmp
 tcpip
  Further Configuration
   Static routes
    Add a Static Route

Make sure to set your "Destination Type" to "net," your "Destination Address" to "default" and set the "Default Gateway" to the IP address of your default router (Same as when you set up your persistent host address in step 4 - Note that it's not shown in the documentation here, but will be there if you follow the smitty-steps down to the correct menu level :)

11. And you're set. Just for caution's sake, you should do some simple testing by running:

host1 # smitty hacmp
 Extended Configuration
  Extended Verify and Synchronization

Accept the defaults here and your changes and setup will be verified and synchronized with every node in your cluster (In our case today, the "other one" ;)

And you're good to go :)

Remember, our little experiment here was a very quick and dirty how-to with very limited scope. If you want an equally expedient, but more comprehensive and wide-ranging, look at setting up HACMP fast, check out TriParadigm's HACMP Configuration Page. It's a lot better than this one, but may be too much if you're just getting started (?) :)

Cheers,

Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Problems Compiling CFEngine With Static Libraries

Here's a little weekend trick for you :)

I ran into this while compiling the latest version of cfengine. If you've ever built it, you know that two of its prerequisites are having access to includes and library files from openssl and Berkely db during configure and compile time. We visited the one program with a heavy take on building the db package in a previous post.

So, with the assumption that you've already installed Berkeley db and openssl and are now ready to configure cfengine, the first hurdle is easy enough to get around. If your libraries and includes are in standard places (like /usr/lib and /usr/include) it isn't even a hurdle at all. However, if you've built your Berkeley db and openssl software into special directories, you'll need to pass at least one of these two options to configure:

host # ./configure --with-berkeleydb=/some/dir/db --with-openssl=/some/dir/openssl

That part's simple to figure out, because

host # ./configure --help

will tell you what flags you need to pass after you get the error during configuration. Of course, reading the INSTALL and/or README file first would help with that, too ;)

The second hurdle is a little less obvious, and may not happen to everyone. It has to do with how you compiled your Berkeley db and openssl source. If you compiled them to have shared libraries, you won't run into this issue. However, if you compiled them with static libraries, the cfengine "make" process will balk when it gets to the point that it call those libraries. You would see an error similar to:

Undefined error dlopen -- undefined symbol...blah, blah, blah

This is because it won't, or can't, deal with the static libraries you've compiled for db and openssl.

The good news is that you can get around this fairly easily by setting the following two variables in your shell and just restarting the "make."

host # BERKELEY_DB_LIB="/some/dir/db/lib/libdb-4.5.a -lrt"
host # OPENSSL_LIB_LIB="/some/dir/openssl/lib/libcrypto.a -ldl"

and then running make with the -e flag, which will give your environment variables (set above) precedence over variables set in the Makefile.

It's important to note that some systems (generally older ones, like Solaris 2.6) don't have the -lrt (librt) library. In these cases, just use -ldl for both the db and openssl evironment variables.

After that, it's all icing :)

Cheers,

, Mike

linux unix internet technology

CC FLAG UPDATE - Building Solaris Packages Quickly - An Old Post Revisited

Hey There,

Updated Some More - Thank you for your emails :) I've updated this once more to hard-code the CC=gcc flag on the configure line. This db package's configure script appears not to pickup gcc more often than it does, so, unless you're using cc, it's best just to force gcc.

Now that we've just gotten through a marathon "how to" on creating Linux RPMS, I'd thought I'd look back at the old post I referenced at the beginning of the week regarding making your own Solaris packages.

That post was, and still is, theoretically sound. Although, looking back at it, I feel that it really lacked the kind of example work-up that would have made the instructions come back and hit home. So with reference to my original (admittedly dry) instruction on how to make your own Solaris pkg files, here's a quick example of the actual process of creating a package from compiling your software to the finished product. Hopefully this will not only flesh-out the original post, but also show how much simpler this is to do in reality than it is in theory :)

And, in an effort to make up for lost time and dispel any possibility of confusion, we're going to use a real freely-available source gzip that anyone can grab for free and walk through this on their own Solaris machine step-by-step, cut-and-paste. Here are the basic requirements in order to reproduce this on your own without having to do any extra work outside of the instructions posted:

1. Have gcc and make installed on your Solaris System.
2. Run Solaris on Sparc (not x_86 or Intel - although this should still work).
3. If possible, use the 64 bit version (As long as the "isainfo" or "isalist" commands return "sparcv9" somewhere in their output, you should be in the same boat I am as I do this compile and package creation. Again, this probably won't be an issue if you use 32 bit.

And that should be it. Now we just need to download the Berkeley DB 4.6.21 source gzipped tarball (available by clicking the name of the product on this line or here ;)

And off we go! I'm going to basically walk through this by example (what I'll actually type during the process) and leave out the tons of output from "configure" and "make," etc, so this doesn't get too hard to read. Hopefully, it will be painfully easy to understand :)

Step 1. Unpack and build your software (grab your Berkeley DB source now if you haven't already!

SPECIAL NOTE: This post has been updated to reflect the need to run configure from the build_unix subdirectory under the db-4.6.21 directory. You may also need to set CC=gcc as an option on your configure command line, but db should pick it up as long as it comes first in your PATH. Apologies for the typo.

host # mkdir /tmp/build
host # cp db-4.6.21.tar.gz /tmp/build/.
host # cd /tmp/build
host # gzip -d -c db-4.6.21.tar.gz|tar xvpf -
... <--- This will be shorthand for obligatory command output that doesn't make a difference to us here.
host # cd db-4.6.21
host # cd build_unix
host # ../dist/configure CC=gcc --prefix=/usr/local/db-4.6.21
...
host # make
...
host # make docdir=/usr/local/db-4.6.21/doc/4.6.21 install
...
host # cd /tmp/build
host # rm -r db-4.6.21

And we're done with the compile :)

Step 2. Create the Solaris pkg file from our installation!

host # cd /usr/local/db-4.6.21
host # find . -print|pkgproto >prototype
host # echo "i pkginfo" >>TEMPFILE
host # echo "i checkinstall" >>TEMPFILE
host # cat prototype >>TEMPFILE
host # mv TEMPFILE prototype
host # vi checkinstall <--- For the two files we need to edit, I'll just indent and put the contents of each below the lines where I use "vi" (my favorite editor). You can edit the file with any editor you want :)

#!/bin/sh

 platform_should_be="sparc"
 platform=`uname -p`
 if [ ${platform} != ${platform_should_be} ]
 then
  echo "PROGRAM can only be installed on ${platform_should_be}"
  exit 1
 fi
 exit

host # vi pkginfo

SUNW_PRODNAME="SunOS"
 SUNW_PRODVERS="5.9"
 SUNW_PKGTYPE="usr"
 PKG="MTdb"
 NAME="Berkely DB-4.6.21"
 VERSION="4.6.21"
 VENDOR="Open Source"
 ARCH="sparc"
 EMAIL="eggi@comcast.net"
 CATEGORY="application"
 BASEDIR=/usr/local/db-4.6.21
 DESC="http://linuxshellaccount.blogspot.com DB-4.6.21"
 PSTAMP="The Linux And Unix Menagerie"
 CLASSES="none"

host # pkgmk -b `pwd` -d /tmp
...
host # pkgtrans -s /tmp /tmp/build/MTdb.pkg MTdb
...
host # rm -r /tmp/MTdb

And now you can move the /tmp/build/MTdb.pkg file anywhere on the file system you want. You should be able to utilize all the applicable "pkginfo" command options against it and it should install as easily (assuming you left it in /tmp/build) as this (of course, it won't be any harder if you install it from anywhere else ;)

host # pkgadd -d /tmp/build/MTdb.pkg

also

host # pkgrm MTdb <--- If you ever want to delete it from the package repository

Hope you enjoyed that whirlwind tour of a Solaris package (pkg) build in almost-real-time. Hopefully, also, it helps complement our previous post on creating Solaris packages by giving you a solid working example to compare against the theory!

Cheers,


, Mike

linux unix internet technology