Cluster Server Failover Testing On Linux And Unix

A fine how do you do :)

WARNING/GUARANTEE: Today's post is the product of a tired mind that just finished working and didn't have much time to think beyond the automatic. If you feel you may be entertained, please continue reading. If you want to learn some really useful tricks to test a two-node cluster's robustness, this article may be for you, too. If you're looking for information you can apply in the workplace without being escorted from the building by armed guards, proceed no further ;)

As today's post title obliquely suggests, I'm going to take another day to properly formulate my response to our "F" reading experiment (not to be confused with the anti-literacy initiative ;) that we began on Monday. I've received a number of very interesting comments on the subject of the article that got the whole idea rolling around in that butter churn in between my ears. Although none of the responses have radically changed my feelings on the subject, they have augmented them and provided some fresh perspective. Although I still intend to throw a little signature "meta" style into the post (because if we all read in the F formation, my post is going to have to play off of that to whatever degree I can manage :), I'm now reconsidering my original rough-draft and, possibly, working some additional angles into it. I've got some emails out there (as I always request permission to use other folks' opinions when they're kind enough to share) and hope to hear back soon. Worst case, I'll post the original tomorrow and add the comments as their own entities (attached, of course) at a later date.

Also, as this post's title overtly suggests, I spent most of my day testing cluster failover scenario's at work. I won't mention any proprietary or freeware brand names, as this post isn't specific enough to warrant the reference, but, after today's exercise (which, of course, I've had to do more than a couple different ways at a couple of different places of employment) I decided to put together a small comprehensive list of two-node cluster disaster/failure/failover scenarios that one should never push a cluster into production without performing.

It goes without saying that the following is a joke. Which is, of course, why I "wrote" it with my lips sealed ;)

Comprehensive Two-Node Cluster Failover Testing Procedure - v0.00001alpha

Main assumption: You have a two-node cluster all set up in a single rack, all service groups and resources are set to critical, no service groups or resources are frozen and pretty much everything should cause flip-flop (technical term ;)

1. Take down one service within each cluster service group (SG), one at a time. Expected result: Each cluster service group should fault and failover to the secondary node. The SG's should show as faulted in your cluster status output on the primary node, and online on the secondary.

2. Turn all the services, for each SG, back on, one by one, on the primary node. Expected result: All of the SG's should offline on the secondary node and come back up on the primary.

3. Do the same thing, but on the secondary. Expected result for the first test: Nothing happens, except the SG's show as faulted on the secondary node. Expected result for the second test: Nothing happens, except the SG's show as back offline on the secondary node.

4. Switch SG's from the primary to secondary node cleanly. Expected result: What did I just write?

5. Switch SG's from the secondary node back to the primary node cleanly. Expected result: Please don't make me repeat myself ;)

6. Unplug all heartbeat cables (serial, high priority ethernet, low priority, disk, etc) except one on the primary node. Expected result: Nothing happens except, if you're on the system console, you can't type anything anymore because the cluster is going freakin' nuts with all of its diagnostic messages!

7. Plug all those cables back in. Expected result: Everything calms down, nothing happens (no cluster failover) except you realize that you accidentally typed a few really harmful commands and may have hit enter while your screen was draped with garbage characters. The primary node may be making strange noises now ;)

8. Do the same thing on the secondary node. Expected result: No cluster failover, but the secondary node may now be making strange low beeping sounds and visibly shaking ;)

9. Pull the power cords out of the primary node. Expected result: Complete cluster failover to the secondary node.

10. Put the plugs back in. Expected result: Complete cluster failback to the primary node.

11. Do the same thing to the secondary node. Expected results for both actions: Absolutely nothing. But you knew this already. Are you just trying to waste the company's time? ;)

12. Kick the rack, containing the primary and secondary node, lightly. Expected results: Hopefully, the noises will stop now...

13. Grab a screwdriver and repeatedly stab the primary node. Expected Result: If you're careful you won't miss and cut yourself on the razor sharp rack mounts. Otherwise, everything should be okay.

14. Pull the fire alarm and run. Expected result: The guy you blame it on may have to spend the night in lock-up ;)

15. Tell everyone everything's fine and the cluster is working as expected. Expected result: General contentment in the ranks of the PMO.

16. Tell everyone something's gone horribly wrong and you have no idea what. Use the console terminal window on your desktop and export it via WebVNC so that everyone can see the output from it. Before exporting your display, start up a program you wrote (possibly using script and running it with the "-t" option to more accurately reflect realistic timing, although a bit faster. Ensure that this program runs in a continuous loop. Expected Result: General pandemonium. Emergency conference calls, 17 or 18 chat sessions asking for status every 5 seconds and dubious reactions to your carefully pitched voice, which should speak in reassuring terms, but tremble just slightly like you're a hair's breadth away from a complete nervous breakdown.

17. Go out to lunch. Expected Result: What do you care? Hopefully, you'll feel full afterward ;)

Cheers,

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Setting Up Basic IPMP On Solaris Unix

Hey there,

I thought I'd start this week out with a yawn ...I mean a bang ;) This is a topic we've never touched on, but one that is used very often in most large computer networks: IP multi-pathing. Simply put, allowing for the "outside world" to have more than one path to your networked server. The closest we've come, to date, would be our post on SunCluster monitoring. I suppose that could be chalked up to the fact that we're concentrating on a particular flavour of Unix, while most of my Linux postings are (or attempt to be) broad-based and satisfy as many distro's as possible.

In any event, here's a quick primer on getting IPMP set up on your Solaris host with the minimum amount of hardware and hassle. Quick and easy (I think ;)

1. Why should I set up IPMP? You don't have to. It's not a requirement of anything except SunCluster (assuming you want to get your setup officially certified - otherwise you can hack your way around that, too). The main benefit to you is that you'll have the comfort of knowing that users will still be able to connect to your host over the network even if one of your network cards goes to pot And, of course, that you won't have to lift a finger to make things go back to the way they were once the disaster is over. Simply put: You'll have network failover working for you in case of a network card failure. The user will never know you had an issue, as they will always be accessing your host via the same IP address.

2. What is required to use IPMP? Generally, you'll want to have as many failover points as possible (or reduce the number of network single-points-of-failure). This would mean having two separate network adapters. This can be done with one (with failover happening between virtual interfaces on the same NIC), but in order to attain your minimum two-points-of-failure, you should have two different NIC's with each of those residing on a different physical bus. It's good practice actually have those network cards hooked up to the network and the links verified before proceeding. Also, you should do most of this through a serial console or ALOM connection. Since you're dealing with networking failure, if you connect via a regular network connection to any IP on the host, there's a good chance you'll be dropped unexpectedly at some point during the process.

SPECIAL NOTE: IPMP, on its own, is not meant to protect you from an entire network segment going down. It will handle failover between NIC's, but they all need to be on the same network segment, or subnet, so if the network goes down (our example 10.10.10.0/Class C), you're still going to be offline.

3. Is it easy to setup IPMP on Solaris? Yeah :) Here's how:

4. At the PROM level, be sure to set the local-mac-address? variable to true

ok > setenv local-mac-address? true

you can also set this at the OS level, using the eeprom command:

host # eeprom local-mac-address?=true

If you choose to make this change using eeprom at the OS level, you should reboot your box before proceeding.

5. Install the required pkg files: This step is really easy, since the in.mpathd binary comes in the SUNWcsr (Core Solaris) pkg file, which your system won't run without. Hopefully it's already installed ;)

6. Alter the FAILURE_DETECTION_TIME value in the /etc/default/mpathd configuration file from 10000 milliseconds to 3 or 4000 milliseconds. This isn't necessary, but it will drop the failure detection time below 10 seconds, which might save you from having to answer any questions if you experience an unexplainable split-second network "burp" - of course, use this to your taste. Setting it too low may cause your virtual interfaces to flap back and forth constantly!

7. In your /etc/hosts file, include information for the "floating IP" (The one everyone else will use to connect to your system) and the other two physical IP's. This isn't absolutely necessary, but it can be helpful later on if you happen to forget what's what on any given machine.

Ex:
10.10.10.1 hostname-phys1
10.10.10.2 hostname-phys2
10.10.10.3 virtualhost

8. Modify the /etc/hostname.* files so that they include the proper information (this is where the meat of the configuration is done, in my opinion. If it's even up for a vote ;)

Ex:

/etc/hostname.hme0 (contents)

hostname-phys1 group ipmpgroup netmask + broadcast + deprecated -failover up
addif virtualhost netmask + broadcast + up

/etc/hostname.qfe0 (contents)

hostname-phys2 group ipmpgroup netmask + broadcast + deprecated -failover standby up

NOTE: You can, in our example, set the /etc/hostname.qfe0 file to be exactly the same as /etc/hostname.hme0 (with the only different being the "hostname-phys2" at the beginning of the entry) and it will work just as well. If you are using Veritas Cluster Server to do the failing over for you, it will not work unless you have one of your failover NIC's set to "standby" rather than "up"

9. Add both of your virtual IP's to an IP group (named ipmpgroup in this case - it can be anything you want) using ifconfig (You could also have done this before (added NIC's to groups before creating your /etc/hostname.* files), but, as long as we haven't started this baby up yet, the exact order doesn't matter):

host # ifconfig hme0 group ipmpgroup
host # ifconfig qfe0 group ipmpgroup

10. Activate: You can do this one of two basic ways (I'm sure there are more if you're creative about it ;) Reboot your machine or make the interfaces active manually. To activate manually, all you really need to do is copy and paste the contents of the two /etc/hostname.* files onto the command line, one after the other. If you have two lines of input in any, or either file, try to fit them all onto one line when executing from the command line.

11. Test: Run a continuous ping against your "floating IP" (10.10.10.3 in our case), and start pulling cables and resetting them. Do this one at a time, of course. If you remove both physical network connections at once, your machine will be off the net :)

You can also use the if_mpadm command to help with testing, if you prefer to "virtually" pull the plug on your physical interfaces. For instance "if_mpadm -d INTERFACE_NAME" will disable an interface, just as if you pulled the cable out. "if_mpadm -r INTERFACE_NAME" will reset the interface to it's "natural" state (however you have it set up; even if that's wrong ;) Check out the if_mpadm man page for more information on this command, although there's not much more to it.

If you're interested in jumping forward and getting into more advanced IPMP configurations, you can also check out the Sun Documentation Site and read the IP Network Multipathing Administration Guide at your leisure.

Cheers,

, Mike

Offlining, Failing Over And Switching in VCS

Hey there,

Today we're going to address a question that's asked commonly enough by foks who use Veritas Cluster Server: What's the difference between offlining, failing over and switching when dealing with service groups across multiple nodes? That doesn't seem like a very common question. I'm sure it's probably hardly ever phrased that way ;)

Anyway, to the point, all three options above are useful (hence your ability to avail yourself of them) and sufficiently distinct that you should be sure you're using the one you want to, depending on what ends you wish to achieve. All of this information is fairly general and should work on VCS for Linux as well as Unix.

We'll deal with them one bye one, outlining what they basically do, with a short example command line to demonstrate, where applicable. None of this stuff is hard to pick up and run with, as long as all the components of your VCS cluster are setup correctly. Occasionally, you may see errors if things aren't exactly perfect, like we noted in this post on recovering indefinitely hung VCS resources.

1. Offlining. The distinction to be made here, most plainly, is that, when you offline a resource or service group, you are "only" doing that. This differs from failover and switching in that the service group you offline with this particular option is not brought online anywhere else as a result. So, when you execute, for instance:

host # hagrp -offline YOUR_SERVICE_GROUP -sys host1

that service group, and its resources, generally, are taken offline on host1, and nothing else happens. If you're operating in an environment where systems don't run service groups concurrently (active/active), you will have effectively "shut down" that service group, and any services it provides, for the entire cluster.

2. Failing Over. This is more of a concept than any particular command. When you have your cluster setup to fail over, if a resource or service group, etc, goes offline on one node (host1, for instance) and it wasn't brought offline on purpose, VCS will naturally attempt to bring it online on the next available node (listed in your main.cf configuration file). Needless to say, if only one host is listed for a particular service group, its failure on one host will mean the failure of the entire service group. It also obviates the use of VCS in the first place ;)

3. Switching. This is what most folks want to do when they "offline" a service group, as in point 1. Although, since VCS automatically switches unexpectedly offlined resources on its own (when it's set up to), it's reasonable for someone new to the product to assume that offlining a service group would engage VCS in a switching activity. Unfortunately, this isn't the case. If you want to switch a service group from host1 to host2, for example, these would be the command options you'd want to give to hagrp:

host # hagrp -switch YOUR_SERVICE_GROUP -to host2 <--- Assuming you're running this from host1.

Hopefully this little guided mini-FAQ helped out with differentiating between the concepts. If you find the command line examples valuable, even better :)

Happy failing! ;) Over.


, Mike

linux unix internet technology