Hey There,
Today's post is going to steer away from the Linux and/or Unix Operating Systems just slightly, and look at a problem a lot of folks run into, but have problems diagnosing, when they first set up a Veritas cluster.
Our only assumptions for this post are that Veritas Cluster Server is installed correctly on a two-node farm, everything is set up to failover and switch correctly in the software and no useful information can be obtained via the standard Veritas status commands (or, in other words, the software thinks everything's fine, yet it's reporting that it's not working correctly ;)
Generally, with issues like this one (the software being unable to diagnose its own condition), the best place to start is at the lowest level. So, we'll add the fact that the physical network cabling and connections have been checked to our list of assumptions.
Our next step would be to take a look at the next layer up on the protocol stack, which would be the LLT (low latency transport protocol) layer (which, coincidentally, shares the same level as the MAC, so you may see it referred to, elsewhere, as MAC/LLT, or just MAC, when LLT is actually meant!) This is the base layer at which Veritas controls how it sends its heartbeat signals.
The layer-2 LLT protocol is most commonly associated with the DLPI (all these initials... man. These stand for the Data Link Provider Interface). Which brings us around to the point of this post ;)
Veritas Cluster Server comes with a utility called "dlpiping" that will specifically test device-to-device (basically NIC-to-NIC or MAC-to-MAC) communication at the LLT layer. Note that if you can't find the dlpiping command, it comes standard as a component in the VRTSllt package and is generally placed in /opt/VRTSllt/ by default. If you want to use it without having to type the entire command, you can just add that directory to your PATH environment variable by typing:
host # PATH=$PATH:/opt/VRTSllt;export PATH
In order to use dlpiping to troubleshoot this issue, you'll need to set up a dlpiping server on at least one node in the cluster. Since we only have two nodes in our imaginary cluster, having it on only one node should be perfect.
To set up the dlpiping server on either node, type the following at the command prompt (unless otherwise noted, all of these Veritas-specific commands are in /opt/VRTSllt and all system information returned, by way of example here, is intentionally bogus):
host # getmac /dev/ce:0 <--- This will give use the MAC address of the NIC we want to set the server up on (ce0, in this instance). For this command, even if your device is actually named ce0, eth0, etc, you need to specify it as "device:instance"
/dev/ce:0 00:00:00:FF:FF:FF
Next, you just have to start it up and configure it slightly, like so (Easy peasy; you're done :)
host # dlpiping -s /dev/ce:0
This command runs in the foreground by default. You can background it if you like, but once you start it running on whichever node you start it on, you're better off leaving that system alone so that anything else you do on it can't possibly affect the outcome of your tests. Since our pretend machine's cluster setup is completely down right now anyway, we'll just let it run in the foreground. You can stop the server, at any time, by simply typing a ctl-C:
^C
host #
Now, on every other server in the cluster, you'll need to run the dlpiping client. We only have one other server in our cluster, but you would, theoretically, repeat this process as many times as necessary; once for each client. Note, also, that for the dlpiping server and client setups, you should repeat the setup-and-test process for at least one NIC on every node in the cluster that forms a distinct heartbeat-chain. You can determine which NIC's these are by looking in the /etc/llttab file.
host # dlpiping -c /dev/ce:0 00:00:00:FF:FF:FF <--- This is the exact output from the getmac command we issued on the dlpiping server host.
If everything is okay with that connection, you'll see a response akin to a Solaris ping reply:
0:00:00:FF:FF:FF is alive
If something is wrong, the output is equally simple to decipher:
no response from 00:00:00:FF:FF:FF
Assuming everything is okay, and you still have problems, you should check out the support site for Veritas Cluster Server and see what they recommend you try next (most likely testing the IP layer functionality - ping! ;)
If things don't work out, and you get the error, that's great (assuming you're a glass-half-full kind of person ;) Getting an error at this layer of the stack greatly reduces the possible-root-cause pool and leaves you with only a few options that are worth looking into. And, since we've already verified physical cabling connectivity (no loose or poorly fitted ethernet cabling in any NIC) and traced the cable (so we know NICA-1 is going to NICB-1, as it should), you can be almost certain that the issue is with the quality or type of your ethernet cabling.
For instance, your cable may be physically damaged or improperly pinned-out (assuming you make your own cables and accidentally made a bad one - mass manufacturers make mistakes, too, though). Also, you may be using a standard ethernet cable, where a crossover (or, in some instances, rollover) cable is required. Of course, whenever you run into a seeming dead-end like this, double check your Veritas Cluster main.cf file to make sure that it's not in any way related to a slight error that you may have missed earlier on in the process.
In any event, you are now very close to your solution. You can opt to leave your dlpiping server running for as long as you want. To my knowledge it doesn't cause any latency issues that are noticeable (at least in clusters with a small number of nodes). Once you've done your testing, however, it's also completely useless unless you enjoy running that command a lot ;)
Cheers,
, Mike
Friday, May 30, 2008
Troubleshooting Veritas Cluster Server LLT Issues On Linux and Unix
Saturday, December 8, 2007
Adding Storage using Volume Manager in a Veritas Cluster
Today, we're going to take a look at adding extra disk to a Veritas Cluster using Veritas Volume Manager. We'll assume for the purposes of this post, that you've been asked to add 35gb of space to a Veritas volume (filesystem: /veritas/a - volume name: veritas_a). Now we'll walk, step by step, through determining whether we have the disk available to add, and adding it if we do.
The first thing you'll want to do is determine what disk group(s) the filesystem/volume belongs to. Generally, you'll be told this when you're asked to add disk to an existing system, but we'll just assume the very worst ;) In any event, even if you are told, it's good practice to verify that the filesystem/volume does, in fact, belong to the disk group(s) the request is asking to have you augment.
Now, since we're dealing with a Veritas Cluster, if the disk group is shared between more than one server node, you can run:
vxdctl -c mode
and that will show you what the master node is. This server is where you'll want to execute all of your Veritas Volume Manager commands. If the cluster is inactive, vxdctl will tell you that, and you'll need to execute your commands on the server you're already on.
Now, either on the Cluster's master node, or the machine you're on if the cluster is inactive, get a list of all the disk groups, like so:
vxdg list
NAME STATE ID
rootdg enabled 1199919191.1025.host1
mydg enabled 8373777737.1314.host2
Since the user requested more filespace under /veritas/a, we'll need to see what disk group this filesystem/volume belongs to:
df -kl | grep veritas
/dev/vx/dsk/mydg/veritas_a 34598912 24632328 9903320 72% /veritas/a
/dev/vx/dsk/mydg/veritas_b 34598912 649112 33687984 2% /veritas/b
/dev/vx/dsk/mydg/veritas_c 35350080 17696 35056360 1% /veritas/c
We can see that, under /dev/vx/dsk, mydg is showing as the disk group for our /veritas/a filesystem, and veritas_a is showing as the volume. In cases like this, it's a time saver if you just set the default group once on the command line, so you don't have to use the "-g" option for every command you run:
vxdctl defaultdg mydg
Now you'll run some standard commands to make sure that your system can see all the disk available to it and make sure that the data is fresh:
devfsadm -Cv (note that the cluster will only update the volume manager configuration for cluster nodes that use the mydg diskgroup -- we don't have to specify that here, because we defined it as the default above)
vxdctl enable (this is basically the second part of the same update. The devfsadm command is Solaris specific while vxdctl is Veritas specific)
Now we need to check for free space, to see if we can accommodate the request with what we have available to us at present:
vxdg -g mydg list <-- Again "-g mydg" is optional because it's the default.
DISK DEVICE TAG OFFSET LENGTH FLAGS
DISK1 c5t27d27s2 c5t27d27 69198720 1501440 n
DISK2 c5t27d30s2 c5t27d30 69198720 1501440 n
DISK3 c5t27d28s2 c5t27d28 69198720 1501440 n
This output is, admittedly, a pain to read, because it expects you to be able to calculate disk offsets in your head. For this output, if the offset were zero and length was 70700160 and flags were '-' then that DISK would have 35 GB of free space available to us. Here all the listed storage has been partially used up. Hence, there is not enough storage space available!. We can also verify this by running
vxdg free (optionally: vxdg -g mydg free) to see if there is any free space left in the disk group. Either way, we're out of luck, it would seem.
So, now we're going to have to look into Veritas' disk configuration and see if there isn't a spare disk (hope) that we can pick up and import into the mydg diskgroup so that we can add the space (One thing to note in the output below is that all online disk shows its disk group membership even though it's imported locally):
vxdisk -o alldgs list <--- (abbreviated to keep this post bearable ;)
DEVICE TYPE DISK GROUP STATUS
c0t0d0s2 sliced - - error
c0t2d0s2 sliced disk02 rootdg online
c1t2d0s2 sliced disk01 rootdg online
c5t27d40s2 sliced - (mydg200) online
c5t27d41s2 sliced - (mydg2) online
c5t27d300s2 sliced - (mydg300) online
c5t20d99s2 sliced - - error
Luckily, it appears that disk c5t20d99s2 is available and doesn't belong to any existing disk groups! We'll use the command line to initialize this free disk and add it to our disk group (mydg):
vxdiskadd c5t20d99s2
You will be asked if you want to initialize the disk (or re-initialize it, if it's already initialized). Answer yes to either option, to keep things simple. Then it will ask you what disk group you would like to add the disk to. Answer: mydg. It will ask you if you want to use a default disk name. Depending on your situation, this may be okay, but we'll say no and add it as "DISK4" to keep with our naming standard. Then you will be prompted if you want to add the disk as a spare for the mydg disk group. Answer no to this, or you won't be able to use it for the additional storage you want! Answer no again, when it asks you if you want to encapsulate the disk, and then answer yes when it asks you if you want to initalize it. You can then choose to perform a surface analysis before adding it (It says it's highly recommended but my experience is it's generally not necessary and takes a very very very long time). Phew... that wasn't too much to info to have to feed to a single command ;)
Now, we'll want to grow the mydg disk group's volume and filesystem size. We've added the new disk, so we have more space, but the user can't see it or use it at this point. Now, common wisdom is to use the vxassist command. It's common, because it can be a great command to use (very simple), but I prefer to use "vxresize" because, unless your filesystem is vxfs, running vxassist to grow the volume (/veritas_a) can cause serious issues when growing the filesystem (if, for instance, you're using the Veritas Volume Manager but have your volume filesystem type set to Solaris "ufs")!
Calculate the maximum length of your new volume using vxassist (this part is okay - forget about what I just said above - that only applies to using vxassist to do the actually growing):
vxassist -g mydg (optional, if default is already set) maxsize <--- This will be the NEW_LENGTH argument for the next command.
vxresize -F (vxfs or ufs,usually) -g mydg (optional, still) -x (to make sure that the operation fails if the volume doesn't increase in length) veritas_a NEW_LENGTH <-- The last time I checked, - and another reason I prefer this method - this command grows your filesystem with the volume so you don't need to run any more steps after this.
Of course, you can keep tabs on all this, after you're done typing, using the vxtask command. At no point during this entire operation will you need to unmount your filesystem. I believe it's "recommended" (because something could always go wrong), but I've never found it to be necessary (technically, it's not supposed to fail, but Veritas has to cover itself, just in case)
Once the task is finished, your "df -k /veritas/a" output should show your new disk available and ready for use!
Cheers,
, Mike
linux unix internet technology
Tuesday, September 30, 2008
How To Resolve Veritas Disk Group Cluster Volume Management Problems On Linux or Unix
Hey There,
Today we're going to look at an issue that, while it doesn't happen all that often, happens just enough to make it post-worthy. I've only seen it a few times in my "career," but I don't always have access to the fancy software, so this problem may be more widespread than I've been lead to believe ;) The issue we'll deal with today is: What do you do when disk groups, within a cluster, conflict with one another? Or, more correctly, what do you do when disk groups within a cluster conflict with one another even though all the disk is being shared by every node in the cluster? If that still doesn't make sense (and I'm not judging "you," it just doesn't sound right to me, yet ;) what do you do in a situation where every node in a cluster shares a common disk group and, for some bizarre reason, this creates a conflict between nodes in the cluster and some of them refuse to use the disk even though it's supposed to be accessible through every single node? Enough questions... ;)
Check out these links for a smattering of other posts we've done on dealing with Veritas Volume Manager and fussing with Veritas Cluster Server. Some of the material covered may be useful if you have problems with any of the concepts glossed over in the problem resolution at the end.
Like I mentioned, this "does" happen from time to time, and not for the reasons you might generally suspect (like one node having a lock on the disk group and refusing to share, etc). In fact, the reason this happens sometimes (in this very particular case) is quite interesting. Even quite disturbing, since you'd expect that this shouldn't be able to happen.
Here's the setup, and another reason this problem seems kind of confusing. A disk group (we'll call it DiskGroupDG1 because we're all about creativity over here ;) is being shared between 2 nodes in a 2 node cluster. Both nodes have Veritas Cluster Server (VCS) set up correctly and no other problems with Veritas exist. If the DiskGroupDG1 disk group is imported on Node1, using the Cluster Volume Manager (CVM), it can be mounted and accessed by Node2 without any issues. However, if DiskGroupDG1 is imported on Node2, using CVM, it cannot be mounted and/or access by Node1.
All things being equal, this doesn't readily make much sense. There are no disparities between the nodes (insofar as the Veritas Cluster and Volume Management setup are concerned) and things should be just peachy going one way or the other. So, what's the deal, then?
The problem, actually, has very little to do with VCS and/or CVM (Although they're totally relevant and deserve to be in the title of the post -- standard disclaimer ;). The actual issue has to do, mostly, with minor disk numbering on the Node1 and Node2 servers. What???
Here's what happens:
In the first scenario (where everything's hunky and most everything's dorey) the DiskGroupDG1 disk group is imported by CVM on Node1 and Node1 notices that the "minor numbers" of the disks in the disk group are exactly the same as the "minor numbers" on disk it already has mounted locally. You can always tell a disk's (or any other device's) minor number by using the ls command on Linux or Unix, like so:
<-- In this instance, the device's "major number" is 32 and the device's "minor number" is 0. Generally, with virtual disks, etc, you won't see numbers that low.host # /dev/dsk # ls -ls c0t0d0s0
2 lrwxrwxrwx 1 root root 41 May 11 2001 c0t0d0s0 -> ../../devices/pci@1f,4000/scsi@3/sd@0,0:a
host # /dev/dsk # ls -ls ../../devices/pci@1f,4000/scsi@3/sd@0,0:a
0 brw-r----- 1 root sys 32, 0 May 11 2001 ../../devices/pci@1f,4000/scsi@3/sd@0,0:a
Now, on Node1, since it recognizes this conflict on import, does what Veritas VM naturally does to avoid conflict; it renumbers the imported volumes ("minor number" only) so that the imported volumes won't conflict with volumes in another disk group that's already resident on the system it's managing. Therefore, when Node2 attempts to mount, with CVM, the command is successful.
In the second scenario (where thing are a little bit hunky, but not at all dorey), Node2 imports the DiskGroupDG1 disk group and none of the minor numbers in that disk group's volumes conflict with any of its local (or already mounted) disk. The disk group volumes are imported with no error, but, the "minor numbers" are not temporarily changed, either. You see where this is going. It's a freakin' train wreck waiting to happen ;)
Now, when Node1 attempts to mount, it determines there's a conflict, but can't renumber the "minor numbers" on the disk group's volumes (since they're already imported and mounted on Node2) and, therefore, takes the only other course of action it can think of and bails completely.
So, how do you get around this for once and all time? Well, I'm not sure it's entirely possible to anticipate this problem with a variable number of nodes in a cluster, all with independent disk groups and, also, sharing volume groups between nodes, although you could take simple measures to prevent it most of the time (like running ls against every volume in every disk group in a cluster every now and again and making sure no conflicts existed. The script should be pretty easy to whip up).
Basically, in this instance (and any like it), the solution involves doing what Veritas VM did in the first scenario; except doing it all-the-way. No temporary-changing of "minor numbers." For our purposes, we'd like to change them permanently, so that they never conflict again! It can be done in a few simple steps.
1. Stop VCS on the problem node first.
2. Stop any applications using the local disk group whose "minor numbers" conflict with the "minor numbers" of the volumes in DiskGroupDG1.
3. Unmount (umount) the filesystems and deport the affected disk group.
4. Now, pick a new "minor number" that won't conflict with the DiskGroupDG1 "minor numbers." Higher is generally better, but I'd check the minor numbers on all the devices in my device tree just to be sure.
5. Run the following command against your local disk group (named, aptly, LocalDG1 ;) :
host # vxdg reminor LocalDG1 3900 <-- Note that this number is the base, so every volume, past the initial, within the disk group will have a "minor number" one integer higher than the last (3900, 3901, etc)
6. Reimport the LocalDG1 disk group
7. Remount your filesystems, restart your applications and restart VCS on the affected node.
8. You don't have to, but I'd do the same thing on all the nodes, if I had a window in which to do it.
And, that would be that. Problem solved.
You may never ever see this issue in your lifetime. But, if you do, hopefully, this page (or one like it) will still be cyber-flotsam on the info-sea ;)
Cheers,
, Mike
Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.
Monday, March 23, 2009
How To Modify Live VCS Clusters On Linux And Unix Without Completely Alienating Your Co-Workers
Hey there,
A happy Monday to everyone and here's hoping you had a halfway-decent weekend. There's nothing quite like 2 days of misery leading up to a Monday, which is why I like to do something fun every weekend that I can. This weekend I slept a lot. It was a blast. I'm awake now, but I'm getting right back up on that horse as soon as I'm done proofing this post ;)
If you work on Veritas Cluster Server a lot (VCS from here on in, and on out ;), or even if you just work on them a little (or sit next to someone who does), you probably know how incredibly annoying it can be to make fixes to them once you've got them up and online. They are, almost by definition, expected to provide high availability for whatever twisted perverse software you run on them and "high availability" almost always translates into "high accountability." Even though a perfectly constructed cluster (of however many individual nodes) is supposed to "make your life easier," I've found that this is hardly ever the case. People tend to go ape$h1t whenever a cluster node experiences an issue, even when it's something as mundane as a flip-flop between NIC's on a single node that are sharing a local floating-IP that is but one of many virtual IP's that the entire cluster uses to float its own virtual IP on top of. If you work on clusters, you're statistically likely to be either incredibly apathetic about these sorts of issues, or a ring-tone away from a total nervous breakdown ;)
In the event that you do have to work on an active Veritas cluster (whether on Linux or Unix or even Windows), there are a few ways you can keep those alarm bells (that you have no choice but to trigger in order to get your work done) from reaching their intended recipients. The optimal end result, of course, being that you get your job done in a timely and efficient manner and your co-workers (who aren't in your department and don't need to know - HUGE ASSUMPTION ;) don't lose any sleep worrying about the money the company is conceivably losing while nothing really "bad" is happening.
We'll run down this short list in order from "best practice" to crude severing of email and other notification avenues and/or faking "upness." You may need to do the group of "last things" on this list, no matter how great your setup is. If you're working in an environment where outside agents (HP OpenView, etc) report on your cluster's condition, turning off notification the proper way may not do you any good at all, anyway ;)
Prior to making any changes to your main.cf (assuming you're just changing it with the intention of restoring it to its exact same state), I would recommend either saving off your main.cf file using simple copy:
host # cp main.cf main.cf.`hostname`.`date +%m%d%y%H%M%S`
or creating a command file from your cf file, so that you can just rebuild it again if anything terrible happens, like so:
host # hacf -cftocmd . <-- assuming you're in the main Veritas configuration directory. If you're not, and even if you are, you can pass the hacf command the full path to your configuration directory, if you prefer:
host # hacf -cftocmd /etc/VRTSvcs/conf/config <-- The same thing as the previous command, in most situations.
You can then use the .cmd file to recreate your .cf file using the "-cmdtocf" argument, with the rest of the hacf command generally remaining the same.
1. The first thing you would "normally" do (on the primary node in your cluster), would be to make sure that VCS's notification system redirects its error messages to an alternate source (like your email instead of everyone else's). So, assuming you have the NotifierMngr resource set up to mail errors to everyone@mycompany.com, you can easily modify this without alarming anyone, like so:
host # haconf -makerw
host # hares -modify NotifierMngr SmtpRecipients "me@nobodyelse.com = Error"
host # haconf -dump -makero
Of course, your setup may be more, or less, complicated, but that should give you the gist of it.
2. The next thing you should consider (still going by the VCS playbook) is that some of your resources may have specific "owners" assigned. These people will be notified of issues with certain resources, as well, and should be neutralized whether or not it actually makes a difference ;) You can do this in approximately the exact same way, like this:
host # haconf -makerw
host # hares -modify SOME_RESOURCE ResourceOwner "me@nobodyelse.com"
host # haconf -makero
3. Those first two steps will only insulate you from VCS's notification services. You still have to worry about the system's you're running on. This step can be a big headache (especially when you see what's coming up) and won't always do you any good. I'm including it for completeness' sake. The next thing you could do, would be to disable sendmail/mail on your systems. This can be done a variety of different ways, depending upon what OS and version you're running, but (generally), this simple method should do the trick:
a. Determine the PID's of your active sendmail/mailer daemons and kill them. Alternately, play it safe and shut them down with their respective shutdown scripts (not as much fun ;)
b. Move your sendmail/mail configuration files from their regular location to a different one (or just rename them) so that they can't restart if they try to.
c. Make sure nothing is running on your system, in cron, perhaps, (like cfengine) that will pro-actively fix the problem you've created to keep mail from getting off of the servers.
4. As noted, step 3 can be a hassle and may be a huge waste of time. Now you have to consider that outside, and inside, systems may also be monitoring your cluster's health. Programs like HP OpenView may have agents installed on your local systems and other computer systems may be set up outside of your cluster in order to test its availability from alternate locations. Any combination of these may still trigger errors and alerts once you start faulting a VCS resource on purpose.
At this point you can go with what might be the best option of them all (combined with steps 1 and 2): Stop your cluster dead, properly, so that none of the resources show as faulted or are affected in any way. You can do this very simply by running the following on your cluster's primary node:
host # hastop -all -force
This will bring down your cluster so that you can work on the configuration file, etc, but will not take down any of the resources that it manages (which should buy you some time, since none of them will report as "faulted"). There's still the outside chance that someone (or something) may be monitoring the "state" of your cluster externally. In that cause, you're probably screwed and should let whomever's keeping tabs know what you intend to do so that they will feel more comfortable ignoring it ;) Keep in mind, of course, that, when you bring your nodes back online, you should try to bring up the primary node first. This will avoid any issues with your primary node doing a "remote build" of its configuration file from any secondary, tertiary, etc, nodes. Again, depending on the complexity of your setup, this may make very little difference.
5. (Bonus step - Not recommended) Some folks prefer to "freeze" the cluster when they want to do maintenance. This is perfectly acceptable practice, as it will allow you to operate on resources within service groups in VCS without risking taking down the cluster or causing undesired failover of resources. It should be noted, however, that if you cause any of the basic functionality built into a resource to fail (which may happen if you're re-tooling a resource to a good degree), that resource will still show up in VCS as faulted until its basic functionality (as understood by VCS) is restored. This faulted state will most probably cause the NotifierMngr and ResourceOwner's to be notified of the error condition!
And, there you have it. I realize this was somewhat of a quick glossing over of the subject, but, as those of us who work on VCS have probably quite-painfully learned, trying to explain VCS (even at the most basic level) can become a huge task if you want to cover every possible aspect or angle. That's why so many 5000 page books are published every year on how to open the box and find the serial number ;)
Cheers,
, Mike
Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.
Wednesday, April 8, 2009
A Simple Way To Get Better Symantec NetBackup Support For VCS
Hey there,
This post is a little gift for those of you out there who run clustered NetBackup on VCS to go along with our earlier posts on emulating the NetBackup activity monitor on the command line and copying and modifying NetBackup policies, schedules and clients between hosts:)
As most of you who've run NetBackup in a VCS cluster may know, sometimes getting support for NetBackup (used with VCS) can be a royal pain; never mind that NetBackup (Symantec or, previously, Veritas) can be "purchased" as a VCS cluster add-on. The problem (and, to me, this is inexplicable, since NetBackup used to be a Veritas product) is that, if you ever experience issues with your NetBackup cluster and need to call Symantec for help, they generally have an entirely contrary attitude if they find out that you're actually running NetBackup in a VCS cluster that includes other "certified" add-on components (?)
And, yes, you read that correctly: You're going to have problems getting reasonable support for NetBackup if you've incorporated any other VCS modules in your cluster. If you want to get decent support for your clustered NetBackup setup (in my experience) your cluster should "only" include the NetBackup add-on Module. This is hardly a realistic scenario, but (for instance), if you're running the Oracle and NetBackup add-on modules in the same cluster, your support experience will be less than pleasant.
Hurdling right over the argument that Symantec "should" enthusiastically support NetBackup in a Cluster, no matter what else is clustered with it, we'll proceed directly to the solution. It's not exactly Kosher, but it works :)
Let's say, for instance, that you have a 2 node VCS cluster. Let's also say that your cluster has both Oracle and NetBackup add-on modules installed and operational. Then, finally, let's say the NetBackup component begins to fail miserably. What then?
As a little aside, in the spirit of Airplane:"For instance, that you have a 2 node VCS cluster. That your cluster has both Oracle and NetBackup add-on modules installed and operational. The NetBackup component begins to fail miserably. What then?"
Sorry; couldn't help it ;)
Basically, in the situation described above (and repeated, per instruction, directly below ;) you'll get half-hearted support for your NetBackup problem, at best, if (and this is important) you "tell the truth about your setup." I'm not advocating dishonesty (although, as they say, it "is" the second best policy ;), but, if you want Symantec to fully support your NetBackup issue and not waste a lot of time pointing fingers at Veritas and, generally, giving you the run-around, you'll need to assure them that you're "only" running NetBackup on your cluster (no matter how cost-inefficient that may be ;)
Please be sure, before you use the method below, that you have conclusively ruled out other add-on modules in your cluster as being a part of (or the entirety of) the problem.
Once you've determined that NetBackup is, indeed, the issue, you'll need to call Symantec for that support you're "paying for" (another thing that makes this whole ordeal ridiculous). And, in order to get that full support, you'll need to implement the following two methods:
1. Always play your cards very close to the vest. Don't divulge unnecessary information and only provide very specific responses to questions asked. For instance, if NetBackup Support wants to see the permissions of your main.cf file, only do an "ls -l" of the one specific main.cf. If they want to see the contents, only show them that "one" file's contents. And, of course, don't let them "take over" your machine via WebEx or any other "remote control" mechanism.
2. Make sure that you have a "prop" main.cf to show them (This is the gift attached below. It may only be a template that needs customizing to suit your own site's naming conventions, etc, but it's a nice gesture, at least ;)
Feel free to manipulate the following workable example as necessary. The only things you should need to change would be the specific "names" that you've given your resources (and NIC device names, IP addresses, etc). And, remember, this main.cf is for "support eyes only" - it probably doesn't do everything you need to do in your "real" cluster configuration (which is why you should be sure you rename it to something else before you fire up your first node ;), but it "does" do the most important thing of all when it comes to getting support for NetBackup on VCS: It describes a VCS cluster that contains the NetBackup add-on module and no others!
Enjoy :)
include "types.cf"
include "/usr/openv/netbackup/bin/cluster/vcs/NetBackupTypes.cf"
cluster MyClusterName (
)
system MyHost1 (
)
system MyHost2 (
)
group MyMainServiceGroup (
SystemList = { MyHost1 = 0, MyHost2 = 1 }
AutoStartList = { MyHost1 }
)
// resource dependency tree
//
// group MyMainServiceGroup
// {
// }
group MyNetBackupServiceGroup (
SystemList = { MyHost1 = 0, MyHost2 = 1 }
AutoStartList = { MyHost1, MyHost2 }
)
DiskGroup MyNetBackupDiskGroup (
DiskGroup = netbackupdg
)
IP MyNetBackupIP (
Device = ce0
Address = "10.99.99.99"
NetMask = "255.255.255.0"
)
Mount MyNetBackupMount (
MountPoint = "/shared/opt/VRTSnbu"
BlockDevice = "/dev/vx/dsk/netbackupdg/sharedoptvrtsnbu"
FSType = vxfs
MountOpt = largefiles
FsckOpt = "-y"
)
NIC MyNetBackupNIC (
Device = ce0
)
NetBackup MyNetBackupServer (
ServerName = MyNetBackupServer
ServerType = NBUMaster
)
Volume MyNetBackupVolume (
Volume = sharedoptvrtsnbu
DiskGroup = netbackupdg
)
MyNetBackupIP requires MyNetBackupNIC
MyNetBackupMount requires MyNetBackupVolume
MyNetBackupServer requires MyNetBackupIP
MyNetBackupServer requires MyNetBackupMount
MyNetBackupVolume requires MyNetBackupDiskGroup
// resource dependency tree
//
// group MyNetBackupServiceGroup
// {
// NetBackup MyNetBackupServer
// {
// IP MyNetBackupIP
// {
// NIC MyNetBackupNIC
// }
// Mount MyNetBackupMount
// {
// Volume MyNetBackupVolume
// {
// DiskGroup MyNetBackupDiskGroup
// }
// }
// }
// }
, Mike
Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.
Monday, February 9, 2009
Getting Faster Support For Your VCS-Clustered NetBackup Servers
Hey There,
Today's post is a little trick that anyone running Veritas/Symantec NetBackup (Linux or Unix) on VCS - Veritas Cluster Server - should know. As the title suggests, doing this one little thing will almost guarantee you more responsive support from Symantec (given the highly specific situation outlined in the first sentence, of course ;). The funny thing, though, is that many people have this problem already; they just may not have had to deal with Symantec, with regards to it, yet.
The problem you have (or will have) to face is that Symantec "does not" fully support VCS Clustered NetBackup installations if that same cluster has other Service Groups and/or major resources active on it. I'd almost say that they don't support it at all, but that's not entirely true (just keep complaining. You'll see ;). However, Symantec's stated position is that they really "don't" support it at all (and this is only hearsay, of course. Something I've heard while on the phone with another person who answered the phone at the number listed on our contract for Symantec NetBackup Support. So, of course, I may not be entirely correct, here ...).
Basically, what this means is that Symantec has VCS Cluster support for NetBackup (They own both NetBackup and Veritas Cluster Server), have VCS modules (types files, pkg's, etc) for NetBackup and even have full documentation online to help you setup your NetBackup Server(s) in VCS. Even given that fact, they do not support NetBackup in a VCS cluster that also runs, say, NFS or Oracle or even a shared mountpoint that isn't NB-related. NOTE: They especially don't support IPmultiNIC_B (I'm not sure why, since NIC auto-failback seems like a great idea to me)!! Their contention is that NetBackup is meant to function on VCS as the "only service or resource." This is fine, if you can spare 2 or 3 servers to just run NetBackup (larger companies, and data security/storage/backup companies, probably have no issue with this). However, most businesses can't afford to spare the extra servers "just" for NetBackup and will often have NetBackup in the same VCS cluster that runs an NFS group/resource, other shared mountpoints, web servers, databases, etc. Hopefully not "all" of those on one machine, but times are tight; you never know ;)
Now, here's the trick (it's not even a trick, really, just some good advice to help you out) to getting speedy support from Symantec when you call them about an issue with NetBackup on VCS: You're going to have to do a little bit of prep work and (even if it hurts) lie a little bit. If you're not comfortable "bending the truth," just remember that dishonesty is the second-best policy. Things could be worse ;)
1. Before you call, have a backup main.cf ready to go. And, this is very important if you're working on a cluster that's been around for a while, make sure that you do one of two things when Symantec Support asks for anything like an "ls" of your /etc/VRTSvcs/conf/config directory:
a. Do a very specific ls:
host # ls /etc/VRTSvcs/conf/config/main.cf <-- They don't need to see the other 500 hundred backups that are three times the size of your faked-up file ;)
b. Create a staging directory, copy only the minimal files you want into it, and then give them an "ls" of that:
host # cd /etc/VRTSvcs/conf/config/stage <-- Don't show them this!
host # ls -l *
2. As noted above, you need a clean main.cf for NetBackup Support, or you're going to end up getting bounced around from department to department, at best. If you're running anything aside from NetBackup in one cluster, you'll need to remove it (not literally, of course). Here's a quick example (This setup is going to be very basic, just to keep it short - which means the commented dependency trees that VCS puts into the conf file are removed, for this example, as well) - BTW, feel free to use this exact main.cf (just change a few names and numbers) if it works for you. It's been verified on the system I'm goofing with, but it probably won't work out-of-the-box on your system :)
Since this example is so long, I'll bid you farewell for the day, now. Hope this helps you if you ever get in a "We don't support such-and-such" or "You really shouldn't be doing such-and-such" or "It's a VCS Problem <--> It's a NetBackup Problem" sort-of-situation. And, remember, even if you goof up once and tell them the truth (who amongst us isn't guilty of being upfront every once in a while ;), you can always ask them to cancel your request ("Oh wait, I see what the problem is... Sorry, you can close the ticket" etc) and then call back later and go through normal channels so you'll hopefully get a different person (and, when you do, be sure to intimate that the problem you're having now is totally unrelated to the previous ticket, if they're efficient and happen to look for, and find, it. It's worth a shot ;)
Cheers,
--- Stripped-Down main.cf ---include "types.cf"
include "/usr/openv/netbackup/bin/cluster/vcs/NetBackupTypes.cf"
cluster VCScluster1 (
UserNames = { admin = EncryptedPassword }
ClusterAddress = "10.10.10.199"
Administrators = { admin }
)
system VCShost1 (
)
system VCShost2 (
)
group ClusterService (
SystemList = { VCShost1 = 0, VCShost2 = 1 }
AutoStartList = { VCShost1 }
)
IP webip (
Device = bge0
Address = "10.10.10.199"
NetMask = "255.255.255.0"
)
NIC webnic (
Device = bge0
)
webip requires webnic
group SG_VCSNB1 (
SystemList = { VCShost1 = 0, VCShost2 = 1 }
AutoStartList = { VCShost1 }
)
IPMultiNIC nbuIP (
Address = "10.10.10.199"
MultiNICResName = mnic
)
MultiNICA mnic (
Device @VCShost1 = { bge1 = "10.10.10.197", bge2 = "10.10.10.198" }
Device @VCShost2 = { bge2 = "10.10.10.198", bge1 = "10.10.10.197" }
ArpDelay = 5
IfconfigTwice = 1
PingOptimize = 0
)
DiskGroup VCShostDG (
DiskGroup = VCShostDG
)
Mount MNT_NBmount_VCSNB1 (
MountPoint = "/opt/VRTSnbu"
BlockDevice = "/dev/vx/dsk/VCShostDG/NBmount"
FSType = vxfs
MountOpt = largefiles
FsckOpt = "-y"
)
NetBackup APP_nbu_VCSNB1 (
Critical = 0
ServerName = NBU_Server
ServerType = NBUMaster
)
Volume VOL_NBmount_VCSNB1 (
Volume = NBmount
DiskGroup = VCShostDG
)
APP_nbu_VCSNB1 requires nbuIP
APP_nbu_VCSNB1 requires MNT_NBmount_VCSNB1
MNT_NBmount_VCSNB1 requires VOL_NBmount_VCSNB1
VOL_NBmount_VCSNB1 requires VCShostDG
nbuIP requires mnic
, Mike
Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.
Friday, November 14, 2008
Basic Veritas Cluster Server Troubleshooting
Hey There,
For the end of the week, we're going to continue with the theme of sparse-but-hopefully useful information. Quick little "crib sheets" (preceding by paragraphs and paragraphs of stilted ramblings by the lunatic who pens this blog's content ;) For this Friday, we're going to come back around and take a look at Veritas Cluster Server (VCS) troubleshooting. If you're interested in more specific examples of problems, solutions and suggestions, with regards to VCS, check out all the VCS related posts from the past year or so. Hopefully you'll be able to find something useful in our archives, as well. These simple suggestions should work equally well for Unix as well as Linux, if you choose to go the VCS route rather than some less costly one :)
And, here we go again; quick, pointed bullets of info. Bite-sized bits of troubleshooting advice that focus on solving the problem, rather than understanding it. That sounds awful, I know, but, sometimes, you have to get things done and, let's face it, if it's the job or your arse, who cares about the why? Leave that for philosophers and academics. Plus, since you fix problems so fast, you'll have plenty of time to read up on the ramifications of your actions later ;)
The setup: Your site is down. It's a small cluster configuration with only two nodes and redundant nic's, attached network disk, etc. All you know is that the problem is with VCS (although it's probably indirectly due to a hardware issue). Something has gone wrong with VCS and it's, obviously, not responding correctly to whatever terrible accident of nature has occurred. You don't have much more to go on than that. The person you receive your briefing from thinks the entire clustered server set up (hardware, software, cabling, power, etc) is a bookmark in IE ;)
Now, one by one, in a fashion that zigs on purpose, but has a tendency to zag, here are a few things to look at right off the bat when assessing a situation like this one. Perhaps next week, we'll look into more advanced troubleshooting (and, of course, you can find lots of specific "weird VCS problem" solutions in our VCS archives)
1. Check if the cluster is working at all.
Log into one of the cluster nodes as root (or a user with equivalent privilege - who shouldn't exist ;) and run
host1 # hastatus –summary
or
host1 # hasum <-- both do the same thing, basicallyEx:
host1 # hastatus -summary
-- SYSTEM STATE
-- System State Frozen
A host1 RUNNING 0
A host2 RUNNING 0
-- GROUP STATE
-- Group System Probed AutoDisabled State
B ClusterService host1 Y N OFFLINE
B ClusterService host2 Y N ONLINE
B SG_NIC host1 Y N ONLINE
B SG_NIC host2 Y N OFFLINE
B SG_ONE host1 Y N ONLINE
B SG_ONE host2 Y N OFFLINE
B SG_TWO host1 Y N OFFLINE
B SG_TWO host2 Y N OFFLINE
Clearly, your situation is bad: A normal VCS status should indicate that all nodes in the cluster are “RUNNING” (which these are). However, it should also show all service groups as being ONLINE on at least one of the nodes, which isn't the case above with SG_TWO (Service Group 2).
2. Check for cluster communication problems. Here we want to determine if a service group is failing because of any heartbeat failure (The VCS cluster, that is, not another administrator ;)
Check on GAB first, by running:
host1 # gabconfig -aEx:
host1 # gabconfig -a
GAB Port Memberships
===============================================================
Port a gen 3a1501 membership 01
Port h gen 3a1505 membership 01
This output is okay. You would know you had a problem at this point if any of the following conditions were true:
if no port “a” memberships were present (0 and 1 above), this could indicate a problem with gab or llt (Looked at next)
If no port "h" memberships were present (0 and 1 above), this could indicate a problem with had.
If starting llt causes it to stop immediately, check your heartbeat cabling and llt setup.
Try starting gab, if it's down, with:
host1 # /etc/init.d/gab start
If you're running the command on a node that isn't operational, gab won't be seeded, which means you'll need to force it, like so:
host1 # /sbin/gabconfig -x
3. Check on LLT, now, since there may be something wrong there (even though it wasn't indicated above)
LLT will most obviously present as a crucial part of the problem if your "hastatus -summary" gives you a message that it "can't connect to the server." This will prompt you to check all cluster communication mechanisms (some of which we've already covered).
First, bang out a quick:
host1 # lltconfig
on the command line to see if llt is running at all.
If llt isn't running, be sure to check your console, system messages file (syslog, possibly messages and any logs in /var/log/VRTSvcs/... - usually the "engine log" is worth a quick look) As a rule, I usually do
host1 # ls -tr
when I'm in the VCS log directory to see which log got written to last, and work backward from there. This puts the most recently updated file last in the listing. My assumption is that any pertinent errors got written to one of the fresher log files :) Look in these logs for any messages about bad llt configurations or files, such as /etc/llttab, /etc/llthost and /etc/VRTSvcs/conf/sysname. Also, make sure those three files contain valid entries that "match" <-- This is very important. If you refer to the same facility by 3 different names, even though they all point back to the same IP, VCS can become addled and drop-the-ball.
Examples of invalid entries in LLT config files would include "node numbers" outside the range of 0 to 31 and "cluster numbers" outside the range of 0 to 255.
Now, if LLT "is" running, check its status, like so:
host # lltstat -wn <-- This will let you know if llt on the separate nodes within the cluster can communicate with one another.
Of course, verify physical connections, as well. Also, see our previous post on dlpiping for more low-level-connection VCS troubleshooting tips.Ex:
host1 # lltstat -vvn
LLT node information:
Node State Link Status Address
0 prsbn012 OPEN
ce0 DOWN
ce1 DOWN
HB172.1 UP 00:03:BA:9D:57:91
HB172.2 UP 00:03:BA:0E:F1:DE
HB173.1 UP 00:03:BA:9D:57:92
HB173.2 UP 00:03:BA:0E:D0:BE
1 prsbn015 OPEN
ce3 UP 00:03:BA:0E:CE:09
ce5 UP 00:03:BA:0E:F4:6B
HB172.1 UP 00:03:BA:9D:5C:69
HB172.2 UP 00:03:BA:0E:CE:08
HB173.1 UP 00:03:BA:0E:F4:6A
HB173.2 UP 00:03:BA:9D:5C:6A
host1 # cat /etc/llttab <-- pardon the lack of low-pri links. We had to build this cluster on the cheap ;)
set-node /etc/VRTSvcs/conf/sysname
set-cluster 100
link ce0 /dev/ce:0 - ether 0x1051 -
link ce1 /dev/ce:1 - ether 0x1052 -
exclude 7-31
host1 # cat /etc/llthosts
0 host1
1 host2
host1 # cat /etc/VRTSvcs/conf/sysname
host1
If llt is down, or you think it might be the problem, either start it or restart it with:
host1 # /etc/init.d/llt.rc start
or
host1 # /etc/init.d/llt.rc stop
host1 # /etc/init.d/llt.rc start
And, that's where we'll end it today. There's still a lot more to cover (we haven't even given the logs more than their minimum due), but that's for next week.
Until then, have a pleasant and relaxing weekend :)
Cheers,
, Mike
Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.
Monday, December 17, 2007
Simple Sun Cluster Monitoring Script
Hey There,
Today, I've included a script I wrote to monitor a small SunCluster 3.1 environment (Running Oracle Parallel Server - OPS) we have set up at our shop. It basically runs through the output of various "scstat" command variations and reports on all combinations of errors it encounters. I built some functionality in it to be as specific as possible about the error states and to make sure that it returns an "All Clear" once the error condition no longer exists.
I also wrote this to run in cron. I run it every 5 minutes. Any time period is acceptable, depending upon what you need, and, of course, you could always put this script in a wrapper so that it runs constantly (although that generally necessitates writing another script to make sure that this script is running, restarting it as necessary and vice versa - For Veritas Cluster Server fans out there, this is a cheap and quick way to imitate the relationship between had and hashadow).
This was also written for a small environment (2 machines with 2 "live" network connections each). I didn't include monitoring of the heartbeats, since the script is meant to be run locally (with "exactly" the same values in the customizable section) on all nodes in a cluster, and scstat's indication of failure on any of these tests is, in and of itself, a guarantee that loss of a heartbeat connection is the very least of your problems ;)
Enjoy!
This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License#!/usr/bin/perl
###################################
# suncluster_mon - check cluster health
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
# simple parser for:
# 1. scstat -n
# 2. scstat -i
# 3. scstat -g
# 4. scstat -D
#
###################################
###################################
# CUSTOMIZE SETTINGS HERE
# $debug can be set with the -d
# switch when invoking the script
###################################
$primary_node="";
$secondary_node="";
$primary_eth="";
$secondary_eth="";
$date=`/usr/bin/date "+%A, %B %d - %H:%M:%S"`;
$hostname=`/usr/bin/hostname`;
$debug=0;
###################################
# PLEASE DO NOT EDIT BELOW
# EXCEPT MAIL SETTINGS AT END
###################################
chomp($date);
if ( $#ARGV >= 0 ) {
foreach $arg (@ARGV) {
if ( $arg =~ /-d/ ) {
$debug=1;
print "Debug Output Set\n";
} else {
push(@nogo, $arg)
}
}
if ( @nogo > 0 ) {
print "Unrecognized options: @nogo\n";
print "Ignoring and continuing\n";
}
}
@scstatn=`/usr/cluster/bin/scstat -n|/usr/bin/egrep 'node:'`;
for $nodestat (@scstatn) {
if ( $nodestat =~ /Cluster node:/ ) {
$nodestat =~ s/Cluster node://;
@nodestat = split(" ", $nodestat);
if ( $nodestat[1] eq "Offline" && $nodestat[0] eq $secondary_node ) {
if ( $nodefailure ) {
$alert="\nCluster Alert:\n$primary_node and $secondary_node are both in Offline state. Cluster Failed\n";
} else {
$alert="\nCluster Alert:\n$primary_node is in Offline state. $secondary_node is Online - Cluster Crippled\n";
}
$alert="\nCluster Alert:\n$nodestat[0] has switched to $nodestat[1] Status - Cluster Crippled - Checking For Dual Failure\n";
$nodefailure=1;
push(@alert, $alert);
} elsif ( $nodestat[1] eq "Offline" && $nodestat[0] eq $primary_node ) {
if ( $nodefailure ) {
$alert="\nCluster Alert:\n$primary_node and $secondary_node are both in Offline state. Cluster Failed\n";
} else {
$alert="\nCluster Alert:\n$primary_node is in Offline state. $secondary_node is Online - Cluster Crippled\n";
}
$alert="\nCluster Alert:\n$nodestat[0] has switched to $nodestat[1] Status - Cluster Crippled - Checking For Dual Failure\n";
$nodefailure=1;
push(@alert, $alert);
} else {
if ( $debug ) {
$alert="Cluster Info: $nodestat[0] and $nodestat[1] OK\n";
push(@debug, $alert);
}
$beautiful++;
}
} else {
print "WTF ---- SCSTAT -D\n";
}
}
if ( $beautiful == 2 ) {
if ( $debug ) {
$alert="Cluster Info: Both nodes OK\n";
push(@debug, $alert);
}
}
@scstati=`/usr/cluster/bin/scstat -i|/usr/bin/egrep 'Group:'`;
for $ipmpstat (@scstati) {
if ( $ipmpstat =~ /IPMP Group:/ ) {
$ipmpstat =~ s/IPMP Group://;
@ipmpstat = split(" ", $ipmpstat);
if ( $ipmpstat[0] == $primary_node ) {
if ( $ipmpstat[3] eq $primary_eth && $ipmpstat[4] ne "Online" ) {
$alert="\nIPMP Alert:\n$ipmpstat[3] is in $ipmpstat[4] mode on $ipmpstat[0] - Checking for Online of $ipmpstat[0] $secondary_eth\n";
$ethprimaryfailure=1;
push(@alert, $alert);
} elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Online" && $ethprimaryfailure == 1 ) {
$alert="\nIPMP Alert:\n$ipmpstat[0] $secondary_eth is not in an Online state on $ipmpstat[0] - IPMP Group Down on both Interfaces!\n";
push(@alert, $alert);
} elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Standby" && ! $ethprimaryfailure ) {
$alert="\nIPMP Alert:\n$ipmpstat[3] is in $ipmpstat[4] mode on $ipmpstat[0] - Failover interface is not UP - IPMP Group Crippled on $ipmpstat[0]!\n";
push(@alert, $alert);
} else {
if ( $debug ) {
$alert="IPMP Info: IPMP Group $ipmpstat[0] $ipmpstat[3] OK\n";
push(@debug, $alert);
}
}
} elsif ( $ipmpstat[0] == $secondary_node ) {
if ( $ipmpstat[3] eq $primary_eth && $ipmpstat[4] ne "Online" ) {
$alert="\nIPMP Alert:\n$primary_eth is in $ipmpstat[4] mode on $ipmpstat[0] - Checking for Online of $ipmpstat[0] $secondary_eth\n";
$ethprimaryfailure=1;
push(@alert, $alert);
} elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Online" && $ethprimaryfailure == 1 ) {
$alert="\nIPMP Alert:\n$ipmpstat[0] ipmpstat[3] not in an Online state on ipmpstat[0] - IPMP Group Down on both Interfaces!\n";
push(@alert, $alert);
} elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Standby" && ! $ethprimaryfailure ) {
$alert="\nIPMP Alert:\n$ipmpstat[3] is in $ipmpstat[4] mode on $ipmpstat[0] - Failover interface is not UP - IPMP Group Crippled on $ipmpstat[0]!\n";
push(@alert, $alert);
} else {
if ( $debug ) {
$alert="IPMP Info: IPMP Group $ipmpstat[0] $ipmpstat[3] OK\n";
push(@debug, $alert);
}
}
}
} else {
print "WTF ---- SCSTAT -I\n";
}
}
@scstatg=`/usr/cluster/bin/scstat -g|/usr/bin/egrep 'Resource:|Resources:|Group:'`;
for $rgstat (@scstatg) {
if ( $rgstat =~ /Resources:/ ) {
$rgstat =~ s/Resources://;
@rgstat = split(" ", $rgstat);
if ( @rgstat != 5 ) {
shift @rgstat;
$alert="\nOracle Resource Alert:\nOracle resource group $rgstat[0] is not running all resources on the cluster\n--Only running -- @rgstat\n";
push(@alert, $alert);
} else {
if ( $debug ) {
$alert="Oracle Resource Info: Oracle resource group $rgstat[0] OK -- @rgstat OK\n";
push(@debug, $alert);
}
}
} elsif ( $rgstat =~ /Group:/ ) {
$rgstat =~ s/Group://;
@rgstat = split(" ", $rgstat);
if ( $rgstat[2] ne "Online" && $rgstat[1] eq $primary_node ) {
$alert="\nOracle Resource Alert:\nResource Group $rgstat[0] is in $rgstat[2] state on primary node $rgstat[1] - Checking for failover\n";
$rgprimaryfailure=1;
push(@alert, $alert);
} elsif ( $rgstat[2] ne "Online" && $rgstat[1] eq $secondary_node && $rgprimaryfailure == 1 ) {
$alert="\nOracle Resource Alert:\nResource Group $rgstat[0] is in $rgstat[2] state on cluster - Resource Group Down on both Nodes!\n";
push(@alert, $alert);
} else {
if ( $debug ) {
$alert="Oracle Resource Info: $rgstat[1] Resource Group $rgstat[0] OK\n";
push(@debug, $alert);
}
}
} elsif ( $rgstat =~ /Resource:/ ) {
$rgstat =~ s/Resource://;
@rgstat = split(" ", $rgstat);
if ( $rgstat[2] ne "Online" && $rgstat[1] eq $primary_node ) {
$alert="\nOracle Resource Alert:\nResource$rgstat[0] is in $rgstat[2] state on primary node $rgstat[1] - Checking for failover\n";
$rgprimaryfailure=1;
push(@alert, $alert);
} elsif ( $rgstat[2] ne "Online" && $rgstat[1] eq $secondary_node && $rgprimaryfailure == 1 ) {
$alert="\nOracle Resource Alert:\nResource $rgstat[0] is in $rgstat[2] state on cluster - Resource Group Down on both Nodes!\n";
push(@alert, $alert);
} else {
if ( $debug ) {
$alert="Oracle Resource Info: $rgstat[1] Resource $rgstat[0] OK\n";
push(@debug, $alert);
}
}
} else {
print "WTF ---- SCSTAT -G\n";
}
}
@scstatD=`/usr/cluster/bin/scstat -D|/usr/bin/egrep 'servers|status'`;
for $diskstat (@scstatD) {
if ( $diskstat =~ /servers/ ) {
$diskstat =~ s/Device group servers://;
@diskstat = split(" ", $diskstat);
if ( $diskstat[1] ne $primary_node ) {
$alert="\nDisk Resource Alert:\n$diskstat[0] has switched primary node to $diskstat[1] from $diskstat[2]\n";
push(@alert, $alert);
} elsif ( $diskstat[1] ne $primary_node && $diskstat[1] ne $secondary_node ) {
$alert="\nDisk Resource Alert:\n$diskstat[0] has failed on all nodes!\n";
push(@alert, $alert);
} else {
if ( $debug ) {
$alert="Disk Resource Info: $diskstat[1] primary - $diskstat[0] OK\n";
push(@debug, $alert);
}
}
} elsif ( $diskstat =~ /status/ ) {
$diskstat =~ s/Device group status://;
@diskstat = split(" ", $diskstat);
if ( $diskstat[1] ne "Online" ) {
$alert="\nDisk Resource Group Alert:\n$diskstat[0] has switched to $diskstat[1] state on the cluster\n";
push(@alert, $alert);
} elsif ( $diskstat[1] eq "Online" ) {
if ( $debug) {
$alert="Disk Resource Group Info: $diskstat[0] OK\n";
push(@debug, $alert);
}
}
} else {
print "WTF ---- SCSTAT -D\n";
}
}
###################################
# EDIT To: Reply-To: and From:
# if you want mail to go somewhere
# useful and-or helpful!
###################################
if ( @alert > 0 || @debug > 0 ) {
open(CMAIL, "|/usr/lib/sendmail -t");
print CMAIL "Subject: CLUSTER ALERT - $hostname - $date\n";
print CMAIL "From: you\@yourdomain.com\n";
print CMAIL "Reply-To: you\@yourdomain.com\n";
print CMAIL "To: recipients\@yourdomain.com\n";
print CMAIL "\n\n";
foreach $warning (@alert) {
print CMAIL $warning;
}
if ( $debug > 0 ) {
foreach $message (@debug) {
print CMAIL $message;
}
}
close(CMAIL);
system("touch /tmp/cfail_ihot_stat");
} elsif ( @alert == 0 && -f "/tmp/cfail_ihot_stat" ) {
open(CMAIL, "|/usr/lib/sendmail -t");
print CMAIL "Subject: CLUSTER RESTORED - $hostname - $date\n";
print CMAIL "From: you\@yourdomain.com\n";
print CMAIL "Reply-To: you\@yourdomain.com\n";
print CMAIL "To: recipients\@yourdomain.com\n";
print CMAIL "\n\n";
print CMAIL "All Cluster Services Back To Good State\n";
print CMAIL "All Cluster Nodes: OK\n";
print CMAIL "All IPMP Groups: OK\n";
print CMAIL "All Oracle Resources: OK\n";
print CMAIL "All Storage Groups: OK\n";
close(CMAIL);
unlink("/tmp/cfail_ihot_stat");
}
, Mike
linux unix internet technology
Wednesday, January 14, 2009
How Many Different Ways Can You Stop VCS On Linux Or Unix?
Hey there,
This little rundown may seem trivial in comparison to some of our older posts on Veritas Cluster Server (VCS), as they all attacked a very specific aspect of VCS functionality and, mostly, were aimed at either explaining more advanced concepts or showing you how to get away with stuff that you're not supposed to do ;)
However, even in its simplicity, today's topic is just as valid as our previous posts, and (quite probably) more useful to the admin or user who wants to know enough to administrate VCS well, but doesn't care about all the nitty-gritty. Because, let's face it, the nitty-gritty doesn't really matter all the time. I find that I almost never think of the nitty gritty every morning when I start my car (I reserve that special time for praying that it won't explode ;)
So, before I veer too far off the beaten path, we'll get down to today's subject: How to shutdown a Veritas Cluster, using VCS, the many different ways to do it and what they all mean.
Of course, the base command for stopping VCS (Sometimes also referred to as HAD for the High Availability Daemon, which explains why all the commands for VCS start with "ha" :) is "hastop." hastop has a number of ways in which you can call it; all of which can have a significant impact on your day if used arbitrarily ;)
1. hastop -local. This usage stops VCS on your local machine (or the machine on which you're running it - same thing). It also causes any service groups on your local system to be taken offline.
2. hastop -local -evacuate. This invocation is very similar to the previous, except that it migrates (if possible) all of your service groups to the default failover system before it stops the VCS services on your local machine. This option isn't available if you use the -all flag (since, obviously, there won't be any systems left up in the cluster to fail over to ;)
3. hastop -local -noautodisable. The -noautodisable option (also not available for use in conjunction with the -all argument) makes it so that none of your service groups are set to disabled.
4. As an aside of sorts, it should be noted that you can use -evacuate and -noautodisable together, although you can't use either with other arguments, like -force or -all.
3. hastop (-all|-local) -force. These options, when presented to hastop, cause it to stop VCS services on all systems in your cluster (-all) or just your local system (-local), but it does not do anything to your service groups or offline their resources. This is convenient when you have planned maintenance on VCS-managed resources and don't want to create a fault condition (and all its accompanying alarms, bells and whistles ;) When VCS is stopped in this manner, hastart merely resumes running VCS services as normal (It doesn't bring VCS up with an overt assumption that something wrong has happened). One downside of using the -force option is that it doesn't check whether or not you may be editing the main.cf file at the time you decide to call it quits. If you have it open, in read/write mode, and you stop VCS this way, you may be in for an unpleasant restart.
4. hastop -all. This stops VCS on all systems in your cluster, as the previous command does, but it doesn't ignore your system groups and will take them offline.
5. hastop -sys SYSTEMNAME. This way of executing hastop causes VCS to react in much the same way as the -local flag does (including having the -evacuate, -noautodisable and -force arguments - the same rules, listed above, apply to using combinations of -evacuate and -noautodisable (ok) and -force (not ok to mix with either of the others). One thing to note is that you can't use the -local and/or -all flags with the -sys flag (again, for obvious reasons. It's nice to know that some things in this world make sense ;), since the -sys flag requires you to specify a specific system (or host) in your cluster on which to execute the command.
6. hastop -help. This is a great option to use when you have no idea who you are, where you are, how you got there and/or what to do ;)
One final (and, of course, interesting ;) note about modifying hastop's behaviour is that you can modify the EngineShutdown "cluster attribute" (outside the scope of this little post) to set different default behaviours for hastop. This can be a big help if you always do things "the not normal way" ;) The EngineShutdown cluster attribute(s) (ESCA from now on, since I'm getting tired of mistyping the name of this thing ;) that you can set (with the haclus command) are:
a. Enable: This ESCA indicates that all hastop commands should be processed. This is the default.
b. Disable: This ESCA indicates that all hastop commands will not be processed. The only exception to this setting is if you use the -force argument to hastop.
c. DisableClusStop: This ESCA makes it so that the "hastop -all" command is rejected or not processed. All other hastop commands are processed as they normally would be.
d. PromptClusStop: This ESCA will cause the operator/administrator to be prompted (Do you really want to do this? Are you sure? C'mon, seriously? ;) before it executes the "hastop -all" command. All other hastop commands are processed normally and don't require answering any prompts.
e. PromptLocal: This ESCA prompts the admin whenever "hastop -local" is invoked (Are you positive you want to do this? Have you given this any thought at all? Remember; it's you're arse. Do you still want to do this? ;). All other hastop commands are processed normally.
f. PromptAlways: This ESCA (which is also referred to in non-academic circles as the Big PITA ;) causes the admin/qualified-user to be prompted when executing any and all hastop commands. When you get sick of answering questions, and you have the permissions, you might want to turn this off since it's not default and someone you work with may have turned it on just to eventually drive you nuts ;)
And, there you have it; hastop in a big complicated nutshell (Or outside of a nutshell ...whatever the opposite of in-a-nutshell is ;) One thing I should mention if you work with cluster attributes or any parts of the main.cf, types.cf or other VCS configuration files; it's always a good idea to make sure you know what the defaults for your version are. VCS's configuration file style will just "not include" most values that are set to their defaults, so you would never see the Enable variable and value in the ESCA, since it's the default.
A somewhat cool trick (if you're not sure if a particular variable/value pair is set to its default) is to add it to your configuration file(s). After you rebuild your config files the proper way (or manually edit it/them the somewhat-frowned-upon way), if the variable/value pair you entered is not set to default, it will have been removed!
Cheers,
, Mike
Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.
Friday, June 27, 2008
SunCluster Quick Command Reference
Hey There,
Well, sitting back and reflecting today, I realized that we've done our fair share of posts on cheat-sheets and such (Our LVM command reference, for example). However, when it comes to clustering systems, we've probably been unfair in our bias toward Veritas Clustering as opposed to SunCluster, for which we've only put up, I believe, one post with our Simple SunCluster Monitoring Script (Thankfully linked back to from The SunCluster Wiki). What's a poor blog to do? ;)
With that in mind, today we're going to lay down a quick command reference for SunCluster. Hopefully you'll find it useful and enjoy it :) Of course, it assumes that you know, mostly, how to use SunCluster (and are familiar with basic clustering concepts), but it can be used as an attractive wallpaper, if nothing else ;)
All of this information, and much more, can be found at Sun's Online Documentation Center for SunCluster.
Enjoy,
Quorum Commands:
host # clquorum add device - Add a SCSI Quorum Device
host # clquorum add -t netapp_nas -p filer =nasdevicename,lun_id =IDnumdevice Nasdevice - Add a NAS Quorum Device
host # clquorum add -t quorumserver -p qshost =IPaddress, port =portnumber quorumservername - Add a Quorum Server
host # clquorum remove device - Remove a Quorum Device
Resource Type Commands:
host # clresourcetype register type - Register a Resource Type
host # clresourcetype unregister type - Remove a Resource Type
Resource Group Commands:
host # clresourcegroup create group - Create a Failover Resource Group
host # clresourcegroup create -S group - Create a Scalable Resource Group
host # clresourcegroup online + - Bring Online All Resource Groups
host # clresourcegroup delete group - Delete a Resource Group
host # clresourcegroup delete -F group - Delete a Resource Group and All of Its Resources
host # clresourcegroup switch -n nodename group - Switch the Current Primary Node of a Resource Group
host # clresourcegroup unmanage group - Move a Resource Group Into the UNMANAGED State
host # clresourcegroup suspend group - Suspend Automatic Recovery of a Resource Group
host # clresourcegroup resume group - Resume Automatic Recovery of a Resource Group
host # clresourcegroup set -p Failback=true + name=value - Change a Resource Group Property
host # clresourcegroup add-node -n nodename group - Add a Node To a Resource Group
host # clresourcegroup remove-node -n nodename group - Remove a Node From a Resource Group
Resource Administration Commands:
host # clreslogicalhostname create -g group lh-resource - Create a Logical Hostname Resource
host # clressharedaddress create -g group sa-resource - Create a Shared Address Resource
host # clresource create -g group -t type resource - Create a Resource
host # clresource delete resource - Remove a Resource
host # clresource disable resource - Disable a Resource
host # clresource set -t type -p name=value + - Change a Single-Value Resource Property
host # clresource set -p name+=value resource - Add a Value to a List of Property Values; Existing values in the list are unchanged.
host # clresource create -t HAStoragePlus -g group -p FileSystemMountPoints=mount-point-list -p Affinityon=true rs-hasp - Create an HAStorage Plus Resource
host # clresource clear -f STOP_FAILED resource - Clear the STOP_FAILED Error Flag on a Resource
Device Administration Commands:
host # cldevicegroup create -t vxvm -n node-list -p failback=true vxdevgrp - Add a VxVM Device Group
host # cldevicegroup delete devgrp - Remove a Device Group
host # cldevicegroup switch -n nodename devgrp - Switch a Device Group to a New Node
host # cldevicegroup offline devgrp - Bring Offline a Device Group
host # cldevice refresh diskname - Update Device IDs for the Cluster
Additional Administration and Monitoring Commands:
To add a Node to Cluster:
From the node to be added, which has access:
host # clnode add -c clustername -n nodename -e endpoint1, endpoint2 - Use this only if the node has access to the cluster
To remove a Node From the Cluster:
From the node to be removed, which is in noncluster
mode and has access:
host # clnode remove - Use this only if the node has access to the cluster
host # clnode evacuate nodename - Switch All Resource Groups and Device Groups Off of a Node
host # clinterconnect disable nodename:endpoint - Manage the Interconnect Interfaces
host # clinterconnect enable nodename:endpoint - To disable a cable so that maintenance can be performed, then enable the same cable afterward.
host # cluster status - Display the Status of All Cluster Components
host # command status - Display the Status of One Type of Cluster Component
host # cluster show - Display the Complete Cluster Configuration
host # command show Component - Display the Configuration of One Type of Cluster
host # command list - List One Type of Cluster Component
host # clnode show-rev -v - Display Sun Cluster Release and Version Information
To list the software versionson the current node.
host # clnode show | grep nodename - Map Node ID to Node Name
host # cltelemetryattribute enable -t disk rbyte.rate wbyte.rate read.rate write.rate - Enable Disk Attribute Monitoring on All Cluster Disks
host # cltelemetryattribute disable -t disk rbyte.rate wbyte.rate read.rate write.rate - Disable Disk Attribute Monitoring on All Cluster Disks
SHUTTING DOWN AND BOOTING A CLUSTER
To shut Down the Entire Cluster:
host # cluster shutdown
From one node:
host # clnode evacuate
host # shutdown - Shut Down a Single Node
To boot a Single Node from the OBP:
ok> boot
To reboot a Node Into Noncluster Mode from the OBP:
ok> boot -x
, Mike
Tuesday, February 26, 2008
Offlining, Failing Over And Switching in VCS
Hey there,
Today we're going to address a question that's asked commonly enough by foks who use Veritas Cluster Server: What's the difference between offlining, failing over and switching when dealing with service groups across multiple nodes? That doesn't seem like a very common question. I'm sure it's probably hardly ever phrased that way ;)
Anyway, to the point, all three options above are useful (hence your ability to avail yourself of them) and sufficiently distinct that you should be sure you're using the one you want to, depending on what ends you wish to achieve. All of this information is fairly general and should work on VCS for Linux as well as Unix.
We'll deal with them one bye one, outlining what they basically do, with a short example command line to demonstrate, where applicable. None of this stuff is hard to pick up and run with, as long as all the components of your VCS cluster are setup correctly. Occasionally, you may see errors if things aren't exactly perfect, like we noted in this post on recovering indefinitely hung VCS resources.
1. Offlining. The distinction to be made here, most plainly, is that, when you offline a resource or service group, you are "only" doing that. This differs from failover and switching in that the service group you offline with this particular option is not brought online anywhere else as a result. So, when you execute, for instance:
host # hagrp -offline YOUR_SERVICE_GROUP -sys host1
that service group, and its resources, generally, are taken offline on host1, and nothing else happens. If you're operating in an environment where systems don't run service groups concurrently (active/active), you will have effectively "shut down" that service group, and any services it provides, for the entire cluster.
2. Failing Over. This is more of a concept than any particular command. When you have your cluster setup to fail over, if a resource or service group, etc, goes offline on one node (host1, for instance) and it wasn't brought offline on purpose, VCS will naturally attempt to bring it online on the next available node (listed in your main.cf configuration file). Needless to say, if only one host is listed for a particular service group, its failure on one host will mean the failure of the entire service group. It also obviates the use of VCS in the first place ;)
3. Switching. This is what most folks want to do when they "offline" a service group, as in point 1. Although, since VCS automatically switches unexpectedly offlined resources on its own (when it's set up to), it's reasonable for someone new to the product to assume that offlining a service group would engage VCS in a switching activity. Unfortunately, this isn't the case. If you want to switch a service group from host1 to host2, for example, these would be the command options you'd want to give to hagrp:
host # hagrp -switch YOUR_SERVICE_GROUP -to host2 <--- Assuming you're running this from host1.
Hopefully this little guided mini-FAQ helped out with differentiating between the concepts. If you find the command line examples valuable, even better :)
Happy failing! ;) Over.
, Mike
linux unix internet technology
