Showing posts sorted by relevance for query vcs cluster server. Sort by date Show all posts
Showing posts sorted by relevance for query vcs cluster server. Sort by date Show all posts

Monday, February 9, 2009

Getting Faster Support For Your VCS-Clustered NetBackup Servers

Hey There,

Today's post is a little trick that anyone running Veritas/Symantec NetBackup (Linux or Unix) on VCS - Veritas Cluster Server - should know. As the title suggests, doing this one little thing will almost guarantee you more responsive support from Symantec (given the highly specific situation outlined in the first sentence, of course ;). The funny thing, though, is that many people have this problem already; they just may not have had to deal with Symantec, with regards to it, yet.

The problem you have (or will have) to face is that Symantec "does not" fully support VCS Clustered NetBackup installations if that same cluster has other Service Groups and/or major resources active on it. I'd almost say that they don't support it at all, but that's not entirely true (just keep complaining. You'll see ;). However, Symantec's stated position is that they really "don't" support it at all (and this is only hearsay, of course. Something I've heard while on the phone with another person who answered the phone at the number listed on our contract for Symantec NetBackup Support. So, of course, I may not be entirely correct, here ...).

Basically, what this means is that Symantec has VCS Cluster support for NetBackup (They own both NetBackup and Veritas Cluster Server), have VCS modules (types files, pkg's, etc) for NetBackup and even have full documentation online to help you setup your NetBackup Server(s) in VCS. Even given that fact, they do not support NetBackup in a VCS cluster that also runs, say, NFS or Oracle or even a shared mountpoint that isn't NB-related. NOTE: They especially don't support IPmultiNIC_B (I'm not sure why, since NIC auto-failback seems like a great idea to me)!! Their contention is that NetBackup is meant to function on VCS as the "only service or resource." This is fine, if you can spare 2 or 3 servers to just run NetBackup (larger companies, and data security/storage/backup companies, probably have no issue with this). However, most businesses can't afford to spare the extra servers "just" for NetBackup and will often have NetBackup in the same VCS cluster that runs an NFS group/resource, other shared mountpoints, web servers, databases, etc. Hopefully not "all" of those on one machine, but times are tight; you never know ;)

Now, here's the trick (it's not even a trick, really, just some good advice to help you out) to getting speedy support from Symantec when you call them about an issue with NetBackup on VCS: You're going to have to do a little bit of prep work and (even if it hurts) lie a little bit. If you're not comfortable "bending the truth," just remember that dishonesty is the second-best policy. Things could be worse ;)

1. Before you call, have a backup main.cf ready to go. And, this is very important if you're working on a cluster that's been around for a while, make sure that you do one of two things when Symantec Support asks for anything like an "ls" of your /etc/VRTSvcs/conf/config directory:

a. Do a very specific ls:

host # ls /etc/VRTSvcs/conf/config/main.cf <-- They don't need to see the other 500 hundred backups that are three times the size of your faked-up file ;)

b. Create a staging directory, copy only the minimal files you want into it, and then give them an "ls" of that:

host # cd /etc/VRTSvcs/conf/config/stage <-- Don't show them this!
host # ls -l *

2. As noted above, you need a clean main.cf for NetBackup Support, or you're going to end up getting bounced around from department to department, at best. If you're running anything aside from NetBackup in one cluster, you'll need to remove it (not literally, of course). Here's a quick example (This setup is going to be very basic, just to keep it short - which means the commented dependency trees that VCS puts into the conf file are removed, for this example, as well) - BTW, feel free to use this exact main.cf (just change a few names and numbers) if it works for you. It's been verified on the system I'm goofing with, but it probably won't work out-of-the-box on your system :)

Since this example is so long, I'll bid you farewell for the day, now. Hope this helps you if you ever get in a "We don't support such-and-such" or "You really shouldn't be doing such-and-such" or "It's a VCS Problem <--> It's a NetBackup Problem" sort-of-situation. And, remember, even if you goof up once and tell them the truth (who amongst us isn't guilty of being upfront every once in a while ;), you can always ask them to cancel your request ("Oh wait, I see what the problem is... Sorry, you can close the ticket" etc) and then call back later and go through normal channels so you'll hopefully get a different person (and, when you do, be sure to intimate that the problem you're having now is totally unrelated to the previous ticket, if they're efficient and happen to look for, and find, it. It's worth a shot ;)

Cheers,

--- Stripped-Down main.cf ---

include "types.cf"
include "/usr/openv/netbackup/bin/cluster/vcs/NetBackupTypes.cf"

cluster VCScluster1 (
UserNames = { admin = EncryptedPassword }
ClusterAddress = "10.10.10.199"
Administrators = { admin }
)

system VCShost1 (
)

system VCShost2 (
)

group ClusterService (
SystemList = { VCShost1 = 0, VCShost2 = 1 }
AutoStartList = { VCShost1 }
)

IP webip (
Device = bge0
Address = "10.10.10.199"
NetMask = "255.255.255.0"
)

NIC webnic (
Device = bge0
)

webip requires webnic

group SG_VCSNB1 (
SystemList = { VCShost1 = 0, VCShost2 = 1 }
AutoStartList = { VCShost1 }
)

IPMultiNIC nbuIP (
Address = "10.10.10.199"
MultiNICResName = mnic
)

MultiNICA mnic (
Device @VCShost1 = { bge1 = "10.10.10.197", bge2 = "10.10.10.198" }
Device @VCShost2 = { bge2 = "10.10.10.198", bge1 = "10.10.10.197" }
ArpDelay = 5
IfconfigTwice = 1
PingOptimize = 0
)

DiskGroup VCShostDG (
DiskGroup = VCShostDG
)

Mount MNT_NBmount_VCSNB1 (
MountPoint = "/opt/VRTSnbu"
BlockDevice = "/dev/vx/dsk/VCShostDG/NBmount"
FSType = vxfs
MountOpt = largefiles
FsckOpt = "-y"
)

NetBackup APP_nbu_VCSNB1 (
Critical = 0
ServerName = NBU_Server
ServerType = NBUMaster
)

Volume VOL_NBmount_VCSNB1 (
Volume = NBmount
DiskGroup = VCShostDG
)

APP_nbu_VCSNB1 requires nbuIP
APP_nbu_VCSNB1 requires MNT_NBmount_VCSNB1
MNT_NBmount_VCSNB1 requires VOL_NBmount_VCSNB1
VOL_NBmount_VCSNB1 requires VCShostDG
nbuIP requires mnic


, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Friday, November 14, 2008

Basic Veritas Cluster Server Troubleshooting

Hey There,

For the end of the week, we're going to continue with the theme of sparse-but-hopefully useful information. Quick little "crib sheets" (preceding by paragraphs and paragraphs of stilted ramblings by the lunatic who pens this blog's content ;) For this Friday, we're going to come back around and take a look at Veritas Cluster Server (VCS) troubleshooting. If you're interested in more specific examples of problems, solutions and suggestions, with regards to VCS, check out all the VCS related posts from the past year or so. Hopefully you'll be able to find something useful in our archives, as well. These simple suggestions should work equally well for Unix as well as Linux, if you choose to go the VCS route rather than some less costly one :)

And, here we go again; quick, pointed bullets of info. Bite-sized bits of troubleshooting advice that focus on solving the problem, rather than understanding it. That sounds awful, I know, but, sometimes, you have to get things done and, let's face it, if it's the job or your arse, who cares about the why? Leave that for philosophers and academics. Plus, since you fix problems so fast, you'll have plenty of time to read up on the ramifications of your actions later ;)

The setup: Your site is down. It's a small cluster configuration with only two nodes and redundant nic's, attached network disk, etc. All you know is that the problem is with VCS (although it's probably indirectly due to a hardware issue). Something has gone wrong with VCS and it's, obviously, not responding correctly to whatever terrible accident of nature has occurred. You don't have much more to go on than that. The person you receive your briefing from thinks the entire clustered server set up (hardware, software, cabling, power, etc) is a bookmark in IE ;)

Now, one by one, in a fashion that zigs on purpose, but has a tendency to zag, here are a few things to look at right off the bat when assessing a situation like this one. Perhaps next week, we'll look into more advanced troubleshooting (and, of course, you can find lots of specific "weird VCS problem" solutions in our VCS archives)

1. Check if the cluster is working at all.

Log into one of the cluster nodes as root (or a user with equivalent privilege - who shouldn't exist ;) and run

host1 # hastatus –summary

or

host1 # hasum <-- both do the same thing, basically

Ex:

host1 # hastatus -summary

-- SYSTEM STATE
-- System State Frozen

A host1 RUNNING 0
A host2 RUNNING 0

-- GROUP STATE
-- Group System Probed AutoDisabled State

B ClusterService host1 Y N OFFLINE
B ClusterService host2 Y N ONLINE
B SG_NIC host1 Y N ONLINE
B SG_NIC host2 Y N OFFLINE
B SG_ONE host1 Y N ONLINE
B SG_ONE host2 Y N OFFLINE
B SG_TWO host1 Y N OFFLINE
B SG_TWO host2 Y N OFFLINE


Clearly, your situation is bad: A normal VCS status should indicate that all nodes in the cluster are “RUNNING” (which these are). However, it should also show all service groups as being ONLINE on at least one of the nodes, which isn't the case above with SG_TWO (Service Group 2).

2. Check for cluster communication problems. Here we want to determine if a service group is failing because of any heartbeat failure (The VCS cluster, that is, not another administrator ;)

Check on GAB first, by running:

host1 # gabconfig -a

Ex:

host1 # gabconfig -a
GAB Port Memberships
===============================================================
Port a gen 3a1501 membership 01
Port h gen 3a1505 membership 01


This output is okay. You would know you had a problem at this point if any of the following conditions were true:

if no port “a” memberships were present (0 and 1 above), this could indicate a problem with gab or llt (Looked at next)

If no port "h" memberships were present (0 and 1 above), this could indicate a problem with had.

If starting llt causes it to stop immediately, check your heartbeat cabling and llt setup.

Try starting gab, if it's down, with:

host1 # /etc/init.d/gab start

If you're running the command on a node that isn't operational, gab won't be seeded, which means you'll need to force it, like so:

host1 # /sbin/gabconfig -x

3. Check on LLT, now, since there may be something wrong there (even though it wasn't indicated above)

LLT will most obviously present as a crucial part of the problem if your "hastatus -summary" gives you a message that it "can't connect to the server." This will prompt you to check all cluster communication mechanisms (some of which we've already covered).

First, bang out a quick:

host1 # lltconfig

on the command line to see if llt is running at all.

If llt isn't running, be sure to check your console, system messages file (syslog, possibly messages and any logs in /var/log/VRTSvcs/... - usually the "engine log" is worth a quick look) As a rule, I usually do

host1 # ls -tr

when I'm in the VCS log directory to see which log got written to last, and work backward from there. This puts the most recently updated file last in the listing. My assumption is that any pertinent errors got written to one of the fresher log files :) Look in these logs for any messages about bad llt configurations or files, such as /etc/llttab, /etc/llthost and /etc/VRTSvcs/conf/sysname. Also, make sure those three files contain valid entries that "match" <-- This is very important. If you refer to the same facility by 3 different names, even though they all point back to the same IP, VCS can become addled and drop-the-ball.

Examples of invalid entries in LLT config files would include "node numbers" outside the range of 0 to 31 and "cluster numbers" outside the range of 0 to 255.

Now, if LLT "is" running, check its status, like so:

host # lltstat -wn <-- This will let you know if llt on the separate nodes within the cluster can communicate with one another.

Of course, verify physical connections, as well. Also, see our previous post on dlpiping for more low-level-connection VCS troubleshooting tips.

Ex:

host1 # lltstat -vvn
LLT node information:
Node State Link Status Address
0 prsbn012 OPEN
ce0 DOWN
ce1 DOWN
HB172.1 UP 00:03:BA:9D:57:91
HB172.2 UP 00:03:BA:0E:F1:DE
HB173.1 UP 00:03:BA:9D:57:92
HB173.2 UP 00:03:BA:0E:D0:BE
1 prsbn015 OPEN
ce3 UP 00:03:BA:0E:CE:09
ce5 UP 00:03:BA:0E:F4:6B
HB172.1 UP 00:03:BA:9D:5C:69
HB172.2 UP 00:03:BA:0E:CE:08
HB173.1 UP 00:03:BA:0E:F4:6A
HB173.2 UP 00:03:BA:9D:5C:6A


host1 # cat /etc/llttab <-- pardon the lack of low-pri links. We had to build this cluster on the cheap ;)

set-node /etc/VRTSvcs/conf/sysname
set-cluster 100
link ce0 /dev/ce:0 - ether 0x1051 -
link ce1 /dev/ce:1 - ether 0x1052 -
exclude 7-31
host1 # cat /etc/llthosts
0 host1
1 host2
host1 # cat /etc/VRTSvcs/conf/sysname
host1


If llt is down, or you think it might be the problem, either start it or restart it with:

host1 # /etc/init.d/llt.rc start

or

host1 # /etc/init.d/llt.rc stop
host1 # /etc/init.d/llt.rc start


And, that's where we'll end it today. There's still a lot more to cover (we haven't even given the logs more than their minimum due), but that's for next week.

Until then, have a pleasant and relaxing weekend :)

Cheers,

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Monday, April 20, 2009

Prepping For Setting Up VCS NFS Clustering On Solaris 10

Hey there,

Hope your work week is beginning swimmingly :) Mine is kind of like the end of last week, although that sentence is a bit of a non sequitur. I actually haven't stopped working, so it still is last week this week (???) You know what I mean (Although, I hope that you don't ;)

Today we're going to look at preparing your Solaris 10 system for clustering NFS (The Network File System ... the "The" is silent ;) on VCS (Veritas Cluster Server). In many ways, it's the same as setting it up on previous versions of Solaris, but it differs in many ways, as well. Apparently, I spend way too much time looking at this issue in many many ways ;)

NOTE: This post is kind of a wrapper around our previous posts on adding NFS to an existing VCS cluster and adding NFS to a VCS cluster with no down time. You can check either of those out if you want to read up on doing the VCS configuration part of the "VCS NFS" setup. This post is of purely a preparatory nature (with, admittedly, some post-installation test steps and a pointer to another old post on an uncommon Solaris 10 VCS NFS error and how to fix it).

1. First, we'll do the Solaris 10 setup. This is very important, since the SMF (Soul Macerating Futility or, possibly, Service Management Facility) has changed the way in which "services" and "run levels" are either dealt with or completely subverted ;)

a. If you're going to be depending on VCS for NFS management, it will interest you to know that VCS won't have anything to do with NFS if you want to use it on your own (on the same machine). For that reason, we're going to use svccfg to "delete" the following services, rather than using svcadm to "disable" them.

host # svccfg delete -f svc:/network/nfs/server:default
host # svccfg delete -f svc:/network/nfs/mapid:default
host # svccfg delete -f svc:/network/nfs/status:default
host # svccfg delete -f svc:/network/nfs/nlockmgr:default


b. Doing the above may (actually, should) kill the lockd and statd daemons that are probably already running (I'm trying not to be too presumptuous ;) If that's the case, you'll need to start those up again.

host # (/usr/lib/nfs/lockd &)
host # (/usr/lib/nfs/statd &)


VCS will take care of starting them once it's good to go.

If the above way of starting those services from the command line seems goofy, check out our aging post on what to do when nohup hangs up anyway from way back when. It's a fair read and may still be interesting to a certain degree ;)

c. Finally, you just need to make one directory, for convenience's sake (and also so that the "NFSRestart" resource will actually work):

host # mkdir /opt/VRTSvcs/lock

2. Now (whooshing right past the "actual" VCS NFS setup (referenced above from two previous posts on that subject), you're ready to do a few simple tests.

a. Once you have NFS running on VCS on your primary node, pick another node (we'll just assume you picked the secondary) and test that the NFS mount is up and working like VCS says it is (You can't always take it at its word):

host2 # showmount -e host
export list for host:
/that/vcs/nfs/directory/share (everyone)


Then just make sure you can actually mount the share.

b. Then fail over to your secondary node and run the same test on the primary:

host # showmount -e host2
export list for host2:
/that/vcs/nfs/directory/share (everyone)


Again, mount the share just to be sure everything's in working order.

c. If you encounter an error while running "showmount" on either server, like this:

showmount: host: RPC: Rpcbind failure - RPC: Authentication error

accompanied by your being able to generate this error (although, not necessarily):

host # rpcinfo -p host
rpcinfo: can't contact portmapper: RPC: Authentication error; why = Failed (unspecified error)


be sure to check out our previous post on this little Solaris 10 VCS NFS gotcha and, hopefully, you'll end up knowing more than you ever wanted to about how to straighten that out :)

And, finally, in an out-of-sequence-series of only four posts, you're finally done setting up NFS in VCS on Solaris 10. Hopefully, you finished a long time ago. It's been six months since some of the referenced posts were originally published. Over time, it gets getting harder and harder to dot all the i's on this blog ;)

Cheers,

, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, January 14, 2009

How Many Different Ways Can You Stop VCS On Linux Or Unix?

Hey there,

This little rundown may seem trivial in comparison to some of our older posts on Veritas Cluster Server (VCS), as they all attacked a very specific aspect of VCS functionality and, mostly, were aimed at either explaining more advanced concepts or showing you how to get away with stuff that you're not supposed to do ;)

However, even in its simplicity, today's topic is just as valid as our previous posts, and (quite probably) more useful to the admin or user who wants to know enough to administrate VCS well, but doesn't care about all the nitty-gritty. Because, let's face it, the nitty-gritty doesn't really matter all the time. I find that I almost never think of the nitty gritty every morning when I start my car (I reserve that special time for praying that it won't explode ;)

So, before I veer too far off the beaten path, we'll get down to today's subject: How to shutdown a Veritas Cluster, using VCS, the many different ways to do it and what they all mean.

Of course, the base command for stopping VCS (Sometimes also referred to as HAD for the High Availability Daemon, which explains why all the commands for VCS start with "ha" :) is "hastop." hastop has a number of ways in which you can call it; all of which can have a significant impact on your day if used arbitrarily ;)

1. hastop -local. This usage stops VCS on your local machine (or the machine on which you're running it - same thing). It also causes any service groups on your local system to be taken offline.

2. hastop -local -evacuate. This invocation is very similar to the previous, except that it migrates (if possible) all of your service groups to the default failover system before it stops the VCS services on your local machine. This option isn't available if you use the -all flag (since, obviously, there won't be any systems left up in the cluster to fail over to ;)

3. hastop -local -noautodisable. The -noautodisable option (also not available for use in conjunction with the -all argument) makes it so that none of your service groups are set to disabled.

4. As an aside of sorts, it should be noted that you can use -evacuate and -noautodisable together, although you can't use either with other arguments, like -force or -all.

3. hastop (-all|-local) -force. These options, when presented to hastop, cause it to stop VCS services on all systems in your cluster (-all) or just your local system (-local), but it does not do anything to your service groups or offline their resources. This is convenient when you have planned maintenance on VCS-managed resources and don't want to create a fault condition (and all its accompanying alarms, bells and whistles ;) When VCS is stopped in this manner, hastart merely resumes running VCS services as normal (It doesn't bring VCS up with an overt assumption that something wrong has happened). One downside of using the -force option is that it doesn't check whether or not you may be editing the main.cf file at the time you decide to call it quits. If you have it open, in read/write mode, and you stop VCS this way, you may be in for an unpleasant restart.

4. hastop -all. This stops VCS on all systems in your cluster, as the previous command does, but it doesn't ignore your system groups and will take them offline.

5. hastop -sys SYSTEMNAME. This way of executing hastop causes VCS to react in much the same way as the -local flag does (including having the -evacuate, -noautodisable and -force arguments - the same rules, listed above, apply to using combinations of -evacuate and -noautodisable (ok) and -force (not ok to mix with either of the others). One thing to note is that you can't use the -local and/or -all flags with the -sys flag (again, for obvious reasons. It's nice to know that some things in this world make sense ;), since the -sys flag requires you to specify a specific system (or host) in your cluster on which to execute the command.

6. hastop -help. This is a great option to use when you have no idea who you are, where you are, how you got there and/or what to do ;)

One final (and, of course, interesting ;) note about modifying hastop's behaviour is that you can modify the EngineShutdown "cluster attribute" (outside the scope of this little post) to set different default behaviours for hastop. This can be a big help if you always do things "the not normal way" ;) The EngineShutdown cluster attribute(s) (ESCA from now on, since I'm getting tired of mistyping the name of this thing ;) that you can set (with the haclus command) are:

a. Enable: This ESCA indicates that all hastop commands should be processed. This is the default.

b. Disable: This ESCA indicates that all hastop commands will not be processed. The only exception to this setting is if you use the -force argument to hastop.

c. DisableClusStop: This ESCA makes it so that the "hastop -all" command is rejected or not processed. All other hastop commands are processed as they normally would be.

d. PromptClusStop: This ESCA will cause the operator/administrator to be prompted (Do you really want to do this? Are you sure? C'mon, seriously? ;) before it executes the "hastop -all" command. All other hastop commands are processed normally and don't require answering any prompts.

e. PromptLocal: This ESCA prompts the admin whenever "hastop -local" is invoked (Are you positive you want to do this? Have you given this any thought at all? Remember; it's you're arse. Do you still want to do this? ;). All other hastop commands are processed normally.

f. PromptAlways: This ESCA (which is also referred to in non-academic circles as the Big PITA ;) causes the admin/qualified-user to be prompted when executing any and all hastop commands. When you get sick of answering questions, and you have the permissions, you might want to turn this off since it's not default and someone you work with may have turned it on just to eventually drive you nuts ;)

And, there you have it; hastop in a big complicated nutshell (Or outside of a nutshell ...whatever the opposite of in-a-nutshell is ;) One thing I should mention if you work with cluster attributes or any parts of the main.cf, types.cf or other VCS configuration files; it's always a good idea to make sure you know what the defaults for your version are. VCS's configuration file style will just "not include" most values that are set to their defaults, so you would never see the Enable variable and value in the ESCA, since it's the default.

A somewhat cool trick (if you're not sure if a particular variable/value pair is set to its default) is to add it to your configuration file(s). After you rebuild your config files the proper way (or manually edit it/them the somewhat-frowned-upon way), if the variable/value pair you entered is not set to default, it will have been removed!

Cheers,

, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Monday, March 23, 2009

How To Modify Live VCS Clusters On Linux And Unix Without Completely Alienating Your Co-Workers

Hey there,

A happy Monday to everyone and here's hoping you had a halfway-decent weekend. There's nothing quite like 2 days of misery leading up to a Monday, which is why I like to do something fun every weekend that I can. This weekend I slept a lot. It was a blast. I'm awake now, but I'm getting right back up on that horse as soon as I'm done proofing this post ;)

If you work on Veritas Cluster Server a lot (VCS from here on in, and on out ;), or even if you just work on them a little (or sit next to someone who does), you probably know how incredibly annoying it can be to make fixes to them once you've got them up and online. They are, almost by definition, expected to provide high availability for whatever twisted perverse software you run on them and "high availability" almost always translates into "high accountability." Even though a perfectly constructed cluster (of however many individual nodes) is supposed to "make your life easier," I've found that this is hardly ever the case. People tend to go ape$h1t whenever a cluster node experiences an issue, even when it's something as mundane as a flip-flop between NIC's on a single node that are sharing a local floating-IP that is but one of many virtual IP's that the entire cluster uses to float its own virtual IP on top of. If you work on clusters, you're statistically likely to be either incredibly apathetic about these sorts of issues, or a ring-tone away from a total nervous breakdown ;)

In the event that you do have to work on an active Veritas cluster (whether on Linux or Unix or even Windows), there are a few ways you can keep those alarm bells (that you have no choice but to trigger in order to get your work done) from reaching their intended recipients. The optimal end result, of course, being that you get your job done in a timely and efficient manner and your co-workers (who aren't in your department and don't need to know - HUGE ASSUMPTION ;) don't lose any sleep worrying about the money the company is conceivably losing while nothing really "bad" is happening.

We'll run down this short list in order from "best practice" to crude severing of email and other notification avenues and/or faking "upness." You may need to do the group of "last things" on this list, no matter how great your setup is. If you're working in an environment where outside agents (HP OpenView, etc) report on your cluster's condition, turning off notification the proper way may not do you any good at all, anyway ;)

Prior to making any changes to your main.cf (assuming you're just changing it with the intention of restoring it to its exact same state), I would recommend either saving off your main.cf file using simple copy:

host # cp main.cf main.cf.`hostname`.`date +%m%d%y%H%M%S`

or creating a command file from your cf file, so that you can just rebuild it again if anything terrible happens, like so:

host # hacf -cftocmd . <-- assuming you're in the main Veritas configuration directory. If you're not, and even if you are, you can pass the hacf command the full path to your configuration directory, if you prefer:
host # hacf -cftocmd /etc/VRTSvcs/conf/config <-- The same thing as the previous command, in most situations.

You can then use the .cmd file to recreate your .cf file using the "-cmdtocf" argument, with the rest of the hacf command generally remaining the same.

1. The first thing you would "normally" do (on the primary node in your cluster), would be to make sure that VCS's notification system redirects its error messages to an alternate source (like your email instead of everyone else's). So, assuming you have the NotifierMngr resource set up to mail errors to everyone@mycompany.com, you can easily modify this without alarming anyone, like so:

host # haconf -makerw
host # hares -modify NotifierMngr SmtpRecipients "me@nobodyelse.com = Error"
host # haconf -dump -makero


Of course, your setup may be more, or less, complicated, but that should give you the gist of it.

2. The next thing you should consider (still going by the VCS playbook) is that some of your resources may have specific "owners" assigned. These people will be notified of issues with certain resources, as well, and should be neutralized whether or not it actually makes a difference ;) You can do this in approximately the exact same way, like this:

host # haconf -makerw
host # hares -modify SOME_RESOURCE ResourceOwner "me@nobodyelse.com"
host # haconf -makero


3. Those first two steps will only insulate you from VCS's notification services. You still have to worry about the system's you're running on. This step can be a big headache (especially when you see what's coming up) and won't always do you any good. I'm including it for completeness' sake. The next thing you could do, would be to disable sendmail/mail on your systems. This can be done a variety of different ways, depending upon what OS and version you're running, but (generally), this simple method should do the trick:

a. Determine the PID's of your active sendmail/mailer daemons and kill them. Alternately, play it safe and shut them down with their respective shutdown scripts (not as much fun ;)
b. Move your sendmail/mail configuration files from their regular location to a different one (or just rename them) so that they can't restart if they try to.
c. Make sure nothing is running on your system, in cron, perhaps, (like cfengine) that will pro-actively fix the problem you've created to keep mail from getting off of the servers.

4. As noted, step 3 can be a hassle and may be a huge waste of time. Now you have to consider that outside, and inside, systems may also be monitoring your cluster's health. Programs like HP OpenView may have agents installed on your local systems and other computer systems may be set up outside of your cluster in order to test its availability from alternate locations. Any combination of these may still trigger errors and alerts once you start faulting a VCS resource on purpose.

At this point you can go with what might be the best option of them all (combined with steps 1 and 2): Stop your cluster dead, properly, so that none of the resources show as faulted or are affected in any way. You can do this very simply by running the following on your cluster's primary node:

host # hastop -all -force

This will bring down your cluster so that you can work on the configuration file, etc, but will not take down any of the resources that it manages (which should buy you some time, since none of them will report as "faulted"). There's still the outside chance that someone (or something) may be monitoring the "state" of your cluster externally. In that cause, you're probably screwed and should let whomever's keeping tabs know what you intend to do so that they will feel more comfortable ignoring it ;) Keep in mind, of course, that, when you bring your nodes back online, you should try to bring up the primary node first. This will avoid any issues with your primary node doing a "remote build" of its configuration file from any secondary, tertiary, etc, nodes. Again, depending on the complexity of your setup, this may make very little difference.

5. (Bonus step - Not recommended) Some folks prefer to "freeze" the cluster when they want to do maintenance. This is perfectly acceptable practice, as it will allow you to operate on resources within service groups in VCS without risking taking down the cluster or causing undesired failover of resources. It should be noted, however, that if you cause any of the basic functionality built into a resource to fail (which may happen if you're re-tooling a resource to a good degree), that resource will still show up in VCS as faulted until its basic functionality (as understood by VCS) is restored. This faulted state will most probably cause the NotifierMngr and ResourceOwner's to be notified of the error condition!

And, there you have it. I realize this was somewhat of a quick glossing over of the subject, but, as those of us who work on VCS have probably quite-painfully learned, trying to explain VCS (even at the most basic level) can become a huge task if you want to cover every possible aspect or angle. That's why so many 5000 page books are published every year on how to open the box and find the serial number ;)

Cheers,

, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Tuesday, February 12, 2008

Manually Editing The Main VCS Configuration File The Right Way

Hey again,

If you're like me and work on all sorts of things in the course of your sysadmin duties, working with Veritas Cluster Server has its pros and cons. On the one hand, it's great once it's all set up and working correctly, and troubleshooting it is fairly easy if you're working off of a solid foundation. On the other hand, making simple additions to the configuration file can be a lengthy and confusing ordeal.

Here's a little trick I picked up along the way, in my experience working with VCS. It's not exactly kosher (if you like to follow rules to the letter), but it works and it's 99% legitimate :)

Let's say, for instance, you wanted to add an NFS mount resource to your VCS cluster configuration. For ease of example, we'll assume you either know the syntax of the main configuration file (main.cf) or can find another cluster setup somewhere else on your network that you can "borrow" from (always more convenient ;) and modify.

Note that this entire process will keep VCS happy and won't cause any sort of split-brain condition that might result from your just manually editing the configuration file and rebooting your server (If things are working correctly, VCS will probably just overwrite your changes if you do this, anyway)

The first thing you'll want to do is to open the main.cf file for editing in memory. You can do this with the haconf command, like so:

host # haconf -makerw

Now, copy off ( just in case ) and open the main.cf file (Generally located in the /etc/VRTSvcs/conf/config directory) in your favorite editor and paste that other configuration addition to the bottom of your config file. We'll assume it's something simple, like this (Note that I haven't included the entire entry for brevity's sake and that you'll also not want to include the "dependency tree" lines. The ones that start with "//" - These will be created for you during this process):

group nfs_sg (
SystemList = { host1 = 0, host2 = 1 }
AutoStartList = { host1, host2 }
)

DiskGroup nfsdg (
DiskGroup = nfsdg
)
...


Now, you'll want to modify the parts that need modifying to suit your particular setup and move your augmented main.cf to another directory:

host # cp /etc/VRTSvcs/conf/config/main.cf /temporary/directory/main.cf

and verify that it's okay (Technically, you could still do this while it's in the regular directory) by running hacf:

host # hacf -verify /temporary/directory <--- This command will assume that there is a main.cf file in the directory you specify

Then, and this is the part that takes the most work out of your job, use hacf to create a commands file (These are commands you would normally have to type at the command line. There are so many of them involved in almost any addition, that I always update this way, just so there's no way I can forget one!):

host # hacf -cftocmd /temporary/directory

This will create a file called main.cmd in the /temporary/directory. This is chock full of commands that you would normally have to execute in order to augment your VCS configuration.

After backing up the main.cmd file in /etc/VRTSvcs/conf/config to, say, main.cmd.bak (Note: Never use the suffix .previous as this is the VCS default and your backup may get overwritten if you run subsequent haconf commands!), move your new main.cmd command into the VCS configuration directory, like so:

host # cp /temporary/directory/main.cmd /etc/VRTSvcs/conf/config/main.cmd

Now, instead of using the "hacf -cmdtocf" command (which should copy the main.cmd file to your configuration, but doesn't always work correctly), do this instead: Simply execute the main.cmd file you just created:

host # sh /etc/VRTSvcs/conf/config/main.cmd

Note: You're going to get a million errors flying across the screen. These are all warning you that you are attempting to add or modify a resource that already exists or has the property you're attempting to change it to. Don't worry about these at all. The errors result in the commands not being executed. Running the entire main.cmd file, and just ignoring those errors, ensures that every "new" command you need to enter will be entered :)

Now that you're all done "running all those commands" you can close the main.cf file in memory the proper way with haconf again, like this:

host # haconf -dump -makero

Now your new resource will have been added to the main configuration file, VCS will be happy and propagate the change to all the nodes in your cluster and you should be able to begin monitoring your new resource and, possibly, troubleshooting any errors that occur if you got any little part of your new entry wrong. Fun ;)

Best wishes,

, Mike




Monday, January 26, 2009

Adding NFS Management To An Existing VCS Cluster - Part One

Hey there,

Today we're going to take a look at a subject we've covered a bit in the past. If, after reading today's post, you want to things the quick and dirty way, check out our previous post on manually editing the main.cf file.

Today's task, although it sounds grim (since it involves VCS, mostly ;), is actually fairly simple to do and can be done in at least two different ways.

For today's purposes, we're going to show one of those ways and we're going to assume (per the title) that we're going to add an NFS resource to an existing VCS cluster on Linux or Unix (with special notes for any OS-specific steps). We're also going to assume that we're dealing with a two-node cluster since that's easier to write about and cuts down on my over-explanation ;) Also, we've disabled all NFS services on the host OS (meaning that we've stopped them and removed them from init scripts so that the OS won't try to manage them itself! A simple "ps -ef|grep nfs" check before you proceed should show you any remaining running nfs programs which you should kill)

1. The simple way (according to me :)

a. Bring down your entire VCS cluster (all nodes) from the node that you'll be doing the modification of the main.cf file on. It's VERY IMPORTANT that you use the -force flag when running the "hastop -all" command, as this will cause VCS to go down, but won't affect any of the applications running. To put it another way, if you just run "hastop -all" without the -force flag, VCS will go down on all of your cluster's nodes cleanly and bring down all the resources it manages with it. We want to do this as covertly as possible.

host # hastop -all -force

b. Now, proceed to edit your main.cf file. To totally cover our arses, we're going to make a quick backup of main.cf and copy the existing version into a new directory to do our edits:

host # cp main.cf main.cf.`date +%m%d%y`
host # pwd
/etc/VRTSvcs/conf/config
host # mkdir tmp
host # cp main.cf tmp/.
host # cd /etc/VRTSvcs/conf/config/tmp
host # vi main.cf


c. Make the changes and save (Note that you don't have to add any additional "includes" - like types.cf - to the file, as the NFS resource is already supported in the existing types.cf include file). All additions here should be made within the service group that the NFS resource will be a member of. Our service group is going to be called SG (there I go, getting all creative again ;). Add the following lines within the service group definition. These should be added after the SystemList definition, which I've kept in the example (but you don't need to modify), here:

group SG (
SystemList = { host1 = 0, host2 = 1 }
AutoStartList = { host1 }
)

NFS NFS_server_host1 (
Critical = 0
)

NFSRestart NFS_nfsrestart_host1 (
Critical = 0
NFSRes = NFS_server_host1
LocksPathName = "/opt/VRTSvcs/lock"
NFSLockFailover = 1
)

Share SHR_sharename_host1 (
Critical = 0
PathName = "/your/nfs/shared/directory"
Options = "rw,anon=0"
)

NFS_nfsrestart_host1 requires hostip1
SHR_sharename_host1 requires MNT_mount1_host1
SHR_sharename_host1 requires NFS_server_host1


You'll note that I set all of the new entries to non-critical (Critical = 0), since VCS's default is to make the resource critical, which would mean that, if we made mistake, the NFS resource bombing would cause a failover that we don't want. You can, and should, remove this line once you know everything is good. Also, check out the 3 new dependency-tree lines I've added. You don't need to modify the pretty commented version that VCS creates automatically; just the 3 basic dependencies (or requirements) that your NFS resource should rely on.

d. Now, we just need to check our new configuration to make sure it's syntactically correct:

host # hacf -verify /etc/VRTSvcs/conf/config/tmp/main.cf

if this shows any errors, fix them and run this command again until it just returns you to a blank prompt. Although it can be aggravating, VCS' hacf command doesn't tell you anything if it thinks the main.cf is fine ;)

e. Since everything went well, we just need to move our new main.cf file into its proper location (/etc/VRTSvcs/conf/config/). We've already backed up the original, earlier, so - if things get nuts, or the addition causes problems - we can switch back to the working configuration fairly quickly.

host # pwd
/etc/VRTSvcs/conf/config/tmp
host # mv main.cf ../.


f. Now, we just need to start up VCS again.

IMPORTANT NOTE: Only start VCS on the node on which you did your edits. Do not start it on any other nodes until the node you're working on is up and running VCS without errors. You can check the status of your host's VCS processes using either "hastatus -sum" over and over again, or just "hastatus" which provides information in real time, but is slightly more confusing to read than the output of "hastatus -sum." If you get an error about the "lock" file, just touch (create it yourself) the file you named as your lockfile in the new main.cf. You can touch it even if it exists, just to be sure, without causing any issues:

host # touch /opt/VRTSvcs/lock
host # hastart


g. Now that we've established the VCS is running fine on our main node, we can start up the other node(s). "hastatus -sum" and "hastatus" will show them (at an early point) in a REMOTE_BUILD state. This indicates that they're updating their main.cf's to match the one that we just changed.

Voila! All set. Once you're happy, be sure to set the NFS resource to "critical," if you want that. It's as easy as typing these commands on the command line (again, on your main node, if possible):

host # haconf -makerw
host # hares -modify NFS_server_host1 Critical 1
host # hares -modify NFS_nfsrestart_host1 Critical 1
host # hares -modify SHR_sharename_host1 Critical 1
host # haconf -dump -makero


A special note for Solaris 10 users: If you want to use VCS to manage your NFS resources, you'll need to delete certain resources from the SMF framework. The following commands should do the trick for you:

host # svccfg delete -f svc:/network/nfs/server:default
host # svccfg delete -f svc:/network/nfs/status:default
host # svccfg delete -f svc:/network/nfs/nlockmgr:default


Tomorrow, we'll take a look at another way to do this without taking VCS down :)

Cheers,

, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Tuesday, February 26, 2008

Offlining, Failing Over And Switching in VCS

Hey there,

Today we're going to address a question that's asked commonly enough by foks who use Veritas Cluster Server: What's the difference between offlining, failing over and switching when dealing with service groups across multiple nodes? That doesn't seem like a very common question. I'm sure it's probably hardly ever phrased that way ;)

Anyway, to the point, all three options above are useful (hence your ability to avail yourself of them) and sufficiently distinct that you should be sure you're using the one you want to, depending on what ends you wish to achieve. All of this information is fairly general and should work on VCS for Linux as well as Unix.

We'll deal with them one bye one, outlining what they basically do, with a short example command line to demonstrate, where applicable. None of this stuff is hard to pick up and run with, as long as all the components of your VCS cluster are setup correctly. Occasionally, you may see errors if things aren't exactly perfect, like we noted in this post on recovering indefinitely hung VCS resources.

1. Offlining. The distinction to be made here, most plainly, is that, when you offline a resource or service group, you are "only" doing that. This differs from failover and switching in that the service group you offline with this particular option is not brought online anywhere else as a result. So, when you execute, for instance:

host # hagrp -offline YOUR_SERVICE_GROUP -sys host1

that service group, and its resources, generally, are taken offline on host1, and nothing else happens. If you're operating in an environment where systems don't run service groups concurrently (active/active), you will have effectively "shut down" that service group, and any services it provides, for the entire cluster.

2. Failing Over. This is more of a concept than any particular command. When you have your cluster setup to fail over, if a resource or service group, etc, goes offline on one node (host1, for instance) and it wasn't brought offline on purpose, VCS will naturally attempt to bring it online on the next available node (listed in your main.cf configuration file). Needless to say, if only one host is listed for a particular service group, its failure on one host will mean the failure of the entire service group. It also obviates the use of VCS in the first place ;)

3. Switching. This is what most folks want to do when they "offline" a service group, as in point 1. Although, since VCS automatically switches unexpectedly offlined resources on its own (when it's set up to), it's reasonable for someone new to the product to assume that offlining a service group would engage VCS in a switching activity. Unfortunately, this isn't the case. If you want to switch a service group from host1 to host2, for example, these would be the command options you'd want to give to hagrp:

host # hagrp -switch YOUR_SERVICE_GROUP -to host2 <--- Assuming you're running this from host1.

Hopefully this little guided mini-FAQ helped out with differentiating between the concepts. If you find the command line examples valuable, even better :)

Happy failing! ;) Over.


, Mike




Tuesday, September 30, 2008

How To Resolve Veritas Disk Group Cluster Volume Management Problems On Linux or Unix

Hey There,

Today we're going to look at an issue that, while it doesn't happen all that often, happens just enough to make it post-worthy. I've only seen it a few times in my "career," but I don't always have access to the fancy software, so this problem may be more widespread than I've been lead to believe ;) The issue we'll deal with today is: What do you do when disk groups, within a cluster, conflict with one another? Or, more correctly, what do you do when disk groups within a cluster conflict with one another even though all the disk is being shared by every node in the cluster? If that still doesn't make sense (and I'm not judging "you," it just doesn't sound right to me, yet ;) what do you do in a situation where every node in a cluster shares a common disk group and, for some bizarre reason, this creates a conflict between nodes in the cluster and some of them refuse to use the disk even though it's supposed to be accessible through every single node? Enough questions... ;)

Check out these links for a smattering of other posts we've done on dealing with Veritas Volume Manager and fussing with Veritas Cluster Server. Some of the material covered may be useful if you have problems with any of the concepts glossed over in the problem resolution at the end.

Like I mentioned, this "does" happen from time to time, and not for the reasons you might generally suspect (like one node having a lock on the disk group and refusing to share, etc). In fact, the reason this happens sometimes (in this very particular case) is quite interesting. Even quite disturbing, since you'd expect that this shouldn't be able to happen.

Here's the setup, and another reason this problem seems kind of confusing. A disk group (we'll call it DiskGroupDG1 because we're all about creativity over here ;) is being shared between 2 nodes in a 2 node cluster. Both nodes have Veritas Cluster Server (VCS) set up correctly and no other problems with Veritas exist. If the DiskGroupDG1 disk group is imported on Node1, using the Cluster Volume Manager (CVM), it can be mounted and accessed by Node2 without any issues. However, if DiskGroupDG1 is imported on Node2, using CVM, it cannot be mounted and/or access by Node1.

All things being equal, this doesn't readily make much sense. There are no disparities between the nodes (insofar as the Veritas Cluster and Volume Management setup are concerned) and things should be just peachy going one way or the other. So, what's the deal, then?

The problem, actually, has very little to do with VCS and/or CVM (Although they're totally relevant and deserve to be in the title of the post -- standard disclaimer ;). The actual issue has to do, mostly, with minor disk numbering on the Node1 and Node2 servers. What???

Here's what happens:
In the first scenario (where everything's hunky and most everything's dorey) the DiskGroupDG1 disk group is imported by CVM on Node1 and Node1 notices that the "minor numbers" of the disks in the disk group are exactly the same as the "minor numbers" on disk it already has mounted locally. You can always tell a disk's (or any other device's) minor number by using the ls command on Linux or Unix, like so:

host # /dev/dsk # ls -ls c0t0d0s0
2 lrwxrwxrwx 1 root root 41 May 11 2001 c0t0d0s0 -> ../../devices/pci@1f,4000/scsi@3/sd@0,0:a
host # /dev/dsk # ls -ls ../../devices/pci@1f,4000/scsi@3/sd@0,0:a
0 brw-r----- 1 root sys 32, 0 May 11 2001 ../../devices/pci@1f,4000/scsi@3/sd@0,0:a
<-- In this instance, the device's "major number" is 32 and the device's "minor number" is 0. Generally, with virtual disks, etc, you won't see numbers that low.

Now, on Node1, since it recognizes this conflict on import, does what Veritas VM naturally does to avoid conflict; it renumbers the imported volumes ("minor number" only) so that the imported volumes won't conflict with volumes in another disk group that's already resident on the system it's managing. Therefore, when Node2 attempts to mount, with CVM, the command is successful.
In the second scenario (where thing are a little bit hunky, but not at all dorey), Node2 imports the DiskGroupDG1 disk group and none of the minor numbers in that disk group's volumes conflict with any of its local (or already mounted) disk. The disk group volumes are imported with no error, but, the "minor numbers" are not temporarily changed, either. You see where this is going. It's a freakin' train wreck waiting to happen ;)

Now, when Node1 attempts to mount, it determines there's a conflict, but can't renumber the "minor numbers" on the disk group's volumes (since they're already imported and mounted on Node2) and, therefore, takes the only other course of action it can think of and bails completely.

So, how do you get around this for once and all time? Well, I'm not sure it's entirely possible to anticipate this problem with a variable number of nodes in a cluster, all with independent disk groups and, also, sharing volume groups between nodes, although you could take simple measures to prevent it most of the time (like running ls against every volume in every disk group in a cluster every now and again and making sure no conflicts existed. The script should be pretty easy to whip up).
Basically, in this instance (and any like it), the solution involves doing what Veritas VM did in the first scenario; except doing it all-the-way. No temporary-changing of "minor numbers." For our purposes, we'd like to change them permanently, so that they never conflict again! It can be done in a few simple steps.

1. Stop VCS on the problem node first.

2. Stop any applications using the local disk group whose "minor numbers" conflict with the "minor numbers" of the volumes in DiskGroupDG1.

3. Unmount (umount) the filesystems and deport the affected disk group.

4. Now, pick a new "minor number" that won't conflict with the DiskGroupDG1 "minor numbers." Higher is generally better, but I'd check the minor numbers on all the devices in my device tree just to be sure.

5. Run the following command against your local disk group (named, aptly, LocalDG1 ;) :

host # vxdg reminor LocalDG1 3900 <-- Note that this number is the base, so every volume, past the initial, within the disk group will have a "minor number" one integer higher than the last (3900, 3901, etc)

6. Reimport the LocalDG1 disk group

7. Remount your filesystems, restart your applications and restart VCS on the affected node.

8. You don't have to, but I'd do the same thing on all the nodes, if I had a window in which to do it.

And, that would be that. Problem solved.

You may never ever see this issue in your lifetime. But, if you do, hopefully, this page (or one like it) will still be cyber-flotsam on the info-sea ;)

Cheers,

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, June 17, 2009

Easily Removing VCS resources From A Running Configuration On Unix Or Linux

Hey there,

Today's post is going to be a bit brief (which is a relative statement. If you've never been here before, this post may seem incredibly long ;). The subject matter (as noted in the Subject ;) deals with how to easily, and safely, remove resources from a running VCS (Veritas Cluster Server) configuration. And, just as a matter of course, I feel I must qualify the previous statement by noting that if you remove a vital resource from your running VCS configuration, no matter how safely and correctly you do it, you may end up with a major headache ;) (Please see our other posts on VCS for Linux or Unix if any part of this run-down requires further explanation. Hopefully, we've already covered it :)

And here we go. The method to doing this is so simple, I will be writing the rest of this post in "Dick and Jane" style. See VCS configuration. See VCS configuration running. See Sysadmin. See Sysadmin answer 15 Instant Messages and respond to 3 emails while fielding questions on a conference call... No. That won't work ;)

Here we go, for real. Removing VCS resources from a running VCS configuration can be extremely simple. In fact, in order to make sure that you not only do it simply, but also correctly, you're going to make use of a file that VCS creates automatically when you save your main.cf file. That file is called "main.cmd" The "main.cmd" file should exist already, but if there's any reason you have to doubt that your main.cmd is correct (or you just don't have one - it should be located in /etc/VRTSvcs/conf/config), you can always create one from the running configuration by doing the following with the "hacf" command (I do this, usually, just to be sure):

host # cp /etc/VRTSvcs/conf/config/main.cf /var/tmp/main.cf
host # hacf -cftocmd /var/tmp


The above sequence of commands will create a file named "main.cmd" in your /var/tmp directory. This file contains every single command line (from the most basic to the most specific) that you would need to completely recreate your VCS configuration (of course, using the "-cmdtocf" flag might be a bit easier ;) As such, you can, of course, also use this file as a guide to easily "add" resources to your running VCS config, but that's beyond the scope of this post. "hacf" also allows you to use the "-dest" option so you can put the main.cmd that you created in a separate location, like so:

host # hacf -cftocmd /var/tmp -dest /home/user1/.

And here's where it gets easy :)

Let's say that you wanted to remove a Veritas disk volume from your running VCS configuration (which would, also, usually mean that you would want to remove the mount point from VCS control and remove any dependencies). Assuming that you know the volume's name, you can find out everything you need to know about the volume and the mountpoint, including dependency linkage, by simply using the "grep" command against your main.cmd file (Note that, even if you don't know your volume's name, you can figure that out just as easily by simply grepping for "Volume" - the VCS resource type, which is case sensitive - in your main.cmd file, and looking for the name of the disk volume that way).

Then let's say that your volume's name is VOL_volume54_host. And let's also say that you have the privilege and means to execute the VCS commands you'll be discovering (I'm heavily fighting a strong Airplane joke urge right now ;)

host # grep VOL_volume54 /var/tmp/main.cmd
hares -add VOL_volume54_host Volume SG_host
hares -modify VOL_volume54_host Critical 0
hares -modify VOL_volume54_host Volume volume54
hares -modify VOL_volume54_host DiskGroup hostdg
hares -modify VOL_volume54_host Enabled 1
hares -link MNT_mount54_host VOL_volume54_host
hares -link VOL_volume54_host DG_hostdg_host


For now, we'll just use this information to find the disk mount (Mount) name and grep for that, as well:

host # grep MNT_mount54_host /var/tmp/main.cmd
hares -add MNT_mount54_host Mount SG_host
hares -modify MNT_mount54_host Critical 0
hares -modify MNT_mount54_host MountPoint "/disk54/files"
hares -modify MNT_mount54_host BlockDevice "/dev/vx/dsk/hostdg/volume54"
hares -modify MNT_mount54_host FSType vxfs
hares -modify MNT_mount54_host MountOpt largefiles
hares -modify MNT_mount54_host FsckOpt "%-y"
hares -modify MNT_mount54_host CkptUmount 1
hares -modify MNT_mount54_host SecondLevelMonitor 0
hares -modify MNT_mount54_host SecondLevelTimeout 30
hares -modify MNT_mount54_host Enabled 1
hares -link MNT_mount54_host VOL_volume54_host
hares -link RANDOMRESOURCE_resourcename_host MNT_mount54_host


Now you'll put these all together in order (I prefer to remove link dependencies first, then remove the mount followed by the volume - doing otherwise may cause problems for you). Note that one dependency link showed up in both greps. You can run the same command twice without doing any harm, but I'm stripping the duplicate out for neatness' sake. You're almost ready to go. At this point, you'll end up with this list of VCS commands (which you shouldn't run, but - if you do, accidentally - won't cause any harm):

hares -link MNT_mount54_host VOL_volume54_host
hares -link VOL_volume54_host DG_hostdg_host
hares -link RANDOMRESOURCE_resourcename_host MNT_mount54_host
hares -add MNT_mount54_host Mount SG_host
hares -modify MNT_mount54_host Critical 0
hares -modify MNT_mount54_host MountPoint "/disk54/files"
hares -modify MNT_mount54_host BlockDevice "/dev/vx/dsk/hostdg/volume54"
hares -modify MNT_mount54_host FSType vxfs
hares -modify MNT_mount54_host MountOpt largefiles
hares -modify MNT_mount54_host FsckOpt "%-y"
hares -modify MNT_mount54_host CkptUmount 1
hares -modify MNT_mount54_host SecondLevelMonitor 0
hares -modify MNT_mount54_host SecondLevelTimeout 30
hares -modify MNT_mount54_host Enabled 1
hares -add VOL_volume54_host Volume SG_host
hares -modify VOL_volume54_host Critical 0
hares -modify VOL_volume54_host Volume volume54
hares -modify VOL_volume54_host DiskGroup hostdg
hares -modify VOL_volume54_host Enabled 1


Now, you'll strip this down and "reverse" the commands. And by "reverse" I mean reverse the intent, and not the order ;) So, all -link options will become -unlink options, etc. This will leave you with the following (we're removing every "unnecessary" command, since, for instance, removing a volume automatically removes all of its attributes):

hares -unlink MNT_mount54_host VOL_volume54_host
hares -unlink VOL_volume54_host DG_hostdg_host
hares -unlink RANDOMRESOURCE_resourcename_host MNT_mount54_host
hares -delete MNT_mount54_host
hares -delete VOL_volume54_host


and that's a much shorter, and much nicer, list. Now all you have to do to remove the resources successfully is to do the following (Also, consider that you may want to shut down VCS on all but the primary node, if possible, so that your new configuration doesn't get overwritten, and that all your secondary, tertiary, etc, VCS nodes do a remote build of the new configuration from your primary node):

host # haconf -makerw
host # hares -unlink MNT_mount54_host VOL_volume54_host
host # hares -unlink VOL_volume54_host DG_hostdg_host
host # hares -unlink RANDOMRESOURCE_resourcename_host MNT_mount54_host
host # hares -delete MNT_mount54_host
host # hares -delete VOL_volume54_host
host # haconf -dump -makero


And you're all set. Easy Peasy :)

Cheers,

, Mike


Banner for College Student-Oriented Sites (728 version 1)



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Thursday, February 5, 2009

A Little VCS NFS Gotcha On Solaris 10

Hey again,

We're going back to the Solaris 10 Unix well, reaching back a little (as opposed to the 14 month reach-back we did yesterday ;) and adding a little something to our posts on adding NFS management to an existing VCS cluster as well as the follow up on how to do the exact same thing without taking your VCS cluster offline. Today's post is actually another little bit of fix-it knowledge to keep in the back of your hat (if that's even an expression anyone's ever used... if not, consider it © ® ™ us ;). And, of course, this piece of knowledge came to everyone by accident. Actually, by virtue of an accident... the answer was found methodically... I think ;)

In any event, after extensively field-testing the methods espoused in the earlier posts referenced above, a deployment of clustered servers to an offsite location ended up having an issue that we weren't able to anticipate (or cause to occur in previous cookie-cutter-similar deployments). For some reason, when we rolled this cluster out, NFS just refused to work in a failover capacity. Actually, it only failed, specifically, to allow the main node to mount on the failover node. This problem seems pedestrian (even still ;) - the only odd thing was that it had never happened before under equal circumstances.

Here's what we figured out along the way (and how to fix it, too ;) For our purposes today (and the way it was then) the NFS cluster component works fine on node-b, but node-a can't mount the NFS resource when it fails over to node-b.

1. The first thing most people do in any investigation is to see if the basic stuff is all up and running. We don't like to be different, so we duly checked that all of the required VCS resources were up and online. They were; which explained the puzzling ONLINE state ;)

2. We then proceeded to ensure that, in fact, node-b was sharing out the NFS resource. Commands like showmount indicated that it, indeed, was. A little research into the subject showed that the issue we ended up having can indicate an RPC failure at this point, as well, but it's best to try step 3, too, just to be sure the problem isn't confined to a single server (although the fix for it is the same no matter which way your story goes ;)

3. Then we finally struck gold, and got an actual error, when we tried to hit the mount from node-a:

node-a # showmount -e node-b
showmount: node-a: RPC: Rpcbind failure - RPC: Authentication error
node-a # rpcinfo -p node-b
rpcinfo: can't contact portmapper: RPC: Authentication error; why = Failed (unspecified error)


4. Unspecified errors are the best kind of errors you can get since there are a much wider variety of possible solutions you can come up with... Or, maybe I have that backwards... There's really not much more to step 4. This step is a practice in surrealism ;)

5. It turns out that the answer lay in setting rpcbind properties (away from the defaults on both servers). The answer to the problem (or the fix, if you will) actually makes more sense than the way things "usually" work. The first thing we did was to set rpcbind to "global" on both nodes. By default, it was set to "local_only." We double confirmed that this is still the case on other cluster setups we have running, in which everything is hunky-dory. You also need to do these steps on both nodes (or all nodes) in your cluster, while, here, we're only showing what we typed on the active NFS resource-sharing node:

node-b # svcprop network/rpc/bind:default | grep local_only <-- See if the local_only property is set
config/local_only boolean true <-- and there it is!

then move on to fixing the problem (again on both nodes) by setting the rpcbind configuration value to global (which, in the instance of rpcbind, actually means setting the local_only attribute to "false"):

node=b # svccfg
svc:> select network/rpc/bind
svc:/network/rpc/bind> setprop config/local_only=false
svc:/network/rpc/bind> quit


6. Then, just double check to make sure you've gotten it all set up correctly:

node-b # svcprop network/rpc/bind:default | grep local_only
config/local_only boolean true


...well, that's not right, but don't give up just yet! Keep typing. Type, Forrest, Type! ;)

node-b # svcadm refresh network/rpc/bind:default
node-b # svcprop network/rpc/bind:default | grep local_only
config/local_only boolean false


there... that's better.

7. Finally, just make sure you can mount your NFS resource from whichever node isn't currently hosting the NFS resource. You don't necessarily have to test it on both nodes, once you've fixed this issue on both, but why risk the near-future embarrassment?

node-a # showmount -e node-b
export list for node-b:
/our/shared/directory (everyone)


And that's that. You should be good to go :) Since all's well that ends well, we'll try not to leave you with any clichés in our farewell. Parting is, after all, such sweet sorrow. At least until tomorrow :)

Cheers,

, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Monday, November 19, 2007

Recovering Faulted VCS Resources In An Indefinitely Hung State!

If you've ever found yourself in a situation where a Veritas Cluster Server (VCS) resource/service group is reporting faulted, you may have seen the situation described here. Today we're going to focus on one specific condition where the most common wisdom I've seen on the web is to do an "hastop -local -force" and follow up with an "hastart" to solve the problem. Common sense dictates that this is the last thing you'll want to do, unless you're on a testing environment, as you'll be completely downing the cluster in the process.

The reason the stop/start answer is the most common is because, since the resource/service group's "resource" is waiting to go OFFLINE, while the resource/service "group" is trying to start, or go ONLINE, simply attempting to "online" or "offline" either only results in infinite hang. This is because each entity's successful action depends on the other entity's failure and VCS won't fail if it can wait instead.

You'll know you're looking at this sort of error if you see the following (this generally can be found by running "hasum" or "hastatus -summary"):

A. The resource/service "group" will be showing in a "STARTING|PARTIAL" state
B. An individual "resource" within that resource/service "group" will be showing in the "W_OFFLINE" state.

The following steps to resolution are certainly not the only way to get everything back to an "up" state and, also, assume that there is nothing really "wrong" with the individual resource or resource/service group. That sort of troubleshooting is outside the scope of this blog-sized how-to.

So, again, our assumptions here are:
1. An Oracle database resource/service group, named "oracledb_sg" has faulted on the server "vcsclusterserver1.xyz.com."
2. An individual resource, a member of the resource/service group "oracledb_sg," named "oracledb_oracle_u11192007," is really the only thing that's failed, or shows as failing in "oracledb_sg."
3. There is actually nothing wrong with the resource or the resource/service group. Somehow, while flipping service groups back and forth between machines, somebody made an error in execution or VCS ran into a state problem that it caused by itself ("split brain" or some similar condition).
4. Note that we've reached these assumptions based partially on the fact that the resource is waiting to go OFFLINE, and the resource/service group is waiting to go ONLINE (stuck in STARTING|PARTIAL), on the same server!

And the following are the steps we could take to resolve this issue and get on with our lives:

1. First, get a summary of the resource group that you've been alerted as failed or faulted, like this:

hastatus -summary|grep oracledb_sg|grep vcsclusterserver1 (or some variation thereof, depending on how much information you want to get back)

B oracledb_sg vcsclusterserver1.xyz.com Y N STARTING|PARTIAL
C oracledb_sg Oracle oracledb_oracle_u11192007 vcsclusterserver1.xyz.com W_OFFLINE


2. If you want to be doubly sure, you'll check that you get the same resource id ( oracledb_oracle_u11192007 ) from the "hares" command, and also get the same status:

hares -display|grep oracledb|grep "waiting to" --> to get the resource id a second way.

3. At this point you still can't "online" the resource until you "clear" it (of the W_OFFLINE state flag).

hares -clear oracledb_oracle_u11192007 -sys vcsclusterserver1.xyz.com
hagrp -online oracledb_sg -sys vcsclusterserver1.xyz.com


Now your resource/service group should be back in the straight ONLINE state, and you shouldn't see any messages (In "hasum" or "hastatus -summary" output) regarding the individual resource. Time to relax :)

, Mike





Friday, February 13, 2009

Correcting Auto-Disabled Service Groups In Veritas Cluster Server

Hey there,

Today's post is going to be fairly specific. I'm either tapped-out on creativity or I'm writing this post after working into the night, or both ;) Here's a little something I picked up today concerning VCS and service groups for either the Unix or Linux version.

If you've ever wondered why, every once in a while, when you have a VCS service group problem and you get this error when you try to correct it:

service group is auto-disabled in the cluster!

you may have had a nervous breakdown at some point in your life ;) Adding to the confusion is that the default help output for the hagrp command only shows an autodisable option (? Either that or I was really tired while I was troubleshooting)

Now, there are good reasons for your service group to go into this state. It doesn't just do it to piss you off ;) For instance, if the basic VCS service (or engine) isn't running on a particular node in your cluster, this attribute gets set. That way, if it comes back online unexpectedly (or someone unexpectedly brings it online while you're working on something else in the VCS config) it won't start contending for resources and causing a possibly more confusing situation. It'll also do this if none of the resources in a service group have been probed (also generally indicative of a "bad bad failure") and if you lose all of your high priority heartbeats. Technically, I think it only does this when you only have disk heartbeats left, but I believe it also does it if you get down to having just one low priority link active and don't use disk heartbeats (I could test this out, but then I might have to keep working until tomorrow morning. I'm at the point where senseless voodoo-thinking has taken over ;)

Luckily, it's a really simple situation to fix, if you know the service group shouldn't be disabled and/or you need to bring an autodisabled service group up. The command line that does that is:

host # hagrp -autoenable YOUR_SERVICE_GROUP_NAME -sys THE_NODE_YOU_WANT_TO_AUTOENABLE_THE_SG_ON

Another way to get around this is to manually probe all of the resources in that service group. It seems like a waste of time to me, but I'm glad the option exists, just in case the above command line fails to do the trick for one reason or another.

host # hares -probe RESOURCE_NAME -sys NODE_YOU_WANT_TO_PROBE_RESOURCE_ON

I find this can be made somewhat simpler by using "hares -display," and piping that to a grep and awk statement, although doing this could result in errors that can end up making your life even more miserable if you don't put in extra echo statements to list out what exact resources are being probed!

And, as my exit nears, another way to do this is to simply use your /etc/VRTSvcs/conf/config/main.cf (assuming it has service groups in it that aren't autodisabled) and get the command line (same as above) by using hacf:

host # hacf -cftocmd /etc/VRTSvcs/config/config/

This will produce a file named "main.cmd", and you can then just "grep -i able" out of it and you should have one line in there, at least, that prints out the exact command you need to run (with your specific service group and system name some times) to unset that flag (or attribute)

Here's to a happy Valentine's day, tomorrow, for almost all of you ;)

Cheers

, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.