Easily Removing VCS resources From A Running Configuration On Unix Or Linux

Hey there,

Today's post is going to be a bit brief (which is a relative statement. If you've never been here before, this post may seem incredibly long ;). The subject matter (as noted in the Subject ;) deals with how to easily, and safely, remove resources from a running VCS (Veritas Cluster Server) configuration. And, just as a matter of course, I feel I must qualify the previous statement by noting that if you remove a vital resource from your running VCS configuration, no matter how safely and correctly you do it, you may end up with a major headache ;) (Please see our other posts on VCS for Linux or Unix if any part of this run-down requires further explanation. Hopefully, we've already covered it :)

And here we go. The method to doing this is so simple, I will be writing the rest of this post in "Dick and Jane" style. See VCS configuration. See VCS configuration running. See Sysadmin. See Sysadmin answer 15 Instant Messages and respond to 3 emails while fielding questions on a conference call... No. That won't work ;)

Here we go, for real. Removing VCS resources from a running VCS configuration can be extremely simple. In fact, in order to make sure that you not only do it simply, but also correctly, you're going to make use of a file that VCS creates automatically when you save your main.cf file. That file is called "main.cmd" The "main.cmd" file should exist already, but if there's any reason you have to doubt that your main.cmd is correct (or you just don't have one - it should be located in /etc/VRTSvcs/conf/config), you can always create one from the running configuration by doing the following with the "hacf" command (I do this, usually, just to be sure):

host # cp /etc/VRTSvcs/conf/config/main.cf /var/tmp/main.cf
host # hacf -cftocmd /var/tmp

The above sequence of commands will create a file named "main.cmd" in your /var/tmp directory. This file contains every single command line (from the most basic to the most specific) that you would need to completely recreate your VCS configuration (of course, using the "-cmdtocf" flag might be a bit easier ;) As such, you can, of course, also use this file as a guide to easily "add" resources to your running VCS config, but that's beyond the scope of this post. "hacf" also allows you to use the "-dest" option so you can put the main.cmd that you created in a separate location, like so:

host # hacf -cftocmd /var/tmp -dest /home/user1/.

And here's where it gets easy :)

Let's say that you wanted to remove a Veritas disk volume from your running VCS configuration (which would, also, usually mean that you would want to remove the mount point from VCS control and remove any dependencies). Assuming that you know the volume's name, you can find out everything you need to know about the volume and the mountpoint, including dependency linkage, by simply using the "grep" command against your main.cmd file (Note that, even if you don't know your volume's name, you can figure that out just as easily by simply grepping for "Volume" - the VCS resource type, which is case sensitive - in your main.cmd file, and looking for the name of the disk volume that way).

Then let's say that your volume's name is VOL_volume54_host. And let's also say that you have the privilege and means to execute the VCS commands you'll be discovering (I'm heavily fighting a strong Airplane joke urge right now ;)

host # grep VOL_volume54 /var/tmp/main.cmd
hares -add VOL_volume54_host Volume SG_host
hares -modify VOL_volume54_host Critical 0
hares -modify VOL_volume54_host Volume volume54
hares -modify VOL_volume54_host DiskGroup hostdg
hares -modify VOL_volume54_host Enabled 1
hares -link MNT_mount54_host VOL_volume54_host
hares -link VOL_volume54_host DG_hostdg_host

For now, we'll just use this information to find the disk mount (Mount) name and grep for that, as well:

host # grep MNT_mount54_host /var/tmp/main.cmd
hares -add MNT_mount54_host Mount SG_host
hares -modify MNT_mount54_host Critical 0
hares -modify MNT_mount54_host MountPoint "/disk54/files"
hares -modify MNT_mount54_host BlockDevice "/dev/vx/dsk/hostdg/volume54"
hares -modify MNT_mount54_host FSType vxfs
hares -modify MNT_mount54_host MountOpt largefiles
hares -modify MNT_mount54_host FsckOpt "%-y"
hares -modify MNT_mount54_host CkptUmount 1
hares -modify MNT_mount54_host SecondLevelMonitor 0
hares -modify MNT_mount54_host SecondLevelTimeout 30
hares -modify MNT_mount54_host Enabled 1
hares -link MNT_mount54_host VOL_volume54_host
hares -link RANDOMRESOURCE_resourcename_host MNT_mount54_host

Now you'll put these all together in order (I prefer to remove link dependencies first, then remove the mount followed by the volume - doing otherwise may cause problems for you). Note that one dependency link showed up in both greps. You can run the same command twice without doing any harm, but I'm stripping the duplicate out for neatness' sake. You're almost ready to go. At this point, you'll end up with this list of VCS commands (which you shouldn't run, but - if you do, accidentally - won't cause any harm):

hares -link MNT_mount54_host VOL_volume54_host
hares -link VOL_volume54_host DG_hostdg_host
hares -link RANDOMRESOURCE_resourcename_host MNT_mount54_host
hares -add MNT_mount54_host Mount SG_host
hares -modify MNT_mount54_host Critical 0
hares -modify MNT_mount54_host MountPoint "/disk54/files"
hares -modify MNT_mount54_host BlockDevice "/dev/vx/dsk/hostdg/volume54"
hares -modify MNT_mount54_host FSType vxfs
hares -modify MNT_mount54_host MountOpt largefiles
hares -modify MNT_mount54_host FsckOpt "%-y"
hares -modify MNT_mount54_host CkptUmount 1
hares -modify MNT_mount54_host SecondLevelMonitor 0
hares -modify MNT_mount54_host SecondLevelTimeout 30
hares -modify MNT_mount54_host Enabled 1
hares -add VOL_volume54_host Volume SG_host
hares -modify VOL_volume54_host Critical 0
hares -modify VOL_volume54_host Volume volume54
hares -modify VOL_volume54_host DiskGroup hostdg
hares -modify VOL_volume54_host Enabled 1

Now, you'll strip this down and "reverse" the commands. And by "reverse" I mean reverse the intent, and not the order ;) So, all -link options will become -unlink options, etc. This will leave you with the following (we're removing every "unnecessary" command, since, for instance, removing a volume automatically removes all of its attributes):

hares -unlink MNT_mount54_host VOL_volume54_host
hares -unlink VOL_volume54_host DG_hostdg_host
hares -unlink RANDOMRESOURCE_resourcename_host MNT_mount54_host
hares -delete MNT_mount54_host
hares -delete VOL_volume54_host

and that's a much shorter, and much nicer, list. Now all you have to do to remove the resources successfully is to do the following (Also, consider that you may want to shut down VCS on all but the primary node, if possible, so that your new configuration doesn't get overwritten, and that all your secondary, tertiary, etc, VCS nodes do a remote build of the new configuration from your primary node):

host # haconf -makerw
host # hares -unlink MNT_mount54_host VOL_volume54_host
host # hares -unlink VOL_volume54_host DG_hostdg_host
host # hares -unlink RANDOMRESOURCE_resourcename_host MNT_mount54_host
host # hares -delete MNT_mount54_host
host # hares -delete VOL_volume54_host
host # haconf -dump -makero

And you're all set. Easy Peasy :)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Another Joke Server - The SPUD - Much Better Than The Other Gigantic One

Happy Saturday :)

I found another funny "hoax" on the web (Technically, I remembered that I'd found it a long time ago and then forgot to remember that I'd found it ;), even better than that Gigantic Old Server that I was silly enough to not research before I put the picture up here on this blog. In respect to the honesty of this online rag, I'm never going to retract that post and will leave it there, for however long Google lets me, to remind everyone that I'm a frickin' idiot some times (no further comments, please ;)

Enjoy The SPUD Server - They say it's vegetable powered, runs a perverted Linux kernel (keep it away from the kids ;), and that it's a hoax, but, even though a tuber is a type of vegetable, I'm still paranoid about being wrong. But not so much that I won't risk my reputation to get a post out here today ;)

Enjoy the pictures and check out the The main SPUD server site. There are links to a lot of other interesting projects there, as well :)

Cheers,

---

---

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Sun M-Series Enterprise Unix Servers And DSCP - Why?

SPECIAL NOTE: Friday's post on the absorption of knowledge was written on Blogspot in IE7 and suffered from some horrible display problems on FireFox and other browsers. It has been completely retooled. Apologies for any inconvenience that post caused. Even though I write some goofy rants, missing text wasn't supposed to be part of that post's intentionally cryptic style. Thanks :) - Mike

Hey there,

Today, we're going to look at DSCP (The Domain to Service processor Communication Protocol) and why it was introduced with the newer M series Enterprise servers from Sun. The first time I ran into it, I thought approximately the following: All right, here's something else that can make my life more complicated ;) Now, not only do I have to hop onto a private network (added security at work) to connect to the service processor (SP), but then I have to be sure that an entirely separate network is functioning correctly in order for me to connect to a domain on my machine if I want to get there directly from the SP. I'm naturally negative, but I keep it inside until I've given myself the time to realize that things are either not as bad or better than I initially assumed, which goes a long way toward maintaining good relationships with other human beings ;)

DSCP, although it "is" another layer of complication, is actually a pretty good idea. In essence, it's an attempt at damage control and resource sharing (which almost never works entirely), on par with earlier "HUGE" Sun computers, which does a pretty decent job at a relatively low cost. And, I use the word "cost" in reference to the amount of work you have to put into getting it all set up and working correctly. It's practically free, in that respect, so kudos (The plural, pronounced kyu-doze) to Sun for adding a little extra protection and making it an additon to an existing model that doesn't encumber the admin or user to a noticeable degree. Once it's set up, it may as well be non-existent. ...unless you have problems with it, but that's a general rule when it comes to anything.

Inter-process communication on the 6800/6900, etc boxes was handled pretty decently as well, but didn't really lay ground for the possiblity of scaling easily in the future. On those boxes, shared memory (for example) was "stuck" to a single SP, so if you had multiple SP's, you essentially had multiple machines, each alienated from the other with regards to system resources. The 6800/6900's were, as one would expect, worse in this regard than the 15k-and-up models (check out the price tags. Some times it can be an indicator of quality. Is the difference in ease of sharing worth it? Depends on your situation)

The 15k-and-up servers ran physical ethernet from the service processor(s) to the domain controllers (whereas, if I forgot to mention above, the older mailbox-architecture was imbedded and much more difficult to "change"). This setup made it slightly easier to re-allocate resources since you basically had a MAN setup on your Sun system (Maintenance Area Network for those of us who still care ;). Since the MAN operated at the application layer of the stack, all you had to do to scale up when you added a new product or application was assign a new TCP port. And that was that: Sharing made much simpler. There's a lot of nitty-gritty behind it all, but who wants to read about that? If you do, check out docs.sun.com and go nuts ;)

So, now, we've made it all the way up to the almost-present (we'll just consider it the "now" for now ;) and the M-Series servers. Since the MAN configuration of the 15k-and-up servers worked out so beneficially, that concept was guaranteed to be built in to the next generation of Enterprise computers. The one big drawback, from an architectural perspective, was the fact that the 15k-and-up servers had a whole bunch of externally exposed cables patched all over the place to maintain that separate MAN network (lots of ways to goof that up). The DSCP is Sun's way of taking care of that issue while maintaining the extended reliablity (and security) introduced with the MAN concept.

DSCP reproduces the 15k-and-up MAN using shared RAM, a pseudo-serial driver and PPP. To the user, this means all the benefits of a MAN, without all of the cords ;) Actually, it's a very nice implementation of physical-to-virtual transformation. And, as we all know, in about 10 to 15 years, we'll all be virtual and the only people left alive on the planet earth will be maintenance technicians ;)

The best thing about DSCP is how incredibly complicated it is to set up. Just kidding ;) It's actually incredibly simple. So simple that, if you're like me, you're wondering when a monkey is going to be able to start doing your job ;) Assuming an unlimited amount of Domains attached to a service processor, setup of DSCP is as simple as this:

XSCF> setdscp -i 10.0.0.1 -m 255.255.255.0
Commit these changes to the database? [y|n] : y

or, even simpler:

XSCF> setdscp
DSCP network  [10.0.0.1       ] >
DSCP netmask  [255.255.255.0  ] >
XSCF address  [10.0.0.2       ] >
Domain #00 address  [10.0.0.3       ] >
Domain #01 address  [10.0.0.4       ] >

These examples are from an M4000 with only 2 domains, but the flavour stays the same on the larger boxes in the series. The only thing you really have to worry about (just like your MAN networks) is that you pick a network that doesn't get used. Generally, as shown above, using a non-routable network is best practice. Although I accidentally typed a 24-bit netmask for a class C IP (and used 1 for the network instead of 0), it doesn't really make a difference.

Of course, displaying the information is just as simple (should you forget):

XSCF> showdscp

DSCP Configuration:

Network: 10.0.0.1
Netmask: 255.255.255.0

Location     Address
----------   ---------
XSCF         10.0.0.2
Domain #00   10.0.0.3
Domain #01   10.0.0.4

The key benefit, aside from the easier resource sharing that carries over from the MAN days, is the extra protection each domain is provided. Assuming an exploit is commited against one domain, whoever's gotten onto your box and screwed up that configuration will have to work harder to get to the other connected domains, since they'll have to go through the XSCF to get there. There is no absolute direct connection between domains. Although, since I know somebody out there is thinking this, it "is" still possible to attack all the domains on an M-Series machine at once; just not in an overt fashion. For instance (and I'm not encouraging this behaviour in any way whatsover) if you can create a situation whereby the administrator needs to power-cycle his/her M-Series server to restore functionality to the exploited domain, you've just brought all of the domains down in one fell swoop.

And, in closing, if you can't get to the XSCF to check out the DSCP configuration and you have the proper privilege and access to a domain hosted on an M-Series server, you can obtain that same information using the prtdscp command. Even more convenient, you can SSH (assuming you've set it up) directly from your domain to the DSCP IP using a command similar to:

host # ssh `prtdscp -s`

If you work somewhere that can afford it, enjoy the convenience :)

Cheers,

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Adding Slightly Different Types In VCS On Linux And Unix

Hey there,

Today we're going to take a look at creating new "types" for use with Veritas Cluster Server (VCS). In a broad sense of the term, almost everything you'll ever define in your main.cf (the main configuration file for VCS) is based on a specific "type," which is actually described in the only standard include file in that configuration file: types.cf - Note that both main.cf and types.cf are located in /etc/VRTSvcs/conf/config. You could move the types.cf to an alternate location fairly easily and only have to modify the "include" line in main.cf, but it's not recommended. VCS has the potential to make your life miserable in many built-in ways ;)

For instance, if you had an entry like this in your main.cf:

Apache httpd_server ( 
 httpdDir = "/usr/local/apache" 
 HostName = www
 User = nobody
 ConfigFile = "/usr/local/apache/conf/httpd.conf" 
 )

that Apache instance you described (its name "httpd_server" is arbitrary and/or up to you, but is how you would reference that instance of that type later on in the config file) would actually be based on the "Apache" type (all types in VCS are "cAse SensiTIVe ;) in types.cf, which is described thusly (as you can see, it has many attributes that, mostly, are left at their defaults. We've italicized the points that we've specifically defined above):

type Apache (
        static str ArgList[] = { ResLogLevel, State, IState, httpdDir, SharedObj
Dir, EnvFile, HostName, Port, User, SecondLevelMonitor, SecondLevelTimeout, Conf
igFile, EnableSSL, DirectiveAfter, DirectiveBefore }
        str ResLogLevel = INFO
        str httpdDir
        str SharedObjDir
        str EnvFile
        str HostName
        int Port = 80
        str User
        boolean SecondLevelMonitor = 0
        int SecondLevelTimeout = 30
        str ConfigFile
        boolean EnableSSL = 0
        str DirectiveAfter{}
        str DirectiveBefore{}
)

As you may have noted, a lot of defaults are set in the standard type definition. For instance, the Port is set to 80 by default. You could override that in your main.cf by simply including a line in your "Apache httpd_server {" definition that read:

 Port = 8080

or whatever you preferred.

Assuming that you will only be running Apache web servers on either port 80 or 8080 (we're going to skip 443, since it "silently" gets defined if you set "EnableSSL = 1" which includes that port automatically - although we may be putting that in a way that seems slightly off ;) you can either do things the easy way and just describe two differently-named Apache instances in your main.cf, like so (Be sure to check out our older posts on modifying the main.cf file properly if you're uncertain as to whether or not you're updating and/or distributing config files appropriately):

Apache regular_server ( 
 httpdDir = "/usr/local/apache" 
 HostName = www
 User = nobody
 Port = 80
 ConfigFile = "/usr/local/apache/conf/httpd.conf" 
 )
 
Apache alternate_server ( 
 httpdDir = "/usr/local/apache" 
 HostName = www
 User = nobody
 Port = 8080
 ConfigFile = "/usr/local/apache/conf/httpd.conf" 
 )

Or do things the hard way. The hard way can be dangerous (especially if you make typos and/or any other serious errors editing the types.cf file - back it up before you muck with it ;) and is generally not necessary. We just made it the topic of today's post to show you how to do it in the event that you want to customize to that degree and/or need to. Keep in mind that (if you change, or add to, types.cf) you should always keep a backup of both the original and your modified version of types.cf handy. I you ever apply a patch or service/maintenance pack from Veritas, it may very well overwrite your custom types.cf file.

Assuming you've read the preceding paragraph and are willing to take the risk, you might modify your types.cf file to include two different versions of the Apache type: One for servers running on port 80 (the default) and one for servers running on port 8080. As we mentioned, types in types.cf are "case sensitive," which makes them simpler to make similarly unique. This works out well in Unix and Linux, since most types are associated with an actual binary directory in /opt/VRTSvcs/bin (which gets the OS involved, and the OS is case sensitive).

So, assuming we wanted to add an "ApachE8080" type to types.cf, our first move would be to duplicate/modify the binary directory in /opt/VRTSvcs. In our example, this is very simplistic, since we're not creating a new type from scratch and doing everything the hack'n'slash way (If you prefer order over speedy-chaos, check out the "hatype," "haattr" and "hares" commands, although not necessarily in that order ;)

host # ls /opt/VRTSvcs/bin/Apache
Apache.pm    ApacheAgent  monitor      online
Apache.xml   clean        offline

The /opt/VRTSvcs/bin/Apache directory contains a number of programs and generic "methods" required by VCS for most resource types (As a counter-example, online-only types like NIC don't require the "start" or "stop" scripts/methods listed under Apache, since they're "always on" in theory ;). In order for us to create our ApachE8080 type, we'll need to do this one simple step first:

host # cp -pr /opt/VRTSvcs/bin/Apache /opt/VRTSvcs/bin/ApachE8080
host # ls /opt/VRTSvcs/bin/ApachE8080
Apache.pm    ApacheAgent  monitor      online
Apache.xml   clean        offline
host # diff -r /opt/VRTSvcs/bin/Apache /opt/VRTSvcs/bin/ApachE8080
host # <-- In this case, no news is good news :)

Now, all we have to do is modify our types.cf file. We're not sure if it's required, but we always duplicate the empty lines between type definitions just in case (It doesn't hurt). This is what our new type will look like (Note that the only line changed - aside from the name of the type - is the "int Port" line which we've italicized again):

type ApachE8080 (
        static str ArgList[] = { ResLogLevel, State, IState, httpdDir, SharedObj
Dir, EnvFile, HostName, Port, User, SecondLevelMonitor, SecondLevelTimeout, Conf
igFile, EnableSSL, DirectiveAfter, DirectiveBefore }
        str ResLogLevel = INFO
        str httpdDir
        str SharedObjDir
        str EnvFile
        str HostName
        int Port = 8080
        str User
        boolean SecondLevelMonitor = 0
        int SecondLevelTimeout = 30
        str ConfigFile
        boolean EnableSSL = 0
        str DirectiveAfter{}
        str DirectiveBefore{}
)

And, you're all set. Now you can change your main.cf file to look like the following and everything should work just as if you had done it the easy way (Be sure to check out our older posts on modifying the main.cf file properly if you're uncertain as to whether or not you're updating and/or distributing config files appropriately):

Apache regular_server ( 
 httpdDir = "/usr/local/apache" 
 HostName = www
 User = nobody
 ConfigFile = "/usr/local/apache/conf/httpd.conf" 
 )
 
ApachE8080 alternate_server ( 
 httpdDir = "/usr/local/apache2" 
 HostName = www
 User = nobody
 ConfigFile = "/usr/local/apache2/conf/httpd.conf" 
 )

and, now, you no longer have to go through all the trouble of adding that "Port = 80" (or "Port = 8080") line to your main.cf type specification...

Six of one, half dozen of another? Whatever works for you, depending on your situation, is our basic rule. Or, in other words, 13 of one, baker's dozen of another ;)

Cheers,

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

How Many Different Ways Can You Stop VCS On Linux Or Unix?

Hey there,

This little rundown may seem trivial in comparison to some of our older posts on Veritas Cluster Server (VCS), as they all attacked a very specific aspect of VCS functionality and, mostly, were aimed at either explaining more advanced concepts or showing you how to get away with stuff that you're not supposed to do ;)

However, even in its simplicity, today's topic is just as valid as our previous posts, and (quite probably) more useful to the admin or user who wants to know enough to administrate VCS well, but doesn't care about all the nitty-gritty. Because, let's face it, the nitty-gritty doesn't really matter all the time. I find that I almost never think of the nitty gritty every morning when I start my car (I reserve that special time for praying that it won't explode ;)

So, before I veer too far off the beaten path, we'll get down to today's subject: How to shutdown a Veritas Cluster, using VCS, the many different ways to do it and what they all mean.

Of course, the base command for stopping VCS (Sometimes also referred to as HAD for the High Availability Daemon, which explains why all the commands for VCS start with "ha" :) is "hastop." hastop has a number of ways in which you can call it; all of which can have a significant impact on your day if used arbitrarily ;)

1. hastop -local. This usage stops VCS on your local machine (or the machine on which you're running it - same thing). It also causes any service groups on your local system to be taken offline.

2. hastop -local -evacuate. This invocation is very similar to the previous, except that it migrates (if possible) all of your service groups to the default failover system before it stops the VCS services on your local machine. This option isn't available if you use the -all flag (since, obviously, there won't be any systems left up in the cluster to fail over to ;)

3. hastop -local -noautodisable. The -noautodisable option (also not available for use in conjunction with the -all argument) makes it so that none of your service groups are set to disabled.

4. As an aside of sorts, it should be noted that you can use -evacuate and -noautodisable together, although you can't use either with other arguments, like -force or -all.

3. hastop (-all|-local) -force. These options, when presented to hastop, cause it to stop VCS services on all systems in your cluster (-all) or just your local system (-local), but it does not do anything to your service groups or offline their resources. This is convenient when you have planned maintenance on VCS-managed resources and don't want to create a fault condition (and all its accompanying alarms, bells and whistles ;) When VCS is stopped in this manner, hastart merely resumes running VCS services as normal (It doesn't bring VCS up with an overt assumption that something wrong has happened). One downside of using the -force option is that it doesn't check whether or not you may be editing the main.cf file at the time you decide to call it quits. If you have it open, in read/write mode, and you stop VCS this way, you may be in for an unpleasant restart.

4. hastop -all. This stops VCS on all systems in your cluster, as the previous command does, but it doesn't ignore your system groups and will take them offline.

5. hastop -sys SYSTEMNAME. This way of executing hastop causes VCS to react in much the same way as the -local flag does (including having the -evacuate, -noautodisable and -force arguments - the same rules, listed above, apply to using combinations of -evacuate and -noautodisable (ok) and -force (not ok to mix with either of the others). One thing to note is that you can't use the -local and/or -all flags with the -sys flag (again, for obvious reasons. It's nice to know that some things in this world make sense ;), since the -sys flag requires you to specify a specific system (or host) in your cluster on which to execute the command.

6. hastop -help. This is a great option to use when you have no idea who you are, where you are, how you got there and/or what to do ;)

One final (and, of course, interesting ;) note about modifying hastop's behaviour is that you can modify the EngineShutdown "cluster attribute" (outside the scope of this little post) to set different default behaviours for hastop. This can be a big help if you always do things "the not normal way" ;) The EngineShutdown cluster attribute(s) (ESCA from now on, since I'm getting tired of mistyping the name of this thing ;) that you can set (with the haclus command) are:

a. Enable: This ESCA indicates that all hastop commands should be processed. This is the default.

b. Disable: This ESCA indicates that all hastop commands will not be processed. The only exception to this setting is if you use the -force argument to hastop.

c. DisableClusStop: This ESCA makes it so that the "hastop -all" command is rejected or not processed. All other hastop commands are processed as they normally would be.

d. PromptClusStop: This ESCA will cause the operator/administrator to be prompted (Do you really want to do this? Are you sure? C'mon, seriously? ;) before it executes the "hastop -all" command. All other hastop commands are processed normally and don't require answering any prompts.

e. PromptLocal: This ESCA prompts the admin whenever "hastop -local" is invoked (Are you positive you want to do this? Have you given this any thought at all? Remember; it's you're arse. Do you still want to do this? ;). All other hastop commands are processed normally.

f. PromptAlways: This ESCA (which is also referred to in non-academic circles as the Big PITA ;) causes the admin/qualified-user to be prompted when executing any and all hastop commands. When you get sick of answering questions, and you have the permissions, you might want to turn this off since it's not default and someone you work with may have turned it on just to eventually drive you nuts ;)

And, there you have it; hastop in a big complicated nutshell (Or outside of a nutshell ...whatever the opposite of in-a-nutshell is ;) One thing I should mention if you work with cluster attributes or any parts of the main.cf, types.cf or other VCS configuration files; it's always a good idea to make sure you know what the defaults for your version are. VCS's configuration file style will just "not include" most values that are set to their defaults, so you would never see the Enable variable and value in the ESCA, since it's the default.

A somewhat cool trick (if you're not sure if a particular variable/value pair is set to its default) is to add it to your configuration file(s). After you rebuild your config files the proper way (or manually edit it/them the somewhat-frowned-upon way), if the variable/value pair you entered is not set to default, it will have been removed!

Cheers,

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

How To Set Up A Headless X Server On Redhat Linux

Hey there,

Today we're going to throw out another Linux tip somewhat related to networking. Or, more correctly, we're going to throw another quick tip out there ;) Today we're going to look at setting up a headless X Server. It's not quite as difficult, or as scary, as it sounds. Running a headless X Server is kind of like Night Of The Living Dead (the original; although the remake was kind of fun) except without all the zombies and with exported displays instead of excoriated remains (best I could do in the almost-rhyming department on that one ;)

If you stick to the following steps (Tested on RedHat Enterprise Linux Update 4) and adjust slightly if your distro is either completely different, your system architecture varies or your RedHat release is just a version or so off, you should be up and running in no time.

1. Make sure that you install all of the following rpm's on the machine on which you want to run the headless X Server (ignore errors that state you already have any of the packages installed. If you run into dependency conflicts, run rpm with the --nodeps option. If you need to update any of the files, run rpm with the --update option. You may need to install these one by one, but you might get lucky and just be able to run rpm -ivh in a loop and have them all install the first time. You can also get around dependency issues by installing all of the rpm's on one line - i.e. rpm -ivh rpm1.rpm rpm2.rpm ... rpmN.rpm):

XFree-libs-data-4.3.0-78.EL.i386.rpm
XFree-xfs-4.3.0-78.EL.i386.rpm
XFree-libs-4.3.0-78.EL.i386.rpm
XFree86-xauth-4.3.0-35.EL.i386.rpm
XFree86-base-fonts-4.3.0-35.EL.i386.rpm
XFree86-4.3.0-78.EL.i386.rpm
cpp-3.2.3-47.i386.rpm
xinitrc-3.32-1.noarch.rpm
switchdesk-3.98.17.i386.rpm
desktop-file-utils-0.3-10.i386.rpm
XFree86-Xvfb-4.3.0-78.EL.i386.rpm
XFree86-ISO8859-14-100dpi-fonts-4.3.0-97.i386.rpm

2. Start up X's virtual frame buffer server like so:

Xvfb :0 –pn –nolisten tcp –audit 4 –screen 0 800x600x16 & <-- The 800x600 is the screen resolution and the x16 (16 bit)is the color depth. You can change these settings to suit your taste or satisfy any limitations your graphics driver might impose).

NOTE: If you have problems with the X Server going offline when you logoff, try running it with nohup and/or check out our previous post on what to do when nohup hangs up anyway!

And, how about that? I think this is, blessedly, the shortest post I've ever written :) Just pretend there are a couple more paragraphs of "ipsum lorem dolor sit amet" if you find yourself suffering from information underload ;)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Simple Perl Script To Ease Console Server Use On Linux And Unix

Howdy,

Well, it's been a good long time (almost a week) since I've posted anything but numbered lists of, hopefully, useful information. It's about time for a little scripting. So, just to maintain my sanity (if not your own - assuming you follow this blog a bit, for which I thank you), today I'm putting up a little Perl script I wrote to maintain connections to a simple Cyclades console server. It can be re-worked fairly easily to connect to multiple console servers of any kind.

Technically, this script could be used for any server which accepts connections over telnet (Which this script is set up to use, even though I'm pretty sure I fought the good fight arguing against it). For more information on why it's a bad idea to use telnet for anything you don't have to, check out our old posts on using tcpdump on Linux or snoop on Solaris to easily grab user passwords. It should also be noted that I wrote this for a non-primary employer (contract). The IRS can rest assured that they'll get their cut of my money at the end of the quarter, but I can't, of course, reveal the identity of the client at this point in time (and, for ethical reasons, I won't later, either, since, given the setup, it would be like setting up a turkey-shoot ;)

The script can be run simply, like most of the scripts we put out here, from the command line. It only requires that you supply a hostname and will spit out help information if you just run it with no arguments, like so:

host # ./dconsole.pl
Usage: ./dconsole.pl [-h|-l|hostname]

This Perl script is fairly limited for a reason, but it's very easy to expand upon and only requires minor adjustment to set it up to use SSH. Since, of course, SSH would require some sort of key-or-host-based authentication method set up before this script would work, be sure to do that before you just run this out-of-the-box. Believe it or not, we've got a post on distributing and setting up SSH keys network-wide on this site, as well ;)

Sometimes I feel like I'm beginning to write in a mobius strip ;)

Anyway, enjoy, and best wishes,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# dconsole.pl - Connect to your console server or modify to connect to a whole bunch!
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if ( $#ARGV == -1 || $#ARGV > 0 ) {
 print "Usage: $0 [-h|-l|hostname]\n";
 exit(1);
}

# This is a simple array to list out all hosts connected to the console server

@hosts = qw(host1 host2 host3);

if ( $ARGV[0] =~ /^-h$/ ) {
 print "\n";
 print "dconsole.pl - Mike Golvach - eggi\@comcast.net\n";
 print "\n";
 print "-h : This piddly screen.\n";
 print "-l : List hosts available.\n";
 print "hostname : host to connect to.\n";
 print "\n";
 exit;
} elsif ( $ARGV[0] =~ /^-l$/ ) {
 print "\n";
 print "@hosts";
 print "\n";
 exit;
}

$goodtogo = 0;
for $hostname (@hosts) {
 if ( $ARGV[0] eq $hostname ) {
  $goodtogo = 1;
 }
}

if ( ! $goodtogo ) {
 print "\n";
 print "$ARGV[0]?  Never heard of it.\n";
 print "\n";
 exit(1);
}

# This is a simple hash to associate connected hosts to their ports on the console server

%whatitis = qw(host1 2 host2 3 host3 4);

$conport = $whatitis{$ARGV[0]};
print "\n"; 
print "Opening $ARGV[0]...\n";
print "\n";
system("/usr/bin/telnet THECONSOLESERVER $conport");
exit;

, Mike

Troubleshooting Veritas Cluster Server LLT Issues On Linux and Unix

Hey There,

Today's post is going to steer away from the Linux and/or Unix Operating Systems just slightly, and look at a problem a lot of folks run into, but have problems diagnosing, when they first set up a Veritas cluster.

Our only assumptions for this post are that Veritas Cluster Server is installed correctly on a two-node farm, everything is set up to failover and switch correctly in the software and no useful information can be obtained via the standard Veritas status commands (or, in other words, the software thinks everything's fine, yet it's reporting that it's not working correctly ;)

Generally, with issues like this one (the software being unable to diagnose its own condition), the best place to start is at the lowest level. So, we'll add the fact that the physical network cabling and connections have been checked to our list of assumptions.

Our next step would be to take a look at the next layer up on the protocol stack, which would be the LLT (low latency transport protocol) layer (which, coincidentally, shares the same level as the MAC, so you may see it referred to, elsewhere, as MAC/LLT, or just MAC, when LLT is actually meant!) This is the base layer at which Veritas controls how it sends its heartbeat signals.

The layer-2 LLT protocol is most commonly associated with the DLPI (all these initials... man. These stand for the Data Link Provider Interface). Which brings us around to the point of this post ;)

Veritas Cluster Server comes with a utility called "dlpiping" that will specifically test device-to-device (basically NIC-to-NIC or MAC-to-MAC) communication at the LLT layer. Note that if you can't find the dlpiping command, it comes standard as a component in the VRTSllt package and is generally placed in /opt/VRTSllt/ by default. If you want to use it without having to type the entire command, you can just add that directory to your PATH environment variable by typing:

host # PATH=$PATH:/opt/VRTSllt;export PATH

In order to use dlpiping to troubleshoot this issue, you'll need to set up a dlpiping server on at least one node in the cluster. Since we only have two nodes in our imaginary cluster, having it on only one node should be perfect.

To set up the dlpiping server on either node, type the following at the command prompt (unless otherwise noted, all of these Veritas-specific commands are in /opt/VRTSllt and all system information returned, by way of example here, is intentionally bogus):

host # getmac /dev/ce:0 <--- This will give use the MAC address of the NIC we want to set the server up on (ce0, in this instance). For this command, even if your device is actually named ce0, eth0, etc, you need to specify it as "device:instance"
/dev/ce:0 00:00:00:FF:FF:FF

Next, you just have to start it up and configure it slightly, like so (Easy peasy; you're done :)

host # dlpiping -s /dev/ce:0

This command runs in the foreground by default. You can background it if you like, but once you start it running on whichever node you start it on, you're better off leaving that system alone so that anything else you do on it can't possibly affect the outcome of your tests. Since our pretend machine's cluster setup is completely down right now anyway, we'll just let it run in the foreground. You can stop the server, at any time, by simply typing a ctl-C:

^C
host #

Now, on every other server in the cluster, you'll need to run the dlpiping client. We only have one other server in our cluster, but you would, theoretically, repeat this process as many times as necessary; once for each client. Note, also, that for the dlpiping server and client setups, you should repeat the setup-and-test process for at least one NIC on every node in the cluster that forms a distinct heartbeat-chain. You can determine which NIC's these are by looking in the /etc/llttab file.

host # dlpiping -c /dev/ce:0 00:00:00:FF:FF:FF <--- This is the exact output from the getmac command we issued on the dlpiping server host.

If everything is okay with that connection, you'll see a response akin to a Solaris ping reply:

0:00:00:FF:FF:FF is alive

If something is wrong, the output is equally simple to decipher:

no response from 00:00:00:FF:FF:FF

Assuming everything is okay, and you still have problems, you should check out the support site for Veritas Cluster Server and see what they recommend you try next (most likely testing the IP layer functionality - ping! ;)

If things don't work out, and you get the error, that's great (assuming you're a glass-half-full kind of person ;) Getting an error at this layer of the stack greatly reduces the possible-root-cause pool and leaves you with only a few options that are worth looking into. And, since we've already verified physical cabling connectivity (no loose or poorly fitted ethernet cabling in any NIC) and traced the cable (so we know NICA-1 is going to NICB-1, as it should), you can be almost certain that the issue is with the quality or type of your ethernet cabling.

For instance, your cable may be physically damaged or improperly pinned-out (assuming you make your own cables and accidentally made a bad one - mass manufacturers make mistakes, too, though). Also, you may be using a standard ethernet cable, where a crossover (or, in some instances, rollover) cable is required. Of course, whenever you run into a seeming dead-end like this, double check your Veritas Cluster main.cf file to make sure that it's not in any way related to a slight error that you may have missed earlier on in the process.

In any event, you are now very close to your solution. You can opt to leave your dlpiping server running for as long as you want. To my knowledge it doesn't cause any latency issues that are noticeable (at least in clusters with a small number of nodes). Once you've done your testing, however, it's also completely useless unless you enjoy running that command a lot ;)

Cheers,

, Mike

Masking Your HTTP Make And Version In Apache For Linux Or Unix

Hey there,

In an older post, we went into some small detail about ignoring HTTP headers when using bash to access the network. Today we're going to look at that from another angle and consider how you can, at least somewhat, protect yourself from a clearly defined outside attack by mocking up your HTTP headers using Apache.

For this simple test, we used the 3 latest versions of the three separate build trees of the Apache HTTPD Server: versions 1.3.41, 2.0.63 and 2.2.8. All three versions were compiled on RedHat Linux, SUSE Linux and Solaris Unix.

While all of these versions provide for some measure of protection within the httpd.conf file (such as turning the server signature on or off, allowing you to fake your hostname, letting you fool with server tokens, etc), if you want to truly utilize security-through-obscurity with Apache, your starting point should be at the source code level. Of course, if you use a pre-packaged binary, you might no be able to do any of this, short of recompiling a vendor package, which might cause more harm than good...

As a quick note to source-code builders who run Apache with mod_ssl, please keep in mind that mod_ssl generally checks the Apache include files to determine whether or not the version of Apache it's being compiled against is the correct one. If you "do" use this sort of setup, be sure to compile mod_ssl first, make these changes and then compile Apache. Or you could compile Apache this way, change the include back to the way it was, compile mod_ssl and go from there. Whatever suits your particular style is OK :)

The main (and by main, I mean "only" ;) trick here is to change what the Apache HTTPD server thinks it is (The version that it spits out of the httpd binary). All of the configuration file mangling you do won't stop Apache from reporting its true version given a particular set of circumstances.

Luckily, this modification is very easily achieved, and can be set in one file (before running configure and make). Depending on how crazy you want to get, you can mislead attackers so that they attempt to use outdated exploits against your site or (if you get too creative) let them know that they can't be sure what version of Apache you're running... if you're running Apache at all. The biggest trick in all this, after a while, might just be remembering what version you really "are" running ;)

For Apache 1.3.x, you can change these fields in the source_dir (wherever you untar the source files) under the src/include directory in the httpd.h file:

#define SERVER_BASEVENDOR "Apache Group"
#define SERVER_BASEPRODUCT "Apache"
#define SERVER_BASEREVISION "1.3.41"

If you want, you could make this:

#define SERVER_BASEVENDOR "HTTPD Consortium Group"
#define SERVER_BASEPRODUCT "SpyGlass"
#define SERVER_BASEREVISION "3.2"

And, instead of a user seeing this when they hit the right error page (or nail your server directly):

Server: Apache/2.2.4 (Unix)

They'd see this:

Server: SpyGlass/3.2 (Unix)

And that's just from a telnet to port 80. The other information could be gotten other ways. In any event, whoever gets it will be misinformed :)

For both the 2.0.x and 2.2.x strains of Apache's HTTPD server, the file you'll need to modify is in the source_dir, in the include directory, in the file ap_release.h.

For 2.0.x - Modify these lines to suit your taste:

#define AP_SERVER_BASEVENDOR "Apache Software Foundation"
#define AP_SERVER_BASEPRODUCT "Apache"
#define AP_SERVER_MAJORVERSION_NUMBER 2
#define AP_SERVER_MINORVERSION_NUMBER 0
#define AP_SERVER_PATCHLEVEL_NUMBER 63
#define AP_SERVER_ADD_STRING ""

For 2.2.x, the file name and location are the same, and there is only one additional line added that you can/should manipulate:

#define AP_SERVER_BASEVENDOR "Apache Software Foundation"
#define AP_SERVER_BASEPROJECT "Apache HTTP Server"
#define AP_SERVER_BASEPRODUCT "Apache"
#define AP_SERVER_MAJORVERSION_NUMBER 2
#define AP_SERVER_MINORVERSION_NUMBER 2
#define AP_SERVER_PATCHLEVEL_NUMBER 8
#define AP_SERVER_DEVBUILD_BOOLEAN 0

For the AP_SERVER_DEVUILD_BOOLEAN, you can change the value to 1 and the string "-dev" will be added. This is essentially the same as the AP_SERVER_ADD_STRING variable in 2.0.x, but slightly less flexible.

Here's to having fun making up server names or (even better) resurrecting old ones that haven't been around since I was a teenager :)

Cheers,

, Mike

linux unix internet technology

Perl Script To List Out Linux or Unix Crontabs In HTML

Hello again,

Today's script is a simple, but useful, one. A while back, we ran a series of posts on graphing out disk, paging, memory and cpu statistics for Unix, with a big roundup for paging, memory and cpu on Linux.

This time, we're going to go in a bit of a different direction and put together an example script that creates an HTML page, also (hopefully) providing some useful information for you, the sysem admin or user.

As with the other scripts, there are a few words of caution that go with putting any sort of script like this "out there." While this doesn't have the destructive potential of the Linux graphing scripts (or the Unix ones), as they were interfacing directly with system commands, you should still be careful whenever you publish information in this way. Since our script goes to the filesystem to read its information from a system output file, it affords access that may not necessarily be granted to a user logged on to that system.

In any event, all "gloom and doom" aside, we're starting out simple with today's "system information" script by just reporting on crontab contents. The script below does need to be run from the command line (unless you expressly allow execution of .pl files in your web server's configuration and modify it somewhat), which creates an interesting cyclical loop if you run it from cron ;) It's also kind enough to remove all comment lines from the crontabs so they're, hopefully, more readable!

You can execute it simply by running it from the command line with no arguments, like so:

host # ./cron.pl

You'll probably need to modify a few variables (like the "chdir" line, to indicate the correct path to your crontabs directory, the output file's destination and the section where I purposefully ignore the basic "system accounts" (including root) and then print to the HTML page that they're "currently unused"), but it should run right out-of-the-box, otherwise.

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# cron.pl --- Easily scan through a server's cron jobs
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

chomp($cur_dir=`pwd`);
chdir("/var/spool/cron/crontabs");
@crons = `ls`;
$filename = "${cur_dir}/cron.html";
$hostname = `hostname`;

foreach $job (@crons) {
 chomp $job;
 if ( $job =~ /(^adm$|^lp$|^sys$|^uucp$|^root$)/ ) {
  next;
 } elsif ( $job =~ /\.pl|html|passwd/ ) {
  next;
 } else {
  push(@parcel, $job);
 }
}

open(OUTPUT, ">$filename");
print OUTPUT "<html><head><title>$hostname Cron Services</title></head>\n";
print OUTPUT "<body BGCOLOR=\"#000000\" TEXT=\"#00FF00\" LINK=\"#FFFF00\" VLINK=\"#FF0000\" ALINK=\"#BECD8C\">\n";
print OUTPUT "<FONT COLOR=\"FFFF00\">$hostname -- CRON SERVICES:</FONT>\n";
print OUTPUT "<br><hr><br>\n";
print OUTPUT "<FONT COLOR=\"FFFF00\">root:</FONT><br>Currently unused. - System Supplied.<br>\n";
print OUTPUT "<br><hr><br>\n";
print OUTPUT "<FONT COLOR=\"FFFF00\">adm:</FONT><br>Currently unused. - System Supplied.<br>\n";
print OUTPUT "<br><hr><br>\n";
print OUTPUT "<FONT COLOR=\"FFFF00\">lp:</FONT><br>Currently unused. System Supplied.<br>\n";
print OUTPUT "<br><hr><br>\n";
print OUTPUT "<FONT COLOR=\"FFFF00\">sys:</FONT><br>Currently unused. - System Supplied.<br>\n";
print OUTPUT "<br><hr><br>\n";
print OUTPUT "<FONT COLOR=\"FFFF00\">uucp:</FONT><br>Currently unused - System Supplied.<br>\n";
print OUTPUT "<br><hr><br>\n";

foreach $part (@parcel) {
 open(PACKAGE, "<$part");
 @package = <PACKAGE>;
 close(PACKAGE);
 print OUTPUT "<FONT COLOR=\"FFFF00\">$part</FONT>\n";
 print OUTPUT "<br>\n";

 foreach $pretty (@package) {
  if ( $pretty =~ /^#/ ) {
   next;
  } else {
   @picture = split(/ /, $pretty);
   $frame = join(" ", @picture);
   chomp $frame;
   print OUTPUT "$frame<br>\n";
  }
 }
 print OUTPUT "<br><hr><br>\n";
} 
close(OUTPUT);

, Mike

linux unix internet technology

Bypassing Boot Server Installation When Jumpstarting Across Subnets

Greetings,

Today I thought we'd go with a little "trickery," although this isn't really a trick; it's just not the way you're supposed to use Solaris' Jumpstart when Jumpstarting across subnets. It has it's benefits, though, aside from the obvious (I'll get to that at the end :) Note that some of this process varies in Solaris 10 (insofar as the starting up of services goes, but we'll hit that in a later post, since they've completely changed the procedure). This will work, almost word for word in Solaris versions up to and including 9.

Now, to get started:

In order to install and configure a system across subnets, it is recommended by Sun that the Jumpstart client be configured to run through a Boot Server on that client's subnet (and on a separate server for every subnet that you need to hop through). This involves executing the add_install_client command on the Boot Server on the same subnet as the client, and installing all the software required to run a Boot Server.

It is possible, however, to use any server on the client's subnet as a conduit through which to pass the boot, installation and configuration information between the client and server without expending the time, or wasting the resources, that installing a Boot Server on the intermediate client requires.

In order to make this seamless connection between subnets possible, and allow your Jumpstart client to install and configure itself through the server, you will need to modify certain files and ensure that several daemon processes are active on the gateway computer.

1. First, be sure to update the gateway computer's /etc/hosts and /etc/ethers files to include entries for the install client. The gateway computer is were we'd normally have to consume the disk space to actually set up a legitimate Boot Server.

2. Ensure that the proper daemons are active on the gateway computer. You may need to start up rarpd, bootparamd and tftpd (which is, by default, commented out of the /etc/inetd.conf file, so you'll need to "kill -1" or "kill -HUP" the inetd process after uncommenting that entry).

3. Create an entry for the install client in the /etc/bootparams file. You can generally copy the entry that exists in your Jumpstart server's /etc/bootparams file, but you will need to make a few changes:

a. Translate the hostname of the server (if entered in this fashion) to it's I.P. Address.

b. Adding an ns entry won't hurt. Assuming a Jumpstart server with I.P. Address 192.168.0.12, no supported nameservice active and a netmask of 255.255.255.0, you can add this to the end of the /etc/bootparams entry for your client (on the gateway computer, of course):

Ex:

ns=192.168.0.12:none(255.255.255.0) <-- Add a colon to the beginning of this addition if the /etc/bootparams entry doesn't end with one!

4. Create a directory off of the root partition named tftpboot.

Ex:

mkdir /tftpboot

5. Populate your /tftpboot directory with the contents of the /tftpboot directory on your Jumpstart server host and ensure all file and directory ownership and permissions are the same. If you know exactly what to look for, you really only need to copy the files that are relevant to the server you're trying to Jumpstart, but you can't go wrong by copying them all.

Now, the intermediate host/server should be prepared to accept your Jumpstart client's rarp requests and transparently forward them to your Jumpstart server on the alternate subnet!

Note that you can do this on as many servers as you like (if, for instance, you have to take multiple hops across subnets to get to your Jumpstart server). Just duplicate on each intermediate host. The only thing you'll have to change (and sometimes you won't even have to do this) is the Jumpstart host listed in /etc/bootparams on the servers that come before you hit the server that can get to the Jumpstart server. You can create a simple daisy chain by replacing the Jumpstart server name with the IP of the next host along the way in the /etc/bootparams file. Hopefully, you won't ever have to hop more than once to get to an internal network that has a Jumpstart server on it.

And finally, as promised, the extra benefit: If you ever get stuck at home and have to boot a Solaris box that won't come up, the box is on a subnet without a Jumpstart (or Boot) server and there's no one at the office to put in a CD for you to boot off of, you can use this exact same procedure to hop subnets and get to single user mode with:

boot net -s

Cheers,

, Mike

linux unix internet technology

Simple Sun Cluster Monitoring Script

Hey There,

Today, I've included a script I wrote to monitor a small SunCluster 3.1 environment (Running Oracle Parallel Server - OPS) we have set up at our shop. It basically runs through the output of various "scstat" command variations and reports on all combinations of errors it encounters. I built some functionality in it to be as specific as possible about the error states and to make sure that it returns an "All Clear" once the error condition no longer exists.

I also wrote this to run in cron. I run it every 5 minutes. Any time period is acceptable, depending upon what you need, and, of course, you could always put this script in a wrapper so that it runs constantly (although that generally necessitates writing another script to make sure that this script is running, restarting it as necessary and vice versa - For Veritas Cluster Server fans out there, this is a cheap and quick way to imitate the relationship between had and hashadow).

This was also written for a small environment (2 machines with 2 "live" network connections each). I didn't include monitoring of the heartbeats, since the script is meant to be run locally (with "exactly" the same values in the customizable section) on all nodes in a cluster, and scstat's indication of failure on any of these tests is, in and of itself, a guarantee that loss of a heartbeat connection is the very least of your problems ;)

Enjoy!





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

###################################
# suncluster_mon - check cluster health
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
# simple parser for:
# 1. scstat -n
# 2. scstat -i
# 3. scstat -g
# 4. scstat -D
#
###################################
 
###################################
# CUSTOMIZE SETTINGS HERE
# $debug can be set with the -d
#  switch when invoking the script
###################################
$primary_node="";
$secondary_node="";
$primary_eth="";
$secondary_eth="";
$date=`/usr/bin/date "+%A, %B %d - %H:%M:%S"`;
$hostname=`/usr/bin/hostname`;
$debug=0;
###################################
# PLEASE DO NOT EDIT BELOW
# EXCEPT MAIL SETTINGS AT END
###################################

chomp($date);

if ( $#ARGV >= 0 ) {
 foreach $arg (@ARGV) {
  if ( $arg =~ /-d/ ) {
   $debug=1;
   print "Debug Output Set\n";
  } else {
   push(@nogo, $arg)
  }
 }
 if ( @nogo > 0 ) {
  print "Unrecognized options: @nogo\n";
  print "Ignoring and continuing\n";
 }
}

@scstatn=`/usr/cluster/bin/scstat -n|/usr/bin/egrep 'node:'`;
for $nodestat (@scstatn) {
 if ( $nodestat =~ /Cluster node:/ ) {
  $nodestat =~ s/Cluster node://;
  @nodestat = split(" ", $nodestat);
  if ( $nodestat[1] eq "Offline" && $nodestat[0] eq $secondary_node ) {
   if ( $nodefailure ) {
    $alert="\nCluster Alert:\n$primary_node and $secondary_node are both in Offline state.  Cluster Failed\n";
   } else {
    $alert="\nCluster Alert:\n$primary_node is in Offline state.  $secondary_node is Online - Cluster Crippled\n";
   }
   $alert="\nCluster Alert:\n$nodestat[0] has switched to $nodestat[1] Status - Cluster Crippled - Checking For Dual Failure\n";
   $nodefailure=1;
   push(@alert, $alert);
  } elsif ( $nodestat[1] eq "Offline" && $nodestat[0] eq $primary_node ) {
   if ( $nodefailure ) {
    $alert="\nCluster Alert:\n$primary_node and $secondary_node are both in Offline state.  Cluster Failed\n";
   } else {
    $alert="\nCluster Alert:\n$primary_node is in Offline state.  $secondary_node is Online - Cluster Crippled\n";
   }
   $alert="\nCluster Alert:\n$nodestat[0] has switched to $nodestat[1] Status - Cluster Crippled - Checking For Dual Failure\n";
   $nodefailure=1;
   push(@alert, $alert);
  } else {
   if ( $debug ) {
    $alert="Cluster Info: $nodestat[0] and $nodestat[1] OK\n";
    push(@debug, $alert);
   }
   $beautiful++;
  }
 } else {
  print "WTF ---- SCSTAT -D\n";
 }
}
if ( $beautiful == 2 ) {
  if ( $debug )   {
   $alert="Cluster Info: Both nodes OK\n";
   push(@debug, $alert);
  }
}

@scstati=`/usr/cluster/bin/scstat -i|/usr/bin/egrep 'Group:'`;
for $ipmpstat (@scstati) {
 if ( $ipmpstat =~ /IPMP Group:/ ) {
  $ipmpstat =~ s/IPMP Group://;
  @ipmpstat = split(" ", $ipmpstat);
  if ( $ipmpstat[0] == $primary_node ) {
   if ( $ipmpstat[3] eq $primary_eth && $ipmpstat[4] ne "Online" ) {
    $alert="\nIPMP Alert:\n$ipmpstat[3] is in $ipmpstat[4] mode on $ipmpstat[0] - Checking for Online of $ipmpstat[0] $secondary_eth\n";
    $ethprimaryfailure=1;
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Online" && $ethprimaryfailure == 1 ) {
    $alert="\nIPMP Alert:\n$ipmpstat[0] $secondary_eth is not in an Online state on $ipmpstat[0] - IPMP Group Down on both Interfaces!\n";
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Standby" && ! $ethprimaryfailure ) {
    $alert="\nIPMP Alert:\n$ipmpstat[3] is in $ipmpstat[4] mode on $ipmpstat[0] - Failover interface is not UP - IPMP Group Crippled on $ipmpstat[0]!\n";
    push(@alert, $alert);
   } else {
    if ( $debug )   {
     $alert="IPMP Info: IPMP Group $ipmpstat[0] $ipmpstat[3] OK\n";
     push(@debug, $alert);
    }
   }
  } elsif ( $ipmpstat[0] == $secondary_node ) {
   if ( $ipmpstat[3] eq $primary_eth && $ipmpstat[4] ne "Online" ) {
    $alert="\nIPMP Alert:\n$primary_eth is in $ipmpstat[4] mode on $ipmpstat[0] - Checking for Online of $ipmpstat[0] $secondary_eth\n";
    $ethprimaryfailure=1;
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Online" && $ethprimaryfailure == 1 ) {
    $alert="\nIPMP Alert:\n$ipmpstat[0] ipmpstat[3] not in an Online state on ipmpstat[0] - IPMP Group Down on both Interfaces!\n";
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Standby" && ! $ethprimaryfailure ) {
    $alert="\nIPMP Alert:\n$ipmpstat[3] is in $ipmpstat[4] mode on $ipmpstat[0] - Failover interface is not UP - IPMP Group Crippled on $ipmpstat[0]!\n";
    push(@alert, $alert);
   } else {
    if ( $debug )   {
     $alert="IPMP Info: IPMP Group $ipmpstat[0] $ipmpstat[3] OK\n";
     push(@debug, $alert);
    }
   }
  }
 } else {
  print "WTF ---- SCSTAT -I\n";
 }
}

@scstatg=`/usr/cluster/bin/scstat -g|/usr/bin/egrep 'Resource:|Resources:|Group:'`;
for $rgstat (@scstatg) {
 if ( $rgstat =~ /Resources:/ ) {
  $rgstat =~ s/Resources://;
  @rgstat = split(" ", $rgstat);
  if ( @rgstat != 5 ) {
   shift @rgstat;
   $alert="\nOracle Resource Alert:\nOracle resource group $rgstat[0] is not running all resources on the cluster\n--Only running -- @rgstat\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Oracle Resource Info: Oracle resource group $rgstat[0] OK -- @rgstat OK\n";
    push(@debug, $alert);
   }
  }
 } elsif ( $rgstat =~ /Group:/ ) {
  $rgstat =~ s/Group://;
  @rgstat = split(" ", $rgstat);
  if ( $rgstat[2] ne "Online" && $rgstat[1] eq $primary_node ) {
   $alert="\nOracle Resource Alert:\nResource Group $rgstat[0] is in $rgstat[2] state on primary node $rgstat[1] - Checking for failover\n";
   $rgprimaryfailure=1;
   push(@alert, $alert);
  } elsif ( $rgstat[2] ne "Online" && $rgstat[1] eq $secondary_node && $rgprimaryfailure == 1 ) {
   $alert="\nOracle Resource Alert:\nResource Group $rgstat[0] is in $rgstat[2] state on cluster - Resource Group Down on both Nodes!\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Oracle Resource Info: $rgstat[1] Resource Group $rgstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } elsif ( $rgstat =~ /Resource:/ ) {
  $rgstat =~ s/Resource://;
  @rgstat = split(" ", $rgstat);
  if ( $rgstat[2] ne "Online" && $rgstat[1] eq $primary_node ) {
   $alert="\nOracle Resource Alert:\nResource$rgstat[0] is in $rgstat[2] state on primary node $rgstat[1] - Checking for failover\n";
   $rgprimaryfailure=1;
   push(@alert, $alert);
  } elsif ( $rgstat[2] ne "Online" && $rgstat[1] eq $secondary_node && $rgprimaryfailure == 1 ) {
   $alert="\nOracle Resource Alert:\nResource $rgstat[0] is in $rgstat[2] state on cluster - Resource Group Down on both Nodes!\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Oracle Resource Info: $rgstat[1] Resource $rgstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } else {
  print "WTF ---- SCSTAT -G\n";
 }
}

@scstatD=`/usr/cluster/bin/scstat -D|/usr/bin/egrep 'servers|status'`;
for $diskstat (@scstatD) {
 if ( $diskstat =~ /servers/ ) {
  $diskstat =~ s/Device group servers://;
  @diskstat = split(" ", $diskstat);
  if ( $diskstat[1] ne $primary_node ) {
   $alert="\nDisk Resource Alert:\n$diskstat[0] has switched primary node to $diskstat[1] from $diskstat[2]\n";
   push(@alert, $alert);
  } elsif ( $diskstat[1] ne $primary_node && $diskstat[1] ne $secondary_node ) {
   $alert="\nDisk Resource Alert:\n$diskstat[0] has failed on all nodes!\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Disk Resource Info: $diskstat[1] primary - $diskstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } elsif ( $diskstat =~ /status/ ) {
  $diskstat =~ s/Device group status://;
  @diskstat = split(" ", $diskstat);
  if ( $diskstat[1] ne "Online" ) {
   $alert="\nDisk Resource Group Alert:\n$diskstat[0] has switched to $diskstat[1] state on the cluster\n";
   push(@alert, $alert);
  } elsif ( $diskstat[1] eq "Online" ) {
   if ( $debug) {
    $alert="Disk Resource Group Info: $diskstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } else {
  print "WTF ---- SCSTAT -D\n";
 }
}

###################################
# EDIT To: Reply-To: and From:
# if you want mail to go somewhere
# useful and-or helpful!
###################################

if ( @alert > 0 || @debug > 0 ) {
 open(CMAIL, "|/usr/lib/sendmail -t");
 print CMAIL "Subject: CLUSTER ALERT - $hostname - $date\n";
 print CMAIL "From: you\@yourdomain.com\n";
 print CMAIL "Reply-To: you\@yourdomain.com\n";
 print CMAIL "To: recipients\@yourdomain.com\n";
 print CMAIL "\n\n";
 foreach $warning (@alert) {
  print CMAIL $warning;
 }
 if ( $debug > 0 ) {
  foreach $message (@debug) {
   print CMAIL $message;
  }
 }
 close(CMAIL);
 system("touch /tmp/cfail_ihot_stat");
} elsif ( @alert == 0 && -f "/tmp/cfail_ihot_stat" ) {
 open(CMAIL, "|/usr/lib/sendmail -t");
 print CMAIL "Subject: CLUSTER RESTORED - $hostname - $date\n";
 print CMAIL "From: you\@yourdomain.com\n";
 print CMAIL "Reply-To: you\@yourdomain.com\n";
 print CMAIL "To: recipients\@yourdomain.com\n";
 print CMAIL "\n\n";
 print CMAIL "All Cluster Services Back To Good State\n";
 print CMAIL "All Cluster Nodes: OK\n";
 print CMAIL "All IPMP Groups: OK\n";
 print CMAIL "All Oracle Resources: OK\n";
 print CMAIL "All Storage Groups: OK\n";
 close(CMAIL);
 unlink("/tmp/cfail_ihot_stat");
}

, Mike

linux unix internet technology

Checking If A Mail Server Is An Open Relay Using Perl and Berkeley Sockets!

SPAM is common enough nowadays that you can probably find some version of this basic check somewhere on the web (probably a lot of places), but our intent here is to take a look at what's behind the pretty looking web pages and/or software, used in determining whether or not the mail server you're runnning (or any mail server, for that matter) is an open relay.

Below is a little script I put together using Perl (and its Socket module - based on Berkeley Sockets) that will do this check for you. In a future post, we'll explore Berkeley Sockets programming at a more granular level (as it's far outside the scope of this simple blog post), but for now check out the script below:





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/perl

#
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

use Socket;

$host = "whateverMailServerYouWantToCheck.xyz.com";
$relaydomain = "xyz.com";
$sender = "eggi@noOneWillEverPickThisDomain.com";
$adressee = "info@hopefullyThisMailServerIsNotInTheDomainWereTesting.zyx.com";
$protocol = "tcp";
$port = 25;

if ( $port =~ /\D/) {
$port = getservbyname($port, $protocol);
}
unless ( $port ) {
print "No port : ${port}/$protocol\n";
}
$inet_address = inet_aton($host) || die "No host: ${host}!";
$port_address = sockaddr_in($port, $inet_address);
$protocol_num = getprotobyname('$protocol');
socket(SOCKET, PF_INET, SOCK_STREAM, $protocol_num) || die "Socket: $!";
connect(SOCKET, $port_address) || die "Socket $!";
send(SOCKET, $line, 'SOCK_STREAM') == length($line) || die "Cannot Send Message!: $!\n";
print SOCKET "HELO $relaydomain\n";
print SOCKET "MAIL From: $sender\n";
print SOCKET "RCPT To: $adressee\n";
print SOCKET "DATA\n";
print SOCKET "Subject: If this didn't get rejected - you may be running an open relay!\n";
print SOCKET ".\n";
print SOCKET "quit\n";
close(SOCKET) || die "Close $!";
exit(0);

At a very basic level, there are just a few things (covered in the script above) that you need to do to quickly determine if a mail server is an open relay. First, make sure you're sending from a different domain than the mail server's domain. Second, connect to the mail server and attempt to send mail to another (different) domain. If this works, the mail server is acting as an open relay.

Of course, like I said, it might be easier to just go to any number of sites on the web and have that free service do the check for you. But, in any event, this way can be more fun and you can customize it to do whatever (hopefully, good and decent) things you want it to :)

, Mike

linux unix internet technology

Using Solaris Volume Manager To Help With Server Moves

Howdy,

For the basics of setting Solaris Volume Manager up for basic root disk mirroring, you can refer back to an older post on this site (just search for those bolded terms) or try this link.

Here we're going to do discuss another good use for Solaris Volume Manager mirroring: Server moves. Especially server moves where you're required to migrate the server from one server platform to another (Say, from a 6800 to a v490 or something like that).

Using Solaris Volume Manager, it's very simple to set yourself up to be able to roll back all your changes if your migration fails, without losing anything in the process (except maybe a lot of your time since, if it fails, you'll have to do it again at some point).

Sucessfully rolling back is paramount to any server move, or project. If you can't go back to the way things were once you begin to enact a change, you're taking a huge risk if anything goes wrong!

Basically, you'll be guaranteeing your security in three simple steps. Our assumption is that your basic mirroring setup is as such: metadevice d0 consists of metadevice mirrors d10 and d20, d1 consists of d11 and d21, etc)

1. Assuming a simple disk-to-disk mirror setup, detach the secondary mirror for each slice on the root disk (of course, this would apply to all other disks and slices, but I'm trying not to take up too much space today, so we'll stick to the first two slices on the first two disks ;)

metadetach d0 d20
metadetach d1 d21

2. Now remove the secondary disk (leaving in the one that's actively running your system!) and replace it with a new third disk. Format this disk exactly the same as the disk already in the system (using prtvtoc and fmthard, or just using select in the format partition command menu) and create your tertiary mirrors.

metainit -f d30 1 1 c0t1d0s0
metainit -f d31 1 1 c0t1d0s0

Note that this disk is in the same logical disk location as your original mirror disk. That's not a problem and will work to your benefit when you put back the original mirrors.

3. Now attach those new metadevices to the original mirror devices and wait while they sync up your primary disk

metattach d0 d30
metattach d1 d31

Once you're done with this, simply go through this whole process again, but detach the new mirrors, remove your new disk, replace your old disk and reattach the old mirrors (you won't have to metainit your old mirrors, since SVM still thinks they exist but aren't in use).

Now you have a disk that, should everything go horribly wrong, you can pop back in your original server and get things right back to the way they were. You may need to re-install the bootblock on your backup disk (from a single user cd boot), but this will generally work without extra effort :)

, Mike

linux unix internet technology