Sed Manpage Markup Language Translation Script

As I note in the header of this script it's somewhat imperfect. In fact, for the most part it may be almost completely obviated by the last few years of advancements in OS translation of the manpage itself. It is, however, another example of just how crazy you can get with sed.

Back in the day (and I still see this from time to time on some current Unix and Linux systems), if you redirected the output of "man whatever" to a file, like so, in order to get the manpage without using sed, or some other information-massager:

man whatever > OUTPUTFILE

you'd end up with a script full of garbage. Not that it was completely unreadable, but you'd have to filter out all the markup language graffiti on your own (in your head, if possible ;)

That's what prompted me to begin work on stripping down a garbage-output manpage to a simple, and easily readable, file using sed. You may notice, if you need to do this kind of translation and use this method, that a few markup language remnants remain. Feel free to embellish this work to remove them as well. The final section of the script is the easiest place in which to do this. Which also answers the question: Why so many individual replacements and not just one compact regular expression? As noted above, this is a work in progress that I'm trying to make work for a lot of different flavors (and versions) of Linux and Unix. Once I feel that I've run down every possible marker, I'll make it a lot tidier and wrap it up with a bow ;)

Enjoy,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh
#
# Semi-Imperfect Man-Source To Regular-File Translator
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

echo >> $1.txt 

sed -e '1,/NAME/d' -e '/^\.\\"/d' -e 's/^\.if//' -e '/^\.TH/d' -e '/^\.de/d' -e '/^\.ds/d' -e '/^\.ll/d' -e '/^\.in/d' -e '/^\.ti/d' -e '/^\.ie/d' -e '/^\.br/d' -e '/^\.el/d' -e 's/^\.\.//' -e 's/^\.SH\(.*\)/\1\
/' -e '/^\.nr/d' -e '/^\\f/d' -e '/^\.}f/d' -e '/SYNOPSIS/i\
' -e '/COPYRIGHT/i\
' -e '/DESCRIPTION/i\
' -e '/SEE ALSO/i\
' -e 's/\.PP//' -e '/^\.PD/d' -e 's/^\.BR//' -e '/^\.BI \\/{
N
s/\.BI \\/ /g
s/\n/ /g
}' -e '/^\.B \\/{
N
s/\.B \\/ /g
s/\n/ /g
}' -e '/^\.B/{
N
s/\.B/ /g
s/\n/ /g
}' -e '/^\.IR/{
N
s/\.IR//g
s/\n/ /g
}' -e 's/\.SB//' -e '/\.TS/d' -e '/\.TE/d' -e 's/^\.IX//' -e '/\.LP/d' -e 's/\.TP *[0123456789]*//g' -e 's/\\fB//g' -e 's/\\fR//g' -e 's/\\fP//g' -e 's/\\fI//g' -e 's/\\| *//g' -e 's/ n //g' -e 's/ t //g' -e 's/\\^ *//g' -e 's/\\//g' -e 's/\.FN//g' -e 's/\.I//g' -e '/^\.SM/d' -e 's/^\.RS//' -e 's/^\.RE//' -e 's/\.SS//g' $1 >> $1.txt

, Mike

linux unix internet technology

Script to Join Letters In An Array

Today's script is a follow up to yesterday's post in which we'll join letters of a word that we previously split up into an array. In a Unix shell script, it's relatively simple to do this using any number of methods. For our purposes today, we're going to make it difficult ;)

In today's Unix shell script, we've again written it in sh, for maximum portability between systems. You'll also note that, because of this, we're, again, going to use some very basic methods to get the results we want. As noted, the Bourne shell doesn't provide a lot of the conveniences we've come to expect from the more advanced shells, which necessitates a bit more scripting on our part.

Take a look at today's script and notice how we deal with arrays. Since the Bourne shell does not provide a facility for creating or using arrays, we (essentially) have to fake them. As Unix shell scripting goes, this can be a confusing way to attack the problem. Although, a more accurate statement would probably be that mastering these sorts of Unix scripting methods, and being able to fall back on them, will put you in a position where you will always be able to write a script to accomplish what's required. Who needs all those fancy high-level shell built-in's anyway ;)

Hopefully, you'll find this interesting and helpful.

Best Wishes,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

########################################
# shjoin - mash arrays into strings
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Usage - shjoin IFS ${array[@]}
#
# Notes - If IFS is a space, or other 
# shell meta-character, be sure to quote
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
########################################

argvcount=$#

if [ $argvcount -lt 2 ]
then
 exit 1
fi 

TMPIFS=$1
shift
ARRAY=$@

string=`for x in $ARRAY
do
 if [ $x = "\0" ]
 then
  echo " $TMPIFS\c"
 else
 echo "$x$TMPIFS\c"
 fi
done`
newstring=${string}

echo $newstring

, Mike

linux unix internet technology

Script to Split Words On A Null Delimiter

Today's script is going to deal with a problem I've run into from time to time when trying to split words into an array. In a Unix shell script, it's easy, using tools like awk, to split lines into arrays of words; but trying to split a word into an array of characters can sometimes be difficult, if not impossible, given the limitations of the tools at your disposal.

In today's Unix shell script, you'll see that we've written it in sh, for maximum portability between systems. You'll also note that, because of this, we're forced to use some old-style methods to get the results we want. The Bourne (and/or Posix) shell, as wonderful as it is, doesn't provide a lot of the conveniences we've come to expect from the more advanced shells.

Take a look at today's script and notice the prevalent use of expr. There are a million ways you can use this, as a tool in your Unix shell scripting arsenal, to simulate anything the more advanced shells can do. In fact, it would probably be more correct to state that the more advanced shells create their user-friendly built-in commands using these sorts of Unix scripting methods and hiding them from the user. It is, after all, a matter of convenience. No sense in re-inventing the wheel unless you need to ;)

Hopefully, you'll find this interesting and useful. Tomorrow, we'll look at an equally nitty-gritty script that will do the exact opposite.

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

############################################
# shsplit - split words with null delimiter.
#
# 2008 - Mike Golvach - eggi@comcast.net #
#
# Usage - shsplit string
#
# Notes - If string contains spaces, be sure
# to quote it. If you're trying to split a
# string with a delimiter, use awk.
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
############################################

argvcount=$#
expr=/path/to/your/expr

if [ $argvcount -eq 0 -o $argvcount -gt 1 ]
then
exit 1
else
string=$1
letters=`$expr "$1" : '.*'`
fi

basecount=0
dotcount=1

while [ $basecount -ne $letters ]
do
dots=`$expr substr "$string" $dotcount 1`
spacetest=`$expr "$dots" : ' '`
if [ $spacetest -eq 1 ]
then
eval array$basecount="\\\0"
else
eval array$basecount="$dots"
fi
dotcount=`$expr $dotcount + 1`
basecount=`$expr $basecount + 1`
done

basecount=0

while [ $basecount -ne $letters ]
do
eval echo "\$array$basecount"
basecount=`$expr $basecount + 1`
done

, Mike

linux unix internet technology

Simple Factorial Generation - Perl versus Bash

Hey there,

I've seen this floating around the boards, so I thought I'd add my 2 cents. Lots of folks (more homework? When will it end?) are looking for scripts to help them find the factorial of any given number.

For those of you who may not know, the factorial of a number is the number itself multiplied by all the numbers from 1 up to that number. So, the factorial of 3 is: 1 times 2 times 3 = 6

Some of the scripts I see are severely convoluted, so I thought I'd put this up here as a little homework help. It can be solved with Perl in 10 lines (Could be less if I wasn't so hung up on formatting ;)

Interestingly enough - it can be done with the same amount of lines in Linux's bash shell, like so (assuming a recursive function). Or, as I wrote in a previous post, you "could" do it in 1 ;)

factorial ()  {

local number=$1
if [ "$number" -eq 0 ]
then
    factorial=1
else
    let "next = number - 1"
    factorial $next
    let "factorial = $number * $?"
fi
return $factorial
}

This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

print "factorial of: ";
chomp($factorial = <STDIN>);
$number = $factorial;
if ( $factorial == 0 ) {
 $factorial = 1;
}
for ( $factor = $factorial - 1; $factor >= 1; --$factor ) {
 $factorial *= $factor;
}
printf("Factorial of %d is : %g\n", $number, $factorial);

Enjoy,

, Mike

linux unix internet technology



l

Corrected Some Blogspot Code Auto-Rewrites in Previous Two Posts

Just a quick shout.

I noticed a few errors caused by Blogspot's tag interpretation that made the code I posted for the last 2 days unworkable. I've fixed these errors.

If you notice any others (they almost always have to do with the < and > characters), please feel free to email me and let me know, and I'll be happy to revise whatever didn't make it to the post page as I intended.

You can check out this post regarding some of the problems posting code to blogspot. Maybe it will help you out a bit, too :)

, Mike

linux unix internet technology

Using Linux strace To Debug Application Issues

Today's post is a little bit of a walkthrough of using RedHat Linux strace to debug (and find the root cause of ) a system issue and a little bit of caution about how much information you should really share with application project managers if you don't want to be stuck supporting a hacked solution for longer than anyone should. As far as job security goes, it can't hurt, but certain situations really do require that folks upgrade their software to the vendor supported version, no matter how creative your solution ends up being.

Note: If you're looking for an in-depth examination of strace, this post isn't it. Not to turn any readers off, as we'll certainly be exploring that in a future post, but this is more of a walkthrough of a problem resolution involving strace rather than a dissertation on the command itself.

In this particular instance, we were working with a product (heretofore referred to as the Product or Product) that, to my knowledge, had just begun having "issues." This generally means to me that somebody did something they weren't supposed to or were trying to do something they weren't supposed to. I'm cynical, but I try to have the common courtesy to keep it to myself ;)

The Product was a client/server application running on the Java platform that had suddenly begun dropping connections from the application server to the backend database. I did a quick check of the /proc/****/status file (and, of course, gave a quick nod to "top" and "netstat -an"), confirming that they were dropping quite a few, like so:

host # cat /proc/14653/status | grep Threads
Threads: 311

host # top <-- Truncated to just show the top process, which was the "failing" one.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14653 user 16 0 1634m 1.1g 40m S 120 14.1 296:46.54 java

I found the "total time used" (listed under the TIME+ column) somewhat alarming since the process had only been up for about a half an hour! It couldn't possibly be correct.

The next thing I did, since I like to go to the guts when I get brought in on an emergency issue regarding a Product I've never used before, was to shut it down and restart it with strace; teeing the information off to a file in /tmp (since I like to watch the characters fly by), like so:

host # strace -vF ./product arguments 2>&1|tee /tmp/OUTPUT

Note that this could have also been accomplished by running the process, getting it's pid from the ps tables and adding the -p PID flag to the strace command in order to attach to the process after starting it. I prefer to start the process with strace, if I can, in case something critical happens during the initial load-up.

Long story short (It is Sunday - Our day of rest - after all ;), I found that the Product process was dumping tons of FUTEX_WAIT errors. At this point, I checked the version of the OS with uname for the only part I really cared about at that point:

x86-64

The machine was running RedHat Advanced Server 5 - 64 bit. The FUTEX_WAIT errors have to do with the implementation of Fast Userspace Mutex locking implemented in the stable version of the RedHat Linux kernel 2.6 and up. This raised a red flag, and caused me to ask the inevitable question: When did this start happening?

As it turns out, the Product had previously been hosted on another machine running RedHat Advanced Server 4, 32 bit. This made perfect sense, since the Mutex locking mechanism at the kernel level they "used" to run on was completely different, and not compatible, since the Product was built to use the old style Mutex locking mechanisms.

Futile attempts were made to add lame fixes, like:

LD_ASSUME_KERNEL=2.4.whatever (Apologies; I don't remember what it was off-hand since it didn't end up helping at all ;)

to the Product's startup scripts, but this didn't resolve the problem. The "actual" (and correct) resolution would have been to upgrade to the vendor-supported version of the Product. The manufacturer of the Product actually had a more recent version available that was built specifically to address this issue and run on RedHat AS 5 - 64 bit.

And, here's the kicker (as per the cautionary message at the beginning of this post): As some of you may know, a bug in strace (It's actually more of a "feature" since it almost "has" to happen) makes it slow down a process slightly due to all the extra work it has to do to parse all the system calls, opens, reads, writes, errors, library loads, etc.

strace slowed the process down enough that the FUTEX_WAIT errors stopped occurring and the Product began functioning normally again. We ended up adding it to the init scripts after much debate over spending a few bucks to avoid possible total confusion in the future.

In retrospect, I'm glad that I was able to help identify the issue and find the root cause. On the other hand, I wish there was some way I could have avoided identifying the problem without providing a cheap and quick "out" for the end user. I'm sure, in the future, someone else will have to work on this Product (maybe a problem with the version of Java or something more insidious) and they'll be given about as much information as I was. Documentation was laid down, but I don't expect anyone will read it. Sometimes, the solution is as bad as the problem, in a way or two ;)

Enjoy your Sunday :)

, Mike

linux unix internet technology

Working with Linux RPM's

This post is a continuation, of sorts, of my last post. This is more of a general-audience post. Most experienced admins know most of this stuff already. Like I mentioned previously, I try to write this blog with an appreciation for what it was like when I first started out in the business. I owe my success to a great many patient and helpful people.

In this post, I wanted to hit on the basics of working with RPM's in Linux (RPM stands for the Redhat Package Management system - basically, they're the software packages that make up your system). In later posts we'll go into some neat tricks... But for now, we'll stick with the basics. Knowing the basics in any field of interest is invaluable in growing and mastering that skillset, just like knowing your ABC's can really help if you ever intend to read or write :)

Check the bottom for a recap of all the RPM options we're going to use and their literal meanings:

1. To display the basic information for any RPM, just type:

host # rpm -qi RPM_NAME - like:

host # rpm -qi bash
Name : bash Relocations: /usr
Version : 2.05 Vendor: Red Hat, Inc.
Release : 8.2 Build Date: Mon 28 Jun 2004 10:33:55 AM CDT
Install date: Thu 12 Jan 2006 01:25:27 PM CST Build Host: host.redhat.com
Group : System Environment/Shells Source RPM: bash-2.05-8.2.src.rpm
... EDITED OUT FOR BREVITY'S SAKE!

2. If you're not sure where to start with the above command, just have RPM spit out all the packages it knows about and pipe that to more, like so:

host # rpm -qa|more
redhat-logos-1.1.3-1
glibc-2.2.4-32.18
cracklib-2.7-12
dosfstools-2.7-1
gdbm-1.8.0-11
...

3. Now that you've figured out what package you want to inspect (Note that you don't have to include the full name to get the information from RPM. The redhat-logos-1.1.3-1 program can be referred to simply as redhat-logos) and have gotten some basic information about it, you can list out all the files associated with the package like this:

host # rpm -ql bash
/bin/bash
/bin/bash2
/bin/sh
/etc/skel/.bash_logout
...

4. Here's one that doesn't require a lot of output, since it's somewhat of a re-explanation. You can add the -p flag to the examples in points 1 and 3 if you're querying an RPM package, and not the RPM database!

host # rpm -qip bash-2.05-8.2.i386.rpm <--- Listing out information for the RPM package itself.
host # rpm -qlp bash-2.05-8.2.i386.rpm <--- Listing out files associated with the RPM package itself.

5. Of course, you may find a file and want to know what RPM package it belongs to. You can get that by typing:

host # rpm -qif /etc
Name : filesystem Relocations: (not relocateable)
Version : 2.1.6 Vendor: Red Hat, Inc.
Release : 2 Build Date: Mon 20 Aug 2001 03:34:02 PM CDT
Install date: Thu 12 Jan 2006 01:24:41 PM CST Build Host: host.redhat.com
Group : System Environment/Base Source RPM: filesystem-2.1.6-2.src.rpm Vendor: Red Hat, Inc.
... (Just as long as the description in point 1)

6. If you want to install a new RPM, you'll need the package file, and would run RPM like this:

host # rpm -i bash-2.05-8.2.i386.rpm

This isn't very interesting (which may be what you want -- I don't care to look at verbose output "all" the time). You can spice it up by adding the -v and/or -h flag, like so:

host # rpm -ivh bash-2.05-8.2.i386.rpm

7. If you want to uninstall an RPM, you'll just need to know the abbreviated name, like I mentioned in point 4). You can also make this as verbose and visually entertaining as the system will allow with -v and/or -h:

host # rpm -e bash

Note that this command would return an error if you had multiple instances of the bash RPM installed. In that case, you could still abbreviate, but would have to include the version number. So you'd type

host # rpm -e bash-2.05.8.2

instead of just bash.

So, to recap, and possibly explain anything I may have glossed over, these basic commands should get you started working with the RPM package management facility on Linux. The translations of the flags we've covered are as follows:

Major flags (usually the ones preceded with a dash, but you can arrange the flags in whatever order you choose - just be careful - see note in the minor flags):

q = query
i = install
e = remove/uninstall

Minor flags

i = information (not the same as the major flag i. Of course, you'll probably never use -ii or -ei, as the combinations would be redundant and opposite, respectively.
a = all
l = list
p = RPM package file (e.g. whatever.rpm)
f = file
v = verbose
h = hash (prints lots of # symbols while it completes your request :)

Enjoy getting started working with RPM packages. They're one of the foundations of the Linux operating system. In fact, a combination of certain packages actually "is" the operating system. Knowing how to manipulate them and have them work for you can make it easier to explore many other things (like new software you've always wanted to install and try out :)

Best wishes,

, Mike

linux unix internet technology

A Few Linux Networking Tips

Here's a little something for those of us who use Linux (The place I work uses RedHat primarily) on a day to day basis. Lots of shops these days are switching from the more expensive solutions, like those offered by Sun and HP, to cut cost of deployment and maintenance, which opens up a great pathway into the Linux administration field for folks who are eager to learn. I can tell you, after working all angles of the *nix arena for a decade or so, that there's nothing more grating than having to fill out a form and wait for someone to do something you're supposed to be being paid to be good at. I'm not going to sneeze at my paycheck, of course, but I'm worried that, if I work at too many big corporations with big contracts, my mind will atrophy and I'll die a slow miserable death long before my body gives out ;)

In this post, I just wanted to touch on some Linux networking basics. I try to write this blog for users of all skill levels and I think some of my posts assume a lot of pre-knowledge. This isn't a "for dummies" site, by any means (I've never understood how they marketed that series of books so well. I've tried flat-out telling people they're idiots for most of my natural life and it's never ended well ;). That being said, I'm hoping this blog attracts folks with a wealth of experience and is also accessible to those new to the field.

So, today, I'm going to touch on some Linux networking tips at a basic level. As with everything, over time, I'll dive into these subjects in greater detail. But for now, we'll get to the meat of the post. A lot of times, I find, it makes more sense to know what to do and understand it later, rather than the opposite. To that end (This is all pretty much RedHat specific - please don't try this on a Sun box without a healthy ego - the error messages can be blunt ;) here we go:

1. Adding a default router:

You can generally do this one of two different ways:

If the NIC is already configured and UP, all you need to do is use the route command, like so:

host # route add default gw 127.0.0.1 dev eth0

If you want to add a default route permanently, you'll just need to add this line to the existing /etc/sysconfig/network configuration file:

Note that the NETWORKING and HOSTNAME variables should already be in there (If not, assign them values of yes and "whatever your hostname is", respectively. Also, your network may not be up ;)

GATEWAY=127.0.0.1

Of course, if you prefer, you can always use the /etc/init.d/network script to bounce your NIC's and routes (You set these up in the /etc/sysconfig/network and /etc/sysconfig/network-scripts/ifcfg-eth0, etc, scripts). You can also use the service command to bring up your network, bring it down or restart it (which will, again, re-read your configuration).

2. Displaying your NIC's device driver settings:

This is most commonly done with a command called ethtool. To get your NIC's settings you could do the following:

host # ethtool -i eth0
driver: tg3
version: 3.10u6
firmware-version:
bus-info: 05:01.0

You can then use this info to help with problem solving. For instance, if eth0 isn't coming up correctly, perhaps eth0 doesn't have a proper alias setup in /etc/modules.conf, like:

alias eth0 tg3 <--- This line tells us that eth0 is actually an alias referring to the tg3 device driver.

If you look in there and it's:

alias eth0 e1000

you've found the problem right there!)

3. Display your NIC's Speed, Duplex and Negotiation settings (also with ethtool):

This one is just as simple as the command above (ethtool is much nicer than using old-style ifconfig, netstat and/or kstat - although they all have their virtues and are necessary depending on how old your Linux distro is)

host # ethtool eth0
Settings for eth0:
Cannot get device settings: Resource temporarily unavailable
Supports Wake-on: g
Wake-on: d
Current message level: 0x000000ff (255)
Link detected: no

Looks like the link's not up for that one! It's just as easy to spot if all's well (You can sometimes tell from a good distance away ;):

host # ethtool eth1
Settings for eth1:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Current message level: 0x000000ff (255)

4. Now to configure, or initialize, those settings, we'll use ethtool as well:

Assuming we found out that the device associated with eth0 didn't have the cable connected, and we've got that all set up (along with correcting the alias in /etc/modules.conf, if that was wrong), we could fix it all by doing something like this:

ethtool -s eth0 speed 1000 duplex full autoneg off

You can add as many option/value pairs as you need (e.g. speed 1000) to get the job done. Don't forget to update the /etc/sysconfig/network and /etc/sysconfig/network-scripts/ifcfg-eth0, etc, files with your settings. If you don't, you'll have to do this every time the machine reboots!

Hope this has helped you out some or, at least, helped you get started taking on the Linux world :)

Cheers,

, Mike

linux unix internet technology

NetBackup Policy Reporting Script

This is a follow-up to an earlier post regarding NetBackup where we dealt with using scripting to monitor daily NetBackup activity (actually, the time frame of the reporting was open if you wanted to change it).

In this little scriptlet, we deal with reporting on NetBackup Policies. Since this is widely variable (You name your own policies, after all), I made that part of the pattern-matching involved a variable ;). The script parses the bppllist command output using the standard -U option, which is formatted for easier reading and makes for easier line matching. Another variation on this, if you prefer to report on Types rather than Names is running bppllist with the -L option. This will give you output such as:

Policy Type: MS-Exchange-Server

and

Type: FULL SExchange

These options are addressed by including a variables section at the top of the script so that you can customize it in that way rather than deal with working your way back up through script options on top of bppllist options. In short, it's only a little less confusing, but looks a lot better. My feeling is that, in some instances it's better to have 3 slightly differently named versions of the same script that you run when you need to, rather than trying to remember 3 different ways to invoke a script that would require you to remember various patterns you needed to match.

Of course, this is presented for your assistance and, hopefully, convenience. Feel free to modify it to suit your particular needs and style.

Have a great weekend!





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# list_bp_schedules
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

##############################################
# Variables that should probably be edited
$names_or_types = "1"; # 0 = Types - 1 = Names
$policy_name_or_type = "MY_POLICY"; # This is treated as a regexp!
##############################################

if ( $names_or_types )  {
    @bp = `/usr/openv/netbackup/bin/admincmd/bppllist -allpolicies -U`;
}   else    {
    @bp = `/usr/openv/netbackup/bin/admincmd/bppllist -allpolicies -L`;
}

$date = `date +%m%d%y`;
open (OUTPUT, ">>Schedules.$date");
select(OUTPUT);
foreach $line (@bp) {
 if ( ! $daily ) {
     if ( $names_or_types )  {
      print "$line\n" if ( $line =~ /Policy Name/ || $line =~ /Schedule/ );
        }   else    {
            print "$line\n" if ( $line =~ /Policy Type/ || $line =~ /Schedule/ );
        }
  if ( $line =~ /$policy_name_or_type/ ) {
   $daily = 1;
  }
 } elsif ( $daily ) {
  if ( $line =~ /\w/ ) {
   print "$line\n";
  } else {
   $daily = 0;
   next;
  }
 }
}

, Mike

linux unix internet technology

Getting Error Values From The Middle Of A Pipe Chain In Bash

This is something very interesting I found out not too long ago, while hashing out some work with a colleague. As most administrators (or users) who do a fair amount of shell scripting know, the error status or return code (Generally referred to as "errno" in all the man pages) of a process is a fairly common method to use in determining the process flow of a script.

The one thing about the value of "errno" (or, literally, the variable "$?" in most shells) is that it's erased with each consecutive process that gets run. So if you were to run a series of command lines that echoed the return value of the grep command, the following example would be accurate (assuming the string "bob" can't be found in /home/myfile):

host # grep bob /home/myfile >/dev/null 2>&1
host # echo $?
host # 1

while this one would give you misleading information:

host # grep bob /home/myfile >/dev/null 2>&1
host # touch /home/myfile
host # echo $?
host # 0

So, on the first set of command lines, you're actually getting the return code of 1 from grep (indicating that it can't find the string "bob" in /home/myfile), while the second one gets you the return code of 0 from the touch command. "errno" always contains the return value of the last-executed command.

Which brings us around to the topic indicated in the title of this post (I promise to tie in the whole introduction about "errno" at the end; it wasn't a complete waste of your time ;). While it's easy enough to trap "errno" in any series of disconnected commands (for instance, in the second example above, if we'd echoed $? before running touch, it would have given us the correct output), I had always thought it was impossible to grab the correct value from a command in the middle of a pipe chain, like this:

host # grep bob /home/myfile 2>&1|Grep joe|xargs echo
host # echo $?
host # 0

You'll note that I purposefully capitalized the G in grep so that it would return an error code that didn't indicate success, yet - since this is a chain of commands all connected by pipes - "errno" returns the value of the xargs command, since it was the last one executed. Which means I've spent a lot of time jumping through hoops to "reword" any pipe chain so that I could extract the information I needed.

Now (and I'm almost positive this wasn't the case a few years back) the bash shell has actually taken on this predicament and come up with a nice workable solution for it(I'm waiting for it to pop up in sh and ksh, since they've been burned into my psyche over the last decade or so). In bash, if you run a series of piped-together commands, you can actually extract the value of "errno" from any command in the chain by using the shell built-in PIPESTATUS array, like so:

host # grep bob /home/myfile 2>&1|Grep joe|xargs echo
host # echo ${PIPESTATUS[@]}
host # 1 127 0

How nice is that? :) Now you can easily tell the return value of every process in a pipe-chain. The initial grep returns 1 because the string "bob" isn't in /home/myfile, the misspelled Grep returns 127 because the command can't be found and the final xargs returns 0. That solves a lot of problems and can potentially save you lines upon lines of convoluted code.

The one thing about it that can be frustrating is that it behaves in much the same way as "errno" (See, I told you I'd bring it back around ;). If you don't capture the output immediately (or dish it off into another variable), the array will zero out and contain no values as soon as you enter your next command, like so:

host # grep bob /home/myfile 2>&1|Grep joe|xargs echo
host # touch /home/myfile
host # echo ${PIPESTATUS[@]}
host # 0

At this point, after we've executed the touch command, the PIPESTATUS array has been cleared out, just like "errno" gets written over, even though we haven't executed another pipe chain. Its behaviour is basically identical. Below, we show that, once the array has been written over, its size gets reduced to 1 ( The single return value of the last executed command) and we further prove that the array really has been clipped down to one variable by attempting to print the first and second values; the second of which doesn't exist. Continued from above:

host # echo ${#PIPESTATUS[@]} <--- Here we ask bash for the size of the PIPESTATUS array
host # 1
host # echo ${PIPESTATUS[0]} <--- Here we check the first variable in the PIPESTATUS array
host # 0
host # echo ${PIPESTATUS[1]} <--- Here we check the second variable, which now doesn't exist
host #

This is easy enough to get around, however, since - just like "errno" - you can assign that array to another array before you execute another command, like so:

host # grep bob /home/myfile 2>&1|Grep joe|xargs echo
host # new_array=${PIPESTATUS[@]}
host # touch /home/myfile
host # echo ${PIPESTATUS[@]}
host # 0
host # echo ${new_array[@]}
host # 1 127 0

If you knew this already, I envy you the convenience you continue to enjoy. For the rest of us; a pleasant surprise :)

Best Wishes,

, Mike

linux unix internet technology

Non-Maliciously Scanning For Open Network Ports

Generally, port scanning is looked at as a "Black Hat" activity. This isn't so much stereotyping, as a heavily weighed and considered generalization. Entire industries are built around the fact that most, if not all, networked servers (public and private) are under almost constant attack by some individual or group. It's for that reason that most folks, like myself, take pause before ever engaging in such activity, even if our motives are pure. Depending on your situation, the very fact that you do this might get you flagged.

On the flipside, however, is the obvious "need" to be able to check whether ports are open. Reasons range from making sure that services are up and available, to "White Hat" activity, like making sure that all of the network ports that you think are closed actually are; thus enabling you to detect security risks and eliminate them before someone else can take advantage.

The script I'm putting up today is a port scanner in its purest sense. It doesn't attempt to "investigate" the services running on any of the network ports it finds open, which is usually enough to keep you from getting threatening emails from your security department ;) The little reporting it does do on open network ports is variable, as it gets the information from the /etc/services file on the host from which it's being run (using the standard getservbyport call).

You call the script by whatever name you give it (I'll call it "scanner" because it makes sense and, maybe, I'm not feeling very original right now ;) and supply it with the IP Address of the target server and the highest port on it that you want to scan. The script assumes you want to start at port 1. You can change that very easily by modifying the $port variable. You also need to be root to run this script, as it creates sockets (using standard Berkeley Sockets code) and regular users don't have system permission to do that on Unix and Linux systems. If they do, something is horribly wrong :P

A sample run would look like this:

host # ./scanner 127.0.0.1 1000
Open : 80/tcp :
Open : 111/tcp : sunrpc
Open : 443/tcp :
host #

In the above example, it appears that we've got a regular and secure webserver running, as well as the sunrpc service. The reason that the description field on the left is empty for ports 80 and 443 is that there are no entries in the host's /etc/services file for those ports. Simply adding those entries in that file would make the program produce this on an exactly equal run:

host # ./scanner 127.0.0.1 1000
Open : 80/tcp : http
Open : 111/tcp : sunrpc
Open : 443/tcp : https
host #

Hopefully, you'll find this useful with regards to your completely-above-board administration activities :)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/perl

#
# scanner - simple port scanner
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if ( $#ARGV != 1 ) {
 print "Usage: $0 hostnameOrIP highPort\n";
 exit(1);
}

use Socket;

$host = $ARGV[0];
$highport = $ARGV[1];
$port = 1;

OUTERLOOP: while ( $port <= $highport ) {
 $| = 1;
 $line = " ";
 $tcp = 1;
 INNERLOOP: while ($tcp >= 0) {
  if ( $tcp ) {
   $protocol = "tcp";
  } else {
   $protocol = "udp";
  }
#print "${port}/$protocol... ";
  if ( $port =~ /\D/) {
   $port = getservbyname($port, $protocol);
  }
  unless ( $port ) {
   print "No port : ${port}/$protocol\n";
   if ( $tcp ) {
    $tcp--;
    next INNERLOOP;
   } else {
    $port++;
    next OUTERLOOP;
   }
  }
  $inet_address = inet_aton($host) || die "No host: ${host}!";
  $port_address = sockaddr_in($port, $inet_address);
  $protocol_num = getprotobyname('$protocol');
  socket(SOCKET, PF_INET, SOCK_STREAM, $protocol_num) || die "Socket: $!";
  unless ( connect(SOCKET, $port_address) ) {
   if ( $tcp ) {
    $tcp--;
    next INNERLOOP;
   } else {
    $port++;
    $tcp--;
    next OUTERLOOP;
   }
  }
  if ( $tcp ) {
  send(SOCKET, $line, 'SOCK_STREAM') == length($line) || die "Cannot Send Message!: $!\n";
  } else {
  send(SOCKET, $line, 'SOCK_DGRAM', $protocol) == length($line) || die "Cannot Send Message!: $!\n";
  }
  close(SOCKET) || die "Close $!";
  $service = getservbyport($port, $protocol); 
  print "Open : ${port}/$protocol : $service\n";
  $port++;
  $tcp--;
 }
}

, Mike

linux unix internet technology

Script To Convert Solaris Truss Error Output Into Plain English

A fairly common request I get when helping to debug problems with executables, or other types of programs that need to be "trussed" (or have their system calls traced; which is a huge generalization ;), is to "please explain what all the error output means." Sometimes, possession of this knowledge can be the answer to the problem handed to me on a plate.

Solaris' truss (available on Linux as strace or xtrace - although with slightly different options and output) is an excellent tool for debugging and can be used as simplistically or with as much complexity as you're comfortable with. End users of Unix systems (the client or office staff) generally don't want to deal with it at all.

Although it isn't always true, a lot of times the error message that precedes a program's crash is a significant help in determining the root cause of the problem. Just knowing that, for instance, the program tries to write to an output file, and gets an error indicating that there's not enough space left on the device (or partition), immediately before it crashes can solve the case right then and there.

To that end, I threw together today's script. I think we'll definitely delve deeper into using truss (and also xtrace/strace) in a completely separate post. This script serves a very limited purpose, but can be a helpful tool to use as a first step, if you've got a lot going on (or if you believe - like I do - that most problems aren't as complicated as we can make them ;). It takes the arguments of whatever program you would run truss against (along with that program's arguments). So, if you would normally run (and we're keeping the truss simple here, with no options, which isn't the case in the script):

/usr/bin/truss /usr/bin/myprogram -f myconfig

Here, you'd simply run ( I'll name this script error_detail.sh for now):

./error_detail.sh /usr/bin/myprogram -f myconfig

The script essentially strips down the output of truss to the lines that contain system errors (like ENOENT and EIO, etc) and then takes those lines and prints them out, followed by the literal description of the error ( extracted from the contents of Solaris' /usr/include/sys/errno.h).

e.g. EPERM translates into "Operation Not Permitted"

Hopefully, this will help you get to those simple conclusions a little faster (and also remind you of what some of the more obscure system errors actually mean ;)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/ksh
#
# error_detail.sh
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

trap 'exit 1' 1 2 3 9 15
program="$@"

/usr/bin/truss -f $program 2>&1|awk '{ if (( $NF > 0  && $(NF-1) ~ /Err#/ ) || ( $0 ~ /truss: cannot trace/ )) print $0 }'|while read x
do
 cant_truss=`echo $x 2>&1|grep -i "truss: cannot trace" >/dev/null 2>&1`
 if [ $? -eq 0 ]
 then
  print "Truss cannot trace this command and/or it's argument(s)!"
  print $x
  exit 2
 fi
 err_detail=$(grep `echo $x|awk '{ print $NF }'` /usr/include/sys/errno.h|sed 's/\/\*\(.*\)\*\//\1/'|sed 's/^.*[0123456789] * *\(.*\)$/\1/')
 print "$x : $err_detail"
done
exit 0

, Mike

linux unix internet technology

Simple Sun Cluster Monitoring Script

Hey There,

Today, I've included a script I wrote to monitor a small SunCluster 3.1 environment (Running Oracle Parallel Server - OPS) we have set up at our shop. It basically runs through the output of various "scstat" command variations and reports on all combinations of errors it encounters. I built some functionality in it to be as specific as possible about the error states and to make sure that it returns an "All Clear" once the error condition no longer exists.

I also wrote this to run in cron. I run it every 5 minutes. Any time period is acceptable, depending upon what you need, and, of course, you could always put this script in a wrapper so that it runs constantly (although that generally necessitates writing another script to make sure that this script is running, restarting it as necessary and vice versa - For Veritas Cluster Server fans out there, this is a cheap and quick way to imitate the relationship between had and hashadow).

This was also written for a small environment (2 machines with 2 "live" network connections each). I didn't include monitoring of the heartbeats, since the script is meant to be run locally (with "exactly" the same values in the customizable section) on all nodes in a cluster, and scstat's indication of failure on any of these tests is, in and of itself, a guarantee that loss of a heartbeat connection is the very least of your problems ;)

Enjoy!





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

###################################
# suncluster_mon - check cluster health
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
# simple parser for:
# 1. scstat -n
# 2. scstat -i
# 3. scstat -g
# 4. scstat -D
#
###################################
 
###################################
# CUSTOMIZE SETTINGS HERE
# $debug can be set with the -d
#  switch when invoking the script
###################################
$primary_node="";
$secondary_node="";
$primary_eth="";
$secondary_eth="";
$date=`/usr/bin/date "+%A, %B %d - %H:%M:%S"`;
$hostname=`/usr/bin/hostname`;
$debug=0;
###################################
# PLEASE DO NOT EDIT BELOW
# EXCEPT MAIL SETTINGS AT END
###################################

chomp($date);

if ( $#ARGV >= 0 ) {
 foreach $arg (@ARGV) {
  if ( $arg =~ /-d/ ) {
   $debug=1;
   print "Debug Output Set\n";
  } else {
   push(@nogo, $arg)
  }
 }
 if ( @nogo > 0 ) {
  print "Unrecognized options: @nogo\n";
  print "Ignoring and continuing\n";
 }
}

@scstatn=`/usr/cluster/bin/scstat -n|/usr/bin/egrep 'node:'`;
for $nodestat (@scstatn) {
 if ( $nodestat =~ /Cluster node:/ ) {
  $nodestat =~ s/Cluster node://;
  @nodestat = split(" ", $nodestat);
  if ( $nodestat[1] eq "Offline" && $nodestat[0] eq $secondary_node ) {
   if ( $nodefailure ) {
    $alert="\nCluster Alert:\n$primary_node and $secondary_node are both in Offline state.  Cluster Failed\n";
   } else {
    $alert="\nCluster Alert:\n$primary_node is in Offline state.  $secondary_node is Online - Cluster Crippled\n";
   }
   $alert="\nCluster Alert:\n$nodestat[0] has switched to $nodestat[1] Status - Cluster Crippled - Checking For Dual Failure\n";
   $nodefailure=1;
   push(@alert, $alert);
  } elsif ( $nodestat[1] eq "Offline" && $nodestat[0] eq $primary_node ) {
   if ( $nodefailure ) {
    $alert="\nCluster Alert:\n$primary_node and $secondary_node are both in Offline state.  Cluster Failed\n";
   } else {
    $alert="\nCluster Alert:\n$primary_node is in Offline state.  $secondary_node is Online - Cluster Crippled\n";
   }
   $alert="\nCluster Alert:\n$nodestat[0] has switched to $nodestat[1] Status - Cluster Crippled - Checking For Dual Failure\n";
   $nodefailure=1;
   push(@alert, $alert);
  } else {
   if ( $debug ) {
    $alert="Cluster Info: $nodestat[0] and $nodestat[1] OK\n";
    push(@debug, $alert);
   }
   $beautiful++;
  }
 } else {
  print "WTF ---- SCSTAT -D\n";
 }
}
if ( $beautiful == 2 ) {
  if ( $debug )   {
   $alert="Cluster Info: Both nodes OK\n";
   push(@debug, $alert);
  }
}

@scstati=`/usr/cluster/bin/scstat -i|/usr/bin/egrep 'Group:'`;
for $ipmpstat (@scstati) {
 if ( $ipmpstat =~ /IPMP Group:/ ) {
  $ipmpstat =~ s/IPMP Group://;
  @ipmpstat = split(" ", $ipmpstat);
  if ( $ipmpstat[0] == $primary_node ) {
   if ( $ipmpstat[3] eq $primary_eth && $ipmpstat[4] ne "Online" ) {
    $alert="\nIPMP Alert:\n$ipmpstat[3] is in $ipmpstat[4] mode on $ipmpstat[0] - Checking for Online of $ipmpstat[0] $secondary_eth\n";
    $ethprimaryfailure=1;
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Online" && $ethprimaryfailure == 1 ) {
    $alert="\nIPMP Alert:\n$ipmpstat[0] $secondary_eth is not in an Online state on $ipmpstat[0] - IPMP Group Down on both Interfaces!\n";
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Standby" && ! $ethprimaryfailure ) {
    $alert="\nIPMP Alert:\n$ipmpstat[3] is in $ipmpstat[4] mode on $ipmpstat[0] - Failover interface is not UP - IPMP Group Crippled on $ipmpstat[0]!\n";
    push(@alert, $alert);
   } else {
    if ( $debug )   {
     $alert="IPMP Info: IPMP Group $ipmpstat[0] $ipmpstat[3] OK\n";
     push(@debug, $alert);
    }
   }
  } elsif ( $ipmpstat[0] == $secondary_node ) {
   if ( $ipmpstat[3] eq $primary_eth && $ipmpstat[4] ne "Online" ) {
    $alert="\nIPMP Alert:\n$primary_eth is in $ipmpstat[4] mode on $ipmpstat[0] - Checking for Online of $ipmpstat[0] $secondary_eth\n";
    $ethprimaryfailure=1;
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Online" && $ethprimaryfailure == 1 ) {
    $alert="\nIPMP Alert:\n$ipmpstat[0] ipmpstat[3] not in an Online state on ipmpstat[0] - IPMP Group Down on both Interfaces!\n";
    push(@alert, $alert);
   } elsif ( $ipmpstat[3] eq $secondary_eth && $ipmpstat[4] ne "Standby" && ! $ethprimaryfailure ) {
    $alert="\nIPMP Alert:\n$ipmpstat[3] is in $ipmpstat[4] mode on $ipmpstat[0] - Failover interface is not UP - IPMP Group Crippled on $ipmpstat[0]!\n";
    push(@alert, $alert);
   } else {
    if ( $debug )   {
     $alert="IPMP Info: IPMP Group $ipmpstat[0] $ipmpstat[3] OK\n";
     push(@debug, $alert);
    }
   }
  }
 } else {
  print "WTF ---- SCSTAT -I\n";
 }
}

@scstatg=`/usr/cluster/bin/scstat -g|/usr/bin/egrep 'Resource:|Resources:|Group:'`;
for $rgstat (@scstatg) {
 if ( $rgstat =~ /Resources:/ ) {
  $rgstat =~ s/Resources://;
  @rgstat = split(" ", $rgstat);
  if ( @rgstat != 5 ) {
   shift @rgstat;
   $alert="\nOracle Resource Alert:\nOracle resource group $rgstat[0] is not running all resources on the cluster\n--Only running -- @rgstat\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Oracle Resource Info: Oracle resource group $rgstat[0] OK -- @rgstat OK\n";
    push(@debug, $alert);
   }
  }
 } elsif ( $rgstat =~ /Group:/ ) {
  $rgstat =~ s/Group://;
  @rgstat = split(" ", $rgstat);
  if ( $rgstat[2] ne "Online" && $rgstat[1] eq $primary_node ) {
   $alert="\nOracle Resource Alert:\nResource Group $rgstat[0] is in $rgstat[2] state on primary node $rgstat[1] - Checking for failover\n";
   $rgprimaryfailure=1;
   push(@alert, $alert);
  } elsif ( $rgstat[2] ne "Online" && $rgstat[1] eq $secondary_node && $rgprimaryfailure == 1 ) {
   $alert="\nOracle Resource Alert:\nResource Group $rgstat[0] is in $rgstat[2] state on cluster - Resource Group Down on both Nodes!\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Oracle Resource Info: $rgstat[1] Resource Group $rgstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } elsif ( $rgstat =~ /Resource:/ ) {
  $rgstat =~ s/Resource://;
  @rgstat = split(" ", $rgstat);
  if ( $rgstat[2] ne "Online" && $rgstat[1] eq $primary_node ) {
   $alert="\nOracle Resource Alert:\nResource$rgstat[0] is in $rgstat[2] state on primary node $rgstat[1] - Checking for failover\n";
   $rgprimaryfailure=1;
   push(@alert, $alert);
  } elsif ( $rgstat[2] ne "Online" && $rgstat[1] eq $secondary_node && $rgprimaryfailure == 1 ) {
   $alert="\nOracle Resource Alert:\nResource $rgstat[0] is in $rgstat[2] state on cluster - Resource Group Down on both Nodes!\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Oracle Resource Info: $rgstat[1] Resource $rgstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } else {
  print "WTF ---- SCSTAT -G\n";
 }
}

@scstatD=`/usr/cluster/bin/scstat -D|/usr/bin/egrep 'servers|status'`;
for $diskstat (@scstatD) {
 if ( $diskstat =~ /servers/ ) {
  $diskstat =~ s/Device group servers://;
  @diskstat = split(" ", $diskstat);
  if ( $diskstat[1] ne $primary_node ) {
   $alert="\nDisk Resource Alert:\n$diskstat[0] has switched primary node to $diskstat[1] from $diskstat[2]\n";
   push(@alert, $alert);
  } elsif ( $diskstat[1] ne $primary_node && $diskstat[1] ne $secondary_node ) {
   $alert="\nDisk Resource Alert:\n$diskstat[0] has failed on all nodes!\n";
   push(@alert, $alert);
  } else {
   if ( $debug )   {
    $alert="Disk Resource Info: $diskstat[1] primary - $diskstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } elsif ( $diskstat =~ /status/ ) {
  $diskstat =~ s/Device group status://;
  @diskstat = split(" ", $diskstat);
  if ( $diskstat[1] ne "Online" ) {
   $alert="\nDisk Resource Group Alert:\n$diskstat[0] has switched to $diskstat[1] state on the cluster\n";
   push(@alert, $alert);
  } elsif ( $diskstat[1] eq "Online" ) {
   if ( $debug) {
    $alert="Disk Resource Group Info: $diskstat[0] OK\n";
    push(@debug, $alert);
   }
  }
 } else {
  print "WTF ---- SCSTAT -D\n";
 }
}

###################################
# EDIT To: Reply-To: and From:
# if you want mail to go somewhere
# useful and-or helpful!
###################################

if ( @alert > 0 || @debug > 0 ) {
 open(CMAIL, "|/usr/lib/sendmail -t");
 print CMAIL "Subject: CLUSTER ALERT - $hostname - $date\n";
 print CMAIL "From: you\@yourdomain.com\n";
 print CMAIL "Reply-To: you\@yourdomain.com\n";
 print CMAIL "To: recipients\@yourdomain.com\n";
 print CMAIL "\n\n";
 foreach $warning (@alert) {
  print CMAIL $warning;
 }
 if ( $debug > 0 ) {
  foreach $message (@debug) {
   print CMAIL $message;
  }
 }
 close(CMAIL);
 system("touch /tmp/cfail_ihot_stat");
} elsif ( @alert == 0 && -f "/tmp/cfail_ihot_stat" ) {
 open(CMAIL, "|/usr/lib/sendmail -t");
 print CMAIL "Subject: CLUSTER RESTORED - $hostname - $date\n";
 print CMAIL "From: you\@yourdomain.com\n";
 print CMAIL "Reply-To: you\@yourdomain.com\n";
 print CMAIL "To: recipients\@yourdomain.com\n";
 print CMAIL "\n\n";
 print CMAIL "All Cluster Services Back To Good State\n";
 print CMAIL "All Cluster Nodes: OK\n";
 print CMAIL "All IPMP Groups: OK\n";
 print CMAIL "All Oracle Resources: OK\n";
 print CMAIL "All Storage Groups: OK\n";
 close(CMAIL);
 unlink("/tmp/cfail_ihot_stat");
}

, Mike

linux unix internet technology

Simple Encryption and Decryption For Fun And No Profit

Here's another little fun thing for the weekend. It's based on Perl's pack and unpack functions and is a good introduction to using them for other purposes. It's also a good way to rediscover the fun of "passing notes in class," even though you're older and have a real job now ;)

You can use this script any way you want to; in fact it's written with a few things left out (how you want to deal with shell special characters -- another complete post on its own -- and if you want to expand on it to read and/or write to STDIN/STDOUT so you can pipe one instance to another, even though that defeats the purpose to a certain degree).

Hopefully, it will pique your curiosity about Perl and its ability to compress and convert different data types (I'm using hexadecimal and character here, but the available list is voluminous).

I've included usage points in the comments section of the script, but the basic usage would be:

tranz.pl encode your message here
tranz.pl decode 458616e6b6370264f62702659637964796e6760245865602c496e657870216e6460255e6968702d456e6167656279656 <-- The Hex output from a message encoded with this script.

Enjoy, and have a safe Sunday :)





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# 2007 - Mike Golvach - eggi@comcast.net
# Sanitize shell special characters in
# whatever manner you prefer - or just
# backslash them on the command line :)
#
# Sample usage:
# trans.pl encode hi there
# trans.pl encode hi there >FILE
# trans.pl encode `cat FILE`
# trans.pl decode 8696024786562756
# trans.pl decode `cat FILE`
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

$input = $ARGV[0];
shift @ARGV;
$message = join(" ",@ARGV);

if ( $input eq "encode")        {
        $output=unpack("h*",$message);
        @length=$output=~/.{0,256}/g;
        print("$output \n");
}       elsif ( $input eq "decode" )    {
        chomp($message);
        $output.=pack"h*",$message;
        print "\n$output\n";
}       else    {
        print "Usage: $0 [encode|decode] whatever you want to type\n";
        exit(1)
}

, Mike

linux unix internet technology

A Simple Trick To Keep Your SSH Session From Timing Out

Hey there,

I thought I'd write a simple something this Saturday; just a little trick that's served me well over the years.

This has to do with any login session in a terminal window (Telnet, SSH, etc). Although this may not always be the case, most places I've ever worked at have had, at least, a few machines were they enforced time-outs on your login. This can be frustrating if you're in the middle of doing something, get called away to do something else, and come back to a "Disconnected!" dialogue box.

Usually, you should be able to get away with just putting a line like this in your .profile or .bash_profile:

TMOUT=0;export TMOUT

But, I've found that a lot of setups don't honor this shell setting. Instead, they take a measure of your activity and log you out if you don't produce enough in your session. So, when you type:

sqlplus @database_query

Even if that takes all day to run, you'll still get disconnected in 5 minutes.

This simple shell loop has almost always worked for me:

while :;do print -n "* ";sleep 15;done

Substitute either
echo -n "* "
or
echo "*\c" <--- \c is the escape character representation of a space.

for the print statement in that little one-line loop, depending on your shell (sh doesn't support the "print" statement) and its implementation of echo (you might be using the built-in or the system binary; in any event - one of these three varieties should do the trick for you.

I leave the carriage-return out on purpose so I can come back later and look at:

*******_

instead of pages of single *'s along the side of my screen, forcing me to scroll back forever to remember what I was doing ;)

This trick basically works because, even though you're not directly interacting with your terminal, you're sending packets back and forth every time that print (or echo) statement executes.

Enjoy a little less stress. Your terminal windows should now be waiting for you instead of threatening to leave :)

, Mike

linux unix internet technology

Shutting Down Domains on Sun 6800/6900 Servers

Here's a little information that comes in handy every once in a while. As you get used to working with Sun's larger machines, the convenience of some of the more advanced features becomes almost trivial. It's common (at least for me) to forget, from time to time, that ServerA and ServerB actually reside on the same physical server (Server1, for example).

That's one thing you get reminded of very quickly when, say, ServerB has a hardware related problem and you need to fix it. When you're dealing with DataCenter class machines, you generally don't want to make a mistake and accidentally pull a card that belongs to ServerA in your attempts to fix ServerB. The headache-multiplication theory is taken for granted, not to mention that company's generally throw lots of money at humongous hardware so they can house systems of "greater importance" on them. ServerA and ServerB, almost literally, translate into lost-revenue when they go down. When you're stuck in this sort of situation, taking your time and doing things right (even if you need to take a gut-punch and "read the manual" ;) is always more important than trying to blast your way through it and hoping for the best.

Luckily, when you're dealing with the 6800/6900 server series from Sun, dealing with domains and working on a "single" machine on a multi-domain system is pretty simple as long as you take the necessary precautions (never be embarrased to type "help"). Also, just so I can start typing 6800 instead of 6800/6900 from now on; the only real difference between the two is the internal architecture. The 6800's are SCSI-based, while the 6900's are fiber. You'll note that this is the difference with almost all of Sun's server series that relate closely (the v480 and v490, or the v880 and v890 - All of them are just slightly different). The 900's and 90's were released because internal fiber disk is much faster than internal scsi-connected disk. To keep it simple ;)

As a for instance, let's say that ServerB is suffering terrible failures (even Sun can't readily explain them). A number of HBA's on the I/O boards have failed and there's a problem with one of the System Boards. Also, your root mirror disk is giving off errors left and right. This is a potentially horrible scenario for which no resolution will be given. We're just using it to make it so we can walk through the process of bringing ServerB down completely and replacing parts.

The first thing you'll want to do is to connect to the System Controller. This can be done any number of ways. Your site should have documentation related to how they've set it up. Generally it will be an SSH or Telnet connection to the SC. You can also set up direct connects for the Domain Consoles, but, even when we have them, I prefer to connect to the SC, as you can get to all of the Domain Consoles from there, as well as the "Platform Shell"! Assuming you've connected, you'll be at a terminal screen that looks something like this.

System Controller 'Server1':

    Type  0  for Platform Shell

    Type  1  for domain A console
    Type  2  for domain B console
    Type  3  for domain C console
    Type  4  for domain D console

Since this is also very specific to how your machine was set up, we'll go with the assumption that ServerA is on the "Domain A" console and ServerB is on the "Domain B" console. Since we want to work on ServerB, and leave ServerA up and running while we do, we'll type in the following (This may seem counter-intuitive at first, but the fact that I'm logging into the "Platform Shell" rather than the "Domain B" console can offer some enhanced control (when you get around to playing with it) and allows you to connect to any domain directly from it):

Input: 0

Platform Shell

Server1:SC> 

Now we're at the SC prompt, at the Platform Shell level -- Remember, at almost any point along the way you can type "help" to get a list of all available commands. When you get a chance, do so, and you'll see what I mean about the enhanced flexibility that starting off at the Platform Shell offers. To continue, we'll connect directly to the "Domain B" console, which is just like logging into a regular machine serial console:

Server1:SC> console b

Connected to Domain B

ServerB console login: root
Password: ******

And, just like on any other machine, we'll bring it down to an ok> prompt as if it weren't a part of a larger physical organism (Server1 - the big 6800)

ServerB# init 0

You'll get the regular system messages and whatever else gets spit to the screen when you normally shut down, and you're there. Now, we'll want to switch from the "Domain Console" to the "Domain Shell." We can do that like so:

{c} ok

<---------- Here type a literal [ctl]+] (the control key and the right bracket (]) simultaneously) - this will get you to a Telnet or SSH prompt - depending on your setup. Then, you'll send a "break" signal to make the switch from Console to Shell.

telnet> send break

Domain Shell for Domain B - ServerB

ServerB:B> setkeyswitch off <-- This command is the one that will "turn off" ServerB. Note that, if you've looked at the "help" output, you don't want to run "poweroff" - That could seriously ruin your day ;) The "poweroff" command is used for powering off the physical grids. Generally, on a two domain 6800, you'll only have one, so running "poweroff" might bring down both ServerB and ServerA. Sun only requires you to split your 6800 into 2 grids if you want to have 3 or 4 domains!

Powering boards off ...
ServerB:B>

Now your "virtual" server (ServerB) is off, and ServerA is still up and running as if nothing were going on. You're ready to begin replacing parts.

As a quick note; before you completely disconnect from the "Domain Shell," I always find it's good practice to run the following comand:

ServerB:B> showboards

Slot     Pwr Component Type                 State      Status     Domain
----     --- --------------                 -----      ------     ------
/N0/SB1  Off CPU Board                      Assigned   Not tested B
/N0/SB2  Off CPU Board                      Assigned   Not tested B
/N0/SB3  Off CPU Board                      Assigned   Not tested B
/N0/IB7  Off PCI I/O Board                  Assigned   Not tested B
/N0/IB9  Off PCI I/O Board                  Assigned   Not tested B

Write down the left-most column (Slot) and glance over the entries to make sure that they're all in the correct Domain (B, here) and that the "Pwr" (power) column lists them all as off. This will help make doubly sure you don't accidentally affect ServerA, as the CPU Board and I/O Board numbers are listed on the outsides of the devices and, if you've written this information down, you can refer to it and easily locate what part of the system you can safely work with.

And, of course (very quickly) for those of you who want to know how to get everything back up and running, just do the following (A very quick summary of commands and output here, as the concepts are all the same, but done in a logical reverse order; with the exception of the rarely needed "resume" command noted below)

System Controller 'Server1':

    Type  0  for "Platform Shell"

    Type  1  for domain A console
    Type  2  for domain B console
    Type  3  for domain C console
    Type  4  for domain D console

    Input: 0
    
    Platform Shell

Server1:SC> 

Server1:SC> console b

Connected to Domain B

<---------- Here type a literal [ctl]+ ]

telnet> send break

Domain Shell for Domain B - ServerB

ServerB:B> setkeyswitch on
Powering boards on ...
ServerB:B>resume

<--- Note that this command and the following are not usually necessary. Once you power on your system by doing the "setkeyswitch on," the 6800 will run through extensive system tests and boot the OS directly.

ok> boot

Hopefully the amount of time spent reading this will save you much much more in the future :)

Cheers,

, Mike

linux unix internet technology

Why Horrible Sun Boot Problems Aren't Always All That Bad

I had an experience at work recently that had me shaking me head (and wishing I'd left for home a few minutes earlier ;) One of our v490 servers, that was already racked,cabled up and ready to have the OS built and put on the network the following day, decided it just wasn't going to boot up; not even to an ok> prompt!

Without the keyswitch set to run extended diagnostics, the situation looked pretty severe. This is about all I saw before it would power back down to nothing:

1:0>Waiting for master in slave_spin() CPU=0:0, timeout in 29 seconds...
2:0>Waiting for master in slave_spin() CPU=0:0, timeout in 29 seconds...
3:0>Waiting for master in slave_spin() CPU=0:0, timeout in 29 seconds...
1:0>
1:0>ERROR: TEST = Slave Spin
1:0>H/W under test = CPU, Motherboard/Centerplane, I/O board, (system init)
1:0>Repair Instructions: Replace items in order listed by 'H/W under test' above.
1:0>MSG = ERROR :Timeout waiting for master, doing re-config reset.
1:0>END_ERROR

And "nothing!" Anyway, as is our company's policy, I placed a call to Sun Support and their suggestion, as is suggested plainly by the error above, was to have a Field Engineer come out and replace the CPU boards (including the CPU's and memory - which is actually faster), and if that didn't work, replace the motherboard, the centerplane and the I/O board, progressively, until the error went away. You can see why I wasn't too happy, right? We're talking about a potential 10 extra hours of work doing parts replacement, followed by diagnostics, followed by possible extra parts orders, replacements, diagnostics, add infinitum (if not ad naseum ;)

Here's the kicker. After hooking up a laptop to the ALOM port, we started the system up with extended diagnostics. It wasn't looking much better. In fact, it gave a lot of confusing errors, like (and I'm paraphrasing here, because I stopped logging my diag output after a while):

FATAL ERRORS:
This version of v490/890 servers only support Ultra IV Processors
CPU's Online:
cpu #0 - Ultra IV 1500
cpu #2 - Ultra IV 1500

What?? That seemed contradictory to me. So we did what isn't generally a good idea (unless your machine appears to be in a state of complete ruination anyway) and pulled the plug, let it idle and powered it back on with the diagnostic keyswitch set. This time it gave us a little more information, and - lo and behold - in between the thousands of diagnostic messages (in between the FATAL ERRORS and the "slave_spin" errors) this line popped up:

OBP/Flash version 4.16.4 does not support part number ##### (Which happened to be the part number of both of our CPU boards).

This was great news! But how to fix it? Of course, replacing the centerplane (which, if you've ever done it - or even watched it being done - understand that it can be a painstaking and extended process) would fix the problem. On the v490 server, the OBP resides on the centerplane, so that was one option (If we'd have followed Sun's advice, of course, we would have already gone through replacing both CPU boards and, possibly, the motherboard before getting to that point!)

Our system OBP/Flash version was 4.16.4, and for the 1500 CPU - Ultra IV CPU boards, we needed to be up to OBP/Flash version 4.18.1. Clearly the CPU boards had been put in the v490 without regard to whether or not they were actually compatible ;)

Our next step was to take an old CPU board and replace the two new ones with it (just to test) and, magically, the machine booted perfectly. None of the system components listed were in a state of failure, or on their way to failing. The 1350 CPU board we put in only required OBP/Flash version 4.15.6 to be supported, and our centerplane OBP exceeded that level.

Our options boiled down to, as we saw it then, installing the OS on disk while we had the one 1350 CPU board installed, downloading the latest OBP/Flash and installing it, and then shutting down and booting up with the two new 1500 CPU Ultra IV boards (While this was a perfectly workable solution, it seemed like there must be a faster way to do it). Net booting was also an option, but that would require modifying our net boot server and might also cause other unforeseen complicatons. We also didn't want to have to have Sun replace the centerplane, as this wasn't any more guaranteed to work than our system-install method.

We eventually ended up bringing a Sun FE on site and got the surprise of our lives (or at least our present days ;) Luckily, Sun FE's have a CD/DVD (So far as I know, it's been around for about a year and is only available to Sun personnel) called SUE (which stands for Sun Utility Environment - or something like that - I was sneaking peaks). This is a tool that's time came a long while ago. With it, the FE was able to boot us to the ok> prompt (using the 1350 CPU board) and run the OBP/Flash upgrade directly from CD!

That's stretching the truth somewhat - SUE actually creates a mini-boot environment in on-board memory and sets up a temporary alias so that you can reboot and upgrade the OBP/Flash. So, instead of having to install the OS, boot the machine into network mode, download the latest OBP/Flash and then reboot with the new flash file, like so (somewhat abbreviated):

init 0
ok> boot disk /flash-update-v490 <--- or whatever the OBP/Flash upgrade file was called.

We were able to update the OBP/Flash by just booting off of the SUE CD, picking the OBP/Flash upgrade from the list available on the CD and letting it do a :

reboot -- cdrom /flash-update-v490

That was a "huge" time savings! Hopefully, Sun will make this CD, or a CD utility like it, available to users (or, at least, contract holders) in the near future.

So, as it turned out, that absolutely horrible boot problem wasn't really all that bad. Rather than replacing every single piece of hardware on the system until we found the one that was bad, all we had to do was upgrade the OBP/Flash on the system!

Sometimes the most complicated problems have the simplest solutions :)

Best wishes,

, Mike

linux unix internet technology