Emergency Booting RedHat Linux With USB

Hey there,

This post is for any Linux admin out there, who uses RedHat (or any similar distro, really - the differences are generally minor) and works in a shop outfitted with the latest and greatest hardware to "drive the network." I'm not sure what that expression means, but I heard it in a meeting once and it seemed vaguely motivating ;) It's also the first post on using grub since our post on using grub to change the root password.

If you do use RedHat Linux and run it on hardware that's new or "newer," you probably have a couple of servers (or, possibly, even regular PC's) running RedHat that don't have those good old-fashioned PS/2 keyboards and mice (mouses ?? I'll puzzle over that for my remaining years). You may even, in some instances, find yourself in a situation where you're booting a server from a USB drive. You won't see that too often (especially not in enterprise systems), but you'll catch it here and there in development environments with bizarre quadruple-boot configurations. You may not, technically, support them, but (since you're the sys admin) you support them ;)

Basically, today, we're going to take a look at a way you can protect yourself against disaster in the former situation (No PS/2 Keyboard or mouse) and assume that your server is running RedHat Linux and only has USB keyboard and mouse connectors (Really, you don't need the mouse. That's for the GUI, and the GUI shouldn't have anything to do with an "emergency" recovery. If your GUI works, things can't be all that bad, and, if it doesn't, you won't be using it ;). We'll assume also, that you'll need them to work properly when you boot from a disaster situation. This, as most RedHat Linux users and/or admin's can attest to, is an issue directly related to the default OS setup. The issue is that the USB drivers generally (or, by default) get loaded up after the system has begun booting. Since the USB drivers are treated as ancillary, if your machine is completely screwed and you need to do some minimal CLI work (just interacting with "grub" at a high level), having a USB keyboard as your only option means you're screwed, too, unless you get creative and connect to the box via ALOM or Serial or can jerry-rig some other form of TTY connection to emulate keyboard interaction (You'll discover amazing things you never knew you knew when an important box goes down and the boss is building up to a Grand Mal seizure while he breathes down your neck ;)

Of course, the easiest thing to do (in almost any case, for any problem) is to use up that ounce of prevention (If you encounter resistance, you can always spin this as something that will "save the company a pound of cure" ;). And the easiest way to do that in RedHat Linux is to just create an alternate boot image that includes USB drivers (you saw this coming from a mile away, I know). And RedHat (as with most other distro's) makes this very easy to do. Given the version of RedHat Linux you're running, the switches/arguments you have to pass to the various commands may be slightly different, so I'm including multiple options in some cases where only a few (or one) are actually relevant. Check the man pages on your system to see which one works for you and/or if that option even exists. Who knows, you may have more options on your version than I do on mine, which could, potentially, be even better :)

The first thing to do, would be to figure out what USB drivers you have loaded on your system. A simple find command like this, should give you a general idea (use "lsmod" and grep that output if you want to do things "the right way," but you may miss a few things ;)

host # uname -r
2.4.21-37.ELsmp
host # find /lib/modules/`uname -r`/kernel/drivers/usb -type f -name "*usb*"
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/usb-ohci.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/usb-uhci.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/dabusb.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/serial/ir-usb.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/serial/kl5kusb105.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/serial/usbserial.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/hpusbscsi.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/storage/usb-storage.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/usb-midi.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/usbcore.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/usbnet.o

and this one, just because it's always nice to have (details on down the page):

/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/ehci-hcd.o

The second thing to do would be to cozy up your "mkinitrd" command and get comfortable with it ;) On RHEL4 (RedHat Enterprise Linux Release 4), you can get away with the following command line to create a new bootable USB-friendly kernel boot image (Note that the release in these examples is RHEL3):

host # mkinitrd --with=ehci-hcd --with=uhci-hcd /boot/newinitrd-`uname -r`.img `uname -r`

If you're using an older, or different, version of RedHat, check out the options you have available to your "mkinitrd" command. Generally, you'll find at least one of these:

--with-usb
--without-usb (completely contrary to this post, but, included for completeness for those folks who want to make sure they boot with NO USB support ;)
--with=MODULE_NAME
--preload=MODULE_NAME

For instance, this would work on most servers:

host # mkinitrd --preload=ehci-hcd --preload=usb-storage --preload=scsi_mod --preload=sd_mod /boot/usbinitrd.img `uname -r`

From my little find investigation, I found that this would probably work well for my particular server:

host # mkinitrd --preload=ehci-hcd --preload=usb-storage --preload=usb-uhci --preload=usb-ohci --preload=scsi_mod --preload=sd_mod /boot/usbinitrd.img `uname -r`

And, then we'd add a new entry in /etc/grub.conf and give it a test the next time we can sneak in a clean boot up (For those last few examples of "mkinitrd" entries, of course, I'd be doing them all with `uname -r` instead of the actual value of that command, but you don't want to put those backticks in your /etc/grub.conf). A new entry, considering our command above, might look something like this:

title USB RHEL3 (2.6.9-1.667smp)
root (hd0,0)
kernel /vmlinuz-2.6.9-1.667smp
initrd /usbinitrd.img 2.6.9-1.667smp

...basically, if you're just created a different "initrd" image for your same kernel, just copy those lines, change the title slightly and alter the line that begins with "initrd" to point to the file you created with "mkinitrd." Simple :)

and, just in case you're curious about any of the modules we've suggested possibly adding to your new USB boot image, you can usually find out as much about them as you want to know using the modinfo command, like so (I'm leaving out the usb-storage, etc, ones since their name's pretty much say it all :)

host # modinfo ehci-hcd
filename:    /lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/ehci-hcd.o
description: "2003-Jan-22 USB 2.0 'Enhanced' Host Controller (EHCI) Driver"
author:      "David Brownell"
license:     "GPL"
parm:        log2_irq_thresh int, description "log2 IRQ latency, 1-64 microframes"

host # modinfo usb-uhci
filename:    /lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/usb-uhci.o
description: "USB Universal Host Controller Interface driver"
author:      "Georg Acher, Deti Fliegl, Thomas Sailer, Roman Weissgaerber"
license:     "GPL"

host # modinfo usb-ohci
filename:    /lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/usb-ohci.o
description: "USB OHCI Host Controller Driver"
author:      "Roman Weissgaerber <weissg@vienna.at>, David Brownell"
license:     "GPL"

Here's hoping you either never have to use this, or that you have it ready to roll if you ever do :)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Shell Script To Monitor Disk Usage On Linux and Unix

Hey There,

Today, we're going to take a look at a simple shell script to monitor disk space usage. It's been quite a while since we've touch on that, going back to a post from last November regarding finding space hogs on overlay mounts. The script has been kept simple (basically checking every partition for one fixed percentage full) to highlight other features.

The main intent here was to set up a monitor that would be able to handle a variety of Linux and Unix Operating Systems (all dependant on the "uname -s" output from that system) and focus on that area distinctly. We've limited our initial list to HP-UX, Solaris, SCO and OpenBSD.

In this case we're using a simple case statement to enumerate through the four *nix's we have listed here. Obviously, we could easily add more operating systems, and their variations of the "df" command, to our list and, if it ever got too big, either roll them into an array or simplify the script so that more OS's would fall under the same umbrella.

Also, notice that we're stepping through parsing of the df output more tediously than is actually necessary. For instance, the creation of the df output could be parsed with sed all in one fell swoop. Again, although it is generally considered best practice to compact your script/code, our hope here was that this would be easy to follow for as many people as possible. Some folks learn better by tackling the tough-stuff and working back to basics and some of us learn better by starting with the basics and putting them all together to create the tough-stuff. It's a long and convoluted statement of philosophy, to be sure, but fairly descriptive of what we actually mean ;)

If you prefer, on the line where we parse the TABLE file and trim it with sed, you can take out this part (or add to it) as it was placed in there as an example of how to ignore a specific partition (/usr):

-e '/usr$/d'

So, the line:

sed -e '1d' -e '/usr$/d' ${BASEDIR}/TABLE >> ${BASEDIR}/TABLE2

could be changed to:

sed -e '1d' ${BASEDIR}/TABLE >> ${BASEDIR}/TABLE2

which could then be simplified even further (since you don't need to use -e, even though it's okay to, if you only have one instruction to pass to sed) to become this:

sed '1d' ${BASEDIR}/TABLE >> ${BASEDIR}/TABLE2

And the entire script could be thusly compacted and streamlined, etc.
In any event, I hope this can be of some help (or, at least, an inspiration to reach higher ;) to you!

Best wishes,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/ksh

#
# dfvk.sh - Check partition % full
# across multiple OS'
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

trap 'rm -f ${BASEDIR}/TABLE ${BASEDIR}/TABLE2 ${BASEDIR}/MAILER;exit 1' 1 2 3 15

#BASEDIR="/tmp"
BASEDIR="tmp"
LOGHOST=`hostname`
MAILEES="you@yourhost.com"
LIMIT=90
OSVERSION=`uname -s`

case $OSVERSION in

        SunOS | OpenBSD )
                df -k >> ${BASEDIR}/TABLE
                ;;
        HP-UX )
                bdf >> ${BASEDIR}/TABLE
                ;;
        * )
                df -v >> ${BASEDIR}/TABLE
                ;;

esac

echo "$LOGHOST is getting ready to bother you!" >> ${BASEDIR}/MAILER
echo >> ${BASEDIR}/MAILER

sed -e '1d' -e '/usr$/d' ${BASEDIR}/TABLE >> ${BASEDIR}/TABLE2
sed 's/%//g' ${BASEDIR}/TABLE2 > ${BASEDIR}/TABLE
cat ${BASEDIR}/TABLE |while read ONE TWO THREE FOUR FIVE SIX
do

        case $OSVERSION in

                HP-UX | SunOS | OpenBSD )
                                if [ $FIVE -gt $LIMIT ]
                                then
                                        print "$SIX partition at ${FIVE}% capacity" >> ${BASEDIR}/MAILER
                                else
                                        continue
                                fi
                        break;;
                * )
                                if [ $SIX -gt $LIMIT ]
                                then
                                        print "$ONE partition at ${SIX}% capacity" >> ${BASEDIR}/MAILER
                                else
                                        continue
                                fi
                        break;;

        esac

done

MAILCOUNT=`cat ${BASEDIR}/MAILER |wc -l`
if [ $MAILCOUNT -gt 2 ]
then
        cat ${BASEDIR}/MAILER |mailx -s "$LOGHOST : Potential Paging Threat!" $MAILEES
fi

rm -f ${BASEDIR}/TABLE ${BASEDIR}/TABLE2 ${BASEDIR}/MAILER

, Mike

Using Linux strace To Debug Application Issues

Today's post is a little bit of a walkthrough of using RedHat Linux strace to debug (and find the root cause of ) a system issue and a little bit of caution about how much information you should really share with application project managers if you don't want to be stuck supporting a hacked solution for longer than anyone should. As far as job security goes, it can't hurt, but certain situations really do require that folks upgrade their software to the vendor supported version, no matter how creative your solution ends up being.

Note: If you're looking for an in-depth examination of strace, this post isn't it. Not to turn any readers off, as we'll certainly be exploring that in a future post, but this is more of a walkthrough of a problem resolution involving strace rather than a dissertation on the command itself.

In this particular instance, we were working with a product (heretofore referred to as the Product or Product) that, to my knowledge, had just begun having "issues." This generally means to me that somebody did something they weren't supposed to or were trying to do something they weren't supposed to. I'm cynical, but I try to have the common courtesy to keep it to myself ;)

The Product was a client/server application running on the Java platform that had suddenly begun dropping connections from the application server to the backend database. I did a quick check of the /proc/****/status file (and, of course, gave a quick nod to "top" and "netstat -an"), confirming that they were dropping quite a few, like so:

host # cat /proc/14653/status | grep Threads
Threads: 311

host # top <-- Truncated to just show the top process, which was the "failing" one.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14653 user 16 0 1634m 1.1g 40m S 120 14.1 296:46.54 java

I found the "total time used" (listed under the TIME+ column) somewhat alarming since the process had only been up for about a half an hour! It couldn't possibly be correct.

The next thing I did, since I like to go to the guts when I get brought in on an emergency issue regarding a Product I've never used before, was to shut it down and restart it with strace; teeing the information off to a file in /tmp (since I like to watch the characters fly by), like so:

host # strace -vF ./product arguments 2>&1|tee /tmp/OUTPUT

Note that this could have also been accomplished by running the process, getting it's pid from the ps tables and adding the -p PID flag to the strace command in order to attach to the process after starting it. I prefer to start the process with strace, if I can, in case something critical happens during the initial load-up.

Long story short (It is Sunday - Our day of rest - after all ;), I found that the Product process was dumping tons of FUTEX_WAIT errors. At this point, I checked the version of the OS with uname for the only part I really cared about at that point:

x86-64

The machine was running RedHat Advanced Server 5 - 64 bit. The FUTEX_WAIT errors have to do with the implementation of Fast Userspace Mutex locking implemented in the stable version of the RedHat Linux kernel 2.6 and up. This raised a red flag, and caused me to ask the inevitable question: When did this start happening?

As it turns out, the Product had previously been hosted on another machine running RedHat Advanced Server 4, 32 bit. This made perfect sense, since the Mutex locking mechanism at the kernel level they "used" to run on was completely different, and not compatible, since the Product was built to use the old style Mutex locking mechanisms.

Futile attempts were made to add lame fixes, like:

LD_ASSUME_KERNEL=2.4.whatever (Apologies; I don't remember what it was off-hand since it didn't end up helping at all ;)

to the Product's startup scripts, but this didn't resolve the problem. The "actual" (and correct) resolution would have been to upgrade to the vendor-supported version of the Product. The manufacturer of the Product actually had a more recent version available that was built specifically to address this issue and run on RedHat AS 5 - 64 bit.

And, here's the kicker (as per the cautionary message at the beginning of this post): As some of you may know, a bug in strace (It's actually more of a "feature" since it almost "has" to happen) makes it slow down a process slightly due to all the extra work it has to do to parse all the system calls, opens, reads, writes, errors, library loads, etc.

strace slowed the process down enough that the FUTEX_WAIT errors stopped occurring and the Product began functioning normally again. We ended up adding it to the init scripts after much debate over spending a few bucks to avoid possible total confusion in the future.

In retrospect, I'm glad that I was able to help identify the issue and find the root cause. On the other hand, I wish there was some way I could have avoided identifying the problem without providing a cheap and quick "out" for the end user. I'm sure, in the future, someone else will have to work on this Product (maybe a problem with the version of Java or something more insidious) and they'll be given about as much information as I was. Documentation was laid down, but I don't expect anyone will read it. Sometimes, the solution is as bad as the problem, in a way or two ;)

Enjoy your Sunday :)

, Mike

linux unix internet technology

What A Night - Changing Solaris Machine Names

Hey there,

Some more possibly boring, but definitely useful, trivia about working on Solaris. Just got off an emergency shift re-purposing a whole bunch of boxes.

It's actually really simple work to change a solaris hosts name. Just need to make sure you hit all the bases.

The files you have to edit are the same for Solaris 8/9 and 10, except for one.

All you need to edit, and run, to change your hostname is:

Edit: /etc/hosts - make your main hostname change here. If possible include an entry for your defaultrouter!
/etc/hostname.NIC (e.g. hostname.ce0) - may be more than one of these depending.
/etc/nodename - the main hostname of your server
/etc/inet/net/*/hosts (Solaris 8/9) - just change the hostname in the hosts files.
/etc/inet/ipnodes (Solaris 10) - change the hostname here
/etc/defaultrouter - you can enter an IP here, but I prefer to enter a hostname and list that in /etc/hosts (above)

Then execute:
hostname NewHostname (as root)
uname -S NewHostname (as root)

And reboot if you like. There's always running sys-unconfig, but it takes out every modification you've ever made. It'll even zero out your resolv.conf file (why???)

Anyway - Enjoy your rest - I have to be back at work in a few minutes ;)

, Mike

linux unix internet technology