Tuesday, September 23, 2008

Emergency Booting RedHat Linux With USB

Hey there,

This post is for any Linux admin out there, who uses RedHat (or any similar distro, really - the differences are generally minor) and works in a shop outfitted with the latest and greatest hardware to "drive the network." I'm not sure what that expression means, but I heard it in a meeting once and it seemed vaguely motivating ;) It's also the first post on using grub since our post on using grub to change the root password.

If you do use RedHat Linux and run it on hardware that's new or "newer," you probably have a couple of servers (or, possibly, even regular PC's) running RedHat that don't have those good old-fashioned PS/2 keyboards and mice (mouses ?? I'll puzzle over that for my remaining years). You may even, in some instances, find yourself in a situation where you're booting a server from a USB drive. You won't see that too often (especially not in enterprise systems), but you'll catch it here and there in development environments with bizarre quadruple-boot configurations. You may not, technically, support them, but (since you're the sys admin) you support them ;)

Basically, today, we're going to take a look at a way you can protect yourself against disaster in the former situation (No PS/2 Keyboard or mouse) and assume that your server is running RedHat Linux and only has USB keyboard and mouse connectors (Really, you don't need the mouse. That's for the GUI, and the GUI shouldn't have anything to do with an "emergency" recovery. If your GUI works, things can't be all that bad, and, if it doesn't, you won't be using it ;). We'll assume also, that you'll need them to work properly when you boot from a disaster situation. This, as most RedHat Linux users and/or admin's can attest to, is an issue directly related to the default OS setup. The issue is that the USB drivers generally (or, by default) get loaded up after the system has begun booting. Since the USB drivers are treated as ancillary, if your machine is completely screwed and you need to do some minimal CLI work (just interacting with "grub" at a high level), having a USB keyboard as your only option means you're screwed, too, unless you get creative and connect to the box via ALOM or Serial or can jerry-rig some other form of TTY connection to emulate keyboard interaction (You'll discover amazing things you never knew you knew when an important box goes down and the boss is building up to a Grand Mal seizure while he breathes down your neck ;)

Of course, the easiest thing to do (in almost any case, for any problem) is to use up that ounce of prevention (If you encounter resistance, you can always spin this as something that will "save the company a pound of cure" ;). And the easiest way to do that in RedHat Linux is to just create an alternate boot image that includes USB drivers (you saw this coming from a mile away, I know). And RedHat (as with most other distro's) makes this very easy to do. Given the version of RedHat Linux you're running, the switches/arguments you have to pass to the various commands may be slightly different, so I'm including multiple options in some cases where only a few (or one) are actually relevant. Check the man pages on your system to see which one works for you and/or if that option even exists. Who knows, you may have more options on your version than I do on mine, which could, potentially, be even better :)

The first thing to do, would be to figure out what USB drivers you have loaded on your system. A simple find command like this, should give you a general idea (use "lsmod" and grep that output if you want to do things "the right way," but you may miss a few things ;)

host # uname -r
2.4.21-37.ELsmp
host # find /lib/modules/`uname -r`/kernel/drivers/usb -type f -name "*usb*"
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/usb-ohci.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/usb-uhci.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/dabusb.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/serial/ir-usb.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/serial/kl5kusb105.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/serial/usbserial.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/hpusbscsi.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/storage/usb-storage.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/usb-midi.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/usbcore.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/usbnet.o


and this one, just because it's always nice to have (details on down the page):

/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/ehci-hcd.o

The second thing to do would be to cozy up your "mkinitrd" command and get comfortable with it ;) On RHEL4 (RedHat Enterprise Linux Release 4), you can get away with the following command line to create a new bootable USB-friendly kernel boot image (Note that the release in these examples is RHEL3):

host # mkinitrd --with=ehci-hcd --with=uhci-hcd /boot/newinitrd-`uname -r`.img `uname -r`

If you're using an older, or different, version of RedHat, check out the options you have available to your "mkinitrd" command. Generally, you'll find at least one of these:

--with-usb
--without-usb
(completely contrary to this post, but, included for completeness for those folks who want to make sure they boot with NO USB support ;)
--with=MODULE_NAME
--preload=MODULE_NAME


For instance, this would work on most servers:

host # mkinitrd --preload=ehci-hcd --preload=usb-storage --preload=scsi_mod --preload=sd_mod /boot/usbinitrd.img `uname -r`

From my little find investigation, I found that this would probably work well for my particular server:

host # mkinitrd --preload=ehci-hcd --preload=usb-storage --preload=usb-uhci --preload=usb-ohci --preload=scsi_mod --preload=sd_mod /boot/usbinitrd.img `uname -r`

And, then we'd add a new entry in /etc/grub.conf and give it a test the next time we can sneak in a clean boot up (For those last few examples of "mkinitrd" entries, of course, I'd be doing them all with `uname -r` instead of the actual value of that command, but you don't want to put those backticks in your /etc/grub.conf). A new entry, considering our command above, might look something like this:

title USB RHEL3 (2.6.9-1.667smp)
root (hd0,0)
kernel /vmlinuz-2.6.9-1.667smp
initrd /usbinitrd.img 2.6.9-1.667smp


...basically, if you're just created a different "initrd" image for your same kernel, just copy those lines, change the title slightly and alter the line that begins with "initrd" to point to the file you created with "mkinitrd." Simple :)

and, just in case you're curious about any of the modules we've suggested possibly adding to your new USB boot image, you can usually find out as much about them as you want to know using the modinfo command, like so (I'm leaving out the usb-storage, etc, ones since their name's pretty much say it all :)

host # modinfo ehci-hcd
filename: /lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/ehci-hcd.o
description: "2003-Jan-22 USB 2.0 'Enhanced' Host Controller (EHCI) Driver"
author: "David Brownell"
license: "GPL"
parm: log2_irq_thresh int, description "log2 IRQ latency, 1-64 microframes"

host # modinfo usb-uhci
filename: /lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/usb-uhci.o
description: "USB Universal Host Controller Interface driver"
author: "Georg Acher, Deti Fliegl, Thomas Sailer, Roman Weissgaerber"
license: "GPL"

host # modinfo usb-ohci
filename: /lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/usb-ohci.o
description: "USB OHCI Host Controller Driver"
author: "Roman Weissgaerber <weissg@vienna.at>, David Brownell"
license: "GPL"


Here's hoping you either never have to use this, or that you have it ready to roll if you ever do :)

Cheers,

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.