Emergency Booting RedHat Linux With USB

Hey there,

This post is for any Linux admin out there, who uses RedHat (or any similar distro, really - the differences are generally minor) and works in a shop outfitted with the latest and greatest hardware to "drive the network." I'm not sure what that expression means, but I heard it in a meeting once and it seemed vaguely motivating ;) It's also the first post on using grub since our post on using grub to change the root password.

If you do use RedHat Linux and run it on hardware that's new or "newer," you probably have a couple of servers (or, possibly, even regular PC's) running RedHat that don't have those good old-fashioned PS/2 keyboards and mice (mouses ?? I'll puzzle over that for my remaining years). You may even, in some instances, find yourself in a situation where you're booting a server from a USB drive. You won't see that too often (especially not in enterprise systems), but you'll catch it here and there in development environments with bizarre quadruple-boot configurations. You may not, technically, support them, but (since you're the sys admin) you support them ;)

Basically, today, we're going to take a look at a way you can protect yourself against disaster in the former situation (No PS/2 Keyboard or mouse) and assume that your server is running RedHat Linux and only has USB keyboard and mouse connectors (Really, you don't need the mouse. That's for the GUI, and the GUI shouldn't have anything to do with an "emergency" recovery. If your GUI works, things can't be all that bad, and, if it doesn't, you won't be using it ;). We'll assume also, that you'll need them to work properly when you boot from a disaster situation. This, as most RedHat Linux users and/or admin's can attest to, is an issue directly related to the default OS setup. The issue is that the USB drivers generally (or, by default) get loaded up after the system has begun booting. Since the USB drivers are treated as ancillary, if your machine is completely screwed and you need to do some minimal CLI work (just interacting with "grub" at a high level), having a USB keyboard as your only option means you're screwed, too, unless you get creative and connect to the box via ALOM or Serial or can jerry-rig some other form of TTY connection to emulate keyboard interaction (You'll discover amazing things you never knew you knew when an important box goes down and the boss is building up to a Grand Mal seizure while he breathes down your neck ;)

Of course, the easiest thing to do (in almost any case, for any problem) is to use up that ounce of prevention (If you encounter resistance, you can always spin this as something that will "save the company a pound of cure" ;). And the easiest way to do that in RedHat Linux is to just create an alternate boot image that includes USB drivers (you saw this coming from a mile away, I know). And RedHat (as with most other distro's) makes this very easy to do. Given the version of RedHat Linux you're running, the switches/arguments you have to pass to the various commands may be slightly different, so I'm including multiple options in some cases where only a few (or one) are actually relevant. Check the man pages on your system to see which one works for you and/or if that option even exists. Who knows, you may have more options on your version than I do on mine, which could, potentially, be even better :)

The first thing to do, would be to figure out what USB drivers you have loaded on your system. A simple find command like this, should give you a general idea (use "lsmod" and grep that output if you want to do things "the right way," but you may miss a few things ;)

host # uname -r
2.4.21-37.ELsmp
host # find /lib/modules/`uname -r`/kernel/drivers/usb -type f -name "*usb*"
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/usb-ohci.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/usb-uhci.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/dabusb.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/serial/ir-usb.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/serial/kl5kusb105.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/serial/usbserial.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/hpusbscsi.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/storage/usb-storage.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/usb-midi.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/usbcore.o
/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/usbnet.o

and this one, just because it's always nice to have (details on down the page):

/lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/ehci-hcd.o

The second thing to do would be to cozy up your "mkinitrd" command and get comfortable with it ;) On RHEL4 (RedHat Enterprise Linux Release 4), you can get away with the following command line to create a new bootable USB-friendly kernel boot image (Note that the release in these examples is RHEL3):

host # mkinitrd --with=ehci-hcd --with=uhci-hcd /boot/newinitrd-`uname -r`.img `uname -r`

If you're using an older, or different, version of RedHat, check out the options you have available to your "mkinitrd" command. Generally, you'll find at least one of these:

--with-usb
--without-usb (completely contrary to this post, but, included for completeness for those folks who want to make sure they boot with NO USB support ;)
--with=MODULE_NAME
--preload=MODULE_NAME

For instance, this would work on most servers:

host # mkinitrd --preload=ehci-hcd --preload=usb-storage --preload=scsi_mod --preload=sd_mod /boot/usbinitrd.img `uname -r`

From my little find investigation, I found that this would probably work well for my particular server:

host # mkinitrd --preload=ehci-hcd --preload=usb-storage --preload=usb-uhci --preload=usb-ohci --preload=scsi_mod --preload=sd_mod /boot/usbinitrd.img `uname -r`

And, then we'd add a new entry in /etc/grub.conf and give it a test the next time we can sneak in a clean boot up (For those last few examples of "mkinitrd" entries, of course, I'd be doing them all with `uname -r` instead of the actual value of that command, but you don't want to put those backticks in your /etc/grub.conf). A new entry, considering our command above, might look something like this:

title USB RHEL3 (2.6.9-1.667smp)
root (hd0,0)
kernel /vmlinuz-2.6.9-1.667smp
initrd /usbinitrd.img 2.6.9-1.667smp

...basically, if you're just created a different "initrd" image for your same kernel, just copy those lines, change the title slightly and alter the line that begins with "initrd" to point to the file you created with "mkinitrd." Simple :)

and, just in case you're curious about any of the modules we've suggested possibly adding to your new USB boot image, you can usually find out as much about them as you want to know using the modinfo command, like so (I'm leaving out the usb-storage, etc, ones since their name's pretty much say it all :)

host # modinfo ehci-hcd
filename:    /lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/ehci-hcd.o
description: "2003-Jan-22 USB 2.0 'Enhanced' Host Controller (EHCI) Driver"
author:      "David Brownell"
license:     "GPL"
parm:        log2_irq_thresh int, description "log2 IRQ latency, 1-64 microframes"

host # modinfo usb-uhci
filename:    /lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/usb-uhci.o
description: "USB Universal Host Controller Interface driver"
author:      "Georg Acher, Deti Fliegl, Thomas Sailer, Roman Weissgaerber"
license:     "GPL"

host # modinfo usb-ohci
filename:    /lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/usb-ohci.o
description: "USB OHCI Host Controller Driver"
author:      "Roman Weissgaerber <weissg@vienna.at>, David Brownell"
license:     "GPL"

Here's hoping you either never have to use this, or that you have it ready to roll if you ever do :)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Using Grub To Change RedHat Linux's Root Password

Hey There,

I'm back from the dead and no longer cursing Vista ( I've always thought it was strange that most people, including myself, feel disappointment or sorrow when they lose something they didn't have in the first place. The opposite seems perfectly logical. I feel very relieved that I no longer have something that I never had before ;)

Today, we're going to take a look at a quick way to get back your root password (assuming its yours) if, for some reason, you get locked out of your machine. It happens... more than I'd care to admit ;) The course of action I see followed most often is booting from CD, temporarily mounting the physical root drive, and editing /etc/passwd. This is a time-tested solution, and works on pretty much any version of Linux or Unix I've worked with, but I think this way is more fun (and slightly less dangerous). Plus, it saves you a little time (not a lot; just a little). We haven't taken a look at grub since our post on recovering failed raid disks, so I guess this post is about due.

The trick in question has only been tested on RedHat Linux ES, so I can't speak for whether it works on, say, CentOS or Fedora (although I imagine it would work on any system that uses grub for boot loading). Basically, what we're going to do is use grub's boot options to allow us to obtain root access. And, if your machine is properly secured (the /boot directory in particular), you shouldn't be able to edit /etc/grub.conf as a normal user, so physical access (or console/ALOM-type access) to the machine in question is required. It's a pretty simple procedure and goes something like the following:

1. Login to the console on the machine and type "reboot" or "shutdown -r," etc, if you have an account with privileges to initiate such an action. If you don't have an account with suitable privilege, try control-alt-delete (and power off) or hard power-down your machine (you may need to fsck later, but that's a given), take a deep breath and count to 11 ;)

2. Power on the server and wait for the grub boot screen to come up. You'll may not need the GUI for this to work, but it's the only way I've done it. When the grub boot menu comes up, hit the up or down arrow key at least once to stop the automatic boot countdown timer. If you have multiple boot options, choose the one you know (or believe) is the one currently in use (actually, this shouldn't matter, but loading up an older kernel might cause issues) and press the "e" key to enter edit mode.

3. After you enter edit mode, you'll be presented with a few lines of text (dependent on how you have your grub.conf populated). Using your arrow key again, navigate to the line that starts with "kernel." Press the "e" key again, and your cursor should show up at the end of the "kernel" line (if it doesn't, you can move it to where you need it by using the left and/or right arrow keys as necessary).

4. Now that you're in edit mode, and your cursor is in the correct position, type a "space" character followed by "single." So if your boot command line was:

kernel /boot/vmlinuz-2.6.9-34.ELsmp ro root=/dev/sda1
it would now be:
kernel /boot/vmlinuz-2.6.9-34.ELsmp ro root=/dev/sda1 single

5. Now type "b" to continue the boot process and you'll be dumped into a limited shell, as root, passwordlessly. Sometimes this has seemed not to work for me if I changed my edit-focus to a line other than the "kernel" line before typing "b", but that could just be superstition on my part. Thankfully, I don't have to do this all that often :)

6. The rest is gravy. You're root, so all you have to do is type "passwd", set the root password to whatever you like and reboot using your preferred method (reboot, shutdown -r, init 6, whatever works, etc). Since you're in a single-user shell, you can also instantiate a reboot by just typing "exit."
And you should be all set. Now you no longer have an excuse to avoid fixing problems on that machine (the downside ;)
Cheers,

, Mike

---

08/06/2008 - Thanks for this Additional Useful Information From zcat:

On many distros the 'single' or 'rescue' boot will still ask for a
password. You can get around this by starting linux without starting
initd, just launch a shell instead; and it's blindingly fast.

'e' to edit the boot entry, select the kernel line and press 'e'
again, then type "init=/bin/bash", enter, press 'b' to boot it. You
end up at a root prompt with / mounted read-only. (depending on the
distro, you might need /bin/sh instead)

# mount / -o remount,rw
# passwd
<change your root password here>
# mount / -o remount,ro
<three-finger salute or hit the reset button>


It's also useful for fixing up boot problems, if you're silly enough
to have put commands in various init scripts that don't actually exit
or daemonize...

---

Thanks for these comments from Laurent regarding an alternate way to get to single user:

Nice to see that method that I was used to apply. Especially with some servers that have been hardened with password aging implementation. And when it is stable you don't need to log on for more than 60 days sometimes....

You could also add that grub can (should?) be password protected.

Cheers

Laurent

UnConfusing The Issue Of Disabling Root On Linux or Unix

Hey There,

Loosely following up on a few previous posts regarding securing Linux and Unix and even more fun with Unix and Linux security, today we're going to look at something vaguely security-related, and also vaguely scary ;)

It has to do with the root account (or user id 0) and security hardening. I've got nothing against either, so this won't be an insane diatribe, but (especially, no offense to Ubuntu) with more Linux distro's coming straight out-of-the-box with root disabled, I see the issue of security and the root user get a little confused. A lot ;)

There are a numbers of things at issue here, and they're all loosely tied-together by the common thread of an earnest striving for better total-system security. Today, I'm just feeling like clearing some of them up to the best of my ability, before they make me nuts.

1. Having a disabled root account and removing a root account are two entirely separate things. As I mentioned above, Ubuntu comes with the root account disabled (basically, you can do this on any OS by running "passwd -l root" as the root user, hopefully setting a regular user up with a sudo rule so they can su to root if they need to later, and then exiting out of the root account).

This is perfectly fine. It upsets quite a few people who don't know what to make of it, initially, but it's not a bad idea. The root user's password is locked, so no one can su to it without sudo (or some other privilege-escalation vehicle), but the account still exists and runs as normal.

It goes without saying, at this point in the post, that actually removing root (userdel root will fail - probably too late - but you can accomplish almost the same thing by editing /etc/passwd, /etc/group, /etc/security/*, etc, as root and logging out) is a bad idea. The problem then becomes that, not only can no one su to root, it's actually gone, and everything that used to run as uid 0 will begin to exhibit bizarre behaviour. Without intervention you can expect to lose your system fairly quickly and have the boot process crash on you as soon as it gets to running init, at best. By default, on almost every system I know, init runs as root, spawning "init" (process 1 - the mother of them all). If that can't start, what else can? I've heard tell of systems that will revert to the nobody account to run this, but I've never seen it happen and I can't imagine how it would work...

2. There are ways to run your system without having a root user, but almost all of them are not worth the effort. As a blanket statement, if you replace the root account with another account with uid 0 (Except in such instances as FreeBSD, where they've gone through the trouble of actually coding out a "toor" user as a root variant), you're opening up the door to any number of problems you may have not foreseen. Replacing uid 0 with uid 100034 (pick a number) would be even more probable to cause stress-related fatalities ;) Liken it to the old-style firewall security method of "block what you don't want and let everything else in." That philosophy was eventually replaced by the more sensible "block everything and only let in what you want" method. If you mess with root, or uid 0, you may have a huge list of things you've accounted for, but you're almost guaranteed to stung by something you'd (and anybody else probably would have) forgotten or never even considered.

3. Changing init in Linux using the boot loader (with arguments like init=/bin/bash) doesn't actually address the issue with running init without root, or a user with uid 0. This trick can come in very handy when, for instance, you want to subvert LILO or Grub security and boot into bash. Of course, / will be mounted read-only, but it's only a minor inconvenience to run "mount -o rw,remount" on it to begin editing files you shouldn't be editing. Of course, you'd only end up doing this if you were in it deep and had to manually edit your /etc/shadow file or something, right? Yes. That's the ethical answer :) As far as hardening security goes, I'm not sure how this misconception got started.

4. Having more than one user with uid 0, or more than one group with gid 0, is not only insane, it's lazy-insane ;) Seriously, I've seen this proposed as an alternative to locking root's password and using sudo in order for certain users to access root to make changes as the superuser. There are a few issues with this sort of convenience, none of which are good (except, that 5 or 6 guys on your system have uid 0 and root privilege which is great! ...I mean bad). Aside from the simple "plus 1" factor of risk-of-accidental-keystroke-downing-your-machine incidents per user with said privilege, having multiple users with the same uid and/or gid can cause erratic behaviour with programs that do name-to-uid and/or uid-to-name mapping (like the login command).

All of this comes back to locking up the root user password. This is (assuming you have to pick from that and the 4 options above) the most sane choice. I'm not gonna shine you on about how I've never used root as my main account for years on end and haven't ever accidentally deleted the /dev filesystem (well, at least as much as it would let me before there was no device left to do the deleting ;) I've done lots of crazy and irresponsible things in my career and, according to some folks (like the U.S. military and heavy-hitting banking and/or securities-trading companies) I still do, no matter how good I try to be ;)

But I think, if you must tighten the screws on the root account, then look to Ubuntu's example and have the default setup with a locked root password and a user account that can "sudo su" to it. There's no need to be any more extreme or creative and, if you decide that you must and haven't completely profiled your operating system's kernel and shell code before implementation to ensure safety from latent dependency bugs (or somehow managed to figure out a way to defend against an army of guys/gals with root privilege easily and with guaranteed low-downtime), may luck be with you. And, if it's not, may you have an understanding superior.

Hopefully, this hasn't confused the issue even further. Whatever the issue was... ;)

Cheers,

, Mike

Recovering failed RAID disks on Linux

Hey there once more,

As I mentioned in yesterday's post regarding creating RAID disk sets we'll be looking at how to deal with RAID disk failures on Linux in today's post. Once we're done here, I think we can call it a wrap on this subject for a while (Are those sighs of relief I'm hearing? ;) At least until we come up with some good shell scripts to expedite some of this stuff.

Disk failure, on the hardware side, is too broad to cover in any great detail here, but the following basic steps should be followed (of course, as noted, your setup may require otherwise). The scenario here is that one of your disks has just gone "bad." It's beyond recovery.

1. If the disk is hot-swappable, simply remove it. If it isn't, you'll need to schedule downtime and remove the disk then. If the failed disk is your boot disk, you'll already have your downtime ;)

2. Replace the failed disk and restart your machine (and skip to step 6) if your failed disk isn't the boot disk.

3. If your failed disk is the boot disk, you'll next want to boot off of CD (We're using RedHat Linux AS), mount the root filesystem under a temporary mountpoint, and do the following:

host # mkdir /tmp/recovery
host # mount /dev/hda0 /tmp/recovery
host # chroot /tmp/recovery
host # grub --batch (This may take a while as grub probes and tries to guess where all of your drives are)

4. Once grub is finished probing, do the following at the "grub>" prompt:

grub> root (hd0,0)
root (hd0,0)
Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd0)
setup (hd0)
...
Running "install /grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/grub/stage2
/grub/grub.conf"... succeeded
grub> exit

5. Now take a second and verify that all is well while still running off of the CD, like so:

host # cat /boot/grub/device.map:
(fd0) /dev/fd0
(hd0) /dev/hda

host # df -k (should show the /dev/md0 mount)
host # umount /tmp/recovery
host # reboot (Be sure to set the grub device map for hd0 to /dev/hdc if /dev/hda has gone bye-bye)

6. Now that you have the disk physically replaced and you've booted back up, check the content of /proc/mdstat, like so:

host # cat /proc/mdstat

Personalities : [raid1]
read_ahead 1024 sectors
...
md0 : active raid1 hda1[0]
     307328 blocks [2/1] [U_]
...
unused devices: <none>

Again, the [2/1] listing indicates that one part of the mirror is not active in this two-way mirror set!

7. If this has been driving you crazy from the first post, we're now going to recreate the mirror so it's not incomplete (apologies for any stress-related reading injuries. Sometimes I go a long way to make a point ;). The first thing we'll do is repartition the disk, again, with fdisk (instructions in our previous post regarding creating RAID disk sets ) and we should end up with our partition table looking exactly the same:

host # fdisk -l /dev/hda (The partition table should look almost identical for /dev/hdc)

Disk /dev/hda: 16 heads, 63 sectors, 77520 cylinders 
Units = cylinders of 1008 * 512 bytes 

   Device Boot    Start       End    Blocks   Id  System 
/dev/hda1   *         1      6095   3071848+  fd  Linux raid autodetect 
/dev/hda2          6096     67047  30719808   fd  Linux raid autodetect 
/dev/hda3         67048     73142   3071880   fd  Linux raid autodetect
/dev/hda4         75175     77206   1024096+  82  Linux swap

8. As a matter of course, just to be sure, check /etc/raidtab to see what devices need to be mirrored with what devices:

host # cat /etc/raidtab

raiddev             /dev/md0
raid-level                  1
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
   device          /dev/hda1
   raid-disk     0
   device          /dev/hdc1
   raid-disk     1
...

9. Now, we just need to add back all the partitions we lost:

host # raidhotadd -a /dev/md0 /dev/hda1
...

And keep tabs on the process by checking on /proc/mdstat:

host # cat /proc/mdstat

Personalities : [raid1]
read_ahead 1024 sectors
...
md0 : active raid1 hda1[2] hdc1[1] 
      30716160 blocks [2/1] [_U] 
      [===========>..........]  recovery = 45.9% (34790000/61432320) finish=98.7min speed=8452K/sec
unused devices: <none>

Once the RAID sync is done, you should be good to go! I'd reboot one more time, just to be sure. Especially if you were forced to take downtime in the first place :P

Take care,

, Mike

linux unix internet technology