Recovering failed RAID disks on Linux

Hey there once more,

As I mentioned in yesterday's post regarding creating RAID disk sets we'll be looking at how to deal with RAID disk failures on Linux in today's post. Once we're done here, I think we can call it a wrap on this subject for a while (Are those sighs of relief I'm hearing? ;) At least until we come up with some good shell scripts to expedite some of this stuff.

Disk failure, on the hardware side, is too broad to cover in any great detail here, but the following basic steps should be followed (of course, as noted, your setup may require otherwise). The scenario here is that one of your disks has just gone "bad." It's beyond recovery.

1. If the disk is hot-swappable, simply remove it. If it isn't, you'll need to schedule downtime and remove the disk then. If the failed disk is your boot disk, you'll already have your downtime ;)

2. Replace the failed disk and restart your machine (and skip to step 6) if your failed disk isn't the boot disk.

3. If your failed disk is the boot disk, you'll next want to boot off of CD (We're using RedHat Linux AS), mount the root filesystem under a temporary mountpoint, and do the following:

host # mkdir /tmp/recovery
host # mount /dev/hda0 /tmp/recovery
host # chroot /tmp/recovery
host # grub --batch (This may take a while as grub probes and tries to guess where all of your drives are)

4. Once grub is finished probing, do the following at the "grub>" prompt:

grub> root (hd0,0)
root (hd0,0)
Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd0)
setup (hd0)
...
Running "install /grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/grub/stage2
/grub/grub.conf"... succeeded
grub> exit

5. Now take a second and verify that all is well while still running off of the CD, like so:

host # cat /boot/grub/device.map:
(fd0) /dev/fd0
(hd0) /dev/hda

host # df -k (should show the /dev/md0 mount)
host # umount /tmp/recovery
host # reboot (Be sure to set the grub device map for hd0 to /dev/hdc if /dev/hda has gone bye-bye)

6. Now that you have the disk physically replaced and you've booted back up, check the content of /proc/mdstat, like so:

host # cat /proc/mdstat

Personalities : [raid1]
read_ahead 1024 sectors
...
md0 : active raid1 hda1[0]
     307328 blocks [2/1] [U_]
...
unused devices: <none>

Again, the [2/1] listing indicates that one part of the mirror is not active in this two-way mirror set!

7. If this has been driving you crazy from the first post, we're now going to recreate the mirror so it's not incomplete (apologies for any stress-related reading injuries. Sometimes I go a long way to make a point ;). The first thing we'll do is repartition the disk, again, with fdisk (instructions in our previous post regarding creating RAID disk sets ) and we should end up with our partition table looking exactly the same:

host # fdisk -l /dev/hda (The partition table should look almost identical for /dev/hdc)

Disk /dev/hda: 16 heads, 63 sectors, 77520 cylinders 
Units = cylinders of 1008 * 512 bytes 

   Device Boot    Start       End    Blocks   Id  System 
/dev/hda1   *         1      6095   3071848+  fd  Linux raid autodetect 
/dev/hda2          6096     67047  30719808   fd  Linux raid autodetect 
/dev/hda3         67048     73142   3071880   fd  Linux raid autodetect
/dev/hda4         75175     77206   1024096+  82  Linux swap

8. As a matter of course, just to be sure, check /etc/raidtab to see what devices need to be mirrored with what devices:

host # cat /etc/raidtab

raiddev             /dev/md0
raid-level                  1
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
   device          /dev/hda1
   raid-disk     0
   device          /dev/hdc1
   raid-disk     1
...

9. Now, we just need to add back all the partitions we lost:

host # raidhotadd -a /dev/md0 /dev/hda1
...

And keep tabs on the process by checking on /proc/mdstat:

host # cat /proc/mdstat

Personalities : [raid1]
read_ahead 1024 sectors
...
md0 : active raid1 hda1[2] hdc1[1] 
      30716160 blocks [2/1] [_U] 
      [===========>..........]  recovery = 45.9% (34790000/61432320) finish=98.7min speed=8452K/sec
unused devices: <none>

Once the RAID sync is done, you should be good to go! I'd reboot one more time, just to be sure. Especially if you were forced to take downtime in the first place :P

Take care,

, Mike

linux unix internet technology

Creating RAID disk sets on Linux

Today's post is a somewhat-continuation of yesterday's post on RAID disk monitoring. We looked at monitoring then, and (the reason I call this a somewhat-continuation) now we're going to take a step back and look at setting up those RAID disk sets (a simple mirror set in this example)in the first place ;) Note that all the examples I'm using today are taken from RedHat Linux AS and can be run in bash or any native shell. Scripting some of this out is near impossible, but someday soon I may give it a go.

In any event, once we've covered all this ground, we'll definitely look at how a handy shell script or two can save a whole lot of time on these processes :) So, off we go:

The first thing you'll want to do is to create your partitions on each physical disk, using the fdisk utility. I personally prefer it from the command line, so if you use Disk Druid or any other GUI interface, it should be easy enough to translate this to that so I don't have to over-explain.

The first thing we'll do is to create the disk partitions and assign them to an appropriate RAID device. The steps should roughly be the same as these:

1. Invoke fdisk at that command line.
2. Select "n" to add a new partition (anything from a slice of disk for mirroring two disks, to an entire disk for higher level RAID and mega-storage).
3. Once added, select "t" to change the partition flag and select "Linux Raid Autodetect" (selection "fd" on my console, possibly different on yours).
4. Select "a" to add a bootable flag to the partition, if necessary.
5. Save the new disk layout, and quit, by selecting "w"
6. Repeat as necessary

It's important, since we're doing this from the command line down-and-dirty, that we make sure that all partitions on both disks are "exactly" the same size. Unless your secondary mirror disk is larger, in which case you just need to make sure that you only use partitions larger than the originals to mirror to (It only makes sense. You can't copy a larger filesystem to a smaller one without losing data - The RAID utilities will discourage you from doing so, anyway).

Now, you'll need to put filesystems on all of those partitions. Generally, you should be able to take care of this using mkfs:

host # mkfs -t ext3 /dev/hda1

In the bizarre event that this fails, or you're using an older or different OS than I am, this should also work:

host # mkfs -t ext2 -j /dev/hda1

Now we'll use set up our /etc/raidtab file, with the following (assuming the same for all slices):

host # raiddev /dev/md0
        raid-level      linear
        nr-raid-disks   2
        nr-spare-disks  0
        chunk-size      64
        persistent-superblock 1
        device          /dev/hda1
        raid-disk       0
        device          /dev/hdc1
        raid-disk       1
...

Then we'll initialize the RAID device, like so:

host # mkraid /dev/md0

Once again, per yesterday's post on RAID disk monitoring, we can check on the RAID device's progress by simply doing the following:

host # cat /proc/mdstat

and we'll know instantly if the RAID device is in a good or bad state, and how far along it is if it's still synching. You can use the above command to check status whenever you make a change to your RAID configuration for any valid RAID md devices.

If you ever want, or need, to remove any RAID partitions, you can do that like this (In this example, the hda1 slice is having issues on the md0 device):

host # raidsetfaulty /dev/md0 /dev/hda1
host # raidhotremove /dev/md0 /dev/hda1

Sometimes, as a failsafe (this depends on your situation and what device you're removing) you may want to re-install grub on your disk device, like so:

host # grub-install /dev/hda

And, finally for today, if you do end up having to remove a RAID mirror partition and it can be replaced inline (A whole disk in itself, for instance - possibly just a partition on a disk if bizarre filesystem corruption was the only issue and you've recreated the filesystem), do the following:

1. Add the new RAID devices as before (create, set filesystem type, assign same partition names if possible, etc) and check the status of the RAID groups. [2/2] means that the RAID group is ok, [2/1] means that the RAID group needs to be resynced (you'll note from yesterday that the RAID group we're working on is in a [2/1] state, with only one active mirror. If you noted that this could be, and should be, fixed - as well as the fact that this failure would drop that RAID partition because of this condition - you're sprinting ahead of my tutorial skills ;)

host # cat /proc/mdstat 
Personalities : [raid1] 
read_ahead 1024 sectors 
md0 : active raid1 hda1[0] 
      30716160 blocks [2/1] [U_] 
...
unused devices: <none>

2. Determine your RAID group and mirrors from /etc/raidtab:

host # cat /etc/raidtab 
...
raiddev             /dev/md0 
raid-level                  1 
nr-raid-disks               2 
chunk-size                  64k 
persistent-superblock       1 
nr-spare-disks              0 
    device          /dev/hda1 
    raid-disk     0 
    device          /dev/hdc1 
    raid-disk     1 
...

3. Now, you'll want to resync the RAID mirror sets that got goofed up when we had to replace the RAID parititions:

host # raidhotadd /dev/md0 /dev/hda1

4. Assuming you've lost more than 1 mirror, just repeat the procedure above as many times as necessary.

5. Lastly, assuming all went well, kick back and enjoy the view ;) - Note that Linux will normally queue resyncs and do them one at a time, so don't panic if you only see one out of 5 disk groups being worked on when you check this out!

host # cat /proc/mdstat  
...     
md3 : active raid1 hda1[2] hdc1[1] 
      30716160 blocks [2/1] [_U] 
      [===========>..........]  recovery = 45.9% (34790000/61432320) finish=98.7min speed=8452K/sec 
...      
unused devices: <none>

And that's it for today. In another near future post (probably tomorrow) we'll look at the fun involved with RAID disk failure ;)

Best Wishes,

A Few Linux RAID Disk Monitoring Tips

Hey there,

For today's post, I thought I'd put together a few Linux RAID disk monitoring tips. This list isn't meant to be too specific, but more of a catch-all of stuff you probably have to do over and over again. But sometimes not often enough that it burns into your brain ;)

At the very basic level, you can manage your disk partitions using fdisk, much like in Windows, although completely different (type "m" to get a menu listing of available commands once you bring this little utility up). You can use the "-l" option to just print out the partition menu and not use it in interactive mode:

host # fdisk -l /dev/hda 

Disk /dev/hda: 16 heads, 63 sectors, 77520 cylinders 
Units = cylinders of 1008 * 512 bytes 

   Device Boot    Start       End    Blocks   Id  System 
/dev/hda1   *         1      6095   3071848+  fd  Linux raid autodetect 
/dev/hda2          6096     67047  30719808   fd  Linux raid autodetect 
/dev/hda3         67048     73142   3071880   fd  Linux raid autodetect
/dev/hda4         75175     77206   1024096+  82  Linux swap

If you want to display the status of your RAID devices, much the way metastat does on Solaris, it's much simpler. One of the great things about Linux is the way they've taken the /proc filesystem and turned into a really useful tool.

Entering the following will display the information you need. Note that this is a two way mirror with only one active mirror (I take the examples I can find on our network ;). Ideally, you'd want to have a [2/2] configuration for better HA.

host # cat /proc/mdstat

Personalities : [raid1]
read_ahead 1024 sectors
md2 : active raid1 hda3[0]
     62912 blocks [2/1] [U_]

md1 : active raid1 hda2[0]
     153152 blocks [2/1] [U_]

md0 : active raid1 hda1[0]
     307328 blocks [2/1] [U_]

If you're syncing up your RAID devices and want to check on the progress, just use the same command, like so:

host # cat /proc/mdstat 
Personalities : [raid1] 
read_ahead 1024 sectors 
md2 : active raid1 hdc3[1] 
      1020032 blocks [2/1] [_U] 
      
md0 : active raid1 hdc1[1] 
      3068288 blocks [2/1] [_U]
      
md1 : active raid1 hda2[2] hdc2[1] 
      3068288 blocks [2/1] [_U] 
      [==>..................]  recovery = 13.7% (894298/6136576) finish=10.2min speed=7833K/sec 
unused devices: 

Now, assuming all that is taken care of and you want to check out your RAID configuration, you'll just need to read your RAID configuration table (/etc/raidtab):

host # cat /etc/raidtab

raiddev             /dev/md0
raid-level                  1
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
   device          /dev/hda1
   raid-disk     0
   device          /dev/hdc1
   raid-disk     1
raiddev             /dev/md1
raid-level                  1
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
   device          /dev/hda2
   raid-disk     0
   device          /dev/hdc2
   raid-disk     1
raiddev             /dev/md2
raid-level                  1
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
   device          /dev/hda3
   raid-disk     0
   device          /dev/hdc3
   raid-disk     1

If you want to get more detailed with your analysis, there are (of course) built in commands to enable you to do so. lsraid is one of my favorites :)

To display a short listing of the md0 device, type the following:

# lsraid -A -a /dev/md0
[dev 9, 0] /dev/md0 8F88ACB0.7D06B4C4.FA677344.C3448700 online
[dev 3, 3] /dev/hda1 8F88ACB0.7D06B4C4.FA677344.C3448700 good
[dev 22, 3] /dev/hdc1 8F88ACB0.7D06B4C4.FA677344.C3448700 good

If you'd like to display a short listing of the RAID array that the disk hda1 belongs to (essentially the same output in this case), just type:

# lsraid -A -d /dev/hda1
[dev 9, 0] /dev/md0 8F88ACB0.7D06B4C4.FA677344.C3448700 online
[dev 3, 3] /dev/hda1 8F88ACB0.7D06B4C4.FA677344.C3448700 good
[dev 22, 3] /dev/hdc1 8F88ACB0.7D06B4C4.FA677344.C3448700 good

lsraid also comes in handy if you want to list out any faulty devices. This is something to consider putting in cron, or running regularly, so that you get notified of any failures in a timely fashion. No sense in having RAID's HA setup if you wait for all the components to fail before you replace any of them ;)

For a short listing, go with:

# lsraid -A -f -a /dev/md0
[dev 9, 0] /dev/md0 8F88ACB0.7D06B4C4.FA677344.C3448700 online

If you want, or need, all the gruesome details, lsraid will be happy to oblige. Just invoke it like this:

lsraid -D -l -a /dev/md0 
[dev 3, 3] /dev/hda1: 
        md version              = 0.90.0 
        superblock uuid         = 8F88ACB0.7D06B4C4.FA677344.C3448700 
        md minor number         = 0 
        created                 = 1169715493 (Fri Jan 25 08:58:13 2007) 
        last updated            = 1169933481 (Sun Jan 27 21:31:21 2007) 
        raid level              = 1 
        chunk size              = 64 KB 
        apparent disk size      = 3068288 KB 
        disks in array          = 2 
        required disks          = 2 
        active disks            = 2 
        working disks           = 2 
        failed disks            = 0 
        spare disks             = 0 
        position in disk list   = 0 
        position in md device   = 0 
        state                   = good 

[dev 22, 3] /dev/hdc1: 
...

In a near future post (maybe tomorrow), we'll follow up with some tips on setting up your RAID arrays in Linux and walk through that process together.

Hope this helps you out some in the meantime :)

Cheers,

, Mike

linux unix internet technology