DiskSuite/VolumeManager or Zpool Mirroring On Solaris: Pros and Cons

Hey There,

Today we're going to look at two different ways to mirror disk on Solaris (both free - but distinguished from freeware in that they're distributed for use on Solaris' commercial and (often) proprietary filesystems and OS).

The old way, probably every Solaris Sysadmin knows backward and forward. Using the Solaris DiskSuite set of tools (meta-whathaveyou ;), which was, at one point, changed to Solaris Volume Manager (which introduced some feature enhancements, but not the kind I was expecting. The name Volume Manager has a direct connection in my brain to Veritas and the improvements weren't about coming closer to working seamlessly with that product).

The somewhat-new way (using the zpool command) won't work - to my knowledge - on any OS prior to Solaris 10, but with Solaris 8 and 9 reaching end of life in the not-too-distant future, every Solaris Sysadmin will have some measure of choice.

With that in mind let's take a look at a simple two disk mirror. We'll look at how to create one and review it in terms of ease-of-implementation and cost (insofar as work is considered expensive if it takes a long time... which leaves one to wonder why I'm not comparing the two methods in terms of time ;)

Both setups will assume that you've already installed your operating system, and all required packages, and that the only task before you is to create a mirror of your root disk and have it available for failover (which it should be by default)

The DiskSuite/VolumeManager Way:

1. Since you just installed your OS, you wouldn't need to check if your disks were mirrored. In the event that you're picking up where someone else left off (and it isn't blatantly obvious - I mean "as usual" ;), you can check the status of your mirror using the metastat command:

host # metastat -p

You'll get errors because nothing is set up. Cool :)

2. The first thing you'll want to do is to ensure that both disks have exactly the same partition table. The same-ness has to be "exact," as in down to the cylinder. If you're off even slightly, you could be causing yourself major headaches. Luckily, it's very easy to make your second (soon to be a mirror) layout exactly the same as your base OS disk. You actually have at least two options:

a. You can run format, select the disk you have the OS installed on, type label (if format tells you the disk isn't labeled), then select your second disk, type partition, type select and pick the number of the label of your original disk. A lot of times these labels will be very generic (especially if you just typed "y" when format asked you to label the disk or format already did it for you during install) and you may have more than one to choose from. It's simple enough to figure out which one is the right one though (as long as you remember your partition map from the original disk and have made is sufficiently different from the default 2 or 3 partition layout). Just choose select, pick one, then choose print. If you've got the right one, then type label. Otherwise, repeat until you've gone through all of your selections. One of them has to be it, unless you never labeled your primary disk.

b. You can use two command (fmthard and prtvtoc) and just get it over with:

host # prtvtoc /dev/rdsk/c0t0d0s2 |fmthard -s - /dev/rdsk/c1t0d0s2

3. Then you'll want to mirror all of your "slices" (or partitions; whatever you want to call them. We'll assume you have 6 slices set up (s0, s1, s3, s4, s5 and s6) for use and slice 7 (s7) partitioned with about 5 Mb of space. You can probably get away with less. You just need to set this up for DiskSuite/VolumeManager to be able to keep track of itself.

Firstly, you'll need to initialize the minimum number of "databases," set up the mirror group and add the primary disk slices as the first mirrors in the mirror-set (even though, at this point, they're not mirroring anything, nor are they mirrors of anything ;) Note that it's considered best practice to not attach the secondary mirror slices to the mirror device, even though you can do it for some of your slices. You'll have to reboot to get root to work anyway, so you may as well do them all at once and be as efficient as is possible:

host # metadb -a -f /dev/rdsk/c0t0d0s7
host # metadb -a /dev/rdsk/c1t0d0s7
host # metainit -f d10 1 1 c0t0d0s0
host # metainit -f d20 1 1 c1t0d0d0
host # metainit -d0 -m d10
host # metainit -f d11 1 1 c0t0d0s1
host # metainit -f d21 1 1 c1t0d0d1
host # metainit -d1 -m d11
host # metainit -f d13 1 1 c0t0d0s3
host # metainit -f d23 1 1 c1t0d0d3
host # metainit -d3 -m d13
host # metainit -f d14 1 1 c0t0d0s4
host # metainit -f d24 1 1 c1t0d0d4
host # metainit -d4 -m d14
host # metainit -f d15 1 1 c0t0d0s5
host # metainit -f d25 1 1 c1t0d0d5
host # metainit -d5 -m d15
host # metainit -f d16 1 1 c0t0d0s6
host # metainit -f d26 1 1 c1t0d0d6
host # metainit -d6 -m d16

4. Now you'll run the "metaroot" command, which will add some lines to your /etc/system file and modify your /etc/vfstab to list the metadevice for your root slice, rather than the plain old slice (/dev/dsk/c0t0d0s0, /dev/rdsk/c0t0d0s0):

host # metaroot

5. Then, you'll need to manually edit /etc/vfstab to replace all of the other slices' regular logical device entries with the new metadevice entries. You can use the root line (done for you) as an example. For instance, this line:

/dev/dsk/c0t0d0s6 /dev/rdsk/c0t0d0s6 /users ufs 1 yes -

would need to be changed to:

/dev/md/dsk/d6 /dev/md/rdsk/d6 /users ufs 1 yes -

and, once that's done you can reboot. If you didn't make any mistakes, everything will come up normally.

6. Once you're back up and logged in, you need to attach the secondary mirror slices. This is fairly simple and where the actual syncing up of the disk begins. Continuing from our example above, you'd just need to type:

host # metattach d0 d20
host # metattach d1 d21
host # metattach d3 d23
host # metattach d4 d24
host # metattach d5 d25
host # metattach d6 d26

The syncing work will go on in the background, and may take some time depending upon how large your hard drives and slices are. Note that, if you reboot during a sync, that sync will fail and it will start from 0% on reboot with the affected primary mirror slices remaining intact and the secondary mirror slices automatically resyncing. You can use the "metastat" command to check out the progress of your syncing slices.

And, oh yeah... I almost forgot this part of the post:

The Zpool way:

1. First you'll want to do exactly what you did with DiskSuite/VolumeManager (since both disks have to be exactly the same). We'll assume you're insanely practical, and will just use this command to make sure your disks are both formatted exactly the same (just like above):

host # prtvtoc /dev/rdsk/c0t0d0s2 |fmthard -s - /dev/rdsk/c1t0d0s2

2. Now we'll need create a pool, add your disks to it (all slices as one) and mirror them:

host # zpool create mypool mirror c0t0d0 c1t0d0

3. Wait for the mirror to sync up all the slices. You can check the progress with "zpool status POOLNAME" - like:

host # zpool status mypool

And that's that. The choice is yours, unless you're still using Solaris 9 or older. This post isn't meant to condemn the SDS/SVM way. It works reliably and is really easy to script out (and when both of these methods are scripted out, they're just as easy to run and the only hassle the old way gets you is the forced reboot).

It's good to see that things are getting easier and more efficient. Although, hopefully, that won't make today's Sysadmins tomorrows bathroom attendants ;)

Cheers,

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

LVM's Roots - Mirroring Your Boot Disk On HP-UX 10 Unix

Hey There,

If you read this blog every once in a while (or if you just happen to have ever searched for - or queried the tag named - LVM in our growing library of questionably-valuable articles ;) you've probably noted that, although mentioned in passing, none of them has ever dealt directly with HP-UX. AIX and, of course, Linux have received their fair share of attention. Even Solaris Volume Manager (or, if you still prefer it, Solstice Disk Suite) and Veritas Volume Manager have been covered in some detail with reference to their similarity to LVM. It's about time that HP-UX (arguably one of the mothers of LVM (or LVM2) as we know it today) should get some sort of treatment. This blog is, after all, dedicated to Linux and Unix (both terms being purposefully generic so we can write about whatever *nix machines we can get our hands on :)

Today's entry is a bit of a quick introduction to HP's Logical Volume Manager and was written specifically for an HP-UX 10.x box. We haven't specifically tested this against 11.x or 11i, but, from our experience working with both, this script should work with little-or-no modification on 11.x. Now that we've got a few HP servers to have fun with (I mean... work really hard on ;), we'll give HP-UX it's due and run through the essentials of LVM. We'll try to make it as short and sweet as possible, while not skimming over the basics, so that the posts themselves can serve as a decent reference for a straight-up HP-UX user. Actually, if you're an HP-UX user (Experience here ranges from 9.x through 11i - Old 800 series K class towers ( with matching WYSE terminals) to SuperDomes and some of the newer 9000 series), you're also well familiar with the huge differences in the basic functionality between versions of the OS and the ISL and GSP/BCH underpinnings (which you could, somewhat, liken to a difference between the Domain Console/System Controller setup on the big Sun 3800 through 25k servers and the newer XCP/XSCF setup on the Mx000 series).

Until that day, here's a little script to help mirror your root disks on HP-UX 10. This was actually tested and used on a K100 Server (Refurbished, of course, but smokin' fast with 4 100 MHz CPU's. Actually, pretty decent once it boots up okay :)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

#
# hpmirrordisk.sh - Double check this before you run it.  Seriously :)
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

echo "You have installed the MirrorDisk-HPUX Product already, right?"
echo "If you still need to, hit ctl-C and quit running this script!"
read oblig
echo
echo "This should show only one bootable disk to start out..."
echo
lvlnboot -v
echo
echo "New bootable physical volume name [c?t?d?]"
echo
read disk
pvcreate -B /dev/rdsk/$disk
echo
echo "Volume group you'll be mirroring [vg??]"
echo
read vgroup
vgextend /dev/$vgroup /dev/dsk/$disk
echo
echo "Making Mirror Disk Bootable..."
echo
mkboot /dev/rdsk/$disk
mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/dsk/$disk
echo
echo "This result output should give you back \"hpux -lq (;0)/stand/vmunix\""
echo
lifcp /dev/rdsk/${disk}:AUTO -
echo
echo "Return to continue"
echo
read oblig
echo
echo "Extending all logical volumes..."
echo
for x in 1 2 3 4 5 6 7 8 9
do
 /usr/sbin/lvextend -m 1 /dev/${vgroup}/lvol$x /dev/dsk/$disk
done
echo
echo "This should show two bootable disks now..."
echo
lvlnboot -v
echo
echo "Okay? Hit return"
echo
read oblig
echo
echo "For these tests, the Physical Extents should map to the"
echo "Logical Extents for each logical volume."
echo
for x in 1 2 3 4 5 6 7 8 9
do
echo
 echo "lvol$x"
 echo
 lvdisplay -v /dev/${vgroup}/lvol$x |sed -n '/Distribution/,/Logical/p'|sed '$d'
 echo
 echo "Okay? Hit return"
 echo
 read oblig
done

echo
echo "All you need to do now is reboot and be sure to halt the"
echo "system at the boot menu.  Make sure the Primary Boot Path"
echo "is set to the mirrored disk.  If not, you will have to set"
echo "the correct path in the COnfiguration menu.  Also, under the"
echo "COnfiguration menu, set Auto Search equal to ON."
echo "Finally, be sure to set the Alternate Boot Path to the"
echo "original disk"
echo

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Mirroring Your Boot Disk Using Veritas On Sun Without Encapsulation

Hey There,

Today, we're going to take a look at a using Veritas to mirror your boot disks, without using encapsulation. There's raging debate about whether or not Veritas root disk encapsulation is bad (some folks are great with it because it takes care of everything, while some folks hate it because it's very hard to deal with when it doesn't take care of everything ;)

The following procedure was introduced quite a while ago by a pair of gentleman named Gene Trantham and John S. Howard in an article they wrote for Sun Blueprints Online (A site run by Sun that highlights best-practices for many, many things Solaris). I believe it's now simply referred to as the "Trantham/Howard Method."

Although I've used this method many times in my career, there's no point in re-writing the instructions (although I did, to a certain degree, to make them more concise and accessible). I did, however, want to have this information included on my blog, to complement (and complete) our earlier posts on patching your Veritas root disk, Veritas Volume Group setup, etc. The full document can be read here in the August 2000 Online Edition of Sun Blueprints.

So, without further ado, and, hopefully you'll find this helpful as well, the "Trantham/Howard Method" for mirroring boot disks using Veritas on Sun Solaris (somewhat modified and truncated):

Enjoy,

1. Install the basic Veritas Volume Manager Software.

2. Install any required Veritas licenses.

3. Use vxinstall to encapsulate the root disk c0t0d0 (Or whatever cxtxdx disk name accurately reflects your setup). Choose not to encapsulate the root disk and don't mirror it. For this post's sake, name it rootdisk Once the root disk is encapsulated, simply reboot and check "vxprint -ht" - You should note that only your rootdisk (and its Veritas name) is listed out.

4. Initialize the root mirror disk c0t1d0 (again, substitute with your own specific controller-target-disk name).

host # /usr/lib/vxvm/bin/vxdisksetup -i c0t1d0
host # vxdg adddisk rootmirror=c0t1d0

5. Manually attach the mirror plexes to the root volumes with "vxrootmir" (manually mirror the swapvol, usr and var volumes, if necessary, as well):

host # /usr/lib/vxvm/bin/vxrootmir rootmirror
host # vxassist mirror swapvol alloc=rootmirror
host # vxassist mirror usr alloc=rootmirror
host # vxassist mirror var alloc=rootmirror

While you wait for this to finish, you can check on the progress with the vxtask command.

6. Once your root disk is mirrored, disassociate the original plexes on c0t0d0, remove them, and remove the encapsulated rootdisk from the rootdg diskgroup:

host # vxplex dis rootvol-01 swapvol-01 var-01 usr-01
host # vxdg rmdisk rootdisk

Now your operating system is entirely running off of the c0t1d0 disk.
At this point, the operating system is based just on the initialized disk c0t1d0, which you can see by looking at the output of "vxprint -ht" again.

7. Now initialize, as opposed to encapsulating, c0t0d0:

host # /usr/lib/vxvm/bin/vxdisksetup -i c0t0d0
host # vxdg adddisk rootdisk=c0t0d0

8. Now, mirror everything from the rootmirror disk back to the rootdisk:

host # /usr/lib/vxvm/bin/vxrootmir rootdisk
host # vxassist mirror swapvol alloc=rootdisk
host # vxassist mirror var alloc=rootdisk
host # vxassist mirror usr alloc=rootdisk

Again, you can use the vxtask command to view the progress of this command, which may take some time to complete. Also, this process removes the "rootdisk-B0" subdisk. This is used by Veritas to protect the boot block, but isn't necessary when you're not encapsulating your disks.

9. Using the output of "vxprint -ht", create disk slices corresponding to the Veritas subdisks. The usage for vxmksdpart command is:

vxmksdpart [-f] [-g diskgroup] subdisk sliceno [tag flags]

You won't need to run through this for the root slice since vxrootmir has already done this for you. You will need to do this for the other slices, though. But, don't use vxmksdpart just yet.

10. On the "new" root disk, first create the slices for the partitions you're going to mirror:

host # /usr/lib/vxvm/bin/vxmksdpart rootdisk-02 1 0x03 0x01
host # /usr/lib/vxvm/bin/vxmksdpart rootmirror-02 1 0x03 0x01

and so on for each required partition (again, you can get this output from "vxprint -ht"). The "sliceno" part of the command syntax is very important here!

host # /usr/lib/vxvm/bin/vxmksdpart rootdisk-03 5 0x07 0x00
host # /usr/lib/vxvm/bin/vxmksdpart rootmirror-03 5 0x07 0x00
...

11. Among other things (like backing up /etc/vfstab, etc) be sure to capture the current disk partition information for both disks, before rebooting, using vxprtvtoc:

host # /usr/lib/vxvm/bin/vxprtvtoc -f /tmp/disk1 /dev/rdsk/c0t0d0s2
host # /usr/lib/vxvm/bin/vxprtvtoc -f /tmp/disk2 /dev/rdsk/c0t1d0s2
host # cat /tmp/disk1 /tmp/disk2 > /etc/vtoc

12. Update your system dump configuration to use the swap partition on the primary disk:

host # dumpadm -d /dev/dsk/c0t0d0s1
Dump content: kernel pages
Dump device: /dev/dsk/c0t0d0s1 (swap)
Savecore directory: /var/crash/host
Savecore enabled: yes

13. In the OBP, be sure to setup both disks using devalias (name them disk and rootdisk, if you want, or whatever makes the most sense to you) and set your configuration to have the boot-device set up to make it easy for your system to fail over if the primary disk goes bad:

ok> setenv boot-device disk, rootdisk, net

14. Be sure to test booting from the OBP for both disk and rootdisk. Assuming that goes well, you're all set :)

, Mike

Safely Patching Your Veritas Root Mirror Disk On Linux Or Unix

Hey there,

It's been a long time since we've taken a look at anything "Veritas" (almost a few months now since we published a few posts regarding disk groups and volume groups in Veritas Volume Manager for Linux and/or Unix. Given the relatively broad nature of this blog, I sometimes wonder how we can ever stay entirely focused on any "one" thing for too long ;)

But, enough about us... For this "Lazy Sunday" post we're going to take a look at patching a root (or boot) mirror disk in VxVM safely. And by safely, we mean that you'll be able to fail-back to your root mirror disk as if nothing ever happened. That is, if something awful actually does happen.

The basic concept is simple, and applies to all brands and methods of root disk mirroring. When you're faced with having to apply patches to your OS (which invariably involves changes to your root disk), you always want to make sure that your root mirror is "golden" before you begin. You also want to make sure that it's taken out of the equation for the initial patch run, so you'll have a perfect failback device (less sweat, no accounting for tears ;)

The first thing you'll want to do, as per above, is to validate your root disk's mirror disk. For Veritas Volume Manager, every volume associated with the root disk must (well, technically, "should") have, at least, a single subdisk for each and every plex on the root disk and the root mirror disk.

For our example today, we'll consider that our root disk is c0t0d0s2 and its mirror is c1t0d0s02. They both belong to the default Veritas Volume: rootdg. Please also note that a lot of this output is "mocked up" to a certain degree since I'm not in a position to actually disassociate volumes on the computers I'm using for the sake of this post :)

You can check the state of your volumes with the "vxprint" command, like so (we'll use the ellipses (...) to indicate output that I've trimmed to keep this post under 50,000 words ;) :

host # vxprint -htqg rootdg <--- This output has been truncated as well, to highlight the mostly one-to-one relationship between subdisks (sd) and plexes (pl). As you can see, each of our two volumes on our rootdisk has at least one subdisk associated with each plex. We're going to ignore root_disk-B0 for this post (or not go into it too much) as this isn't really a "volume" but a way Veritas gets around the fact that it uses the part of the disk that most operating systems reserve (the bootblock - This, again, is enough material for another post entirely)

Disk group: rootdg 

dg rootdg       default ...
dm root_disk    c0t0d0s2 ...
dm root_mirror  c1t0d0s2 ...

sd root_diskPriv        - ...

v root_volume       - ...
pl root_volume-01   root_volume ...
sd root_disk-B0 root_volume-01 ...
sd root_disk-02 root_volume-01 ...
pl root_volume-02   root_volume ...
sd root_mirror-01       root_volume-02 ...

v swap_volume       - ...
pl swap_volume-01   swap_volume ...
sd root_disk-01 swap_volume-01 ...
pl swap_volume-02   swap_volume ...
sd root_mirror-02       swap_volume-02 ...

Now that we know we're good, even though it may have already been done, I find it's always good practice to install a new bootblock on the root mirror disk from the main root disk. The worst case scenario (assuming no typos ;) would be that you updated an existing bootblock with one that should, theoretically, be an exact match for your primary root disk (which is what we want) :

host # /usr/lib/vxvm/bin/vxbootsetup -g rootdg root_mirror

If you have other partitions on your root disk, that aren't listed in your vxprint output of the rootdg above, you can define them with the vxmksdpart command. You might have your /opt partition on the root disk, but not in the rootdg. Sometimes you'll see /home or even /var on the rootdisk but not associated with the rootdg. While it's considered "best practice" by Veritas to add these partitions to the rootdg before separating the disks, I've found that it's never actually been "necessary." The idea is that you associate the partitions, just so you can disassociate them a few minutes later (???)

Next, we'll disassociate (see what I mean ;) the root mirror disk plexes from the root disk, like so (you can verify that, for instance, swap_volume-02 is associated with the mirror disk in the vxprint output above):

host # vxplex -g rootdg dis root_volume-02
host # vxplex -g rootdg dis swap_volume-02

Now, well simply mount the root filesystem from the disassociated mirror disk on a temporary directory on the root disk and make a few quick file backups and edits, like so (Note that, for most Linux flavours, /etc/system noted below is actually /etc/sysctl.conf and /etc/vfstab is /etc/fstab):

host # mkdir /vxtmp
host # mount /dev/dsk/c1t0d0s0 /vxtmp
host # cp /vxtmp/etc/system /vxtmp/etc/system.old
host # cp /vxtmp/etc/vfstab /vxtmp/etc/vfstab.old
host # cp /vxtmp/etc/vfstab.prevm /vxtmp/etc/vfstab
host # touch /mnt/etc/vx/reconfig.d/state.d/install-db

Now, in the /vxtmp/etc/system file, we'll comment out the following two lines (remember that in the /etc/system file the "*" is the comment character. You probably already know that, but I feel responsible ;) -- Edit the following two lines so that they are now commented:

* rootdev ...
* set vxio ...

Then we'll unmount the root mirror disk on /vxtmp:

host # umount /vxtmp

and we're ready to patch! Assuming that everything goes swimmingly, all we need to do is reattach the root mirror disk plexes to the root disk, like this:

host # vxplex -g rootdg att root_volume root_volume-02
host # vxplex -g rootdg att swap_volume swap_volume-02

The root disk should sync itself up so that the root mirror disk gets updated (which you can monitor with "vxtask") And, you're all set :)

Now... If things go bad... The official explanation is so long and ridiculous (and differs for versions up to 3.5 and newer versions), that I'll refer you to an actual official document from Veritas online support that will show you a neat trick to get around having to jump through 15 or 16 hoops to get this all over with ;) Another glorious example of the system raging against itself :)

Cheers,

, Mike

ZFS Command Sheet For Solaris Unix 10 - Pool And File System Creation

Hey There,

Today, we're going back to the Solaris 10 Unix well and slapping together a few useful commands (or, at least, a few commands that you'll probably use a lot ;). We've already covered ZFS, and Solaris 10 zones, in our previous posts on creating storage pools for ZFS and patching Solaris 10 Unix zones, but those were more specific, while this post is meant to be a little quick-stop command repository (and only part one, today). This series also is going to focus more on ZFS and less on the "zone" aspect of the Solaris 10 OS.

Apologies if the explanations aren't as long as my normal posts are. ...Then again, some of you may be thanking me for the very same thing ;)

So, without further ado, some Solaris 10-specific commands that will hopefully help you in a pinch :) Note that for all commands where I specify a virtual device or storage pool, you can get a full listing of all available devices/pools by "not specifying" any storage pool. I'm just trying to keep the output to the point so this doesn't get out of hand.

Today we're going to take storage pools and ZFS file systems and look at creation-based commands, tomorrow we'll look at maintenance/usage commands, and then we'll dig on destructive commands and cleaning up the mess :)

1. To create virtual devices (vdevs), which can, technically, be virtual (disk made from a part, or parts, of real disk) or "real" disk if you have it available to you, you can do this:

host # mkfile 1g vdev1 vdev2 vdev3
host # # ls -l vdev[123]
-rw------T   1 root     root     1073741824 May  5 09:47 vdev1
-rw------T   1 root     root     1073741824 May  5 09:47 vdev2
-rw------T   1 root     root     1073741824 May  5 09:48 vdev3

2. To create a storage pool, and check it out, you can do the following:

# zpool create zvdevs /vdev1 /vdev2 /vdev3
# zpool list zvdevs <--- Don't specify the name of the pool if you want to get a listing of all storage pools!

NAME                    SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
zvdevs                 2.98G     90K   2.98G     0%  ONLINE     -

3. If you want to create a mirror of two vdev's of different size, this can be done, but you'll be stuck with the smallest possible mirror (as it would be physically impossible to put more information on one disk that it can contain. That seems like common sense ;)

host # zpool create -f vzdevs mirror /vdev1 /smaller_vdev <--- The mirrored storage pool will be the size of the "smaller_vdev"

4. If you want to create a mirror, with all the disks (or vdevs) the same size (like they should be :), you can do it like this:

host # zpool create zvdevs mirror /vdev1 /vdev2 /vdev3 /vdevn... <--- I haven't hit the max yet, but I know you can create a "lot" of mirrors in the same set. Of course, you'd be wasting a lot of disk and it would probably make data access slower...

# zpool list
NAME          SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
myzfs        95.5M    112K   95.4M     0%  ONLINE     -
host # zpool status -v zvdevs
  pool: zvdevs
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zvdevs      ONLINE       0     0     0
          mirror    ONLINE       0     0     0
            /vdev1  ONLINE       0     0     0
            /vdev2  ONLINE       0     0     0
            /vdev3  ONLINE       0     0     0

errors: No known data errors

5. You can create new directories, add file systems on them and mount them in your storage pool very easily. All you need to do is "create" them with the "zfs" command. Three tasks in one! (as easy as creating a pool with the zpool command):

host # zfs create zvdevs/vusers
host # df -h zvdevs/vusers
Filesystem             size   used  avail capacity  Mounted on
zvdevs/vusers          984M    24K   984M     1%    /zvdevs/vusers

6. If you need to create additional ZFS file systems, the command is the same, just lather rinse and repeat ;)

host # zfs create zvdevs/vusers2
host # zfs create zvdevs/vusers3
host # zfs list |grep zvdevs
zvdevs                             182K   984M  27.5K  /zvdevs
zvdevs/vusers                     24.5K   984M  24.5K  /zvdevs/vusers
zvdevs/vusers2                    24.5K   984M  24.5K  /zvdevs/vusers2
zvdevs/vusers3                    24.5K   984M  24.5K  /zvdevs/vusers3

See you tomorrow, for more fun with Solaris 10 ZFS/Storage Pool maintenance/usage commands :)

Cheers,

, Mike

linux unix internet technology

Perl Script To Mirror Lines In A File On Linux Or Unix

Good afternoon-morning-day-evening :)

In much the same vein as our previously posted script to do weak encryption with Octal Dump, today we're throwing out another Perl script to do something possibly equally worthless, but still somewhat entertaining ;) Since it's written in Perl, and uses that language's specific constructs (i.e. No "system" calls), it should run equally well on Linux or Unix. The logic in the script is simple enough that it should probably work going back a few major versions.

As much as this may not seem to be of any use to you now, knowing how to mirror (or reverse, since many people, including myself, don't have any mirror-friendly fonts ;) each line in a file can be beneficial. Although no one in the office is likely to come up to you and say "Mr. sysadmin, sir (of course, I'm exaggerating. People are much more formal than that normally ;), Can you print every single line of this report reading from right-to-left instead of the standard left-to-right?", it's even more unlikely that, if this were to happen, any business/war jargon would be used. I can't imagine something like this ever being "mission critical" or required for any sort of "code red" situation. If it ever is, you'll be really glad you know how to do this...

But, at a fundamental level, it's good to understand the basic functionality of Perl and how to deal with scalar variables (or string variables), arrays and how to muck around with them (or make them work for you ;) In this particular script, the building blocks of some very useful functions of Perl are employed (to a dubious end, I'll admit) and, hopefully, presented in an easy to understand fashion. Over time, we'll dig into every little thing there is to know (If that can be cranked out in one life time ;) with regards to each function. In the mean time, check out the use of these four functions (Note that, for all, we'll assume that the variable $variable is defined as "abcd" and the array @array consists of four members: a, b, c and d:

split - This function will take a scalar variable and "split" it, on a delimiter, into an array:

Ex: @array = split(//, $variable); <--- Now @array has four members (a, b, c and d) that it got from $variable

join - This function will take an array and "join" it into one scalar variable:

Ex: $variable = join(/ /, @array); <--- Now $variable equals "abcd" since it contains all 4 members of @array joined together by an empty delimiter "/ /" (that is, each character, or space, is considered a separate member of the array)

undef - This function will "undefine" our array, in this case. It can be used on scalar, array and hash variables as well.

Ex: undef(@array); <--- Now @array is not just empty, it isn't even defined. It may as well not exist.

push - This function will "push" a variable (scalar, array, hash, or references to same, and more -- getting way off-topic ;) onto the left side of an array. Subsequent pushes of extra variables are added from the left, so if you push three variables into an array (a, b and c, for instance) in one order, they'll actually end up in the array in the opposite order.

Ex:
push(@array, "a"); <--- @array now equals "a"
push(@array, "b"); <--- @array now equals "b" "a"
push(@array, "c"); <--- @array now equals "c" "b" "a"

In any event, enjoy, and I hope this helps you out if you're beginning to learn the Perl scripting language!

Best wishes,

SAMPLE RUN:

host # cat words|head -3;cat words|tail -3 <--- The beginning and end of the file we're going to "mirror"
Aarhus
Aaron
Ababa
Zulu
Zulus
Zurich
host # ./mirror.pl words
host # ls
. .. mirror.pl words words.mirror <--- Our newly created file is called "words.mirror"
host # cat words.mirror|head -3;cat words.mirror|tail -3 <--- And (good deal), the mirroring seems to have worked!
suhraA
noraA
ababA
uluZ
suluZ
hciruZ
host # wc -l words* <--- Just double checking here to make sure that the number of characters in our original file and the mirror file are exactly the same, which they should be.
45378 words
45378 words.mirror
90756 total





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# mirror.pl - print an entire file backward, line by line
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if ( $#ARGV != 0 )      {
        print "Usage: $0 text_file\n";
        exit(1);
}

$text_file=$ARGV[0];

open(TXT, "<$text_file");
@txt = <TXT>;
close(TXT);

open(RTXT, ">${text_file}.mirror");
foreach $ln (@txt)      {
        chomp($ln);
        @ln = split(//, $ln);
        $ln_len = @ln;
        undef(@rev);
        while ( $ln_len > 0 )   {
                push(@rev, $ln[${ln_len}-1]);
                $ln_len--;
        }
        $rev = join(/ /, @rev);
        print RTXT "$rev\n";
}
close(RTXT);
exit(0);

, Mike

linux unix internet technology

Recovering failed RAID disks on Linux

Hey there once more,

As I mentioned in yesterday's post regarding creating RAID disk sets we'll be looking at how to deal with RAID disk failures on Linux in today's post. Once we're done here, I think we can call it a wrap on this subject for a while (Are those sighs of relief I'm hearing? ;) At least until we come up with some good shell scripts to expedite some of this stuff.

Disk failure, on the hardware side, is too broad to cover in any great detail here, but the following basic steps should be followed (of course, as noted, your setup may require otherwise). The scenario here is that one of your disks has just gone "bad." It's beyond recovery.

1. If the disk is hot-swappable, simply remove it. If it isn't, you'll need to schedule downtime and remove the disk then. If the failed disk is your boot disk, you'll already have your downtime ;)

2. Replace the failed disk and restart your machine (and skip to step 6) if your failed disk isn't the boot disk.

3. If your failed disk is the boot disk, you'll next want to boot off of CD (We're using RedHat Linux AS), mount the root filesystem under a temporary mountpoint, and do the following:

host # mkdir /tmp/recovery
host # mount /dev/hda0 /tmp/recovery
host # chroot /tmp/recovery
host # grub --batch (This may take a while as grub probes and tries to guess where all of your drives are)

4. Once grub is finished probing, do the following at the "grub>" prompt:

grub> root (hd0,0)
root (hd0,0)
Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd0)
setup (hd0)
...
Running "install /grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/grub/stage2
/grub/grub.conf"... succeeded
grub> exit

5. Now take a second and verify that all is well while still running off of the CD, like so:

host # cat /boot/grub/device.map:
(fd0) /dev/fd0
(hd0) /dev/hda

host # df -k (should show the /dev/md0 mount)
host # umount /tmp/recovery
host # reboot (Be sure to set the grub device map for hd0 to /dev/hdc if /dev/hda has gone bye-bye)

6. Now that you have the disk physically replaced and you've booted back up, check the content of /proc/mdstat, like so:

host # cat /proc/mdstat

Personalities : [raid1]
read_ahead 1024 sectors
...
md0 : active raid1 hda1[0]
     307328 blocks [2/1] [U_]
...
unused devices: <none>

Again, the [2/1] listing indicates that one part of the mirror is not active in this two-way mirror set!

7. If this has been driving you crazy from the first post, we're now going to recreate the mirror so it's not incomplete (apologies for any stress-related reading injuries. Sometimes I go a long way to make a point ;). The first thing we'll do is repartition the disk, again, with fdisk (instructions in our previous post regarding creating RAID disk sets ) and we should end up with our partition table looking exactly the same:

host # fdisk -l /dev/hda (The partition table should look almost identical for /dev/hdc)

Disk /dev/hda: 16 heads, 63 sectors, 77520 cylinders 
Units = cylinders of 1008 * 512 bytes 

   Device Boot    Start       End    Blocks   Id  System 
/dev/hda1   *         1      6095   3071848+  fd  Linux raid autodetect 
/dev/hda2          6096     67047  30719808   fd  Linux raid autodetect 
/dev/hda3         67048     73142   3071880   fd  Linux raid autodetect
/dev/hda4         75175     77206   1024096+  82  Linux swap

8. As a matter of course, just to be sure, check /etc/raidtab to see what devices need to be mirrored with what devices:

host # cat /etc/raidtab

raiddev             /dev/md0
raid-level                  1
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
   device          /dev/hda1
   raid-disk     0
   device          /dev/hdc1
   raid-disk     1
...

9. Now, we just need to add back all the partitions we lost:

host # raidhotadd -a /dev/md0 /dev/hda1
...

And keep tabs on the process by checking on /proc/mdstat:

host # cat /proc/mdstat

Personalities : [raid1]
read_ahead 1024 sectors
...
md0 : active raid1 hda1[2] hdc1[1] 
      30716160 blocks [2/1] [_U] 
      [===========>..........]  recovery = 45.9% (34790000/61432320) finish=98.7min speed=8452K/sec
unused devices: <none>

Once the RAID sync is done, you should be good to go! I'd reboot one more time, just to be sure. Especially if you were forced to take downtime in the first place :P

Take care,

, Mike

linux unix internet technology

Creating RAID disk sets on Linux

Today's post is a somewhat-continuation of yesterday's post on RAID disk monitoring. We looked at monitoring then, and (the reason I call this a somewhat-continuation) now we're going to take a step back and look at setting up those RAID disk sets (a simple mirror set in this example)in the first place ;) Note that all the examples I'm using today are taken from RedHat Linux AS and can be run in bash or any native shell. Scripting some of this out is near impossible, but someday soon I may give it a go.

In any event, once we've covered all this ground, we'll definitely look at how a handy shell script or two can save a whole lot of time on these processes :) So, off we go:

The first thing you'll want to do is to create your partitions on each physical disk, using the fdisk utility. I personally prefer it from the command line, so if you use Disk Druid or any other GUI interface, it should be easy enough to translate this to that so I don't have to over-explain.

The first thing we'll do is to create the disk partitions and assign them to an appropriate RAID device. The steps should roughly be the same as these:

1. Invoke fdisk at that command line.
2. Select "n" to add a new partition (anything from a slice of disk for mirroring two disks, to an entire disk for higher level RAID and mega-storage).
3. Once added, select "t" to change the partition flag and select "Linux Raid Autodetect" (selection "fd" on my console, possibly different on yours).
4. Select "a" to add a bootable flag to the partition, if necessary.
5. Save the new disk layout, and quit, by selecting "w"
6. Repeat as necessary

It's important, since we're doing this from the command line down-and-dirty, that we make sure that all partitions on both disks are "exactly" the same size. Unless your secondary mirror disk is larger, in which case you just need to make sure that you only use partitions larger than the originals to mirror to (It only makes sense. You can't copy a larger filesystem to a smaller one without losing data - The RAID utilities will discourage you from doing so, anyway).

Now, you'll need to put filesystems on all of those partitions. Generally, you should be able to take care of this using mkfs:

host # mkfs -t ext3 /dev/hda1

In the bizarre event that this fails, or you're using an older or different OS than I am, this should also work:

host # mkfs -t ext2 -j /dev/hda1

Now we'll use set up our /etc/raidtab file, with the following (assuming the same for all slices):

host # raiddev /dev/md0
        raid-level      linear
        nr-raid-disks   2
        nr-spare-disks  0
        chunk-size      64
        persistent-superblock 1
        device          /dev/hda1
        raid-disk       0
        device          /dev/hdc1
        raid-disk       1
...

Then we'll initialize the RAID device, like so:

host # mkraid /dev/md0

Once again, per yesterday's post on RAID disk monitoring, we can check on the RAID device's progress by simply doing the following:

host # cat /proc/mdstat

and we'll know instantly if the RAID device is in a good or bad state, and how far along it is if it's still synching. You can use the above command to check status whenever you make a change to your RAID configuration for any valid RAID md devices.

If you ever want, or need, to remove any RAID partitions, you can do that like this (In this example, the hda1 slice is having issues on the md0 device):

host # raidsetfaulty /dev/md0 /dev/hda1
host # raidhotremove /dev/md0 /dev/hda1

Sometimes, as a failsafe (this depends on your situation and what device you're removing) you may want to re-install grub on your disk device, like so:

host # grub-install /dev/hda

And, finally for today, if you do end up having to remove a RAID mirror partition and it can be replaced inline (A whole disk in itself, for instance - possibly just a partition on a disk if bizarre filesystem corruption was the only issue and you've recreated the filesystem), do the following:

1. Add the new RAID devices as before (create, set filesystem type, assign same partition names if possible, etc) and check the status of the RAID groups. [2/2] means that the RAID group is ok, [2/1] means that the RAID group needs to be resynced (you'll note from yesterday that the RAID group we're working on is in a [2/1] state, with only one active mirror. If you noted that this could be, and should be, fixed - as well as the fact that this failure would drop that RAID partition because of this condition - you're sprinting ahead of my tutorial skills ;)

host # cat /proc/mdstat 
Personalities : [raid1] 
read_ahead 1024 sectors 
md0 : active raid1 hda1[0] 
      30716160 blocks [2/1] [U_] 
...
unused devices: <none>

2. Determine your RAID group and mirrors from /etc/raidtab:

host # cat /etc/raidtab 
...
raiddev             /dev/md0 
raid-level                  1 
nr-raid-disks               2 
chunk-size                  64k 
persistent-superblock       1 
nr-spare-disks              0 
    device          /dev/hda1 
    raid-disk     0 
    device          /dev/hdc1 
    raid-disk     1 
...

3. Now, you'll want to resync the RAID mirror sets that got goofed up when we had to replace the RAID parititions:

host # raidhotadd /dev/md0 /dev/hda1

4. Assuming you've lost more than 1 mirror, just repeat the procedure above as many times as necessary.

5. Lastly, assuming all went well, kick back and enjoy the view ;) - Note that Linux will normally queue resyncs and do them one at a time, so don't panic if you only see one out of 5 disk groups being worked on when you check this out!

host # cat /proc/mdstat  
...     
md3 : active raid1 hda1[2] hdc1[1] 
      30716160 blocks [2/1] [_U] 
      [===========>..........]  recovery = 45.9% (34790000/61432320) finish=98.7min speed=8452K/sec 
...      
unused devices: <none>

And that's it for today. In another near future post (probably tomorrow) we'll look at the fun involved with RAID disk failure ;)

Best Wishes,