How Avoid Solaris Panics When Using Savecore With Veritas Volume Manager

Hey There,

Ever since Solaris 7, the savecore command has been able to act somewhat like "gcore," but better. The introduction of the "-L" flag allowed users to take a crash dump of the entire running Solaris system (much in the same way gcore does for individual processes). When invoked with this flag, savecore will take a snapshot of the live system and write it to the system dump device. This dump device has to be "dedicated," and defined in /etc/dumpadm.conf. Out of the box, this dedicated dump device is set to your system swap partition, so you may have it set up correctly even if you've never given it a second thought (or a first ;). This, in and of itself, isn't anything spectacular, since you can use "fssnap" to do pretty much the same thing and have finer control over what gets "snapped" and where it gets put. The beauty of "savecore -L /some/other/output/directory" is that, after writing the snapshot to your dedicated dump device, it will suck up that data and create crash dump files wherever you've instructed it to. It's kind of like a combination of "fssnap" and "gcore," if you could manage to get those two programs to cooperate with each other and create a crash dump of your live system.

Of course, the point of today's post isn't to debate the merits of using this feature or even (as it may seem) glorifying it in any way. The preceding paragraph was simply meant as introduction to a point-of-disaster that is still just waiting to happen on Solaris boxes all over the world. If you prefer to keep your machines at the "ok>" prompt, this post may not be for you.

The disaster, itself, is not brought on solely by savecore and the fault doesn't really lie with the program itself. However, if you opt to manage your filesystems with Veritas Volume Manager, depending on how you go about things, the combination of the two can be "interesting" at best ;) Of course, in order to cause this panic/crash/freak-out to happen, things in your Solaris dumpadm configuration and Veritas Volume Manager setup have to be constructed in a very specific way. Unfortunately, that very specific way is the default for a lot of shops that use Volume Manager to encapsulate the root disks on their systems. That, and having a default (or even a modified dumpadm.conf that doesn't stray too far from convention) is all that's required. If this situation could only occur on the third Tuesday of every month, on alternate leap years during a "classic syzygy" (when the sun moon and earth lie in a straight line with one another), I probably wouldn't be writing this post right now. So, in a way, it's a good thing, because I was initially going to make today's bullet-list about 50 ways you can save money, and avoid responsibility, by catching up on sleep ;)

So, after that lengthy preamble, here's a breakdown of one way to "break down" your Solaris system running VVM, step by step:

1. Ensure that your dumpadm.conf is "normal": This is a default /etc/dumpadm.conf (put together for us by Solaris during the installation process):

host # cat /etc/dumpadm.conf
#
# dumpadm.conf
#
# Configuration parameters for system crash dump.
# Do NOT edit this file by hand -- use dumpadm(1m) instead.
#
DUMPADM_DEVICE=/dev/dsk/c0t0d0s1
DUMPADM_SAVDIR=/var/crash/host
DUMPADM_CONTENT=kernel
DUMPADM_ENABLE=yes

It looks good. According to the way things have always been done, it's just about perfect. Solaris, as far back as I can remember, has always recommended using your swap partition as your dump device if you enabled savecore (which you had to manually set up before Solaris 7), and the save directory is the default that it's always been (/var/crash/yourhostname).

NOTE: If you stray from this setup by changing your DUMPADM_DEVICE to another unused partition (preferably on a separate disk), you'll never experience this disaster (at least not for the same reasons...)

2. Encapsulate your root disk using Veritas Volume Manager: You may recall, from our past post on how to mirror your root disk using Veritas that our setup went through pains to make sure that the end product was "not" encapsulated. The reasons for this were different, but, if you followed those (or other similar) instructions, you won't be able to recreate this exact "magnificent failure" either.

3. Run "savecore -L /whatever/directory/you/want" on a machine in which the above two conditions exist.

And, that's it! Your results may vary, but general system panic is the most accurate way to describe the plethora of confusing hex error messages and resulting system crash that will most likely occur.

Now, let's take a look at the reasons "why" this can happen:

1. If you're using Veritas Volume Manager and your root disk (assuming this is where your DUMPADM_DEVICE, or swap partition, resides) is "not" encapsulated, dumpadm would recognize that the device it has listed as a dump location is being used by Veritas and can't be used by itself. This results in dumpadm refusing to run "savecore -L." Since the root disk is encapsulated, dumpadm has no idea that the DUMPADM_DEVICE is being accessed by Veritas through a different logical device. Basically, Solaris checks the swaplist and determines that the DUMPADM_DEVICE is not listed as a swap device (it "is" a swap device, but encapsulation only considers the Volume to be a swap device, and not the partition associated with it). Since the partition underlying the Veritas Volume is not listed as a swap device, it shows up as "dedicated." Therefore, your "savecore -L" command is processed as though all were well.

2. Actually, this is just an extension of point 1. But, the disaster happens when you run "savecore -L." Since the DUMPADM_DEVICE is pointing to the same region of disk that the Veritas swap Volume is located, when the command gets run, information stored in the swap partition and /tmp filesystem can easily become corrupted. It's virtually guaranteed to, since Solaris and Veritas are both acting independently of each other and reading/writing from the exact same place (both assuming they're the only one with exclusive access).

This problem is highly prevalent in shops that use "defaults" and recommended "best practices." (e.g. the default /etc/dumpadm.conf on a Solaris box directly falls into contradiction with Veritas Volume Manager's preference toward encapsulating the root disk). Fortunately, it is very easy to fix in one, or a few ways:

1. Change your /etc/dumpadm.conf file so that the DUMPADM_DEVICE is on a disk other than the root disk (or its mirror, if you have that set up). /usr/sbin/dumpadm only runs when it's called, so that's all you have to do. Veritas (I mean Symantec) themselves recommend that you "never" run "savecore -L" on an encapsulated root disk.

2. Follow the instructions in our old post on mirroring your root disk with Volume Manager and dumpadm will be able to determine that its default DUMPADM_DEVICE can't be used, which will nudge you toward implementing the first solution if you really want to make use of this aspect of savecore's functionality.

...and then any of various combinations of the two (given the variety of site installations of Solaris and different ways folks like to set their disks up with VVM). All you really have to do is avoid just one of these specific conditions that will almost always cause the problem.

Here's to not spending our nights and weekends at work :)

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Mirroring Your Boot Disk Using Veritas On Sun Without Encapsulation

Hey There,

Today, we're going to take a look at a using Veritas to mirror your boot disks, without using encapsulation. There's raging debate about whether or not Veritas root disk encapsulation is bad (some folks are great with it because it takes care of everything, while some folks hate it because it's very hard to deal with when it doesn't take care of everything ;)

The following procedure was introduced quite a while ago by a pair of gentleman named Gene Trantham and John S. Howard in an article they wrote for Sun Blueprints Online (A site run by Sun that highlights best-practices for many, many things Solaris). I believe it's now simply referred to as the "Trantham/Howard Method."

Although I've used this method many times in my career, there's no point in re-writing the instructions (although I did, to a certain degree, to make them more concise and accessible). I did, however, want to have this information included on my blog, to complement (and complete) our earlier posts on patching your Veritas root disk, Veritas Volume Group setup, etc. The full document can be read here in the August 2000 Online Edition of Sun Blueprints.

So, without further ado, and, hopefully you'll find this helpful as well, the "Trantham/Howard Method" for mirroring boot disks using Veritas on Sun Solaris (somewhat modified and truncated):

Enjoy,

1. Install the basic Veritas Volume Manager Software.

2. Install any required Veritas licenses.

3. Use vxinstall to encapsulate the root disk c0t0d0 (Or whatever cxtxdx disk name accurately reflects your setup). Choose not to encapsulate the root disk and don't mirror it. For this post's sake, name it rootdisk Once the root disk is encapsulated, simply reboot and check "vxprint -ht" - You should note that only your rootdisk (and its Veritas name) is listed out.

4. Initialize the root mirror disk c0t1d0 (again, substitute with your own specific controller-target-disk name).

host # /usr/lib/vxvm/bin/vxdisksetup -i c0t1d0
host # vxdg adddisk rootmirror=c0t1d0

5. Manually attach the mirror plexes to the root volumes with "vxrootmir" (manually mirror the swapvol, usr and var volumes, if necessary, as well):

host # /usr/lib/vxvm/bin/vxrootmir rootmirror
host # vxassist mirror swapvol alloc=rootmirror
host # vxassist mirror usr alloc=rootmirror
host # vxassist mirror var alloc=rootmirror

While you wait for this to finish, you can check on the progress with the vxtask command.

6. Once your root disk is mirrored, disassociate the original plexes on c0t0d0, remove them, and remove the encapsulated rootdisk from the rootdg diskgroup:

host # vxplex dis rootvol-01 swapvol-01 var-01 usr-01
host # vxdg rmdisk rootdisk

Now your operating system is entirely running off of the c0t1d0 disk.
At this point, the operating system is based just on the initialized disk c0t1d0, which you can see by looking at the output of "vxprint -ht" again.

7. Now initialize, as opposed to encapsulating, c0t0d0:

host # /usr/lib/vxvm/bin/vxdisksetup -i c0t0d0
host # vxdg adddisk rootdisk=c0t0d0

8. Now, mirror everything from the rootmirror disk back to the rootdisk:

host # /usr/lib/vxvm/bin/vxrootmir rootdisk
host # vxassist mirror swapvol alloc=rootdisk
host # vxassist mirror var alloc=rootdisk
host # vxassist mirror usr alloc=rootdisk

Again, you can use the vxtask command to view the progress of this command, which may take some time to complete. Also, this process removes the "rootdisk-B0" subdisk. This is used by Veritas to protect the boot block, but isn't necessary when you're not encapsulating your disks.

9. Using the output of "vxprint -ht", create disk slices corresponding to the Veritas subdisks. The usage for vxmksdpart command is:

vxmksdpart [-f] [-g diskgroup] subdisk sliceno [tag flags]

You won't need to run through this for the root slice since vxrootmir has already done this for you. You will need to do this for the other slices, though. But, don't use vxmksdpart just yet.

10. On the "new" root disk, first create the slices for the partitions you're going to mirror:

host # /usr/lib/vxvm/bin/vxmksdpart rootdisk-02 1 0x03 0x01
host # /usr/lib/vxvm/bin/vxmksdpart rootmirror-02 1 0x03 0x01

and so on for each required partition (again, you can get this output from "vxprint -ht"). The "sliceno" part of the command syntax is very important here!

host # /usr/lib/vxvm/bin/vxmksdpart rootdisk-03 5 0x07 0x00
host # /usr/lib/vxvm/bin/vxmksdpart rootmirror-03 5 0x07 0x00
...

11. Among other things (like backing up /etc/vfstab, etc) be sure to capture the current disk partition information for both disks, before rebooting, using vxprtvtoc:

host # /usr/lib/vxvm/bin/vxprtvtoc -f /tmp/disk1 /dev/rdsk/c0t0d0s2
host # /usr/lib/vxvm/bin/vxprtvtoc -f /tmp/disk2 /dev/rdsk/c0t1d0s2
host # cat /tmp/disk1 /tmp/disk2 > /etc/vtoc

12. Update your system dump configuration to use the swap partition on the primary disk:

host # dumpadm -d /dev/dsk/c0t0d0s1
Dump content: kernel pages
Dump device: /dev/dsk/c0t0d0s1 (swap)
Savecore directory: /var/crash/host
Savecore enabled: yes

13. In the OBP, be sure to setup both disks using devalias (name them disk and rootdisk, if you want, or whatever makes the most sense to you) and set your configuration to have the boot-device set up to make it easy for your system to fail over if the primary disk goes bad:

ok> setenv boot-device disk, rootdisk, net

14. Be sure to test booting from the OBP for both disk and rootdisk. Assuming that goes well, you're all set :)

, Mike

Monitoring and Display Commands For LVM On Linux And Unix

Hey there,

Today, we're going to follow up on our previous post regarding getting started with LVM logical volume management and go over a few (perhaps) boring but essential commands you'll want to know in order to keep tabs on your setup both in good times and in bad.

Refer back to our initial post on getting started with LVM for any additional details, if you'd like them, but, for today's purposes, we're just going to use the outcome of that post as the simple setup that we'll be monitoring/displaying today. In a nutshell, we have one physical volume composed of one volume group which, in turn, contains only one logical volume (or you could think of it the other way around). Unless I'm completely off-center here today, our output from today should be extremely reminiscent of yesterday's ;)

Note: Based on your particular version, or vendor distribution of LVM, the output of these commands may be slightly different. Not so much so that it should make a significant difference.

Basically, our monitoring and displaying commands are broken down into three groups: Those dealing with "physical volumes," those dealing with "logical volumes" and those dealing with "volume groups." We'll step through them (not over and/or around them, even though that would be more polite ;) one by one (in twos):

Physical Volumes:

The two commands we'll be using here are pvscan and pvdisplay.

pvscan, as with all of the following commands, pretty much does what the name implies. It scans your system for LVM physical volumes. When used straight-up, it will list out all the physical volumes it can find on the system, including those "not" associated with volume groups (output truncated to save on space):

host # pvscan
pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- ACTIVE  PV "/dev/hda1"   is in no VG [512 MB]
...
pvscan -- ACTIVE  PV "/dev/hdd1" of VG "vg01"[512 MB / 266 MB free]
...

Next, we'll use pvdisplay to display our only physical volume:

host # pvdisplay /dev/hdd1 <-- Note that you can leave the /dev/hdd1, or any specification, off of the command line if you want to display all of your physical volumes. We just happen to know we only have one and are being particular ;)

...
PV Name   /dev/hdd1
VG Name   vg01
PV Size   512 MB
...

Other output should include whether or not the physical volume is allocatable (or "can be used" ;), total physical extents (see our post on getting started with LVM for a little more information on PE's), free physical extents, allocated physical extents and the physical volume's UUID (Identifier).

Volume Groups:

The two commands we'll be using here are vgscan and vgdisplay.

vgscan will report on all existing volume groups, as well as create a file (generally) called /etc/lvmtab (Some versions will create an /etc/lvmtab.d directory as well):

host # vgscan
vgscan -- reading all physical volumes (this may take a while...)
vgscan -- found active volume group "vg01"
...

vgdisplay can be used to check on the state and condition of our volume group(s). Again, we're specifying our volume group on the command line, but this is not necessary:

host # vgdisplay vg01
...
VG Name               vg01
...
VG Size               246 MB
...

this command gives even more effusive output. Everything from the maximum logical volumes the volume group can contain (including how many it currently does and how many of those are open), separate (yet similar) information with regards to the physical volumes it can encompass, all of the information you've come to expect about the physical extents and, of course, each volume's UUID.

Logical volumes:

The two commands we'll be using here are (you guessed it ;) lvscan and lvdisplay.

lvscan will give us a report on all of our logical volumes, and their state (ACTIVE/INACTIVE or possibly in ERROR) on our system, like so:

host # lvscan
...
lvscan -- ACTIVE            "/dev/vg01/lvol01" [246 MB]
lvscan -- 1 logical volumes with 246 MB total in 1 volume group
lvscan -- 1 active logical volume

Once this step has been completed (and it should be completed at least once!), we can use lvdisplay to check out our logical volume(s). We'll be specifically looking at lvol01:

host # lvdisplay lvol01
...
LV Name                /dev/vg01/lvol01
VG Name                vg01
...

and we'll get our standard UUID information, read/write access, availability status and information on the logical volume size (among other useful information, hopefully including the block device the logical volume is associated with :)

It should be noted that the logical volume size is one of the areas we generally look at first. If, for some reason, the output from "df" seems less than what you intended, this will give you a confident indication of whether or not your logical volume is too small, or you just need to grow the filesystem, resident on it, to use all of the logical volume's space!

...And that about wraps up the "information gathering" section of this series of posts on LVM. In future posts we'll be looking at ways to manipulate volumes (both in groups and individually in their physical and logical aspects; although the physical volume is actually a logical representation of a logical representation of a physical device... but I think we went into too much detail about that gordian knot in our previous post, and, even then, haven't begun to scratch the surface ;). In future posts we'll also be looking at LVM on Unix and Linux and the relationship between the two, which is becoming less complex every day!

Cheers,

, Mike

How To Get Started With Logical Volume Management In Linux

Hey again,

Today, since past posts regarding disk management software have mostly been limited to Veritas Volume Manager, I thought we'd look at LVM on Linux (In a future post, we'll check out the differences between LVM 1 and 2, but for now I figured we could keep it simple and broad).

If you use any of the "major player" Linux flavours out there, you've probably noticed (over the last year or so) that standard installs show all your disks listed out in df, like:

/dev/mapper/vg01-lvol01 246M   75.2M    158M 32%    /dir

rather than the old-style:

/dev/hdd1      246M   75.2M    158M    32%    /dir

that you've (if you're anything like me and only have two hard drives on the home PC you're running Linux on ;) grown to love.

And, while I think it's fantastic that Linux has grown into a mostly-user-friendly operating system with GUI's to make all of the extended partitioning and volume management more accessible, I also think there's something to be said for knowing how to be able to set all that stuff up on your own. You never know when your main display (or X-windows session) will die on you and, at some point, you may end up having to do some disk repair on the lonely old command line.

With that in mind, we'll step through the most basic LVM setup: Getting from /dev/hdd to /dev/mapper/vg01-lvol01 from the command prompt.

Our basic (and huge assumption) here is that we've got a primary drive to work from (although we could do this just as easily from a rescue, or live, CD-ROM if we needed to perform this work on our primary disk during the Linux installation process or a freakish-nightmare of a fix)

First, we'll want to use fdisk to create a "new" "primary" partition that consists of the entire disk (512Mb in size). Barring a primer on fdisk, we'll want to set up our disk to have one primary partition that consists of all cylinders (first and last should be the default start and end values) and is set to the "Linux LVM" partition type.

Once we're done with the stuff we'd "normally" do (for the most part) to set up any other type of partition table on our /dev/hdd disk, we can begin to play around with the LVM commands. Note that if you chose any partition type other than LVM when setting up your partition, using the following commands will, at best, result in error messages and accomplish nothing :)

If you've ever used HP-UX before, a lot of these commands will seem familiar. There's a reason, but the legalities of discussing it (and possibly getting bits of it wrong) fall outside the scope of my pocketbook ;)

The first thing we'll do is create a "physical volume," like so:

host # pvcreate /dev/hdd1

Yes, it was a physical volume already. The logical-physical relationship sometimes doesn't hold up to scrutiny outside of the "rules" ;)

Next, well create a "volume group." This is necessary, because "logical volumes" run in packs... pardon my lack of propriety ;) We could use the "-s" option here to define the "physical extent" size, but we'll leave that alone. To say the least, specifying this requires some research and planning. If you get that number wrong, things can go South later, and you'll have to do some ugly backtracking. We just want to define a "volume group" so we have something to work with. We'll call it vg01, for lack of a more original name.

host # vgcreate vg01 /dev/hdd1

Next, we'll create a "logical volume" (we'll name this one "lvol01" to keep things consistent) of 246 Megabytes size. That should fit nicely inside vg01 (which we're basically using to represent the entire partition /dev/hdd1). We could make our "logical volume" as large as the entire partition and/or "volume group", but that would defeat the purpose of this whole exercise. The aim, of course, is to end up with flexible volumes so we can resize them later if we need to.

host # lvcreate -L 246M -n lvol01 vg01

This is where the /dev/mapper stuff comes into play (it never made sense to me the first time I saw it... really. It's because of my past experience with HP-UX ;) The "lvcreate" command creates a symlink between /dev/vg01/lvol01 and the character device /dev/mapper/vg01-lvol01.

And, next, all we have to do is format that bad boy:

host # mkfs -t ext3 /dev/vg01/lvol01

Note that, above, you can set a reserved block count (based on percentage) using the -m flag. I would usually set this to "-m 5" so that 5% of the allocated space would be reserved for root (although you can set it as low as 1). This way, if someone jams your filesystem up to 100% and no one can get any work done, you can still log in as root and fix the issue. -v is another useful flag if you want to see verbose output. You can insert these two flags anywhere in the above command line except at the very end (after the logical device name) or in between the -t flag and its argument. Common sense, I know, but I feel compelled to over-explain ;)

Now, we'll make a mount point, or, really, just make a new directory off of the root filesystem (although you can do overlays if you wanna get creative ;)

host # mkdir /dir

and then we'll mount our logical volume up and check it out with df:

host # mount -t ext3 /dev/vg01/lvol01 /dir

host # df -h /dir
...
/dev/mapper/vg0-lvol0 246M   75.2M    158M 32%    /dir

Tomorrow, or maybe the day after that, we'll take a look at some more ways to manipulate your existing volumes, add more volumes, do mirroring and lots of other fun stuff that LVM allows you to do.

For now, we'll just thank our deity-of-choice that this all worked out ;)

Cheers,

, Mike

Safely Patching Your Veritas Root Mirror Disk On Linux Or Unix

Hey there,

It's been a long time since we've taken a look at anything "Veritas" (almost a few months now since we published a few posts regarding disk groups and volume groups in Veritas Volume Manager for Linux and/or Unix. Given the relatively broad nature of this blog, I sometimes wonder how we can ever stay entirely focused on any "one" thing for too long ;)

But, enough about us... For this "Lazy Sunday" post we're going to take a look at patching a root (or boot) mirror disk in VxVM safely. And by safely, we mean that you'll be able to fail-back to your root mirror disk as if nothing ever happened. That is, if something awful actually does happen.

The basic concept is simple, and applies to all brands and methods of root disk mirroring. When you're faced with having to apply patches to your OS (which invariably involves changes to your root disk), you always want to make sure that your root mirror is "golden" before you begin. You also want to make sure that it's taken out of the equation for the initial patch run, so you'll have a perfect failback device (less sweat, no accounting for tears ;)

The first thing you'll want to do, as per above, is to validate your root disk's mirror disk. For Veritas Volume Manager, every volume associated with the root disk must (well, technically, "should") have, at least, a single subdisk for each and every plex on the root disk and the root mirror disk.

For our example today, we'll consider that our root disk is c0t0d0s2 and its mirror is c1t0d0s02. They both belong to the default Veritas Volume: rootdg. Please also note that a lot of this output is "mocked up" to a certain degree since I'm not in a position to actually disassociate volumes on the computers I'm using for the sake of this post :)

You can check the state of your volumes with the "vxprint" command, like so (we'll use the ellipses (...) to indicate output that I've trimmed to keep this post under 50,000 words ;) :

host # vxprint -htqg rootdg <--- This output has been truncated as well, to highlight the mostly one-to-one relationship between subdisks (sd) and plexes (pl). As you can see, each of our two volumes on our rootdisk has at least one subdisk associated with each plex. We're going to ignore root_disk-B0 for this post (or not go into it too much) as this isn't really a "volume" but a way Veritas gets around the fact that it uses the part of the disk that most operating systems reserve (the bootblock - This, again, is enough material for another post entirely)

Disk group: rootdg 

dg rootdg       default ...
dm root_disk    c0t0d0s2 ...
dm root_mirror  c1t0d0s2 ...

sd root_diskPriv        - ...

v root_volume       - ...
pl root_volume-01   root_volume ...
sd root_disk-B0 root_volume-01 ...
sd root_disk-02 root_volume-01 ...
pl root_volume-02   root_volume ...
sd root_mirror-01       root_volume-02 ...

v swap_volume       - ...
pl swap_volume-01   swap_volume ...
sd root_disk-01 swap_volume-01 ...
pl swap_volume-02   swap_volume ...
sd root_mirror-02       swap_volume-02 ...

Now that we know we're good, even though it may have already been done, I find it's always good practice to install a new bootblock on the root mirror disk from the main root disk. The worst case scenario (assuming no typos ;) would be that you updated an existing bootblock with one that should, theoretically, be an exact match for your primary root disk (which is what we want) :

host # /usr/lib/vxvm/bin/vxbootsetup -g rootdg root_mirror

If you have other partitions on your root disk, that aren't listed in your vxprint output of the rootdg above, you can define them with the vxmksdpart command. You might have your /opt partition on the root disk, but not in the rootdg. Sometimes you'll see /home or even /var on the rootdisk but not associated with the rootdg. While it's considered "best practice" by Veritas to add these partitions to the rootdg before separating the disks, I've found that it's never actually been "necessary." The idea is that you associate the partitions, just so you can disassociate them a few minutes later (???)

Next, we'll disassociate (see what I mean ;) the root mirror disk plexes from the root disk, like so (you can verify that, for instance, swap_volume-02 is associated with the mirror disk in the vxprint output above):

host # vxplex -g rootdg dis root_volume-02
host # vxplex -g rootdg dis swap_volume-02

Now, well simply mount the root filesystem from the disassociated mirror disk on a temporary directory on the root disk and make a few quick file backups and edits, like so (Note that, for most Linux flavours, /etc/system noted below is actually /etc/sysctl.conf and /etc/vfstab is /etc/fstab):

host # mkdir /vxtmp
host # mount /dev/dsk/c1t0d0s0 /vxtmp
host # cp /vxtmp/etc/system /vxtmp/etc/system.old
host # cp /vxtmp/etc/vfstab /vxtmp/etc/vfstab.old
host # cp /vxtmp/etc/vfstab.prevm /vxtmp/etc/vfstab
host # touch /mnt/etc/vx/reconfig.d/state.d/install-db

Now, in the /vxtmp/etc/system file, we'll comment out the following two lines (remember that in the /etc/system file the "*" is the comment character. You probably already know that, but I feel responsible ;) -- Edit the following two lines so that they are now commented:

* rootdev ...
* set vxio ...

Then we'll unmount the root mirror disk on /vxtmp:

host # umount /vxtmp

and we're ready to patch! Assuming that everything goes swimmingly, all we need to do is reattach the root mirror disk plexes to the root disk, like this:

host # vxplex -g rootdg att root_volume root_volume-02
host # vxplex -g rootdg att swap_volume swap_volume-02

The root disk should sync itself up so that the root mirror disk gets updated (which you can monitor with "vxtask") And, you're all set :)

Now... If things go bad... The official explanation is so long and ridiculous (and differs for versions up to 3.5 and newer versions), that I'll refer you to an actual official document from Veritas online support that will show you a neat trick to get around having to jump through 15 or 16 hoops to get this all over with ;) Another glorious example of the system raging against itself :)

Cheers,

, Mike

Volume Groups Setup With Veritas Volume Manager

Hey There,

Today we're going to set up volume groups using the disks, and disk groups, that we setup yesterday in our post on creating disk groups with Veritas Volume Manager.

Our assumptions for the disks, and disk groups, we've setup are also spelled out in that same post about initializing disks and creating those disk groups. If you have any questions regarding the setup we're working from, they should be answered there :)

Now, we're ready to take the disk groups we created yesterday (again, not recommended to have all 3 of those exist at one time in reality. But highly recommended for the purposes of explaining the basic commands and concepts in today's post ;)

We're going to create the 4 most basic types of volumes today. These are referred to by their RAID (Redundant Array of Indepedent Disks) numbers, which translate as follows:

RAID 0 = Striped/Concatenated disk <--- Use this if you just want to combine any number of disks into one giant volume. Stripes of data are written across multiple disks, but this only enhances usability and doesn't provide any protection from disaster if a disk in the volume group fails.

RAID 1 = Mirrored disk <--- Use this if you want to mirror your disk, so that you have a simple mirror to fail over to if your primary disk fails.

RAID 5 = Striped disk with parity <--- Use this if you want to make your volume group highly available and recoverable. You'll need at least 4 disks in your RAID 5 group for this setup to be worthwhile, but it can make uptime and recovery a lot less of a hassle if you've got the disk to throw at your storage needs.

RAID 1+0 = Concatenated Mirrored disk <--- (as opposed to Mirrored Concatenated disk). The distinction to be made here is that RAID 1+0 (Striped Mirrors) consists of several disk mirrors, each reflection of which is concatenated/striped to create a larger volume. In RAID 0+1 (Mirrored Stripes), you have a set of disks Concatenated/Striped first and then Mirrored back to an equal amount of concatenated disk. Hopefully, I'm explaining this well enough, and not over-explaining it ;) Basically, with RAID 1+0, you can afford to lose a lot more disk and remain "up" than you can with RAID 0+1 (In RAID 0+1, loss of any disk A on one side of the mirror and disk C on the other side will bring you down, since each loss of a single disk, on either side, takes down that whole Concatenated miror. RAID 0+1 is like creating a much larger simple Mirrored disk than you could create with RAID 1).

For the following examples, we're going to use the command "vxassist." While you can use vxmake, etc, to perform these same functions (and, eventually, may have to if you need to do things more precisely), using vxassist is optimal when you're working with fresh disk and want to get up and running quickly :)

1. Creating a RAID 0 Volume Group named volume_group1 (From disk_group2, which contains hdc, hdd, hde and hdf):

host # /usr/sbin/vxassist -g disk_group2 -b make volume_group1 768000s layout=striped stripeunit=128 ncolumn=4

2. Creating a RAID 1 Volume Group name volume_group2 (from disk_group1, which contains hda and hdb):

host # /usr/sbin/vxassist -g disk_group1 -b make volume_group2 512000s layout=nostripe,log nmirror=2

3. Creating a RAID 5 Volume Group name volume_group3 (from disk_group3, which contains all 6 disks):

host # /usr/sbin/vxassist -g disk_group3 -b make volume_group3 1280000 layout=raid5,log stripeunit=32 ncolumn=6

4. Creating a RAID 1+0 Volume Group name volume_group4 (from disk_group3, which contains all 6 disks):

host # /usr/sbin/vxassist -g disk_group3 -b make volume_group4 768000s layout=stripe-mirror,log nmirror=2 stripeunit=128 ncolumn=2

As a bit of explanation, for each instance of vxassist that we invoke, you'll notice some common elements:

The "-g" flag simply asks us to name the disk group that we'll be using to create the volume group.

The volume group name (after "-b make") is simply what we want to call the particular volume group we're creating.

The following field is the size of the volume in sectors.

The field after that is the "layout" that we're looking to create. For RAID 0, we chose "striped." For RAID 1, "nostripe, log." For RAID 5, "raid5, log." And, for RAID 1+0, "stripe-mirror,log."

The "stripeunit" is only required for striped volumes and you can leave this out if you want to accept your VVM version's default.

The final column is also variable. For RAID 0, 5 and 1+0 we have the "ncolumn" attribute which basically translates to the number of disks in our volume group. For RAID 1 and 1+0 we have the "nmirror" attribute which translates to how many mirror devices we want to have. Two mirrors is actually the minimum you can have and really means "one disk, mirrored to one other disk." I initially found that confusing. Assuming I'm not the only guy who wondered if I was triple-mirroring by specifying two mirrors, I thought I'd just add that on there ;)

Hopefully this has been helpful to you. I'll be writing more on this, Solaris Volume Manager and LVM for Linux in posts to come :)

Best Wishes,

, Mike

linux unix internet technology

Setting Up Disk Groups In Veritas Volume Manager

Good day,

Today we're going to take a look at the very basics of getting started with Veritas Volume Manager. We're not going to limit ourselves to any specific operating system, as all of these commands work almost exactly the same on Unix and Linux. We'll be using RedHat disk identifiers, for brevity's sake, but it should be noted that wherever we list out a disk like "sda," you can substitute c0t0d0 (or what not) if you're using Solaris. Of course, if you're running RedHat, you're probably already using LVM, in which case VVM might not be worth the hefty asking price.

Tomorrow, we'll take a look at creating volumes with these disk groups , but for now, I thing we'll be chewing up sufficient HTML landscape just running down these basic commands ;) We'll be concentrating on the command line functionality in this small series of posts. You can do all of this stuff using the "vxdiskadm" command also, which is a tty gui that pretty much walks you through the steps very easily in plain English.

Our assumptions for today's exercise (and tomorrow's), are that you have 6 disks on your system of equal size, manufacturer, etc, and that you've already installed and (at least temporarily) licensed Veritas Volume Manager on the OS. Our only other obvious assumption will be that you may or may not have installed the Veritas File System. These commands should work equally well for vxfs, ufs, ext3, etc.

The first thing you'll want to do to set up your disk group(s) is to initialize all of your disks so that Volume Manager recognizes them. This is very simply done on the command line, like so, for our six disks (/dev/hda, /dev/hdb, /dev/hdc, /dev/hdd, /etc/hde, /dev/hdf):

host # /etc/vx/bin/vxdisksetup -if hda
host # /etc/vx/bin/vxdisksetup -if hdb
host # /etc/vx/bin/vxdisksetup -if hdc
host # /etc/vx/bin/vxdisksetup -if hdd
host # /etc/vx/bin/vxdisksetup -if hde
host # /etc/vx/bin/vxdisksetup -if hdf

Note: For all of these operations, you can check on the progress (if you run a command that takes a good while to complete) using the "vxtask" command!

It's generally good practice to include /etc/vx/bin in your PATH environment variable (even on Linux, for backward compatability's sake). At the very least, it will save your fingers from type-cramping ;)

The command line options we've added here, before the disk name declarations (which don't need to be fully qualified with /dev... ), are "-i" to initialize the disks (by writing a disk header instead of just creating a partition for VVM use) and "-f" to force the initialization (This is technically an option to the "vxdisk" command, but can be used with "vxdisksetup").

Now, to create your disk group, or groups, you just simply have to run the following. Let's say we want to create two disk groups (disk_group1 and disk_group2); the first with hda and hdb, and the second with the remaining disks we've initialized. We can take care of that by executing:

host # /usr/sbin/vxdg init disk_group1 disk_group1_01=hda disk_group1_2=hdb
host # /usr/sbin/vxdg init disk_group2 disk_group2_01=hdc disk_group2_2=hdd disk_group2_3=hde disk_group2_4=hdf

We'll also create a disk group , called disk_group3, consisting of all 6 disks (Of course, it's not recommended to have any single Veritas disk belong to more than one disk group, but we're going to do it here, just so we have a disk group that we can use in tomorrow's examples on setting up volume groups).

hosts # /usr/sbin/vxdg init disk_group3 disk_group3_01=hda disk_group3_2=hdb disk_group3_3=hdc disk_group3_4=hdd disk_group3_5=hde disk_group3_6=hdf

And we're all set. Now all six of our disks are initialized for use with Veritas Volume Manager and we've got them ordered into disk groups.

Tomorrow, we'll take a look at creating different types of volumes from the disks, and disk groups, that we've set up today.

Until then,


, Mike

linux unix internet technology

Adding Storage using Volume Manager in a Veritas Cluster

Today, we're going to take a look at adding extra disk to a Veritas Cluster using Veritas Volume Manager. We'll assume for the purposes of this post, that you've been asked to add 35gb of space to a Veritas volume (filesystem: /veritas/a - volume name: veritas_a). Now we'll walk, step by step, through determining whether we have the disk available to add, and adding it if we do.

The first thing you'll want to do is determine what disk group(s) the filesystem/volume belongs to. Generally, you'll be told this when you're asked to add disk to an existing system, but we'll just assume the very worst ;) In any event, even if you are told, it's good practice to verify that the filesystem/volume does, in fact, belong to the disk group(s) the request is asking to have you augment.

Now, since we're dealing with a Veritas Cluster, if the disk group is shared between more than one server node, you can run:

vxdctl -c mode

and that will show you what the master node is. This server is where you'll want to execute all of your Veritas Volume Manager commands. If the cluster is inactive, vxdctl will tell you that, and you'll need to execute your commands on the server you're already on.

Now, either on the Cluster's master node, or the machine you're on if the cluster is inactive, get a list of all the disk groups, like so:

vxdg list
NAME STATE ID
rootdg enabled 1199919191.1025.host1
mydg enabled 8373777737.1314.host2

Since the user requested more filespace under /veritas/a, we'll need to see what disk group this filesystem/volume belongs to:

df -kl | grep veritas
/dev/vx/dsk/mydg/veritas_a 34598912 24632328 9903320 72% /veritas/a
/dev/vx/dsk/mydg/veritas_b 34598912 649112 33687984 2% /veritas/b
/dev/vx/dsk/mydg/veritas_c 35350080 17696 35056360 1% /veritas/c

We can see that, under /dev/vx/dsk, mydg is showing as the disk group for our /veritas/a filesystem, and veritas_a is showing as the volume. In cases like this, it's a time saver if you just set the default group once on the command line, so you don't have to use the "-g" option for every command you run:

vxdctl defaultdg mydg

Now you'll run some standard commands to make sure that your system can see all the disk available to it and make sure that the data is fresh:

devfsadm -Cv (note that the cluster will only update the volume manager configuration for cluster nodes that use the mydg diskgroup -- we don't have to specify that here, because we defined it as the default above)
vxdctl enable (this is basically the second part of the same update. The devfsadm command is Solaris specific while vxdctl is Veritas specific)

Now we need to check for free space, to see if we can accommodate the request with what we have available to us at present:

vxdg -g mydg list <-- Again "-g mydg" is optional because it's the default.
DISK DEVICE TAG OFFSET LENGTH FLAGS
DISK1 c5t27d27s2 c5t27d27 69198720 1501440 n
DISK2 c5t27d30s2 c5t27d30 69198720 1501440 n
DISK3 c5t27d28s2 c5t27d28 69198720 1501440 n

This output is, admittedly, a pain to read, because it expects you to be able to calculate disk offsets in your head. For this output, if the offset were zero and length was 70700160 and flags were '-' then that DISK would have 35 GB of free space available to us. Here all the listed storage has been partially used up. Hence, there is not enough storage space available!. We can also verify this by running

vxdg free (optionally: vxdg -g mydg free) to see if there is any free space left in the disk group. Either way, we're out of luck, it would seem.

So, now we're going to have to look into Veritas' disk configuration and see if there isn't a spare disk (hope) that we can pick up and import into the mydg diskgroup so that we can add the space (One thing to note in the output below is that all online disk shows its disk group membership even though it's imported locally):

vxdisk -o alldgs list <--- (abbreviated to keep this post bearable ;)
DEVICE TYPE DISK GROUP STATUS
c0t0d0s2 sliced - - error
c0t2d0s2 sliced disk02 rootdg online
c1t2d0s2 sliced disk01 rootdg online
c5t27d40s2 sliced - (mydg200) online
c5t27d41s2 sliced - (mydg2) online
c5t27d300s2 sliced - (mydg300) online
c5t20d99s2 sliced - - error

Luckily, it appears that disk c5t20d99s2 is available and doesn't belong to any existing disk groups! We'll use the command line to initialize this free disk and add it to our disk group (mydg):

vxdiskadd c5t20d99s2

You will be asked if you want to initialize the disk (or re-initialize it, if it's already initialized). Answer yes to either option, to keep things simple. Then it will ask you what disk group you would like to add the disk to. Answer: mydg. It will ask you if you want to use a default disk name. Depending on your situation, this may be okay, but we'll say no and add it as "DISK4" to keep with our naming standard. Then you will be prompted if you want to add the disk as a spare for the mydg disk group. Answer no to this, or you won't be able to use it for the additional storage you want! Answer no again, when it asks you if you want to encapsulate the disk, and then answer yes when it asks you if you want to initalize it. You can then choose to perform a surface analysis before adding it (It says it's highly recommended but my experience is it's generally not necessary and takes a very very very long time). Phew... that wasn't too much to info to have to feed to a single command ;)

Now, we'll want to grow the mydg disk group's volume and filesystem size. We've added the new disk, so we have more space, but the user can't see it or use it at this point. Now, common wisdom is to use the vxassist command. It's common, because it can be a great command to use (very simple), but I prefer to use "vxresize" because, unless your filesystem is vxfs, running vxassist to grow the volume (/veritas_a) can cause serious issues when growing the filesystem (if, for instance, you're using the Veritas Volume Manager but have your volume filesystem type set to Solaris "ufs")!

Calculate the maximum length of your new volume using vxassist (this part is okay - forget about what I just said above - that only applies to using vxassist to do the actually growing):

vxassist -g mydg (optional, if default is already set) maxsize <--- This will be the NEW_LENGTH argument for the next command.

vxresize -F (vxfs or ufs,usually) -g mydg (optional, still) -x (to make sure that the operation fails if the volume doesn't increase in length) veritas_a NEW_LENGTH <-- The last time I checked, - and another reason I prefer this method - this command grows your filesystem with the volume so you don't need to run any more steps after this.

Of course, you can keep tabs on all this, after you're done typing, using the vxtask command. At no point during this entire operation will you need to unmount your filesystem. I believe it's "recommended" (because something could always go wrong), but I've never found it to be necessary (technically, it's not supposed to fail, but Veritas has to cover itself, just in case)

Once the task is finished, your "df -k /veritas/a" output should show your new disk available and ready for use!

Cheers,

, Mike

linux unix internet technology

Using Solaris Volume Manager To Help With Server Moves

Howdy,

For the basics of setting Solaris Volume Manager up for basic root disk mirroring, you can refer back to an older post on this site (just search for those bolded terms) or try this link.

Here we're going to do discuss another good use for Solaris Volume Manager mirroring: Server moves. Especially server moves where you're required to migrate the server from one server platform to another (Say, from a 6800 to a v490 or something like that).

Using Solaris Volume Manager, it's very simple to set yourself up to be able to roll back all your changes if your migration fails, without losing anything in the process (except maybe a lot of your time since, if it fails, you'll have to do it again at some point).

Sucessfully rolling back is paramount to any server move, or project. If you can't go back to the way things were once you begin to enact a change, you're taking a huge risk if anything goes wrong!

Basically, you'll be guaranteeing your security in three simple steps. Our assumption is that your basic mirroring setup is as such: metadevice d0 consists of metadevice mirrors d10 and d20, d1 consists of d11 and d21, etc)

1. Assuming a simple disk-to-disk mirror setup, detach the secondary mirror for each slice on the root disk (of course, this would apply to all other disks and slices, but I'm trying not to take up too much space today, so we'll stick to the first two slices on the first two disks ;)

metadetach d0 d20
metadetach d1 d21

2. Now remove the secondary disk (leaving in the one that's actively running your system!) and replace it with a new third disk. Format this disk exactly the same as the disk already in the system (using prtvtoc and fmthard, or just using select in the format partition command menu) and create your tertiary mirrors.

metainit -f d30 1 1 c0t1d0s0
metainit -f d31 1 1 c0t1d0s0

Note that this disk is in the same logical disk location as your original mirror disk. That's not a problem and will work to your benefit when you put back the original mirrors.

3. Now attach those new metadevices to the original mirror devices and wait while they sync up your primary disk

metattach d0 d30
metattach d1 d31

Once you're done with this, simply go through this whole process again, but detach the new mirrors, remove your new disk, replace your old disk and reattach the old mirrors (you won't have to metainit your old mirrors, since SVM still thinks they exist but aren't in use).

Now you have a disk that, should everything go horribly wrong, you can pop back in your original server and get things right back to the way they were. You may need to re-install the bootblock on your backup disk (from a single user cd boot), but this will generally work without extra effort :)

, Mike

linux unix internet technology

Basic Root Disk Mirroring With Solaris Volume Manager

Hello again!

Solaris Volume Manager (SVM - Used to be called Solaris Disk Suite - SDS - pretty much only the name changed) is an excellent OS standard software RAID management tool used by a lot of IT shops. Even shops that use Veritas Volume Manager software for serious storage management, still use SVM to take care of the base disks.

The reason most shops utilize SVM to manage the root disks is that it's easier to manage than Veritas for this limited purpose. Another reason is that, in coordination with its ease of use, it's integrated into Solaris. The thinking around this is generally: Why not use Solaris' tool to manage our Solaris OS root disk(s)? If you've ever had to deal with Sun support, you know another reason why using their product on their product is a good thing :)

Setting up RAID groups, and managing them, is generally very simple and doesn't require rebooting the machine, etc, when you want to add new disk (with some exceptions). One of the exceptions to this rule is when you use SVM to set up management of your root disks.

Managing your root disks with SVM is a two-part process. Generally, it's used for mirorring the root disk for quick failover and is configured right after installation. Configuring it after is also easy, but I like to get my necessary reboots completed before I hand over product to a customer.

The process is fairly simple and is accomplished like this:

1. The two disks that will be used for mirroring have to be formatted "exactly" alike. The easiest way to do this is to partition one root disk and then either use the output from "prtvtoc" to seed "fmthard" or, more simply, use Solaris' "format" command to format the initial disk and then "select" the same layout from the list of available partition setups when formatting the second disk (labeling the first makes its partition table a selectable option!)

2. Meta Databases need to be created on both disks. The first needs to be forced, and any additional do not. We're using slice 7 for our example here:

metadb -a -f /dev/dsk/c0t0d0s7
metadb -a /dev/dsk/c0t1d0s7

3. Next, the metadevices need to be setup. For each slice that we want to mirror, we need to create two stripe-concat metadevices, as such (shown for only two, here, but done for every slice we want to mirror - which should be every slice on the disk except slice 2!):

metainit -f d10 1 1 c0t0d0s0
metainit -f d20 1 1 c0t1d0s0

metainit -f d11 1 1 c0t0d0s1
metainit -f d21 1 1 c0t1d0s1

4. Next, the metadevice that will be composed of the two mirrors needs to be created. You can do this by attaching one of the mirrors to the new metadevice. Note that, you can do this all at once, normally, but for the root disk, you can only attach one mirror slice at this point for each slice, as such:

metainit d0 -m d10
metainit d1 -m d11

5. The root partition is special in that SVM actually has a command to change its value to the metadevice value (d0) in /etc/vfstab. It also adds some information in /etc/system for you. That command is:

metaroot d0

6. Now, sadly enough you'll need to edit your /etc/vfstab to reflect the new metadevices. So for the entry for metadevice d1, you'd change:

/dev/dsk/c0t0d0s1 to /dev/md/dsk/d1
/dev/rdsk/c0t0d0s1 to /dev/md/rdsk/d1

7. Once that's completed and all commands have returned successfully (you can check that all's well by running "metastat -p") you will need to reboot. I would suggest actually shutting your machine down to init level 0 so that you can change your alternate boot device to the secondary (mirror) disk from the default of "net." This way, once you're set up, if your primary disk fails your system will automatically boot off of the mirror.

8. Now to the final step. You need to attach that second mirror slice to your mirror metadevice, as such:

metattach d0 d20
metattach d1 d21

Again, as long as you receive no errors, you should be all set. It will take SVM a little while to sync up the two disk (the actual mirroring process from the first disk to the second). You can check on the status by running "metastat" on its own. I personally prefer to run a command line loop to keep me up to date, like:

while true;do metastat -t|grep -i sync;sleep 60;done

then, when all the sync's are complete, I'll know right away. It's always nice to know when you can stop worrying :)

, Mike

linux unix internet technology