How Avoid Solaris Panics When Using Savecore With Veritas Volume Manager

Hey There,

Ever since Solaris 7, the savecore command has been able to act somewhat like "gcore," but better. The introduction of the "-L" flag allowed users to take a crash dump of the entire running Solaris system (much in the same way gcore does for individual processes). When invoked with this flag, savecore will take a snapshot of the live system and write it to the system dump device. This dump device has to be "dedicated," and defined in /etc/dumpadm.conf. Out of the box, this dedicated dump device is set to your system swap partition, so you may have it set up correctly even if you've never given it a second thought (or a first ;). This, in and of itself, isn't anything spectacular, since you can use "fssnap" to do pretty much the same thing and have finer control over what gets "snapped" and where it gets put. The beauty of "savecore -L /some/other/output/directory" is that, after writing the snapshot to your dedicated dump device, it will suck up that data and create crash dump files wherever you've instructed it to. It's kind of like a combination of "fssnap" and "gcore," if you could manage to get those two programs to cooperate with each other and create a crash dump of your live system.

Of course, the point of today's post isn't to debate the merits of using this feature or even (as it may seem) glorifying it in any way. The preceding paragraph was simply meant as introduction to a point-of-disaster that is still just waiting to happen on Solaris boxes all over the world. If you prefer to keep your machines at the "ok>" prompt, this post may not be for you.

The disaster, itself, is not brought on solely by savecore and the fault doesn't really lie with the program itself. However, if you opt to manage your filesystems with Veritas Volume Manager, depending on how you go about things, the combination of the two can be "interesting" at best ;) Of course, in order to cause this panic/crash/freak-out to happen, things in your Solaris dumpadm configuration and Veritas Volume Manager setup have to be constructed in a very specific way. Unfortunately, that very specific way is the default for a lot of shops that use Volume Manager to encapsulate the root disks on their systems. That, and having a default (or even a modified dumpadm.conf that doesn't stray too far from convention) is all that's required. If this situation could only occur on the third Tuesday of every month, on alternate leap years during a "classic syzygy" (when the sun moon and earth lie in a straight line with one another), I probably wouldn't be writing this post right now. So, in a way, it's a good thing, because I was initially going to make today's bullet-list about 50 ways you can save money, and avoid responsibility, by catching up on sleep ;)

So, after that lengthy preamble, here's a breakdown of one way to "break down" your Solaris system running VVM, step by step:

1. Ensure that your dumpadm.conf is "normal": This is a default /etc/dumpadm.conf (put together for us by Solaris during the installation process):

host # cat /etc/dumpadm.conf
#
# dumpadm.conf
#
# Configuration parameters for system crash dump.
# Do NOT edit this file by hand -- use dumpadm(1m) instead.
#
DUMPADM_DEVICE=/dev/dsk/c0t0d0s1
DUMPADM_SAVDIR=/var/crash/host
DUMPADM_CONTENT=kernel
DUMPADM_ENABLE=yes

It looks good. According to the way things have always been done, it's just about perfect. Solaris, as far back as I can remember, has always recommended using your swap partition as your dump device if you enabled savecore (which you had to manually set up before Solaris 7), and the save directory is the default that it's always been (/var/crash/yourhostname).

NOTE: If you stray from this setup by changing your DUMPADM_DEVICE to another unused partition (preferably on a separate disk), you'll never experience this disaster (at least not for the same reasons...)

2. Encapsulate your root disk using Veritas Volume Manager: You may recall, from our past post on how to mirror your root disk using Veritas that our setup went through pains to make sure that the end product was "not" encapsulated. The reasons for this were different, but, if you followed those (or other similar) instructions, you won't be able to recreate this exact "magnificent failure" either.

3. Run "savecore -L /whatever/directory/you/want" on a machine in which the above two conditions exist.

And, that's it! Your results may vary, but general system panic is the most accurate way to describe the plethora of confusing hex error messages and resulting system crash that will most likely occur.

Now, let's take a look at the reasons "why" this can happen:

1. If you're using Veritas Volume Manager and your root disk (assuming this is where your DUMPADM_DEVICE, or swap partition, resides) is "not" encapsulated, dumpadm would recognize that the device it has listed as a dump location is being used by Veritas and can't be used by itself. This results in dumpadm refusing to run "savecore -L." Since the root disk is encapsulated, dumpadm has no idea that the DUMPADM_DEVICE is being accessed by Veritas through a different logical device. Basically, Solaris checks the swaplist and determines that the DUMPADM_DEVICE is not listed as a swap device (it "is" a swap device, but encapsulation only considers the Volume to be a swap device, and not the partition associated with it). Since the partition underlying the Veritas Volume is not listed as a swap device, it shows up as "dedicated." Therefore, your "savecore -L" command is processed as though all were well.

2. Actually, this is just an extension of point 1. But, the disaster happens when you run "savecore -L." Since the DUMPADM_DEVICE is pointing to the same region of disk that the Veritas swap Volume is located, when the command gets run, information stored in the swap partition and /tmp filesystem can easily become corrupted. It's virtually guaranteed to, since Solaris and Veritas are both acting independently of each other and reading/writing from the exact same place (both assuming they're the only one with exclusive access).

This problem is highly prevalent in shops that use "defaults" and recommended "best practices." (e.g. the default /etc/dumpadm.conf on a Solaris box directly falls into contradiction with Veritas Volume Manager's preference toward encapsulating the root disk). Fortunately, it is very easy to fix in one, or a few ways:

1. Change your /etc/dumpadm.conf file so that the DUMPADM_DEVICE is on a disk other than the root disk (or its mirror, if you have that set up). /usr/sbin/dumpadm only runs when it's called, so that's all you have to do. Veritas (I mean Symantec) themselves recommend that you "never" run "savecore -L" on an encapsulated root disk.

2. Follow the instructions in our old post on mirroring your root disk with Volume Manager and dumpadm will be able to determine that its default DUMPADM_DEVICE can't be used, which will nudge you toward implementing the first solution if you really want to make use of this aspect of savecore's functionality.

...and then any of various combinations of the two (given the variety of site installations of Solaris and different ways folks like to set their disks up with VVM). All you really have to do is avoid just one of these specific conditions that will almost always cause the problem.

Here's to not spending our nights and weekends at work :)

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Mirroring Your Boot Disk Using Veritas On Sun Without Encapsulation

Hey There,

Today, we're going to take a look at a using Veritas to mirror your boot disks, without using encapsulation. There's raging debate about whether or not Veritas root disk encapsulation is bad (some folks are great with it because it takes care of everything, while some folks hate it because it's very hard to deal with when it doesn't take care of everything ;)

The following procedure was introduced quite a while ago by a pair of gentleman named Gene Trantham and John S. Howard in an article they wrote for Sun Blueprints Online (A site run by Sun that highlights best-practices for many, many things Solaris). I believe it's now simply referred to as the "Trantham/Howard Method."

Although I've used this method many times in my career, there's no point in re-writing the instructions (although I did, to a certain degree, to make them more concise and accessible). I did, however, want to have this information included on my blog, to complement (and complete) our earlier posts on patching your Veritas root disk, Veritas Volume Group setup, etc. The full document can be read here in the August 2000 Online Edition of Sun Blueprints.

So, without further ado, and, hopefully you'll find this helpful as well, the "Trantham/Howard Method" for mirroring boot disks using Veritas on Sun Solaris (somewhat modified and truncated):

Enjoy,

1. Install the basic Veritas Volume Manager Software.

2. Install any required Veritas licenses.

3. Use vxinstall to encapsulate the root disk c0t0d0 (Or whatever cxtxdx disk name accurately reflects your setup). Choose not to encapsulate the root disk and don't mirror it. For this post's sake, name it rootdisk Once the root disk is encapsulated, simply reboot and check "vxprint -ht" - You should note that only your rootdisk (and its Veritas name) is listed out.

4. Initialize the root mirror disk c0t1d0 (again, substitute with your own specific controller-target-disk name).

host # /usr/lib/vxvm/bin/vxdisksetup -i c0t1d0
host # vxdg adddisk rootmirror=c0t1d0

5. Manually attach the mirror plexes to the root volumes with "vxrootmir" (manually mirror the swapvol, usr and var volumes, if necessary, as well):

host # /usr/lib/vxvm/bin/vxrootmir rootmirror
host # vxassist mirror swapvol alloc=rootmirror
host # vxassist mirror usr alloc=rootmirror
host # vxassist mirror var alloc=rootmirror

While you wait for this to finish, you can check on the progress with the vxtask command.

6. Once your root disk is mirrored, disassociate the original plexes on c0t0d0, remove them, and remove the encapsulated rootdisk from the rootdg diskgroup:

host # vxplex dis rootvol-01 swapvol-01 var-01 usr-01
host # vxdg rmdisk rootdisk

Now your operating system is entirely running off of the c0t1d0 disk.
At this point, the operating system is based just on the initialized disk c0t1d0, which you can see by looking at the output of "vxprint -ht" again.

7. Now initialize, as opposed to encapsulating, c0t0d0:

host # /usr/lib/vxvm/bin/vxdisksetup -i c0t0d0
host # vxdg adddisk rootdisk=c0t0d0

8. Now, mirror everything from the rootmirror disk back to the rootdisk:

host # /usr/lib/vxvm/bin/vxrootmir rootdisk
host # vxassist mirror swapvol alloc=rootdisk
host # vxassist mirror var alloc=rootdisk
host # vxassist mirror usr alloc=rootdisk

Again, you can use the vxtask command to view the progress of this command, which may take some time to complete. Also, this process removes the "rootdisk-B0" subdisk. This is used by Veritas to protect the boot block, but isn't necessary when you're not encapsulating your disks.

9. Using the output of "vxprint -ht", create disk slices corresponding to the Veritas subdisks. The usage for vxmksdpart command is:

vxmksdpart [-f] [-g diskgroup] subdisk sliceno [tag flags]

You won't need to run through this for the root slice since vxrootmir has already done this for you. You will need to do this for the other slices, though. But, don't use vxmksdpart just yet.

10. On the "new" root disk, first create the slices for the partitions you're going to mirror:

host # /usr/lib/vxvm/bin/vxmksdpart rootdisk-02 1 0x03 0x01
host # /usr/lib/vxvm/bin/vxmksdpart rootmirror-02 1 0x03 0x01

and so on for each required partition (again, you can get this output from "vxprint -ht"). The "sliceno" part of the command syntax is very important here!

host # /usr/lib/vxvm/bin/vxmksdpart rootdisk-03 5 0x07 0x00
host # /usr/lib/vxvm/bin/vxmksdpart rootmirror-03 5 0x07 0x00
...

11. Among other things (like backing up /etc/vfstab, etc) be sure to capture the current disk partition information for both disks, before rebooting, using vxprtvtoc:

host # /usr/lib/vxvm/bin/vxprtvtoc -f /tmp/disk1 /dev/rdsk/c0t0d0s2
host # /usr/lib/vxvm/bin/vxprtvtoc -f /tmp/disk2 /dev/rdsk/c0t1d0s2
host # cat /tmp/disk1 /tmp/disk2 > /etc/vtoc

12. Update your system dump configuration to use the swap partition on the primary disk:

host # dumpadm -d /dev/dsk/c0t0d0s1
Dump content: kernel pages
Dump device: /dev/dsk/c0t0d0s1 (swap)
Savecore directory: /var/crash/host
Savecore enabled: yes

13. In the OBP, be sure to setup both disks using devalias (name them disk and rootdisk, if you want, or whatever makes the most sense to you) and set your configuration to have the boot-device set up to make it easy for your system to fail over if the primary disk goes bad:

ok> setenv boot-device disk, rootdisk, net

14. Be sure to test booting from the OBP for both disk and rootdisk. Assuming that goes well, you're all set :)

, Mike