Safely Patching Your Veritas Root Mirror Disk On Linux Or Unix

Hey there,

It's been a long time since we've taken a look at anything "Veritas" (almost a few months now since we published a few posts regarding disk groups and volume groups in Veritas Volume Manager for Linux and/or Unix. Given the relatively broad nature of this blog, I sometimes wonder how we can ever stay entirely focused on any "one" thing for too long ;)

But, enough about us... For this "Lazy Sunday" post we're going to take a look at patching a root (or boot) mirror disk in VxVM safely. And by safely, we mean that you'll be able to fail-back to your root mirror disk as if nothing ever happened. That is, if something awful actually does happen.

The basic concept is simple, and applies to all brands and methods of root disk mirroring. When you're faced with having to apply patches to your OS (which invariably involves changes to your root disk), you always want to make sure that your root mirror is "golden" before you begin. You also want to make sure that it's taken out of the equation for the initial patch run, so you'll have a perfect failback device (less sweat, no accounting for tears ;)

The first thing you'll want to do, as per above, is to validate your root disk's mirror disk. For Veritas Volume Manager, every volume associated with the root disk must (well, technically, "should") have, at least, a single subdisk for each and every plex on the root disk and the root mirror disk.

For our example today, we'll consider that our root disk is c0t0d0s2 and its mirror is c1t0d0s02. They both belong to the default Veritas Volume: rootdg. Please also note that a lot of this output is "mocked up" to a certain degree since I'm not in a position to actually disassociate volumes on the computers I'm using for the sake of this post :)

You can check the state of your volumes with the "vxprint" command, like so (we'll use the ellipses (...) to indicate output that I've trimmed to keep this post under 50,000 words ;) :

host # vxprint -htqg rootdg <--- This output has been truncated as well, to highlight the mostly one-to-one relationship between subdisks (sd) and plexes (pl). As you can see, each of our two volumes on our rootdisk has at least one subdisk associated with each plex. We're going to ignore root_disk-B0 for this post (or not go into it too much) as this isn't really a "volume" but a way Veritas gets around the fact that it uses the part of the disk that most operating systems reserve (the bootblock - This, again, is enough material for another post entirely)

Disk group: rootdg 

dg rootdg       default ...
dm root_disk    c0t0d0s2 ...
dm root_mirror  c1t0d0s2 ...

sd root_diskPriv        - ...

v root_volume       - ...
pl root_volume-01   root_volume ...
sd root_disk-B0 root_volume-01 ...
sd root_disk-02 root_volume-01 ...
pl root_volume-02   root_volume ...
sd root_mirror-01       root_volume-02 ...

v swap_volume       - ...
pl swap_volume-01   swap_volume ...
sd root_disk-01 swap_volume-01 ...
pl swap_volume-02   swap_volume ...
sd root_mirror-02       swap_volume-02 ...

Now that we know we're good, even though it may have already been done, I find it's always good practice to install a new bootblock on the root mirror disk from the main root disk. The worst case scenario (assuming no typos ;) would be that you updated an existing bootblock with one that should, theoretically, be an exact match for your primary root disk (which is what we want) :

host # /usr/lib/vxvm/bin/vxbootsetup -g rootdg root_mirror

If you have other partitions on your root disk, that aren't listed in your vxprint output of the rootdg above, you can define them with the vxmksdpart command. You might have your /opt partition on the root disk, but not in the rootdg. Sometimes you'll see /home or even /var on the rootdisk but not associated with the rootdg. While it's considered "best practice" by Veritas to add these partitions to the rootdg before separating the disks, I've found that it's never actually been "necessary." The idea is that you associate the partitions, just so you can disassociate them a few minutes later (???)

Next, we'll disassociate (see what I mean ;) the root mirror disk plexes from the root disk, like so (you can verify that, for instance, swap_volume-02 is associated with the mirror disk in the vxprint output above):

host # vxplex -g rootdg dis root_volume-02
host # vxplex -g rootdg dis swap_volume-02

Now, well simply mount the root filesystem from the disassociated mirror disk on a temporary directory on the root disk and make a few quick file backups and edits, like so (Note that, for most Linux flavours, /etc/system noted below is actually /etc/sysctl.conf and /etc/vfstab is /etc/fstab):

host # mkdir /vxtmp
host # mount /dev/dsk/c1t0d0s0 /vxtmp
host # cp /vxtmp/etc/system /vxtmp/etc/system.old
host # cp /vxtmp/etc/vfstab /vxtmp/etc/vfstab.old
host # cp /vxtmp/etc/vfstab.prevm /vxtmp/etc/vfstab
host # touch /mnt/etc/vx/reconfig.d/state.d/install-db

Now, in the /vxtmp/etc/system file, we'll comment out the following two lines (remember that in the /etc/system file the "*" is the comment character. You probably already know that, but I feel responsible ;) -- Edit the following two lines so that they are now commented:

* rootdev ...
* set vxio ...

Then we'll unmount the root mirror disk on /vxtmp:

host # umount /vxtmp

and we're ready to patch! Assuming that everything goes swimmingly, all we need to do is reattach the root mirror disk plexes to the root disk, like this:

host # vxplex -g rootdg att root_volume root_volume-02
host # vxplex -g rootdg att swap_volume swap_volume-02

The root disk should sync itself up so that the root mirror disk gets updated (which you can monitor with "vxtask") And, you're all set :)

Now... If things go bad... The official explanation is so long and ridiculous (and differs for versions up to 3.5 and newer versions), that I'll refer you to an actual official document from Veritas online support that will show you a neat trick to get around having to jump through 15 or 16 hoops to get this all over with ;) Another glorious example of the system raging against itself :)

Cheers,

, Mike