Locating New Backup Hardware Using Veritas NetBackup On The Solaris Unix Command Line

Hey there,

Yesterday we took a look at the basics of using Veritas/Symantec NetBackup to add a new TLD and drives to your existing machine. Today, we're going to go just one step beyond and assume a fairly commonplace situation, which has somehow, inexplicably arisen from THE SITUATION we found ourselves in yesterday. For some reason (and this hardly ever happens ...not sure which word to emphasize to obtain the maximum sarcastic drippage) after we connected our new Tape Loading Device (TLD or Tape Robot), and the two drives it contains, to our backup server, NetBackup - and, possibly, the server itself, is failing to recognize the new device(s). Again, we're going to assume that both the server, TLD, drives and all other hardware are absolutely fine and that all required connections between the devices are set up properly.

NOTE: Today's post is going to assume that some tried and true methods will get you to "good." Tomorrow's post will look at some other ways to make NetBackup recognize and work with your "known good" (and compliant) setup.

If we take the same direct route to initial discovery that we did yesterday, we'd run the same sgscan (which is, as one reader noted, shorthand for "sgscan all") command initially, like so (pardon the error output. I can't afford to create the situation I want to display so I'm doing it from memory):

host # /usr/openv/volmgr/bin/sgscan
/dev/sg/c0t0l0: Disk (/dev/rdsk/c0t0d0): "SUZUKI MBB2147RCSUN146G"
/dev/sg/c0t1l0: Disk (/dev/rdsk/c0t1d0): "SUZUKI MBB2147RCSUN146G"
/dev/sg/c0t2l0: Tape (???): "Unknown"
/dev/sg/c0t3l0: Cdrom: "Hyundai    DV-W28E-R"
/dev/sg/c1t0l0: Changer: "Unknown"
/dev/sg/c1t1l0: Tape (???): "Unknown"

Basically, every line where it says "Unknown" is where we're interested in looking. The system can't find our TLD or its drives, so now we have to try to discover them ourselves (with and/or without NetBackup) and then come back around and use NetBackup to verify that we're okay. These steps are pretty dry, but I think if you follow them in a somewhat linear order (skipping some or doing some before others, if you're comfortable) they should get you to where you want to be. Fat, happy and with a TLD your backup server recognizes. Okay, maybe not happy ;)

Note: If you feel uncomfortable about running any of the commands below, please enlist the assistance of someone who is either able to provide guidance (since each case is unique) and/or will get in trouble instead of you if things to go to Hell ;) j.k.

And, here we go. These steps won't be numbered, so I can't possibly screw that aspect up, but should be easy to follow since each command will be separated by space and begin with the "host # " prompt. Some of these commands, as the title of today's post suggests, may not exist on a flavour of Unix or Linux that isn't Solaris.

First, we'll take a look at our device tree. Do the device links listed in sgscan exist? Also, is /dev/rmt populated at all?

host # ls /dev/sg/c0t2l0 /dev/sg/c1t1l0 /dev/sg/c1t0l0 /dev/rmt
/dev/sg/c0t2l0  /dev/sg/c1t0l0  /dev/sg/c1t1l0

/dev/rmt:
0     0cb   0hb   0lb   0mb   0u    1     1cb   1hb   1lb   1mb   1u
0b    0cbn  0hbn  0lbn  0mbn  0ub   1b    1cbn  1hbn  1lbn  1mbn  1ub
0bn   0cn   0hn   0ln   0mn   0ubn  1bn   1cn   1hn   1ln   1mn   1ubn
0c    0h    0l    0m    0n    0un   1c    1h    1l    1m    1n    1un

They appear to be there, but they're probably bad. Let's try devfsadm, all on its lonesome and check sgscan again (From now on we'll just assume the output is the same as the train-wreck we witnessed above, until we get to the end. Hopefully, your journey will come to a close sooner!):

host # devfsadm

If this fails to produce results, you can try to run the same command with the "-C" option to remove stale links that no longer point to a valid physical device path:

host # devfsadm -C

Of course, if you know that you only had two tape drives before (/dev/rmt/0 and 1) and believe sgscan when it says it can't recognize the paths we listed, you can delete all of that stuff and try those two steps again. Sometimes it helps to force Solaris to recreate the dev links:

host # rm /dev/rmt/*
host # devfsadm -C

should be enough, but you can almost certainly do this, as well:

host # rm /dev/rmt/* /dev/sg/c0t2l0 /dev/sg/c1t1l0 /dev/sg/c1t0l0
host # devfsadm -C

Running the "ls /dev/sg/c0t2l0 /dev/sg/c1t1l0 /dev/sg/c1t0l0 /dev/rmt" listed above will, almost always, give you the same results once you've completed these steps.

You might also run this command if you have the drivers installed:

host # cfgadm -al

If you find a section with /dev/rmt1, /dev/rmt0 and the /dev/sg path to your Changer in it, and one or some of them are showing unconfigured (all the sections start with a controller number and a colon - in our setup the output is "c2:xxxx") you can either specifically configure any of the entries listed behind the controller number, by using the entire device name your rmt and disk changer devices are listed beside, or you can just configure the whole shebang. Why not?:

host # cfgadm -c configure c2

Listing it again with "cfgadm -al" should show all the appropriate devices as "configured." If it doesn't; don't worry. It probably doesn't matter, but was worth a shot.

Both "tpconfig-d" and "tpconfig -dl" will give you back the same results as sgscan (although formatted differently and limited to the tape and TLD information) if the problem still hasn't resolved. To save space and prevent carpal-knuckle syndrome, full versions of the output of these commands, as run against a working setup, are located at the bottom of yesterday's posts as a series of in-page hyperlinks. The only things that will be different in your execution of:

host # tpconfig -d

and

host # tpconfig -dl

output will be that the drives will usually either show up as DOWN ( possibly with an identifier - for us, hcart2 - and path like /dev/rmt/0) or you will get virtually no output at all ...yeah, I guess that's a "huge" difference :) If you notice that tpconfig returns a listing for you, this is positive, even if it shows your drives as "down." We won't go crazy yet, since we were going to run the next command, regardless:

host # vmoprcmd

Now we may get results that show "HOST STATUS" as <NONE> or, hopefully ACTIVE (good to go!), ACTIVE-DISK (can do local disk backups), ACTIVE-TAPE (can backup to tape, but, for some reason, can't backup to local disk) or even DEACTIVATED (either it's off or NetBackup thinks it is) or OFFLINE (Same as the last, except substitute offline for off ;) Your drives will also show as either non-existent, UP, UP-TLD, RESTART or DOWN (perhaps a few others, but all of them self-explanatory). As long as the tape drive type (hcart2 for us) is shown, you're on the way.

And the final things we'll try today will be to react to the output produced for the Tape Drives. If your TLD is still not showing, that's something for tomorrow. If you see your tapes in a DOWN state, but correctly identified as the types of tapes they are, this will probably do the trick for you:

host # vmoprcmd -up 0
host # vmoprcmd -up 1

for the first (0) and second (1) instance of the drive, listed in the first "Id" column of "tpconfig -d". You can also do this, which is easier (at least for me) to remember, since you can directly map it from the vmoprcmd output without squinting ;)

host # vmoprcmd -upbyname Drive000
host # vmoprcmd -upbyname Drive001

from the vmoprcmd output in the "Drive Name" column, which also happens to be the first column in the "vmoprcmd" output.

When you're done with that, or if your tape drives show as RESTART, do yourself a favor and stop and start NetBackup. You may not get a chance once you let everyone know it's fixed. If you don't have other startup scripts set up, you can use:

host # /usr/openv/netbackup/bin/goodies/netbackup stop

then run:

host # /usr/openv/netbackup/bin/bpps -a

and, if everything is gone (unless you're running the GUI - it's okay to not kill those PID's), start 'er up again, like so:

host # /usr/openv/netbackup/bin/goodies/netbackup start

and do another "bpps -a" to make sure all of the appropriate daemons are running. Then, just to make yourself feel better, and so you're absolutely sure, do one more "sgscan." All should look as it did in yesterday's post (see link-back above) and you should be all set. At least, you'll be ready to test some backups and pray that your troubles are over ;)

We'll be back tomorrow to look at some ways to deal with really pernicious and aggravating software and OS failures. Until then,

Cheers,

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Booting A Known-Good Solaris Disk On A Different Server Type

Hey There,

For the most part, if you work on Sun (or any other kind of) servers routinely, when really bad things happen (like system crashes, etc), part of the process of recovery may ultimately include parts replacement. At its worst, the process may include a complete replacement machine (This has actually happened to me many times, after 15 or 16 parts replacement visits ;)

Generally, this isn't an issue, as Sun will want to replace, for instance, a v490 with a new v490 and you can just pop your old operating system disks in there and boot right up.

Occassionally, however, you'll run into a situation where the server you need replaced is no longer supported. In these instances (and in instances where your company has just decided to upgrade its servers), you find yourself in a little bit of a pickle. Nothing terrible, but you can't just pop the old disks in the new box and boot up without issue (We'll assume, for this post, that the hardware, itself, is compatible. In a lot of instances, new machines have different physical connection interfaces, which adds another layer of complication to the issue, but I digress...)

So, now that you've got your new box (which has OS compatible parts, compatible hardware, etc) and your OS disk from your old Sun server, you have to address how to get that new box up and running with your old OS disk. A lot of times, this can be a life saver, especially if you keep peripheral data on your boot disk!

The good news is, you only need to do three things to get yourself back up and running and not have to hassle with re-installing your OS, copying back from the original disk and hoping you get everything you need :)

First, after inserting your hard drive in the new server, you'll need to either boot off of cdrom or the network (Check out this post on extended boot options for tips on doing that). Whatever media you do decide to boot off of, local or remote, it's very important that you boot from an image that's as close to the OS on your original disk as possible. In a perfect world, you'll have a CD or netboot image that is the exact release of the OS you have on your original disk. At the very least, it should be the same major version. As a for instance, trying to complete the following steps using a Solaris 10 image to get your Solaris 8 disk to boot will seem to work, but will ultimately result in disappointment.

At the "ok>" prompt (or PROM) level on your new box, type something similar to the following (Basically, boot into single user mode):

ok> boot cdrom -s

Second, you'll want to mount your old disk on the mini Solaris environment you end up in after booting into single user mode, like so:

host # mkdir /tmp/a <--- You'll probably have to make your mount point in /tmp as all the other filesystems should be read-only in single user mode. /mnt may also be available to mount your physical hard drive on.

host # mount /dev/dsk/c0t0d0s0 /tmp/a <--- Note that the exact disk device might be different. You'll know as soon as you try to mount it and run an "ls" on "/tmp/a" whether or not you've gotten the right one. You may even find out sooner if the system outputs errors like it should.

Thirdly, you'll need to execute two crucial commands. These will make the difference between your system booting to a bizarre error, and resultant crash, or a successful start up.

From the single user command prompt, type:

host # devfsadm -C -r /tmp/a -p /tmp/a/etc/path_to_inst <--- This command will rebuild the device tree on your boot disk and create a new /etc/path_to_inst file.

host # installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t0d0s0 <--- This will install a new boot block on your old hard drive. This is the one step where it really matters what version of the OS your boot CD is. As referenced above, if you try to install a Solaris 10 boot block on a Solaris 8 OS disk, the command will succeed, but the following boot attempt will fail!

Once you've completed these three steps, you should be all set to bring the machine back down to run level 0 and boot up successfully! In the odd case you may need to make sure STDIN, STDOUT and STDERR are setup correctly. A simple fix for this is to just symlink in your mounted root's dev directory:

host # cd /tmp/a/dev
host # ln -s fd/0 stdin
host # ln -s fd/1 stdout
host # ln -s fd/2 stderr

Then boot (you can skip the above step most of the time)

host # init 0
ok > boot

And all should be well again :)

Best wishes,


, Mike

linux unix internet technology

Adding Storage using Volume Manager in a Veritas Cluster

Today, we're going to take a look at adding extra disk to a Veritas Cluster using Veritas Volume Manager. We'll assume for the purposes of this post, that you've been asked to add 35gb of space to a Veritas volume (filesystem: /veritas/a - volume name: veritas_a). Now we'll walk, step by step, through determining whether we have the disk available to add, and adding it if we do.

The first thing you'll want to do is determine what disk group(s) the filesystem/volume belongs to. Generally, you'll be told this when you're asked to add disk to an existing system, but we'll just assume the very worst ;) In any event, even if you are told, it's good practice to verify that the filesystem/volume does, in fact, belong to the disk group(s) the request is asking to have you augment.

Now, since we're dealing with a Veritas Cluster, if the disk group is shared between more than one server node, you can run:

vxdctl -c mode

and that will show you what the master node is. This server is where you'll want to execute all of your Veritas Volume Manager commands. If the cluster is inactive, vxdctl will tell you that, and you'll need to execute your commands on the server you're already on.

Now, either on the Cluster's master node, or the machine you're on if the cluster is inactive, get a list of all the disk groups, like so:

vxdg list
NAME STATE ID
rootdg enabled 1199919191.1025.host1
mydg enabled 8373777737.1314.host2

Since the user requested more filespace under /veritas/a, we'll need to see what disk group this filesystem/volume belongs to:

df -kl | grep veritas
/dev/vx/dsk/mydg/veritas_a 34598912 24632328 9903320 72% /veritas/a
/dev/vx/dsk/mydg/veritas_b 34598912 649112 33687984 2% /veritas/b
/dev/vx/dsk/mydg/veritas_c 35350080 17696 35056360 1% /veritas/c

We can see that, under /dev/vx/dsk, mydg is showing as the disk group for our /veritas/a filesystem, and veritas_a is showing as the volume. In cases like this, it's a time saver if you just set the default group once on the command line, so you don't have to use the "-g" option for every command you run:

vxdctl defaultdg mydg

Now you'll run some standard commands to make sure that your system can see all the disk available to it and make sure that the data is fresh:

devfsadm -Cv (note that the cluster will only update the volume manager configuration for cluster nodes that use the mydg diskgroup -- we don't have to specify that here, because we defined it as the default above)
vxdctl enable (this is basically the second part of the same update. The devfsadm command is Solaris specific while vxdctl is Veritas specific)

Now we need to check for free space, to see if we can accommodate the request with what we have available to us at present:

vxdg -g mydg list <-- Again "-g mydg" is optional because it's the default.
DISK DEVICE TAG OFFSET LENGTH FLAGS
DISK1 c5t27d27s2 c5t27d27 69198720 1501440 n
DISK2 c5t27d30s2 c5t27d30 69198720 1501440 n
DISK3 c5t27d28s2 c5t27d28 69198720 1501440 n

This output is, admittedly, a pain to read, because it expects you to be able to calculate disk offsets in your head. For this output, if the offset were zero and length was 70700160 and flags were '-' then that DISK would have 35 GB of free space available to us. Here all the listed storage has been partially used up. Hence, there is not enough storage space available!. We can also verify this by running

vxdg free (optionally: vxdg -g mydg free) to see if there is any free space left in the disk group. Either way, we're out of luck, it would seem.

So, now we're going to have to look into Veritas' disk configuration and see if there isn't a spare disk (hope) that we can pick up and import into the mydg diskgroup so that we can add the space (One thing to note in the output below is that all online disk shows its disk group membership even though it's imported locally):

vxdisk -o alldgs list <--- (abbreviated to keep this post bearable ;)
DEVICE TYPE DISK GROUP STATUS
c0t0d0s2 sliced - - error
c0t2d0s2 sliced disk02 rootdg online
c1t2d0s2 sliced disk01 rootdg online
c5t27d40s2 sliced - (mydg200) online
c5t27d41s2 sliced - (mydg2) online
c5t27d300s2 sliced - (mydg300) online
c5t20d99s2 sliced - - error

Luckily, it appears that disk c5t20d99s2 is available and doesn't belong to any existing disk groups! We'll use the command line to initialize this free disk and add it to our disk group (mydg):

vxdiskadd c5t20d99s2

You will be asked if you want to initialize the disk (or re-initialize it, if it's already initialized). Answer yes to either option, to keep things simple. Then it will ask you what disk group you would like to add the disk to. Answer: mydg. It will ask you if you want to use a default disk name. Depending on your situation, this may be okay, but we'll say no and add it as "DISK4" to keep with our naming standard. Then you will be prompted if you want to add the disk as a spare for the mydg disk group. Answer no to this, or you won't be able to use it for the additional storage you want! Answer no again, when it asks you if you want to encapsulate the disk, and then answer yes when it asks you if you want to initalize it. You can then choose to perform a surface analysis before adding it (It says it's highly recommended but my experience is it's generally not necessary and takes a very very very long time). Phew... that wasn't too much to info to have to feed to a single command ;)

Now, we'll want to grow the mydg disk group's volume and filesystem size. We've added the new disk, so we have more space, but the user can't see it or use it at this point. Now, common wisdom is to use the vxassist command. It's common, because it can be a great command to use (very simple), but I prefer to use "vxresize" because, unless your filesystem is vxfs, running vxassist to grow the volume (/veritas_a) can cause serious issues when growing the filesystem (if, for instance, you're using the Veritas Volume Manager but have your volume filesystem type set to Solaris "ufs")!

Calculate the maximum length of your new volume using vxassist (this part is okay - forget about what I just said above - that only applies to using vxassist to do the actually growing):

vxassist -g mydg (optional, if default is already set) maxsize <--- This will be the NEW_LENGTH argument for the next command.

vxresize -F (vxfs or ufs,usually) -g mydg (optional, still) -x (to make sure that the operation fails if the volume doesn't increase in length) veritas_a NEW_LENGTH <-- The last time I checked, - and another reason I prefer this method - this command grows your filesystem with the volume so you don't need to run any more steps after this.

Of course, you can keep tabs on all this, after you're done typing, using the vxtask command. At no point during this entire operation will you need to unmount your filesystem. I believe it's "recommended" (because something could always go wrong), but I've never found it to be necessary (technically, it's not supposed to fail, but Veritas has to cover itself, just in case)

Once the task is finished, your "df -k /veritas/a" output should show your new disk available and ready for use!

Cheers,

, Mike

linux unix internet technology