Fixing NetBackup Media Management Issues Through Mental Exhaustion

Hey there,

If you're ever setting up a new NetBackup server, and you're pretty sure you've done everything correctly, you very well may have ;)

Today's post is drawn directly from the 3 or 4 hours of insanity I went through trying to fix a broken setup "correctly" (which may work for you :) and then the neat little trick I pulled, as a last ditch effort, that actually ended up solving the problem. This is somewhat of an informal post, so if any of the steps I describe below aren't specific enough, check out our other posts on NetBackup troubleshooting and Set up (and other NetBackup Tips - with some bleed-over) where we've spent more time on the nitty-gritty :)

Note: I spent a lot of time reading NetBackup forums on and off of Veritas' site (or is it Symantec now? Nothing's changed ;) and some of the help will be very valuable to me in the future, I'm sure. However, and this drives me nuts, about 80% of the time, if I found someone who had the "exact" same problem as me, the responses were either:

1. Non-existent. A bummer, but, actually, somewhat comforting, since I could feel like no on else in the world knew what the fuck the deal was with that problem ;)

2. Argumentative and off-topic. This, I will never understand. Not so much that people will write random replies (often spiteful) that are of no use to anyone whatsoever, but that forum moderators (especially on the official NBU site) don't just delete that crap. I realize that reading someone else's opinionated bullshit is absolutely free and I don't "have to read it" (just like everyone who reads this blog is free to check out another web site at any time ;), but I question a world in which opinion, itself, is elevated above substance in the improper venue. If there isn't one out there already, I'm going to start a forum for people who need to vent. I'll keep the topics legitimate sounding, but the whole site will be all about belittling your fellow man for no reason any more noble than inflating your own floundering and pathetic ego. Is that the sound of hate mail, already? ;)

Anyway, back to NetBackup.

THE PROBLEM: All systems came up good. Drives recognized at the OS and Software layer and everything, seemingly, good to go. Unfortunately, every backup resulted in an error 96 (Unable to allocate new media for backup). This was on a server that had been setup in a disciplined manner (I must qualify that with "I think" since I didn't verify that folks were actually following procedure, but the build was close enough that they almost "had" to be) and it actually did have fresh media to allocate).

The Problem's (somewhat) human translation: The EMM (Enterprise Media Manager) database cannot find an entry for any of the media in your tape robot that it can use to satisfy your request for a backup. On a good day, the solution may be as easy as just inventorying your robot (using the "update" flag or checkbox) or reassigning your tapes to the volume pools that your policies belong to).

Assuming that's not the issue, they recommend that you get rid of everything and re-add it. Although this sounds like a big pain in the ass, it's really not that bad of an idea, and makes sense. Especially since I was working with a new setup and didn't have to worry about preserving any old backups. Burn it all down and rebuild. No problem :)

Rather than lifting, and rewording, the content from This Veritas Knowledge Base Page For Error 96, I would suggest you check out that page first if you run into this issue (It's a good jumping off point to various other avenues if your problem doesn't get resolved, as it has a few tests that may help narrow down your issue). I visited a number of other helpful places, but most of the stuff I ended up doing there didn't help me (today) and I don't want this post to get too confusing.

And finally...

THE SOLUTION THAT WORKED FOR ME: After trying to expire media and having NetBackup tell me that it wasn't in the EMM database, and then trying to add it to the EMM database and having the EMM database complain that an entry for the media already existed (a contrary position that I arrived at by taking many different paths ;), I eventually threw my hand up in the air (not literally) and just deleted the "volume group" that all the "volume pools" belonged to, because it was the single stupidest thing I could think of doing at the time ;)

As it turns out, once I whacked that thing (the only volume group we had), my next plaintive re-inventory actually updated the robot. Of course, after doing that, none of the tapes belonged to any of our volume groups anymore, but all I had to do was add the media back to their respective volume groups and, lo and behold, backups started working.

The moral of the story? Don't be afraid to give up. Every once in a while, it works ;)

Cheers,

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Unix and Linux Workplace Sadomasochism: Possibly As Bad As It Sounds.

Hey there,

Ever Feel Like Killing Your Boss? Me neither, officially, but after a day like today, I can't get that old Feederz album out my head. Great tunes, if you're feeling punky and vengeful ;)

As I sit and reflect on my work-day (and the non-stop Unix and Linux weekend suck-a-thon coming up) I think (for only about the third or fourth time in my life) that just hiring someone to kick me in the nuts every 10 minutes or so (at random intervals, to keep it interesting ;) has actually surpassed showing up at work, and being figuratively kneed in the sac, on my list of top ten ways to spend my day.

While I ice my bruised ego and get mentally detached enough to VPN in to do even more work tonight (so I can be almost on-track by tomorrow morning) I thought I'd put up this great piece of work-related humor that's eerily similar to what I've been thinking all week. The site I got this from is called WhyWorkSucks.com and is well worth checking out. I'm going to be hopping on this gentleman's mailing list, because I can't wait to read the book he's working on. As he states on his page, he's not looking to sucker you; just keep you informed. Kind of like we do here. No spam, just updates if you want them (In the right hand column; handled through Google - who have acquired FeedBurner - so we never know who you are or what your email address is).

Enjoy the following and find out if you've got the stuff it takes to make it in management! Also, check out whyworksucks.com. There's a lot of great stuff on that site, including interactive pages like the "Wheel Of Policy" and the "CEOmatic 2000," just in case you ever find yourself at a loss for that important sounding, but ultimately meaningless, declaration that may or may not serve to "rally the troops" (bet the other way ;)

Cheers,

NOTE: Clicking on the link, at the bottom of the excerpt below, will take you to the whyworksucks.com website to take the quiz. Hopefully, you don't have it in you. And if you do, please quit wearing the steel-toed boots ;) ...the following has only been modified to show the pictures the hosting company won't allow us to create remote-links to and to make this legible on BlogSpot.

---

This is a test to see if you're ready
— or even ever capable — of holding a position in management.
This highly accurate scientific, standardized test was developed
by some highly paid consultants who in turn consulted with a
number of focus group participants who in turn consulted with
an inner demon named Lester.

Before we get started
with the test, take a quick look at your writing utensil. If
that writing utensil is a number 2 pencil, put down the test
right now and give yourself an "F." Coming to class
prepared?! How could you be so incompetent! You're never going
to succeed that way!

If — even now
— the thought of finding something to write with hasn't
crossed your mind, give yourself a 1,000-point head start.

Are you ready now?
Click here to begin the management quiz.

© Copyright 2000 Why Work
Sucks. All Rights Reserved.

---

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Monitoring and Display Commands For LVM On Linux And Unix

Hey there,

Today, we're going to follow up on our previous post regarding getting started with LVM logical volume management and go over a few (perhaps) boring but essential commands you'll want to know in order to keep tabs on your setup both in good times and in bad.

Refer back to our initial post on getting started with LVM for any additional details, if you'd like them, but, for today's purposes, we're just going to use the outcome of that post as the simple setup that we'll be monitoring/displaying today. In a nutshell, we have one physical volume composed of one volume group which, in turn, contains only one logical volume (or you could think of it the other way around). Unless I'm completely off-center here today, our output from today should be extremely reminiscent of yesterday's ;)

Note: Based on your particular version, or vendor distribution of LVM, the output of these commands may be slightly different. Not so much so that it should make a significant difference.

Basically, our monitoring and displaying commands are broken down into three groups: Those dealing with "physical volumes," those dealing with "logical volumes" and those dealing with "volume groups." We'll step through them (not over and/or around them, even though that would be more polite ;) one by one (in twos):

Physical Volumes:

The two commands we'll be using here are pvscan and pvdisplay.

pvscan, as with all of the following commands, pretty much does what the name implies. It scans your system for LVM physical volumes. When used straight-up, it will list out all the physical volumes it can find on the system, including those "not" associated with volume groups (output truncated to save on space):

host # pvscan
pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- ACTIVE  PV "/dev/hda1"   is in no VG [512 MB]
...
pvscan -- ACTIVE  PV "/dev/hdd1" of VG "vg01"[512 MB / 266 MB free]
...

Next, we'll use pvdisplay to display our only physical volume:

host # pvdisplay /dev/hdd1 <-- Note that you can leave the /dev/hdd1, or any specification, off of the command line if you want to display all of your physical volumes. We just happen to know we only have one and are being particular ;)

...
PV Name   /dev/hdd1
VG Name   vg01
PV Size   512 MB
...

Other output should include whether or not the physical volume is allocatable (or "can be used" ;), total physical extents (see our post on getting started with LVM for a little more information on PE's), free physical extents, allocated physical extents and the physical volume's UUID (Identifier).

Volume Groups:

The two commands we'll be using here are vgscan and vgdisplay.

vgscan will report on all existing volume groups, as well as create a file (generally) called /etc/lvmtab (Some versions will create an /etc/lvmtab.d directory as well):

host # vgscan
vgscan -- reading all physical volumes (this may take a while...)
vgscan -- found active volume group "vg01"
...

vgdisplay can be used to check on the state and condition of our volume group(s). Again, we're specifying our volume group on the command line, but this is not necessary:

host # vgdisplay vg01
...
VG Name               vg01
...
VG Size               246 MB
...

this command gives even more effusive output. Everything from the maximum logical volumes the volume group can contain (including how many it currently does and how many of those are open), separate (yet similar) information with regards to the physical volumes it can encompass, all of the information you've come to expect about the physical extents and, of course, each volume's UUID.

Logical volumes:

The two commands we'll be using here are (you guessed it ;) lvscan and lvdisplay.

lvscan will give us a report on all of our logical volumes, and their state (ACTIVE/INACTIVE or possibly in ERROR) on our system, like so:

host # lvscan
...
lvscan -- ACTIVE            "/dev/vg01/lvol01" [246 MB]
lvscan -- 1 logical volumes with 246 MB total in 1 volume group
lvscan -- 1 active logical volume

Once this step has been completed (and it should be completed at least once!), we can use lvdisplay to check out our logical volume(s). We'll be specifically looking at lvol01:

host # lvdisplay lvol01
...
LV Name                /dev/vg01/lvol01
VG Name                vg01
...

and we'll get our standard UUID information, read/write access, availability status and information on the logical volume size (among other useful information, hopefully including the block device the logical volume is associated with :)

It should be noted that the logical volume size is one of the areas we generally look at first. If, for some reason, the output from "df" seems less than what you intended, this will give you a confident indication of whether or not your logical volume is too small, or you just need to grow the filesystem, resident on it, to use all of the logical volume's space!

...And that about wraps up the "information gathering" section of this series of posts on LVM. In future posts we'll be looking at ways to manipulate volumes (both in groups and individually in their physical and logical aspects; although the physical volume is actually a logical representation of a logical representation of a physical device... but I think we went into too much detail about that gordian knot in our previous post, and, even then, haven't begun to scratch the surface ;). In future posts we'll also be looking at LVM on Unix and Linux and the relationship between the two, which is becoming less complex every day!

Cheers,

, Mike

How To Get Started With Logical Volume Management In Linux

Hey again,

Today, since past posts regarding disk management software have mostly been limited to Veritas Volume Manager, I thought we'd look at LVM on Linux (In a future post, we'll check out the differences between LVM 1 and 2, but for now I figured we could keep it simple and broad).

If you use any of the "major player" Linux flavours out there, you've probably noticed (over the last year or so) that standard installs show all your disks listed out in df, like:

/dev/mapper/vg01-lvol01 246M   75.2M    158M 32%    /dir

rather than the old-style:

/dev/hdd1      246M   75.2M    158M    32%    /dir

that you've (if you're anything like me and only have two hard drives on the home PC you're running Linux on ;) grown to love.

And, while I think it's fantastic that Linux has grown into a mostly-user-friendly operating system with GUI's to make all of the extended partitioning and volume management more accessible, I also think there's something to be said for knowing how to be able to set all that stuff up on your own. You never know when your main display (or X-windows session) will die on you and, at some point, you may end up having to do some disk repair on the lonely old command line.

With that in mind, we'll step through the most basic LVM setup: Getting from /dev/hdd to /dev/mapper/vg01-lvol01 from the command prompt.

Our basic (and huge assumption) here is that we've got a primary drive to work from (although we could do this just as easily from a rescue, or live, CD-ROM if we needed to perform this work on our primary disk during the Linux installation process or a freakish-nightmare of a fix)

First, we'll want to use fdisk to create a "new" "primary" partition that consists of the entire disk (512Mb in size). Barring a primer on fdisk, we'll want to set up our disk to have one primary partition that consists of all cylinders (first and last should be the default start and end values) and is set to the "Linux LVM" partition type.

Once we're done with the stuff we'd "normally" do (for the most part) to set up any other type of partition table on our /dev/hdd disk, we can begin to play around with the LVM commands. Note that if you chose any partition type other than LVM when setting up your partition, using the following commands will, at best, result in error messages and accomplish nothing :)

If you've ever used HP-UX before, a lot of these commands will seem familiar. There's a reason, but the legalities of discussing it (and possibly getting bits of it wrong) fall outside the scope of my pocketbook ;)

The first thing we'll do is create a "physical volume," like so:

host # pvcreate /dev/hdd1

Yes, it was a physical volume already. The logical-physical relationship sometimes doesn't hold up to scrutiny outside of the "rules" ;)

Next, well create a "volume group." This is necessary, because "logical volumes" run in packs... pardon my lack of propriety ;) We could use the "-s" option here to define the "physical extent" size, but we'll leave that alone. To say the least, specifying this requires some research and planning. If you get that number wrong, things can go South later, and you'll have to do some ugly backtracking. We just want to define a "volume group" so we have something to work with. We'll call it vg01, for lack of a more original name.

host # vgcreate vg01 /dev/hdd1

Next, we'll create a "logical volume" (we'll name this one "lvol01" to keep things consistent) of 246 Megabytes size. That should fit nicely inside vg01 (which we're basically using to represent the entire partition /dev/hdd1). We could make our "logical volume" as large as the entire partition and/or "volume group", but that would defeat the purpose of this whole exercise. The aim, of course, is to end up with flexible volumes so we can resize them later if we need to.

host # lvcreate -L 246M -n lvol01 vg01

This is where the /dev/mapper stuff comes into play (it never made sense to me the first time I saw it... really. It's because of my past experience with HP-UX ;) The "lvcreate" command creates a symlink between /dev/vg01/lvol01 and the character device /dev/mapper/vg01-lvol01.

And, next, all we have to do is format that bad boy:

host # mkfs -t ext3 /dev/vg01/lvol01

Note that, above, you can set a reserved block count (based on percentage) using the -m flag. I would usually set this to "-m 5" so that 5% of the allocated space would be reserved for root (although you can set it as low as 1). This way, if someone jams your filesystem up to 100% and no one can get any work done, you can still log in as root and fix the issue. -v is another useful flag if you want to see verbose output. You can insert these two flags anywhere in the above command line except at the very end (after the logical device name) or in between the -t flag and its argument. Common sense, I know, but I feel compelled to over-explain ;)

Now, we'll make a mount point, or, really, just make a new directory off of the root filesystem (although you can do overlays if you wanna get creative ;)

host # mkdir /dir

and then we'll mount our logical volume up and check it out with df:

host # mount -t ext3 /dev/vg01/lvol01 /dir

host # df -h /dir
...
/dev/mapper/vg0-lvol0 246M   75.2M    158M 32%    /dir

Tomorrow, or maybe the day after that, we'll take a look at some more ways to manipulate your existing volumes, add more volumes, do mirroring and lots of other fun stuff that LVM allows you to do.

For now, we'll just thank our deity-of-choice that this all worked out ;)

Cheers,

, Mike

Summary Post Regarding Sun 6800/6900 Server Component Management

Hey There,

I'm putting this "lazy Sunday" post out to tie up all of our previous posts regarding dealing with board, power, and component management on Sun 6800/6900 Servers.

This post truly deserves the designation of a "lazy Sunday" post because it takes care of two things at once:

1. It lays down a roadmap of all of our previous posts on this topic, in order, in one easy to find location.
2. It allows me to enjoy my Sunday (insofar as that's possible, after working a weekend Friday/night/Saturday shift. We all have to sleep sometime, and I have a Sunday very-early-morning job still coming up ;)

Initially, take a look at our post on shutting down domains on 6800/6900 servers. It's important to read this one first, as the next two entries, although more specific, assume some of this information.

Then, follow up with our post on installing system boards on 6800/6900 servers.

Finally, hit yesterday's post on moving boards between domains on 6800/6900 servers and everything's wrapped up with a bow :)

Hope you're all enjoying a restful Sunday :)

, Mike

linux unix internet technology

Working with Linux RPM's

This post is a continuation, of sorts, of my last post. This is more of a general-audience post. Most experienced admins know most of this stuff already. Like I mentioned previously, I try to write this blog with an appreciation for what it was like when I first started out in the business. I owe my success to a great many patient and helpful people.

In this post, I wanted to hit on the basics of working with RPM's in Linux (RPM stands for the Redhat Package Management system - basically, they're the software packages that make up your system). In later posts we'll go into some neat tricks... But for now, we'll stick with the basics. Knowing the basics in any field of interest is invaluable in growing and mastering that skillset, just like knowing your ABC's can really help if you ever intend to read or write :)

Check the bottom for a recap of all the RPM options we're going to use and their literal meanings:

1. To display the basic information for any RPM, just type:

host # rpm -qi RPM_NAME - like:

host # rpm -qi bash
Name : bash Relocations: /usr
Version : 2.05 Vendor: Red Hat, Inc.
Release : 8.2 Build Date: Mon 28 Jun 2004 10:33:55 AM CDT
Install date: Thu 12 Jan 2006 01:25:27 PM CST Build Host: host.redhat.com
Group : System Environment/Shells Source RPM: bash-2.05-8.2.src.rpm
... EDITED OUT FOR BREVITY'S SAKE!

2. If you're not sure where to start with the above command, just have RPM spit out all the packages it knows about and pipe that to more, like so:

host # rpm -qa|more
redhat-logos-1.1.3-1
glibc-2.2.4-32.18
cracklib-2.7-12
dosfstools-2.7-1
gdbm-1.8.0-11
...

3. Now that you've figured out what package you want to inspect (Note that you don't have to include the full name to get the information from RPM. The redhat-logos-1.1.3-1 program can be referred to simply as redhat-logos) and have gotten some basic information about it, you can list out all the files associated with the package like this:

host # rpm -ql bash
/bin/bash
/bin/bash2
/bin/sh
/etc/skel/.bash_logout
...

4. Here's one that doesn't require a lot of output, since it's somewhat of a re-explanation. You can add the -p flag to the examples in points 1 and 3 if you're querying an RPM package, and not the RPM database!

host # rpm -qip bash-2.05-8.2.i386.rpm <--- Listing out information for the RPM package itself.
host # rpm -qlp bash-2.05-8.2.i386.rpm <--- Listing out files associated with the RPM package itself.

5. Of course, you may find a file and want to know what RPM package it belongs to. You can get that by typing:

host # rpm -qif /etc
Name : filesystem Relocations: (not relocateable)
Version : 2.1.6 Vendor: Red Hat, Inc.
Release : 2 Build Date: Mon 20 Aug 2001 03:34:02 PM CDT
Install date: Thu 12 Jan 2006 01:24:41 PM CST Build Host: host.redhat.com
Group : System Environment/Base Source RPM: filesystem-2.1.6-2.src.rpm Vendor: Red Hat, Inc.
... (Just as long as the description in point 1)

6. If you want to install a new RPM, you'll need the package file, and would run RPM like this:

host # rpm -i bash-2.05-8.2.i386.rpm

This isn't very interesting (which may be what you want -- I don't care to look at verbose output "all" the time). You can spice it up by adding the -v and/or -h flag, like so:

host # rpm -ivh bash-2.05-8.2.i386.rpm

7. If you want to uninstall an RPM, you'll just need to know the abbreviated name, like I mentioned in point 4). You can also make this as verbose and visually entertaining as the system will allow with -v and/or -h:

host # rpm -e bash

Note that this command would return an error if you had multiple instances of the bash RPM installed. In that case, you could still abbreviate, but would have to include the version number. So you'd type

host # rpm -e bash-2.05.8.2

instead of just bash.

So, to recap, and possibly explain anything I may have glossed over, these basic commands should get you started working with the RPM package management facility on Linux. The translations of the flags we've covered are as follows:

Major flags (usually the ones preceded with a dash, but you can arrange the flags in whatever order you choose - just be careful - see note in the minor flags):

q = query
i = install
e = remove/uninstall

Minor flags

i = information (not the same as the major flag i. Of course, you'll probably never use -ii or -ei, as the combinations would be redundant and opposite, respectively.
a = all
l = list
p = RPM package file (e.g. whatever.rpm)
f = file
v = verbose
h = hash (prints lots of # symbols while it completes your request :)

Enjoy getting started working with RPM packages. They're one of the foundations of the Linux operating system. In fact, a combination of certain packages actually "is" the operating system. Knowing how to manipulate them and have them work for you can make it easier to explore many other things (like new software you've always wanted to install and try out :)

Best wishes,

, Mike

linux unix internet technology

SSH Command Runner To Help With Those Big Tiresome Tasks!

You may recall that we looked at one of the core components of this script in an earlier post, located here.

Today, in preparation for Thanksgiving, I'm putting up a little script that can help you run almost any command line you can dream up, and save the output for you in a relatively nicely formatted report. I'm calling it "scmd" (short for SSH command) but you can call it whatever you want :)

Now, of course, this script comes with a pre-requisite. For instance, it won't really help you save time if you don't have ssh-key-based passwordless logins set up for yourself on all the servers you manage. For Thanksgiving day, we'll go over how to easily set yourself up with those.

For today, assuming you've got your passwordless logins all set, feel free to modify this script as you wish, and use it to kick back and relax when the boss asks you to verify the version of Veritas Foundation Suite on all 300 servers in your farm. Otherwise, until tomorrow, enjoy only having to type in your password over and over and over again (at least you won't have to keep re-typing the command ;)

The script is posted below, for your enjoyment. Note that the "hostfile" is set to be of the format: hostname colon ip_address (e.g. host.xyz.com : 192.168.0.1). The hostname is the only necessary component of each line and all lines that begin with pound symbols (#'s) will be skipped. One host per line, please.

Again, tomorrow, we'll go over how to easily set up ssh keys for yourself network wide, and feel free to modify this little script to make your worklife as effortless as possible:





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/ksh

#
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

trap 'rm tmpfile.$pid;echo "Caught Signal. Cleaning Up And Quitting...";exit 3"' 1 2 3 9 15

pid=$$

if [ $# -ne 1 ]
then
echo "Usage: $0 \"quoted command\""
exit 1
fi

if [ -f hostfile ]
then
hostfile="hostfile"
elif [ -f /export/home/bob/data/hostfile ]
then
hostfile="/export/home/bob/data/hostfile"
else
echo "Can't find hostfile. No hosts to ping. Out..."
exit 2
fi

command=$1
squashed_command=`echo $command|/bin/sed -e 's/ * *//g' -e 's/|//' -e 's/\///g'`
buffered_command="${command};echo";

print "" >>tmpfile.$pid
print "Report Output for \"${command}\"" >>tmpfile.$pid
print "______________________________" >>tmpfile.$pid
print "" >>tmpfile.$pid

cat $hostfile|while read name colon address
do
if [ $name == "#" ]
then
:
else
print "\nRunning \"${command}\" on \c" >>tmpfile.$pid
print "$name... " >>tmpfile.$pid
print "$name... "
/usr/bin/ssh -n $address $buffered_command 2>/dev/null |tee -a tmpfile.$pid
fi
done

mv tmpfile.$pid OUTPUT.${squashed_command}.$pid

After a successful run of, for instance - scmd "/usr/sbin/pkginfo VRTSvcs" - you'll end up with a nicely formatted report with the output from all of the servers in your "hostfile," named OUTPUT.usrsbinpkginfoVRTSvcs.14756; the only real variable in the output report name will be the process id tacked on the end so you can run the same script multiple times and not overwrite your old data.

And, oh yes, don't forget to backslash all your special characters :)

, Mike

linux unix internet technology