Showing posts with label solaris. Show all posts
Showing posts with label solaris. Show all posts

Tuesday, May 5, 2009

Wtmpx Log Rolling On Unix or Linux: Practical Application Of Fwtmp

Hey There,

In yesterday's post on getting the year from wtmpx, we took a look at a great built-in program called fwtmp (that I somehow managed to not notice for several years ;) and examined some uses of it from a high level perspective.

Today we're going to look it from an opposite angle and look at something very specific that it can help you do. And, just to dot the i's that I can (and this list is, of course, incomplete), you can find fwtmp in /usr/lib/acct on Solaris and in /usr/sbin/acct on RedHat Linux (AS 5.2) and on SUSE Linux 9. Of course, all of the operating systems require you to have the correct pkg/rpm/dpkg files installed in order for the command to exist on your system at all :)

Below is a really simple shell script to illustrate the functionality of fwtmp. It's basically a log rotation script written specifically to highlight the use of fwtmp to rotate your wtmpx/wtmp/btmp file. It's meant to be run in cron and is simple to execute since (as it stands) it takes no arguments. Feel free to embellish for your own environment or to make it more accessible across a wide variety of different OS's. The basic cron entry I would add would be something like:

58 23 * * * /usr/local/bin/wtmp_rotate >/dev/null 2>&1


which basically just tells the cron daemon to run /usr/local/bin/wtmp_rotate (the place I like to put all my custom scripts) at 11:58pm every day and to dump any output from the command into the bit-bucket (redirecting both STDOUT and STDERR to /dev/null)

Hope this script helps you out some. You may want to test it by making a temporary directory and copying your wtmpx file into there first. I've included some commented lines to indicate the parts of the script you'd want to modify to ensure that your testing "doesn't" use the real system file.

And to answer the question of why I compress the files after converting them back to binary; I found, in my testing, that the opposite of what seemed logical was true. The binary files compacted to a much greater degree than the fwtmp-generated ASCII files. I didn't investigate it much further since it is what it is, but, if I had to throw out a possible reason it may be that fwtmp pads that ASCII file with a lot of extra bits that can't be stripped (That brush-off has middle-management written all over it ;)

Enjoy and cheers :)


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# wtmpx_rotate - rotate your user login logs... wheee :)
#
# 2009 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

# COMMENTED OUT SECTIONS SHOULD BE SWAPPED WITH THEIR UNCOMMENTED COUNTERPARTS TO DO LOCAL-DIRECTORY TESTING WITH A COPY OF WTMPX
# THESE SWAPPABLE SECTIONS WILL BE SEPARATED FROM LIKE PARTS OF THE SCRIPT BY A SINGLE BLANK LINE

if [[ ! -d /var/adm/backup_log_dir ]]
then
mkdir /var/adm/backup_log_dir
fi


#if [[ ! -d backup_log_dir ]]
#then
# mkdir backup_log_dir
#fi


wtmpx="/var/adm/wtmpx" # This may be /var/log/wtmpx or /var/log/lastlog depending on your setup
#wtmpx="wtmpx"


fwtmp="/usr/lib/acct/fwtmp" # This may be /usr/sbin/fwtmp depending on your setup. Not using "which" since /usr/lib/acct isn't a standard directory.
sed=`which sed`
rm=`which rm`
compress=`which compress` # Or gzip, bzip2, whatever you prefer

grep_date=$(date "+%a %b %e")
grep_date_ext=$(date "+%a %b %e"|$sed 's/ //g')
grep_year=$(date +%Y)
variable_ext1=$(echo ${RANDOM}`date "+%S"`)
variable_ext2=$(echo ${RANDOM}`date "+%S"`)
variable_ext3=$(echo ${RANDOM}`date "+%S"`)
wtmpx_plus_variable_ext1=${wtmpx}.$variable_ext1
wtmpx_plus_variable_ext2=${wtmpx}.$variable_ext2
wtmpx_plus_variable_ext3=${wtmpx}.$variable_ext3
backup_log_dir_file=${wtmpx}.${grep_date_ext}.$grep_year


backup_log_dir_dir="/var/adm/backup_log_dir"
#backup_log_dir_dir="backup_log_dir"


$fwtmp < $wtmpx > $wtmpx_plus_variable_ext1

$sed -n "/$grep_date.*$grep_year$/p" $wtmpx_plus_variable_ext1 > $wtmpx_plus_variable_ext2
$sed "/$grep_date.*$grep_year$/d" $wtmpx_plus_variable_ext1 > $wtmpx_plus_variable_ext3

$rm $wtmpx $wtmpx_plus_variable_ext1

$fwtmp -ic < $wtmpx_plus_variable_ext2 > $wtmpx
$fwtmp -ic < $wtmpx_plus_variable_ext3 > $backup_log_dir_file

$rm $wtmpx_plus_variable_ext2 $wtmpx_plus_variable_ext3
$compress $backup_log_dir_file
mv ${backup_log_dir_file}.Z $backup_log_dir_dir


, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Monday, May 4, 2009

Getting The Year From Wtmpx/Wtmp - Part 2 - The Easy Way

Hey there,

Today's subject matter hearkens back to a post we did in 2007 about looking around inside Solaris' wtmpx. In that post, the ultimate objective was to get the "year" information, since commands like "last" don't include that information in their output and the lack of that year information can make for confusing output if you don't roll your wtmpx file every year.

Once I stumbled over this command (which is one thing that always amazes me, since I've been using Solaris since 2.4 - there's always something new to learn, even if it's old ;) I was extremely pleased. Of course, knowing how to crack open wtmpx and get at anything and everything inside it is probably a more powerful skill to possess, but, if you're a simplest-is-best kind of guy like me, using a command that will allow you to get the last output with the year attached is perfect if you're looking to get your last output with the year attached. Seems almost self-evident ;)

The command you can use is called "fwtmp" and exists in the "/usr/lib/acct" directory on Solaris and in many and varying places depending on your Linux distro (assuming your db is updated, commands like locate or slocate should make it easy to find). The command has actually been around since Solaris 7, but (yes, I did some research on it when I realized I'd never realized it existed before ;) had some issues that made it unacceptable. Those same issues persisted through most of Solaris 8, but were eventually ironed out by the time Solaris 9 came out (which is also like saying "By the time you had your Solaris 8 system patched to the gills," since a fully patched 64-bit installation of Solaris 8 was, basically, Solaris 9). The command still stands in good stead on Solaris 10 and is working well on every Linux distro I've been able to test.

Even better still, the command's application is extremely simple; allowing you to use its output for whatever diabolical purposes you have in mind (I believe it supports altruism as well ;) Many of the wtmp/btmp implementations on Linux already have a way around this by allowing you to specify time frames (with the -t flag) when you invoke last. Most distro's also include this command (and its fellow commands) by default, as well. This old post on getting the year from last on Linux also demonstrates how Linux makes this whole process less convoluted by allowing you to get the same information from the lastlog file.

For a quick example of how easy this command is to use, check out the following command line session. First we create a temporary file using fwtmp and then we read a line from it to verify that it worked correctly (it doesn't get much simpler than that... unless they just start including the year in last's output ;) :

host # /usr/lib/acct/fwtmp < /var/adm/wtmpx > tmp_wtmpx_file
host # head -1 tmp_wtmpx_file
user123 sshd 1258 7 0000 0000 1226072918 230489 0 29 host123.desktop.ourhost.com Fri Nov 7 09:48:38 2008


Simplicity at its finest. There's actually a whole separate binary to fix wtmpx, which basically takes care of cleaning operations you could do yourself.

As a for instance, assuming you had the privileges to do so, and wanted to remove that login listed above from the wtmpx file, you could run that initial command to create the tmp_wtmpx_file, delete that line in it (using vi, emacs, what-have-you) and then convert the text file you got from fwtmp back into a binary wtmpx file, using the same command (The -ic options instruct fwtmp that its input is in ASCII format and needs to be converted to binary):

host # /usr/lib/acct/fwtmp -ic tmp_wtmpx_file > /var/adm/wtmpx


Of course, this is a simplistic look at how to do something like that (and certainly not a recommendation to mess with your security logs). You should probably back up files before overwriting them, etc. Still, it's kind of cool to know you can get all this information from wtmpx without having to learn the Perl pack signature (or know anything about C :)

The programming possibilities are only limited to what you need to do and the information fwtmp can provide!

Cheers!

, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Tuesday, April 28, 2009

DiskSuite/VolumeManager or Zpool Mirroring On Solaris: Pros and Cons

Hey There,

Today we're going to look at two different ways to mirror disk on Solaris (both free - but distinguished from freeware in that they're distributed for use on Solaris' commercial and (often) proprietary filesystems and OS).

The old way, probably every Solaris Sysadmin knows backward and forward. Using the Solaris DiskSuite set of tools (meta-whathaveyou ;), which was, at one point, changed to Solaris Volume Manager (which introduced some feature enhancements, but not the kind I was expecting. The name Volume Manager has a direct connection in my brain to Veritas and the improvements weren't about coming closer to working seamlessly with that product).

The somewhat-new way (using the zpool command) won't work - to my knowledge - on any OS prior to Solaris 10, but with Solaris 8 and 9 reaching end of life in the not-too-distant future, every Solaris Sysadmin will have some measure of choice.

With that in mind let's take a look at a simple two disk mirror. We'll look at how to create one and review it in terms of ease-of-implementation and cost (insofar as work is considered expensive if it takes a long time... which leaves one to wonder why I'm not comparing the two methods in terms of time ;)

Both setups will assume that you've already installed your operating system, and all required packages, and that the only task before you is to create a mirror of your root disk and have it available for failover (which it should be by default)

The DiskSuite/VolumeManager Way:

1. Since you just installed your OS, you wouldn't need to check if your disks were mirrored. In the event that you're picking up where someone else left off (and it isn't blatantly obvious - I mean "as usual" ;), you can check the status of your mirror using the metastat command:

host # metastat -p

You'll get errors because nothing is set up. Cool :)

2. The first thing you'll want to do is to ensure that both disks have exactly the same partition table. The same-ness has to be "exact," as in down to the cylinder. If you're off even slightly, you could be causing yourself major headaches. Luckily, it's very easy to make your second (soon to be a mirror) layout exactly the same as your base OS disk. You actually have at least two options:

a. You can run format, select the disk you have the OS installed on, type label (if format tells you the disk isn't labeled), then select your second disk, type partition, type select and pick the number of the label of your original disk. A lot of times these labels will be very generic (especially if you just typed "y" when format asked you to label the disk or format already did it for you during install) and you may have more than one to choose from. It's simple enough to figure out which one is the right one though (as long as you remember your partition map from the original disk and have made is sufficiently different from the default 2 or 3 partition layout). Just choose select, pick one, then choose print. If you've got the right one, then type label. Otherwise, repeat until you've gone through all of your selections. One of them has to be it, unless you never labeled your primary disk.

b. You can use two command (fmthard and prtvtoc) and just get it over with:

host # prtvtoc /dev/rdsk/c0t0d0s2 |fmthard -s - /dev/rdsk/c1t0d0s2

3. Then you'll want to mirror all of your "slices" (or partitions; whatever you want to call them. We'll assume you have 6 slices set up (s0, s1, s3, s4, s5 and s6) for use and slice 7 (s7) partitioned with about 5 Mb of space. You can probably get away with less. You just need to set this up for DiskSuite/VolumeManager to be able to keep track of itself.

Firstly, you'll need to initialize the minimum number of "databases," set up the mirror group and add the primary disk slices as the first mirrors in the mirror-set (even though, at this point, they're not mirroring anything, nor are they mirrors of anything ;) Note that it's considered best practice to not attach the secondary mirror slices to the mirror device, even though you can do it for some of your slices. You'll have to reboot to get root to work anyway, so you may as well do them all at once and be as efficient as is possible:

host # metadb -a -f /dev/rdsk/c0t0d0s7
host # metadb -a /dev/rdsk/c1t0d0s7
host # metainit -f d10 1 1 c0t0d0s0
host # metainit -f d20 1 1 c1t0d0d0
host # metainit -d0 -m d10
host # metainit -f d11 1 1 c0t0d0s1
host # metainit -f d21 1 1 c1t0d0d1
host # metainit -d1 -m d11
host # metainit -f d13 1 1 c0t0d0s3
host # metainit -f d23 1 1 c1t0d0d3
host # metainit -d3 -m d13
host # metainit -f d14 1 1 c0t0d0s4
host # metainit -f d24 1 1 c1t0d0d4
host # metainit -d4 -m d14
host # metainit -f d15 1 1 c0t0d0s5
host # metainit -f d25 1 1 c1t0d0d5
host # metainit -d5 -m d15
host # metainit -f d16 1 1 c0t0d0s6
host # metainit -f d26 1 1 c1t0d0d6
host # metainit -d6 -m d16


4. Now you'll run the "metaroot" command, which will add some lines to your /etc/system file and modify your /etc/vfstab to list the metadevice for your root slice, rather than the plain old slice (/dev/dsk/c0t0d0s0, /dev/rdsk/c0t0d0s0):

host # metaroot

5. Then, you'll need to manually edit /etc/vfstab to replace all of the other slices' regular logical device entries with the new metadevice entries. You can use the root line (done for you) as an example. For instance, this line:

/dev/dsk/c0t0d0s6 /dev/rdsk/c0t0d0s6 /users ufs 1 yes -


would need to be changed to:

/dev/md/dsk/d6 /dev/md/rdsk/d6 /users ufs 1 yes -


and, once that's done you can reboot. If you didn't make any mistakes, everything will come up normally.

6. Once you're back up and logged in, you need to attach the secondary mirror slices. This is fairly simple and where the actual syncing up of the disk begins. Continuing from our example above, you'd just need to type:

host # metattach d0 d20
host # metattach d1 d21
host # metattach d3 d23
host # metattach d4 d24
host # metattach d5 d25
host # metattach d6 d26


The syncing work will go on in the background, and may take some time depending upon how large your hard drives and slices are. Note that, if you reboot during a sync, that sync will fail and it will start from 0% on reboot with the affected primary mirror slices remaining intact and the secondary mirror slices automatically resyncing. You can use the "metastat" command to check out the progress of your syncing slices.

And, oh yeah... I almost forgot this part of the post:

The Zpool way:

1. First you'll want to do exactly what you did with DiskSuite/VolumeManager (since both disks have to be exactly the same). We'll assume you're insanely practical, and will just use this command to make sure your disks are both formatted exactly the same (just like above):

host # prtvtoc /dev/rdsk/c0t0d0s2 |fmthard -s - /dev/rdsk/c1t0d0s2

2. Now we'll need create a pool, add your disks to it (all slices as one) and mirror them:

host # zpool create mypool mirror c0t0d0 c1t0d0

3. Wait for the mirror to sync up all the slices. You can check the progress with "zpool status POOLNAME" - like:

host # zpool status mypool

And that's that. The choice is yours, unless you're still using Solaris 9 or older. This post isn't meant to condemn the SDS/SVM way. It works reliably and is really easy to script out (and when both of these methods are scripted out, they're just as easy to run and the only hassle the old way gets you is the forced reboot).

It's good to see that things are getting easier and more efficient. Although, hopefully, that won't make today's Sysadmins tomorrows bathroom attendants ;)

Cheers,

, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, April 15, 2009

Having Fun With Other Solaris Sys Admins

Hey There,

Today I'm speeding around, working my arse off so that I can work all weekend (When does it end? ...oh yeah; with the worms ;) and I wrote this little shell script (in Solaris' /bin/sh) to lighten the mood a little.

It's another in a string of a thousand fake error messages, but it worked pretty well for my purposes. I made sure to install it up front in the /etc/rc2.d directory (from JumpStart) so that any admin watching a new installation come up for the first time (to verify its integrity) would be guaranteed to see it. Man, it was fun until I had to fix the problem and pretend I had no idea what happened ;)

Enjoy the video of the script output and, hopefully, you can have some fun with this shell script (or a variation of it) on your own!

Please note that the only fun thing other than the error message are the traps set on the interrupts. It's a reboot-a-thon in the making ;)








Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

#
# bad_news.sh - OMFG
#
# 2009 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

trap 'tput rc' 0 1 2 3 15
echo
echo "EXCEPTION: kern.alert !!!"
echo "SYSTEM BOOT ERROR!"
sleep 1
echo "SYSTEM BOOT ERROR!"
echo "Resampling Solaris Boot Loader ..."
sleep 2
echo "Extracting libC boot loader kernel modules"
sleep 1
echo
echo "Fixing /usr/platform/"`uname -i`
echo "This should just take a moment before resuming..."
echo
echo "YOU WILL NEED TO REBOOT THE SYSTEM WHEN THIS PROCESS COMPLETES!"
sleep 1
echo
echo "RUN PHASE 1: Removing files from /usr/platform/"`uname -i`"/ ...\c"
tput sc
while :
do
echo "| \c"
tput rc
echo "/ \c"
tput rc
echo "- \c"
tput rc
echo "\\ \c"
tput rc
done


Cheers,



Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, March 11, 2009

Locating New Backup Hardware Using Veritas NetBackup On The Solaris Unix Command Line

Hey there,

Yesterday we took a look at the basics of using Veritas/Symantec NetBackup to add a new TLD and drives to your existing machine. Today, we're going to go just one step beyond and assume a fairly commonplace situation, which has somehow, inexplicably arisen from THE SITUATION we found ourselves in yesterday. For some reason (and this hardly ever happens ...not sure which word to emphasize to obtain the maximum sarcastic drippage) after we connected our new Tape Loading Device (TLD or Tape Robot), and the two drives it contains, to our backup server, NetBackup - and, possibly, the server itself, is failing to recognize the new device(s). Again, we're going to assume that both the server, TLD, drives and all other hardware are absolutely fine and that all required connections between the devices are set up properly.

NOTE: Today's post is going to assume that some tried and true methods will get you to "good." Tomorrow's post will look at some other ways to make NetBackup recognize and work with your "known good" (and compliant) setup.

If we take the same direct route to initial discovery that we did yesterday, we'd run the same sgscan (which is, as one reader noted, shorthand for "sgscan all") command initially, like so (pardon the error output. I can't afford to create the situation I want to display so I'm doing it from memory):

host # /usr/openv/volmgr/bin/sgscan
/dev/sg/c0t0l0: Disk (/dev/rdsk/c0t0d0): "SUZUKI MBB2147RCSUN146G"
/dev/sg/c0t1l0: Disk (/dev/rdsk/c0t1d0): "SUZUKI MBB2147RCSUN146G"
/dev/sg/c0t2l0: Tape (???): "Unknown"
/dev/sg/c0t3l0: Cdrom: "Hyundai DV-W28E-R"
/dev/sg/c1t0l0: Changer: "Unknown"
/dev/sg/c1t1l0: Tape (???): "Unknown"


Basically, every line where it says "Unknown" is where we're interested in looking. The system can't find our TLD or its drives, so now we have to try to discover them ourselves (with and/or without NetBackup) and then come back around and use NetBackup to verify that we're okay. These steps are pretty dry, but I think if you follow them in a somewhat linear order (skipping some or doing some before others, if you're comfortable) they should get you to where you want to be. Fat, happy and with a TLD your backup server recognizes. Okay, maybe not happy ;)

Note:
If you feel uncomfortable about running any of the commands below, please enlist the assistance of someone who is either able to provide guidance (since each case is unique) and/or will get in trouble instead of you if things to go to Hell ;) j.k.

And, here we go. These steps won't be numbered, so I can't possibly screw that aspect up, but should be easy to follow since each command will be separated by space and begin with the "host # " prompt. Some of these commands, as the title of today's post suggests, may not exist on a flavour of Unix or Linux that isn't Solaris.

First, we'll take a look at our device tree. Do the device links listed in sgscan exist? Also, is /dev/rmt populated at all?

host # ls /dev/sg/c0t2l0 /dev/sg/c1t1l0 /dev/sg/c1t0l0 /dev/rmt
/dev/sg/c0t2l0 /dev/sg/c1t0l0 /dev/sg/c1t1l0

/dev/rmt:
0 0cb 0hb 0lb 0mb 0u 1 1cb 1hb 1lb 1mb 1u
0b 0cbn 0hbn 0lbn 0mbn 0ub 1b 1cbn 1hbn 1lbn 1mbn 1ub
0bn 0cn 0hn 0ln 0mn 0ubn 1bn 1cn 1hn 1ln 1mn 1ubn
0c 0h 0l 0m 0n 0un 1c 1h 1l 1m 1n 1un


They appear to be there, but they're probably bad. Let's try devfsadm, all on its lonesome and check sgscan again (From now on we'll just assume the output is the same as the train-wreck we witnessed above, until we get to the end. Hopefully, your journey will come to a close sooner!):

host # devfsadm

If this fails to produce results, you can try to run the same command with the "-C" option to remove stale links that no longer point to a valid physical device path:

host # devfsadm -C

Of course, if you know that you only had two tape drives before (/dev/rmt/0 and 1) and believe sgscan when it says it can't recognize the paths we listed, you can delete all of that stuff and try those two steps again. Sometimes it helps to force Solaris to recreate the dev links:

host # rm /dev/rmt/*
host # devfsadm -C


should be enough, but you can almost certainly do this, as well:

host # rm /dev/rmt/* /dev/sg/c0t2l0 /dev/sg/c1t1l0 /dev/sg/c1t0l0
host # devfsadm -C


Running the "ls /dev/sg/c0t2l0 /dev/sg/c1t1l0 /dev/sg/c1t0l0 /dev/rmt" listed above will, almost always, give you the same results once you've completed these steps.

You might also run this command if you have the drivers installed:

host # cfgadm -al

If you find a section with /dev/rmt1, /dev/rmt0 and the /dev/sg path to your Changer in it, and one or some of them are showing unconfigured (all the sections start with a controller number and a colon - in our setup the output is "c2:xxxx") you can either specifically configure any of the entries listed behind the controller number, by using the entire device name your rmt and disk changer devices are listed beside, or you can just configure the whole shebang. Why not?:

host # cfgadm -c configure c2

Listing it again with "cfgadm -al" should show all the appropriate devices as "configured." If it doesn't; don't worry. It probably doesn't matter, but was worth a shot.

Both "tpconfig-d" and "tpconfig -dl" will give you back the same results as sgscan (although formatted differently and limited to the tape and TLD information) if the problem still hasn't resolved. To save space and prevent carpal-knuckle syndrome, full versions of the output of these commands, as run against a working setup, are located at the bottom of yesterday's posts as a series of in-page hyperlinks. The only things that will be different in your execution of:

host # tpconfig -d

and

host # tpconfig -dl

output will be that the drives will usually either show up as DOWN ( possibly with an identifier - for us, hcart2 - and path like /dev/rmt/0) or you will get virtually no output at all ...yeah, I guess that's a "huge" difference :) If you notice that tpconfig returns a listing for you, this is positive, even if it shows your drives as "down." We won't go crazy yet, since we were going to run the next command, regardless:

host # vmoprcmd

Now we may get results that show "HOST STATUS" as <NONE> or, hopefully ACTIVE (good to go!), ACTIVE-DISK (can do local disk backups), ACTIVE-TAPE (can backup to tape, but, for some reason, can't backup to local disk) or even DEACTIVATED (either it's off or NetBackup thinks it is) or OFFLINE (Same as the last, except substitute offline for off ;) Your drives will also show as either non-existent, UP, UP-TLD, RESTART or DOWN (perhaps a few others, but all of them self-explanatory). As long as the tape drive type (hcart2 for us) is shown, you're on the way.

And the final things we'll try today will be to react to the output produced for the Tape Drives. If your TLD is still not showing, that's something for tomorrow. If you see your tapes in a DOWN state, but correctly identified as the types of tapes they are, this will probably do the trick for you:

host # vmoprcmd -up 0
host # vmoprcmd -up 1


for the first (0) and second (1) instance of the drive, listed in the first "Id" column of "tpconfig -d". You can also do this, which is easier (at least for me) to remember, since you can directly map it from the vmoprcmd output without squinting ;)

host # vmoprcmd -upbyname Drive000
host # vmoprcmd -upbyname Drive001


from the vmoprcmd output in the "Drive Name" column, which also happens to be the first column in the "vmoprcmd" output.

When you're done with that, or if your tape drives show as RESTART, do yourself a favor and stop and start NetBackup. You may not get a chance once you let everyone know it's fixed. If you don't have other startup scripts set up, you can use:

host # /usr/openv/netbackup/bin/goodies/netbackup stop

then run:

host # /usr/openv/netbackup/bin/bpps -a

and, if everything is gone (unless you're running the GUI - it's okay to not kill those PID's), start 'er up again, like so:

host # /usr/openv/netbackup/bin/goodies/netbackup start

and do another "bpps -a" to make sure all of the appropriate daemons are running. Then, just to make yourself feel better, and so you're absolutely sure, do one more "sgscan." All should look as it did in yesterday's post (see link-back above) and you should be all set. At least, you'll be ready to test some backups and pray that your troubles are over ;)

We'll be back tomorrow to look at some ways to deal with really pernicious and aggravating software and OS failures. Until then,

Cheers,

, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Tuesday, March 10, 2009

Using Veritas NetBackup To Add A Changed-Robot-And-Drive On The Solaris Unix Command Line

Hey there,

Fans of useful information, skip straight to THE SITUATION (hyperlinked and in caps for easy case-sensitive searching, NOT BECAUSE I'M SCREAMING OR ANYTHING!!! ;) as soon as you grow tired of the preamble.

I think today's post's title represents the first time I've ever so blatantly hyphenated part of an awkward sentence in order to make it seem intelligible (and not so incredibly long). As I read it now, I wonder what I was thinking. Much the same as I do every time I proofread my work. The only certain thing in this world is that I will find fault with this sentence ( probably, yes, this entire paragraph ) when I proof-read it later ;) Now that my neuroses have been shoved out the back-door, we'll get on with it.

Today we're going to look at a quick way to adapt an existing Veritas/Symantec NetBackup setup to a new set of peripherals on Solaris Unix. Due to the sheer amount of output options (which may or may not be of any use to you ever) available in almost all of NetBackup's commands, I've made my examples somewhat lengthy (although, I've cut a lot out. Those points are somtimes highlighted by ellipses - not to be confused with the ellipses used to convey the sense that my mind is wandering and my inner-or-outer-voice is trailing off into an inaudible mutter... Ellipses are really hard to convey emotion with... Thank God for the winky-eyed thingy - here's hoping I've been doing it to the correct side my entire career ;)

So, for today, we're going to assume you're running Veritas/Symantec NetBackup, approximately version 6.5.x. We'll also assume that you have NetBackup fully installed on Solaris x.x and may or may not have any number of add-on's (like Oracle or Vault modules, etc). The add-on's generally won't play into this, so we're going to forget they exist. Removing extraneous elements from the equation (if you can reasonably afford to discount them) is the best way to troubleshoot (which we won't be doing today, but I had to include that line to set up the next post... ellipses again ;) Ideally, you'd only have NetBackup running, with no add-on's or extras except the Standard License. I don't know how often that happens. I've never worked any place that didn't, at least, have multiple licenses for backing up multiple OS's. But, there you have it. Your basic setup.

Also, probably important, today we're going to look at the way things "should" go for you (with multiple commands or options to explore where possible). Tomorrow, and/or the day after, we'll look at how to figure out (and fix) what's wrong when things "don't" go as they should.

THE SITUATION: Your old tape robot, along with the drives inside it has inexplicably gone bad, you've spent hours and exhausted your support contract trying to fix it somehow, but, ultimately, you're left facing the fact that your trusty old steel-and-plastic-jukebox just isn't going to come back. Ever. If you're lucky, your warranty allows for replacement of the tape robot (A Tape Loading Device or TLD) and its internal drives (2, for now, to keep things simple - hcart2 drives, just because). Worst case you've purchased suitable replacements that match the specifications listed in the previous sentence.

Probably, your /dev/rmt directory is populated and you may even have some other logical paths created on your Solaris system that are no longer valid. Once you've connected your "MaxTape24" TLD (which exists only in my imagination, with it's two internal "FACTOTUM-TD2" drives, both working properly according to the on-board diagnostics), you should be able to verify that your system (at the very least) can recognize the TLD and, hopefully, the drives inside it. Assuming all of the equipment is good, and that it's been hooked up (however you like to daisy-chain it) properly, this shouldn't be an issue. You may choose to run:

host # devfsadm -C

before proceeding, to check for new symbolic links that need to made in your hardware device tree (and, with the -C option, remove ones you no longer need - Operating System's discretion, unfortunately), although it may not be necessary.

FINDING THE NEW HARDWARE WITH NETBACKUP FIRST: Now, contrary to what it seemed like I was leading in to, we're going to try to get NetBackup to do all the OS-work for us today (because, if it works, it's f'ing brilliant. Good job. Go home and relax :) Actually, you could probably look at this more as a way of giving NetBackup a good kick in the arse. The kind of kick that makes it stand up and take account of its surroundings ;) A good way to get started is to run the following at the command line (Oh yes, there will be no GUI instruction in these posts. If you use the GUI - which is okay - just right click on the type of thing you want to do something to and select whatever seems to be the most reasonable option from the drop-down menu. ...last on that:)

host # /usr/openv/volmgr/bin/sgscan <-- I would recommend including /usr/openv/volmgr/bin, /usr/openv/netbackup/bin and /usr/openv/netbackup/bin/admincmd in your PATH variable if you spend a lot of time working with NetBackup at the command line.

/dev/sg/c0t0l0: Disk (/dev/rdsk/c0t0d0): "SUZUKI MBB2147RCSUN146G"
/dev/sg/c0t1l0: Disk (/dev/rdsk/c0t1d0): "SUZUKI MBB2147RCSUN146G"
/dev/sg/c0t2l0: Tape (/dev/rmt/1): "BMI FACTOTUM-TD2"
/dev/sg/c0t3l0: Cdrom: "Hyundai DV-W28E-R"
/dev/sg/c1t0l0: Changer: "TLDHAUS MaxTape24"
/dev/sg/c1t1l0: Tape (/dev/rmt/0): "BMI FACTOTUM-TD2"


Your output may differ (even if you run this command on the same box, since I faked up the output to protect the guilty ;), but basically, this output is positive. You'll notice that sgscan has picked up a bit more than just your new TLD and its drives but that's okay. As it stands, this output is very positive, in that you can see that /dev/rmt/0 and /dev/rmt/1 have been properly mapped to the TLD's internal tape drives and the "TLDHAUS MaxTape24" TLD has been properly identified.

Other commands you could use to, basically, get the same information (or peace of mind) would include (but not be limited to) vmoprcmd, tpconfig and tpautoconf. A few examples at the bottom of the post, with the same setup as above (some whitespace has been clipped to save the virtual trees).

And that's it for today. Tomorrow we'll look at several commands (including some we're using today, but with different options) that can be used to "find" those drives if the system doesn't discover them automatically (the first thing you can try is "devfsadm -C" as noted above, followed by another sgscan).

Until then, enjoy the output and we'll continue on tomorrow. Here are a couple of handy anchor-href's for you, so you don't have to try to figure out where the command you're interested in is hiding out amongst all the flotsam below :)

vmoprcmd
tpconfig -d
tpconfig -dl
tpautoconf -t
tpautoconf -a


Cheers,

vmoprcmd
host # vmoprcmd
HOST STATUS

Host Name Version Host Status

========================================= ======= ===========

host 652000 ACTIVE

host 652000 ACTIVE-DISK

PENDING REQUESTS



DRIVE STATUS

Drive Name Label Ready RecMID ExtMID Wr.Enbl. Type

Host DrivePath Status

=============================================================================

Drive000 No No No hcart2

host /dev/rmt/0cbn TLD



Drive001 No No No hcart2

host /dev/rmt/1cbn TLD



tpconfig -d
host # tpconfig -d

Id DriveName Type Residence

Drive Path Status

****************************************************************************

0 Drive000 hcart2 TLD(0) DRIVE=1

/dev/rmt/0cbn

1 Drive001 hcart2 TLD(0) DRIVE=2

/dev/rmt/1cbn DOWN

Currently defined robotics are:

TLD(0) robotic path = /dev/sg/c1t0l0

EMM Server = host



tpconfig -dl
host # tpconfig -dl

Currently defined drives and robots are:

Drive Name Drive000

Index 0

NonRewindDrivePath /dev/rmt/0cbn

Type hcart2

Status UP

SCSI Protection SR (Global)

Shared Access No

TLD(0) Definition DRIVE=1

Serial Number 1234567890



Drive Name Drive001

Index 1

NonRewindDrivePath /dev/rmt/1cbn

Type hcart2

Status UP

SCSI Protection SR (Global)

Shared Access No

TLD(0) Definition DRIVE=2

Serial Number 0987654321

Currently defined robotics are:

TLD(0) robotic path = /dev/sg/c1t0l0

EMM Server = host



tpautoconf -t
host # tpautoconf -t

TPAC60 BMI FACTOTUM-TD2 4C60 1110365040 -1 -1 -1 -1 /dev/rmt/0cbn - -

TPAC60 BMI FACTOTUM-TD2 4C60 1110364981 -1 -1 -1 -1 /dev/rmt/1cbn - -



tpautoconf -a
host # tpautoconf -a

TPAC60 - - - -1~-1~-1~-1 2 - - - 0 - - - - - - - 0 0 - - host 4 - - 0 - - - -

TPAC60 Drive000 BMI~~~~~FACTOTUM-TD2~~~~~4C60 /dev/rmt/0cbn -1~-1~-1~-1 1 0 0 2 8 10 - - - 1110364981 - 3 0 0 - - host 0 - - 0 - - - -
TPAC60 Drive001 BMI~~~~~FACTOTUM-TD2~~~~~4C60 /dev/rmt/0cbn -1~-1~-1~-1 1 0 0 1 8 10 - - - 1110365040 - 3 0 0 - - host 0 - - 0 - - - -
TPAC60 0 TLDHAUS~~~~MaxTape24~~~~~~~310A /dev/sg/c1t0l0 -1~-1~-1~-1 0 0 0 - 8 - 23 2 1 TLDHAUS_1_9A0206A12 - 2 0 - host host host 0 - - 0 - - - -


, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Monday, February 23, 2009

Sun M-Series Enterprise Unix Servers And DSCP - Why?

SPECIAL NOTE: Friday's post on the absorption of knowledge was written on Blogspot in IE7 and suffered from some horrible display problems on FireFox and other browsers. It has been completely retooled. Apologies for any inconvenience that post caused. Even though I write some goofy rants, missing text wasn't supposed to be part of that post's intentionally cryptic style. Thanks :) - Mike

Hey there,

Today, we're going to look at DSCP (The Domain to Service processor Communication Protocol) and why it was introduced with the newer M series Enterprise servers from Sun. The first time I ran into it, I thought approximately the following: All right, here's something else that can make my life more complicated ;) Now, not only do I have to hop onto a private network (added security at work) to connect to the service processor (SP), but then I have to be sure that an entirely separate network is functioning correctly in order for me to connect to a domain on my machine if I want to get there directly from the SP. I'm naturally negative, but I keep it inside until I've given myself the time to realize that things are either not as bad or better than I initially assumed, which goes a long way toward maintaining good relationships with other human beings ;)

DSCP, although it "is" another layer of complication, is actually a pretty good idea. In essence, it's an attempt at damage control and resource sharing (which almost never works entirely), on par with earlier "HUGE" Sun computers, which does a pretty decent job at a relatively low cost. And, I use the word "cost" in reference to the amount of work you have to put into getting it all set up and working correctly. It's practically free, in that respect, so kudos (The plural, pronounced kyu-doze) to Sun for adding a little extra protection and making it an additon to an existing model that doesn't encumber the admin or user to a noticeable degree. Once it's set up, it may as well be non-existent. ...unless you have problems with it, but that's a general rule when it comes to anything.

Inter-process communication on the 6800/6900, etc boxes was handled pretty decently as well, but didn't really lay ground for the possiblity of scaling easily in the future. On those boxes, shared memory (for example) was "stuck" to a single SP, so if you had multiple SP's, you essentially had multiple machines, each alienated from the other with regards to system resources. The 6800/6900's were, as one would expect, worse in this regard than the 15k-and-up models (check out the price tags. Some times it can be an indicator of quality. Is the difference in ease of sharing worth it? Depends on your situation)

The 15k-and-up servers ran physical ethernet from the service processor(s) to the domain controllers (whereas, if I forgot to mention above, the older mailbox-architecture was imbedded and much more difficult to "change"). This setup made it slightly easier to re-allocate resources since you basically had a MAN setup on your Sun system (Maintenance Area Network for those of us who still care ;). Since the MAN operated at the application layer of the stack, all you had to do to scale up when you added a new product or application was assign a new TCP port. And that was that: Sharing made much simpler. There's a lot of nitty-gritty behind it all, but who wants to read about that? If you do, check out docs.sun.com and go nuts ;)

So, now, we've made it all the way up to the almost-present (we'll just consider it the "now" for now ;) and the M-Series servers. Since the MAN configuration of the 15k-and-up servers worked out so beneficially, that concept was guaranteed to be built in to the next generation of Enterprise computers. The one big drawback, from an architectural perspective, was the fact that the 15k-and-up servers had a whole bunch of externally exposed cables patched all over the place to maintain that separate MAN network (lots of ways to goof that up). The DSCP is Sun's way of taking care of that issue while maintaining the extended reliablity (and security) introduced with the MAN concept.

DSCP reproduces the 15k-and-up MAN using shared RAM, a pseudo-serial driver and PPP. To the user, this means all the benefits of a MAN, without all of the cords ;) Actually, it's a very nice implementation of physical-to-virtual transformation. And, as we all know, in about 10 to 15 years, we'll all be virtual and the only people left alive on the planet earth will be maintenance technicians ;)

The best thing about DSCP is how incredibly complicated it is to set up. Just kidding ;) It's actually incredibly simple. So simple that, if you're like me, you're wondering when a monkey is going to be able to start doing your job ;) Assuming an unlimited amount of Domains attached to a service processor, setup of DSCP is as simple as this:

XSCF> setdscp -i 10.0.0.1 -m 255.255.255.0
Commit these changes to the database? [y|n] : y


or, even simpler:

XSCF> setdscp
DSCP network [10.0.0.1 ] >
DSCP netmask [255.255.255.0 ] >
XSCF address [10.0.0.2 ] >
Domain #00 address [10.0.0.3 ] >
Domain #01 address [10.0.0.4 ] >


These examples are from an M4000 with only 2 domains, but the flavour stays the same on the larger boxes in the series. The only thing you really have to worry about (just like your MAN networks) is that you pick a network that doesn't get used. Generally, as shown above, using a non-routable network is best practice. Although I accidentally typed a 24-bit netmask for a class C IP (and used 1 for the network instead of 0), it doesn't really make a difference.

Of course, displaying the information is just as simple (should you forget):

XSCF> showdscp

DSCP Configuration:

Network: 10.0.0.1
Netmask: 255.255.255.0

Location Address
---------- ---------
XSCF 10.0.0.2
Domain #00 10.0.0.3
Domain #01 10.0.0.4


The key benefit, aside from the easier resource sharing that carries over from the MAN days, is the extra protection each domain is provided. Assuming an exploit is commited against one domain, whoever's gotten onto your box and screwed up that configuration will have to work harder to get to the other connected domains, since they'll have to go through the XSCF to get there. There is no absolute direct connection between domains. Although, since I know somebody out there is thinking this, it "is" still possible to attack all the domains on an M-Series machine at once; just not in an overt fashion. For instance (and I'm not encouraging this behaviour in any way whatsover) if you can create a situation whereby the administrator needs to power-cycle his/her M-Series server to restore functionality to the exploited domain, you've just brought all of the domains down in one fell swoop.

And, in closing, if you can't get to the XSCF to check out the DSCP configuration and you have the proper privilege and access to a domain hosted on an M-Series server, you can obtain that same information using the prtdscp command. Even more convenient, you can SSH (assuming you've set it up) directly from your domain to the DSCP IP using a command similar to:

host # ssh `prtdscp -s`

If you work somewhere that can afford it, enjoy the convenience :)

Cheers,

, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Tuesday, February 10, 2009

Using Kstat To Check Your NIC Settings On Solaris 10 Unix

Aloha,

Hope you're having an enjoyable Friday and are looking forward to getting out of the office just as badly as I am. It's a long story, but I can't even begin to explain how close I came to just snapping like a twig and going incendiary on my cubicle, the office, the parking lot, the CostCo down the street and pretty much the entire business district. I swear; one more day of this and I might have ended up on the national news being brought down in a hail of bullets as entire Police departments and SWAT Teams on loan from neighboring states littered the urban landscape with speeding chunks of molten hot metal (killing the 100 or so odd "innocent" bystanders, of course). Thank God it's Friday. I never thought I'd ever say that and not feel like an imbecile, but, there you have it. I'm wasted. Worn out. Done. Through. The world has taken its sweet time this week and pummelled me like a crash-test dummy. One day at a time; that's what they say, right?

Hold the phone... I'm getting a late breaking news flash. It seems that today is actually Tuesday (??) Wait... no, it's Monday, definitely Monday (Dustin Hoffman, Rain Man, Judge Wapner, Jeopardy) and I'm writing Tuesday's copy. Okay... that, yes... that does mean that ...the work week has just begun... Man... that's... just... depressing... I... All right; buck up. Keep repeating: "I need this job. I need this job." Don't question authority, do what you're told, toe the line, don't buck the system, don't pass the buck, because the buck stops here. I guess I'll have to live up to the title of this post now... It appears I'm going to need Friday's paycheck. I can't possibly finance the kind of epic malfeasance I'm going to be burning to unleash, four days from now, on the change in my pocket and the fumes in my gas tank. So, hey :) Let's learn something we may already know :)

BTW (which means By The Way if you're too lazy to type the whole thing out, although the explanation kind of defeats the whole purpose... ;), the preceding two paragraphs were in jest. Seriously. I was only kidding, and I apologize to each and every one of you that came here for the few lines of actual cut-and-paste code and technical writing that follow. Of course, if you're a regular reader, you either enjoyed the above, hated it but couldn't stop reading (like not being able to look away from a car wreck, except, this is a just a bunch of words on a "page" and there aren't any cars involved ;) or just skipped right past it. If you're one of the last bunch, you're not even reading this right now, which means I could use this opportunity to make a few totally-uncalled-for remarks, but I won't. Just this once... On with the show :)

Today's we're going to take a look at using kstat on Solaris 10 ( It's actually available on Solaris 9, as well - I'm sure not sure, off the top of my head, about 8, but I think they were still using "netstat -k") to find out the three things you most often want to know about your network interface cards. Of course, the three things you'll want to be able to check on, arise from three separate problem-areas (which almost always get lumped together to some degree). Basically, you'd never need to know this stuff if your NIC's came, out-of-the-box, all cranking at the highest speed possible, transmitting as efficiently as possible and only engaging in worthwhile speed/transfer/protocol negotiations with upstream/downstream routers. But, hey, sometimes the world's not perfect ;)

1. How to check your NIC's link speed with kstat:

They say they sent you a Gig card, but you could swear it's chunking along at 100 meg. You're probably right (or really really really impatient ;), but it never hurts to know for sure (insofar as you can know anything for sure - BTW, be sure to check out my post-existential-emo-nihilist poetry at PleaseKillMeButDon'tMakeItHurt.com ;). This command line should give you that info (-m is the argument for your NIC driver type - might be ce or something else, and -i is for the instance of that driver. In this instance we're using the device driver bge0. -s is for the statistic you're looking up, if you want to specify that; which we do, for now):

host # kstat -m bge -i 0 -s link_speed|grep link_speed|awk '{print $2}'
1000


And, YES, according to the OS, that NIC (or that port, anyway) is performing to expectations. Running at a gig. Fantastic!

2. But, what about the duplex? This is a legitimate concern. I've actually never seen any card run at 1 gig half duplex, but it "is" possible, or they wouldn't account for it. You'll usually call duplex into question when you're running at 100 meg (on a gig-capable card and network) or 10 meg, given the same circumstances. This is easy to find out, too. Just hire a team of monkeys to hammer the keyboard until they come up with this sequence of characters ;)

host # kstat -m bge -i 0 -s link_duplex|grep link_duplex|awk '{print $2}'
2


Yet again, SUCCESS! If you wanted full duplex, anyway. That's what the 2 stands for. You could also end up with a 1 (half-duplex) or a 0 (which, technically, means your link is down. If you're not connected through serial console, ALOM, or something of that sort and you get this result, your version of kstat is either lying to you or it's hurt and confused ;)

3. And, finally, we should probably figure out if our NIC is set to auto-negotiate. This setting is less "definite" than the other two, since the folks who configure your network may require that your NIC be set to auto-negotiate, or (this happens a lot) train specifically to a determined speed and duplex. Auto-negotiation is always the easiest thing for both parties involved, if it works, but if you need to force your NIC to run at 100 meg full duplex and turn off auto-negotiation in order for your server to run on the network, that's that. You either do it or people start to wonder why they're paying you ;)

host # kstat -m bge -i 0 -s cap_autoneg|grep cap_autoneg|awk '{print $2}'
1


This is either great news or the beginning of a headache that could last weeks and drain you of whatever shattered will you have left ;) If you get a 1 back from this command, auto-negotiation is enabled (In a perfect world, you're Golden) and, if a 0 comes back, you're not doing any auto-negotiation.

If you want to change any of these values, check out this really old post we did on figuring out your NIC's speed and duplex on Solaris (to set the properties, just change the -get option for "ndd" to -set). It'll get you all sorted out :) And, yes, that post is somewhat similar to this one, but it's from November 2007 and we were still all hung up on using "netstat -k" to get our info back then ;)

Keep smiling. Even if it hurts ;)

Cheers,

, Mike




tbuskey noted this with regards to the kstat command and link speed attribute. Thanks!


This still depends on the hardware. EDITOR NOTE: For this card, tbuskey notes that ifspeed is what you should be looking for!

I have an nge card. kstat -m nge -i 0 | grep link
link_asmpause 0
link_autoneg 1
link_duplex 2
link_pause 0
link_state 1
link_up 1
link_duplex 2
link_state 1
$ kstat -m nge -i 0 | grep speed
bus_speed ifspeed 1000000000
ifspeed 1000000000



Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, December 24, 2008

Taking Full Advantage Of "Who" On Solaris

Hey there,

First things first: Happy Christmas Eve :) Our apologies to our readers who celebrate different festivities this time of year. It's not that we don't want you to enjoy them (although our wishes are probably belated for some), just that we almost forgot about our own... What does that say about us? We don't know either ;) In the spirit of "it's never too late" - Happy Holidays (whatever they may be) from all of us :)

This post is more of an expansion, than a follow-up, to our May 2008 post on using who to find out what and when on Linux and Unix (picking only the options that appeared in most distro's version of "who"). It's also a bit more restrictive than our previous post, as we're restricting the discussion to newer Solaris' (8/9/10) implementation of the classic command (Okay, "according to Hoyle" (and I'm using that idiom loosely, since this post has nothing to do with the game of Whist ;), it can't be a classic for another 12 years, but if TNT can televise "new classics" that just came out last year, the meaning of the word has either been completely devalued or we're making liberal use of artistic license. We prefer the latter justification ;)

Solaris' implementation of who is a great deal better than in older versions, and turns what used to be a "mostly" informational command into a command that you can use to really get to the bottom of things. Following, a list of Solaris who's usage parameters and a brief discussion of each. One thing that should be noted is that the Solaris who command has no "help" switch (-h, --help, nothing), and it will produce output if you just run it without arguments. We usually go with "who -h" (since -h isn't an option) to get usage/help output. You'll have to subject yourself to the humiliating error message, but it's generally worth it ;)

Ladies and gentleman, The Options :)

1. Who straight-up (what better way to start a list of options than with the no-option option? This is what you get if you settle. Four columns: User ID (NAME), terminal (LINE), date (TIME) and local-screen/IP-address:

host # who
user001 console Dec 8 09:48 (:0)
user001 pts/3 Dec 8 09:50 (:0.0)
user001 pts/5 Dec 23 08:56 (10.99.99.99)


2. -a. This option is fairly obvious. It stands for "all" and is the equivalent to using "-Abdlprtu" which, the astute among you may have noted, does not comprise the entirety of the options available ;) We'll go over all the options that are lumped together in this option shortly (If I had more time to proofread, I wouldn't have made the word option a tag for this post ;).

host # who -a
. system boot Dec 8 09:46
. run-level 3 Dec 8 09:47 3 0 S
zsmon . Dec 8 09:47 old 307
LOGIN console Dec 8 09:47 old 331
user001 + console Dec 8 09:48 old 663 (:0)
user001 + pts/3 Dec 8 09:50 old 1015 (:0.0)
user001 pts/4 Dec 8 10:37 old 0 id= /4 term=0 exit=0
user001 + pts/5 Dec 23 08:56 . 17914 (10.99.99.99)
user001 pts/4 Dec 11 16:15 old 28003 id=ts/4 term=0 exit=0
user001 pts/6 Dec 19 10:39 old 11406 id=ts/6 term=0 exit=0
user001 pts/7 Dec 18 16:40 old 9997 id=ts/7 term=0 exit=0
user001 pts/8 Dec 16 14:05 old 5804 id=ts/8 term=0 exit=0


3. -b. This option will show you the last time the system booted. This can be very helpful (especially when used in combination with "last -x" to make use of last's full potential):

host # who -b
. system boot Dec 8 09:46


4. -d. This option purports to list out "dead" processes. The reason we phrase it in that way is that, generally, these processes (or pseudo terminals) may have been used before, but this doesn't mean that they're hanging around like a bunch of zombie processes. For instance, the following output lists 5 "dead" pseudo terminals, although none of them can be found in the output of either "ps" or "lsof" (???) In any event, it's a cool feature :)

host # who -d
user001 pts/4 Dec 8 10:37
user001 pts/4 Dec 11 16:15
user001 pts/6 Dec 19 10:39
user001 pts/7 Dec 18 16:40
user001 pts/8 Dec 16 14:05


5. -H. This one is a fantastically fun joyride through the land of the obvious. It forces who to print out the header for each column it reports on (although it does forget about the display/IP column noted in straight-up "who" output from point 1):

host # who -H
NAME LINE TIME
user001 console Dec 8 09:48 (:0)
user001 pts/3 Dec 8 09:50 (:0.0)
user001 pts/5 Dec 23 08:56 (10.99.99.99)


6. -l. Using who with the -l option lists out only "login" processes. Basically, it will only report on logins that are logged in (or appear to be logged in) to the localhost directly (no external pseudo terminals):

host # who -l
zsmon . Dec 8 09:47 old 307
LOGIN console Dec 8 09:47 old 331


7. -q. This will perform a quick who (only showing the NAME field), and is the only option that the -n option works with. If you use -n with -q, you can specify the number of returned processes you want to see per line of output, at most. By default, who -q tries to return as many results as possible on a single line:

host # who -q
user001 user001 user001

who -q -n 2
user001 user001
user001


8. -r. This option will let you know what run level your system is currently at (again, check out our previous post on using who for more specifics on all of the output "who -r" produces:

host # who -r
. run-level 3 Dec 8 09:47 3 0 S


9. -s. This option is considered the "short form," since it doesn't report any "time since last login," session activity status or PID output. who, run with this option alone is actually the default output. The one time this comes in handy is when you're using it with -a (and -H for the headers, if you want), and want to trim that output a bit. Otherwise, using this option wouldn't make sense, since you'd have to specify the flags to print the two fields you want removed ;)

host # who -s
user001 console Dec 8 09:48 (:0)
user001 pts/3 Dec 8 09:50 (:0.0)
user001 pts/5 Dec 23 08:56 (10.99.99.99)

host # who -asH

NAME LINE TIME
. system boot Dec 8 09:46
. run-level 3 Dec 8 09:47 3 0 S
zsmon . Dec 8 09:47
LOGIN console Dec 8 09:47
user001 + console Dec 8 09:48 (:0)
user001 + pts/3 Dec 8 09:50 (:0.0)
user001 pts/4 Dec 8 10:37
user001 + pts/5 Dec 23 08:56 (10.99.99.99)
user001 pts/4 Dec 11 16:15
user001 pts/6 Dec 19 10:39
user001 pts/7 Dec 18 16:40
user001 pts/8 Dec 16 14:05


10. -t: This option will show you all the times that your system clock was reset (and, yes, sometimes this output can be empty, for reasons that require no explanation ;)

host # who -t
host #


11. -T. This flag shows your tty status (referred to also, above, as session activity status). The + symbol indicates that the tty's status is "writable," the - symbol indicates that the tty's status is "not writable" and the ? symbol indicates general confusion ;) It just means that the system has no idea what the tty's status is, which generally means that it's hung:

host # who -T
user001 + console Dec 8 09:48 old 663 (:0)
user001 + pts/3 Dec 8 09:50 old 1015 (:0.0)
user001 + pts/5 Dec 23 08:56 . 17914 (10.99.99.99)


12. -u. This flag lists out (and I'm quoting from the "usage" output) "useful information." That isn't to say that any other output you can get from who is completely useless. Although, the terminology does seem to cast a shadow... ;)

host # who -u
user001 console Dec 8 09:48 old 663 (:0)
user001 pts/3 Dec 8 09:50 old 1015 (:0.0)
user001 pts/5 Dec 23 08:56 . 17914 (10.99.99.99)


13. -m. This flag limits the information to the current terminal session only. As you can see below, we're logged in using pseudo tty /dev/pts/5:

host # who -m
user001 pts/5 Dec 23 08:56 (10.99.99.99)


14. And, to begin the wrap-up, Solaris' who makes up for the fact that it doesn't have a built in handler to deal with being called as "whoami" by providing two different options to get that same information. In an alarming show of disregard for proper capitalization, both of these versions work ;) Note that this output is almost always exactly the same as the output from "who -m":

host # who am i
user001 pts/5 Dec 23 08:56 (10.99.99.99)
host # who am I
user001 pts/5 Dec 23 08:56 (10.99.99.99)


follow that up with the question that, statistically, follows "who am I?" most often, give it a little bit of "Talking Heads" flavour, and you've got yourself a command that's completely useless ;)

host # my God, what have I done?
-bash: my: command not found


15. Back off the Road To Nowhere (David Byrne, again. Make him stop!! ;)... Lastly (no pun intended, as you'll understand by the end of this paragraph), you can use who, using any combination of options (with the exception of -n, which only works with -q), and follow it all up with a different utmpx file (if, for instance, your old one got to big and you copied it off somewhere). Straight up who on Solaris makes use of /var/adm/utmpx, but you can tell who to use any utmpx-like file (including wtmpx, which can make the "who" command emulate the "last" command to a basic degree):

host # who /var/adm/wtmpx
root console Nov 6 13:39
root console Nov 6 13:48 (:0)
root pts/3 Nov 6 13:51 (:0.0)
root pts/3 Nov 6 14:38 (:0.0)
root pts/4 Nov 6 14:39 (:0.0)
user001 sshd Nov 7 09:48 (host.subnet.domain.com)
user001 pts/4 Nov 7 09:48 (host.subnet.domain.com)
user001 sshd Nov 7 09:54 (host1.subnet2.domain3.com)
user001 pts/5 Nov 7 09:54 (host1.subnet2.domain3.com)
...


Here's hoping today's post help shed a bit more light on Solaris' who options than the standard usage screen does, pointed out a number of reasons it can be a great tool to have in your troubleshooting arsenal and (perhaps) taught you a trick or two :)

Cheers,

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Thursday, December 18, 2008

Quick XSCF Setup Walk-through For The Sun M4000 Server

Hey there,

You may have noticed (sometime in the last year or so ;) that I like to write about Sun computers and Solaris almost as much as I like to write about the many variants of Linux out there, without which I might not even have gotten interested in computing in the first place. Fortunately, or unfortunately, Sun still doesn't pay me residuals to write about their stuff. Admittedly I haven't, yet, screwed up the nerve to attempt brokering a pay-per-post deal with them (or anyone, for that matter). The end result would, of course, be the sound of a closing door, at best. Perhaps I underestimate the kindness of multi-million dollar corporations. One day, I'll do it, just for kicks (the worst that could possibly happen is that they'd crush me under their platinum boot-heels ;). You'll be the first to know the degree to which I'm ignored :)

In any event, since I use their products so often in my work, writing up how-to and tips-type posts regarding the use of them just makes sense. I think of myself as fairly creative (evidenced by the fact that I still exist, in spite of all the stupid, and hazardous, things I did in my youth ;), but every once in a while I have to defer some of the solid tech stuff to get it out of my system and give my noggin a break. By "solid tech stuff" I mean step-by-step walk-throughs of processes and practices, etc. Things that you can read in a manual that differ very little from what I've actually done. The facts (unlike the methods) can't be disputed. I'll admit that, for me, sometimes writing these kinds of posts is a bit mentally-boring but, if I started writing posts about all the goofy sh## I think about every day, this blog would quickly lose focus. I'd have to rename it "The Mongolian Cluster Fugg Menagerie" or something equally descriptive of what you could expect to not rely on reading every day ;) But, enough about my trivial problems. Life is generally good. My main gripe is that I have to work in order to be able to afford to feed my family. Crocodile tears ;)

This link will take you to a walk-through of the basic setup of the XSCF controller (Comparable to ALOM on some Sun systems, and the Service Controller on some others) on the new(ish) M4000 servers. Believe it or not, the hardest part was figuring out how to get started, even though I've worked on them, replacing system boards before. How embarrassing ;) These few pointers might be helpful to you (and, in the process, expose the possibly entertaining inner workings of a sleep-deprived mind ;)

1. Rather than a straight Ethernet connection, like on some of Sun's servers, the M4000 XSCF controller needs to be connected to with a serial cable (they include one with your purchase as a way of saying "Thank You" for spending 10's of thousands of dollars ;)

2. The correct port is not identified, typically, as a serial port. For ease of location, it's the second port (RJ45) in from the right, if you are looking at the server from the back.

3. Even though the XSCF has power running to it, you actually need to turn on the M4000 in order to do the initial setup through the M4000's serial connection to the XSCF (This one had me stumped for about a half an hour while I tried different cables and setups in HyperTerminal. There's just no substitute for reading the manuals that it takes forever to locate online ;) They didn't come with the physical product this time, so I just assumed a few things that ended up making my day go by much faster :)

4. For some reason, I couldn't get these basic manuals without a login to Sunsolve, although they're probably available on docs.sun.com somewhere. If you need to do any hardware work on M4000 or M5000 servers, I put up two essential guides (in PDF format) on one of my web hosting providers. These are worth their weight in paper ;)

The M4000/5000 Server Information Guide

The M4000/5000 Server Service Manual


Hope you can get some use out of those. My wife is obsessed with painting the kitchen - which is why I'm reasonably sure she won't ever read this - and I'm stuck with putting the kids to bed, which means an early night for me and more hallucinogenic dreams about Sponge Bob ;)

To leave you with another quick tip (if you need to install and get the Hell outta there ;), during our initial configuration, we intentionally opted not to set up the DSCP (Domain To Service Processor) protocol during our setup of the XSCF (Extended Control Facility). Once you have the basic networking set up, you can ssh in and set that up later. The DSCP is "important" because it's the protocol XSCF uses to communicate with your server (i.e. if it's not running, you can connect to your domain or console, but good luck getting it to manipulate your M4000 ;)

Hope you're having a peaceful evening, and that the official Sun XSCF Setup Documentation helps you some :)

Cheers,

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Tuesday, December 2, 2008

Predicting Solaris 10 TCP Sequence Numbers Part 1: Initial Discovery

Hey there,

Note: All examples today were produced using Solaris 10. As such, commands like snoop may need to be replaced with ethereal or tcpdump, etc, depending on your OS and/or preference.

Today, we're going to look at a subject that gets plenty of attention (but, maybe not as much publicity) in the world of Unix and Linux network computing: TCP packet sequence number prediction, how it's used to protect network transmissions and whether or not, with the advent of packet checksumming, etc, it's even a factor in basic network security anymore. See here, or at the very bottom of the post, for something potentially interesting about the latest Solaris' network protocol enhancements including SDP.

Not all that long ago (which might mean a long time ago to you, since I'm growin' old ;) TCP number sequencing was pretty simple to predict. In essence, the sequence numbering of TCP packets is required since TCP does its own error-correction and will attempt to resend packets that it doesn't "know" reached their specified endpoint. Numbering, at the most basic level, was/is a great way to ensure that all packets sent actually get received. And, believe it or not, at one point in time that numbering was simply incremental. That means that, given a point-to-point TCP transaction, a simple transmission (from the sender's point-of-view, for instance) might consist of 15 packets. If the first packet was numbered 1 (a gross oversimplification, to be sure, but not too far from being exact ;), the next would be numbered 2, then 3 and so forth until the 15th packet (numbered 15) was successfully sent.

Given the above scenario, and with the aid of hindsight, TCP connection hijacking and other spoofing-based attacks were made quite a bit simpler than they might have been. Of course, once all this "bad" activity began happening, the specifications for TCP packet sequence numbering began to change and become more complicated. Many proprietary OS manufacturer's also began to implement their own proprietary sequence numbering algorithms (in accordance with standards so they could communicate with "unlike" systems). Unfortunately, most of these next-gen specifications were based on simplistic rules like (another gross oversimplification) simple mathematic derivations which didn't use any sort of "outside influence" to affect (or make more unpredictable) the sequence numbering. If you could figure out the equation, you could easily guess what packet number would be next, given one or two sequential packets, and then the whole mess would start all over again. Wikipedia actually has a very nice page on TCP with links to all the relevant RFC's.

This is actually still true today, although OS manufacturers, the Open Source community and white/black-hat hackers are stepping up their games to greater degrees and at a much more condensed pace (by which I mean things now, generally, change more often within a shorter period of time - another gross oversimplification ;)

So, now that it's the present, let's take a look at Solaris 10 and check out what they're doing with regard to this matter (From release 10/08 - the most recent to this date, as far as I know). Assuming we have a connection established with a Solaris 10 server (simple SSH), we can use snoop to gather the information we need to even begin attempting to figure out the TCP packet sequence numbering algorithm they're using (assuming - thinking positive thoughts :) that we can. A simple chunk of output is shown below:

host # snoop -r -d eri0 from 10.99.99.99 to 10.98.98.98
Using device /dev/eri (promiscuous mode)
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625835 Seq=673983919 Len=52 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625887 Seq=673983971 Len=52 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673984023 Len=52 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673984075 Len=340 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673984415 Len=148 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673984563 Len=148 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673984711 Len=148 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673984859 Len=148 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673985007 Len=148 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673985155 Len=148 Win=50400


Of course, the column we're most interested in (at first, anyway) is the 9th column from the left (assuming spaces and tabs as the default field separator). The column beginning with "Seq=" is what we'll look at first. For the sake of this exercise, we've saved all the packets in a file called "packets." We didn't use -o on the command line, but this is a pretty simple cut-and-paste ;). To isolate those sequence numbers, we could just do the following:

host # awk '{print $9}' packets|sed 's/Seq=//'
673983919
673983971
673984023
673984075
673984415
673984563
673984711
673984859
673985007
673985155


And, here's where it becomes difficult. The first test of a reasonably stable (by which I mean, unpredictable) sequence numbering algorithm is to subject it two basic tests. That is to say, is it really secure or is it just a simple numeric or mathematic progression in sheep's clothing?

As we bring part 1 of this endeavour to an end, we'll do the quick and dirty testing (If Solaris fails these, then I may have made a big mistake when I named this post before I finished writing it ;)

1. Are the numbers in simple sequence (e.g. +1 or +2) over and over again?
Answer: No. The numeric difference between the packet pairs (1-2, 2-3, etc) are not infinitely exact.
Proof: The 9 differences in order:
673983971-673983919 = 52
673984023-673983971 = 52
673984075-673984023 = 52
673984415-673984075 = 340
673984563-673984415 = 148
673984711-673984563 = 148
673984859-673984711 = 148
673985007-673984859 = 148
673985155-673985007 = 148


2. Is there any direct correlation between the differences between packet sequence numbers and any other property of the packets?
Answer: Yes. If we scroll up the page and look at the basic snoop output, we can see that the difference in sequence numbers correlates exactly with the length of the packet.
Proof: The packet lengths in order:
host # awk '{print $10}' packets|sed 's/Len=//'
52
52
52
340
148
148
148
148
148
148


It seems as though we have something going here, but it must be harder than this to guess the sequence numbers. It's possible, also, that additional measures are being used that obviate the somewhat-flimsy dependence on convoluted number sequences to safeguard a transmission from "packet injection" or hijacking. For instance, Solaris 10 now makes use of SDP as part of its data transmission security (see, also, the Sockets Direct Protocol Manpage). It may also be that Devil is in the details and we only looked at a surface scan of our packets. We'll look at all of these possibilities in a near-future post. This isn't over yet :)

Cheers,

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.