Quick And Easy Local Filesystem Troubleshooting For SUSE Linux

Hey There,

Today we're going to take a look at some quick and easy ways to determine if you have a problem with your local filesystem on SUSE Linux (tested on 8.x and 9.x). Of course, we're assuming that you have some sort of an i/o wait issue and the users are blaming it on the local disk. While it's not always the case (i/o wait can occur because of CPU, memory and even network latency), it never hurts to be able to put out a fire when you need to. And, when the mob's pounding on your door with lit torches, that analogue is never more appropriate ;)

Just as in previous "quick troubleshooting" posts, like the week before last's on basic VCS troubleshooting, we'll be running through this with quick bullets. This won't be too in-depth, but it should cover the basics.

1. Figure out where you are and what OS you're on:

Generally, something as simple as:

host # uname -a

will get you the info you need. For instance, with SUSE Linux (and most others), you'll get output like:

Linux “hostname” kernel-version blah..blah Date Architecture... yada yada

The kernel version in that string is your best indicator. Generally, a kernel-version starting with 2.4.x will be fore SUSE 8.x and 2.6.x will be for SUSE 9.x. Of course, also avail yourselves of the, possibly available, /etc/release, /etc/issue, /etc/motd, /etc/issue.net files and others like them. It's important that you know what you're working with when you get started. Even if it doesn't matter now, it might later :)

2. Figure out how many local disks and/or volume groups you have active and running on your system:

Determine your server model, number of disks and volume groups. Since you're on SUSE, you may as well use the "hwinfo" command. I never know how much I'm going to need to know about the system when I first tackle a problem, so I'll generally dump it all into a file and then extract it from there as needed. See our really old post for a script that Lists out hardware information on SUSE Linux in a more pleasant format:

host # hwinfo >/var/tmp/hwinfo.out
host # grep system.product /var/tmp/hwinfo.out
system.product = 'ProLiant DL380 G4'

Now, I know what I'm working with. If this specific of a grep doesn't work for you, try "grep -i product" - you'll get a lot more information than you need, but your machine's model and number will be in there and much easier to find than if you looked through the entire output file.

Then, go ahead and check out /proc/partitions. This will give you the layout of your disk:

host # /proc # cat /proc/partitions
major minor  #blocks  name

 104     0   35561280 cciss/c0d0
 104     1     265041 cciss/c0d0p1
 104     2   35294805 cciss/c0d0p2
 104    16   35561280 cciss/c0d1
 104    17   35559846 cciss/c0d1p1
 253     0    6291456 dm-0
 253     1    6291456 dm-1
 253     2    2097152 dm-2
 253     3    6291456 dm-3
 253     4   10485760 dm-4
 253     5    3145728 dm-5
 253     6    2097152 dm-6

"cciss/c0d0" and "cciss/c0d1" show you that you have two disks (most probably mirrored, which we can infer from the dm-x output). Depending upon how your local disk is managed, you may see lines that indicate, clearly, that LVM is being used to manage the disk (because the lines contain hints like "lvma," "lvmb" and so forth ;)

  58     0    6291456 lvma 0 0 0 0 0 0 0 0 0 0 0
  58     1    6291456 lvmb 0 0 0 0 0 0 0 0 0 0 0

3. Check out your local filesystems and fix anything you find that's broken:

Although it's boring, and manual, it's a good idea do take the output of:

host # df -l

and compare that with the contents of your /etc/fstab. This will clear up any obvious errors like mounts that are supposed to be up but aren't or mounts that aren't supposed to up that are, etc... You can whittle down your output from /etc/fstab to show (mostly) only local filesystems by doing a reverse grep on the colon character (:) - This is generally found in remote mounts and almost never found in local filesystem listings.

host # grep -v ":" /etc/fstab

4. Keep hammering away at the obvious:

Check the USED% column in the output of your "df -l" command. If any filesystems are at 100%, some cleanup is in order. It may seem silly, but usually the simplest problems get missed when one too many managers begin breathing down your neck ;) Also, check the inodes column and ensure that those aren't all being used up either.

Mount any filesystems that are supposed to be mounted but aren't, and unmount any filesystems that are mounted but (according to /etc/fstab) shouldn't be). Someone will complain about the latter at some point (almost guaranteed), which will put you in a perfect position to request that it either be put in the /etc/fstab file or not mounted at all.

You're most likely to have an issue here with mounting the unmounted filesystem that's supposed to be mounted. If you try to mount and get an error that indicates the mountpoint can't be found in /etc/fstab or /etc/mnttab, the mount probably isn't listed in /etc/fstab or there is an issue with the syntax of that particular line (could even be a "ghost" control character). You should also check to make sure the mount point being referenced actually exists, although you should get an entirely different (and very self-explanatory) error message in the event that you have that problem.

If you still can't mount, after correcting any of these errors (of course, you could always avoid the previous step and mount from the command line using logical device names instead of paths from /etc/vfstab, but it's always nice to know that what you fix will probably stay fixed for a while ;), you may need to "fix" the disk. This will range in complexity from the very simple to the moderately un-simple ;) The simple (Note: If you're running ReiserFS, use reiserfsck instead of plain fsck for all the following examples. I'm just trying to save myself some typing):

host # umount /uselessFileSystem
host # fsck -y /uselessFileSystem
....
host # mount /

which, you may note, would be impossible to do (or, I should say, I'd highly recommend you DON'T do) on used-and-mounted filesystems or any special filesystems, like root "/" - In cases like that, if you need to fsck the filesystem, you should optimally do it when booted up off of a cdrom or, at the very least, in single user mode (although you still run a risk if you run fsck against a mounted root filesystem).

For the moderately un-simple, we'll assume a "managed file system," like one under LVM control. In this case you could check a volume that refuses to mount (assuming you tried most of the other stuff above and it didn't do you any good) by first scanning all of them (just in case):

host # vgscan
Reading all physical volumes.  This may take a while...
  Found volume group "usvol" using metadata type lvm2
  Found volume group "themvol" using metadata type lvm2

If "usvol" (or any of them) is showing up as inactive, or is completely missing from your output, you can try the following:

host # vgchange –a y

to use the brute-force method of trying to activate all volume groups that are either missing or inactive. If this command gives you errors, or it doesn't and vgscan still gives you errors, you most likely have a hardware related problem. Time to walk over to the server room and check out the situation more closely. Look for amber lights on most servers. I've yet to work on one where "green" meant trouble ;)

If doing the above sorts you out and fixes you up, you just need to scan for logical volumes within the volume group, like so:

host # lvscan 
ACTIVE            '/dev/usvol/usfs02' [32.00 GB] inherit
....

And (is this starting to sound familiar or am I just repeating myself ;), if this gives you errors, try:

host # lvchange –a y

If the logical volume throws you into an error loop, or it doesn't complain but a repeated run of "lvscan" fails, you've got a problem outside the scope of this post. But, at least you know pretty much where it is!

If you manage to make it through the logical volume scan, and everything seems okay, you just need to remount the filesystem as you normally would. Of course, that could also fail... (Does the misery never end? ;)

At that point, give fsck (or reiserfsck) another shot and, if it doesn't do any good, you'll have to dig deeper and look at possible filesystem corruption so awful you may as well restore the server from a backup or server image (ghost).

And, that's that! Hopefully it wasn't too much or too little, and helps you out in one way or another :)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Determining Volume Group Disk Usage On AIX Unix Using Sh And Awk

Hey there,

Today's topic, and accompanying script, have to do with a more specific function already described in our series on working with AIX LVM (Actually, that link points to the last entry in the series. It's a strange thing about serialized content; the only way I can hook you up with a link that will get you links from part 4 to parts 3, 2 and 1 is to give you the last link. If only I could predetermine my following day's post URL without having to actually write the post... Actually, it probably can be done... I'll remind myself to work that into a post here soon. Hopefully I'll send myself an email to remind me before I forget ;)

This post has, loosely, to do with work. And I only mention this because I read in the online news that Google just canned some guy for writing about work in his blog! I'm not going to b.s. you and pretend like I read the entire thing, but, hopefully, he was posting company secrets. I'd hate to think what kind of an oppressive environment the corporate world would become if you could get your walking papers for ruminating over your day in an online diary. A lot of people have to work so much, they don't have much else to write about. Especially at Google. Why do you think they advertise all the games and cafeteria's and cots? Because it's a great place to work? They say; but perhaps it's because you're never ever going home again. The circle of madness has to be broken at some point ;)

In any event, working on an old AIX box, I was presented with a problem I hadn't considered for a while (on 5.x, anyway - I've been stuck in 3 and 4.x land for far too long ;) and, as I looked around for answers, I noticed that the information required to complete a pretty basic piece of work (and very useful, as well) just wasn't out there anywhere. Of course, the work involved using 2 (count 'em, 2 ;) separate commands, but... nothing. And that brings us to here.

The situation: Given a number of physical volumes, which have striped volume groups written across them, how can we figure out how much disk space (on how many different physical volumes and altogether) does that take up. I'm being intentionally vague, since the value we arrive at isn't as consequential as just being able to arrive at the value. Let's just say that we need to know because we need to replicate the disk space requirements elsewhere.

The answer: Surprisingly simple, once you get to it (although, perhaps because of my in-and-out AIX experience, confusing from time-to-time along the way). In order to extract this information, we just need to cruise through the following steps:

1. List out all available volume groups with the lsvg command:

host # lsvg
rootvg
number1vg
number2vg
...

2. Find out what physical volumes are associated with those volume groups using the same command and the "-p" switch. This time we'll need to name one of the volume groups. Steps from this point out would repeat for each individual volume group (and, later, physical volume) that comprise the logical and physical disk we're interested in (ellipses will be used to shorten lines for readability's sake):

host # lsvg -p rootvg
rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
xdisk0 active ...
xdisk1 active ...

3. Then, get the number of physical partitions, and the set physical partition size for each physical volume listed out with the "lsvg -p" command above (The only two lines we really need from this output are the only two that aren't cropped with more ellipses), using the lspv command:

host # lspv xdisk0
PHYSICAL VOLUME: xdisk0...
PV IDENTIFIER: 000...
PV STATE: acti...
STALE PARTITIONS: 0 A...
PP SIZE: 64 megabyte(s) LOGICAL VOLUMES: 11
TOTAL PPs: 542 (34688 megabytes) VG DESCRIPTORS: 2
FREE PPs: 3....
USED PPs: 1....
FREE DISTRIBUTION: 1....
USED DISTRIBUTION: 0....

and we're all set :) Now, we just have a big sloppy mess of output to deal with. Lots and lots of lines of garbage, with the numbers we need sandwiched in, that we can manually add together. So, we'll move on to the natural next phase and script this out. For that volume group "rootvg" (for instance), rather than seeing all the above times 2, 3 or however much it is, this is much nicer to look at:

host # ./pvinfo.sh
Volume Group: rootvg
-----------
Physical Volume: xdisk0
Number of Physical Partitions: 542 Physical Partition Size: 64 Segment Total 34688 Mb
-----------
Physical Volume: hdisk1
Number of Physical Partitions: 542 Physical Partition Size: 64 Segment Total 34688 Mb
-----------
Physical Volume Grand Total : 69376 Mb
...

and so forth for all of our volume groups. Now we'll know how much disk space is allocated to our volume group, in total, and how much is allocated from each physical volume associated with the volume group. Feeling much better now ;)

Hope this helps you out some!

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

#
# pvinfo.sh - get your AIX physical volume totals here!
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

for x in `lsvg|awk '{print $1}`
do
echo "Volume Group: $x"
echo "-----------"
lsvg -p $x|sed 1d|while read one two three
do
if [ $two == "active" ]
then
echo "Physical Volume: $one"
lspv $one|awk 'BEGIN{sum1=0;sum2=0}{ if ( $1 ~ /TOTAL/ && $2 ~ /PPs/ ){ sum1 += $3 } else if ( $1 ~ /PP/ && $2 ~ /SIZE/ ) { sum2 += $3 }}END{sum3=sum1*sum2; print "Number of Physical Partitions: " sum1 " Physical Partition Size: " sum2 " Segment Total " sum3 " Mb";print "-----------"}'
fi
done
lsvg -p $x|sed 1d|while read one two three
do
if [ $two == "active" ]
then
lspv $one|awk 'BEGIN{sum1=0;sum2=0}{ if ( $1 ~ /TOTAL/ && $2 ~ /PPs/ ){ sum1 += $3 } else if ( $1 ~ /PP/ && $2 ~ /SIZE/ ) { sum2 += $3 }}END{sum3=sum1*sum2;print sum3}'
fi
done|awk '{ sum4 += $1 }END{print "Physical Volume Grand Total : " sum4 " Mb";print "-----------"}'
done

, Mike

---

Chris Gibson wrote in with this helpful shortcut. He's got quite a wealth of AIX info, and we've posted additional comments from him on our LVM Series page :)

---

BTW, you can also do the thing same with the lsvg command:

$ lsvg rootvg | grep 'TOTAL PPs'
VG PERMISSION: read/write TOTAL PPs: 1150 (73600 megabytes)

In the example above, we have 73600Mb in rootvg.

Cheers.

Chris

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

LVM's Roots - Mirroring Your Boot Disk On HP-UX 10 Unix

Hey There,

If you read this blog every once in a while (or if you just happen to have ever searched for - or queried the tag named - LVM in our growing library of questionably-valuable articles ;) you've probably noted that, although mentioned in passing, none of them has ever dealt directly with HP-UX. AIX and, of course, Linux have received their fair share of attention. Even Solaris Volume Manager (or, if you still prefer it, Solstice Disk Suite) and Veritas Volume Manager have been covered in some detail with reference to their similarity to LVM. It's about time that HP-UX (arguably one of the mothers of LVM (or LVM2) as we know it today) should get some sort of treatment. This blog is, after all, dedicated to Linux and Unix (both terms being purposefully generic so we can write about whatever *nix machines we can get our hands on :)

Today's entry is a bit of a quick introduction to HP's Logical Volume Manager and was written specifically for an HP-UX 10.x box. We haven't specifically tested this against 11.x or 11i, but, from our experience working with both, this script should work with little-or-no modification on 11.x. Now that we've got a few HP servers to have fun with (I mean... work really hard on ;), we'll give HP-UX it's due and run through the essentials of LVM. We'll try to make it as short and sweet as possible, while not skimming over the basics, so that the posts themselves can serve as a decent reference for a straight-up HP-UX user. Actually, if you're an HP-UX user (Experience here ranges from 9.x through 11i - Old 800 series K class towers ( with matching WYSE terminals) to SuperDomes and some of the newer 9000 series), you're also well familiar with the huge differences in the basic functionality between versions of the OS and the ISL and GSP/BCH underpinnings (which you could, somewhat, liken to a difference between the Domain Console/System Controller setup on the big Sun 3800 through 25k servers and the newer XCP/XSCF setup on the Mx000 series).

Until that day, here's a little script to help mirror your root disks on HP-UX 10. This was actually tested and used on a K100 Server (Refurbished, of course, but smokin' fast with 4 100 MHz CPU's. Actually, pretty decent once it boots up okay :)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

#
# hpmirrordisk.sh - Double check this before you run it.  Seriously :)
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

echo "You have installed the MirrorDisk-HPUX Product already, right?"
echo "If you still need to, hit ctl-C and quit running this script!"
read oblig
echo
echo "This should show only one bootable disk to start out..."
echo
lvlnboot -v
echo
echo "New bootable physical volume name [c?t?d?]"
echo
read disk
pvcreate -B /dev/rdsk/$disk
echo
echo "Volume group you'll be mirroring [vg??]"
echo
read vgroup
vgextend /dev/$vgroup /dev/dsk/$disk
echo
echo "Making Mirror Disk Bootable..."
echo
mkboot /dev/rdsk/$disk
mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/dsk/$disk
echo
echo "This result output should give you back \"hpux -lq (;0)/stand/vmunix\""
echo
lifcp /dev/rdsk/${disk}:AUTO -
echo
echo "Return to continue"
echo
read oblig
echo
echo "Extending all logical volumes..."
echo
for x in 1 2 3 4 5 6 7 8 9
do
 /usr/sbin/lvextend -m 1 /dev/${vgroup}/lvol$x /dev/dsk/$disk
done
echo
echo "This should show two bootable disks now..."
echo
lvlnboot -v
echo
echo "Okay? Hit return"
echo
read oblig
echo
echo "For these tests, the Physical Extents should map to the"
echo "Logical Extents for each logical volume."
echo
for x in 1 2 3 4 5 6 7 8 9
do
echo
 echo "lvol$x"
 echo
 lvdisplay -v /dev/${vgroup}/lvol$x |sed -n '/Distribution/,/Logical/p'|sed '$d'
 echo
 echo "Okay? Hit return"
 echo
 read oblig
done

echo
echo "All you need to do now is reboot and be sure to halt the"
echo "system at the boot menu.  Make sure the Primary Boot Path"
echo "is set to the mirrored disk.  If not, you will have to set"
echo "the correct path in the COnfiguration menu.  Also, under the"
echo "COnfiguration menu, set Auto Search equal to ON."
echo "Finally, be sure to set the Alternate Boot Path to the"
echo "original disk"
echo

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

AIX Unix LVM Destruction Commands. Part Two Of Two

Hey there,

Today, we're finishing up what we started yesterday with our post on using LVM on AIX Linux. Today we're going to move on to Logical Volume, Volume Group and Physical Volume destruction commands. I also opted not to list out the usage commands (since there are quite a few of them, and we haven't covered this subject when looking at Linux LVM yet). It's fairly short and simple, but this is a "Lazy Sunday" :)

If you need help with some of the basic concepts of LVM, please check out our older post on getting started with LVM.

Happy, and hopefully Lazy, Sunday :)

AIX LVM Destruction commands:

Remove a logical volume:

host # rmlv MylogicalVolume

Remove a Volume Group (Note that, again it doesn't agree too much with the Linux or HP versions, we're just turning off the volume group to destroy it):

host # varyoffvg VolumeGroupName

De-Allocate a Physical Volume (Note that Physical Volumes, in AIX, already exist - they just need to be allocated to Volume Groups):

There is no specific pv command for this operation. PV allocation and de-allocation actually, for better or worse, happen as a result of the vg and lv destruction commands. However, you can do the following to "destroy" the device:

host # rmdev DEVICE

, Mike

---

Chris Gibson wrote in with these additional helpful comments. Thanks, Chris!

---

With regard to "Removing a volume group", I use this procedure:

# umount //datafs
/# rmfs //datafs/
# varyoffvg /datavg/
# reducevg datavg /hdisk2 hdisk3 hdisk4/

Reducevg will remove the volume group from the ODM.

Then I can remove the hdisk devices:

# rmdev -dl /hdisk2/
# rmdev -dl /hdisk3/
# rmdev -dl /hdisk4/

Also, if you want to move the VG to another system (or just temporarily remove the VG definition from the system), I use:

# varyoffvg /datavg/
# exportvg /datavg/

I can import it again later on the same or different system using:

# importvg -y /datavg hdisk2 hdisk3 hdisk4/

And one last comment, when I remove an LV I usually want the associated FS to be removed also, so I do:

# rmfs/ /datafs/

This will remove the file systems and the underlying logical volume.

Cheers

Chris

---

Using LVM on AIX Unix. Part One Of No More Than Two.

Hey there,

As you may recall, we just completed a short series of posts on using LVM on Linux and Unix. At the time that I wrote it, I made mention (probably several times) of the similarities between LVM for Linux and HP-UX's brand. Also at that time, since I had hardly any experience with AIX, I was completely unaware that LVM was being used on that operating system.

If you need help with some of the basic concepts of LVM, please check out our older post on getting started with LVM and then check out the series listed above.

Now, I think I'm not allowed to get "too" excited about this, since this blog is mostly about Linux flavours and Solaris Unix (peppered with a little HP and one old post on AIX commands for the Unix user), but I thought I'd put this little series of posts (possibly not in totally sequential order -- Sometimes things come up that I'm compelled to write about ;) up for the AIX enthuisast who is thinking of using, or just beginning to use, LVM on AIX.

I hope this post is helpful to you. In any event, it should make for a good reference and might possibly be useful "later" :) Today, we'll look at LVM creation and display/monitoring commands for AIX and our next post will be on usage and destruction! Please note that all examples are mocked up and any variable that starts with Optional (such as OptionalVolumeGroupName) is an argument that you don't necessarily need to pass to the command.

Enjoy :)

AIX LVM Creation commands:

Create a Physical Volume (Note that Physical Volumes, in AIX, already exist - they just need to be allocated to Volume Groups):

host # lsvpcfg <-- To show the disks/vpaths and any Volume Groups they may already be associated with.
host # lspv|grep None <-- To show available Physical Volumes
host # cfgmgr <-- Use this interactively to allocate the Physical Volume for use with LVM.

Create a Volume Group:

host # mkvg -y VolumeGroupName HardDiskNumber <-- You can get the hard disk number from the lsdev command
host # varyonvg VolumeGroupName

Create a logical volume within a volume group:

host # mklv -y LogicalVolumeName -t FileSystemType VolumeGroup PhysicalPartion(PP) OptionalSpecificHardDriveDefinition
Ex:
host # mklv -y MylogicalVolume -t jfs MyVolumeGroup 8 <-- Note that the PP size is in MB.
or
host # mklv -y MylogicalVolume -t jfs MyVolumeGroup 8 /dev/DiskName

AIX LVM Display/Monitoring commands:

List out information about physical volumes:

host # lspv OptionalPhysicalVolumeName

List out Physical Volume information by Hard Disk:

host # lspv HardDiskNumber

List out Logical Volumes Within a Physical Volume:

host # lspv -l HardDiskNumber

List out Physical Partitions within a Physical Volume:

host # lspv -p HardDiskNumber

List out information about volume groups:

host # lsvg OptionalVolumeGroupName

List out properties of logical volumes within a volume group:

host # lsvg -l VolumeGroupName

List out properties, including hard drives, within a volume group:

host # lsvg -p VolumeGroupName

List out properties of a logical volume:

host # lslv OptionalLogicalVolumeName

List out Logical Volumes within a Physical Volume:

host # lslv -l OptionalLogicalVolumeName

List out Logical Volumes by Physical Volume:

host # lslv -p OptionalPhysicalVolumeName

List out mirrored Logical Volumes:

host # lslv -m OptionalLogicalVolumeName

List out hard disks in a vpath (Virtual Path). Necessary after Physical Volume Allocation to ensure everything went ok:

host # lsvpcfg

List out Fiber Channel adapters:

host # lsdev -Cc adapter
host # scfg -vl Number of fiber channel card

, Mike

SunCluster Quick Command Reference

Hey There,

Well, sitting back and reflecting today, I realized that we've done our fair share of posts on cheat-sheets and such (Our LVM command reference, for example). However, when it comes to clustering systems, we've probably been unfair in our bias toward Veritas Clustering as opposed to SunCluster, for which we've only put up, I believe, one post with our Simple SunCluster Monitoring Script (Thankfully linked back to from The SunCluster Wiki). What's a poor blog to do? ;)

With that in mind, today we're going to lay down a quick command reference for SunCluster. Hopefully you'll find it useful and enjoy it :) Of course, it assumes that you know, mostly, how to use SunCluster (and are familiar with basic clustering concepts), but it can be used as an attractive wallpaper, if nothing else ;)

All of this information, and much more, can be found at Sun's Online Documentation Center for SunCluster.

Enjoy,

Quorum Commands:

host # clquorum add device - Add a SCSI Quorum Device
host # clquorum add -t netapp_nas -p filer =nasdevicename,lun_id =IDnumdevice Nasdevice - Add a NAS Quorum Device
host # clquorum add -t quorumserver -p qshost =IPaddress, port =portnumber quorumservername - Add a Quorum Server
host # clquorum remove device - Remove a Quorum Device

Resource Type Commands:

host # clresourcetype register type - Register a Resource Type
host # clresourcetype unregister type - Remove a Resource Type

Resource Group Commands:

host # clresourcegroup create group - Create a Failover Resource Group
host # clresourcegroup create -S group - Create a Scalable Resource Group
host # clresourcegroup online + - Bring Online All Resource Groups
host # clresourcegroup delete group - Delete a Resource Group
host # clresourcegroup delete -F group - Delete a Resource Group and All of Its Resources
host # clresourcegroup switch -n nodename group - Switch the Current Primary Node of a Resource Group
host # clresourcegroup unmanage group - Move a Resource Group Into the UNMANAGED State
host # clresourcegroup suspend group - Suspend Automatic Recovery of a Resource Group
host # clresourcegroup resume group - Resume Automatic Recovery of a Resource Group
host # clresourcegroup set -p Failback=true + name=value - Change a Resource Group Property
host # clresourcegroup add-node -n nodename group - Add a Node To a Resource Group
host # clresourcegroup remove-node -n nodename group - Remove a Node From a Resource Group

Resource Administration Commands:

host # clreslogicalhostname create -g group lh-resource - Create a Logical Hostname Resource
host # clressharedaddress create -g group sa-resource - Create a Shared Address Resource
host # clresource create -g group -t type resource - Create a Resource
host # clresource delete resource - Remove a Resource
host # clresource disable resource - Disable a Resource
host # clresource set -t type -p name=value + - Change a Single-Value Resource Property
host # clresource set -p name+=value resource - Add a Value to a List of Property Values; Existing values in the list are unchanged.
host # clresource create -t HAStoragePlus -g group -p FileSystemMountPoints=mount-point-list -p Affinityon=true rs-hasp - Create an HAStorage Plus Resource
host # clresource clear -f STOP_FAILED resource - Clear the STOP_FAILED Error Flag on a Resource

Device Administration Commands:

host # cldevicegroup create -t vxvm -n node-list -p failback=true vxdevgrp - Add a VxVM Device Group
host # cldevicegroup delete devgrp - Remove a Device Group
host # cldevicegroup switch -n nodename devgrp - Switch a Device Group to a New Node
host # cldevicegroup offline devgrp - Bring Offline a Device Group
host # cldevice refresh diskname - Update Device IDs for the Cluster

Additional Administration and Monitoring Commands:

To add a Node to Cluster:

From the node to be added, which has access:

host # clnode add -c clustername -n nodename -e endpoint1, endpoint2 - Use this only if the node has access to the cluster

To remove a Node From the Cluster:

From the node to be removed, which is in noncluster
mode and has access:

host # clnode remove - Use this only if the node has access to the cluster

host # clnode evacuate nodename - Switch All Resource Groups and Device Groups Off of a Node
host # clinterconnect disable nodename:endpoint - Manage the Interconnect Interfaces
host # clinterconnect enable nodename:endpoint - To disable a cable so that maintenance can be performed, then enable the same cable afterward.
host # cluster status - Display the Status of All Cluster Components
host # command status - Display the Status of One Type of Cluster Component
host # cluster show - Display the Complete Cluster Configuration
host # command show Component - Display the Configuration of One Type of Cluster
host # command list - List One Type of Cluster Component
host # clnode show-rev -v - Display Sun Cluster Release and Version Information

To list the software versionson the current node.

host # clnode show | grep nodename - Map Node ID to Node Name
host # cltelemetryattribute enable -t disk rbyte.rate wbyte.rate read.rate write.rate - Enable Disk Attribute Monitoring on All Cluster Disks
host # cltelemetryattribute disable -t disk rbyte.rate wbyte.rate read.rate write.rate - Disable Disk Attribute Monitoring on All Cluster Disks

SHUTTING DOWN AND BOOTING A CLUSTER

To shut Down the Entire Cluster:

host # cluster shutdown

From one node:

host # clnode evacuate
host # shutdown - Shut Down a Single Node

To boot a Single Node from the OBP:

ok> boot

To reboot a Node Into Noncluster Mode from the OBP:

ok> boot -x

, Mike

LVM Quick Command Reference For Linux And Unix

Hello again,

Today, as promised in yesterday's post (where we came full circle from getting started with LVM, and using monitoring and display commands to removing those same LVM devices), today we've got a little quick command reference of the most commonly used LVM commands.

You may have noticed that I named Linux and Unix in the title of this post. That's because HP-UX's implementation of LVM is very similar to Linux's, and this reference basically works for both of them :) If you're interested in specifics on most of these commands, please refer to our previous posts. We'll note commands in this reference that we "haven't" covered yet.

And, off we go. Hopefully this will help out if you ever need a one-stop-shop for that LVM command you've been looking for. In future posts, we'll definitely be looking at more advanced LVM concepts. For instance, we haven't even begun to look at things like setting up RAID or any form of advanced manipulation of the physical volumes, logical volumes and volume groups we created. You have our promise that posts on theses subjects will be forthcoming. Just maybe not tomorrow ;)

1. LVM Basic relationships. A quick run-down on how the different parts are related

Physical volume - This consists of one, or many, partitions (or physical extent groups) on a physical drive.
Volume group - This is composed of one or more physical volumes and contains one or more logical volumes.
Logical volume - This is contained within a volume group.

2. LVM creation commands (These commands are used to initialize, or create, new logical objects) - Note that we have yet to explore these fully, as they can be used to do much more than we've demonstrated so far in our simple setup.

pvcreate - Used to create physical volumes.
vgcreate - Used to create volume groups.
lvcreate - Used to create logical volumes.

3. LVM monitoring and display commands (These commands are used to discover, and display the properties of, existing logical objects). Note that some of these commands include cross-referenced information. For instance, pvdisplay includes information about volume groups associated with the physical volume.

pvscan - Used to scan the OS for physical volumes.
vgscan - Used to scan the OS for volume groups.
lvscan - Used to scan the OS for logical volumes.
pvdisplay - Used to display information about physical volumes.
vgdisplay - Used to display information about volume groups.
lvdisplay - Used to display information about logical volumes.

4. LVM destruction or removal commands (These commands are used to ensure that logical objects are not allocable anymore and/or remove them entirely) Note, again, that we haven't fully explored the possibilities with these commands either. The "change" commands in particular are good for a lot more than just prepping a logical object for destruction.

pvchange - Used to change the status of a physical volume.
vgchange - Used to change the status of a volume group.
lvchange - Used to change the status of a logical volume.
pvremove - Used to wipe the disk label of a physical drive so that LVM does not recognize it as a physical volume.
vgremove - Used to remove a volume group.
lvremove - Used to remove a logical volume.

5. Manipulation commands (These commands allow you to play around with your existing logical objects. We haven't posted on "any" of these commands yet - Some of them can be extremely dangerous to goof with for no reason)

pvextend - Used to add physical devices (or partition(s) of same) to a physical volume.
pvreduce - Used to remove physical devices (or partition(s) of same) from a physical volume.
vgextend - Used to add new physical disk (or partition(s) of same) to a volume group.
vgreduce - Used to remove physical disk (or partition(s) of same) from a volume group.
lvextend - Used to increase the size of a logical volume.
lvreduce - Used to decrease the size of a logical volume.

You may note that some of these commands vary slightly from LVM1 to LVM2 and also between the pure Linux implementation and HP-UX's Unix version. We'll be looking at the differences, and similarities, between all of these in future posts, for sure.

Until then. Enjoy :)

, Mike

Removing Logical Devices Using LVM On Linux And Unix

Hey there,

A long weekend of getting confused over factorials and prime numbers is finally over, so today, we're going to finish one part of what we started in our previous posts regarding getting started with LVM logical volume management and making use of LVM's monitoring and display commands, and close this little circle by using those same monitoring commands to aid in disk removal with LVM.

Refer back to those two initial posts for any additional details, if you'd like them, but, for today's purposes, we're just going to use their outcome as the assumed setup which we'll be removing.

In a nutshell, we have one physical volume composed of one volume group which, in turn, contains only one logical volume (or you could think of it the other way around). Hopefully this is all coming back to us (including me ;)

Note: Based on your particular vendor distribution , or version, of LVM, the output of these commands may be slightly different. You shouldn't notice any significant difference, though.

The first thing we'll want to do, to undo all the hard work we've put into creating this little setup, is to remove the logical volume (lvol01) from the volume group (vg01). This is done easily with the lvchange command, like so:

host # lvchange -a n /dev/vg01/lvol01

lvchange doesn't spit any output to the terminal by default, if everything goes well. But, if we double-check that the return code from the above command is 0, then we've successfully made the lvol01 logical volume unavailable (-a for availability status with n following it to indicate that we want that status set to no).

Next, we'll remove the logical volume, allowing the naturally occuring backup of the volume group, from vg01:

host # lvremove -f /dev/vg01/lvol01
...
lvremove -- doing automatic backup of volume group "vg01"
lvremove -- logical volume "/dev/vg01/lvol01" successfully removed

The next two things to do will be to change the volume group's status and remove the volume group using (yes, it's true ;) vgchange and vgremove, like this:

host # vgchange -a n /dev/vg01
vgchange -- volume group "vg01" successfully deactivated

host # vgremove /dev/vg01
vgremove -- volume group "vg01" successfully removed

The most assured way to finish this (unless your flavour of Linux doesn't allow it) would be to run pvchange and then pvremove to finish off the process.

host # pvchange -x n /dev/hdd1 <-- This will tell LVM not to allow allocation of any physical extents from this physical volume (See our original post on getting started with LVM for a slightly more in-depth explanation of these).

host # pvremove -f /dev/hdd1 <-- This will wipe the label on the physical drive so that LVM will no longer recognize it as a physical volume.

And you're all set! Your monitoring and display commands (pvscan, pvdisplay, vgscan, vgdisplay, lvscan and lvdisplay) will now show you disheartening results (that's assuming you didn't want to do this, which goes contrary to the intent of the post ;)

Tomorrow, we'll put a together a quick command reference for LVM, including (of course) a few extra commands so it's not "all" fluff ;)

Cheers,

Mike

Monitoring and Display Commands For LVM On Linux And Unix

Hey there,

Today, we're going to follow up on our previous post regarding getting started with LVM logical volume management and go over a few (perhaps) boring but essential commands you'll want to know in order to keep tabs on your setup both in good times and in bad.

Refer back to our initial post on getting started with LVM for any additional details, if you'd like them, but, for today's purposes, we're just going to use the outcome of that post as the simple setup that we'll be monitoring/displaying today. In a nutshell, we have one physical volume composed of one volume group which, in turn, contains only one logical volume (or you could think of it the other way around). Unless I'm completely off-center here today, our output from today should be extremely reminiscent of yesterday's ;)

Note: Based on your particular version, or vendor distribution of LVM, the output of these commands may be slightly different. Not so much so that it should make a significant difference.

Basically, our monitoring and displaying commands are broken down into three groups: Those dealing with "physical volumes," those dealing with "logical volumes" and those dealing with "volume groups." We'll step through them (not over and/or around them, even though that would be more polite ;) one by one (in twos):

Physical Volumes:

The two commands we'll be using here are pvscan and pvdisplay.

pvscan, as with all of the following commands, pretty much does what the name implies. It scans your system for LVM physical volumes. When used straight-up, it will list out all the physical volumes it can find on the system, including those "not" associated with volume groups (output truncated to save on space):

host # pvscan
pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- ACTIVE  PV "/dev/hda1"   is in no VG [512 MB]
...
pvscan -- ACTIVE  PV "/dev/hdd1" of VG "vg01"[512 MB / 266 MB free]
...

Next, we'll use pvdisplay to display our only physical volume:

host # pvdisplay /dev/hdd1 <-- Note that you can leave the /dev/hdd1, or any specification, off of the command line if you want to display all of your physical volumes. We just happen to know we only have one and are being particular ;)

...
PV Name   /dev/hdd1
VG Name   vg01
PV Size   512 MB
...

Other output should include whether or not the physical volume is allocatable (or "can be used" ;), total physical extents (see our post on getting started with LVM for a little more information on PE's), free physical extents, allocated physical extents and the physical volume's UUID (Identifier).

Volume Groups:

The two commands we'll be using here are vgscan and vgdisplay.

vgscan will report on all existing volume groups, as well as create a file (generally) called /etc/lvmtab (Some versions will create an /etc/lvmtab.d directory as well):

host # vgscan
vgscan -- reading all physical volumes (this may take a while...)
vgscan -- found active volume group "vg01"
...

vgdisplay can be used to check on the state and condition of our volume group(s). Again, we're specifying our volume group on the command line, but this is not necessary:

host # vgdisplay vg01
...
VG Name               vg01
...
VG Size               246 MB
...

this command gives even more effusive output. Everything from the maximum logical volumes the volume group can contain (including how many it currently does and how many of those are open), separate (yet similar) information with regards to the physical volumes it can encompass, all of the information you've come to expect about the physical extents and, of course, each volume's UUID.

Logical volumes:

The two commands we'll be using here are (you guessed it ;) lvscan and lvdisplay.

lvscan will give us a report on all of our logical volumes, and their state (ACTIVE/INACTIVE or possibly in ERROR) on our system, like so:

host # lvscan
...
lvscan -- ACTIVE            "/dev/vg01/lvol01" [246 MB]
lvscan -- 1 logical volumes with 246 MB total in 1 volume group
lvscan -- 1 active logical volume

Once this step has been completed (and it should be completed at least once!), we can use lvdisplay to check out our logical volume(s). We'll be specifically looking at lvol01:

host # lvdisplay lvol01
...
LV Name                /dev/vg01/lvol01
VG Name                vg01
...

and we'll get our standard UUID information, read/write access, availability status and information on the logical volume size (among other useful information, hopefully including the block device the logical volume is associated with :)

It should be noted that the logical volume size is one of the areas we generally look at first. If, for some reason, the output from "df" seems less than what you intended, this will give you a confident indication of whether or not your logical volume is too small, or you just need to grow the filesystem, resident on it, to use all of the logical volume's space!

...And that about wraps up the "information gathering" section of this series of posts on LVM. In future posts we'll be looking at ways to manipulate volumes (both in groups and individually in their physical and logical aspects; although the physical volume is actually a logical representation of a logical representation of a physical device... but I think we went into too much detail about that gordian knot in our previous post, and, even then, haven't begun to scratch the surface ;). In future posts we'll also be looking at LVM on Unix and Linux and the relationship between the two, which is becoming less complex every day!

Cheers,

, Mike

How To Get Started With Logical Volume Management In Linux

Hey again,

Today, since past posts regarding disk management software have mostly been limited to Veritas Volume Manager, I thought we'd look at LVM on Linux (In a future post, we'll check out the differences between LVM 1 and 2, but for now I figured we could keep it simple and broad).

If you use any of the "major player" Linux flavours out there, you've probably noticed (over the last year or so) that standard installs show all your disks listed out in df, like:

/dev/mapper/vg01-lvol01 246M   75.2M    158M 32%    /dir

rather than the old-style:

/dev/hdd1      246M   75.2M    158M    32%    /dir

that you've (if you're anything like me and only have two hard drives on the home PC you're running Linux on ;) grown to love.

And, while I think it's fantastic that Linux has grown into a mostly-user-friendly operating system with GUI's to make all of the extended partitioning and volume management more accessible, I also think there's something to be said for knowing how to be able to set all that stuff up on your own. You never know when your main display (or X-windows session) will die on you and, at some point, you may end up having to do some disk repair on the lonely old command line.

With that in mind, we'll step through the most basic LVM setup: Getting from /dev/hdd to /dev/mapper/vg01-lvol01 from the command prompt.

Our basic (and huge assumption) here is that we've got a primary drive to work from (although we could do this just as easily from a rescue, or live, CD-ROM if we needed to perform this work on our primary disk during the Linux installation process or a freakish-nightmare of a fix)

First, we'll want to use fdisk to create a "new" "primary" partition that consists of the entire disk (512Mb in size). Barring a primer on fdisk, we'll want to set up our disk to have one primary partition that consists of all cylinders (first and last should be the default start and end values) and is set to the "Linux LVM" partition type.

Once we're done with the stuff we'd "normally" do (for the most part) to set up any other type of partition table on our /dev/hdd disk, we can begin to play around with the LVM commands. Note that if you chose any partition type other than LVM when setting up your partition, using the following commands will, at best, result in error messages and accomplish nothing :)

If you've ever used HP-UX before, a lot of these commands will seem familiar. There's a reason, but the legalities of discussing it (and possibly getting bits of it wrong) fall outside the scope of my pocketbook ;)

The first thing we'll do is create a "physical volume," like so:

host # pvcreate /dev/hdd1

Yes, it was a physical volume already. The logical-physical relationship sometimes doesn't hold up to scrutiny outside of the "rules" ;)

Next, well create a "volume group." This is necessary, because "logical volumes" run in packs... pardon my lack of propriety ;) We could use the "-s" option here to define the "physical extent" size, but we'll leave that alone. To say the least, specifying this requires some research and planning. If you get that number wrong, things can go South later, and you'll have to do some ugly backtracking. We just want to define a "volume group" so we have something to work with. We'll call it vg01, for lack of a more original name.

host # vgcreate vg01 /dev/hdd1

Next, we'll create a "logical volume" (we'll name this one "lvol01" to keep things consistent) of 246 Megabytes size. That should fit nicely inside vg01 (which we're basically using to represent the entire partition /dev/hdd1). We could make our "logical volume" as large as the entire partition and/or "volume group", but that would defeat the purpose of this whole exercise. The aim, of course, is to end up with flexible volumes so we can resize them later if we need to.

host # lvcreate -L 246M -n lvol01 vg01

This is where the /dev/mapper stuff comes into play (it never made sense to me the first time I saw it... really. It's because of my past experience with HP-UX ;) The "lvcreate" command creates a symlink between /dev/vg01/lvol01 and the character device /dev/mapper/vg01-lvol01.

And, next, all we have to do is format that bad boy:

host # mkfs -t ext3 /dev/vg01/lvol01

Note that, above, you can set a reserved block count (based on percentage) using the -m flag. I would usually set this to "-m 5" so that 5% of the allocated space would be reserved for root (although you can set it as low as 1). This way, if someone jams your filesystem up to 100% and no one can get any work done, you can still log in as root and fix the issue. -v is another useful flag if you want to see verbose output. You can insert these two flags anywhere in the above command line except at the very end (after the logical device name) or in between the -t flag and its argument. Common sense, I know, but I feel compelled to over-explain ;)

Now, we'll make a mount point, or, really, just make a new directory off of the root filesystem (although you can do overlays if you wanna get creative ;)

host # mkdir /dir

and then we'll mount our logical volume up and check it out with df:

host # mount -t ext3 /dev/vg01/lvol01 /dir

host # df -h /dir
...
/dev/mapper/vg0-lvol0 246M   75.2M    158M 32%    /dir

Tomorrow, or maybe the day after that, we'll take a look at some more ways to manipulate your existing volumes, add more volumes, do mirroring and lots of other fun stuff that LVM allows you to do.

For now, we'll just thank our deity-of-choice that this all worked out ;)

Cheers,

, Mike