Hey again,
Today, we're back with part two of what was going to be a three part series on working with ZFS and storage pools. Actually, this was originally going to be one post, but (luckily ?) it's grown into four gigantic ones ;) This one, and tomorrow's, are going to be the "big daddies" of the bunch.
Yesterday we looked at ZFS storage pool and file system creation and today we're going to move on to commands that we can use to manipulate those storage pools and file systems that we've made.
Please note, again, that for all commands where I specifically name a virtual device or storage pool, you can get a full listing of all available devices by simply not specifying any storage pool at all.
And, without a moment to spare, here come those maintenance/usage commands (please enjoy responsibly ;)
1. If you need to know as much as possible about your storage pools, you can use this command:host # zpool status -v zvdevs
pool: zvdevs
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
zvdevs ONLINE 0 0 0
/vdev1 ONLINE 0 0 0
/vdev2 ONLINE 0 0 0
/vdev3 ONLINE 0 0 0
errors: No known data errors
2. In Solaris 10 storage pool management, you can also "online" and "offline" virtual devices. You might need to do this from time to time if you need to replace an "actual" device that may be faulty. Here's an example of offlining and then onlining a vdev. Note that, if you use the "-t" flag when offlining, the device will only be temporarily disabled. Normal offlining is persistent and the storage pool will maintain the vdev in an offline state even after a reboot:host # zpool offline zvdevs /vdev2
Bringing device /vdev2 offline
host # zpool status -v zvdevs
pool: zvdevs
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scrub: resilver completed with 0 errors on Mon May 5 10:05:33 2008
config:
NAME STATE READ WRITE CKSUM
zvdevs DEGRADED 0 0 0
mirror DEGRADED 0 0 0
/vdev1 ONLINE 0 0 0
/vdev2 OFFLINE 0 0 0
errors: No known data errors
host # zpool online zvdevs /vdev2
Bringing device /vdev2 online
host # zpool status -v zvdevs
pool: zvdevs
state: ONLINE
scrub: resilver completed with 0 errors on Mon May 5 10:18:34 2008
config:
NAME STATE READ WRITE CKSUM
zvdevs ONLINE 0 0 0
mirror ONLINE 0 0 0
/vdev1 ONLINE 0 0 0
/vdev2 ONLINE 0 0 0
errors: No known data errors
3. If you want to attach another disk to your storage pool mirror, it's just as simple. This process will create a simple mirror if you only have one device in your pool (???) or create a triplicate, quadruplicate, etc, mirror if you already have a simple mirror set up:
host # zpool attach zvdevs /vdev1 /vdev3 <--- Note that we're specifically saying we want to mirror /vdev1 and not /vdev2. It doesn't really matter, since they're both mirrors of each other, but you can't just attach a device without naming a device to mirror!host # zpool status -v zvdevs
pool: zvdevs
state: ONLINE
scrub: resilver completed with 0 errors on Mon May 5 10:05:33 2008
config:
NAME STATE READ WRITE CKSUM
zvdevs ONLINE 0 0 0
mirror ONLINE 0 0 0
/vdev1 ONLINE 0 0 0
/vdev2 ONLINE 0 0 0
/vdev3 ONLINE 0 0 0
errors: No known data errors
5. For a little shell-game, if you ever need to replace a vdev in your storage pool (say, with a hot spare), you can do it easily, like this:
host # zpool add zvdevs spare /vdev3 <--- This may not be necessary, but I removed the spare and am re-adding it.host # zpool replace zvdevs /vdev1 /vdev3
host # zpool status -v zvdevs
pool: zvdevs
state: ONLINE
scrub: resilver completed with 0 errors on Mon May 5 10:20:58 2008
config:
NAME STATE READ WRITE CKSUM
zvdevs ONLINE 0 0 0
mirror ONLINE 0 0 0
spare ONLINE 0 0 0
/vdev1 ONLINE 0 0 0
/vdev3 ONLINE 0 0 0
/vdev2 ONLINE 0 0 0
spares
/vdev3 INUSE currently in use
errors: No known data errors
6. If you suspect file system damage, you can "scrub" your storage pool. zpool will verify that everything is okay (if it is ;) and will auto-repair any problems on mirror or raid pools :)
host # zpool scrub zvdevs <--- Depending on how much disk you have and how full it is, this can take a while and chew up I/O cycles like nuts.
7. If you want to share storage pools between systems (real or virtual), you can "export" your storage pool like so (Note that this will make the storage pool not show up as being "owned" by your system anymore, although you can reimport it):host # zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
datadg 33.8G 191M 33.6G 0% ONLINE -
rootdg 9.75G 5.11G 4.64G 52% ONLINE -
zvdevs 1016M 127K 1016M 0% ONLINE -
host # zpool export zvdevs
host # zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
datadg 33.8G 191M 33.6G 0% ONLINE -
rootdg 9.75G 5.11G 4.64G 52% ONLINE -
8. In order to "import" an exported filesystem, you just need to have the system permission to do so (being root in the global zone is the perfect place to be when you try this. Just be careful ;)
host # zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
datadg 33.8G 191M 33.6G 0% ONLINE -
rootdg 9.75G 5.11G 4.64G 52% ONLINE -
host # zpool import -d / zvdevs
host # zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
datadg 33.8G 191M 33.6G 0% ONLINE -
rootdg 9.75G 5.11G 4.64G 52% ONLINE -
zvdevs 1016M 92K 1016M 0% ONLINE -
You can run this command without -d (to specify the base directory of the storage pool) and zpool will search /dev/dsk for a place to import. In our case it won't find it, like this:
host # zpool import zvdevs
cannot import 'zvdevs': no such pool available
9. If you need to know what version of ZFS your system is using, you can use zpool's "upgrade" option. Don't worry. Without the proper flags, it just lets you know what version is running and some other information, like this:
host # zpool upgrade
This system is currently running ZFS version 4.
All pools are formatted using this version.
10. If you want to see what features your version of ZFS has, you can use the "upgrade" option with the -v flag. Everything is retro, so (in our case), since we're running version 4, we have all the capabilities of versions 3, 2 and 1 but not of versions 5 and higher (At the time of this post, ZFS is up to version 10):host # zpool upgrade -v
This system is currently running ZFS version 4.
The following versions are supported:
VER DESCRIPTION
--- --------------------------------------------------------
1 Initial ZFS version
2 Ditto blocks (replicated metadata)
3 Hot spares and double parity RAID-Z
4 zpool history
...
Now, go and have fun with those commands :)
Until we meet again,
, Mike
linux unix internet technology
Wednesday, May 7, 2008
Maintenance And Usage Commands For ZFS On Solaris 10 Unix
Tuesday, February 26, 2008
Offlining, Failing Over And Switching in VCS
Hey there,
Today we're going to address a question that's asked commonly enough by foks who use Veritas Cluster Server: What's the difference between offlining, failing over and switching when dealing with service groups across multiple nodes? That doesn't seem like a very common question. I'm sure it's probably hardly ever phrased that way ;)
Anyway, to the point, all three options above are useful (hence your ability to avail yourself of them) and sufficiently distinct that you should be sure you're using the one you want to, depending on what ends you wish to achieve. All of this information is fairly general and should work on VCS for Linux as well as Unix.
We'll deal with them one bye one, outlining what they basically do, with a short example command line to demonstrate, where applicable. None of this stuff is hard to pick up and run with, as long as all the components of your VCS cluster are setup correctly. Occasionally, you may see errors if things aren't exactly perfect, like we noted in this post on recovering indefinitely hung VCS resources.
1. Offlining. The distinction to be made here, most plainly, is that, when you offline a resource or service group, you are "only" doing that. This differs from failover and switching in that the service group you offline with this particular option is not brought online anywhere else as a result. So, when you execute, for instance:
host # hagrp -offline YOUR_SERVICE_GROUP -sys host1
that service group, and its resources, generally, are taken offline on host1, and nothing else happens. If you're operating in an environment where systems don't run service groups concurrently (active/active), you will have effectively "shut down" that service group, and any services it provides, for the entire cluster.
2. Failing Over. This is more of a concept than any particular command. When you have your cluster setup to fail over, if a resource or service group, etc, goes offline on one node (host1, for instance) and it wasn't brought offline on purpose, VCS will naturally attempt to bring it online on the next available node (listed in your main.cf configuration file). Needless to say, if only one host is listed for a particular service group, its failure on one host will mean the failure of the entire service group. It also obviates the use of VCS in the first place ;)
3. Switching. This is what most folks want to do when they "offline" a service group, as in point 1. Although, since VCS automatically switches unexpectedly offlined resources on its own (when it's set up to), it's reasonable for someone new to the product to assume that offlining a service group would engage VCS in a switching activity. Unfortunately, this isn't the case. If you want to switch a service group from host1 to host2, for example, these would be the command options you'd want to give to hagrp:
host # hagrp -switch YOUR_SERVICE_GROUP -to host2 <--- Assuming you're running this from host1.
Hopefully this little guided mini-FAQ helped out with differentiating between the concepts. If you find the command line examples valuable, even better :)
Happy failing! ;) Over.
, Mike
linux unix internet technology
Monday, November 19, 2007
Recovering Faulted VCS Resources In An Indefinitely Hung State!
If you've ever found yourself in a situation where a Veritas Cluster Server (VCS) resource/service group is reporting faulted, you may have seen the situation described here. Today we're going to focus on one specific condition where the most common wisdom I've seen on the web is to do an "hastop -local -force" and follow up with an "hastart" to solve the problem. Common sense dictates that this is the last thing you'll want to do, unless you're on a testing environment, as you'll be completely downing the cluster in the process.
The reason the stop/start answer is the most common is because, since the resource/service group's "resource" is waiting to go OFFLINE, while the resource/service "group" is trying to start, or go ONLINE, simply attempting to "online" or "offline" either only results in infinite hang. This is because each entity's successful action depends on the other entity's failure and VCS won't fail if it can wait instead.
You'll know you're looking at this sort of error if you see the following (this generally can be found by running "hasum" or "hastatus -summary"):
A. The resource/service "group" will be showing in a "STARTING|PARTIAL" state
B. An individual "resource" within that resource/service "group" will be showing in the "W_OFFLINE" state.
The following steps to resolution are certainly not the only way to get everything back to an "up" state and, also, assume that there is nothing really "wrong" with the individual resource or resource/service group. That sort of troubleshooting is outside the scope of this blog-sized how-to.
So, again, our assumptions here are:
1. An Oracle database resource/service group, named "oracledb_sg" has faulted on the server "vcsclusterserver1.xyz.com."
2. An individual resource, a member of the resource/service group "oracledb_sg," named "oracledb_oracle_u11192007," is really the only thing that's failed, or shows as failing in "oracledb_sg."
3. There is actually nothing wrong with the resource or the resource/service group. Somehow, while flipping service groups back and forth between machines, somebody made an error in execution or VCS ran into a state problem that it caused by itself ("split brain" or some similar condition).
4. Note that we've reached these assumptions based partially on the fact that the resource is waiting to go OFFLINE, and the resource/service group is waiting to go ONLINE (stuck in STARTING|PARTIAL), on the same server!
And the following are the steps we could take to resolve this issue and get on with our lives:
1. First, get a summary of the resource group that you've been alerted as failed or faulted, like this:
hastatus -summary|grep oracledb_sg|grep vcsclusterserver1 (or some variation thereof, depending on how much information you want to get back)
B oracledb_sg vcsclusterserver1.xyz.com Y N STARTING|PARTIAL
C oracledb_sg Oracle oracledb_oracle_u11192007 vcsclusterserver1.xyz.com W_OFFLINE
2. If you want to be doubly sure, you'll check that you get the same resource id ( oracledb_oracle_u11192007 ) from the "hares" command, and also get the same status:
hares -display|grep oracledb|grep "waiting to" --> to get the resource id a second way.
3. At this point you still can't "online" the resource until you "clear" it (of the W_OFFLINE state flag).
hares -clear oracledb_oracle_u11192007 -sys vcsclusterserver1.xyz.com
hagrp -online oracledb_sg -sys vcsclusterserver1.xyz.com
Now your resource/service group should be back in the straight ONLINE state, and you shouldn't see any messages (In "hasum" or "hastatus -summary" output) regarding the individual resource. Time to relax :)
, Mike
linux unix internet technology
