Dealing With ZFS-Rooted Zones on Solaris 10 Unix

Hey there,

Today, we're going to take a look at a problem that's been haunting Solaris 10 (and, to a degree, Open Solaris) for almost 3 years now. This ties back pretty closely to earlier posts we put out on migrating zones, patching local and global zones and working with zfs filesystems, since it has exactly to do with a problem concerning zfs, Solaris 10 zones and one specific way in which they can be created.

Theoretically, it would seem, that the one way that's causing the most problems is the one way that should be the most desirable way to configure your setup (???)

Here's a little something to think about when considering creating zones with a zfs filesystem. Although this quote is taken out of context, directly from Sun, it was actually put out there as a selling point (in its own context actually):

zones are integrated into the operating system, providing seamless functionality and a smooth upgrade path.

However, as many of you may be aware by now (it being July 2nd, 2008 and, officially, version 5/08 of Solaris 10 is on the market), although creating bootable zfs zones and zones with zfs root filesystems is now finally possible (I believe it was originally introduced, in a small way, back in 6/05 -right before the 06/06 official release), it still suffers from some severe issues, that may not be initially evident. That is to say, if you didn't do your homework before you took advantage of this seemingly great feature, you've probably gotten burned in one fashion
or another, with regards to the upgrade process. Per Sun, again:

Solaris 10 6/06 supports the use of ZFS file systems. It is possible to install a zone into a ZFS fs, but the installer/upgrader program does not yet understand ZFS well enough to upgrade zones that "live" on a ZFS file system.

Because of this (and repeating this ;) upgrading a system that has a zone installed on a ZFS file system is not yet supported. To this day (to my knowledge) the problem still hasn't been completely resolved. Again, from Sun's bug list (And, I know I sound like I'm Sun-bashing here, but I am coming to a positive point. I swear :)

zoneadm attach command Might Fail (6550154)
when you attach a zone, if the original host and the new host have packages at the same patch level but at different intermediate patch histories, the zone attach might fail. various error messages are displayed. The error message depends on the patch histories of the two hosts.
workaround: Ensure that the original host and the new host machines have had the same sequence of patch versions applied for each patch.

Basically, the way things stand now, if you have a zone built on a zfs root filesystem (rather than, say ufs), if you need to upgrade, you officially have 3 options:

l. Be pro-active and "Don't do it!"

2. Go ahead and do it, but be sure to uninstall your zones before upgrading to a new release of Solaris 10, and then reinstall them when your upgrade to the new release is completed.

3. Go ahead and do it, but instead of following the more traditional upgrade-path, completely reinstall the system in order to perform the upgrade. This option makes the least sense, since reinstallation and upgrading aren't synonymous.

Now, for the rainbow after the storm. Yes, rainbows are somewhat illusory and their beauty isn't necessarily the matched-opposite of the horrors of nature you may have had to endure in order for it to bring you to that phenomenon, but it's a lot better than nothing, right ;)

The situation, as you may have guessed, is still pretty much up-in-the air, but there is hope; and in more than one area. For x86 (and possibly Sparc), this initiative is being fast-tracked by Sun for Open Solaris/Solaris 10 (Note that it's dated June 27th, 2008 :) - It basically proposes a -b flag to zoneadm attach, to be used in conjunction with the -u flag, to allow for backing patches out of a zone before an OS update. The full discussion, to date, is located here on openSolaris.org.

Why is this important?

As we noted above, the biggest problem Solaris 10 has with upgrading the OS on machines that have zones with zfs roots is that every single patch and package must be the same after the upgrade in order for it to be considered successful and Solaris 10's update software doesn't work with ZFS well enough to be able to guarantee that patches that get installed in one zone will necessarily get installed in another (global vs. local, zfs vs. ufs/vxfs, even zfs vs. zfs). If we were allowed to ignore certain patches and/or packages in our upgrades, this might make the likelihood of failure drop dramatically!

And here's one more ray of hope (which might be even better by the time you need to apply it). Here's how to upgrade your OS, assuming it has zones mounted on zfs roots, and (possibly) get away with not having to go to the extremes Sun is obligated to recommend. It's actually fairly simple. Mostly because it isn't guaranteed to work ;) The one good thing is that, if it doesn't work, you'll have your data saved off, so (if this procedure fails) you can still do it the hard way and not lose anything, except time, by trying :)

Do the following. We'll assume you've read our previous posts on migrating and patching both local and global zones and understand the basic system-down commands that would necessarily precede the following:

l. Halt and detach each of your zones that sits on a zfs root (I'd personally do this for all of my zones), like so:

host # zoneadm -z ZONENAME1 halt
host # zoneadm -z ZONENAME1 detach

2. Export all of your zfs pools:

host # zpool export ZONENAME1 <-Make sure that you note the names here, for the reverse process, just in case!
host # zpool export ZONENAME2

3. Perform your upgrade however you prefer.

4. Import all of your zfs pools

host # zpool import <--If this doesn't work, use the names you specified during the export previously.

or

host # zpool import ZONENAME1
host # zpool import ZONENAME2

5. Reattach and boot/install your zones using the -F flag to force the issue (of course you can leave it out if you want, just to see what happens. Sometimes forcing makes things work that are flagged as errors, but aren't really. You can also use the -n flag to do a dry-run):

host# zoneadm -z ZONENAME1 attach -F
host# zoneadm -z ZONENAME1 boot -F <--Note that for the zoneadm command. if you don't list a zone name with the -z flag, the subcommand (halt, detach, attach, boot, etc) would apply to all zones!

And you should either be all set or have a more limited set of issues to deal with (probably mostly patch related).

Later on in the week (if we don't run out of screen space ;) we'll look at ways to troubleshoot a Solaris 10 zfs-root zone upgrade gone bad.

Until that bright and sunny day:)

, Mike

Creating Storage Pools For Solaris ZFS

Hey there,

As a way of getting on to addressing Solaris 10 Unix issues beyond patching zones and migrating zones, today we're going to put together a slam-bang setup of a Solaris ZFS storage pool. For those of you unfamiliar with ZFS (and I only mention this because I still barely ever use it -- Lots of folks are stuck on either Solaris 8 or 9 and are having a hard time letting go ;), it simply stands for "Zettabyte File System."

A Zettabyte = 1024 Exabytes.
An Exabyte = 1024 Petabytes.
A Petabyte = 1024 Terabytes.
A Terabyte = 1024 Gigabytes.
A Gigabyte = 1024 Megabytes.

And, at about this point the number scale makes sense to most of us. And, even though a Zettabyte seems like it's an unbelievably large amount of space, the next generation of operating systems will require twice as much RAM and disk space in order to respond to your key presses in under a minute ;)

Anyway, now that we have that out of the way, let's set up a ZFS pool.

First we'll create the pool, in a pretty straightforward fashion, using the zpool command and adding the disks (which can be in either cXtXdX notation, or specified by slice with cXtXdXsX notation. You can also use full logical path notation if you want (e.g. /dev/disk/cXtXdXsX). We'll assume two entire disks (unmirrored), create them and mount them for use:

host # zpool create MyPoolName c0t0d0 c0t1d0

Note that you can run "zpool create -n" to do a dry-run. This will allow you to find any mistakes you may be making in your command line as zpool doesn't "really" create the pool when you run it with the "-n" flag.

We also won't have to do anything extra to create the mount point, as it defaults to the name of the disk group. So, in this case, we would have a storage pool with a default mount point of /MyPoolName. This directory has to either not exist (in which case it will be automatically created) or exist and be empty (in which case the root dataset will be mounted over the existing directory). If you want to specify a different mount point, you can use the "-m" flag for zpool, like so:

host # zpool create -m /pools/MyPool MyPoolName c0t0d0 c0t1d0

And your ZFS storage pool is created! You can destroy it just as easily by running:

host # zpool destroy MyPoolName

or, if the pool is in a faulted state (or you experience any other error, but know you want to get rid of it), you can always force the issue ;)

host # zpool destroy -f MyPoolName

And you're just about ready to use your new "device." All you need to do is ( just like you would when mounting a regular disk device ) create a filesytem on your storage pool. You can do this pretty quickly with the zfs command, like so:

host # zfs create /MyPoolName/MyFileSystem

Note that the pool name directory is the base of the storage pool and your file system is actually created at the next level up. Basically, /MyPoolName is not your file system, but /MyPoolName/MyFileSystem is. And you can manage this file system, in much the same way you manage a regular ufs file system, using the zfs command.

Again, you can always opt-out and destroy it if you don't like it (here, as well, you have the option to "force" if you want):

host # zfs destroy /MyPoolName/MyFileSystem

Aside from "-f" (to force execution), the most common flags to use with "zfs destroy" are "-r" to do a recursive destroy or "-R" to recursively destroy all the regular descendants in the file system tree, plus all cloned file systems outside of your target directory (Be careful with that switch!)

We've only just brushed on getting started with zfs, but, hopefully, it's been somewhat helpful and, at the very least, readable ;)

Enjoy,

, Mike

linux unix internet technology

Patching Solaris 10 Zones - Global And Local Issues

Hello Again,

In a long overdue continuation of our coverage of Solaris 10 (as in our previous post on migrating between local and global zones) today, we're going to take a look at patching on Solaris 10, insofar as it relates to zones. There are a lot of questions going around about how to patch appropriately and what's permissible. This is completely understandable since, if you do it wrong, the consequences can be disastrous and irreversible (and none of us wants to stay up all night... working ;)

We'll start off with the concept of patching and zones. As far as patching is concerned, each zone actually has it's own patch (and package) database. This makes it possible for you to, theoretically, patch single zones on a host individually or patch all of them at once by patching the "global" zone.

There are a few things to keep in mind though, before you go ahead and apply your patches, either way...

1. Even if you have "umpteen" number of zones on a single machine, they all run off of the same kernel, so if you have to patch your kernel (or anything kernel-related) you need to do that from the "global" zone. If you apply a kernel patch on a local zone, and bring its kernel patch revision to a different level than the "global" zone's (and other zones') calamity will ensue (Let's see how many different ways I can write that something bad is going to happen ;)

2. "patchadd" will eventually drive you nuts, anyway, so the following proviso's should be no surprise ;).

a. If you run patchadd with -G in a "global" zone, any packages that have SUNW_PKG_ALLZONES set to true will cause the entire patch operation to fail.

b. If you run patchadd with -G in a "global" zone and "no" packages have SUNW_PKG_ALLZONES set to true, you should be able to install the patch to all zones from the "global" zone.

c. If you run patchadd without -G in a "global" zone, regardless of the setting of SUNW_PKG_ALLZONES, you can install the appropriate patches to any individual zones or the global zone (which will patch all the zones, by default).

d. If you run patchadd in a "local" zone, with or without -G specified, any packages that have SUNW_PKG_ALLZONES set to true will fail and not install, and if none have SUNW_PKG_ALLZONES set to true, everything should work in each "local" zone you apply the patch to.

3. Any software that can be installed at the "local" zone level, can also be patched independently at the "local" zone level, on whatever zones it was installed. This is true regardless of your zone type (whole root or sparse root).

4. The "-G" option to patchadd doesn't stand for "global zone," rather it stands for "the current zone." I can't think of a good mnemonic for this so "G"ood Luck ;) Of course, if you use this flag in the "global" zone, you can pretend it makes sense and stands for that. I do...

5. The "-t" option to patchadd is available for people who got used to the old patch error code numbers and know them by heart (like me. Well... most of them). Even on a system with zones, they make it so only a return code of "0" indicates absolute success. Any other number indicates a problem.

6. The "-R" option to patchadd (with zones enabled) cannot be used to reference the root filesystem in any zone other than the "global" zone. If you choose to ignore this warning and use it on a "local" zone anyway, side effects may include damage to the "local" zone's filesystem, damage to the "global" zone's filesystem, security problems with the "global" zone's filesystem, nauseau, fatigue, dry-eye, constipation, arthritis and random "night terrors." Contact your doctor if you have any trouble breathing calmly while patching "global" zones, as this may be the sign of a rare, but serious, side effect ;)

Happy patching. Here's to eventually being allowed to do it on the weekdays :)

, Mike

linux unix internet technology

Migrating Non-Global Zones Between Machines On Solaris 10

Apologies for the length of this post. I couldn't find any reasonable way to make it any shorter - so I'm including a link to where Sun is officially keeping all their documentation on this subject, should you want to explore it beyond this post. You can acess that info here!

As I always say when working with Solaris 10 (or whatever the latest version of Solaris is) you should do a trial run of a zone migration before you actually move any zone from one machine to another. Make up a bogus zone (like "origzone") set up the basic attributes and use that before you risk your important data!

The move is fairly simple at a high level (I will avoid breaching the subject of Solaris' RBAC to keep this at least somewhat short!):

1. The zone is halted and detached from its current host.
2. The zone is packed up.
3. The zonepath is moved to the target host, where it's attached.

In order to migrate the zone, you have to meet a few pre-requisites:

1. The global zone on the target system has to be running the same Solaris release as the original host.

2. To ensure that the zone will run properly, the target system needs to have the same versions of required OS packages and patches as the original. Any other packages and packages can be different; for instance any third party products and their respective patches don't need to be the same. Any packages that are OS-related and/or zone-inheritable must be included!

3. The host and target systems have to be of the same machine architecture (easily verifiable using "uname -m").

Once that's all been cofirmed, we're off! Running "zoneadm detach" on the original host sets the zone up so it can be attached to the target system. Running "zoneadm attach" on the target system attaches the migrated zone and verifies that the new system can host it.

Now we'll walk step-by-step through migrating the non-global zone from the original machine to the target host. Again, this process must be run as root in the global zone in order to ensure that it works without error (not to mention at all ;)

1. Become the root user and halt the zone to be migrated ("origzone" in this procedure).

orighost.xyz.com# zoneadm -z origzone halt

2. Now detach the zone.

orighost.xyz.com# zoneadm -z my-zone detach

3. Move the zonepath for origzone to the new host.

You can do this any way you want, basically. Just cd into /export/zones, and tar up the contents of the origzone directory. Then move the tarred up zone file to the target server using whatever file transfer facility you prefer, so that you have /export/zones/origzone set up on your target host.

4. On the target host, configure the zone:

targethost.xyz.com# zonecfg -z origzone

You will get a message from Solaris letting you know that origzone doesn't exist on the target machine, and needs to be created. This is only partially true ;)

5. To create the zone origzone on the target host, run "zonecfg -a /export/zones/origzone" on the target host. Then do the following:

zonecfg:origzone> create -a /export/zones/origzone

6. Make any necessary modifications to the configuration. For instance, to borrow from the example on sun.com, the NIC on the target host may be different, or other devices may not be the same. To change the NIC:

zonecfg:origzone> select net physical=ce0
zonecfg:origzone:net> set physical=bge0
zonecfg:origzone:net> end

7. Now, commit the new configuration and exit.

zonecfg:origzone> commit
zonecfg:origzone> exit

8. Attach the zone on the target host using the included validation.

targethost.xyz.com# zoneadm -z origzone attach

Note that, at this point, you'll be warned if you didn't meet any of the pre-requisites regarding required-same packages and patches on the target host, as we mentioned above.

As an additonal end-note, you can force the attach operation without performing the validation if you're absolutely sure that everything is okay and the message is in error. Of course, if the message is correct, and you're not, forcing the attach may result in random and bizarre zone behaviour later on. It's not recommended (By Sun or by me :)

targetzone.xyz.com# zoneadm -z origzone attach -F

Hopefully now you've gotten your zone migrated, and can script out as much as possible, to make mass migrations as simple and expeditious as possible in the future!

, Mike

linux unix internet technology