Prepping For Setting Up VCS NFS Clustering On Solaris 10

Hey there,

Hope your work week is beginning swimmingly :) Mine is kind of like the end of last week, although that sentence is a bit of a non sequitur. I actually haven't stopped working, so it still is last week this week (???) You know what I mean (Although, I hope that you don't ;)

Today we're going to look at preparing your Solaris 10 system for clustering NFS (The Network File System ... the "The" is silent ;) on VCS (Veritas Cluster Server). In many ways, it's the same as setting it up on previous versions of Solaris, but it differs in many ways, as well. Apparently, I spend way too much time looking at this issue in many many ways ;)

NOTE: This post is kind of a wrapper around our previous posts on adding NFS to an existing VCS cluster and adding NFS to a VCS cluster with no down time. You can check either of those out if you want to read up on doing the VCS configuration part of the "VCS NFS" setup. This post is of purely a preparatory nature (with, admittedly, some post-installation test steps and a pointer to another old post on an uncommon Solaris 10 VCS NFS error and how to fix it).

1. First, we'll do the Solaris 10 setup. This is very important, since the SMF (Soul Macerating Futility or, possibly, Service Management Facility) has changed the way in which "services" and "run levels" are either dealt with or completely subverted ;)

a. If you're going to be depending on VCS for NFS management, it will interest you to know that VCS won't have anything to do with NFS if you want to use it on your own (on the same machine). For that reason, we're going to use svccfg to "delete" the following services, rather than using svcadm to "disable" them.

host # svccfg delete -f svc:/network/nfs/server:default
host # svccfg delete -f svc:/network/nfs/mapid:default
host # svccfg delete -f svc:/network/nfs/status:default
host # svccfg delete -f svc:/network/nfs/nlockmgr:default

b. Doing the above may (actually, should) kill the lockd and statd daemons that are probably already running (I'm trying not to be too presumptuous ;) If that's the case, you'll need to start those up again.

host # (/usr/lib/nfs/lockd &)
host # (/usr/lib/nfs/statd &)

VCS will take care of starting them once it's good to go.

If the above way of starting those services from the command line seems goofy, check out our aging post on what to do when nohup hangs up anyway from way back when. It's a fair read and may still be interesting to a certain degree ;)

c. Finally, you just need to make one directory, for convenience's sake (and also so that the "NFSRestart" resource will actually work):

host # mkdir /opt/VRTSvcs/lock

2. Now (whooshing right past the "actual" VCS NFS setup (referenced above from two previous posts on that subject), you're ready to do a few simple tests.

a. Once you have NFS running on VCS on your primary node, pick another node (we'll just assume you picked the secondary) and test that the NFS mount is up and working like VCS says it is (You can't always take it at its word):

host2 # showmount -e host
export list for host:
/that/vcs/nfs/directory/share (everyone)

Then just make sure you can actually mount the share.

b. Then fail over to your secondary node and run the same test on the primary:

host # showmount -e host2
export list for host2:
/that/vcs/nfs/directory/share (everyone)

Again, mount the share just to be sure everything's in working order.

c. If you encounter an error while running "showmount" on either server, like this:

showmount: host: RPC: Rpcbind failure - RPC: Authentication error

accompanied by your being able to generate this error (although, not necessarily):

host # rpcinfo -p host
rpcinfo: can't contact portmapper: RPC: Authentication error; why = Failed (unspecified error)

be sure to check out our previous post on this little Solaris 10 VCS NFS gotcha and, hopefully, you'll end up knowing more than you ever wanted to about how to straighten that out :)

And, finally, in an out-of-sequence-series of only four posts, you're finally done setting up NFS in VCS on Solaris 10. Hopefully, you finished a long time ago. It's been six months since some of the referenced posts were originally published. Over time, it gets getting harder and harder to dot all the i's on this blog ;)

Cheers,

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

A Little VCS NFS Gotcha On Solaris 10

Hey again,

We're going back to the Solaris 10 Unix well, reaching back a little (as opposed to the 14 month reach-back we did yesterday ;) and adding a little something to our posts on adding NFS management to an existing VCS cluster as well as the follow up on how to do the exact same thing without taking your VCS cluster offline. Today's post is actually another little bit of fix-it knowledge to keep in the back of your hat (if that's even an expression anyone's ever used... if not, consider it © ® ™ us ;). And, of course, this piece of knowledge came to everyone by accident. Actually, by virtue of an accident... the answer was found methodically... I think ;)

In any event, after extensively field-testing the methods espoused in the earlier posts referenced above, a deployment of clustered servers to an offsite location ended up having an issue that we weren't able to anticipate (or cause to occur in previous cookie-cutter-similar deployments). For some reason, when we rolled this cluster out, NFS just refused to work in a failover capacity. Actually, it only failed, specifically, to allow the main node to mount on the failover node. This problem seems pedestrian (even still ;) - the only odd thing was that it had never happened before under equal circumstances.

Here's what we figured out along the way (and how to fix it, too ;) For our purposes today (and the way it was then) the NFS cluster component works fine on node-b, but node-a can't mount the NFS resource when it fails over to node-b.

1. The first thing most people do in any investigation is to see if the basic stuff is all up and running. We don't like to be different, so we duly checked that all of the required VCS resources were up and online. They were; which explained the puzzling ONLINE state ;)

2. We then proceeded to ensure that, in fact, node-b was sharing out the NFS resource. Commands like showmount indicated that it, indeed, was. A little research into the subject showed that the issue we ended up having can indicate an RPC failure at this point, as well, but it's best to try step 3, too, just to be sure the problem isn't confined to a single server (although the fix for it is the same no matter which way your story goes ;)

3. Then we finally struck gold, and got an actual error, when we tried to hit the mount from node-a:

node-a # showmount -e node-b
 showmount: node-a: RPC: Rpcbind failure - RPC: Authentication error
node-a # rpcinfo -p node-b
 rpcinfo: can't contact portmapper: RPC: Authentication error; why = Failed (unspecified error)

4. Unspecified errors are the best kind of errors you can get since there are a much wider variety of possible solutions you can come up with... Or, maybe I have that backwards... There's really not much more to step 4. This step is a practice in surrealism ;)

5. It turns out that the answer lay in setting rpcbind properties (away from the defaults on both servers). The answer to the problem (or the fix, if you will) actually makes more sense than the way things "usually" work. The first thing we did was to set rpcbind to "global" on both nodes. By default, it was set to "local_only." We double confirmed that this is still the case on other cluster setups we have running, in which everything is hunky-dory. You also need to do these steps on both nodes (or all nodes) in your cluster, while, here, we're only showing what we typed on the active NFS resource-sharing node:

node-b # svcprop network/rpc/bind:default | grep local_only <-- See if the local_only property is set
config/local_only boolean true <-- and there it is!

then move on to fixing the problem (again on both nodes) by setting the rpcbind configuration value to global (which, in the instance of rpcbind, actually means setting the local_only attribute to "false"):

node=b # svccfg
svc:> select network/rpc/bind
svc:/network/rpc/bind> setprop config/local_only=false
svc:/network/rpc/bind> quit

6. Then, just double check to make sure you've gotten it all set up correctly:

node-b # svcprop network/rpc/bind:default | grep local_only
config/local_only boolean true

...well, that's not right, but don't give up just yet! Keep typing. Type, Forrest, Type! ;)

node-b # svcadm refresh network/rpc/bind:default
node-b # svcprop network/rpc/bind:default | grep local_only
config/local_only boolean false

there... that's better.

7. Finally, just make sure you can mount your NFS resource from whichever node isn't currently hosting the NFS resource. You don't necessarily have to test it on both nodes, once you've fixed this issue on both, but why risk the near-future embarrassment?

node-a # showmount -e node-b
export list for node-b:
/our/shared/directory (everyone)

And that's that. You should be good to go :) Since all's well that ends well, we'll try not to leave you with any clichés in our farewell. Parting is, after all, such sweet sorrow. At least until tomorrow :)

Cheers,

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

JumpStart Symlinks And Solaris 10 Unix

Hey there,

Today's topic covers a little issue I ran into at work (which I actually do from time to time ;) that had me puzzled for a bit. If you're a grizzled Solaris/Slakware veteran like me, some of the newer features of Solaris 10 are pretty much lost on you until you absolutely "need" to understand them ;) Case in point, NFS sharing and JumpStart on Solaris 10 and/or ZFS (a point we seem to have overlooked in the onslaught of information in our post featuring Solaris' 5/08 Release Notes. Do a find for "NFS" - it's a realllllly long release note :)

The issue we had going on was that we'd gotten in the habit of running a JumpStart, and then sometime later (maybe days, weeks, who knows?) running another, etc; money's tight, not too many new machines to build and everything worked fine. Solaris 10 was doing great (That isn't to say that it didn't do great the entire time we had this problem ;) The issue was more of a fundamental misunderstanding and lack of knowledge on my part). Then, a few days ago, we had the time and the means, and I began a two-fisted JumpStart (Nothing violent, just concurrent installations ;)

Then, after I'd run through this a few more times than I'm proud to admit, I finally threw up my hands after yet another attempt at double-JumpStarting failed. The failure was very specific so I was fairly certain I was either doing something wrong or I was doing something wrong ;) Basically, both machines would boot up to the net and begin their JumpStart installations. All would go swimmingly; finding the JumpStart host, grabbing the correct profile, finish scripts and sysidcfg files. The bummer was that, after completing the full install of Solaris 10 (yes, it made us wait until after "all" of the software had been installed) it would get this funky error (hopefully, you've seen it before, and this post might just help you out :) :

Completed software installation

Solaris 10 software installation succeeded

Customizing system files
        - Mount points table (/etc/vfstab)
        - Unselected disk mount points (/var/sadm/system/data/vfstab.unselected)        - Network host addresses (/etc/hosts)

ERROR: Could not open file (/etc/hosts)

ERROR: Could not set up the remote host file (/etc/hosts)

ERROR: System installation failed
Solaris installation program exited.

????????????

Okay, so we were kind of stumped (well, totally stumped until we figured out the solution - by definition, I think ;) It turns out that one of our procedurals before initiating the "boot net - install" portion of single server JumpStarts was actually contributing to our confusion about what the problem really was. The reason for that is because we would always sync our local JumpStart server with the master. Good practice, but (in this case) a bit of a diversion. Anyway, we'll get to why that mattered in a bit ;)

Investigation into the matter (which, after several failed installs, consisted of crashing the install in the mini-root to check the state of the JumpStart temporary mount configuration in real-time) revealed something interesting. If you look up the page a little (or just remember ;) the killer error we got was:

ERROR: Could not open file (/etc/hosts)

ERROR: Could not set up the remote host file (/etc/hosts)

ERROR: System installation failed

Looking at the state of the filesystem, after crashing at the point of error, revealed this directory structure in the temporarily mounted /etc filesystem (stripped down a bit for brevity's sake):

...
-r--r--r-- 1 root sys 99 Feb 2 16:48 hosts
...
-r--r--r-- 1 root sys 91 Feb 2 16:48 ipnodes
...
-r--r--r-- 1 root sys 384 Feb 2 16:48 netmasks
...

The reason that's interesting is that those files should have looked like this:

...
lrwxrwxrwx 1 root other 29 Feb 2 16:56 hosts -> ../../tmp/root/etc/inet/hosts
...
lrwxrwxrwx 1 root other 31 Feb 2 16:56 ipnodes -> ../../tmp/root/etc/inet/ipnodes
...
lrwxrwxrwx 1 root other 32 Feb 2 16:56 netmasks -> ../../tmp/root/etc/inet/netmasks
...

Essentially, Solaris 10 was converting special symlinked files into straight-up flat-files during the JumpStart process. Once that was complete (and the files were corrupted) it really "couldn't" open them up, because the real /etc/hosts file was supposed to be in the /tmp/root/etc directory and not the local one (which is on JumpStart's read-only mini-root filesystem)!

It turns out that this problem (as far as we were interested in figuring out) seems to manifest itself in Solaris 10 for the most part. It may happen in Solaris 9, but we can't go back now!!! :)

And the root cause was... drum roll, please, as I build up to feeling really stupid ;)

The JumpStart mini-root directory was being served up via NFS "read/write"! Doh! And (I'm bringing this back from up top, just as I promised) our procedure for doing JumpStarts had actually made this harder to see. Since we only did one JumpStart at a time, the initial JumpStart would work (even though it left behind a corrupted filesystem). And, per our procedure, right before we kicked off the next one, we'd sync that filesystem up with the known-good master JumpStart server. At no point was this filesystem corruption ever noticed due to the fact that we never had to jump more than one box at a time. Crazy ;)

Anyway, long story short, the quick and simple fix was to unshare the mini-root (/JumpStart/Sol10 for instance, or whatever yours might be) and then reshare it as "read only." If you've been doing this all along (which you should be ;), you'll never have our problem ...probably ;)

Depending upon how your system is setup, you can share NFS a number of ways in Solaris 10. After I ran "unshareall," I was a bit puzzled as to why /etc/dfs/dfstab was empty (See what I meant before? ;). Since I'm such a dinosaur, I wrote it off, figuring somebody had run the share command at the command line and forgot to put that command in an init script or the dfstab file. On many occasions, I would have been correct (and Solaris 10 does still support this type of NFS sharing). The cool thing here is that our JumpStart mini-root was living on a Solaris 10 ZFS dataset. So, all that had to be done to correct the issue (after I uncorrected my incorrect correction ;) was to adjust a property of the dataset in Solaris 10 (actually a very cool feature, I think :) Since the ZFS datasets have a "sharenfs" attribute built-in (set to "no" by default), all we had to do was to change that. A very simple command line to type and a very simple solution to a seemingly complicated issue:

host # zfs set sharenfs=ro,anon=0 maindg/jumpstart/Sol10

Problem solved. And only 3 or 4 hours wasted (I mean, well spent ;) Hopefully this pitiful little tale of woe will get you out of a similar jam sometime :)

Cheers,

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Patching Solaris Unix - The Rules Change Again!

Hey There,

I was going to do a bit today that carried on from yesterday's post on Solaris 10 boot archive patching issues, and walk through using smpatch to keep your machine up to date (especially if you prefer to stay away from the ui gui bits ;)

I reserve the right to do that, maybe tomorrow/maybe not (I've found that posting on a streak - or too many similar posts in a row - doesn't play out very well). In any event, when I went to update my patches on my SunBlade, I got an interesting message from "smpatch." It's going to be the topic of today's post; if you can even really call it a topic. It seems like many of the "old ways" Sun used to allow you to use to patch your machine are going the way of the Dodo :)

And, now, the Announcement Which Services The Public (Sounds much worse than Public Service Announcement ;)

If you manage, or own, any Sun boxes (and use any of their built-in-or-free utilities to keep your machine(s) current) this EOL (End Of Life) notice regarding certain patch updating/management utilities may be of interest to you.

NOTE: I included the entire messages queue for completeness, but Message "2 of 2" (regarding smpatch) is the one that might interest you, unless you've been hassling with trying to get SunConnection working since March of 2008 ;)

Enjoy (in moderation :)

host # smpatch messages -a

        Messages Of The Day

        Message 1 of 2
        Date: 2008/02/29
        Title: Attention all Sun Connection Hosted users
        Description: Users of Sun Update Manager, smpatch, and Sun Connection Enterprise (UCE) are not affected
by this EOL announcement, but should continue to read on to determine if their systems are
enabled for hosted management.

Sun Connection Hosted Customers:  On March 1, 2008, https://sunconnection.sun.com/ will
reach End of Service Life.  You will still be able to receive updates via Sun Update Manager,
smpatch and UCE.  We suggest that you check your systems to ensure the Sun Connection Hosted
transport mechanism is disabled.

To see if your system has been enabled for hosted management, check the value of your system
by using the following command:
     /usr/lib/cc-ccr/bin/ccr -g cns.service.swupPortalMgmt.status
This command will return a value of "disabled", "enabled", or "". If the value is "disabled"
or "", thank you for checking, no further action is required.  If "enabled", please take
action by performing one of the following steps:

You can disable the transport mechanism from Update Manager to Sun Connection hosted by
applying the  patch 121081-08 (sparc)  or 121082-08 (x86) which will be available on Feb.
29, 2008.   If you want more control, you can also disconnect your system from Sun
Connection Hosted and shutdown the associated daemons on your system(s) via either one of
these two options.

1) From the Hosted Management Portal
[Note: This option will not be available once Sun Connection Hosted has been shutdown.]
 a) Go to https://updates.sun.com/ and login to the Hosted management site with your Sun
    Online Account.
 b) Select the Systems tab
 c) For each system listed, click the Edit System Settings icon (second icon to the right
    of the system name)
 d) Scroll to the bottom of the Edit Your System Settings page and click the Delete System
    button
 e) When asked for confirmation click Continue

This option will unregister your systems from the Portal and send a job down to your system
telling it to disable the portal management functionality.

2) Disable Hosted Management functionality from the client
 a) From a terminal window on the target client system,  su to root
 b) Execute the following command to shutdown the local Hosted daemons:
# /usr/lib/cc-cfw/framework/lib/cc-client-adm stop
 c) Execute the following command to prevent the daemons from being restarted at system boot
# /usr/lib/cc-cfw/framework/lib/cc-client-adm disable

Once complete, either option will disassociate your client system with Sun Connection Hosted.

        Message 2 of 2
        Date: 2009/01/13
        Title: Attention all Sun Connection users
        Description: The following patches for Sun Update Connection clients will soon be necessary to validate patch downloads.

Solaris 10 (Sparc) 121118-15
Solaris 10 (x86) 121119-15

Solaris 9 (Sparc) 140476-01
Solaris 9 (x86) 140477-01

Solaris 8 (Sparc) 140475-01

These patches deliver an updated public certificate into the smpatch keystore which will be required to validate patches signed after the expiry of the current patch signing certificate. Run

smpatch update

to ensure you are running with the latest versions of all required patches.  After applying the patch, the default patchset name will become current2. The current2 patchset contains all the patches available today, and will also deliver the newly signed patches once they become available.

Sun Update Connection Proxy (Local Patch Server) users:

A patch will be available shortly to make use of the new certificates.  If you wish, you may alter the patchSigningCertAlias property in the file /var/patchsvr/lps/WEB-INF/applicationContext-lps.xml to

patchsigning:patchsigning2:patchsigning3

After installing the above patch on your system to make use of this new certificate.  A restart of patchsvr will be required to pick up the changes.

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Solaris 10 Unix Patch Update Boot Archive Woes

Ahoy there,

Today we're going to take a look at some "Solaris 10" specific stuff (at least, I hope it's only the OS ;) that's been making me nuts lately. My problem may have to do with the new patch update/management setup, but I've run down enough dead-ends on that hunt that I'm fairly sure it has to do with the implementation of the Solaris 10 "boot archive" and may also be contained within release "10/08," although I've read complaints from users running earlier versions (actually, strangely enough *** heavy sarcasm *** most of the complaints seem to stem from users of the more recent releases ;)

From a fresh install of 10/08, I thought (for once in my life) I'd do the convenient thing on my SunBlade and setup patch notifications. Usually, for my personal boxes, I'll leave everything alone and never fix anything until I notice that it's causing me a problem (which is generally never -- Not that I've ever run any Solaris versions that were bullet-proof, just that I didn't notice any problems I couldn't live with. Like they say; if it ain't broke... ;)

Now that I've gone through my 4th or 5th update using Sun's update manager (which basically just downloads all the patches I need and then runs patchadd in the correct order), I've tried doing the same thing manually - thinking that might be the issue - but ended up in the same quandary. The problem is starting to irritate me. I'm not so much worried about the fact that this issue occurs at all, just that it occurs on my workstation which doesn't have any sort of console connection to it. Ergo, if I run patch updates from home, I have to wait until I get back in the office to get past the single-user-mode hang-up.

The basic issue plays out like this (assuming a simple one disk system with no mirroring, etc):

1. Patches are added in the correct order, patch installs are validated and the system reboots.

2. After cruising past the ok> prompt and starting to boot back up, the system inevitably fails and stops at the dreaded "control-D-or-enter-root-password-for-maintenance" prompt.

3. The error message is always the same, with slight variations denoted by asterisks:

Warning: The following files in / differ from the boot archive:
***
***
...

Immediate fixing of the issue (assuming you're at the console) is very simple to fix, and (to their credit) Sun does include the exact steps you need to go through to take care of it, right after the error message.

Those steps would be:

1. Bring your system down to the PROM level after entering the root password:

host # init 0

or

host # halt (the stop+a keys for those of you who like to get as much bang for the buck from your keystrokes as possible ;)

2. Bring it up in failsafe mode and fix the boot archive problem (which it will, basically, fix for you):

ok> boot -F failsafe <-- with "-Z zpool_dataset" if you're booting a ZFS Root Pool ("boot -L" will list the pools out for you at the ok> prompt)
...
blah, blah, blah

and then:

Do you wish to automatically update this boot archive? [y,n,?] y

and, more often than not, you'll then have to run fsck against your root partition and either exit from single user mode to continue the boot process or do another "init 0" followed by a straight up "ok> boot"

Another option, after your system has failed to boot up properly following patching, is to just clear the boot archive. This works well also (even when your system is live), but is frowned upon in some academic circles. Just enter the root password to get into single user mode and run:

host # svcadm clear system/boot-archive
host # exit

and your machine will come up fine. I've tried applying some basic logic to the problem by executing that command pre-and-post-patching before rebooting, but I still end up in the same boat (??? Why, God? Why!!!!??? ;)

In any event, I'm sorry that we still don't have this site on new hosting. I would "love" for this to be a post that had comments enabled. Someone out there must know the answer (and not any of the regular ones about "known bugs that may never get resolved" ;)

Just as a "maybe/possibly" in closing, I noticed this on my Sparc workstation. At this point I have a sneaking suspicion it's a kernel patch revision issue (based on bootadm's output, tacked on to the end of the post ;), so (if what I've been reading on the message boards is any indication), I'll have this problem for anywhere from "a while" to forever, unless I decide to put my head on the chopping block and try to patchrm my kernel back to a state it's never been in ;)

Hope this post helps you out if you get stuck in the same situation.

Cheers,

host # bootadm list-archive
platform/FJSV,GPUU/kernel
platform/FJSV,GPUZC-L/kernel
platform/FJSV,GPUZC-M/kernel
platform/SUNW,A70/kernel
platform/SUNW,Netra-210/kernel
platform/SUNW,Netra-240/kernel
platform/SUNW,Netra-440/kernel
platform/SUNW,Netra-CP2300/kernel
platform/SUNW,Netra-CP3010/kernel
platform/SUNW,Netra-T12/kernel
platform/SUNW,Netra-T4/kernel
platform/SUNW,SPARC-Enterprise/kernel
platform/SUNW,Serverblade1/kernel
platform/SUNW,Sun-Blade-100/kernel
platform/SUNW,Sun-Blade-1000/kernel
platform/SUNW,Sun-Blade-1500/kernel
platform/SUNW,Sun-Blade-2500/kernel
platform/SUNW,Sun-Fire-15000/kernel
platform/SUNW,Sun-Fire-280R/kernel
platform/SUNW,Sun-Fire-480R/kernel
platform/SUNW,Sun-Fire-880/kernel
platform/SUNW,Sun-Fire-V210/kernel
platform/SUNW,Sun-Fire-V215/kernel
platform/SUNW,Sun-Fire-V240/kernel
platform/SUNW,Sun-Fire-V245/kernel
platform/SUNW,Sun-Fire-V250/kernel
platform/SUNW,Sun-Fire-V440/kernel
platform/SUNW,Sun-Fire-V445/kernel
platform/SUNW,Sun-Fire-V490/kernel
platform/SUNW,Sun-Fire-V890/kernel
platform/SUNW,Sun-Fire/kernel
platform/SUNW,Ultra-1-Engine/kernel
platform/SUNW,Ultra-250/kernel
platform/SUNW,Ultra-4/kernel
platform/SUNW,Ultra-5_10/kernel
platform/SUNW,Ultra-80/kernel
platform/SUNW,Ultra-Enterprise-10000/kernel
platform/SUNW,Ultra-Enterprise/kernel
platform/SUNW,UltraAX-MP/kernel
platform/SUNW,UltraAX-e/kernel
platform/SUNW,UltraAX-e2/kernel
platform/SUNW,UltraAX-i2/kernel
platform/SUNW,UltraSPARC-IIe-NetraCT-40/kernel
platform/SUNW,UltraSPARC-IIe-NetraCT-60/kernel
platform/SUNW,UltraSPARC-IIi-Engine/kernel
platform/SUNW,UltraSPARC-IIi-Netract/kernel
platform/SUNW,UltraSPARC-IIi-cEngine/kernel
platform/SUNW,UltraSPARCengine_CP-20/kernel
platform/SUNW,UltraSPARCengine_CP-40/kernel
platform/SUNW,UltraSPARCengine_CP-60/kernel
platform/SUNW,UltraSPARCengine_CP-80/kernel
platform/TSBW,8000/kernel
platform/TSBW,Ultra-2e/kernel
platform/TSBW,Ultra-2i/kernel
platform/TSBW,Ultra-3i/kernel
platform/sun4u-us3/kernel
platform/sun4u/kernel
platform/sun4us/kernel
etc/cluster/nodeid
etc/dacf.conf
etc/mach
kernel

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Dealing With ZFS-Rooted Zones on Solaris 10 Unix

Hey there,

Today, we're going to take a look at a problem that's been haunting Solaris 10 (and, to a degree, Open Solaris) for almost 3 years now. This ties back pretty closely to earlier posts we put out on migrating zones, patching local and global zones and working with zfs filesystems, since it has exactly to do with a problem concerning zfs, Solaris 10 zones and one specific way in which they can be created.

Theoretically, it would seem, that the one way that's causing the most problems is the one way that should be the most desirable way to configure your setup (???)

Here's a little something to think about when considering creating zones with a zfs filesystem. Although this quote is taken out of context, directly from Sun, it was actually put out there as a selling point (in its own context actually):

zones are integrated into the operating system, providing seamless functionality and a smooth upgrade path.

However, as many of you may be aware by now (it being July 2nd, 2008 and, officially, version 5/08 of Solaris 10 is on the market), although creating bootable zfs zones and zones with zfs root filesystems is now finally possible (I believe it was originally introduced, in a small way, back in 6/05 -right before the 06/06 official release), it still suffers from some severe issues, that may not be initially evident. That is to say, if you didn't do your homework before you took advantage of this seemingly great feature, you've probably gotten burned in one fashion
or another, with regards to the upgrade process. Per Sun, again:

Solaris 10 6/06 supports the use of ZFS file systems. It is possible to install a zone into a ZFS fs, but the installer/upgrader program does not yet understand ZFS well enough to upgrade zones that "live" on a ZFS file system.

Because of this (and repeating this ;) upgrading a system that has a zone installed on a ZFS file system is not yet supported. To this day (to my knowledge) the problem still hasn't been completely resolved. Again, from Sun's bug list (And, I know I sound like I'm Sun-bashing here, but I am coming to a positive point. I swear :)

zoneadm attach command Might Fail (6550154)
when you attach a zone, if the original host and the new host have packages at the same patch level but at different intermediate patch histories, the zone attach might fail. various error messages are displayed. The error message depends on the patch histories of the two hosts.
workaround: Ensure that the original host and the new host machines have had the same sequence of patch versions applied for each patch.

Basically, the way things stand now, if you have a zone built on a zfs root filesystem (rather than, say ufs), if you need to upgrade, you officially have 3 options:

l. Be pro-active and "Don't do it!"

2. Go ahead and do it, but be sure to uninstall your zones before upgrading to a new release of Solaris 10, and then reinstall them when your upgrade to the new release is completed.

3. Go ahead and do it, but instead of following the more traditional upgrade-path, completely reinstall the system in order to perform the upgrade. This option makes the least sense, since reinstallation and upgrading aren't synonymous.

Now, for the rainbow after the storm. Yes, rainbows are somewhat illusory and their beauty isn't necessarily the matched-opposite of the horrors of nature you may have had to endure in order for it to bring you to that phenomenon, but it's a lot better than nothing, right ;)

The situation, as you may have guessed, is still pretty much up-in-the air, but there is hope; and in more than one area. For x86 (and possibly Sparc), this initiative is being fast-tracked by Sun for Open Solaris/Solaris 10 (Note that it's dated June 27th, 2008 :) - It basically proposes a -b flag to zoneadm attach, to be used in conjunction with the -u flag, to allow for backing patches out of a zone before an OS update. The full discussion, to date, is located here on openSolaris.org.

Why is this important?

As we noted above, the biggest problem Solaris 10 has with upgrading the OS on machines that have zones with zfs roots is that every single patch and package must be the same after the upgrade in order for it to be considered successful and Solaris 10's update software doesn't work with ZFS well enough to be able to guarantee that patches that get installed in one zone will necessarily get installed in another (global vs. local, zfs vs. ufs/vxfs, even zfs vs. zfs). If we were allowed to ignore certain patches and/or packages in our upgrades, this might make the likelihood of failure drop dramatically!

And here's one more ray of hope (which might be even better by the time you need to apply it). Here's how to upgrade your OS, assuming it has zones mounted on zfs roots, and (possibly) get away with not having to go to the extremes Sun is obligated to recommend. It's actually fairly simple. Mostly because it isn't guaranteed to work ;) The one good thing is that, if it doesn't work, you'll have your data saved off, so (if this procedure fails) you can still do it the hard way and not lose anything, except time, by trying :)

Do the following. We'll assume you've read our previous posts on migrating and patching both local and global zones and understand the basic system-down commands that would necessarily precede the following:

l. Halt and detach each of your zones that sits on a zfs root (I'd personally do this for all of my zones), like so:

host # zoneadm -z ZONENAME1 halt
host # zoneadm -z ZONENAME1 detach

2. Export all of your zfs pools:

host # zpool export ZONENAME1 <-Make sure that you note the names here, for the reverse process, just in case!
host # zpool export ZONENAME2

3. Perform your upgrade however you prefer.

4. Import all of your zfs pools

host # zpool import <--If this doesn't work, use the names you specified during the export previously.

or

host # zpool import ZONENAME1
host # zpool import ZONENAME2

5. Reattach and boot/install your zones using the -F flag to force the issue (of course you can leave it out if you want, just to see what happens. Sometimes forcing makes things work that are flagged as errors, but aren't really. You can also use the -n flag to do a dry-run):

host# zoneadm -z ZONENAME1 attach -F
host# zoneadm -z ZONENAME1 boot -F <--Note that for the zoneadm command. if you don't list a zone name with the -z flag, the subcommand (halt, detach, attach, boot, etc) would apply to all zones!

And you should either be all set or have a more limited set of issues to deal with (probably mostly patch related).

Later on in the week (if we don't run out of screen space ;) we'll look at ways to troubleshoot a Solaris 10 zfs-root zone upgrade gone bad.

Until that bright and sunny day:)

, Mike

Prtdiag And "Lane Width Failed" Errors On Solaris 10

A lot of admins who work with Solaris 10 Zones may have run into this situation already, but I'm just starting to hear it now from users who like to do their own diagnostics before coming to me with a possible system issue. Don't get me wrong folks; I love it when users show this kind of initiative (as long as they don't go into the data center and start pressing buttons ;)

This problem is similar to the problem Solaris 10 has with old style /usr/ucb/ps, but it isn't quite as prevalent.

The situation that occurs is that a user, trying to gather information on the system that may, or may not be, having a hardware or software issue, runs a pretty standard command called "prtdiag," probably like this:

host # /usr/platform/`uname -i`/sbin/prtdiag

Now, in Solaris 10, this doesn't always cause an error. The reason for this, and how it differs from the old style ps error, is that it only occurs when the user runs prtdiag on Solaris 10 with Zones enabled.

Another thing that makes this unusual error so rare is that, on most Zone-enabled Solaris 10 setups, users accounts are all setup in the non-global Zone and prtdiag will only run in the global Zone. Obviously, the "rules" aren't followed all the time, insofar as system setup and access are concerned. Like they say: the customer is always right, even if he's doing something he probably shouldn't be (or something like that ;)

So far, I've only seen this error on the Mx000 Series Servers from Sun, but that's probably because we use those the most. Generally, the error will present itself in some way similar to the following (Looks scarier than it is and the actual error may vary)

host # /usr/platform/`uname -i`/sbin/prtdiag <-- Stripping the 100 lines preceding the error in my ongoing effort to fight eye-strain ;)

... 
    IO                                                Lane/Frq 
LSB Type  LPID   RvID,DvID,VnID       BDF       State Act,  Max   Name                     Model               
--- ----- ----   ------------------   --------- ----- ----------- ------------------------ ------------------ 
    Logical Path 
    ------------ 
Getting lane width failed for path /pci@3,800000/SUNW,emlxs@0

And, again, just like our ps error on Solaris 10, running the command as root makes everything work just fine:

root@host # /usr/platform/`uname -i`/sbin/prtdiag

...
    IO                                                Lane/Frq 
LSB Type  LPID   RvID,DvID,VnID       BDF       State Act,  Max   Name                     Model               
--- ----- ----   ------------------   --------- ----- ----------- ------------------------ ------------------ 
    Logical Path 
    ------------ 
00  PCIe  3      1, fc21, 10af        1, 0,  0   okay     4,  4   SUNW,emlxs-pci10af,fc21   LPe110094-S           
    /pci@3,800000/SUNW,emlxs@0

Sun's stock answer, for now, is to change the permissions of the prtdiag command so that it runs setuid root (For those of us who are reading this and don't know what that means - A very small portion of the audience that bothered to read this far, I'm sure - when a program is setuid "username," it will run as that user - with that user's privileges - no matter what user actually executes it)

root@host # chmod 4755 /usr/platform/`uname -i`/sbin/prtdiag

or, if you prefer to change your file modes in alpha:

root@host # chmod u+s /usr/platform/`uname -i`/sbin/prtdiag

Probably the best way to work-around this, and keep with Sun's basic security requirement of not running programs like this setuid root, is to make use of a program called sudo (which comes with Solaris 10). Just include a rule like the one below, so that users can only run "/usr/platform/`uname -i`/sbin/prtdiag" straight up and can't run it with any additional flags or switches. This will allow them to get the information they want and safeguard you, the admin, against any unforeseen issues with this work-around. Example rule below:

ALL ALL = (root) /usr/platform/`uname -i`/sbin/prtdiag ""

Another promise from me that, eventually, we will get to a post devoted entirely to sudo. For now, here's a quick rundown on how this rule reads:

ALL <--- All users can use this sudo rule
ALL <--- This command can be used on any host.
= <--- The cement that connects the preceding user and host restrictions with the commands and options to follow
(root) <--- This command will be run as the user root
/usr/platform/`uname -i`/sbin/prtdiag "" <--- This is the only allowed command. This command specifies that /usr/platform/`uname -i`/sbin/prtdiag can only be run with no additional switches or flags (like "-v"). The "" (double-double quotes) indicate that no switches are allowed after the command.

Hopefully, and in all likelihood, this solution should keep everyone happy. You've kept the security flaw from being exploited, retained the original permissions on /usr/platform/`uname -i`/sbin/prtdiag and allowed users to be able to get their diagnostic output.

All that's left to do is thank your user base for helping make your job easier :)

, Mike

linux unix internet technology