Tuesday, September 30, 2008

How To Resolve Veritas Disk Group Cluster Volume Management Problems On Linux or Unix

Hey There,

Today we're going to look at an issue that, while it doesn't happen all that often, happens just enough to make it post-worthy. I've only seen it a few times in my "career," but I don't always have access to the fancy software, so this problem may be more widespread than I've been lead to believe ;) The issue we'll deal with today is: What do you do when disk groups, within a cluster, conflict with one another? Or, more correctly, what do you do when disk groups within a cluster conflict with one another even though all the disk is being shared by every node in the cluster? If that still doesn't make sense (and I'm not judging "you," it just doesn't sound right to me, yet ;) what do you do in a situation where every node in a cluster shares a common disk group and, for some bizarre reason, this creates a conflict between nodes in the cluster and some of them refuse to use the disk even though it's supposed to be accessible through every single node? Enough questions... ;)

Check out these links for a smattering of other posts we've done on dealing with Veritas Volume Manager and fussing with Veritas Cluster Server. Some of the material covered may be useful if you have problems with any of the concepts glossed over in the problem resolution at the end.

Like I mentioned, this "does" happen from time to time, and not for the reasons you might generally suspect (like one node having a lock on the disk group and refusing to share, etc). In fact, the reason this happens sometimes (in this very particular case) is quite interesting. Even quite disturbing, since you'd expect that this shouldn't be able to happen.

Here's the setup, and another reason this problem seems kind of confusing. A disk group (we'll call it DiskGroupDG1 because we're all about creativity over here ;) is being shared between 2 nodes in a 2 node cluster. Both nodes have Veritas Cluster Server (VCS) set up correctly and no other problems with Veritas exist. If the DiskGroupDG1 disk group is imported on Node1, using the Cluster Volume Manager (CVM), it can be mounted and accessed by Node2 without any issues. However, if DiskGroupDG1 is imported on Node2, using CVM, it cannot be mounted and/or access by Node1.

All things being equal, this doesn't readily make much sense. There are no disparities between the nodes (insofar as the Veritas Cluster and Volume Management setup are concerned) and things should be just peachy going one way or the other. So, what's the deal, then?

The problem, actually, has very little to do with VCS and/or CVM (Although they're totally relevant and deserve to be in the title of the post -- standard disclaimer ;). The actual issue has to do, mostly, with minor disk numbering on the Node1 and Node2 servers. What???

Here's what happens:
In the first scenario (where everything's hunky and most everything's dorey) the DiskGroupDG1 disk group is imported by CVM on Node1 and Node1 notices that the "minor numbers" of the disks in the disk group are exactly the same as the "minor numbers" on disk it already has mounted locally. You can always tell a disk's (or any other device's) minor number by using the ls command on Linux or Unix, like so:

host # /dev/dsk # ls -ls c0t0d0s0
2 lrwxrwxrwx 1 root root 41 May 11 2001 c0t0d0s0 -> ../../devices/pci@1f,4000/scsi@3/sd@0,0:a
host # /dev/dsk # ls -ls ../../devices/pci@1f,4000/scsi@3/sd@0,0:a
0 brw-r----- 1 root sys 32, 0 May 11 2001 ../../devices/pci@1f,4000/scsi@3/sd@0,0:a
<-- In this instance, the device's "major number" is 32 and the device's "minor number" is 0. Generally, with virtual disks, etc, you won't see numbers that low.

Now, on Node1, since it recognizes this conflict on import, does what Veritas VM naturally does to avoid conflict; it renumbers the imported volumes ("minor number" only) so that the imported volumes won't conflict with volumes in another disk group that's already resident on the system it's managing. Therefore, when Node2 attempts to mount, with CVM, the command is successful.
In the second scenario (where thing are a little bit hunky, but not at all dorey), Node2 imports the DiskGroupDG1 disk group and none of the minor numbers in that disk group's volumes conflict with any of its local (or already mounted) disk. The disk group volumes are imported with no error, but, the "minor numbers" are not temporarily changed, either. You see where this is going. It's a freakin' train wreck waiting to happen ;)

Now, when Node1 attempts to mount, it determines there's a conflict, but can't renumber the "minor numbers" on the disk group's volumes (since they're already imported and mounted on Node2) and, therefore, takes the only other course of action it can think of and bails completely.

So, how do you get around this for once and all time? Well, I'm not sure it's entirely possible to anticipate this problem with a variable number of nodes in a cluster, all with independent disk groups and, also, sharing volume groups between nodes, although you could take simple measures to prevent it most of the time (like running ls against every volume in every disk group in a cluster every now and again and making sure no conflicts existed. The script should be pretty easy to whip up).
Basically, in this instance (and any like it), the solution involves doing what Veritas VM did in the first scenario; except doing it all-the-way. No temporary-changing of "minor numbers." For our purposes, we'd like to change them permanently, so that they never conflict again! It can be done in a few simple steps.

1. Stop VCS on the problem node first.

2. Stop any applications using the local disk group whose "minor numbers" conflict with the "minor numbers" of the volumes in DiskGroupDG1.

3. Unmount (umount) the filesystems and deport the affected disk group.

4. Now, pick a new "minor number" that won't conflict with the DiskGroupDG1 "minor numbers." Higher is generally better, but I'd check the minor numbers on all the devices in my device tree just to be sure.

5. Run the following command against your local disk group (named, aptly, LocalDG1 ;) :

host # vxdg reminor LocalDG1 3900 <-- Note that this number is the base, so every volume, past the initial, within the disk group will have a "minor number" one integer higher than the last (3900, 3901, etc)

6. Reimport the LocalDG1 disk group

7. Remount your filesystems, restart your applications and restart VCS on the affected node.

8. You don't have to, but I'd do the same thing on all the nodes, if I had a window in which to do it.

And, that would be that. Problem solved.

You may never ever see this issue in your lifetime. But, if you do, hopefully, this page (or one like it) will still be cyber-flotsam on the info-sea ;)


, Mike

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Monday, September 29, 2008

Bash Script To Get Weather Forecasts For Your Zip Code

Hey There,

Today's Linux/Unix bash shell script is probably not the last follow up to the growing stable of scripts we've written to mine the knowledge on tap online. Today's info is grabbed from WeatherBug.com. If you missed any of the others scripts we've jammed out, you can still find them in our older bash script posts to spew out famous quotations on pretty much any subject, do encylopedia lookups, access the online Thesaurus, translate between different languages and, of course, the use the online dictionary. This time we're going to take a crack at getting online weather updates, using zip codes. If you check out the GET variables, you'll notice that the scope is the WORLD, but it doesn't seem to work well with foreign identifiers. Of course, I haven't tried all that hard to get it to work - setting me up for a post on the upgrade ;)

This script is fairly easy to run and somewhat limited. The basic limitations are that it will only work with 5 digit zip codes (no "plus 4") and it will only return a summary weather forecast. If there's more to report, it will indicate this, by letting you know there's ...more information available. You can run it like this:

host # ./weather.sh 45432

Like all of our previous scripts, this script uses wget and sed to get the job done. As per usual, this script is a work in progress and could use some spring cleaning, but, hopefully, it's good enough for you to enjoy :)

Below are a few screen shots of the script's output.

Click on either picture to see it as the artist intended it to be seen before the producers ruined everything ;)

Your Personalized Forecast

When Weather Goes Wrong

Hope you enjoy this script, and can find some use for it. Any suggestions for improvement would be greatly appreciated.

By the way, the forecast for today is cloudy with a chance of all Hell breaking loose in my neighborhood. Here's to your forecast coming back a little more sunny :)


Creative Commons License

This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License


# weather.sh - Now you can be wrong 50 percent of the time, too :)
# 2008 - Mike Golvach - eggi@comcast.net
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

if [ $# -lt 1 ]
echo "Usage: $0 Your Zip Code"
echo "Ex: $0 60015"
exit 1


if [ $# -gt 1 ]
echo "Usage: $0 Your Zip Code"
echo "Ex: $0 60015"
exit 1

$wget -nv -O - "http://weather.weatherbug.com/Common/SearchResults.html?is_search=true&nav_section=1&loc_country=WORLD&zcode=z6286&loc=$args" 2>&1|grep -i "No matches found for your request" >/dev/null 2>&1


if [ $anygood -eq 0 ]
args=`echo $args|sed 's/%20/\+/g'`
echo "No results found for $args"
exit 2

echo -n "Your Forecast:"

$wget -nv -O - "http://weather.weatherbug.com/Common/SearchResults.html?is_search=true&nav_section=1&loc_country=WORLD&zcode=z6286&loc=$args" 2>&1|grep -w For|grep h3|grep $args|sed -e :a -e 's/<[^>]*>/ /g;/</N;//ba' -e 's/$/\n/'

$wget -nv -O - "http://weather.weatherbug.com/Common/SearchResults.html?is_search=true&nav_section=1&loc_country=WORLD&zcode=z6286&loc=$args" 2>&1|sed -e :a -e 's/<[^>]*>/ /g;/</N;//ba' -e 's/$/\n/'|sed -e '1,/Your Forecast/d' -e '/7-Day Forecast/,$d'|sed -e '/^[ \t]*$/d' -e '/^$/N;/\n$/D' -e 's/[ \t]*more/... more\n/'|sed 's/°/Degrees/g'

exit 0

, Mike

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Sunday, September 28, 2008

The Mother Of All Urban Legend Chain Emails And A Little Something For Star Trek Fans

Hey there,

Happy Sunday! If my calculations are correct, it should be around 4pm your time. If it's any earlier, you either didn't stay up late enough last night, you need to get some sleep now that yesterday is today or you should get back in bed and ponder why you have so much energy and what you can do to avoid this sort of awkward situation in the future ;)

For today's joke post, we're posting a double-whammy of humour, since I got both jokes from Joshua's Internet Humor Archive, which isn't all written by Joshua, but is all grouped there, which saves me a HUGE amount of time wasting space typing all about attribution and so forth... I can't believe I blew my one chance to not have to explain that part of a joke post by writing about how lucky I was not to have to explain it. This is quickly becoming a longer explanation of the ineffable feeling of freedom from having misattribution hanging over my head like a vulture than any single post's "this joke was written by this guy," explanation ever has been before. I'll segue out of this confusing, self-referential maelstrom of meta-justifications by the only means left available to me. Anyway...

The first joke is a fake chain letter which combines some of the finest elements of the Internet's most memorable legend and/or chain emails. If it doesn't make you laugh, it will scare you to death. Especially since it really happened to a guy my room-mate used to hang out with in High School ;)

The second is a list of things you'll never ever see on Star Trek. It covers all the way up to "The Next Generation," but I don't officially recognize any show except the original. The "James T. Kirk" Star Trek is the only Star Trek there ever was for many, many reasons. They had no budget, which made for spectacular cardboard "name-your-props", the characters were pretty much all offensive stereo-types (the sleazy womanizing Lothario, the bitter old geezer, the sterile logician, the token black character, token Asian character, token Hispanic character and the fighting mad drunken Scotsman who somehow managed to keep that sh@@box U.S.S. Enterprise moving along even though he had no clue how to cap a pressure valve) and - if you were in the right "frame of mind" - it was one of the most hilarious and entertaining hours of television you could ever watch. To this day, it's still my favorite (so much so that I've refused to watch any of the others ;).

So, we'll knock 'em down one by one. Here we go!

Have a great weekend :)

The Über-Legend

I know this guy whose neighbor (a young man) was home recovering from having been served a rat in his bucket of Kentucky Fried Chicken. So anyway, one day he went to sleep and when he awoke he was in his bathtub and it was full of ice and he was sore all over. When he got out of the tub he realized that his kidneys had been stolen and he saw a note on his mirror that said "Call 911!" But he was afraid to use his phone because it was connected to his computer, and there was a virus on his computer that would destroy his hard drive if he opened an e-mail entitled "Join the crew!"

He knew it wasn't a hoax because he was a computer programmer who was working on software to save us from Armageddon when the year 2000 rolls around. His program will prevent a global disaster in which all the computers get together and distribute the $200 Nieman Marcus cookie recipe under the leadership of Bill Gates. (It's true - I read it all last week in a mass e-mail from Bill Gates Himself, who was also promising me a free Disney World vacation and $5,000 if I would forward the e-mail to everyone I know.)

The poor man then tried to call 911 from a pay phone to report his missing kidneys, but reaching into the coin-return slot he got jabbed with an HIV-infected needle around which was wrapped a note that said, "Welcome to the world of AIDS." Luckily he was only a few blocks from the hospital. It was the very same hospital where that little boy who is dying of cancer we've all heard is being treated. You know the little boy... the one whose last wish is for everyone in the world to send him an e-mail and the American Cancer Society has agreed to pay him a nickel for every e-mail he receives.

Kinda off the subject, I sent him two e-mails and one of them was a bunch of x's and o's in the shape of an angel (if you get it and forward it to twenty people you too will have good luck, but if you forward it to only ten people, you will only have only OK luck, and if you send it to less than ten people you will have "real" bad luck for seven years!).

So anyway the poor guy tried to drive himself to the hospital, but on the way he noticed another car driving along without his lights on. To be helpful, he flashed his lights at him and was promptly shot as part of a gang initiation.

Oh, and another thing, it's a little-known fact that the Y1K problem caused the Dark Ages.

I feel much better for having shared this with you. Thank you for taking the time. I want you to be safe. You know it!!!

Things That Will Never Happen on Star Trek
Author Unkown

1. The Enterprise runs into a mysterious energy field of a type it has encountered several times before.

2. The Enterprise goes to visit a remote outpost of scientists, who are all perfectly all right.


3. Some of the crew visit the holodeck, and it works properly.

4. The crew of the Enterprise discover a totally new life form, which later turns out to be a rather well-known old life form wearing a funny hat.


5. The crew of the Enterprise are struck by a mysterious plague, for which the only cure can be found in the well-stocked Enterprise sick-bay.


6.The Captain has to make a difficult decision about a less advanced people which is made a great deal easier by the Starfleet Prime Directive.


7. The Enterprise successfully ferries an alien VIP from one place to another without serious incident.


8. An enigmatic being composed of pure energy attempts to interface to the Enterprise's computer, only to find out that it has forgotten to bring the right leads.

9. A power surge on the Bridge is rapidly and correctly diagnosed as a faulty capacitor by the highly-trained and competent engineering staff.

10. The Enterprise is captured by a vastly superior alien intelligence which does not put them on trial.

11. The Enterprise is captured by a vastly inferior alien intelligence which they easily pacify by offering it some sweeties.

12. The Enterprise visits an earth-type planet called "Paradise" where everyone is happy all of the time. However, everything is soon revealed to be exactly what it seems.

13. A major Starfleet emergency breaks out near the Enterprise, but fortunately some other ships in the area are able to deal with it to everyone's satisfaction.

14. The Enterprise is involved in a bizarre time-warp experience which is in some way unconnected with the Late 20th Century.

15. Kirk (or Riker) falls in love with a woman on a planet he visits, and isn't tragically separated from her at the end of the episode.

16. Counselor Troi states something other than the blindingly obvious.

17. The warp engines start playing up a bit, but seem to sort themselves out after a while without any intervention from boy genius Wesley Crusher.

18. Wesley Crusher gets beaten up by his classmates for being a smarmy git, and consequently has a go at making some friends of his own age for a change.

19. Spock (or Data) is fired from his high-ranking position for not being able to understand the most basic nuances of about one in three sentences that anyone says to him.

, Mike

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Saturday, September 27, 2008

Converting PI To Binary - More Linux/Unix Humor

Hey there,

Another week has come and gone and it's Saturday again; just like last week. Lather, rinse, repeat ;)

This week's bit of Linux/Unix humour comes from deep within the pages at keithlynch.net. I'd highly recommend you go check it out. It's a very simple page (as in "no flashy graphics") but, once you get through half of his life story, I think you'll find that it's amazing he has a sense of humour left. Seriously; check out his page. He's been through the ringer. Nevertheless, he is also responsible for some pretty funny stuff. Check out the April Fool's section, for the bulk of the jokes on the site. All in all, it's a pretty interesting read. Partly serious, with some levity thrown in for good measure.

Below, I've reprinted one of my favorites that he's written. Since Pi, it's widely believed but not absolutely provable, never repeats itself on the right side of the decimal (although, another piece of Rye Humor (Rye spelled incorrectly on purpose ;) contests that it does repeat at the hyper-thousandth level), the joke below may not be all that far from the truth :)

So I leave you with this pun, that both begs to be written and, at the same time, put out of its misery: You can't have your Pi and eat it 10. <-- 10 is binary 2, like too... Man, that's a groaner. Almost as bad as "There are 10 kind of people in the world. Those that get binary and those that don't." Had to include it for completeness' sake. Sorry. Eat lots of charcoal to get the taste out of your mouth ;)

Enjoy, and have a great Saturday!

Converting pi to binary: Don't do it!

by Keith F. Lynch <kfl@keithlynch.net>

Warning: Do not calculate Pi in binary. It is conjectured that this number is normal, meaning that
it contains all finite bit strings.

If you compute it, you will be guilty of:

  • Copyright infringement (of all books, all short stories, all newspapers, all magazines, all web
    sites, all music, all movies, and all software, including the complete Windows source code)

  • Trademark infringement

  • Possession of child pornography

  • Espionage (unauthorized possession of top secret information)

  • Possession of DVD-cracking software

  • Possession of threats to the President

  • Possession of everyone's SSN, everyone's credit card numbers, everyone's PIN numbers, everyone's
    unlisted phone numbers, and everyone's passwords

  • Defaming Islam. Not technically illegal, but you'll have to go into hiding along with Salman

  • Defaming Scientology. Which is illegal -- just ask Keith Henson.

Also, your computer will contain all of the nastiest known computer viruses. In fact, all of the
nastiest possible computer viruses.

Some of the files on my PC are intensely personal, and I for one don't want you snooping through a
copy of them.

You might get away with computing just a few digits, but why risk it? There's no telling how far
into Pi you can go without finding the secret documents about the JFK assassination, a photograph of
your neighbor's six year old daughter doing the nasty with the family dog, or a complete copy of the
not-yet-released Pearl Harbor movie. So just don't do it.

The same warning applies to e, the square root of 2, Euler's constant, Phi, the cosine of any non-
zero algebraic number, and the vast majority of all other real numbers.

There's a reason why these numbers are always computed and shown in decimal, after all.

Keith F. Lynch - kfl@keithlynch.net - http://keithlynch.net/
I always welcome replies to my e-mail, postings, and web pages, but
unsolicited bulk e-mail sent to thousands of randomly collected
addresses is not acceptable, and I do complain to the spammer's ISP.

, Mike

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Friday, September 26, 2008

Making Solaris 10 Zones Recognize Veritas Raw Devices

Hey there,

Here's something I learned today that I never knew before! ...Actually, when you think about it, everything you ever learn is something you never knew before. Otherwise, you'd already know it, so there'd be no learning involved... Where am I? ;)

Today's short-post is a little trick concerning Solaris 10 and the Veritas File System/Volume Manager. I've been using Veritas with Solaris 10 for quite a while, so this was pretty interesting to find out. In the usual case, I could always set up Solaris zones and hand them Veritas File System partitions with no incident. Usually, something like this:

host # zonecfg -z myNewZone
<-- Here we skip past the create and "set zonepath" type stuff. We have more information on dealing with Solaris zones in this old post
zonecfg:myNewZone> add fs
zonecfg:myNewZone:fs> set dir=/my/directory
zonecfg:myNewZone:fs> set special=/myNewZone/my/directory
zonecfg:myNewZone:fs> set type=lofs
zonecfg:myNewZone:fs> end
zonecfg:myNewZone> commit
zonecfg:myNewZone> exit

and life was beautiful ;)

Out of sheer luck (or horrible misfortune), I've always ended up at places that (while they may make heavy use of Veritas Volume Manager, File System, Cluster Server, etc) either went with some combination of SunCluster, Sun Solaris, HP and Oracle or Informix. Now, there's nothing wrong with any of the possible combinations you could end up with here. In fact, there's - more exactly - nothing wrong, since all of them (well, not HP-UX) can work with Solaris 10 zones using raw devices.

If you're not familiar with basic disk access concepts in Unix, at a very basic level, disk in Unix is generally accessible via two standard gateways: Block devices or Raw/Character devices. Most of the time, you probably deal with the block device interface, as this allows buffered sector reads of the device (generally a hard drive/disk) and is generally faster for most operations. To demonstrate what I mean, if you wanted to read a few bytes from HardDrive1, you could read those few bytes from your block device (almost to the byte) , but you'd have to read the entire device in order to get those few bytes when accessing the information via the raw device interface.

Raw devices do have their benefits, of course, or they probably wouldn't be around any more. Like their name suggests, they allow direct (raw) access to the disk. This can make it possible for you to recover information if your disk becomes corrupted (for instance) but is mostly used by software that likes to manage disk on its own and not have to rely on the Operating System. Most database software uses disk in this manner. Oracle, for instance, creates its own "file system" and (although it will work on UFS or any other filesystem) they say it works a lot more efficiently when it can be allocated raw disk and allowed to manage it on its own. It's probably true to some degree. Does anyone really know? ;)

Anyway (back from that two-paragraph aside ;), what I learned today is that Solaris 10 Zones can't (or won't) deal with raw Veritas devices (WT_?) I also found out that there's a way to get around that issue (hence, the post - I still owe you that one that goes on and on forever and leads nowhere ;).

You can get around this deficiency by doing the following (assuming you've already taken care of the basics of setting up your Veritas Volumes and Filesystem) and have created your zone with a base filesystem, like above, and want to add a raw device.

1. Check out your raw device file in the appropriate directory:

host # ls -l /dev/vx/rdsk/myRawDisk
crw------- 1 root root 289, 45000 Aug 24 11:13 myRawDisk

and make note of the Major and Minor numbers of the device (289 and 45000 respectively, above). You'll need these to feed the mknod command you're going to hammer out next (we'll assume you set your zonepath equal to /myNewZone - oh, yeah, and it's also important that you're not in your "Zone" when you do this):

host # cd /myNewZone/dev
host # mknod myRawDisk c 289 45000

2. And (it seems too simple!!! ;) then you just need to (well... you should ;) log into your zone and check it out, with a simple "dd" test or something:

host # zlogin -l root myNewZone
zone # ls -l /dev/myRawDisk
crw------- 1 root root 289, 45000 Aug 24 11:13 /dev/myRawDisk
zone # dd if=/dev/myRawDisk of=/dev/null
4096+0 records in
4096+0 records out

And, there you go. That problem you never knew you had is solved! ;)


, Mike

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Thursday, September 25, 2008

Replacing System Boards On Sun Mx000 Series Servers

Hey there,

Shifting gears again, today we're going to take a look at doing some hardware maintenance on Sun's (or, technically, Fujitsu's) new Mx000 series servers. At this point, I think there are only 4 variants available; the M4000, M5000, M8000 and M9000. The numbers relatively equate to how much "better" one is than the other, with the highest number being the best (This is a subjective point, though. Depending on your needs an M4000 may be much better for you than an M9000)

I wanted to take a look at Dynamic Reconfiguration (DR) on the Mx000 series, and this seemed like as good an example as any. One thing to keep in mind is that you can't do this on Midrange servers since the replacement of that system board means replacing a motherboard unit (MBU), which can't be done on-the-fly. Why does this matter? I don't know; the M4000 - M9000 are all Enterprise servers that support the DR we're going to do. Just some trivia to keep it interesting ;)

The first thing you'll want to do is to log into the XSCF shell (akin to the Domain Shell or System Controller that we looked at in our old posts on working with Sun 6800 and 6900 Series Servers).

After that, you'll need to check the status of the domain with the "showdcl" command. You just need to pass it one option ( -d ) to identify the domain you want to check out (note the similarities to the 6800/6900 server DR operation. A lot of the commands are identical. That's the last time I'll refer back to those humungous machines. I promise :)

XSCF> showdcl -d 0
00 Running
00 00-0
01 01-0

Then, you'll need (or maybe just want) to check the status of the board that needs to be replaced. This can be done with the "showboards" (So familiar, but I promised not to go there anymore ;) command.

It's important to note that, if the board (itself) doesn't support the DR board deletion command, then - even if you're on an Enterprise system that supports DR - you won't be able to use DR to replace the board. Disregarding other, more eccentric, problems that rarely happen (outside the scope of this post), the thing to look for here is under the "Assignment" column. If a board shows as "Assigned", and meets all the other criteria too Byzantine and awkward to expound upon; this fits the definition of "doesn't support DR board deletion," mentioned above. You'll know for sure that it doesn't work when the command fails (which is another good reason to take an outage no matter how "resilient" your hardware uptime solution is). This is a very easy problem to fix, however. All you usually need to do is add one step before the next (to "unassign" the board) and it will magically support DR board deletion :) We'll group it in with the next step, just to keep things neat and tidy :)

XSCF> showboards 01-0
XSB DID(LSB) Assignment Pwr Conn Conf Test Fault
01-0 00(01) Assigned y y y Passed Normal

Now, we'll delete the system board using the following command

XSCF> deleteboard -c disconnect 01-0

Note, that if the board is Assigned, and doesn't support DR, you'll need to run this variant of the "deleteboard" command before the one above (to unassign it). Note, also, that it doesn't hurt to do this even if the board "does" support DR:

XSCF> deleteboard -c unassign 01-0

No sweat :)
Now, you'll want to check the status of "showboards" again (We're going to pretend that the "Assigned" status is OK, like it usually is, from now on)

XSCF> showboards 01-0
XSB DID(LSB) Assignment Pwr Conn Conf Test Fault
01-0 00(01) Assigned y n n Passed Normal

You'll notice here, now, that the Conn (Connected) and Conf (Configured) columns are showing n (no). This is good since you've deleted the board (logically) from the domain configuration.

Next, you'll need to get your hands dirty and physically replace the board. Actually, you probably won't if you've purchased Sun support (or wear a good sturdy pair of pleather gloves ;), since Sun won't let you touch it if you want them to come back out, at no additional charge, ever again should something actually be "wrong" with the replacement board they send you. We won't go into the boring details of hot-replacing the board, since it's (again) outside the scope of this increasingly long post, and should be performed by a Sun FE if you have no idea how to do it!

Once that's all over with, simply type

XSCF> replacefru

to complete the software part of replacing the "field replaceable unit," and check the status of the system board again. This time, also run:

XSCF> showboards -d 0

to ensure that all the system boards are still registered in the DCL (Domain Components List - Basically a list of all the boards that make up the domain - domain 0 in your case today)

If the system board configuration has changed (like the division type has changed from Uni to Quad for some reason... like you figured out a way to sneak in a system upgrade or something ;), you may need to run the "setupfru" command. You most likely won't, since you're replacing your board with another board that's exactly the same as the old board, except it works ;)

If the replacement system board isn't registered in the DCL, double check to make sure it hasn't assigned itself to a different domain (I've never seen this happen) using:

XSCF> showboards -v -a

In any event, since it's not in the DCL for your domain, you'll just need to add it back by running:

XSCF> setdcl -d 0 -l 01

The -d flag is for the domain and the -l is for the LSB number (listed in your "showboards" output).

Now, you should be on the road to all-the-way-good. But you should check and make sure, just in case:

XSCF> showboards 01-0
XSB DID(LSB) Assignment Pwr Conn Conf Test Fault
01-0 00(01) Assigned y n n Passed Normal

Now, you'll want to check the status of the domain (basically to determine if you want to reboot it or not, which you don't or you'll be directly contradicting everything DR stands for ;)

XSCF> showdcl -d 0
00 Running
00 00-0
01 01-0

and then, finally, you'll add the "new" board back to the domain and "configure" it, as well ("adding" will set the Conn column to y and "configuring" will set the Conf column to y).

XSCF> addboard -c configure -d 0 01-0

Then (and you're almost done - just being really cautious...) check the domain component list status again to make sure everything's cool:

XSCF> showdcl -d 0
00 Running
00 00-0
01 01-0

and run "showboards" on that new board to make sure everything is peachy ( The words Assigned, Passed, Normal and a few letter y's are excellent indicators that things are all well :)

XSCF> showboards 01-0
XSB DID(LSB) Assignment Pwr Conn Conf Test Fault
01-0 00(01) Assigned y y y Passed Normal

Congratulations! You've just completed your DR system board replacement on an M4000, 5000, 8000 or 9000. Now that you know how to do it, re-read these instructions and be amazed that it actually takes you longer to plod through this post than it does to do an actual board replacement ;)

For further perusal, enjoyment and possible confusion, check out The Official DR User's Guide For The Mx000 Series and The Mx000 Server Glossary. They're both fascinating reads that double as powerful sleep-aids ;)


, Mike

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, September 24, 2008

Using LinuxDefender Live To Rescue Your Windows NTFS Drive

Hey There,

Did I actually mention Windows and NTFS in the title of this post? It's a good thing I managed to squeeze a little "Linux" in there or I'd commit myself to the nearest convalescent home immediately ;) Actually, though, if the title is a bit off-putting, this post is all about freeware Linux and how to use it to fix NTFS partitions if you need to. Therefore, this post can be reduced to the following rough verbal equation: "You can use Linux to fix Windows" + "A Fixed Windows Box Equals Happy Windows Users" = "Linux is fantastic and makes it easy for you to fix your loved ones' computers when they're crashed and blue-screening, so they'll be able to leave you alone again as soon as possible ;)"

LinuxDefender Live has been around for a long time, and I do have my own set of duplicate boot CD's that I keep handy just in case my family's Windows box can't find the SYSTEM file and insists on a re-install (which, and this is probably true for everyone, just will not do... too much important stuff on the hard drive and no backups of any kind). For those of you who've never used LDL, it's a special distribution of the Knoppix Linux distro which was made specifically to run off of CD. The idea was, anywhere you went that had a PC, you could probably boot up a working Linux system and access the internet (or the information you carry around with you on a USB stick or some other portable HDD). Knoppix is an excellent solution that does what it advertises, and is actually still good enough to allow you to fix FAT filesystems and many other FS types. It's really the ideal combination of on-the-go functionality and miracle-OS-cure in one.

LinuxDefender Live steps it up a notch with their built in support for NTFS filesystem read/write mounting. LinuxDefender Live is more of a compilation of Knoppix and a lot of other programs that they could squeeze onto a boot CD, than a new version of Knoppix. Relative weights and merits aside, it's still great to have around when you want to fix a Windows NTFS filesystem problem quickly (Letting your Recommended Daily Allowance of Television get ripped and torn away day after day is a slippery slope. Granted, the worst that can happen is you might start reading more books, but it's a serious consideration, nonetheless ;)

The procedure for using LinuxDefender Live to fix your Windows NTFS problem is simple enough, with one catch that I can never seem to get around (except for one way, of course, or I wouldn't be writing about this. Some day, for a goof, I promise to write a post where I offer only questions and no answers, promo a few Get-Rich-Quick scams, ramble on about something else completely unrelated to the topic and end it all in mid sentence ;)

Considering that we've got an easy issue (like the SYSTEM file, in C:\WINNT\ is corrupted and you just need to be able to copy it off and replace it with SYSTEM.BAK, which you can't do without practically reinstalling your Windows OS, because it won't boot), these are the steps we'd take to fix it up and make everything better:

1. Pop open the CD tray while the system is still powered on. If that doesn't work, power it down and use the pinhole-method (sticking a pin in the hole in the front of the CD-ROM drive to manually eject it). Place the LinuxDefender Live CD in there and close it back up. Then power up or restart your machine as your situation dictates.

2. However your system allows you to, push the correct button (f1 or maybe f10/f12) when you power up the machine so that you can get to the system settings and make sure that your CD-ROM drive is listed as a Boot Device and is in the Boot Sequence (preferably first) so that our CD will be able to boot the system from the CD-ROM drive.

3. Power on the machine and kick back. Knoppix (I mean LDL ;) is unusually pleasant to watch, especially when you're in the company of people who don't know what the Hell any of the boot messages mean (as if I understood them all myself ;).

4. Once you're finished booting up and have either your desktop GUI (Gnome or KDE on my version) or the CLI (if you don't mind huge fonts and screen run-off) up and running, just mount the windows hard drive like you'd mount any Linux hard drive, on a temporary mount point. If you prefer to use the GUI, you can mount the disk just like in Windows, but, if you're like most unblessed people, you'll either get errors when you try to mount the disk (complaining about NTFS) or when you try to access it.

5. Here's where the troubleshooting starts. If you open a terminal window in your GUI (so you can keep a decent scrollback buffer and fit more words on the screen), you can try to mount the windows disk to a mountpoint of your own creation, using the specific flags to indicate that you want to mount the disk read/write as filesystem type NTFS. Fortunately, probably, you'll be good to go in a few steps.

6. The first thing to check at this point is that all of the requisite files are available. The NTFS read/write support is provided by the captive project, which requires that you have the ntoskrnl.exe and ntfs.sys files available. You can generally get these by just clicking the huge "Install NTFS Drivers" Icon on your desktop. Alternatively, you can run:

host # sudo captive-install-acquire

from the command line.

7. At this point, if you use wireless networking, you may be in a bit of a jam. I've tried to get around this by creating separate driver floppies but, when all is said and done, the hassle of having to connect an ethernet cable from your router to your PC or laptop is worth the time it takes. If you are already "hard-linked," you shouldn't have any issues with running the command. This problem simply arises from the fact that the LinuxDefender Live CD can't have those two essential Windows files on it for one reason or another (I'm sure it has something to do with licensing costs ;)

8. Even if you are connected to the network successfully, the command may not work the first time. If it cannot complete, it will let you know (in detail) what you need to do. This can be as easy as just typing another command line first, to (this happens often) making "real" disk space available to write the two Windows NTFS drivers to. I usually go with a floppy. My floppy actually already has the two files from 3 different versions of Windows on it, but that's just me ;)

9. When captive is finished installing, you can mount your Windows drive and access it like any regular Linux drive. Again, be sure to pass the options to mount (man mount) to indicate that you want to mount the disk read/write as NTFS. I'm incredibly paranoid, so I just cd directly into the WINNT directory (in this instance), copy off the bad SYSTEM file, copy the SYSTEM.bak file to SYSTEM, cd back to where I was and umount. Actually, if I was really bad, I'd just use absolute path names ;)

10. Now, you just exit or reboot and remove the LinuxDefender Live CD (or vice versa). Windows should come right up and run as poorly as it always has ;)

The biggest pain is that, since LinuxDefender Live is on CD (read-only media), you won't have the two NT drivers the next time you have to save your machine. If you know you're always going to be working on the same machine( and it's possible), carve out a tiny partition (not NTFS) and put them there so they're easily accessible the next time you have to go round and round. Otherwise, copy them to floppy before you quit. Then again, as a third option, you can just copy them to a floppy, or anywhere, after you've got Windows back up and running. I prefer to stick with the versions captive grabs, since they're more likely to be absolutely compatible (strange things change between Service Pack Versions :). Plus, if you have the files on floppy, you won't have to depend on an internet connection to get the files (or other more convoluted methods like booting off of a DOS disk and grabbing them from your hard drive that way... there's always a way!)

Hope this little walk-through was of some help to you. Tons of extra information can be found at the links to the products listed on this page. They're straight-up links to the distro's and company web pages (No sales involved since it's all free ;)

Next up in our "avoiding Windows support" series: How to train yourself to screen calls from friends and family ;)


, Mike

Thanks to an intrepid reader who found this "working" link to the last release of LinuxDefender Live!


Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Tuesday, September 23, 2008

Emergency Booting RedHat Linux With USB

Hey there,

This post is for any Linux admin out there, who uses RedHat (or any similar distro, really - the differences are generally minor) and works in a shop outfitted with the latest and greatest hardware to "drive the network." I'm not sure what that expression means, but I heard it in a meeting once and it seemed vaguely motivating ;) It's also the first post on using grub since our post on using grub to change the root password.

If you do use RedHat Linux and run it on hardware that's new or "newer," you probably have a couple of servers (or, possibly, even regular PC's) running RedHat that don't have those good old-fashioned PS/2 keyboards and mice (mouses ?? I'll puzzle over that for my remaining years). You may even, in some instances, find yourself in a situation where you're booting a server from a USB drive. You won't see that too often (especially not in enterprise systems), but you'll catch it here and there in development environments with bizarre quadruple-boot configurations. You may not, technically, support them, but (since you're the sys admin) you support them ;)

Basically, today, we're going to take a look at a way you can protect yourself against disaster in the former situation (No PS/2 Keyboard or mouse) and assume that your server is running RedHat Linux and only has USB keyboard and mouse connectors (Really, you don't need the mouse. That's for the GUI, and the GUI shouldn't have anything to do with an "emergency" recovery. If your GUI works, things can't be all that bad, and, if it doesn't, you won't be using it ;). We'll assume also, that you'll need them to work properly when you boot from a disaster situation. This, as most RedHat Linux users and/or admin's can attest to, is an issue directly related to the default OS setup. The issue is that the USB drivers generally (or, by default) get loaded up after the system has begun booting. Since the USB drivers are treated as ancillary, if your machine is completely screwed and you need to do some minimal CLI work (just interacting with "grub" at a high level), having a USB keyboard as your only option means you're screwed, too, unless you get creative and connect to the box via ALOM or Serial or can jerry-rig some other form of TTY connection to emulate keyboard interaction (You'll discover amazing things you never knew you knew when an important box goes down and the boss is building up to a Grand Mal seizure while he breathes down your neck ;)

Of course, the easiest thing to do (in almost any case, for any problem) is to use up that ounce of prevention (If you encounter resistance, you can always spin this as something that will "save the company a pound of cure" ;). And the easiest way to do that in RedHat Linux is to just create an alternate boot image that includes USB drivers (you saw this coming from a mile away, I know). And RedHat (as with most other distro's) makes this very easy to do. Given the version of RedHat Linux you're running, the switches/arguments you have to pass to the various commands may be slightly different, so I'm including multiple options in some cases where only a few (or one) are actually relevant. Check the man pages on your system to see which one works for you and/or if that option even exists. Who knows, you may have more options on your version than I do on mine, which could, potentially, be even better :)

The first thing to do, would be to figure out what USB drivers you have loaded on your system. A simple find command like this, should give you a general idea (use "lsmod" and grep that output if you want to do things "the right way," but you may miss a few things ;)

host # uname -r
host # find /lib/modules/`uname -r`/kernel/drivers/usb -type f -name "*usb*"

and this one, just because it's always nice to have (details on down the page):


The second thing to do would be to cozy up your "mkinitrd" command and get comfortable with it ;) On RHEL4 (RedHat Enterprise Linux Release 4), you can get away with the following command line to create a new bootable USB-friendly kernel boot image (Note that the release in these examples is RHEL3):

host # mkinitrd --with=ehci-hcd --with=uhci-hcd /boot/newinitrd-`uname -r`.img `uname -r`

If you're using an older, or different, version of RedHat, check out the options you have available to your "mkinitrd" command. Generally, you'll find at least one of these:

(completely contrary to this post, but, included for completeness for those folks who want to make sure they boot with NO USB support ;)

For instance, this would work on most servers:

host # mkinitrd --preload=ehci-hcd --preload=usb-storage --preload=scsi_mod --preload=sd_mod /boot/usbinitrd.img `uname -r`

From my little find investigation, I found that this would probably work well for my particular server:

host # mkinitrd --preload=ehci-hcd --preload=usb-storage --preload=usb-uhci --preload=usb-ohci --preload=scsi_mod --preload=sd_mod /boot/usbinitrd.img `uname -r`

And, then we'd add a new entry in /etc/grub.conf and give it a test the next time we can sneak in a clean boot up (For those last few examples of "mkinitrd" entries, of course, I'd be doing them all with `uname -r` instead of the actual value of that command, but you don't want to put those backticks in your /etc/grub.conf). A new entry, considering our command above, might look something like this:

title USB RHEL3 (2.6.9-1.667smp)
root (hd0,0)
kernel /vmlinuz-2.6.9-1.667smp
initrd /usbinitrd.img 2.6.9-1.667smp

...basically, if you're just created a different "initrd" image for your same kernel, just copy those lines, change the title slightly and alter the line that begins with "initrd" to point to the file you created with "mkinitrd." Simple :)

and, just in case you're curious about any of the modules we've suggested possibly adding to your new USB boot image, you can usually find out as much about them as you want to know using the modinfo command, like so (I'm leaving out the usb-storage, etc, ones since their name's pretty much say it all :)

host # modinfo ehci-hcd
filename: /lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/ehci-hcd.o
description: "2003-Jan-22 USB 2.0 'Enhanced' Host Controller (EHCI) Driver"
author: "David Brownell"
license: "GPL"
parm: log2_irq_thresh int, description "log2 IRQ latency, 1-64 microframes"

host # modinfo usb-uhci
filename: /lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/usb-uhci.o
description: "USB Universal Host Controller Interface driver"
author: "Georg Acher, Deti Fliegl, Thomas Sailer, Roman Weissgaerber"
license: "GPL"

host # modinfo usb-ohci
filename: /lib/modules/2.4.21-37.ELsmp/kernel/drivers/usb/host/usb-ohci.o
description: "USB OHCI Host Controller Driver"
author: "Roman Weissgaerber <weissg@vienna.at>, David Brownell"
license: "GPL"

Here's hoping you either never have to use this, or that you have it ready to roll if you ever do :)


, Mike

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Monday, September 22, 2008

Famous Quotations Script For Linux and Unix

Hey There,

Today's Linux/Unix bash shell script is yet another follow up to a whole cavalcade of scripts we've written to mine the knowledge on tap at reference.com, although this one is grabbing its material from QuotationsPage.com. If you missed any of the others, you can still find them in our older bash script posts to do encylopedia lookups, access the online Thesaurus, translate between different languages and, of course, the use the online dictionary. This time we're going to take a crack at grabbing some famous quotes for those moments when you just can't think of what someone else may have said about something ;)

This script is fairly easy to run and somewhat limited. The only real limitation is that it will only return the first pages of quotes from QuotationsPage.com. Hopefully that should be enough since they seem to put the best ones near the top of the list. You can run it with one or multiple arguments, like so:

host # ./quote.sh money


host # ./quote.sh love of money

Like all of our previous scripts, this script uses wget and sed to get the job done. As almost-usual, this script is a work in progress and could use some sprucing up, but, hopefully, it's good enough for you to enjoy :)

Below are a few screen shots of the script's output, demonstrating the output of the command examples show above.

Be sure to click on either of these pictures to see in them in Giganti-Size :)

Hope you enjoy this script, and can find some use for it. Any suggestions for improvement would be greatly appreciated.

As Amanda Cross put it: The point of quotations is that one can use another's words to be insulting

Here's to sounding more erudite than I am ;)


Creative Commons License

This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License


# quote.sh - Why not be witty? ;)
# 2008 - Mike Golvach - eggi@comcast.net
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

if [ $# -lt 1 ]
echo "Usage: $0 Your Word(s) To Find A Quote"
echo "May be two or more words separated by spaces"
echo "but only one definition per execution."
echo "Ex: $0 money"
echo "Ex: $0 love of money"
exit 1


if [ $# -gt 1 ]
args=`echo $args|sed 's/ /\+/g'`


$wget -nv -O - "http://www.quotationspage.com/search.php3?Search=$args&startsearch=Search&Author=&C=mgm&C=motivate&C=classic&C=coles&C=poorc&C=lindsly&C=net&C=devils&C=contrib" 2>&1|grep -i "No quotations found" >/dev/null 2>&1


if [ $anygood -eq 0 ]
args=`echo $args|sed 's/%20/\+/g'`
echo "No results found for $args"
exit 2

$wget -nv -O - "http://www.quotationspage.com/search.php3?Search=$args&startsearch=Search&Author=&C=mgm&C=motivate&C=classic&C=coles&C=poorc&C=lindsly&C=net&C=devils&C=contrib" 2>&1|sed -e :a -e 's/<[^>]*>/ /g;/</N;//ba' -e 's/$/\n/'|sed -e '1,/Results from/d' -e '/Results of search for */,$d'|sed 's/^[ \t]*//;s/[ \t]*$//'|sed '/^$/N;/\n$/D'|sed '/Pages: /,$d'|sed '/Results from/,$d'|sed 's/^ *//;s/ *$//;s/ \{1,\}/ /g'|sed 's/More quotations on: .*$//'

exit 0

Sunday, September 21, 2008

More Gutbusting RFC's - Linux and Unix Humor

Hey There,

Hope you're having a great Sunday :) Following up on a joke post we did last week regarding RFC 3093 For the Firewall Enhancement Protocol, I found a site that lists even more of them. In fact, if you visit WYAE you can find a pretty good listing of all the known joke RFC's that have been released (all the way through 2008). Who knew there were so many in-jokes out-there ;)

NOTE: Also check out FAQs.org as they've got their copywrite notice at the bottom of this page on the original site and we're all about giving credit where credit is due!

Below, I've included one of my favorites on The Etymology of "Foo" - RFC 3092. Thank goodness they released this one, as it validates so many examples of pseudo-code :)

Hope you enjoy it and have a nice relaxing weekend :)

RFC3092 - Etymology of "Foo"

Network Working Group D. Eastlake 3rd
Request for Comments: 3092 Motorola
Category: Informational C. Manros
E. Raymond
Open Source Initiative
1 April 2001

Etymology of "Foo"

Status of this Memo

This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this
memo is unlimited.

Copyright Notice

Copyright (C) The Internet Society (2001). All Rights Reserved.


Approximately 212 RFCs so far, starting with RFC 269, contain the
terms `foo', `bar', or `foobar' as metasyntactic variables without
any proper explanation or definition. This document rectifies that

Table of Contents

1. Introduction............................................1
2. Definition and Etymology................................2
3. Acronyms................................................5
Security Considerations...................................11
Authors' Addresses........................................13
Full Copyright Statement..................................14

1. Introduction

Approximately 212 RFCs, or about 7% of RFCs issued so far, starting
with [RFC269], contain the terms `foo', `bar', or `foobar' used as a
metasyntactic variable without any proper explanation or definition.
This may seem trivial, but a number of newcomers, especially if
English is not their native language, have had problems in
understanding the origin of those terms. This document rectifies
that deficiency.

Section 2 below describes the definition and etymology of these words
and Section 3 interprets them as acronyms.

As an Appendix, we include a table of RFC occurrences of these words
as metasyntactic variables.

2. Definition and Etymology

bar /bar/ n. [JARGON]

1. The second metasyntactic variable, after foo and before baz.
"Suppose we have two functions: FOO and BAR. FOO calls BAR...."

2. Often appended to foo to produce foobar.

foo /foo/

1. interj. Term of disgust.

2. Used very generally as a sample name for absolutely anything, esp.
programs and files (esp. scratch files).

3. First on the standard list of metasyntactic variables used in
syntax examples (bar, baz, qux, quux, corge, grault, garply,
waldo, fred, plugh, xyzzy, thud). [JARGON]

When used in connection with `bar' it is generally traced to the
WW II era Army slang acronym FUBAR (`Fucked Up Beyond All
Repair'), later modified to foobar. Early versions of the Jargon
File [JARGON] interpreted this change as a post-war
bowdlerization, but it now seems more likely that FUBAR was itself
a derivative of `foo' perhaps influenced by German `furchtbar'
(terrible) - `foobar' may actually have been the original form.

For, it seems, the word `foo' itself had an immediate prewar
history in comic strips and cartoons. In the 1938 Warner Brothers
cartoon directed by Robert Clampett, "The Daffy Doc", a very early
version of Daffy Duck holds up a sign saying "SILENCE IS FOO!"
`FOO' and `BAR' also occurred in Walt Kelly's "Pogo" strips. The
earliest documented uses were in the surrealist "Smokey Stover"
comic strip by Bill Holman about a fireman. This comic strip
appeared in various American comics including "Everybody's"
between about 1930 and 1952. It frequently included the word
"FOO" on license plates of cars, in nonsense sayings in the
background of some frames such as "He who foos last foos best" or
"Many smoke but foo men chew", and had Smokey say "Where there's
foo, there's fire". Bill Holman, the author of the strip, filled
it with odd jokes and personal contrivances, including other

nonsense phrases such as "Notary Sojac" and "1506 nix nix".
According to the Warner Brothers Cartoon Companion [WBCC] Holman
claimed to have found the word "foo" on the bottom of a Chinese
figurine. This is plausible; Chinese statuettes often have
apotropaic inscriptions, and this may have been the Chinese word
`fu' (sometimes transliterated `foo'), which can mean "happiness"
when spoken with the proper tone (the lion-dog guardians flanking
the steps of many Chinese restaurants are properly called "fu
dogs") [PERS]. English speakers' reception of Holman's `foo'
nonsense word was undoubtedly influenced by Yiddish `feh' and
English `fooey' and `fool'. [JARGON, FOLDOC]

Holman's strip featured a firetruck called the Foomobile that rode
on two wheels. The comic strip was tremendously popular in the
late 1930s, and legend has it that a manufacturer in Indiana even
produced an operable version of Holman's Foomobile. According to
the Encyclopedia of American Comics [EAC], `Foo' fever swept the
U.S., finding its way into popular songs and generating over 500
`Foo Clubs.' The fad left `foo' references embedded in popular
culture (including the couple of appearances in Warner Brothers
cartoons of 1938-39) but with their origins rapidly forgotten.

One place they are known to have remained live is in the U.S.
military during the WWII years. In 1944-45, the term `foo
fighters' [FF] was in use by radar operators for the kind of
mysterious or spurious trace that would later be called a UFO (the
older term resurfaced in popular American usage in 1995 via the
name of one of the better grunge-rock bands [BFF]). Informants
connected the term to the Smokey Stover strip [PERS].

The U.S. and British militaries frequently swapped slang terms
during the war. Period sources reported that `FOO' became a
semi-legendary subject of WWII British-army graffiti more or less
equivalent to the American Kilroy [WORDS]. Where British troops
went, the graffito "FOO was here" or something similar showed up.
Several slang dictionaries aver that FOO probably came from
Forward Observation Officer, but this (like the contemporaneous
"FUBAR") was probably a backronym [JARGON]. Forty years later,
Paul Dickson's excellent book "Words" [WORDS] traced "Foo" to an
unspecified British naval magazine in 1946, quoting as follows:

"Mr. Foo is a mysterious Second World War product, gifted with
bitter omniscience and sarcasm."

Earlier versions of the Jargon File suggested the possibility that
hacker usage actually sprang from "FOO, Lampoons and Parody", the
title of a comic book first issued in September 1958, a joint

project of Charles and Robert Crumb. Though Robert Crumb (then in
his mid-teens) later became one of the most important and
influential artists in underground comics, this venture was hardly
a success; indeed, the brothers later burned most of the existing
copies in disgust. The title FOO was featured in large letters on
the front cover. However, very few copies of this comic actually
circulated, and students of Crumb's `oeuvre' have established that
this title was a reference to the earlier Smokey Stover comics.
The Crumbs may also have been influenced by a short-lived Canadian
parody magazine named `Foo' published in 1951-52. [JARGON]

An old-time member reports that in the 1959 "Dictionary of the
TMRC Language", compiled at TMRC (the Tech Model Railroad Club at
MIT) there was an entry for Foo. The current on-line version, in
which "Foo" is the only word coded to appear red, has the
following [TMRC]:

Foo: The sacred syllable (FOO MANI PADME HUM); to be spoken
only when under obligation to commune with the Deity. Our first
obligation is to keep the Foo Counters turning.

This definition used Bill Holman's nonsense word, then only two
decades old and demonstrably still live in popular culture and
slang, to make a "ha ha only serious" analogy with esoteric
Tibetan Buddhism. Today's hackers would find it difficult to
resist elaborating a joke like that, and it is not likely 1959's
were any less susceptible. [JARGON]

4. [EF] Prince Foo was the last ruler of Pheebor and owner of the
Phee Helm, about 400 years before the reign of Entharion. When
Foo was beheaded by someone he called an "eastern fop" from
Borphee, the glorious age of Pheebor ended, and Borphee rose to
the prominence it now enjoys.

5. [OED] A 13th-16th century usage for the devil or any other enemy.
The earliest citation it gives is from the year 1366, Chaucer A B
C (84): "Lat not our alder foo [devil] make his bobance [boast]".
Chaucer's "Foo" is probably related to modern English "foe".

6. Rare species of dog.

A spitz-type dog discovered to exist after having long been
considered extinct, the Chinese Foo Dog, or Sacred Dog of
Sinkiang, may have originated through a crossing of Northern
European hunting dogs and the ancient Chow Chow from Mongolia or
be the missing link between the Chinese Wolf and the Chow Chow.
It probably derives its name from foochow, of the kind or style

prevalent in Foochow, of or from the city of Foochow (now Minhow)
in southeast China. [DOG]

foobar n.

[JARGON] A widely used metasyntactic variable; see foo for
etymology. Probably originally propagated through DECsystem
manuals by Digital Equipment Corporation (DEC) in 1960s and early
1970s; confirmed sightings there go back to 1972. Hackers do not
generally use this to mean FUBAR in either the slang or jargon
sense. It has been plausibly suggested that "foobar" spread among
early computer engineers partly because of FUBAR and partly
because "foo bar" parses in electronics techspeak as an inverted
foo signal.

foo-fighter n.

World War II term for Unidentified Flying Objects (UFOs) noted by
both German and British military. See [FF] and entry above for

3. Acronyms

The following information is derived primarily from the compilations
at University Cork College <http://www.ucc.ie/acronyms> and Acronym
Finder <http://www.AcronymFinder.com> generally filtered for computer


Generic file extension which is not meant to imply anything about
the file type.


Base Address Register

Buffer Address Register


Forward Observation Observer.

FOO Of Oberlin. An organization whose name is a recursive
acronym. Motto: The FOO, the Proud, the FOO. See

File Open for Output. An NFILE error code [RFC1037].


FTP Operation Over Big Address Records [RFC1639]. (Particularly
appropriate given that the first RFC to use "foo", [RFC269], was
also about file transfer.)


Failed UniBus Address Register - in a VAX, from Digital Equipment
Corporation Engineering.

Fucked Up Beyond All Recognition/Repair - From US Military in
World War II. Sometimes sanitized to "Fouled Up ...".

FUBARD - Past tense of FUBAR.


Below is a table of RFC occurrences of these words as metasyntactic
variables. (This excludes other uses that are reasonably clear like
"vertical bar" or "bar BoF".) Many of these uses are for example
domain names. That usage may decrease with the specification in [RFC
2606] of a Best Current Practice for example domain names.

| RFC# | bar | foo | foo.bar | fubar | # |
| | | | foobar | | |
| 269 | X | X | | | 1 |
| 441 | X | X | | | 2 |
| 614 | | X | | | 3 |
| 686 | | X | | | 4 |
| 691 | | X | | | 5 |
| 733 | X | X | | | 6 |
| 742 | | X | | | 7 |
| 743 | X | X | | | 8 |
| 756 | | X | | | 9 |
| 765 | X | X | | | 10 |
| 772 | X | X | | X | 11 |
| 775 | | | X | | 12 |
| 780 | X | X | | X | 13 |
| 788 | X | X | | | 14 |
| 810 | X | X | X | | 15 |
| 819 | | X | | | 16 |
| 821 | X | X | | | 17 |
| 822 | X | X | | | 18 |
| 882 | X | X | | | 19 |
| 883 | | X | | | 20 |
| 897 | X | X | | | 21 |
| 913 | | X | | | 22 |
| 921 | X | X | | | 23 |
| 934 | | X | | | 24 |
| 952 | X | X | X | | 25 |
| 959 | | | X | | 26 |
| 976 | | | X | | 27 |
| 977 | | X | X | | 28 |
| 987 | | | X | | 29 |
| 1013 | | X | | | 30 |
| 1033 | X | X | | | 31 |
| 1035 | | X | | | 32 |
| 1037 | | X | | | 33 |
| 1056 | X | X | X | | 34 |
| 1068 | | X | | | 35 |
| 1137 | | | X | | 36 |

| 1138 | | X | X | | 37 |
| 1148 | | X | X | | 38 |
| 1173 | | | X | | 39 |
| 1176 | | | X | | 40 |
| 1186 | | X | | | 41 |
| 1194 | | X | | | 42 |
| 1196 | | X | | | 43 |
| 1203 | | X | X | | 44 |
| 1288 | | X | | | 45 |
| 1291 | | X | | | 46 |
| 1309 | | X | | | 47 |
| 1327 | | X | X | | 48 |
| 1341 | X | X | X | | 49 |
| 1343 | | X | X | | 50 |
| 1344 | | X | | | 51 |
| 1348 | | | X | | 52 |
| 1386 | | X | | | 53 |
| 1408 | | X | | | 54 |
| 1411 | | X | | | 55 |
| 1412 | | X | | | 56 |
| 1459 | X | X | X | X | 57 |
| 1480 | | X | | | 58 |
| 1505 | | X | | | 59 |
| 1519 | | X | | | 60 |
| 1521 | X | X | | | 61 |
| 1523 | | X | | | 62 |
| 1524 | | X | X | | 63 |
| 1526 | X | X | | | 64 |
| 1535 | X | X | X | | 65 |
| 1536 | X | | X | | 66 |
| 1537 | | X | X | | 67 |
| 1563 | | X | | | 68 |
| 1564 | | | X | | 69 |
| 1572 | | X | | | 70 |
| 1573 | | X | | | 71 |
| 1622 | | X | | | 72 |
| 1635 | | | X | | 73 |
| 1636 | | X | X | | 74 |
| 1642 | | X | | | 75 |
| 1645 | | | X | | 76 |
| 1649 | | X | | | 77 |
| 1664 | | | X | | 78 |
| 1681 | | | X | | 79 |
| 1697 | | X | | | 80 |
| 1716 | | X | | | 81 |
| 1718 | | X | | | 82 |
| 1730 | X | X | X | | 83 |
| 1734 | | | X | | 84 |

| 1738 | | X | | | 85 |
| 1783 | | | X | | 86 |
| 1784 | | | X | | 87 |
| 1786 | X | X | | | 88 |
| 1813 | X | X | | | 89 |
| 1835 | | X | X | | 90 |
| 1856 | | | X | | 91 |
| 1861 | | | X | | 92 |
| 1866 | | X | | | 93 |
| 1894 | | | X | | 94 |
| 1896 | | X | | | 95 |
| 1898 | | X | | | 96 |
| 1913 | | X | X | | 97 |
| 1945 | X | X | | | 98 |
| 1985 | | X | X | | 99 |
| 2015 | X | X | | | 100 |
| 2017 | | X | | | 101 |
| 2033 | X | X | | | 102 |
| 2045 | | | X | | 103 |
| 2046 | X | X | | | 104 |
| 2049 | X | X | | | 105 |
| 2055 | | X | | | 106 |
| 2060 | X | X | X | | 107 |
| 2065 | | X | | | 108 |
| 2068 | | | X | | 109 |
| 2071 | | X | | | 110 |
| 2088 | | | X | | 111 |
| 2109 | | X | | | 112 |
| 2110 | | X | X | | 113 |
| 2111 | X | X | X | | 114 |
| 2141 | | X | | | 115 |
| 2150 | | X | | | 116 |
| 2152 | | X | | | 117 |
| 2156 | | X | X | | 118 |
| 2163 | | | X | | 119 |
| 2167 | | | X | | 120 |
| 2168 | | | X | | 121 |
| 2169 | | | X | | 122 |
| 2180 | X | X | | | 123 |
| 2193 | X | X | | | 124 |
| 2224 | | X | | | 125 |
| 2227 | X | X | | | 126 |
| 2233 | | X | | | 127 |
| 2234 | X | X | X | | 128 |
| 2243 | | X | | | 129 |
| 2255 | | X | X | | 130 |
| 2280 | X | X | | | 131 |
| 2295 | | X | | | 132 |

| 2302 | | X | | | 133 |
| 2311 | X | | | | 134 |
| 2326 | X | X | X | | 135 |
| 2342 | | X | | | 136 |
| 2348 | | | X | | 137 |
| 2349 | | | X | | 138 |
| 2359 | | | X | | 139 |
| 2369 | X | X | X | | 140 |
| 2378 | | X | | | 141 |
| 2384 | | | X | | 142 |
| 2392 | X | X | X | | 143 |
| 2396 | | | X | | 144 |
| 2401 | | | X | | 145 |
| 2407 | | | X | | 146 |
| 2421 | | X | | | 147 |
| 2425 | | | X | | 148 |
| 2434 | | X | | | 149 |
| 2446 | | X | X | | 150 |
| 2447 | X | X | | | 151 |
| 2458 | | X | X | | 152 |
| 2459 | | | X | | 153 |
| 2476 | | X | | | 154 |
| 2483 | X | X | | | 155 |
| 2486 | | X | | | 156 |
| 2505 | X | X | | | 157 |
| 2518 | X | X | X | | 158 |
| 2535 | | X | | | 159 |
| 2538 | | X | | | 160 |
| 2543 | X | X | X | | 161 |
| 2554 | | | X | | 162 |
| 2557 | | X | X | | 163 |
| 2565 | | X | X | | 164 |
| 2569 | X | X | | | 165 |
| 2593 | X | X | | | 166 |
| 2595 | | X | | | 167 |
| 2608 | | X | | | 168 |
| 2609 | | X | | | 169 |
| 2616 | X | X | X | | 170 |
| 2622 | X | X | | | 171 |
| 2626 | | X | | | 172 |
| 2633 | X | | | | 173 |
| 2640 | | X | X | | 174 |
| 2645 | | | X | | 175 |
| 2650 | X | | | | 176 |
| 2659 | | | X | | 177 |
| 2673 | | X | X | | 178 |
| 2693 | | X | | | 179 |
| 2704 | X | X | | | 180 |

| 2705 | X | X | | | 181 |
| 2717 | | X | X | | 182 |
| 2725 | X | X | | | 183 |
| 2731 | X | X | X | | 184 |
| 2732 | | X | | | 185 |
| 2782 | | X | X | | 186 |
| 2803 | | X | | | 187 |
| 2806 | | X | | | 188 |
| 2812 | X | X | X | X | 189 |
| 2818 | X | X | | | 190 |
| 2828 | | X | X | | 191 |
| 2830 | X | | | | 192 |
| 2831 | X | X | X | | 193 |
| 2839 | | X | | | 194 |
| 2846 | X | X | | | 195 |
| 2853 | | X | | | 196 |
| 2863 | | X | | | 197 |
| 2910 | | X | X | | 198 |
| 2912 | | X | X | | 199 |
| 2915 | | X | | | 200 |
| 2926 | | | X | | 201 |
| 2942 | | X | | | 202 |
| 2965 | | X | | | 203 |
| 2967 | X | X | X | | 204 |
| 2970 | | X | | | 205 |
| 2993 | X | X | | | 206 |
| 3010 | X | X | | | 207 |
| 3023 | | X | | | 208 |
| 3028 | | X | | | 209 |
| 3075 | X | X | | | 210 |
| 3080 | | X | | | 211 |
| 3092 | X | X | X | X | 212 |
| RFC# | bar | foo | foo.bar | fubar | # |
| | | | foobar | | |

Security Considerations

Security issues are not discussed in this memo.


[BFF] "Best of Foo Fighters: Signature Licks", Troy Stetina, Foo
Fighters, October 2000, Hal Leonard Publishing Corporation,
ISBN 063401470.

[DOG] <http://www.rarebreed.com/breeds/foo/foo.html>.

[EAC] "Encyclopedia of American Comics", Ron Goulart, 1990, Facts
on File.

[EF] "Encyclopedia Frobozzica",

[FF] Foo Fighters - "The Rainbow Conspiracy", Brad Steiger,
Sherry Hansen Steiger, December 1998, Kensington Publishing
Corp., ISBN 1575663635. - Computer UFO Network
<http://www.cufon.org> particularly

[FOLDOC] "Free On-Line Dictionary Of Computing",

[JARGON] The Jargon File. See <http://www.jargon.org>. Last
printed as "The New Hacker's Dictionary", Eric S. Raymond,
3rd Edition, MIT Press, ISBN 0-262-68092-0, 1996.

[OED] "The Oxford English Dictionary", J. A. Simpson, 1989,
Oxford University Press, ISBN 0198611862.

[PERS] Personal communications.

[RFC269] Brodie, H., "Some Experience with File Transfer", RFC 269,
December 1971.

[RFC1037] Greenberg, B. and S. Keene, "NFILE - A File Access
Protocol", RFC 1037, December 1987.

[RFC1639] Piscitello, D., "FTP Operation Over Big Address Records
(FOOBAR)", RFC 1639, June 1994.

[RFC2606] Eastlake, D. and A. Panitz, "Reserved Top Level DNS Names",
BCP 32, RFC 2606, June 1999.

[TMRC] The Tech Model Railroad Club (The Model Railroad Club of
the Massachusetts Institute of Technology) Dictionary,

[WBCC] "Warner Brothers Cartoon Companion",

[WORDS] "Words", Paul Dickson, ISBN 0-440-52260-7, Dell, 1982.

Authors' Addresses

The authors of this document are:

Donald E. Eastlake 3rd
155 Beaver Street
Milford, MA 01757 USA

Phone: +1 508-261-5434 (w)
+1 508-634-2066 (h)
Fax: +1 508-261-4777 (w)
EMail: Donald.Eastlake@motorola.com

Carl-Uno Manros
Xerox Corporation
701 Aviation Blvd.
El Segundo, CA 90245 USA

Phone: +1 310-333-8273
Fax: +1 310-333-5514
EMail: manros@cp10.es.xerox.com

Eric S. Raymond
Open Source Initiative
6 Karen Drive
Malvern, PA 19355

Phone: +1 610-296-5718
EMail: esr@thyrsus.com

Full Copyright Statement

Copyright (C) The Internet Society (2001). All Rights Reserved.

This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than

The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an


Funding for the RFC Editor function is currently provided by the
Internet Society.

, Mike

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Excellent Web Hosting At An Affordable Price!
These Are The Guys I Use To Host My Online Presence!