Even More Madcap Manpages - Linux and Unix Laughs

Hey there,

Here's hoping you're enjoying your Sunday and wrapping up a nice 4 day weekend (Or, at the very least, a nice 2 day weekend... sometimes being a sysadmin can be hard on your social life ;)

Today, while out trudging through the wasteland of that thingy they call the Internet, looking far and wide for stuff to make me chuckle, I ran across this awesome collection of fake, and funny, man pages. The clip in today's post is from the New Man Pages page, which you can check out for links to even more humour and an unexpurgated version of this page. As I've almost grown sick of explaining, I had to change a few words to keep this blog in Google's good graces (or, more importantly, to keep from getting an adult warning page slapped on top of a collection of Unix and Linux scripts and tips). I'm still not sure why they're so hard-line on this topic (i.e. you're either "bad" or you're "good"). It doesn't seem like it would take a quantum leap forward in implementation, thinking or coding to only tag certain pages with an "adult" flag while leaving the rest of the site accessible to the general public. I'm sure there are folks out there who find almost everything I write offensive to some degree, but... well, enough with the frustration. As Lenny Bruce used to say, "These are the jokes, folks!" This is good-time Sunday. Yay :)

I, personally, don't have any issues with my employers today, but I thought the bosskill manpage was hilarious :)

Hope you enjoy these funny man pages as much as I still do :)

Cheers,

---

Man Pages

---

alt.sysadmin.recovery man page distribution.
Found on Eric L. Pederson page.

Section 1 - User Commands

lart       Luser Attitude Readjustment Tool
slave      a semi-interactive command for the dirty work
sysadmin   responsible for everything imaginable that may or may not have to do with the system you're using.
           Contraction of "system" and "administrator"
think      you don't have to think, the computer can think for you
whack      mangle requests to a printer or damage a printer

Section 2 - System Calls

people     fetch a structure containing all ttys, whose owner behaves like a human

Section 3 - C Library Functions

chastise   library function to punish users

Section 5 - File Formats

normality  definition of what types of normalities different users may have

Section 8 - Maintenance Commands

bosskill   send a signal to your boss, or terminate your boss
ctluser    control lusers
guru       System Administration
knife      tools to improve network performance via SNIP
luser      process to control the clueless induhviduals who (mis)use computer systems,
           peripheral devices and system administrators. Word play on "loser" and "user"
nuke       launch nuclear weapons at mapped USENET sites
pmsd       Periodically Manic System Daemon. Manages the bizzare and sometimes unexplainable
           behavior exhibited by computers

---

Other man pages

rtfm       read the frickin manual
uubp       Unix-to-Unix beer protocol
dream      suspend execution for an interval while executing random code in memory
sex        have sex
baby       create new process from two parent processes

---

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

The Fedora Girlfriend Test - More Linux and Unix Humor

Hey There,

This Saturday's over-sized slice of, possibly funny, Linux humour comes to use via ITKnowledgeExchange. It's actually a review of an original article entitled "Linux: The Girlfriend Test" from TechRadar.com.

The test itself is fairly self-explanatory, but it's the account of one Linux writer's test of the user-friendliness of Fedora 9 using the perfect test subject: His girlfriend :) As the review of the original article notes (and this is one of the funnier parts, in my opinion), no mention was made of how the writer fared in the "boyfriend shopping test" ;)

Hope you enjoy this, and please check out the original article (a very interesting and funny read) that's referenced in the article below and a few paragraphs above.

Enjoy your weekend :)

---

Fedora gets mixed review in "girlfriend test"

Everyone agrees that the Linux desktop has a lot of work ahead to transform itself from a techie obsession to the intuitive, user-friendly desktop of a Macintosh or Windows machine. Even Mark Shuttleworth, founder of the popular Ubuntu Linux desktop, blogged about the shortcoming in a recent column and vowed to close the gap.

In the meantime, however, U.K.-based TechRadar.com recently published Linux: The Girlfriend Test, and a tough test it was indeed. The goal: to find out if a first-time Linux user, presumably a female college student, could accomplish nine familiar Windows tasks using Fedora 9, the community version of Red Hat.

Here are the tasks and how she fared on each one:

1. Bookmark a website in Firefox. No problem.

2. Write and print a letter in OpenOffice. The first part was easy but the letter wouldn’t print and no error message appeared with a reason or resolution.

3. Rip a CD. Task accomplished. But Fedora failed to identify all the output options.

4. Send an instant message. After several unsuccessful attempts, she succeeded by going to msn.com and inputting user data from her Windows Live Messenger account.

5. Create a pie chart in OpenOffice. No problem.

6. Transfer the ripped CD to her iPod. Attempt failed because of a protocol problem. Again, no error message appeared to identify or fix the difficulty.

7. Move a photo of her head onto a photo of her boyfriend’s body using Photoshop. Easy.

8. Watch a video on YouTube. Failed because Firefox was unable to install Flash player due to a malformed file. There was no work-around explanation.

9. Make an international phone call using Skype. Application installation was successful but audio playback problems prevented communication.

The writer concluded that Linux needs to do more with wizards and pop-up instructions to help new users without a technical background successfully transition from Windows or Macintosh to the Linux desktop.

I agree. But I have to say it was a pretty tough test, and the writer never mentioned how he fared in the “Boyfriend Shopping Test” that was his part of the bargain. Inquiring readers want to know.

---

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Dealing With "Argument list too long" Errors on Linux and Unix

Hey There,

Here's a question that gets asked a lot (and, consequently, answered a lot ;) on the boards. How do you go about dealing with a situation in which you're trying to take care of some business on your Linux or Unix box and you get stopped with the "Argument list too long" error message? It's probably happened to all of us at some point, but it's fairly simple to avoid, and in more than one way. I'm almost positive, even though I know this stuff, that I'll get that error message again at some point in the future. My mind is a terrible thing ;)

The way I try to keep on top of avoiding that error (which, admittedly, isn't all that terrible, assuming its not a predicating factor in a script - like, you don't have any action is your script that might do something destructive if you don't check for errors) is a basic two-prong approach. First, I'll try to figure out what kind of limitations my system can stand. Secondly, I try to stick to some basic methods that guarantee I won't run into the issue no matter what my system's maximum argument length is. Of course, the second part obviates the first part, but I'm a curious guy (take that whatever way you want ;)

1. How can you figure out your systems command line argument length limitations?

The good news is that there a number of ways (and, most probably, much more than I'm going to list here today):

On almost all Unix and Linux systems you can use getconf to find out the answer very simply, like so (all examples today from OpenBSD 4.4-stable):

host # getconf ARG_MAX
262144

RedHat, SUSE, and BSD Linux (just to name a few) also support figuring this out using the sysctl command (for a more in-depth look at this command, check out our old post on using sysctl to work with kernel tunables on Linux:

host # sysctl kern.argmax
kern.argmax=262144

2. How can you avoid ever getting this error in the first place (not to be used as a substitute for doing error checking in your scripts to "make doubly sure" :)

This boils down to, very basically, using three methods: find, xargs or using your shell's basic looping constructs.

For the purposes of all of our examples below we'll assume that you've done something like:

host # ls *

in a directory, with 564,321 files in it, and have received the titular error.

xargs and find work because, used properly, they can reduce that huge list of file names to 564,321 separate lists of one filename each (that's putting it a little simply, but I'm trying to paint a picture here. And, believe me, painting is a lot harder to do while you're typing than it looks ;)

The find command can be used with the -exec option to iterate through the elements of your ls output, like so:

host # find /that/directory -exec ls {} \;

The xargs command will pretty much do the same thing. Some common wisdom (which doesn't work) is to do this:

host # echo /that/directory/*|xargs ls

however this will still give you the error, unless your version of echo can subvert the OS globbing rules and bypass the ARG_MAX without error.

The only real reason to use xargs to get around this issue is if you run into a situation more complicated than find's -exec flag can deal with. This situation doesn't apply, in that respect, so the following solution is, admittedly redundant and overcomplicated:

find /that/directory|xargs ls

Thirdly, you can use shell while looping to get around the error, again using find to make sure you don't get dinged by the maximum argument length error:

find /that/directory|while read x
do
 ls $x
done

Yee-hah :)

Of course, this discourse has been limited to the basics of the shell and one particular situation. If you use Perl, tcl or any other language (under any number of situations) your options will multiply. As with anything else in Linux or Unix, the possible solutions are, to a large degree, limited only to your imagination.

With these little fires lit, go on out there and (to use a dangerous colloquialism) "burn that mother down" ;)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Happy Thanksgiving - Unix/Linux ASCII Art

Happy Turkey Day!

Here's hoping you're all enjoying a day off of work, have nothing whatsoever to do and fully plan on doing it ;)

Below, a little ASCII art script that should run on any Linux or Unix distro. I don't know who created the original ASCII picture and I give credit to whomever it was. If this is your turkey, send me an email and I'll be happy to credit you with the catch!

Happy Thanksgiving and take care :)

Click below to see this gobbler super-sized ;)



...and click here to download the script just in case Blogspot completely destroys all the spacing or any special characters. We put the entire picture in one echo statement in a celebration of whatever anniversary of security through obfuscation this is.

May you be blessed enough to realize all you have to be thankful for.

#!/bin/bash

#
# htg.sh - Happy Thanksgiving :)
#

echo -e "\n                              ,+*^^*+___+++_\n                         ,*^^^^              )\n                      _+*                     ^**+_\n                    +^       _ _++*+_+++_,         )\n        _+^^*+_    (     ,+*^ ^          \+_        )\n       {       )  (    ,(    ,_+--+--,      ^)      ^\\n      { (@)    } f   ,(  ,+-^ __*_*_  ^^\_   ^\       )\n     {:;-/    (_+*-+^^^^^+*+*<_ _++_)_    )    )      /\n    ( /  (    (        ,___    ^*+_+* )   <    <      \\n     U _/     )    *--<  ) ^\-----++__)   )    )       )\n      (      )  _(^)^^))  )  )\^^^^^))^*+/    /       /\n    (      /  (_))_^)) )  )  ))^^^^^))^^^)__/     +^^\n   (     ,/    (^))^))  )  ) ))^^^^^^^))^^)       _)\n    *+__+*       (_))^)  ) ) ))^^^^^^))^^^^^)____*^\n    \             \_)^)_)) ))^^^^^^^^^^))^^^^)\n     (_             ^\__^^^^^^^^^^^^))^^^^^^^)\n       ^\___            ^\__^^^^^^))^^^^^^^^)\\n            ^^^^^\uuu/^^\uuu/^^^^\^\^\^\^\^\^\^\\n               ___) >____) >___   ^\_\_\_\_\_\_\)\n              ^^^//\\_^^//\\_^       ^(\_\_\_\)\n                ^^^ ^^ ^^^ ^^\n"
banner happy;banner thanks;banner giving

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Patching Your Solaris Server From A Network Mount

Hey again,

Today we're going to take a quick look at how to patch your Solaris server from the network (not version-specific, unless you're still using 2.4 in which case I don't remember how to help you, although I'm almost positive the answer is similar ;)

The first question you might be asking yourself is "Why in the world would I ever want, or even need, to do this?" Since I'm a somewhat-honest guy who writes from the hip, I'll tell you that, depending upon where you work and what kind of setup you have there, you may never ever need to do this. If (worst case) you can get a console connection to your server (which you'll need for this method, anyway) and have onsite staff that can pop a cd in a working drive, you probably wouldn't even want to do this. So, the simple answer is: Why not? If the unlikely situation arises (and, if you're reading this in the distant future, it just might have ;), you'll know how to do it. If you set yourself up correctly, it's just one more way to get around having to come into the office in the middle of the night :)

The first thing you'll want to do is to set up a boot server or a jumpstart server. Since that topic is fairly lengthy to go over, and we've already gone over it at length, please check out this old post on begin and finish script variable definitions for Solaris' jumpstart. It's actually the last part of a series of posts on Jumpstart, but, since it wasn't posted in backward chronological order, the last link (this one) connects back to all the previous posts. You can also get a feel for some of your extended options (on both the jumpstart server and the jumpstart client) in these posts on jumpstart booting.

The second thing to do would be to ensure that you have the patch bundle you need either on the local machine, on a partition that would normally get mounted in single user mode (if you can, put them in a subdirectory of / - but only if you have the space. /usr and /var "should" always be good candidates for placement, as well, but I've done this a couple times under conditions so crippling it took me months before I could walk upright again ;). Having your patches available when you get into single user mode (with no networking) is of vital importance when you want to patch your server. Did that sound redundant? Good; because it was ;)

Now, assuming you've prepped your Jumpstart server, and have it all set up to service your client (the machine we're going to patch), it's a simple matter to get things started on your end (We'll assume that you have either a console, ALOM or other such connection to your patch client server so that you won't get disconnected from it when it goes to PROM and can interact with it at any run level or phase). The following should suffice to get the whole shebang (not to be confused with shebang ;) started:

host # id
uid=0(root) gid=0(root)
host # init 0
ok> boot net -s

And we're off. Once your server has successfully booted into single user mode on your Jumpstart, or Boot, server (see above for links to more information regarding those details), you can begin going about your business and, hopefully, finishing well before the time you promised you'd be done.

First (or is it third?), after you've booted into single user mode (no networking) and verified that you can access your patches, be sure to stop any and all unnecessary services. Most of them should be gone, but a simple "ps -ef" and "df -k" to verify that no one snuck in an init script or mount at a run level you wouldn't normally suspect might save you some headache later, and it'll probably only take about 2 or 3 minutes at most, even if you have to "terminate" a process or two. Also be sure to mount any partitions that need to be mounted if, for some reason, they don't get mounted automatically. You should get errors, when you attempt to patch, that will indicate what you need to fix if something is wrong in this regard, but it's usually best just to have things ready and not take the chance that Sun forgot to put an error-check in a script somewhere!

And the rest (much like this post) is academic. The general procedure (assuming you've downloaded a recommended patch bundle from Sun and have it in the directory /patches (or something similar), would go something like this (although possibly slightly differently if you're running Solaris 10 with zones, which case you should check out that hyperlink a few words back:

host # cd /patches
host # ./install_cluster
... < wait and watch, or go away and come back - just make sure you see all the error messages. Most will probably be inconsequential if you're using a generic recommended patch cluster ("software that the patch patches isn't installed" or "greater revision of patch already installed" are technically errors, but can be safely ignored.
host # reboot -- -r

And, that should be that :) Remember, this is actually a much longer post, if you need to go back to the referenced posts and get details for all of the parts. I thought it would be nice to have everything in one place (or linked to one place) finally, with regards to this topic, so I wrote this patch-work wire-frame mockery of a sham ;)

Have fun patching and be sure to do it, at the very least, more often than the Cicadas hatch ;)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Quick And Easy Local Filesystem Troubleshooting For SUSE Linux

Hey There,

Today we're going to take a look at some quick and easy ways to determine if you have a problem with your local filesystem on SUSE Linux (tested on 8.x and 9.x). Of course, we're assuming that you have some sort of an i/o wait issue and the users are blaming it on the local disk. While it's not always the case (i/o wait can occur because of CPU, memory and even network latency), it never hurts to be able to put out a fire when you need to. And, when the mob's pounding on your door with lit torches, that analogue is never more appropriate ;)

Just as in previous "quick troubleshooting" posts, like the week before last's on basic VCS troubleshooting, we'll be running through this with quick bullets. This won't be too in-depth, but it should cover the basics.

1. Figure out where you are and what OS you're on:

Generally, something as simple as:

host # uname -a

will get you the info you need. For instance, with SUSE Linux (and most others), you'll get output like:

Linux “hostname” kernel-version blah..blah Date Architecture... yada yada

The kernel version in that string is your best indicator. Generally, a kernel-version starting with 2.4.x will be fore SUSE 8.x and 2.6.x will be for SUSE 9.x. Of course, also avail yourselves of the, possibly available, /etc/release, /etc/issue, /etc/motd, /etc/issue.net files and others like them. It's important that you know what you're working with when you get started. Even if it doesn't matter now, it might later :)

2. Figure out how many local disks and/or volume groups you have active and running on your system:

Determine your server model, number of disks and volume groups. Since you're on SUSE, you may as well use the "hwinfo" command. I never know how much I'm going to need to know about the system when I first tackle a problem, so I'll generally dump it all into a file and then extract it from there as needed. See our really old post for a script that Lists out hardware information on SUSE Linux in a more pleasant format:

host # hwinfo >/var/tmp/hwinfo.out
host # grep system.product /var/tmp/hwinfo.out
system.product = 'ProLiant DL380 G4'

Now, I know what I'm working with. If this specific of a grep doesn't work for you, try "grep -i product" - you'll get a lot more information than you need, but your machine's model and number will be in there and much easier to find than if you looked through the entire output file.

Then, go ahead and check out /proc/partitions. This will give you the layout of your disk:

host # /proc # cat /proc/partitions
major minor  #blocks  name

 104     0   35561280 cciss/c0d0
 104     1     265041 cciss/c0d0p1
 104     2   35294805 cciss/c0d0p2
 104    16   35561280 cciss/c0d1
 104    17   35559846 cciss/c0d1p1
 253     0    6291456 dm-0
 253     1    6291456 dm-1
 253     2    2097152 dm-2
 253     3    6291456 dm-3
 253     4   10485760 dm-4
 253     5    3145728 dm-5
 253     6    2097152 dm-6

"cciss/c0d0" and "cciss/c0d1" show you that you have two disks (most probably mirrored, which we can infer from the dm-x output). Depending upon how your local disk is managed, you may see lines that indicate, clearly, that LVM is being used to manage the disk (because the lines contain hints like "lvma," "lvmb" and so forth ;)

  58     0    6291456 lvma 0 0 0 0 0 0 0 0 0 0 0
  58     1    6291456 lvmb 0 0 0 0 0 0 0 0 0 0 0

3. Check out your local filesystems and fix anything you find that's broken:

Although it's boring, and manual, it's a good idea do take the output of:

host # df -l

and compare that with the contents of your /etc/fstab. This will clear up any obvious errors like mounts that are supposed to be up but aren't or mounts that aren't supposed to up that are, etc... You can whittle down your output from /etc/fstab to show (mostly) only local filesystems by doing a reverse grep on the colon character (:) - This is generally found in remote mounts and almost never found in local filesystem listings.

host # grep -v ":" /etc/fstab

4. Keep hammering away at the obvious:

Check the USED% column in the output of your "df -l" command. If any filesystems are at 100%, some cleanup is in order. It may seem silly, but usually the simplest problems get missed when one too many managers begin breathing down your neck ;) Also, check the inodes column and ensure that those aren't all being used up either.

Mount any filesystems that are supposed to be mounted but aren't, and unmount any filesystems that are mounted but (according to /etc/fstab) shouldn't be). Someone will complain about the latter at some point (almost guaranteed), which will put you in a perfect position to request that it either be put in the /etc/fstab file or not mounted at all.

You're most likely to have an issue here with mounting the unmounted filesystem that's supposed to be mounted. If you try to mount and get an error that indicates the mountpoint can't be found in /etc/fstab or /etc/mnttab, the mount probably isn't listed in /etc/fstab or there is an issue with the syntax of that particular line (could even be a "ghost" control character). You should also check to make sure the mount point being referenced actually exists, although you should get an entirely different (and very self-explanatory) error message in the event that you have that problem.

If you still can't mount, after correcting any of these errors (of course, you could always avoid the previous step and mount from the command line using logical device names instead of paths from /etc/vfstab, but it's always nice to know that what you fix will probably stay fixed for a while ;), you may need to "fix" the disk. This will range in complexity from the very simple to the moderately un-simple ;) The simple (Note: If you're running ReiserFS, use reiserfsck instead of plain fsck for all the following examples. I'm just trying to save myself some typing):

host # umount /uselessFileSystem
host # fsck -y /uselessFileSystem
....
host # mount /

which, you may note, would be impossible to do (or, I should say, I'd highly recommend you DON'T do) on used-and-mounted filesystems or any special filesystems, like root "/" - In cases like that, if you need to fsck the filesystem, you should optimally do it when booted up off of a cdrom or, at the very least, in single user mode (although you still run a risk if you run fsck against a mounted root filesystem).

For the moderately un-simple, we'll assume a "managed file system," like one under LVM control. In this case you could check a volume that refuses to mount (assuming you tried most of the other stuff above and it didn't do you any good) by first scanning all of them (just in case):

host # vgscan
Reading all physical volumes.  This may take a while...
  Found volume group "usvol" using metadata type lvm2
  Found volume group "themvol" using metadata type lvm2

If "usvol" (or any of them) is showing up as inactive, or is completely missing from your output, you can try the following:

host # vgchange –a y

to use the brute-force method of trying to activate all volume groups that are either missing or inactive. If this command gives you errors, or it doesn't and vgscan still gives you errors, you most likely have a hardware related problem. Time to walk over to the server room and check out the situation more closely. Look for amber lights on most servers. I've yet to work on one where "green" meant trouble ;)

If doing the above sorts you out and fixes you up, you just need to scan for logical volumes within the volume group, like so:

host # lvscan 
ACTIVE            '/dev/usvol/usfs02' [32.00 GB] inherit
....

And (is this starting to sound familiar or am I just repeating myself ;), if this gives you errors, try:

host # lvchange –a y

If the logical volume throws you into an error loop, or it doesn't complain but a repeated run of "lvscan" fails, you've got a problem outside the scope of this post. But, at least you know pretty much where it is!

If you manage to make it through the logical volume scan, and everything seems okay, you just need to remount the filesystem as you normally would. Of course, that could also fail... (Does the misery never end? ;)

At that point, give fsck (or reiserfsck) another shot and, if it doesn't do any good, you'll have to dig deeper and look at possible filesystem corruption so awful you may as well restore the server from a backup or server image (ghost).

And, that's that! Hopefully it wasn't too much or too little, and helps you out in one way or another :)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Cabletv.sh For Linux or Unix - Final Blog Update

Hey There,

IMPORTANT NOTE: As noted in the title, the poll we put up last week showed that a majority of folks who participated in the blog's poll (72%) would rather that this script's development be moved to my SourceForge page and become a separate project which would mean that we won't be doing any more follow-up posts to this one. I must say that I whole-heartedly agree. The first benefit will be that we won't be continually putting up the same material every Monday. The second, and perhaps even better, benefit will be that I'll be able to work on the script on an entirely different schedule. Once it's freed from the daily-post deadline, I'll be able to completely overhaul it and make it much better than it is (and a whole lot more quickly). I want everyone who's written in with suggestions for improvement to know that I've read all your emails (so far) and am very appreciative of your constructive criticism and creative approaches toward making the script into a fuller and more robust "program." I apologize if I haven't gotten back to you yet (for those of you whom I haven't gotten back to ;), but I promise you that I will. BTW (This is how goofy I am), I just realized, as I was doing some backtracking, that I've posted an entry every day since October 23rd 2006. It's a miracle I still have a job and the love of my family. I'm just wondering how I manage to spend time with them... oh, yeah. I don't sleep ;)

LAST NOTE ON THIS: At this point, I'll need to setup my SourceForge page to include the project and, as soon as it's up, I'll be sure to put a link to it on the blog (In the upper right hand corner, with the policies and "send me a comment" links). I'll probably note it in that day's blog listing, too, but you'll know it's ready as soon as you see that link appear. It should be up sometime this week.

For this week's Monday Linux/Unix bash shell script continuation, we're finishing up the blog version of our cabletv.sh script from last week. If you liked that one, and want to check out other versions, please revisit last week's cabletv.sh script, and that page will link back to all the other versions of the script hosted on this blog. If you can see the Blogger search bar (wrapped around the site and not really part of it), searching for "cable tv" should bring up all of them on one or two pages. As usual, skip the next paragraph if you've read it before. It lists out all our other "bringing the web to your CLI" scripts we've cranked out to date. Starting next week, there will finally be more original content!

Our previous web-based Bash scripts, in backward chronological order, include our posts on accessing Wikipedia, accessing the Farmer's Almanac, accessing the International Dictionary, checking out the world's weather, spewing out famous quotations on pretty much any subject, doing encyclopedia lookups, accessing the online Thesaurus, translating between different languages and, of course, using the online dictionary.

This week's improvements are limited to one (I guess that would just make it a singular "improvement" ;). I received several suggestions about reliance on awk's strftime, which helped us get over the in-sequential time problem, but added some minutes to the output. For the record, this fix isn't included in this version, as problems with formatting need to be completely reworked, but, to answer the question, the quicker way to do this without awk is with Gnu date (and it "must" be Gnu date, as, for instance, Solaris' version of date doesn't allow for the same type of formatting). So, in a version I'm not putting up here because it isn't ready yet, we've changed:

strftime("%x - %I:%M %p", TheTimeFromZap2It/1000)) }'

to:

date -d @$((TheTimeFromZap2It/1000)) "+%x - %I:%M %p"

which is much faster!

Our only improvement for this week's sendoff is that we've switched from using wget to using straight-up telnet, since more people have telnet on their machines (by default) than wget (If you recall, the original reason we went with wget was because we didn't want folks to have to run lynx and/or html2text for no reason). This also results in a slight performance improvement (about 30 seconds), but I'll take it :) So, our URL calls that used to look like this:

wget -nv -O - "http://tvlistings.zap2it.com/tvlistings/ZBChooseProvider.do?method=getProviders&zipcode=${zip_code}"

are now set up this way, using telnet to port 80 and passing an HTTP/1.1 request (Zap2it bumps you if you use HTTP 1.0!), like so:

(echo "GET /tvlistings/ZBChooseProvider.do?method=getProviders&zipcode=${zip_code} HTTP/1.1";echo "Host: tvlistings.zap2it.com";echo;sleep 2)|telnet tvlistings.zap2it.com 80

It seems more convoluted (and, technically, I suppose it is), but it works faster. I'm assuming because telnet is so ingrained in Linux and Unix and it saves us the time of loading up another external program that (although it may be "better suited" to the task) can do much more than we need it to and comes with a bit of extra baggage, as would be expected.

Until next time. I look forward to writing something new for you :)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# cabletv.sh - Get your local regular and HD Tv listings
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
# TODO switch to enhanced getopt to deal with variable argument switches
# TODO add config file
# TODO check for bad provider code
# TODO rewrite unix to "pretty" time conversion to speed up returns
# ------------------------------
# DONE switch "wget" to "telnet" for more accessibility and a slight speedup
# DONE End Display Time For Previous day when start time is in zero hour
# DONE check for bad zip code
# DONE added optional time increments - up to 3 hours - for display
# DONE tightened up code using indirect variable references
# DONE date may now be entered in quotes numeric or alpha - e.g. 11/9/08 11/09/2008 "Nov 9 2008" "November 9th 2008" etc
# DONE time may now be entered as 10pm 10 (with am/pm defaulting to system time) 10:00am 23:00 etc
# DONE added error checking for invalid date and time entry - stop before querying online guide
# DONE fixed issue with 00:xx time causing script failure
# DONE added extra error checking
# DONE added better formatting of lineup for easier readability - Thanks to Michael Seeley for the suggestion!
# DONE You have entered an invalid ZIP or postal code
# DONE am to pm or pm to am ends up not in chronological order
# DONE 12 to 1 a/p ends up not in chronological order
#

function usage()
{
 echo
        echo "Usage: $0 ...all options are optional [-h for help]"
 echo "[-z USZipCode] [-p LocalCableOrDishProviderID]"
        echo "[-t Time] [-d date] [-n to leave out HD channel info]"
        echo "[-s ListingTimeSpan 1 for just now, 2 for an additional half"
 echo "hour, 3, 4, 5, 6, in half hour increments, and 7 for 3 hours"
 echo "...defaults to 7 since this is the online listing default]"
 echo
        exit 1
}

while getopts t:z:hs:nd:p: option
do
 case $option in
 t)
  opt_time="$OPTARG"
  ;;
 d)
  opt_date="$OPTARG"
  ;;
 z)
  opt_zip="$OPTARG"
  ;;
 n)
  opt_nohd=1
  ;;
 p)
  opt_provider="$OPTARG"
  ;;
 s)
  opt_timespan="$OPTARG"
  ;;
 h)
  usage
  ;;
 *)
  usage
  ;;
 esac
done

## FEEL FREE TO EDIT HERE - If the path's are wrong, or you use less, rather than more, you can change that here.
telnet=/usr/bin/telnet
pager=/usr/bin/more
## OK, THAT'S ENOUGH OF THAT ;)

if [ ! "$opt_date" ]
then
 nicedate=`date "+%m/%d/%y"`
else
 nicedate=$opt_date
fi

if [ ! $opt_timespan ]
then
 opt_timespan=7
elif [ $opt_timespan -lt 1 -o $opt_timespan -gt 7 ]
then
 usage
fi

if [ ! $opt_time ]
then
 hour=`date "+%I"`
 minute=`date "+%M"`
 ampm=`date "+%p"`
 if [ $hour -lt 12 ]
 then
  if [ $minute -lt 30 ]
  then
   nicetime="${hour}:00 AM"
  else
   nicetime="${hour}:30 AM"
  fi
 else
  if [ $minute -lt 30 ]
  then
   nicetime="${hour}:00 PM"
  else
   nicetime="${hour}:30 PM"
  fi
 fi
 if [ $hour == "00" ]
 then
  cli_time=${hour}:${minute}
 else
  cli_time=${hour}:${minute}${ampm}
 fi
else
 cli_time=$opt_time
fi

date_ok=`date -d "$cli_time" +%s 2>&1`
if [[ "$date_ok" =~ "invalid date" ]]
then
 echo
 echo "INVALID TIME ENTERED: $opt_time"
 echo "Be sure to include full am or pm"
 echo "and do not add am or pm to 24 clock entries"
 echo
 usage
fi

ampm=`echo $opt_time|egrep 'a|p|m' 2>&1`
result2=$?
if [ $result2 -eq 0 ]
then
 date_ok2=`echo $opt_time|egrep 'a|p'|grep m 2>&1`
 result3=$?
 if [[ $result3 -ne 0 ]]
 then
  echo
  echo "INVALID TIME ENTERED: $opt_time"
  echo "Be sure to include full am or pm"
  echo "and do not add am or pm to 24 clock entries"
  echo
  usage
 fi
fi

cli_time2=`date -d "$cli_time" +%s`
let fromtime1=$cli_time2*1000
let fromtime2=$fromtime1+1800000
let fromtime3=$fromtime2+1800000
let fromtime4=$fromtime3+1800000
let fromtime5=$fromtime4+1800000
let fromtime6=$fromtime5+1800000
let fromtime7=$fromtime6+1800000
if [ ! $opt_timespan ]
then
 opt_timespan=7
fi

timespan_relay="$fromtime1"
timespan_counter=2
timespan_limit=$opt_timespan

while [ $timespan_limit -gt 1 ] 
do
 nextup_time=$(eval "echo \$$(echo fromtime${timespan_counter})")
 timespan_relay="$timespan_relay $nextup_time"
 let timespan_counter=$timespan_counter+1
 let timespan_limit=$timespan_limit-1
done

from_timespan=$(eval "echo \$$(echo fromtime${opt_timespan})")
cli_time_end=`echo $from_timespan|awk '{printf("%20s",strftime("%I:%M %p - %x", $0 / 1000))}'`

if [ ! $opt_nohd ]
then
 opt_nohd=0
fi

if [ ! $opt_zip ]
then
 echo -n "Enter Zip Code: "
 read zip_code
else
 zip_code=$opt_zip
fi

(echo "GET /tvlistings/ZBChooseProvider.do?method=getProviders&zipcode=${zip_code} HTTP/1.1";echo "Host: tvlistings.zap2it.com";echo;sleep 2)|telnet tvlistings.zap2it.com 80 2>&1|sed -e :a -e 's/<[^>]*>/ /g;/</N;//ba' |sed -e '/^[ \t]*$/d' 2>&1|grep "You have entered an invalid ZIP or postal code" >/dev/null 2>&1
zip_ok=$?

if [ $zip_ok -eq 0 ]
then
 echo
 echo "INVALID ZIP CODE ENTERED: $opt_zip"
 echo
 usage
fi

if [ ! $opt_provider ]
then
 (echo "GET /tvlistings/ZBChooseProvider.do?method=getProviders&zipcode=${zip_code} HTTP/1.1";echo "Host: tvlistings.zap2it.com";echo;sleep 2)|telnet tvlistings.zap2it.com 80 2>&1|sed 's/<.*&lineupId=\([^&]*\)\">/\1  \c/'|sed -e :a -e 's/<[^>]*>/ /g;/</N;//ba' |sed -e '/^[ \t]*$/d' |sed -e '1,/Choose Your Provider/{x;$!d;x;q;};x;$G' -e '/document.write/,$d' -e "s/'/'/" -e 's/&/\&/' -e 's/^[ \t]*//;s/[ \t]*$//'|sed '/\([A-Za-z0-9_][A-Za-z0-9_]*:[-X]\)/ {
  N
  s/ *\n/ \t/g
 }'
 echo "Enter Provider ID [e.g. \"IL57303:-\" \"4DTV:-\" \"IL12561:X]\""
 read tv_provider
else
 tv_provider=$opt_provider
fi

echo
echo "Television Listings for $nicedate from $cli_time to $cli_time_end"
echo

last=1

if [ $opt_nohd -eq 1 ]
then
 for x in $timespan_relay
 do
  (echo "GET /tvlistings/ZCGrid.do?fromTimeInMillis=${x}&method=decideFwdForLineup&zipcode=${zip_code}&setMyPreference=false&lineupId=${tv_provider} HTTP/1.1";echo "Host: tvlistings.zap2it.com";echo;sleep 2)|telnet tvlistings.zap2it.com 80 2>&1|sed 's/<.*&sch=\([^&]*\)&[^>]*>/\1 /'|sed -e :a -e 's/<[^>]*>/ /g;/</N;//ba' |sed -e '/^[ \t]*$/d' |sed -e '1,/Forgotten password/d' -e '/isFavoritesAvailable/,$d'|sed -e '/[ECMP][SD]T/,+6d' -e "s/'/'/" -e 's/&/\&/' -e '/zc.getAdFrame/d' -e 's/^[ \t]*//;s/[ \t]*$//'| sed -n '/^[0-9][0-9]*$/,+2p'|sed 's/^\([0-9][0-9]*\)$/\n\1/'|sed '/^[0-9][0-9]*$/ {
  N
  N
  s/ *\n/\t/g
  }'|awk '{ if ( $1 ~ /^[0-9]/ ) {printf("%-4s %-8s ", $1,$2);for (i=4;i<NF+1;i++) {printf("%s ", $i)};printf("*** Start - %s", $3);print ""}}'
 done|sort -t'-' -k1,1n -k2,2n | uniq|while read one two;do if [ $one -eq $last ]; then echo "$one $two"|awk '{aline=$NF;$NF="";printf("%s %s\n", $0, strftime("%x - %I:%M %p", aline/1000)) }'|while read one two three;do echo "     $three";done;last=$one;else echo "$one $two"|awk '{aline=$NF;$NF="";printf("%s %s\n", $0, strftime("%x - %I:%M %p", aline/1000))}'|while read one two three;do echo " $one $two";echo "     $three";done;last=$one;fi;done|$pager
else
 for x in $timespan_relay
 do
  (echo "GET /tvlistings/ZCGrid.do?fromTimeInMillis=${x}&method=decideFwdForLineup&zipcode=${zip_code}&setMyPreference=false&lineupId=${tv_provider} HTTP/1.1";echo "Host: tvlistings.zap2it.com";echo;sleep 2)|telnet tvlistings.zap2it.com 80 2>&1|sed 's/<.*&sch=\([^&]*\)&[^>]*>/\1 /'|sed -e :a -e 's/<[^>]*>/ /g;/</N;//ba' |sed -e '/^[ \t]*$/d' |sed -e '1,/Forgotten password/d' -e '/isFavoritesAvailable/,$d'|sed -e '/[ECMP][SD]T/,+6d' -e "s/'/'/" -e 's/&/\&/' -e '/zc.getAdFrame/d' -e 's/^[ \t]*//;s/[ \t]*$//'| sed -n '/^[0-9][0-9]*\.*[0-9]*$/,+2p'|sed -e 's/^\([0-9][0-9]*\.*[0-9]*\)$/\n\1/'|sed '/^[0-9][0-9]*\.*[0-9]*$/ {
  N
  N
  s/ *\n/\t/g
  }'|awk '{ if ( $1 ~ /^[0-9]/ ) {printf("%-4s %-8s ", $1,$2);for (i=4;i<NF+1;i++) {printf("%s ", $i)};printf("*** Start - %s", $3);print ""}}'
 done|sort -t'-' -k1,1n -k2,2n | uniq|while read one two three;do three_one=`echo "$three"|sed -e 's/.*\*\*\* Start - //' -e 's/^[ \t]*//;s/[ \t]*$//'`; let three_two=$three_one/1000; three_three=`echo "$three"|sed 's/^\(.*\*\*\* Start - \).*/\1/'`;three_four=`echo $three_two|awk '{printf("%s", strftime("%x - %I:%M %p", $0)) }'`;if [ $one -eq $last ];then echo "     $three_three $three_four";last=$one;else echo "$one $two";echo "     $three_three $three_four";last=$one;fi;done|$pager
fi

exit 0

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Linux Users Unite ...And Riot: More Funny Fake News

Hey there,

Happy Sunday :) Hope yours is as restful as I expect mine to be. Since I write these posts at least a day in advance I can only say with a slight degree of certainty that I'm feeling better than I ever have and it's been a good long while since I've been quite so relaxed... possibly ;)

In any event, as always, we'll keep the exposition short for our Sunday humor post and turn you on to this bit of fake news from PacketStorm in their Linux and Unix humor section. Check out the rest of site for more laughs and some good security tips!

Enjoy and peace :)

---

Linux Advocates Turn Violent, Go On Rampage

Until yesterday, protesters had kept a quiet, good-natured presence
outside of the Microsoft's new office in downtown Seattle, handing out
free CD's of computer software, and even giving cute little stuffed penguins-their
mascot- to the children of passers-by. "They're such NICE young men," said
78 year old Mildred Matthews, walking her dog nearby. "They would pet my
dog, and we'd talk about Art Bell, Deep Space Nine and the X Files. I think
they just got tired of being repressed and kept down by The Man," said
this great-grandmother when interviewed this morning.

Tear gas fired at crowd.

But these demonstrators, whose numbers have escalated in recent weeks, did in fact turn violent yesterday, clashing with police in the worst street violence in this city's history. "I don't know what set them off," said Microsoft Security Chief Lester Richard.

"They'd been gathering forces for some time, so I guess I should have expected
it. Mr. Gates sent a representative outside to meet with their leaders,
and suddenly there were hundreds of them screaming and breaking out windows.
Once I was safely upstairs, I called police. They plastered their penguin
posters all over our lobby and ate up all my jelly beans" said the 47 year
old, with visible frustration. "They even placed a little stuffed penguin
on my chair!"

Linux nerd wears gasmask hastily fashioned from a garbage bag, work goggles, a pocket protector- and using the insides of a nutty bar as a filter.

By 1:30PM, the streets had turned to utter chaos as police arrived with riot gear, firing teargas at the crowd. One teargas canister, in a bizarre twist of fate, landed in the doorway of the city's Convention Center, where an NRA-sponsored survivalist organization was holding their annual Gun & Knife Show. In the chaos that followed, no one seemed to know who was fighting whom. Making matters worse, a large contingent from a national motorcycle gang just

happened to be passing through town, and was caught in the melee. Injuries
and damage were widespread, with no official estimates yet available. Police
restored order sometime just before midnight.

"Death to Graphical User Interfaces!" screams a penguinhead as he is arrested.

A Peaceful History Shattered By all accounts, the Linux advocates had until this time, been quite peaceful. Jose Suarez, who owns a small convenience store nearby, had only good things to say about them. "Before they even arrived in town, their leaders had contacted me by phone, warning me that I would need to stock very heavily on Mountain Dew, Doctor Pepper, and Jolt Cola. They sure do like that stuff. Geez." Seattle Mayor PauI Schell expressed initial reservations about the influx of penguinheads. "Obviously, when I found out that we were going to have tens of

thousands of outsiders coming to our city, sleeping in their cars, on the
sidewalk- I was very concerned. We hadn't dealt with crowds this size since
the last Grateful Dead show here-
and I'll tell you, these computer nerds are touchy. At least the

hippies stayed mellow till we sent police to the lot to bust their skulls with nightsticks." "I went down and met with them, and they were really nice for a bunch of computer-geeks. They had some sort of cellular hookup or something, and were on the Internet conspiring with people all over the world. They had cellphones, moving lifesize penguin holograms- the penguin is their symbol- and fax machines, some sort of cable satellite TV so they could watch Star Trek- they impressed me as

Protesters begin burning crates of MSN CD's.

good citizens. I'll admit, they fooled me.. When they got tired of playing
soccer or throwing frisbees, they'd get out little homebuilt robots- cute
little things, that play what they call 'fooball.' They said they believe
that technology, if kept in the hands of the people, will be a source of
very good social change. Up until now, our only big problem was with their
all-night laser-tag games all over downtown. Deep down, though, I knew
it wouldn't take much to push them over the edge."

Linux: Technology of The People, or Systems by Satan?

"PeaceFrog" attempts to sell a linux button to the officer who confiscated his soccer ball. He was quickly arrested for vending without a license.

The young enthusiasts of technology that were picketing Microsoft are advocates of Linux, a computer "operating system" that competes with Microsoft's "Windows" software. Linux is licensed under the Free Software Foundation's General Public License and is literally free. Linux is a form of an operating system called Unix, which has become rather controversial in recent years. "Linux is by the people, for the people!" exclaims activist

Richard Stanley Dupp Jr.. "It's the result of people who want technology
to work for you, not for some rich billionaire whose goal is control
of the market and the destruction of the human spirit of creativity." He
then said something unintelligible, causing those nearby to burst out laughing.

"That was Klingon for 'it only took one brick to make that window
drop!'" explains a protester, who gave his name has PenguinBreeze.
PenguinBreeze said he was covering his travel expenses by selling stuffed
penguins, Linux stickers, and "fat way-kind veggie burritos."

The billionaire he's referring to is Microsoft CEO Bill Gates, the Henry
Ford of computer technology. It was Gates who built an empire by putting
PC's in nearly every American home. The Linux activists say that you cannot
purchase a computer without Windows, and seek a refund on the addition
to a PC's price. Neither manufacturers nor Microsoft will give the requested
refund.

But not everyone agrees that this challenge to the Microsoft empire
is based in good intentions. For most of the past week, the picketers have
been the focus of another organization, Seattle Citizens Unix Mobilization,
a group that opposes Unix-oriented systems for religious reasons.

"These cyberpunks are the hardcore subversives- the people who abuse
our technology, hack into our missile sites, question our clergy, defile
our morality, trivialize our art, mock our culture, and pervert innocent
souls with cybersex, pornography, and Disney World," said John Holmes,
the organization's president. "They call their favorite software 'Satan,'
they call their computers 'demons,' and the website of their leaders is
called "satanic.org." Did you know that '666' in Linux means 'give myself
permission, give groups permission, and give everyone permission?'

The dignified-looking, elderly Holmes and his followers had been urging
state and federal authorities to step in, picketing the picketers around
the clock for nearly a week. "They talk about 'the people' and the evils
of capitalism. They're nothing but socialist anarchists working for the
liberal Jewish homosexual environmentalist media!" he added.

Radio talk show host Dr. Laura Schlessinger also jumped into the controversy.
"They talk about freedom but not of personal responsibility, permissiveness
without morality. And what is their symbol? Is it a religious symbol? A
patriotic symbol? A symbol of morality or decency? No. Their symbol is
a penguin- a PENGUIN! An eagle that can't fly, a chicken with no nutritional
value, a clumsy, stupid creature that waddles and wallows in the icy shell
of a cold, lonely hell of isolation," she told listeners yesterday.

Noon: The Nerds Go Nuts

The event that apparently triggered the rioting was, ironically, an act of kindness, generosity, and goodwill on the part of Microsoft. It was not well received, however, among these anarchic backers of free living and free software. Microsoft spokesperson John Mash met with the leaders of the protest, and made them an offer which, he said, was sure to satisfy them. "I couldn't believe it- it HURT" said Linux activist FooManchu, tears welling up in his eyes at the

Protesters charge the front gate, where they covered the Microsoft logo with a Linux logo.

memory. "Do you know what they were offering? They were going to
give each of us an MSN CD and ten free hours. I just totally lost it. We
all did. It got really confusing when all those gun nuts came pouring out
of the Convention Center."

"Those gun nut jerks deserved to be teargassed," said Herman Kriegek,
31, of Worthington Ohio, on condition that he not be identified. "The sick,
bloodthirsty animals came out of the gun show shouting about the UN and
martial law. But they didn't know which side we were on. Our stuffed penguins
seemed to really confuse them. They finally seemed to decide they'd better
kick our asses just to be sure, and they were doing just that when we heard
this sound, like thunder."

That thunderous noise was from the motorcycles of some 300-400 members
of the Judas Disciples motorcycle gang, who were passing through downtown
in formation. They has just attended the funeral of one of their own who
had been gunned down in a barroom brawl.

"I was running from the gun nuts and the police, and was running down
the middle of the street in a panic. I thought it couldn't possibly get
worse. I look up and it's a freakin million bikers wearing gang colors,
coming right at me. The guy asks me why I'm runnin' and I tell him that
they're trying to kill us because we're into Linux. This dude, the meanest,
most evil-looking dude I ever saw in my life looks at me and says 'sheeeit,
Linux? I run Red Hat on my linux box" and nods to the dude next to him.
Turns out that the chief enforcer for the gang had met Linus Torvalds at
Sturgis way back years ago. Linus turned him on to Red Hat, and he's been
writing GNU software ever since."

Police have rounded up most of the organizers, who are now in jail awaiting
bond hearings. Still being sought is the suspected leader of the cult.
"They know that we know about their leader. We're overheard their whisperings.
This "Colonel" guy will be tracked down, and he WILL be brought to justice,"
said Seattle police Lt. Turner Treaques. "He thinks we're stupid, but we're
hot on his trail."

According to jailer Mike Fostquel, the captured nerds are model prisoners.
"They made a crude but listenable crystal radio out of a light bulb, a
crayon, and a square of toilet paper, and a rock. They say they'll have
linux on it by next week. They seem to be having a really good time. Tell
ya the truth, I'll hate to see 'em go."

---

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Linux And Martial Arts Humor - Linus Torvalds Vs. Chuck Norris

Hey there,

Hope your Saturday is going swimmingly, work isn't calling to bug you, your significant other is giving you that alone time you so desperately need or he/she is finally acknowledging your existence again ;)

As per usual, we're doing our best to find some quirky and/or original humor out there on the web, since no one ever wants to work on the weekend, much less read about it. Quite the opposite is true. To paraphrase Loverboy, the weekend is what everybody's working for. Why everybody needs a second chance still has me at a loss, and seems to dilute the purity of the initial truism so eloquently put forth by the band. Perhaps, in a universal sense, Loverboy is saying that, when that weekend we've been working for finally does arrive, we all manage to do something unbelievably stupid and/or harmful on Saturday. I'm assuming the second chance would come on Sunday, when we all (coincidence?) generally attend our religious congregation of choice and beg for redemption. It's also quite possible that we've (in a universal sense) gotten ourselves so deep in the sh** that (and here's where it really becomes clever) the entire reason we're working so hard "for" the weekend is because we won't be getting that second chance until then and "only" then. Of course, when that next weekend comes, we're duty-bound by logic to completely f*** everything up again. I don't know how long you could possibly stand living that kind of roller coaster life. Perhaps Loverboy, again, stated it best later in their career when, having matured to embrace a fuller and more worldly philosophy, they noted that you and me could let it be and just love every minute of it. Which would naturally segue into a much needed call to action... Ready, aim, fire??? Loverboy just doesn't make any sense to me anymore. ...maybe those hot girls really weren't in love? Have I been living a lie all these years? Almost paradise, indeed. I don't know what to believe anymore...

Anyway, back to reality :) I found the original part of this joke at Linuxscrew.com and hopped from there to ChuckNorrisFacts.com I'm not a huge fan of comic adulation, so the I found the Chuck Norris pages quite a bit funnier. The Chuck Norris site certainly does its fair share of bowing to the mighty Chuck, but (if you visit the site and actually read the entire list) the Linus Torvalds jokes get old pretty fast. They're all along the lines of the classic "yada, yada, yada... who him? No, that's God, he just "thinks" he's INSERT NAME OF YOUR HERO HERE." There was a reason they invented the APPLAUSE sign ;)

Hope you enjoy some of these, and definitely check out the sites if you want some more jokes (and some better, but not computer relate at all, ones at the Chuck Norris shrine).

Enjoy your Saturday and be good to one another :)

---

Linus Torvalds once found a segmentation fault in the universe.

Linus Torvalds can run kill -9 and kill Chuck Norris.

Linus Torvalds doesn't die, he simply returns zero.

Linus Torvalds first written program had artificial intelligence.

Linus can divide by zero.

Linus Torvalds runs Linux on his wristwatch and toaster.

Linus Torvalds doesn't receive error messages.

There is no theory of probability, just a list of events that Linus Torvalds allows to occur.

Linus Torvalds takes one look at your desktop and knows which porn sites you visited. In the last ten years.

If you have five dollars and Chuck Norris has five dollars, Chuck Norris has more money than you.

There is no 'ctrl' button on Chuck Norris's computer. Chuck Norris is always in control.

Apple pays Chuck Norris 99 cents every time he listens to a song.

Chuck Norris can sneeze with his eyes open.

There is no theory of evolution. Just a list of creatures Chuck Norris has allowed to live.

Chuck Norris is suing Myspace for taking the name of what he calls everything around you.

Chuck Norris destroyed the periodic table, because he only recognizes the element of surprise.

Chuck Norris can kill two stones with one bird.

There is no chin under Chuck Norris' Beard. There is only another fist.

---

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Another Simple Scriptlet To Make The Unix And Linux CLI More User Friendly

Hey there,

Today we're going to shoot out another quick scriptlet that might be useful (or distracting ;) from time to time. It’s a bit of a follow up on our post from earlier this week on using bash to produce fancy user names for folks logged into your machine, although it’s a little bit longer. Once again, if you go by the 65 character rule, this isn’t a one liner, even though it’s all going on one line ;)

Our little foray into variable interpolation today has to do with making /etc/passwd just a bit more accessible. Most folks are familiar with the 7 colon-separated fields of this file (if not, they’re: The user’s Username, an encrypted password (or usually an “x” if you’re using a shadow password system), the user’s UID, the user’s GID, the user’s gecos information (Also referred to as the “comment field,” usually either empty or populated with a short description to make the account more easily recognizable) , the user’s home directory and, finally, the user’s default shell)

Now, say what you will about the structure and setup of the passwd file, but it’s fairly easy to follow. Almost everything in it is the opposite of cryptic. The password field (field number 2) is, of course, not going to be tailored for your ease of understanding ;) If you get really lucky and you hop on a machine that actually lists the encrypted password in field two of /etc/passwd, you can copy the file to any remote machine and go to work on it. We've posted quite a few scripts to do automated password cracking. You should just see an "x" for a standard user account. If you don’t, have a talk with your SysAdmin or, if you’re him, please move on to a shadowed password system. Too much information exists in the public domain to make the old-style setup anything more than a trifling hassle to someone who wants to exploit your system. Especially a disgruntled employee!

Other than that, the user ID (field number 3) is an easy relation to make, since the username is in the same file (field number 1). The only thing in there that doesn’t practically explain itself is field number 4; the GID (group id) field. It’s not that big of a deal to look a user up in /etc/passwd, glean that group id information and find out what group the user is in by looking at /etc/group. It’s as simple as this:

host # grep user1 /etc/passwd
user1:x:37527:501::/export/home/user1:/bin/bash
host # grep 501 /etc/group
people::501:

and, just like that, we know that the user "user1"'s primary group membership is in the group "people." There are also secondary groups to consider, but finding this out is trivial as well. Just do the following:

host # grep -w user1 /etc/group
guys::600:user1
folks::601:user1

And, yes, I can hear you out there ;) I'm aware of commands like "groups" and "id" ;) They're very good at getting this information, as well (which is why we're going to use "groups" in our one-long-liner. Your output may look slightly different). Note that some versions don't have the -a option for "id" and some have either -g, -G, any combination of the three or all of them - it depends on your distro:

host # groups <-- This is supposed to list groups in order, from left to right, with the leftmost group being the "primary" and every group after that being a "secondary." The ordering of the "secondary" groups in the list is generally "as they appear" in /etc/group. So, since the line with "guys" is on a line closer to the beginning of the /etc/group file than the line with "folks," they're printed in that order. No other relationship exists between them that I'm aware of.
people guys folks
host # id -a
uid=37527(user1) gid=501(people) groups=501(people) <-- On Solaris; not quite as helpful as "id" on other flavours of Linux or Unix.

Anyway, I think I've met my word-count quota ;) Just kidding. I have no idea how much I've typed. I'm not even sure I want to know...

Here's a nice little command line (and, consequently, the topic of this post ;) to print every user's group(s) in the (modified) output of /etc/passwd. Just type it out, like so:

host # IFSBAK=$IFS;export IFS=":";while read one two three four five;do groupnames=`groups $one|sed 's/ /,/g'`;echo "${one}:${two}:${three}:${groupnames}:${five}";done </etc/passwd;export IFS=$IFSBAK


...lots of output left out for reasons of brevity and security :) ...
noaccess:x:60002:noaccess:No Access User:/
user1:x:37527:people,guys,folks::/export/home/user1:/bin/bash
...

And, there we have it. All nice and tidy :) Here's the same thing in script format (below) to make it a little easier to read. Be sure (now matter how you run it) that you don't clip the beginning and end, as you'll probably want to reset your shell's IFS (field separator) to what it was before we changed it to the colon (:) to make parsing /etc/password possible without using awk or cut, etc... Also, if it's of any interest and you noted that I only counted my variables up to "five" it's because "while read" will swallow up the rest of a line in its last variable. So, instead of reading and writing five, six and seven, I just read five, since that would include from field five all the way to the end of the line (at the end of field 7), and field 4 (the group field) was the last field on every line that we were actually interested in :)

#!/bin/bash

IFSBAK=$IFS
export IFS=":"

while read one two three four five
do
 groupnames=`groups $one|sed 's/ /,/g'`
 echo "${one}:${two}:${three}:${groupnames}:${five}"
done </etc/passwd

export IFS=$IFSBAK

Have a great weekend!

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Plain English Explanation Of An Awk Statement For Linux Or Unix

Hey There,

Today's post is going to be a follow-up to yesterday's post on convoluted column arithmetic in awk. I have a nasty habit of listing all my stuff as "beginner" material. Mostly this is because I try to write my material for the middle level to newer user of Unix, Linux, Perl, etc. I have absolutely no interest in picking apart the finer points of quantum mathematics or advanced concepts in programming or operating system design and implementation. I leave that for the academics. Besides, if I did go that route, I'd have to narrow the scope of this blog a "whole lot" ;)

Yesterday's post offered a number of cut-and-paste awk solutions to solving some seemingly difficult data massaging problems and, in my frantic struggle to keep my post under a thousand words, I cut some corners and simply framed the problem, following it with the solution. It's usually a good formula. I'll be the first to admit, though, that the examples were somewhat of a test for me when I first slapped them together and probably deserved to be explained more than they were. To that end, we'll look at one of the examples from yesterday and pick it apart, so that the pieces all make sense to, hopefully, any and every one. I aim to please :)

The example we'll work with is from part one of problem 2, where we looked to add the even columns and, separately, the odd columns and then print those totals at the end of each line. The original awk statement and resultant solution follow (check yesterday's awk post for the original dataset. It doesn't really make a difference insofar as this explanation will be concerned):

host # awk '{sum1=$1;sum2=$2;for (i=3;i<=NF;i++) {if ( i%2 ) {sum1 += $i; printf"%.2f ", $i}else {sum2 += $i; printf"%.2f ", $i}}printf"ODDS %.2f EVENS %.2f\n", sum1, sum2}' DATASET2

51.61 48.39 41.38 58.62 32.00 68.00 56.41 43.59 57.52 42.48 ODDS 284.75 EVENS 315.25
49.06 50.94 42.86 57.14 100.00 0.00 0.00 100.00 11.49 88.51 ODDS 259.96 EVENS 340.04
52.17 47.83 26.83 73.17 61.54 38.46 57.69 42.31 33.33 66.67 ODDS 273.23 EVENS 326.77
34.38 65.62 0.00 100.00 59.26 40.74 100.00 0.00 43.43 56.57 ODDS 257.07 EVENS 342.93
25.53 74.47 35.56 64.44 84.42 15.58 47.53 52.47 0.00 100.00 ODDS 243.04 EVENS 356.96

At first glance, I suppose that awk statement could look imposing. Fortunately, once you pull it apart, it's not all that bad. First, we'll write the awk in "plain English." Here's what the linear thought process would be if we just spelled it out:

To start out, we'll set the variable sum1 to the value of the first field. Then, we'll set the value of the variable sum2 to the value of the second field (This sets us up with sum1 holding the first "odd" field value and sum2 holding the first "even" field value). After this we'll iterate through a looping construct that begins by setting the value of the variable i to 3, continues until the value of the variable i is less than or equal to the number of fields on any given line (NF), and increments the value of i by 1 after every pass.

For each invocation of the loop (which goes until the value of i becomes less than or equal to the number of fields on the line, or record), we check the value of the modulo of i and 2. If you want to look up the definitions of modulo and modulus, I strongly encourage it, but only if you're really curious ;) For our purposes, it will serve to explain that i%2 (i modulo 2) acts as a check on the remainder of a division operation. So, if the value of i is 3, i%2 is equivalent to 3 divided by 2. Since 3 doesn't divide evenly by 2, it leaves a remainder of 1. That remainder is the value of the equation i%2. On the next pass (and every pass where an even number is concerned) the value of i%2 will be 0 since, for instance, 4 divides evenly by two, with a remainder of 0. Hopefully this isn't becoming more confusing than it started out being ;)

So, knowing what we now know, the "if statement" that gets run in every iteration of the outer "for loop" does a little bit of "backward figuring" (that is to say that the logic may not seem to be correct at first glance, although it is ;) The "if conditional" tests the value of i%2. For all even numbers this is equal to 0 and for all odd numbers it is equal to 1. So the statement "if ( i%2 )" is testing the value of that statement. In the case of awk, and most shell equations and their return codes, 1 equals true and 0 equals false. It's almost like saying: If "true" (the case for all odd numbers) do this and if "false" (the case for all even numbers), do the other thing... On to the next part... I never thought this would be so hard to explain ;)

This next part is pretty simple. For the odd numbers, the value of the variable sum1 is made equal to its old value plus the value of the field we're reading on this pass. If the value of i is 3, then we're checking field 3 (or $3, if you prefer). For the even numbers, the value of the variable sum2 is made equal to its old value plus the value of the field we're reading; same as with the odds. If i equals 4, we're adding the value of the i field (field 4, or $4) to the already existing value of sum2. For both instances, we're doing a simple printf to output that value to the terminal.

Now, to end it all, when the value of the variable i becomes less than or equal to the number of fields (NF) on the line (or record), we slip in a quick printf statement to print the values of the odd numbers (sum1) and even numbers (sum2) before moving on to the next line. This can be a confusing point because, as you may know, awk (like sed, etc) operates on every line of a file when you feed it one. It's for this reason that we're so careful with the bracketing. If we weren't able to control where the totals printed out, they'd print only after all the lines in the input file were processed, which isn't what we wanted.

Finally, here's the exact same equation, broken up across several lines (I "chose" to do everything on one line for yesterday's examples, because that makes it easy for me to re-run statements using my line editor in bash :) Hopefully, looking at it in this way (in a proper script-like format) will help shed some more light on the "structure" of the process, just like the previous paragraphs, hopefully, helped elucidate some of the finer points of the "execution" of the process:

awk '{sum1=$1;sum2=$2
for (i=3;i<=NF;i++) {
 if ( i%2 ) {
  sum1 += $i; printf"%.2f ", $i
 }else {
  sum2 += $i; printf"%.2f ", $i
      }
}
printf"ODDS %.2f EVENS %.2f\n", sum1, sum2
}' DATASET2

if you slapped the above in a file, called it "awky.sh" and ran:

host # sh awky.awk

you'd get the exact same results as you got from the one-line version above.

Hopefully, this explanation has been helpful and, if not, please let me know what still remains a point of confusion for you. Unless there are a lot of requests for certain specific bits of info that would warrant another post, I'll be happy to help clear up little bits and pieces either via email or on the boards :)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Convoluted Column Arithmetic Examples Using Awk On Linux Or Unix

Hey There,

Today's going to be a "fun with awk" day. I figure we should have one now because we never had them in school when I was a kid... The topic, as the title suggests, has to do with columnar arithmetic or, less pompously put, performing arithmetic operations on columns or, even more accessibly, adding stuff up ;) It's somewhat like our older post on doing simple cumulative math with awk, but slightly more confusing.

The basic premise here is that you, I and the next guy have all seen the solution to many a question like: "Given a file filled with rows of space delimited numbers, how do I add up all the values for field 2 ($2) in awk so that I can get the total when awk is through processing my file?" ...or something to that effect. Knowing the fundamentals of performing arithmetic in awk can be a great asset to you at work (or, at play, if you like to add, subtract, multiply and divide for kicks ;), but, for the most part, only the most banal instances of its application are either used or queried for on the web (at least as far as my search engine testing has shown so far).

So, without further fanfare, we'll take a look at two sets of data (below) and some various (hopefully, not utterly pedestrian) ways to sort and manipulate the data. These are pulled from a series of posts I answered on the Daniweb.com Shell Scripting Forum regarding odd and even column operations (I try to only plagiarize myself ;). I found the questions interesting to answer and, hopefully, you'll find both the requests and their resolutions of some use, as well :) I've modified some of the solutions that I posted to the board for this blog post, as I found that, at some points, my quick (trying to be helpful) answers could be done a bit better with a little extra attention. Also, the entire thread isn't played out here and some additional problems haven't been included. There's only so much I can type ;)

The setup involves a file that contains these numbers in the columns and rows - or fields and records, as we'll refer to them from here on out - shown below. The initial problem uses this dataset. The result of the initial solution forms the dataset for all the problems to follow (shown below, in, hopefully, sensible order):

Data Set 1:

55 65 48 45 48 68 32 68 44 34 88 65
82 63 52 54 51 68 75 0 0 20 10 77
55 77 60 55 22 60 40 25 75 55 45 90
20 80 33 63 0 64 32 22 75 0 43 56
54 54 12 35 48 87 65 12 77 85 0 15

Problem 1: Given "Data Set 1" (conveniently placed in a file called "DATASET1"), for each pair of successive fields , using awk's built in arithmetic operators, convert each field by dividing it by the sum of the pair and multiplying each by 100 to produce recognizable relative-percentage output. So, for the first column, you would have the results of equations of the following order:

$1/$1+$2, $2/$1+$2, $3/$3+$4, $4/$3+$4, etc....

Solution 1: This can be worked out by the following awk command (technically not a "one-liner" since it exceeds the 65 character limit):

host # awk 'BEGIN{lasti=1}{for (i=1;i<=NF;i++) {if ( i%2==0 ) {y=$lasti+$i;printf("%.2f %.2f ", $lasti/y*100, $i/y*100)}lasti=i }printf"\n" }' DATASET1

This solution produces the following output, which will become "Data Set 2" (which will be put in a file called.... DATASET2. I couldn't come up with anything more clever under pressure ;)

45.83 54.17 51.61 48.39 41.38 58.62 32.00 68.00 56.41 43.59 57.52 42.48
56.55 43.45 49.06 50.94 42.86 57.14 100.00 0.00 0.00 100.00 11.49 88.51
41.67 58.33 52.17 47.83 26.83 73.17 61.54 38.46 57.69 42.31 33.33 66.67
20.00 80.00 34.38 65.62 0.00 100.00 59.26 40.74 100.00 0.00 43.43 56.57
50.00 50.00 25.53 74.47 35.56 64.44 84.42 15.58 47.53 52.47 0.00 100.00

Problem 2: From "Data Set 2," take each line and add up the all the odd fields and all the even fields per record. Then, as an added bonus, add up all the odd fields and all the even fields in all records. This was solved by the following awk statement (Note that, on the board, the sum for the first part of this problem was not printed as part of the solution, since it was being manipulated via an external array elsewhere within a script. For this example, I'm putting the totals for each line at the end of each line and anchoring them with text for easy distinction):

host # awk '{sum1=$1;sum2=$2;for (i=3;i<=NF;i++) {if ( i%2 ) {sum1 += $i; printf"%.2f ", $i}else {sum2 += $i; printf"%.2f ", $i}}printf"ODDS %.2f EVENS %.2f\n", sum1, sum2}' DATASET2

51.61 48.39 41.38 58.62 32.00 68.00 56.41 43.59 57.52 42.48 ODDS 284.75 EVENS 315.25
49.06 50.94 42.86 57.14 100.00 0.00 0.00 100.00 11.49 88.51 ODDS 259.96 EVENS 340.04
52.17 47.83 26.83 73.17 61.54 38.46 57.69 42.31 33.33 66.67 ODDS 273.23 EVENS 326.77
34.38 65.62 0.00 100.00 59.26 40.74 100.00 0.00 43.43 56.57 ODDS 257.07 EVENS 342.93
25.53 74.47 35.56 64.44 84.42 15.58 47.53 52.47 0.00 100.00 ODDS 243.04 EVENS 356.96

and for part 2 of problem 2 - to get the grand total of all even fields and grand total of all odd fields:

host # awk 'BEGIN{sum1=$1;sum2=$2}{for (i=1;i<=NF;i++) {if ( i%2 ) sum1 += $i;else sum2 += $i}}END{print "ODDS " sum1 " EVENS " sum2}' DATASET2

ODDS 1318.05 EVENS 1681.95

Problem 3: Given "Data Set 2," how could you divide every odd field by the grand total of all odd fields, and divide every even field by the grand total of all even fields (with minor modification to set the decimal precision from 2 to 4, since the resulting values will be well below 1% ;) - For example, you would want output that went through the data file and did "$1/1318.05 $2/1681.95, $3/1318.05..." etc. This can be accomplished by the following:

host # awk 'BEGIN{odds=1318.05;evens=1681.95}{for (i=1;i<=NF;i++) {if ( i%2 ) {odd=$i/odds; printf"%.4f ", odd}else {even=$i/evens; printf"%.4f ", even}}printf"\n";}' DATASET2

0.0348 0.0322 0.0392 0.0288 0.0314 0.0349 0.0243 0.0404 0.0428 0.0259 0.0436 0.0253
0.0429 0.0258 0.0372 0.0303 0.0325 0.0340 0.0759 0.0000 0.0000 0.0595 0.0087 0.0526
0.0316 0.0347 0.0396 0.0284 0.0204 0.0435 0.0467 0.0229 0.0438 0.0252 0.0253 0.0396
0.0152 0.0476 0.0261 0.0390 0.0000 0.0595 0.0450 0.0242 0.0759 0.0000 0.0330 0.0336
0.0379 0.0297 0.0194 0.0443 0.0270 0.0383 0.0640 0.0093 0.0361 0.0312 0.0000 0.0595

and here's an alternate way to get the same thing (combining problems 2 and 3 with the assumption that we don't know the exact values of the "odds" and "evens" variables I populated in the BEGIN section of problem 3):

host # echo `awk 'BEGIN{sum1=$1;sum2=$2}{for (i=1;i<=NF;i++) {if ( i%2 ) sum1 += $i;else sum2 += $i}}END{printf"%.4f %.4f", sum1,sum2}' DATASET2`|while read x y;do awk -v odds=$x -v evens=$y '{for (i=1;i<=NF;i++) {if ( i%2 ) {odd=$i/odds; printf"%.4f ", odd}else {even=$i/evens; printf"%.4f ", even}}printf"\n"}' DATASET2;done

0.0348 0.0322 0.0392 0.0288 0.0314 0.0349 0.0243 0.0404 0.0428 0.0259 0.0436 0.0253
0.0429 0.0258 0.0372 0.0303 0.0325 0.0340 0.0759 0.0000 0.0000 0.0595 0.0087 0.0526
0.0316 0.0347 0.0396 0.0284 0.0204 0.0435 0.0467 0.0229 0.0438 0.0252 0.0253 0.0396
0.0152 0.0476 0.0261 0.0390 0.0000 0.0595 0.0450 0.0242 0.0759 0.0000 0.0330 0.0336
0.0379 0.0297 0.0194 0.0443 0.0270 0.0383 0.0640 0.0093 0.0361 0.0312 0.0000 0.0595

And, hopefully, that's enough awk for now. Check out the Daniweb.com Shell Scripting Forum boards to see some of the great solutions other folks come up with (for some pretty bizarre problems) and, if it's just not your day, a few of my other monstrosity's ;)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---