Wednesday, December 31, 2008

New Year,s Eve News: Unix And Linux Users Beware

Hey there,

As part of a non-existent tradition, composed of this year's "New Year's Eve post" (being written as I type ;) and last year's zany and over-the-top network port querying script post, we're going to make this year's EOY post short, to the point, hopefully fun and somewhat worthwhile, so you can leave work early without getting it ok'ed and begin making bad decisions as soon as possible ;)

In all seriousness, if you go out tonight, have one (or maybe three or four ;) on me, but try to stay safe. According to a recent study by law enforcement and the National Association of Situations You'll Regret Later (NASYRL), lots of people get into, sometimes fatal, accidents and/or end up in jail for driving under the influence during the holidays. Everyone here wishes you a safe, happy and fun New Year's Eve and highly encourage you to partake of your legal drug of choice in quantities that will make you question your prior state of mind tomorrow (we can't officially condone the other stuff, but would recommend - if we were in the business of minding yours ;)- that you do whatever you want to do in moderation. Although certain substances have a higher lethality-rate than others, excess is the enemy. Basic food and water will demonstrate this very well any day of the year ;).

And, if possible, if you must go home at the end of the night, please find someone sober to drive you. If you're that sober person, please do what you can to round up a few rummies that need a lift ;)

And here's a little cautionary tale from 3 years ago, ripped from today's headlines! (wait... no... that's this week's episode of Law & Order ;) This piece is actually from LinuxReviews.org regarding the dangers of using a text web browser, especially during times of crisis and/or excess. Hopefully, it will scare the cr@P out of you and you'll begin doing what you're told. If not, and you either enjoy it, hate it or feel absolutely nothing for it, yet still refuse to walk the line, we salute you!

Have fun this New Year's Eve! Best wishes for your safety and maximum enjoyment from all of us :)

Cheers,

NOTE: We left in all the spelling mistakes from the original articles to maintain the integrity of the sources and, also, to simulate drunken journalism in keeping with the night's theme ;)



Solaris-user arrested by British Police for using Lynx


Lynx is a popular text-based browser user by Unix and Linux users. British Telecom, who are the web site administrator for a Tsunami aid site mistook a Lynx web browser user who donated money for a hacker because of the strange user agent the browser leaves in the logs and reported the user to the police.



The police, apparently equally incompetent, raided the private citizen who made the donation swat-style while he was having a quiet lunch.



BBC reported this as Officers from the Metropolitan Police's Computer Crime Unit have begun an inquiry after BT blocked the attempt on New Year's Eve. A 28-year-old man from east London was arrested and released on bail in connection with alleged offences. and this is a good example of how state media reports violent crimes against private citizens when done by the same stats police force.



The persons only crime was to use the non-standard Internet browser Lynx.



From a mailing list:






For donating to a Tsunami appeal using Lynx on Solaris 10. BT (British Telecom) who run the donation management system misread an access log and saw hmm thats a non standard browser not identifying it's type and it's doing strange things. Trace that IP. Arrest that hacker.



Armed police, a van, a police cell and national news later the police have gone in SWAT styley and arrested someone having their lunch.



Out on bail till next week and preparing to make a lot of very bad PR for BT and the Police....



So just goes to show if you use anything other than Firefox or IE and you rely on someone else to interogate access logs or IDS logs you too could be sitting in a paper suit in a cell :(






What is Lynx?



Lynx, Links, Dillo and other rare browsers are used by thousands browsers are used by thousands of Linux and Unix users world wide, but they only make up a very small percentage of visits to mainstream entertainment sites. This may because many such sites frequently use javascript, flash and other garbage non-standard elements that make these sites impossible to render in standard-compliant browsers.



Average Entertainment Website, Browser market share Desember 2004



























202932 visitsBrowser
53.54%Internet Explorer 6.0 (Windows)
27.10%Gecko 1.7.x (Mozilla/Firefox/Galeon/Epiphany/etc)
4.21%Internet Explorer 5.0
4.14%Safari (OSX)
2.54%Opera




The picture is very different for Linux related sites.



Linux Reviews, Browser market share Desember 2004



























56360 visitsBrowser
36.63%Gecko 1.7.x (Mozilla/Firefox/Galeon/Epiphany/etc)
17.58%Internet Explorer 6.0 (Windows)
10.43%Internet Explorer 5.0
3.37%Opera
2.83%Konqueror




Lynx was not in the entertainment site numbers and had 0.53% share at Linux Reviews.



SourceForge, Slashdot, Freshmeat, Lxer and other sites like these have regular visitors who use rare browsers, but if you are using Lynx then you may want to stay clear of official UK government web sites and perhaps also other government web sites in general.






, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Tuesday, December 30, 2008

Unix And Linux Easter Eggs For The Wrong Holiday

Hey there,

Today, since it's just past Christmas and almost New Year's, I figured this would be a great time to trot out some Linux and/or Unix Easter eggs. Actually, it doesn't make sense at all, but if you can put aside your burnt-in sense of the chronological order of the holidays, these can still be fun ;)

I found all of the Easter Eggs for today at a site with the very strange name Eeggs.com. I don't know what an eegg is, and I'm not sure that I want to know, but they have a great collection of Easter Eggs for all manner of OS' ;) I spent most of my time in their Linux section, but you could spend hours on other sections of their site and only occasionally be reminded that you're still at work. Of course, in all seriousness, if you're at work, the thought of driving home as soon as possible is keeping you aware of your location at all times ;)

The following are a few of the cooler ones I ran across (AND could personally verify). If you get a chance, drop by Eeggs.com and submit a support email asking why "eegs" isn't in the dictionary when "ain't" is ;)

1. Fun with PHP. This has worked with every site I've tested it against. The key here is just to find a php-enabled site, and navigate to a php page. Then, all you need to do is pass the php page a few arguments on the browser command line to find these four gems.

For a working example, we'll look at linuxandunixupdates.com's index.php page. Using that URL, we can add the following four strings and get the following four easter eggs. All of the links in this section are set to open up in new windows, so you can click on the link above and add the strings manually, or you can just click on any of the links below. I've also included a picture of the outcome of running those commands below each "magic string" just in case you're worried that I might be luring you into clicking on a redirected link or something else I don't have the time to invest in doing properly right now ;) You should be able to replicate this on any php page on any site anywhere. I haven't been able to fully test the veracity of that claim, but it appears to be true so far!

a. Add ?=PHPE9568F34-D428-11d2-A769-00AA001ACF42 to the end of your URL to see this picture:

php logo

b. Add ?=PHPE9568F35-D428-11d2-A769-00AA001ACF42 to the end of your URL to see this picture:

zend engine 2 logo

c. Add ?=PHPE9568F36-D428-11d2-A769-00AA001ACF42 to the end of your URL to see this picture:

squiggly php logo

d. Add ?=PHPB8B5F2A0-3C92-11d3-A3A9-4C7B08C10000 to the end of your URL to see the PHP Credits. This page looks exactly like the standard info.php page, but lists all the developers who worked on each component. I haven't included it here because it's incredibly long and there are more Easter Eggs to get to before we all forget why we're here :)

2. MAGIC reboot times in the Linux Kernel. This one is interesting, and a bit of a puzzle, since the original entry only gives the answer to the first time (they're all significant to Linux in some way). In any event, you can find these times by looking in /usr/include/linux/*.h and grepping for LINUX_REBOOT_MAGIC. As you can see, below, in our includes, they're all in reboot.h:

host # grep LINUX_REBOOT_MAGIC /usr/include/linux/*.h
/usr/include/linux/reboot.h:#define LINUX_REBOOT_MAGIC1 0xfee1dead
/usr/include/linux/reboot.h:#define LINUX_REBOOT_MAGIC2 672274793
/usr/include/linux/reboot.h:#define LINUX_REBOOT_MAGIC2A 85072278
/usr/include/linux/reboot.h:#define LINUX_REBOOT_MAGIC2B 369367448
/usr/include/linux/reboot.h:#define LINUX_REBOOT_MAGIC2C 537993216


MAGIC2 (as well as the MAGIC2A, B and C) is where you'll find the Easter Egg. If you take any of those values and convert them into regular time (using Perl, for instance), they resolve to an important date in Linux history.

host # perl -e 'print localtime(672274793). "\n";'
Sun Apr 21 17:59:53 1991
host # perl -e 'print localtime(85072278). "\n";'
Mon Sep 11 10:11:18 1972
host # perl -e 'print localtime(369367448). "\n";'
Mon Sep 14 21:04:08 1981
host # perl -e 'print localtime(537993216). "\n";'
Sun Jan 18 12:33:36 1987


Sun Apr 21 17:59:53 1991 is supposedly (and I'm not using the word "supposedly" to cast any more doubt than any reasonable human being would have. I'm not sure if the following is true, so I can only "suppose" that the folks who submitted these Easter Eggs aren't just prepping a new Wikipedia page. Just kidding, of course. Everything in Wikipedia is true ;)) the date Linus Torvalds first began writing Linux. The rest is left up to us to figure out. Something tells me the answers are all somewhere in this Linux Online Timeline.

3. And lastly, so there's plenty more left for you to check out at Eeggs.com, I really enjoyed this last one (actually there were a few others I'm dying to try, along the same lines, but don't have the proper OS' to validate right now) since I'm a "huge" fan of Douglas Adams, even beyond the HitchHiker's Series (although lots and lots of people got really upset over Mostly Harmless when he chose to wrap up the HitchHiker's Trilogy (with the 5th book in the series) in a manner that, apparently, was extremely dissatisfying to ardent fans of the series. I don't begrudge them their opinions. I dug it. I'm only sorry that he passed away and that we'll never know if the The Salmon of Doubt was going to be the sixth HitchHiker's book (answering the fan's complaints, at worst) or the next Dirk Gently novel.

Back to planet earth ;) If you open up vim, and type the following:

host # vim
[esc]:help 42


with the [esc]: being the actual "escape" or "esc" key, followed by the colon (:)

You'll, sadly, not get an explanation of the answer to the meaning of life, the universe and everything, but the payoff's just as pleasant :)

What is the meaning of life, the universe and everything? *42*
Douglas Adams, the only person who knew what this question really was about is
now dead, unfortunately. So now you might wonder what the meaning of death
is...

==============================================================================

Next chapter: |usr_43.txt| Using filetypes
...


Hope you all enjoyed those Easter Eggs and, should you decide to look for more, happy hunting :)

Cheers,

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Monday, December 29, 2008

Finding Your Yahoo Search Index Rank From The Unix Or Linux CLI

Hey There,

Today we're going to continue in our ongoing quest to rank highly in search engine results while simultaneously messing with them a lot ;) Previously, we've put out scripts to find your MSN search index rank from the CLI and to find your Google search index rank from the CLI. This is, of course, another script that, although it fits in the same category as the other two, is distinctive in several ways:

1. This time we're scouring Yahoo's search results.

2. The search results are being parsed differently, so that it makes it easier for the script to detect the point at which Yahoo cuts you off and won't let you do another search for a good 10 or 15 minutes.

3. The parsing code has been minimized, somewhat, so that it may actually be readable by humans who aren't pipe-chain-sed-awk-regular-expression fanatics ;)

4. See the continuation below for a very interesting analysis of Yahoo's robot-tolerance.

BUT FIRST, THIS VERY IMPORTANT NOTE: If you use wget (as we are in this script), or any CLI web-browsing/webpage-grabbing software, and want to fake the User-Agent, please be careful. Please check this online article regarding the likelihood that you may be sued if you masquerade as Mozilla, from the folks who maintain wget themselves.

This Yahoo script, of course, is only slightly different than the original Google and MSN scripts, although the differences are significant enough that most of the core was rewritten completely. The script, itself, operates the same way our Google and MSN search index page rank scripts do, insofar as executing it from the command line goes. There are, at least, three different ways you can call it. The most basic being:

host # ./yrank www.yourdomain.com all these key words

It doesn't matter if they're enclosed in double quotes or not. If you "really" want to get the double quote experience, you just need to backslash your double quotes.

host # ./yrank www.yourdomain.com \"all these key words\"

Other ways include creating files with the URL and keyword information (same format as the command line) and feeding them to the script's STDIN:

host # cat FILE|./mrank
host # ./mrank <FILE


Point 4 (continued from above): Yahoo robot search tolerance as compared with Google. This is actually quite interesting since, I believe, the general assumption is that Google is far less tolerant of seemingly-human interaction with its search than Yahoo is. However, in this case (and we've repeated this experiment over and over again) the opposite is, in fact, true. Check it out! :)

The setup is that we've created a simple file called "searchterms" to feed to both the grank and yrank scripts. It contains the following information:

host # cat searchterms
linuxshellaccount.blogspot.com unix linux
linuxshellaccount.blogspot.com linux unix
linuxshellaccount.blogspot.com unix and linux
linuxshellaccount.blogspot.com linux and unix
linuxshellaccount.blogspot.com unix
linuxshellaccount.blogspot.com linux
linuxshellaccount.blogspot.com perl script
linuxshellaccount.blogspot.com shell script


Then, we put each search engine to the test. Each grabbing results at 100 per page. You'll notice that the Google search engine makes it through the entire bunch without kicking us to the curb ;)

The image below was captured with a rear-view mirror and is, therefore, actually larger than it appears. Click below to see it in "life size" ;)

Google Robot Tolerance Test

And here is the exact same experiment; this time run against Yahoo's search engine. It ...just ...barely ...nope. It doesn't make it, again ;)

This image was taken in "Wallflower-Vision" - sometimes referred to as the "Shrinking-Violet Protocol." To see it in its natural, and unabashedly large, state, just click on it below and it will almost definitely come out of its corner ;)

Yahoo Robot Tolerance Test

Our initial suggestion is to change this line in the script (decreasing the number that you divide RANDOM by will increase the maximum wait time between tries):

let random=${RANDOM}/600


The value of 600 roughly approximates between 0 and 60 seconds of wait time. Reducing that number to 300 will roughly approximate wait times between 0 and 120 seconds, etc. The numbers generated by bash's RANDOM variable may vary depending upon your OS, system architecture, etc.

Another possibility, which we didn't have time to fully test (so it's not included in the script) is that Yahoo may actually object to the direct manipulation of the GET request and would probably respond more favorably if we extracted the URL value for our next successive request from the "Next" button on the search page, rather than moving on to the next valid (although coldly calculated) GET string to bring up the next set of 100 results. Time will tell. Experimentation is ongoing.

Hope you enjoy it, again, and you're still enjoying the holiday's. Even if none of them apply to your religious or moral belief-system, at least you get some paid time off of work :)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# yrank - Get yer Yahoo's Out ;)
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if [ $# -lt 2 -a $# -ne 0 ]
then
echo "Usage: $0 URL Search_Term(s)"
echo "URL with or with http(s)://, ftp://, etc"
echo "Double Quote Search If More Than 1 Term"
exit 1
fi

if [ $# -eq 0 ]
then
while read x y
do
url=$x
search=$y
$0 $x "$y"
done
exit 0
else
url=$1
shift
search=$@
fi

search_terms=`echo $search|sed 's/ /+/g'`
start=1
count=1

echo "Searching for URL $url with search terms: $search"

results=`wget -O - http://search.yahoo.com/search?p=${search_terms}\&ei=UTF-8\&fr=yfp-t-501\&pstart=1\&b=$start 2>/dev/null|sed -n 2p 2>&1|sed 's/^.* of \([0-9,]*\) for .*$/\1/'`

while [ $start -lt 1001 ]
do
wget -O - http://search.yahoo.com/search?p=${search_terms}\&ei=UTF-8\&fr=yfp-t-501\&pstart=1\&b=$start\&n=100 2>&1|grep "error 999" >/dev/null 2>&1
screwed=$?
if [ $screwed -eq 0 ]
then
echo
echo "You have been temporarily barred due to excessive queries."
echo "Please change the \"random\" variable in this script to a"
echo "lower value, to increase wait time between queries, or take"
echo " 5 or 10 minutes before you run this script again!"
echo
exit 1
fi
wget -O - http://search.yahoo.com/search?p=${search_terms}\&ei=UTF-8\&fr=yfp-t-501\&pstart=1\&b=$start\&n=100 2>/dev/null|sed -n 2p 2>&1|sed 's/^.* of [0-9,]* for //'|sed 's/<[^>]*href="\([^"]*\)"[^>]*>/\n\1\n/g'|sed -e :a -e 's/<[^>]*>//g;/</N;//ba'|grep "^http"|sed '/^http[s]*:\/\/[^\.]*\.*[^\.]*\.yahoo.com/d'|sed '/cache?ei/d'|uniq|while read line
do
echo "$line"|grep $url >/dev/null 2>&1
yes=$?
if [ $yes -eq 0 ]
then
echo "Result $count of approximately " $results " results for URL:"
echo "$line"
exit 1
else
let count=$count+1
fi
done
end=$?
if [ $end -eq 1 ]
then
exit 0
else
let start=$start+100
let count=$count+100
let new_limit=$start-1
let random=${RANDOM}/600
echo "Not in first $new_limit results"
echo "waiting $random seconds..."
sleep $random
fi
done


, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Sunday, December 28, 2008

The NetSlave Quiz - Unix And Linux Humor

Hey There,

Todays Unix and Linux humour find brings back a lot of memories from the dotcom days for me. Humungous Stock Option incentives, constant excitement about a possible IPO (Coupled with constant worry about getting laid off next week ;) and, of course, the frequent (and often mandatory) drinking on the job. Yes, there was a "valid" reason software upgrades were published before the QA phase back then ;) This is, strangely enough, a practice that continues to this day... except without the booze. In today's fast paced world, managers are expected to make bad decisions with a clear head ;)

And, before I get myself in too much "more" trouble, here's the bit I found on softpanorama.org called the NetSlave Quiz. Of course, the full version is available on their site, along with a quit a bit more!

Hope you enjoy it and, if you're like me, don't be ashamed if you start feeling a little bit misty ;)

Out of respect for the original publisher's rights, I have not modified any of the links within the piece. They don't look like affiliate links to Amazon.com, but, if they are, he (or she) will be getting paid if you buy. I have a strict no-hijacking policy (more of a moral/ethical thing than any worry about the legality of it. I just wouldn't want anyone to do it to me. Kind of like that "golden rule" everyone keeps talking about ;)

Cheers :)



Softpanorama Open Source Humor Archive



A variation of Steve Baldwin's

 NETSLAVE QUIZ


Adapted from the Amazon.com  NetSlaves
page by Nikolai Bezroukov


PART I: YOUR WORK LIFE


1. AT YOUR LAST JOB INTERVIEW, YOU EXHIBITED:


A. Optimism

B. Mild Wariness

C. Tried to overcome headache. I was really tied

D. Controlled Hostility


2. DESCRIBE YOUR WORKPLACE:


A. An enterprising, dynamic group of individuals laying the groundwork for
tomorrow's economy.

B. A bunch of geeks with questionable social skills.

C. An anxiety-ridden, with long hours and a lot of stress because of
backbiting bunch of finger-pointers.

D. Jerks and PHB


3. DESCRIBE YOUR HOME:


A. Small, but efficient.

B. Shared and dormlike.

C. Rubble-strewn and fetid.

D. I have a personal network at my home with three or more connected
computers and permanent connection to the Internet


PART II: YOUR RELATIONSHIP WITH TECHNOLOGY


4. MY PERSONAL COMPUTER:


A. Maximizes my productivity.

B. Is my best friend.

C. Lets me work all kinds of crazy hours at home.

D. My computers can heat my apartment.


5. MY ALPHANUMERIC PAGER:


A. Is a convenient way to stay in touch with my office.

B. I do not like when it sometimes goes off at night

C. Has been temporarily disabled and I receive warning because of that

D. I have it all the time and always wear a cell phone on my belt.


PART III: YOUR INTERNET KNOWLEDGE


6. WHAT DOES THE "I" IN IPO STAND FOR?


A. Initial

B. Imaginary

C. Impossible

D. Insolvent


7. WHAT DOES THE "E" IN E-COMMERCE STAND FOR?


A. Electronic

B. Exciting

C. Error-prone

D. Exploitation


8. WHAT ARE THE "THREE C'S OF THE INTERNET?"


A. Content, Community and Commerce

B. Cults, Chats and Clutter

C. Cheetos, Chinese Food and Coke

D. Crap, Costs and Chaos


9. WHAT IS THE "WINDOWS ENVIRONMENT"?


A. A graphical operating system.

B. A ploy for world domination.

C. A bug-ridden, proprietary disaster. Open source is the way to go.

D. Linux rulez.


10. WHO IS THE REAL "FATHER OF THE INTERNET"?


A. Vice-president Gore

B. Vinton Cerf

C. Linus Torvalds

D. George Orwell


11. WHY WAS THE INTERNET CREATED?


A. To simplify a way to bid out jobs to an immigrant workers.

B. To boost the Dow over 10,000

C. To create a new bunch of billionaires

D. To create an IPO gold rush.


PART IV: YOUR HOPES AND FEARS FOR THE FUTURE


12. WHY ARE YOU WORKING IN THE INTERNET BUSINESS?


A. To get rich.

B. To get by.

C. Linux rulez

D. Nobody else will take me.


13. DESCRIBE YOURSELF POLITICALLY:


A. Republican

B. Democrat

C. Anarchist

D. Luddite


14. DESCRIBE YOUR RELIGION:


A. Monotheist

B. Atheist

C. Open Source

D. Linux


15. WHAT WILL HAPPEN IN 2000?


A. It's revenge time for brick-and-mortars.

B. Your girlfriend may dump you for a doctor or Venture Capitalist.

C. You are feared that your health insurance may run out and you may not
have new contracts to renew it.

D. You are feared that you will be fired or outsourced with no equity.


HOW TO SCORE YOUR RESULTS



  • If you answered “A” to any of the above questions, your IEQ
    ("Internet Exploitation Quotient") is zero – you’re not a
    Net Slave. Reading The Cathedral and Bazaar can be beneficial

  • If you selected "B," you’ve probably just started working in
    the technology business or lead some sort of charmed (mythical?) existence
    the rest of us only dream about.  Reading NetSlaves
    and Critique
    of CatB
    recommended

  • For those answering mostly  “C” with some D -- you are on your
    way to becoming NetSlave. Reading NetSlaves
    strongly recommended.

  • For those who answered mostly  “D,” -- you are Net Slave. Also
    you should probably contact a mental health professional immediately,
    reading NetSlaves
    probably will not help much, but you can try.





, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Saturday, December 27, 2008

Unix and Linux WebTech Humor: Bourne Shell Server Pages

Happy Saturday :)

Here's a little humour, from way back when, that's as true today as it was whenever it was originally published. It's a joke (or is it?) that puts forth a bold and visionary alternative to more complicated server page technologies, like ASP, JSP, etc. As the author suggests, it may all be a bunch of sh##, but that's by design ;)

This shining example of a Linux and Unix joke that you can actually put into practice, comes from Mark Lindner's Public Domain Software Site (It's actually his software, that he developed... not a bunch of unclaimed stuff that he put together in one place ;). To check out the original version of this page (including extra specifications, quotations and a bunch of other sh## <-- All this scatological prose will tie in nicely when you read the article, I promise ;), check out the SHSP Project Home Page. For this alone, Mark deserves a contribution. Check out his main page and, if you like this sh##, consider funding his future ...stuff ;)

Enjoy and remember to be kind to animals (even other humans ;)



Bourne Shell Server Pages



Introduction



Every work of art begins with a sudden flash of inspiration. In
my case, it all started when I received the following email from
my friend, Steve:






Hey Mark:

I just came up with a totally genius idea: Bourne Shell Server
Pages.

Why should shell languages be the only ones left out of the
${YOUR_LANGUAGE_HERE} Server Pages fad?

The way I see it, the advantages of SHSPs are as follows:


  1. runs everywhere (or at least everywhere on Unix and since I consider
    Windows to be nowhere...)
  2. slo-o-ow
  3. goofy

Now most rational people would consider points 2 & 3 to be drawbacks,
but as far as I can tell they are flat-out requirements for any popular
Web-based technology.

SHSPs are the Wave of the Future, so line up and sign up today!

Tune in next week when we invent 6502 Assembler Server Pages.


—Steve





Even as I was reading this email, my mind was racing,
subconsciously piecing together the implementation details. The
Bourne shell, it turns out, is quite well suited for this type of
task...much more so, I dare say, than Java or C. The solution turned out to be both simple and elegant.



This document chronicles my foray into the not-so-fascinating
world of ${YOUR_LANGUAGE_HERE} Server Pages
technology. All of the code I developed is available for free
download, so you can use Bourne Shell Server Pages to build your
very own killer Web application.



Implementation





The basic idea behind all server page technologies is this:
rather than writing code that generates an HTML document
on-the-fly by writing it out as a series of print
statements, you start with a "skeleton" HTML document and embed
the code right inside it. Voila! Instead of having a tangled,
unreadable, unmaintainable mess of HTML embedded in source code,
you have a tangled, unreadable, unmaintainable mess of source
code embedded in HTML.



Bourne Shell Server Pages are ordinary ASCII text files, with
the special extension .shit, which denotes
"Shell-Interpreted Template." The result of invoking the page
compiler on a .shit file, is, naturally, a shell
script. (It occurred to me that this file extension might seem
objectionable to some, but since it quite accurately—if
unintentionally—conveyed my sentiments toward Web technology in
general, I decided that it should be left unchanged.)



There are three basic elements that can be embedded in HTML:
blocks of code, expressions, and include directives. Each of
these is indicated by special HTML-like tags which are processed
by the page compiler to produce an executable entity (a Java servlet in
the case of JSP, for example), which, when run, produces the
resulting HTML document. I'll discuss each of these elements in
turn.


Code Blocks

Arbitrary blocks of code can be
inserted within an HTML document. Bourne Shell Server Pages
provides the <$ ... $> tags for this
purpose. Shell code that appears between these tags will be
executed, and the output of that code will be inlined in the
HTML in its place. For example:





<html>
<body>
<h1><$ echo "Hello, world!" $></h1>
</body>
</html>



The code can of course span multiple lines, and can be of
arbitrary complexity. What's more, all of the code blocks in a
given Bourne Shell Server Page are in reality part of the same
shell script, so they share a global state: for instance, a
block can reference variables that were assigned in an earlier
block.



Expressions
Arbitrary Bourne Shell expressions
can be inserted into HTML using the <` ... `> tags. These tags serve the same purpose as
do backticks in the Bourne Shell: the expression is
evaluated and the result is inlined in its place. For example,
to set the title of the document based on the contents of the
shell variable $title, one might do this:





<title><` $title `></title>



The above example assumes that a value was assigned to the
variable $title earlier in the page, namely, within
a <$ ... $> block.



Includes
Server pages can be composed of other
server pages using the include directive. The page compiler
evaluates pages recursively, so the nesting can be arbitrarily
deep. In Bourne Shell Server Pages, the tag <^ ... ^> indicates an include. For example the tag:





<^ more.shit ^>




will be replaced with the result of evaluating the Bourne
Shell Server Page more.shit.




The Bourne Shell Server Pages page compiler (shspc.sh),
which is tasked with converting .shit files into shell
scripts, actually consists of a series of sed
commands. These commands perform the following substitutions:


  • All contiguous segments of HTML are wrapped within here
    documents
    .

  • Include directives are replaced with recursive calls to
    shspc.sh to process the included Bourne Shell Server Pages.

  • Expression tags are replaced with a call to the shell's
    eval command.



The result is a script which, when interpreted by the Bourne
Shell, produces the desired HTML page to standard output.



Most other server page technologies rely on the presence of a Web
server, which is itself a complex piece of software that often
requires a substantial amount of maintenance. Bourne Shell Server
Pages has no such requirement, as it includes its own Web server,
implemented as a small shell script (in.httpd.sh) that is launched
from inetd. While certainly not a full-featured Web server,
it does quite an admirable job of serving HTML documents, images,
and of course, Bourne Shell Server Pages.



So What About Web Services?



How does the Bourne Shell Server Pages technology fit into the
bigger picture of Web Services? It's a legitimate question. For
that matter, what the hell are "Web Services" anyway?



I've read quite a bit about Web Services, and have had some
in-depth, first-hand experience with the technologies that form
their underpinnings. To the best of my knowledge, here is an
accurate definition of the term:




Web Services  noun  A software development
meme that espouses the notion of tying together disparate software
components via a crude, non-typesafe, remote procedure call (RPC)
mechanism that consists of sending and receiving data encoded in
an excessively verbose, plaintext format (XML) over a largely
inelegant, stateless file transfer protocol (HTTP).





Whew. That doesn't sound glamorous or exciting at all. It's not
even object oriented. There must be more to this than just
inferior reinterpretations of old ideas? Sadly, there isn't.



One particularly curious aspect of Web Services is that all
communication between components must take place over TCP port
80. The other ports (all 65,534 of them) constitute a veritable
Pandora's Box of perceived dangers, horrors and evils, and so
Thou Shalt Not Bind Them. It's painfully clear that Web
Services exists along three distinct axes: a technical one, an
emotional one, and a decidedly religious one.



It turns out that aside from the fact that it employs HTTP, the
network protocol of the World Wide Web, Web Services really has
nothing to do with the Web. "HTTP Services" would probably be a
more appropriate name for the technology. (Sure, that sounds
pretty silly...but it is what it is.)



The answer to the thesis of this section is thus very simple:
Bourne Shell Server Pages doesn't fit into Web Services,
any more than does any other ${YOUR_LANGUAGE_HERE} Server
Pages technology.


That being said, Bourne Shell Server Pages is a compelling
approach to the problem of providing Web access to existing
software components, which incidentally happens to be the major
focus of Web Services. Other Web technologies require complex
software plumbing in the form of adapters, bridges, and frameworks
to expose legacy software systems to the Web. However, due to the
radically expressive and flexible nature of the Bourne Shell,
Bourne Shell Server Pages makes this problem all but disappear:
any program on the system can be invoked from within a Bourne
Shell Server Page, its output becoming immediately available on
the Web.


For example, providing a Web-based search front-end to a large
CSV-data file is almost trivial with Bourne Shell Server Pages,
since one can rely on the presence of powerful utility programs
for manipulating data, e.g.:






<table>
<tr>
<th>Last Name</th>
<th>First Name</th>
<th>Phone</th>
<th>State</th>
</tr>

<$ IFS=','; grep -i $pattern datafile.csv | \
( while read lname fname phone state;
do
$>

<tr>
<td><` $lname `></td>
<td><` $fname `></td>
<td><` $phone `></td>
<td><` $state `></td>
</tr>

<$ ;; done ) $>

</table>





The true power of Bourne Shell Server Pages stems from the fact
that any program can be executed from within a Server Page,
with no intermediate "glue" logic required. Think of SHSP as "Web
Services without all the layers upon layers of crap."



Downloads





You, too, can build exciting dynamic Web content with Bourne
Shell Server Pages.

There are no pre-requisites for deploying
the technology other than a working Bourne Shell and a decent
operating system to run it on. (Those modest requirements,
unfortunately, rule out these products.)




shspc.sh - page compiler (710 bytes)
in.httpd.sh - Web server (834 bytes)




Installation is simple. Place the above files in
/usr/local/bin, and then install this href="http">http file in /etc/xinet.d (or the
equivalent). Then send a HUP signal to xinetd.

Securing the installation is left as an exercise for the reader.



Credits



Web technology is certainly a controversial topic. Despite the
fact that building Web applications is about as enjoyable as
having one's private parts caught in a cement mixer, it continues
to be an extremely popular software development paradigm. It's
been said that although there is a certain class of problems
for which Web technology is the appropriate solution, your problem
is almost certainly not among them, no matter what your problem
is
. But one would be hard-pressed to convince the industry of
it.



As do most belief systems, the Web technology religion has its
share of rabid followers. In order to avoid their wrath, I remain,
humbly and respectfully yours, Anonymous.



Some astute and enterprising readers have graciously submitted
bugfixes and improvements for SHSP. I have included these into the
official release.




, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Friday, December 26, 2008

The Twelve Days Of Unix

Hey There,

Thanks to the staggered posting system we have here, this joke is post-dated. Despite that, however, we here would like to have wished you and yours a Merry Christmas and/or Happy Holidays ;) Like the falling snow out here in Illinois (which will, by the way, never EVER stop until we're all buried alive) you can expect plenty of fluff over the holiday season, up to, and especially including New Years Day. ...okay, and almost definitely the night before that ;)

Here's a little something for you from Junauza.com from their Unix and Linux Christmas Jokes Page. Be sure to check it out for even more festive funnies and other assorted humor.

Hope you all enjoy it and Happy belated Holidays, again (or, perhaps, soon ;),



The Twelve Days of UNIX
Lyrics: Evan Leibovitch [evan@telly.on.ca]

On the first day I left it, my Unix gave to me:
A burnt-out V.D.T.

On the second day I left it, my Unix gave to me:
Two faulty tapes;
And a burnt-out V.D.T.

On the third day I left it, my Unix gave to me:
Three heads crashed;
Two faulty tapes;
And a burnt-out V.D.T.

On the fourth day I left it, my Unix gave to me:
Four bad blocks;
Three heads crashed;
Two faulty tapes;
And a burnt-out V.D.T.

On the fifth day I left it, my Unix gave to me:
Five core dumps;
Four bad blocks;
Three heads crashed;
Two faulty tapes;
And a burnt-out V.D.T.

On the sixth day I left it, my Unix gave to me:
Six bad controllers;
Five core dumps;
Four bad blocks;
Three heads crashed;
Two faulty tapes;
And a burnt-out V.D.T.

On the seventh day I left it, my Unix gave to me:
Seven blown partitions;
Six bad controllers;
Five core dumps;
Four bad blocks;
Three heads crashed;
Two faulty tapes;
And a burnt-out V.D.T.

On the eighth day I left it, my Unix gave to me:
Eight gettys dying;
Seven blown partitions;
Six bad controllers;
Five core dumps;
Four bad blocks;
Three heads crashed;
Two faulty tapes;
And a burnt-out V.D.T.

On the ninth day I left it, my Unix gave to me:
Nine floppies frying;
Eight gettys dying;
Seven blown partitions;
Six bad controllers;
Five core dumps;
Four bad blocks;
Three heads crashed;
Two faulty tapes;
And a burnt-out V.D.T.

On the tenth day I left it, my Unix gave to me:
Ten ports a-jamming;
Nine floppies frying;
Eight gettys dying;
Seven blown partitions;
Six bad controllers;
Five core dumps;
Four bad blocks;
Three heads crashed;
Two faulty tapes;
And a burnt-out V.D.T.

On the eleventh day I left it, my Unix gave to me:
Eleven chips a-smoking;
Ten ports a-jamming;
Nine floppies frying;
Eight gettys dying;
Seven blown partitions;
Six bad controllers;
Five core dumps;
Four bad blocks;
Three heads crashed;
Two faulty tapes;
And a burnt-out V.D.T.

On the twelfth day I left it, my Unix gave to me:
Twelve boards a-blowing;
Eleven chips a-smoking;
Ten ports a-jamming;
Nine floppies frying;
Eight gettys dying;
Seven blown partitions;
Six bad controllers;
Five core dumps;
Four bad blocks;
Three heads crashed;
Two faulty tapes;
And a burnt-out V.D.T.





, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Thursday, December 25, 2008

Happy Holidays - More ASCII Art

Happy Holidays to you all and Merry Christmas To Some Of You :)

Not much more needs to be said. I hope that you, like me, are spending some time with family and friends or, at least, writing a festive shell script ;)

As usual, click the picture below and it will appear larger than it is!


Happy holidays to one and all

Cheers,

NOTE: ASCII art copy write "Joan Stark" from her ASCII Art Christmas page

echo -ne "\n"
echo -ne " | | . , |_ . ,_ \_/ _,_, |__) _ _ _ _ / \,_\n"
echo -ne " |_|_|_|_/_)| |_|_| |_\_| | (_) |_|_ | _(/_(_|_(__(/_ \__/| |_\n"
echo -ne " __/\n"
echo -ne " __ ___\n"
echo -ne " (_ _ ,_-|-|_ _ ,_ _| | ,_ \_/ _,_, ,_ |__| _ _ ,_-|-\n"
echo -ne " (__(_|_|(_|_| |_ (_|_| |_(_|_ _|_| |_ | (_) |_|_|(_ | |_(/_(_|_|(_|_\n"
echo -ne "\n"
echo -ne " _..._\n"
echo -ne " _..._ .' _ '.\n"
echo -ne " .' _ './ (a) .-'-.\n"
echo -ne " / (e) .-'-. '-,-'\`\n"
echo -ne " | '-,--'\` /\n"
echo -ne " ; / ; |\n"
echo -ne " _ / |/ .\ _\n"
echo -ne " (_\ __..;__ \ ::\ ,/_)._\n"
echo -ne " _;\;\`\`\"\"----...___...---:\"\`\` .-'_ \`'. \ '::.()\__)\n"
echo -ne " (\())(\`._'- \`-._ ,:-\" '. '. - : ; '::\--;__\n"
echo -ne " _\()\`''----...__ \`-._ '-. '._ .' | .:::|()\_)\n"
echo -ne " (_/\`;-. jgs _ ()\`-. \`\"\"--...____..-'\` ;_ .:::| \`.\n"
echo -ne " \_) _;/_)/).;()'._ (\`\.-();/_).__ .::::/().-./\`)\n"
echo -ne " (_\\\`;(()(_/(()/(. __.--;();\`\_)()\__) .::()--;__:()\n"
echo -ne " '-;---()_;\_/()\"-._ (_/ (/ /()';();--._\_::'\`.();\_)\`\n"
echo -ne " ,() (_.-(); \._ \`\`\"\"(_.();_().(\`\.-();__)\`(_/ \`--.__\n"
echo -ne " (_/ (_/() \_) __/(_/|\();__\_)\` \_)\n"
echo -ne " \` (_/ \_)\`-._);\n"


, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, December 24, 2008

Taking Full Advantage Of "Who" On Solaris

Hey there,

First things first: Happy Christmas Eve :) Our apologies to our readers who celebrate different festivities this time of year. It's not that we don't want you to enjoy them (although our wishes are probably belated for some), just that we almost forgot about our own... What does that say about us? We don't know either ;) In the spirit of "it's never too late" - Happy Holidays (whatever they may be) from all of us :)

This post is more of an expansion, than a follow-up, to our May 2008 post on using who to find out what and when on Linux and Unix (picking only the options that appeared in most distro's version of "who"). It's also a bit more restrictive than our previous post, as we're restricting the discussion to newer Solaris' (8/9/10) implementation of the classic command (Okay, "according to Hoyle" (and I'm using that idiom loosely, since this post has nothing to do with the game of Whist ;), it can't be a classic for another 12 years, but if TNT can televise "new classics" that just came out last year, the meaning of the word has either been completely devalued or we're making liberal use of artistic license. We prefer the latter justification ;)

Solaris' implementation of who is a great deal better than in older versions, and turns what used to be a "mostly" informational command into a command that you can use to really get to the bottom of things. Following, a list of Solaris who's usage parameters and a brief discussion of each. One thing that should be noted is that the Solaris who command has no "help" switch (-h, --help, nothing), and it will produce output if you just run it without arguments. We usually go with "who -h" (since -h isn't an option) to get usage/help output. You'll have to subject yourself to the humiliating error message, but it's generally worth it ;)

Ladies and gentleman, The Options :)

1. Who straight-up (what better way to start a list of options than with the no-option option? This is what you get if you settle. Four columns: User ID (NAME), terminal (LINE), date (TIME) and local-screen/IP-address:

host # who
user001 console Dec 8 09:48 (:0)
user001 pts/3 Dec 8 09:50 (:0.0)
user001 pts/5 Dec 23 08:56 (10.99.99.99)


2. -a. This option is fairly obvious. It stands for "all" and is the equivalent to using "-Abdlprtu" which, the astute among you may have noted, does not comprise the entirety of the options available ;) We'll go over all the options that are lumped together in this option shortly (If I had more time to proofread, I wouldn't have made the word option a tag for this post ;).

host # who -a
. system boot Dec 8 09:46
. run-level 3 Dec 8 09:47 3 0 S
zsmon . Dec 8 09:47 old 307
LOGIN console Dec 8 09:47 old 331
user001 + console Dec 8 09:48 old 663 (:0)
user001 + pts/3 Dec 8 09:50 old 1015 (:0.0)
user001 pts/4 Dec 8 10:37 old 0 id= /4 term=0 exit=0
user001 + pts/5 Dec 23 08:56 . 17914 (10.99.99.99)
user001 pts/4 Dec 11 16:15 old 28003 id=ts/4 term=0 exit=0
user001 pts/6 Dec 19 10:39 old 11406 id=ts/6 term=0 exit=0
user001 pts/7 Dec 18 16:40 old 9997 id=ts/7 term=0 exit=0
user001 pts/8 Dec 16 14:05 old 5804 id=ts/8 term=0 exit=0


3. -b. This option will show you the last time the system booted. This can be very helpful (especially when used in combination with "last -x" to make use of last's full potential):

host # who -b
. system boot Dec 8 09:46


4. -d. This option purports to list out "dead" processes. The reason we phrase it in that way is that, generally, these processes (or pseudo terminals) may have been used before, but this doesn't mean that they're hanging around like a bunch of zombie processes. For instance, the following output lists 5 "dead" pseudo terminals, although none of them can be found in the output of either "ps" or "lsof" (???) In any event, it's a cool feature :)

host # who -d
user001 pts/4 Dec 8 10:37
user001 pts/4 Dec 11 16:15
user001 pts/6 Dec 19 10:39
user001 pts/7 Dec 18 16:40
user001 pts/8 Dec 16 14:05


5. -H. This one is a fantastically fun joyride through the land of the obvious. It forces who to print out the header for each column it reports on (although it does forget about the display/IP column noted in straight-up "who" output from point 1):

host # who -H
NAME LINE TIME
user001 console Dec 8 09:48 (:0)
user001 pts/3 Dec 8 09:50 (:0.0)
user001 pts/5 Dec 23 08:56 (10.99.99.99)


6. -l. Using who with the -l option lists out only "login" processes. Basically, it will only report on logins that are logged in (or appear to be logged in) to the localhost directly (no external pseudo terminals):

host # who -l
zsmon . Dec 8 09:47 old 307
LOGIN console Dec 8 09:47 old 331


7. -q. This will perform a quick who (only showing the NAME field), and is the only option that the -n option works with. If you use -n with -q, you can specify the number of returned processes you want to see per line of output, at most. By default, who -q tries to return as many results as possible on a single line:

host # who -q
user001 user001 user001

who -q -n 2
user001 user001
user001


8. -r. This option will let you know what run level your system is currently at (again, check out our previous post on using who for more specifics on all of the output "who -r" produces:

host # who -r
. run-level 3 Dec 8 09:47 3 0 S


9. -s. This option is considered the "short form," since it doesn't report any "time since last login," session activity status or PID output. who, run with this option alone is actually the default output. The one time this comes in handy is when you're using it with -a (and -H for the headers, if you want), and want to trim that output a bit. Otherwise, using this option wouldn't make sense, since you'd have to specify the flags to print the two fields you want removed ;)

host # who -s
user001 console Dec 8 09:48 (:0)
user001 pts/3 Dec 8 09:50 (:0.0)
user001 pts/5 Dec 23 08:56 (10.99.99.99)

host # who -asH

NAME LINE TIME
. system boot Dec 8 09:46
. run-level 3 Dec 8 09:47 3 0 S
zsmon . Dec 8 09:47
LOGIN console Dec 8 09:47
user001 + console Dec 8 09:48 (:0)
user001 + pts/3 Dec 8 09:50 (:0.0)
user001 pts/4 Dec 8 10:37
user001 + pts/5 Dec 23 08:56 (10.99.99.99)
user001 pts/4 Dec 11 16:15
user001 pts/6 Dec 19 10:39
user001 pts/7 Dec 18 16:40
user001 pts/8 Dec 16 14:05


10. -t: This option will show you all the times that your system clock was reset (and, yes, sometimes this output can be empty, for reasons that require no explanation ;)

host # who -t
host #


11. -T. This flag shows your tty status (referred to also, above, as session activity status). The + symbol indicates that the tty's status is "writable," the - symbol indicates that the tty's status is "not writable" and the ? symbol indicates general confusion ;) It just means that the system has no idea what the tty's status is, which generally means that it's hung:

host # who -T
user001 + console Dec 8 09:48 old 663 (:0)
user001 + pts/3 Dec 8 09:50 old 1015 (:0.0)
user001 + pts/5 Dec 23 08:56 . 17914 (10.99.99.99)


12. -u. This flag lists out (and I'm quoting from the "usage" output) "useful information." That isn't to say that any other output you can get from who is completely useless. Although, the terminology does seem to cast a shadow... ;)

host # who -u
user001 console Dec 8 09:48 old 663 (:0)
user001 pts/3 Dec 8 09:50 old 1015 (:0.0)
user001 pts/5 Dec 23 08:56 . 17914 (10.99.99.99)


13. -m. This flag limits the information to the current terminal session only. As you can see below, we're logged in using pseudo tty /dev/pts/5:

host # who -m
user001 pts/5 Dec 23 08:56 (10.99.99.99)


14. And, to begin the wrap-up, Solaris' who makes up for the fact that it doesn't have a built in handler to deal with being called as "whoami" by providing two different options to get that same information. In an alarming show of disregard for proper capitalization, both of these versions work ;) Note that this output is almost always exactly the same as the output from "who -m":

host # who am i
user001 pts/5 Dec 23 08:56 (10.99.99.99)
host # who am I
user001 pts/5 Dec 23 08:56 (10.99.99.99)


follow that up with the question that, statistically, follows "who am I?" most often, give it a little bit of "Talking Heads" flavour, and you've got yourself a command that's completely useless ;)

host # my God, what have I done?
-bash: my: command not found


15. Back off the Road To Nowhere (David Byrne, again. Make him stop!! ;)... Lastly (no pun intended, as you'll understand by the end of this paragraph), you can use who, using any combination of options (with the exception of -n, which only works with -q), and follow it all up with a different utmpx file (if, for instance, your old one got to big and you copied it off somewhere). Straight up who on Solaris makes use of /var/adm/utmpx, but you can tell who to use any utmpx-like file (including wtmpx, which can make the "who" command emulate the "last" command to a basic degree):

host # who /var/adm/wtmpx
root console Nov 6 13:39
root console Nov 6 13:48 (:0)
root pts/3 Nov 6 13:51 (:0.0)
root pts/3 Nov 6 14:38 (:0.0)
root pts/4 Nov 6 14:39 (:0.0)
user001 sshd Nov 7 09:48 (host.subnet.domain.com)
user001 pts/4 Nov 7 09:48 (host.subnet.domain.com)
user001 sshd Nov 7 09:54 (host1.subnet2.domain3.com)
user001 pts/5 Nov 7 09:54 (host1.subnet2.domain3.com)
...


Here's hoping today's post help shed a bit more light on Solaris' who options than the standard usage screen does, pointed out a number of reasons it can be a great tool to have in your troubleshooting arsenal and (perhaps) taught you a trick or two :)

Cheers,

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Tuesday, December 23, 2008

We'll Be Moving Soon: Unix and Linux Lame Encryption Decoded

Hey there,

There are still several hours left to go on the poll (which may be closed at the time of publishing) as I write this post, but (unless something insane happens) it looks as though our desire to remove ourselves from blogspot agrees with the opinion of about 77 percent of the folks who took the time to vote. I won't go through all the reasons things will be better once we get our own URL, although I will provide a handy link back to the original post where we laid out our reasons for moving on to our own host. Once things get ironed out (which should be well before we actually "do" move), the switch will be made. Hopefully, we'll be able to either maintain both sites at once or Google will be good enough to allow us to keep this domain and redirect from it for a while (for free or for a fee; however they do it).

NOTE: If you're one of the potentially thousands of people who stop by here every once in a while (and you haven't written in yet), we'll be putting up a limited subscription email form (you can always email us via the "Send Me A Comment" link at the top right of every page) as soon as possible. To be 100 percent clear, the "limited" part means that it won't be up forever AND that the subscription (as is so often referenced on many white/black/grey-hat marketing sites on the net) is "limited" in that you'll get one email (announcing our new address, and any other information pertinent to the move), after which you will be automatically unsubscribed. This is a one-shot email deal and your email address will not be sold or traded in any way. You always have the option of just following the site, as we should be posting information about the move, as we get it, right here. We probably have something to gain by collecting a bunch of email addresses, but if we ever want to sell you something, we'll let you know that we're trying to sell you something. Wanna buy a bridge? ;)

More on that, as it comes. It appears as though we have some work to do over the holidays. In the end, this move should result in a better experience for both the reader and everyone here, since we'll be able to avail ourselves of conveniences not possible under our current setup. We thank Google for helping us get our start for zero dollars per month, but now it's time for us to move on.

And, to wrap up, since nobody replied regarding our previous confusion and lame encryption treasure-hunt post, we got lucky and don't have to pony-up the prize right away ;) Although, rest assured, on our new site, there will be revision upon revision of our cable TV script, in real-time, as we're able to update it in our CVS repository. We have a personal stake in seeing this through because, after having it work so well for a month, many of us have become entirely dependant on it. Zap2it may have changed their format (and will probably do so again), but we can modify how we extract the information. Also, TVGuide would be a good alternate source of information for getting instant television listings.

BTW, the original script was our Google search index rank script, compressed and goofed around with (although not made unworkable) via methods posted in our series on security through obfuscation and, finally, made even more confusing using our script to do lame encryption using od. Check it out. Why would we lie ;)

In closing, we're looking forward to the move and will put up a mail form as soon as possible. If you prefer, again, just send us email via the "Send Me A Comment" link at the top right of every post if you wish to be notified when the terms of the move are finalized, or you just have a gripe ;)

Cheers :)

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Monday, December 22, 2008

Finding Your MSN/Live Index Rank From The Unix Or Linux CLI

Hey There,

Some of you may recall, a while back, when we did a post on how to find your Google search index rank from the CLI. In that post we took a look at another aspect of site SEO monitoring, where (rather than looking at Google's proprietary "PR" ranking system) we just went out and tried to fake being a human being entering keywords into the search page and then figuring out what result our desired URL actually showed up at (first - we stopped after the first positive result returned, since Google - like most places - will cut you off for, at least, a little while if they suspect you are a non-human ;).

Today's script is a variant on that, although we're going to be trudging through the landscape of MSN/Live's search engine.

IMPORTANT NOTE: Although this warning is on the original Google search rank index page, it bears repeating here and now. If you use wget (as we are in this script), or any CLI web-browsing/webpage-grabbing software, and want to fake the User-Agent, please be careful. Please check this online article regarding the likelihood that you may be sued if you masquerade as Mozilla.

This MSN script, of course, is only slightly different than the original Google script, although the differences are significant enough that I rewrote entire portions so I wouldn't have to back-translate code and then convert it and hope it worked ;) The script, itself, operates the same way our Google search index page rank script works, insofar as executing it from the command line goes. There are, at least, three different ways you can call it. The most basic being:

host # ./mrank www.yourdomain.com all these key words

It doesn't matter if they're enclosed in double quotes or not. If you "really" want to get the double quote experience, you just need to backslash your double quotes.

host # ./mrank www.yourdomain.com \"all these key words\"

Other ways include creating files with the URL and keyword information (same format as the command line) and feeding them to the script's STDIN:

host # cat FILE|./mrank
host # ./mrank <FILE



There is a little bugger that I need to remove from this script that incorrectly returns position #1 as #0 sometimes, but that should be easy to fix. I'd do it if I had the time. In fact, when I do, I'll repost the script and just include a notice in whatever that day's post is.

Now for the pictures :) Following are a few shots of MSN index rank checking as compared to Google index rank checking. Although most people will tell you it should be the opposite, it seems that we have a much better presence on Google than on MSN, after being in business (read: shamelessly self-promoting by answering question in forums, writing articles and submitting a billion sitemaps) for a year or so.

Following those, an EVEN BETTER set of pictures that made me scratch my head and mutter "WTF?" You'll see why. Let's just say that this site ranks lower on MSN for a certain keyword phrase than other sites that point to the post AND the first time our site comes up in that "double quoted" exact-string search, it's for an entirely unrelated post (???) Who has the time to wonder? ;) After that, way down at the bottom, we've somehow managed to remember to tack on the script.

By clicking any of the pictures below you are consenting to the use of our custom shrink ray, which will make each picture appear larger than normal, until you come back to the site ;)

mapquest cli search

adobe cli search

zombie search

And here's the MSN "managing swatch output" double-quote search that I still don't get.

swatch cli search

swatch search not our urls

And here we are, down a bit farther than any references to our post, and listing a page that doesn't contain the post, but probably has it listed in the blog archive on the sidebar ;)

swatch search our url

And, at long last, here's the script. Enjoy and have fun re-tooling it. Just be sure to double-check MSN to make sure you're not trying to fix a problem with your index ranking that exists by design ;)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# mrank - Get your MSN Live Search Ranking Index
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if [ $# -lt 2 -a $# -ne 0 ]
then
echo "Usage: $0 URL Search_Term(s)"
echo "URL with or with http(s)://, ftp://, etc"
echo "Double Quote Search If More Than 1 Term"
exit 1
fi

if [ $# -eq 0 ]
then
while read x y
do
url=$x
search_terms=$y
$0 $x "$y"
done
exit 0
else
url=$1
shift
search_terms=$@
fi

base=0
num=0
start=1
multiple_search=0
not_found=0

for x in $search_terms
do
if [ $multiple_search -eq 0 ]
then
search_string=$x
multiple_search=1
else
search_string="${search_string}+$x"
fi
done

echo "Searching For MSN Index For $url With Search Terms: $search_terms..."
echo

num_results=`wget -q --user-agent=Firefox -O - http://search.msn.com/results.aspx?q=${search_string}\&first=${start}|awk '{ if ( $0 ~ /of [0-9,]* results/ ) print $0 }'|sed 's/^.*of \([0-9,]*\) results.*$/\1/'`

while :;
do
if [ $not_found -eq 1 ]
then
break
fi
wget -q --user-agent=Firefox -O - http://search.msn.com/results.aspx?q=${search_string}\&first=${start} 2>&1|sed 's/<a href=\"\([^\"]*\)\"[^>]*>/\n\1\n/g'|sed -e :a -e 's/<[^>]*>/ /g;/</N;//ba'|sed '1,/See all.../d'|grep http|egrep -v 'cc.msnscache.com' 2>&1|sed '/search.live.com/d' |sed '/search.msn.com/d'|sed '/ocid=/,$d'|sed '/Developers | Help | Feedback/,$d'|awk -v num=$num -v base=$base '{ if ( $1 ~ /^http/ ) print base,num++,$NF }'|awk '{ if ( $2 < 10 ) print "MSN Index Number " $1 $2 " For Page: " $3; else if ( $2 == 10 ) print "MSN Index Number " $1+1 "0 For Page: " $3;else print "MSN Index Number " $1 $2 " For Page: " $3 }'|grep -i $url
if [ $? -ne 0 ]
then
let start=$start+10
let nexthopper=$start-1
if [ $nexthopper -ge 100 ]
then
not_found=1
if [ $not_found -eq 1 ]
then
break
fi
fi
let base=$base+1
num=0
else
break
fi

let sleep_time=${RANDOM}/600
echo "Not In Top $nexthopper Results: Sleeping $sleep_time seconds..."
sleep $sleep_time
done

if [ $not_found -eq 1 ]
then
echo "Not Found In First 100 Index Results!"
echo
fi

echo "Out Of Approximately $num_results Results"
echo
exit 0


, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Sunday, December 21, 2008

Deep Thoughts: What Does This Humor Have To Do With Linux Or Unix?

A glorious Sunday too you :)

Well, it's the end of yet another week, which means I'm slacking off and showcasing funny stuff from other places :) Today, even though I can't make the connection between this piece, Linux and/or Unix, I bring you the (long, but, hopefully, not forgotten) musings of Jack Handey. I first saw this stuff on Saturday Night Live, but, since the folks at PacketStorm Security have this in their humour archives, there's some connection between these hilarious one liners and the world of computing ;)

The original text is available at PacketStorm's Humor Archive.

Read, laugh and enjoy :)



Deep Thoughts
-------------
By Jack Handey

1) Dad always thought laughter was the best medicine, which I guess is why several of us died of tuberculosis.

2) Maybe in order to understand mankind, we have to look at the word itself: "Mankind". Basically, it's made up of two separate words - "mank" and "ind". What do these words mean ? It's a mystery, and that's why so is mankind.

3) I hope if dogs ever take over the world, and they chose a king, they don't just go by size, because I bet there are some Chihuahuas with some good ideas.

4) It takes a big man to cry, but it takes a bigger man to laugh at that man.

5) I guess we were all guilty, in a way. We all shot him, we all skinned him, and we all got a complimentary bumper sticker that said, "I helped skin Bob."

6) I bet the main reason the police keep people away from a plane crash is they don't want anybody walking in and lying down in the crash stuff, then, when somebody comes up, act like they just woke up and go, "What was THAT?!"

7) The face of a child can say it all, especially the mouth part of the face.

8) Ambition is like a frog sitting on a Venus Flytrap. The flytrap can bite and bite, but it won't bother the frog because it only has little tiny plant teeth. But some other stuff could happen and it could be like ambition.

9) I'd rather be rich than stupid.

10) If you were a poor Indian with no weapons, and a bunch of conquistadors came up to you and asked where the gold was, I don't think it would be a good idea to say, "I swallowed it. So sue me."

11) If you define cowardice as running away at the first sign of danger, screaming and tripping and begging for mercy, then yes, Mr. Brave man, I guess I'm a coward.

12) I bet one legend that keeps recurring throughout history, in every culture, is the story of Popeye.

13) When you go in for a job interview, I think a good thing to ask is if they ever press charges.

14) To me, boxing is like a ballet, except there's no music, no choreography, and the dancers hit each other.

15) What is it that makes a complete stranger dive into an icy river to save a solid gold baby? Maybe we'll never know.

16) We tend to scoff at the beliefs of the ancients. But we can't scoff at them personally, to their faces, and this is what annoys me.

17) Probably the earliest flyswatters were nothing more than some sort of striking surface attached to the end of a long stick.

18) I think someone should have had the decency to tell me the luncheon was free. To make someone run out with potato salad in his hand, pretending he's throwing up, is not what I call hospitality.

19) To me, clowns aren't funny. In fact, they're kind of scary. I've wondered where this started and I think it goes back to the time I went to the circus, and a clown killed my dad.

20) As I bit into the nectarine, it had a crisp juiciness about it that was very pleasurable - until I realized it wasn't a nectarine at all, but A HUMAN HEAD!!

21) Most people don't realize that large pieces of coral, which have been painted brown and attached to the skull by common wood screws, can make a child look like a deer.

22) If trees could scream, would we be so cavalier about cutting them down? We might, if they screamed all the time, for no good reason.

23) Better not take a dog on the space shuttle, because if he sticks his head out when you're coming home his face might burn up.

24) You know what would make a good story? Something about a clown who make people happy, but inside he's real sad. Also, he has severe diarrhea.

25) Sometimes when I feel like killing someone, I do a little trick to calm myself down. I'll go over to the persons house and ring the doorbell. When the person comes to the door, I'm gone, but you know what I've left on the porch? A jack-o-lantern with a knife stuck in the side of it's head with a note that says "You." After that I usually feel a lot better, and no harm done.

26) If you're a horse, and someone gets on you, and falls off, and then gets right back on you, I think you should buck him off right away.

27) If you ever teach a yodeling class, probably the hardest thing is to keep the students from just trying to yodel right off. You see, we build to that.

28) If you ever fall off the Sears Tower, just go real limp, because maybe you'll look like a dummy and people will try to catch you because, hey, free dummy.

29) Anytime I see something screech across a room and latch onto someones neck, and the guy screams and tries to get it off, I have to laugh, because what is that thing?

30) He was a cowboy, mister, and he loved the land. He loved it so much he made a woman out of dirt and married her. But when he kissed her, she disintegrated. Later, at the funeral, when the preacher said, "Dust to dust," some people laughed, and the cowboy shot them. At his hanging, he told the others, "I'll be waiting for you in heaven--with a gun."

31) The memories of my family outings are still a source of strength to me. I remember we'd all pile into the car - I forget what kind it was - and drive and drive. I'm not sure where we'd go, but I think there were some trees there. The smell of something was strong in the air as we played whatever sport we played. I remember a bigger, older guy we called "Dad." We'd eat some stuff, or not, and then I think we went home. I guess some things never leave you.

32) If a kid asks where rain comes from, I think a cute thing to tell him is "God is crying." And if he asks why God is crying, another cute thing to tell him is "Probably because of something you did."

33) Contrary to what most people say, the most dangerous animal in the world is not the lion or the tiger or even the elephant. It's a shark riding on an elephant's back, just trampling and eating everything they see.

34) As we were driving, we saw a sign that said "Watch for Rocks." Marta said it should read "Watch for Pretty Rocks." I told her she should write in her suggestion to the highway department, but she started saying it was a joke - just to get out of writing a simple letter! And I thought I was lazy!

35) One thing kids like is to be tricked. For instance, I was going to take my little nephew to DisneyLand, but instead I drove him to an old burned-out warehouse. "Oh, no," I said, "DisneyLand burned down." He cried and cried, but I think that deep down he thought it was a pretty good joke. I started to drive over to the real DisneyLand, but it was getting pretty late.

36) If you saw two guys named Hambone and Flippy, which one would you think liked dolphins the most? I'd say Flippy, wouldn't you? You'd be wrong, though. It's Hambone.

37) Laurie got offended that I used the word "puke." But to me, that's what her dinner tasted like.

38) We used to laugh at Grandpa when he'd head off and go fishing. But we wouldn't be laughing that evening when he'd come back with some whore he picked up in town.

39) I wish a robot would get elected president. That way, when he came to town, we could all take a shot at him and not feel too bad.

40) As the evening sky faded from a salmon color to a sort of flint gray, I thought back to the salmon I caught that morning, and how gray he was, and how I named him Flint.

41) If you're a young Mafia gangster out on your first date, I bet it's real embarrassing if someone tries to kill you.

42) Whenever I see an old lady slip and fall on a wet sidewalk, my first instinct is to laugh. But then I think, what if I was an ant, and she fell on me. Then it wouldn't seem quite so funny.

43) If you go parachuting, and your parachute doesn't open, and you friends are all watching you fall, I think a funny gag would be to pretend you were swimming.

44) When I was a kid my favorite relative was Uncle Caveman. After school we'd all go play in his cave, and every once in a while he would eat one of us. It wasn't until later that I found out that Uncle Caveman was a bear.

45) I think people tend to forget that trees are living creatures. They're sort of like dogs. Huge, quiet, motionless dogs, with bark instead of fur.

46) Happy Fun Ball
- -only $14.95-

Warning: Pregnant women, the elderly and children under 10 should avoid prolonged exposure to Happy Fun Ball.

Caution: Happy Fun Ball may suddenly accelerate to dangerous speeds.

Happy Fun Ball Contains a liquid core, which, if exposed due to rupture, should not be touched, inhaled, or looked at.

Do not use Happy Fun Ball on concrete.

47) Happy Fun Ball
- -only $14.95-

Discontinue use of Happy Fun Ball if any of the following occurs:
*Itching
*Vertigo
*Dizziness
*Tingling in extremities
*Loss of balance or coordination
*Slurred speech
*Temporary Blindness
*Profuse sweating
*Heart Palpitations

48) Happy Fun Ball
- -only $14.95-

If Happy Fun Ball begins to smoke, get away immediately. Seek shelter and cover head.

Happy Fun Ball may stick to certain types of skin.

When not in use, Happy Fun Ball should be returned to its special container and kept under refrigeration...

Failure to do so relieves the makers of Happy Fun Ball, Wacky Products Incorporated, and its parent company Global Chemical Unlimited, of any and all liability.

Ingredients of Happy Fun Ball include an unknown glowing substance which fell to Earth, presumably from outer space.

49) I'd like to see a nude opera, because when they hit those high notes, I bet you can really see it in those genitals.

50) It's really sad when a family can be torn apart by something as simple as a pack of wild dogs.

51) If they ever come up with a swashbuckling School, I think one of the courses should be Laughing, then Jumping Off Something.

52) When you're riding in a time machine way far into the future, don't stick your elbow out the window, or it'll turn into a fossil.

53) One thing kids like is to be tricked. For instance, I was going to take my little nephew to Disneyland, but instead I drove him to an old burned-out warehouse. "Oh, no," I said. "Disneyland burned down." He cried and cried, but I think that deep down, he thought it was a pretty good joke. I started to drive over to the real Disneyland, but it was getting pretty late.

54) A good way to threaten somebody is to light a stick of dynamite. Then you call the guy and hold the burning fuse up to the phone. "Hear that?" you say. "That's dynamite, baby."

55) Why do people in ship mutinies always ask for "better treatment"? I'd ask for a pinball machine, because with all that rocking back and forth you'd probably be able to get a lot of free games.

56) I'd like to be buried Indian-style, where they put you up on a high rack, above the ground. That way, you could get hit by meteorites and not even feel it.

57) If I lived back in the wild west days, instead of carrying a six-gun in my holster, I'd carry a soldering iron. That way, if some smart-aleck cowboy said something like "Hey, look. He's carrying a soldering iron!" and started laughing, and everybody else started laughing, I could just say, "That's right, it's a soldering iron. The soldering iron of justice." Then everybody would get real quiet and ashamed, because they had made fun of the soldering iron of justice, and I could probably hit them up for a free drink.

58) I bet when the neanderthal kids would make a snowman, someone would always end up saying, "Don't forget the thick, heavy brows." Then they would all get embarrassed because they remembered they had the big hunky brows too, and they'd get mad and eat the snowman.

59) Fear can sometimes be a useful emotion. For instance, let's say you're an astronaut on the moon and you fear that your partner has been turned into Dracula. The next time he goes out for the moon pieces, wham!, you just slam the door behind him and blast off. He might call you on the radio and say he's not Dracula, but you just say, "Think again, bat man."

60) Too bad you can't buy a voodoo globe so that you could make the earth spin real fast and freak everybody out.

61) The people in the village were real poor, so none of the children had any toys. But this one little boy had gotten an old enema bag and filled it with rocks, and he would go around and whap the other children across the face with it. Man, I think my heart almost broke. Later the boy came up and offered to give me the toy. This was too much! I reached out my hand, but then he ran away. I chased him down and took the enema bag. He cried a little, but that's the way of these people.

62) I wish I had a Kryptonite cross, because then you could keep both Dracula AND Superman away.

63) I don't think I'm alone when I say I'd like to see more and more planets fall under the ruthless domination of our solar system.

64) Living on Earth may be expensive, but it includes an annual free trip around the Sun.

65) Marta likes to talk about sensuality, but I don't think she would know sensuality if it bit her on the ass.

66) If you drop your keys into molten lava just let 'em go 'cause, man, they're gone.

67) Contrary to popular belief, the most dangerous animal is not the lion or tiger or even the elephant. The most dangerous animal is a shark riding on an elephant, just trampling and eating everything they see.

68) Once when I was in Hawaii, on the island of Kauai, I met a mysterious old stranger. He said he was about to die and wanted to tell someone about the treasure. I said, "Okay, as long as it's not a long story. Some of us have a plane to catch, you know." He stared telling his story, about the treasure and his life and all, and I thought: "This story isn't too long." But then, he kept going, and I started thinking, "Uh-oh, this story is getting long." But then the story was over, and I said to myself: "You know, that story wasn't too long after all." I forget what the story was about, but there was a good movie on the plane. It was a little long, though.

69) Is there anything more beautiful than a beautiful, beautiful flamingo, flying across in front of a beautiful sunset? And he's carrying a very beautiful rose in his beak, and also he's carrying a very beautiful painting with his feet. And also, you're drunk.

70) In weightlifting, I don't think sudden, uncontrolled urination should automatically disqualify you.



, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Saturday, December 20, 2008

More Funny Pictorial Linux And Unix Humor

Happy Saturday to you!

The holidays are looming larger every day now and, I don't know about you, but it's giving me a migraine. Lots of stress and craziness, mostly surrounding who's getting what for whom and constantly wondering if they're overdoing it or not doing enough. This is why, a long long time ago, when I was around 8 or 9 years old, I put a cash-only policy in place and let everyone in the extended family know about it. It's a simple way for me to get past all the holiday worrying about whether I'm going to get underwear, clothes or something else I might "need." To me, the holidays aren't just about giving. They're about giving things to people that they don't "need" (food) but just "want" (20 gallon drum of peanut butter ;)

It works out great for everyone else, too, because they always know that they're getting me exactly what I want. If they ever doubt that what they're giving isn't enough, the fix is simple: just include a little extra cash to go with the cash they're already getting me. If they ever fear that they may be giving me too much, my policy includes a "no such thing" clause, under which "too much" is specifically listed ;)

In any event, I hope your holidays, and the festivities that accompany them, are going well for you. This week's Saturday humor is, yet again, from the folks over at LinuxScrew.com. A few more pictures that should make you laugh-on-the-inside at worst ;) You can, of course, check out the page in its original context at LinuxScrew's Official Weekly Humor Page. Although the material here hasn't been altered (just the formatting), you should stop by there and check out their Linux and Unix humor archives. It's not "all" funny, but the stuff that doesn't make you laugh will make the stuff that does make you laugh seem even funnier! :)

For the record, may favorite picture is the "Russian Linux" mascot ;)

Enjoy and Cheers,




Firefox 4.0


Firefox 4.0


 


 


Debian beer


GNU/Beer


 


 


russian linux


Russian Linux


 


 


Linus Torvald’s toaster


Linus Torvalds’ toaster (runs Linux)




, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Friday, December 19, 2008

Confusion And Lame Encryption On Linux And Unix

Hey there,

Today, since the holidays are bringing me so much joy (heavy, heavy sarcasm ;), I thought it would be fun to write a post that makes use of some of the principles we've posted on this blog over the past year and combine them to create a treasure hunt of sorts. Actually, a lot of my holiday posts are going to be a little experimental. Since readership goes way down during the holiday season, it's a great time to write less conservative posts. Who knows what I'll be able to get away with? Not me; but I can't wait to find out :)

This isn't an ordinary treasure hunt, of course. No buried treasure or, really, anything of value at the end (Sometimes, I think I missed my calling in life when I passed on being a motivational speaker ;). Not much more than the pride in the knowledge that you figured it out and/or the passive-aggressive loathing you'll feel as you obsess over how I'm not so clever as I think, after all (although, hate me in moderation. You don't want to forget to stick another pin in my effigy before you burn it). This is quickly turning into a very dark post ;)

Of course, I, like most people I know who are around my age and have families of their own, dread the holidays. I love the fact that my kids are having fun, but the women (no offense, ladies...) in my family take this whole shebang way too seriously and the long hard journey to the day after the day after Christmas almost always includes at least one emotional breakdown, semi-psychotic episode or in-fighting about who did what better than whom and enlightening conversations that begin with "can you believe what he/she ...blah, blah, blah." I think if we all just got back to the commercialism that the holidays are really all about, everything would be perfectly fine and we'd all be happy again.

Here's a good tip, that I've found always serves the user well: Whenever I start to feel like I'm going to kill a relative during the peak of the festivities, I think to myself: What would Jesus do? I pause, reflect and then just figure "screw it" and try to get on with my life ;)

Anyway, before I lift your spirits up too high, here's what I've got for you today (hint: the next few lines are laden with clues. All that stuff above was crazy talk. And, if you're a family member of mine, of course I didn't mean a word of it ;) The jumble of numbers that looks like an octal dump was encrypted using a perl script that we posted to this blog some time ago. Of course the encrypted bash script has had it's name and attribution removed so those of you who can read octal won't get off easy ;) The script itself is all mashed together using a security method we've practiced in a few posts on this blog which makes scripts less likely to be mucked with by compressing them and, essentially, making them really long one-liners.

If you figure out what the script is and can email me the name (which is listed in the post that the obfuscated and lame-encrypted script originally came from; in the script headers - plain as day), I'll rewrite cabletv.sh for you, since zap2it changed their output format a month after our reader base (and this is impressive, I think) began apparently kicking the cr## out of them 24x7 (on top of the massive count of regular folk who used it through a web browser, as it was intended to be viewed). My apologies to zap2it, of course. Hopefully, they're not too upset. Really, I'd be ecstatic if I could get a Google PR of 7 and an Alexa Rank of 2,977 with 253,590 backlinks (I'm just guesstimating ;).

My time and effort is about the only thing I can afford to give this Christmas. All the money's going toward buying the kids' presents, because I want them to remember their childhood fondly (which gives me panic attacks ;) and not as a time of recession, conservation and perceived punishment for some huge corporation's massive screw-ups. Hopefully, my sheltering doesn't end up producing intolerable adults who'll scorn my existence later in life and/or try to have me declared incompetent so they can take control over my pocket change ;)

Happy Holidays to you all and enjoy the hunt :)

P.S. Check this post for a halfway decent way to download this code without having to manually convert it back from a single line! Look for the step-by-step method I use, when I can't get to the admin interface of this blog, in section 2.

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

043 041 057 142 151 156 057 142 141 163 150 012 012 151 146 040
133 040 044 043 040 055 154 164 040 062 040 055 141 040 044 043
040 055 156 145 040 060 040 135 073 164 150 145 156 040 145 143
150 157 040 042 125 163 141 147 145 072 040 044 060 040 125 122
114 040 123 145 141 162 143 150 137 124 145 162 155 050 163 051
042 073 145 143 150 157 040 042 125 122 114 040 167 151 164 150
040 157 162 040 167 151 164 150 040 150 164 164 160 050 163 051
072 057 057 054 040 146 164 160 072 057 057 054 040 145 164 143
042 073 145 170 151 164 040 061 073 146 151 073 151 146 040 133
040 044 043 040 055 145 161 040 060 040 135 073 164 150 145 156
040 167 150 151 154 145 040 162 145 141 144 040 170 040 171 073
144 157 040 165 162 154 075 044 170 073 040 163 145 141 162 143
150 137 164 145 162 155 163 075 044 171 073 040 044 060 040 044
170 040 042 044 171 042 073 144 157 156 145 073 145 170 151 164
040 060 073 145 154 163 145 040 165 162 154 075 044 061 073 163
150 151 146 164 073 163 145 141 162 143 150 137 164 145 162 155
163 075 044 100 073 146 151 073 142 141 163 145 075 060 073 156
165 155 075 061 073 163 164 141 162 164 075 060 073 155 165 154
164 151 160 154 145 137 163 145 141 162 143 150 075 060 073 156
157 164 137 146 157 165 156 144 075 060 073 146 157 162 040 170
040 151 156 040 044 163 145 141 162 143 150 137 164 145 162 155
163 073 144 157 040 151 146 040 133 040 044 155 165 154 164 151
160 154 145 137 163 145 141 162 143 150 040 055 145 161 040 060
040 135 073 164 150 145 156 040 163 145 141 162 143 150 137 163
164 162 151 156 147 075 044 170 073 155 165 154 164 151 160 154
145 137 163 145 141 162 143 150 075 061 073 145 154 163 145 040
163 145 141 162 143 150 137 163 164 162 151 156 147 075 042 044
173 163 145 141 162 143 150 137 163 164 162 151 156 147 175 053
044 170 042 073 146 151 073 144 157 156 145 073 145 143 150 157
040 042 123 145 141 162 143 150 151 156 147 040 106 157 162 040
107 157 157 147 154 145 040 111 156 144 145 170 040 106 157 162
040 044 165 162 154 040 127 151 164 150 040 123 145 141 162 143
150 040 124 145 162 155 163 072 040 044 163 145 141 162 143 150
137 164 145 162 155 163 056 056 056 042 073 145 143 150 157 073
156 165 155 137 162 145 163 165 154 164 163 075 140 167 147 145
164 040 055 161 040 055 055 165 163 145 162 055 141 147 145 156
164 075 106 151 162 145 146 157 170 040 055 117 040 055 040 150
164 164 160 072 057 057 167 167 167 056 147 157 157 147 154 145
056 143 157 155 057 163 145 141 162 143 150 077 161 075 044 163
145 141 162 143 150 137 163 164 162 151 156 147 134 046 150 154
075 145 156 134 046 163 141 146 145 075 157 146 146 134 046 160
167 163 164 075 061 134 046 163 164 141 162 164 075 044 163 164
141 162 164 134 046 163 141 075 116 174 141 167 153 040 047 173
040 151 146 040 050 040 044 060 040 176 040 057 157 146 040 141
142 157 165 164 040 074 142 076 056 052 074 134 057 142 076 040
146 157 162 057 040 051 040 160 162 151 156 164 040 044 060 040
175 047 174 141 167 153 040 055 106 042 157 146 040 141 142 157
165 164 042 040 047 173 160 162 151 156 164 040 044 062 175 047
174 141 167 153 040 055 106 042 074 142 076 042 040 047 173 160
162 151 156 164 040 044 062 175 047 174 141 167 153 040 055 106
042 074 057 142 076 042 040 047 173 160 162 151 156 164 040 044
061 175 047 140 073 167 150 151 154 145 040 072 073 144 157 040
151 146 040 133 040 044 156 157 164 137 146 157 165 156 144 040
055 145 161 040 061 040 135 073 164 150 145 156 040 142 162 145
141 153 073 146 151 073 167 147 145 164 040 055 161 040 055 055
165 163 145 162 055 141 147 145 156 164 075 106 151 162 145 146
157 170 040 055 117 040 055 040 150 164 164 160 072 057 057 167
167 167 056 147 157 157 147 154 145 056 143 157 155 057 163 145
141 162 143 150 077 161 075 044 163 145 141 162 143 150 137 163
164 162 151 156 147 134 046 156 165 155 075 061 060 060 134 046
150 154 075 145 156 134 046 163 141 146 145 075 157 146 146 134
046 160 167 163 164 075 061 134 046 163 164 141 162 164 075 044
163 164 141 162 164 134 046 163 141 075 116 174 163 145 144 040
047 163 057 074 141 040 150 162 145 146 075 134 042 134 050 133
136 134 042 135 052 134 051 134 042 040 143 154 141 163 163 075
154 076 057 134 156 134 061 134 156 057 147 047 174 141 167 153
040 055 166 040 156 165 155 075 044 156 165 155 040 055 166 040
142 141 163 145 075 044 142 141 163 145 040 047 173 040 151 146
040 050 040 044 061 040 176 040 057 136 150 164 164 160 057 040
051 040 160 162 151 156 164 040 142 141 163 145 054 156 165 155
053 053 054 044 116 106 040 175 047 174 141 167 153 040 047 173
040 151 146 040 050 040 044 062 040 074 040 061 060 040 051 040
160 162 151 156 164 040 042 107 157 157 147 154 145 040 111 156
144 145 170 040 116 165 155 142 145 162 040 042 040 044 061 040
042 060 042 040 044 062 040 042 040 106 157 162 040 120 141 147
145 072 040 042 040 044 063 073 040 145 154 163 145 040 151 146
040 050 040 044 062 040 075 075 040 061 060 060 040 051 040 160
162 151 156 164 040 042 107 157 157 147 154 145 040 111 156 144
145 170 040 116 165 155 142 145 162 040 042 040 044 061 053 061
040 042 060 060 040 106 157 162 040 120 141 147 145 072 040 042
040 044 063 073 145 154 163 145 040 160 162 151 156 164 040 042
107 157 157 147 154 145 040 111 156 144 145 170 040 116 165 155
142 145 162 040 042 040 044 061 040 044 062 040 042 040 106 157
162 040 120 141 147 145 072 040 042 040 044 063 040 175 047 174
147 162 145 160 040 055 151 040 044 165 162 154 073 151 146 040
133 040 044 077 040 055 156 145 040 060 040 135 073 164 150 145
156 040 154 145 164 040 163 164 141 162 164 075 044 163 164 141
162 164 053 061 060 060 073 151 146 040 133 040 044 163 164 141
162 164 040 055 145 161 040 061 060 060 060 040 135 073 164 150
145 156 040 156 157 164 137 146 157 165 156 144 075 061 073 151
146 040 133 040 044 156 157 164 137 146 157 165 156 144 040 055
145 161 040 061 040 135 073 164 150 145 156 040 142 162 145 141
153 073 146 151 073 146 151 073 154 145 164 040 142 141 163 145
075 044 142 141 163 145 053 061 073 146 151 162 163 164 137 160
141 147 145 075 060 073 145 154 163 145 040 142 162 145 141 153
073 146 151 073 154 145 164 040 163 154 145 145 160 137 164 151
155 145 075 044 173 122 101 116 104 117 115 175 057 066 060 060
073 145 143 150 157 040 042 116 157 164 040 111 156 040 124 157
160 040 044 163 164 141 162 164 040 122 145 163 165 154 164 163
072 040 123 154 145 145 160 151 156 147 040 044 163 154 145 145
160 137 164 151 155 145 040 163 145 143 157 156 144 163 056 056
056 042 073 163 154 145 145 160 040 044 163 154 145 145 160 137
164 151 155 145 073 144 157 156 145 073 151 146 040 133 040 044
156 157 164 137 146 157 165 156 144 040 055 145 161 040 061 040
135 073 164 150 145 156 040 145 143 150 157 040 042 116 157 164
040 106 157 165 156 144 040 111 156 040 106 151 162 163 164 040
061 054 060 060 060 040 111 156 144 145 170 040 122 145 163 165
154 164 163 040 055 040 107 157 157 147 154 145 047 163 040 110
141 162 144 040 114 151 155 151 164 042 073 145 143 150 157 073
146 151 073 145 143 150 157 040 042 117 165 164 040 117 146 040
101 160 160 162 157 170 151 155 141 164 145 154 171 040 044 156
165 155 137 162 145 163 165 154 164 163 040 122 145 163 165 154
164 163 042 073 145 143 150 157 073 145 170 151 164 040 060


, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Thursday, December 18, 2008

Quick XSCF Setup Walk-through For The Sun M4000 Server

Hey there,

You may have noticed (sometime in the last year or so ;) that I like to write about Sun computers and Solaris almost as much as I like to write about the many variants of Linux out there, without which I might not even have gotten interested in computing in the first place. Fortunately, or unfortunately, Sun still doesn't pay me residuals to write about their stuff. Admittedly I haven't, yet, screwed up the nerve to attempt brokering a pay-per-post deal with them (or anyone, for that matter). The end result would, of course, be the sound of a closing door, at best. Perhaps I underestimate the kindness of multi-million dollar corporations. One day, I'll do it, just for kicks (the worst that could possibly happen is that they'd crush me under their platinum boot-heels ;). You'll be the first to know the degree to which I'm ignored :)

In any event, since I use their products so often in my work, writing up how-to and tips-type posts regarding the use of them just makes sense. I think of myself as fairly creative (evidenced by the fact that I still exist, in spite of all the stupid, and hazardous, things I did in my youth ;), but every once in a while I have to defer some of the solid tech stuff to get it out of my system and give my noggin a break. By "solid tech stuff" I mean step-by-step walk-throughs of processes and practices, etc. Things that you can read in a manual that differ very little from what I've actually done. The facts (unlike the methods) can't be disputed. I'll admit that, for me, sometimes writing these kinds of posts is a bit mentally-boring but, if I started writing posts about all the goofy sh## I think about every day, this blog would quickly lose focus. I'd have to rename it "The Mongolian Cluster Fugg Menagerie" or something equally descriptive of what you could expect to not rely on reading every day ;) But, enough about my trivial problems. Life is generally good. My main gripe is that I have to work in order to be able to afford to feed my family. Crocodile tears ;)

This link will take you to a walk-through of the basic setup of the XSCF controller (Comparable to ALOM on some Sun systems, and the Service Controller on some others) on the new(ish) M4000 servers. Believe it or not, the hardest part was figuring out how to get started, even though I've worked on them, replacing system boards before. How embarrassing ;) These few pointers might be helpful to you (and, in the process, expose the possibly entertaining inner workings of a sleep-deprived mind ;)

1. Rather than a straight Ethernet connection, like on some of Sun's servers, the M4000 XSCF controller needs to be connected to with a serial cable (they include one with your purchase as a way of saying "Thank You" for spending 10's of thousands of dollars ;)

2. The correct port is not identified, typically, as a serial port. For ease of location, it's the second port (RJ45) in from the right, if you are looking at the server from the back.

3. Even though the XSCF has power running to it, you actually need to turn on the M4000 in order to do the initial setup through the M4000's serial connection to the XSCF (This one had me stumped for about a half an hour while I tried different cables and setups in HyperTerminal. There's just no substitute for reading the manuals that it takes forever to locate online ;) They didn't come with the physical product this time, so I just assumed a few things that ended up making my day go by much faster :)

4. For some reason, I couldn't get these basic manuals without a login to Sunsolve, although they're probably available on docs.sun.com somewhere. If you need to do any hardware work on M4000 or M5000 servers, I put up two essential guides (in PDF format) on one of my web hosting providers. These are worth their weight in paper ;)

The M4000/5000 Server Information Guide

The M4000/5000 Server Service Manual


Hope you can get some use out of those. My wife is obsessed with painting the kitchen - which is why I'm reasonably sure she won't ever read this - and I'm stuck with putting the kids to bed, which means an early night for me and more hallucinogenic dreams about Sponge Bob ;)

To leave you with another quick tip (if you need to install and get the Hell outta there ;), during our initial configuration, we intentionally opted not to set up the DSCP (Domain To Service Processor) protocol during our setup of the XSCF (Extended Control Facility). Once you have the basic networking set up, you can ssh in and set that up later. The DSCP is "important" because it's the protocol XSCF uses to communicate with your server (i.e. if it's not running, you can connect to your domain or console, but good luck getting it to manipulate your M4000 ;)

Hope you're having a peaceful evening, and that the official Sun XSCF Setup Documentation helps you some :)

Cheers,

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, December 17, 2008

Color Completion Using Zsh Modules On Linux Or Unix

Hey There,

How's this for a long-overdue follow up. If my powers of site searching don't fail me, we haven't touched on the Z Shell (zsh) since our post, from July 2008 about multiple stream output in zsh which links back to another post on zsh's extended globbing functionality. This post is more of a follow up to that one than the other, but who's counting? ;)

Here's another cool thing you can do with zsh that involves color. Personally, I'm happy with a two-color scheme that makes my eyes sizzle (like blue on red - or red on blue - both of which look somewhat 3 dimensional), but you have to make room for color in your life. It's not all black and white, even though it seems that way most of the time ;)

In order to make use of this functionality, you'll have to use the modular functionality of zsh. Today, we'll be looking at color completion, but this isn't to say that this is the limit of what you can do with the shell. Just the limit of what I can manage to type today... Doing color completion in zsh requires two modules: complist and colors. The first requires you to set the extendedglob option to true and the second isn't really necessary, unless you don't want to remember what number equals what color in the ansi color scheme. Below, some quick examples of how you can set this up at the command line (or put in your .zshrc if you find you like it and dont' want to hassle with this constantly). Note that both options require that you load up additional modules within zsh:

host # setopt extendedglob
host # zmodload -a colors
host # zmodload -a autocomplete
host # zmodload -a complist


The -a option isn't absolutely necessary, but keeps you from having to remember to type:

zmodload zsh/complist

and

autoload colors

which, as shown, require two different forms to enter. Since they may as well both be auto-loaded, zmodload's -a option makes sense (especially since it stands for autoload... kind of).

Here are a few handy aliases for you to use that can make your zsh shell experience more enjoyable (if color does that for you ;). Of course, we didn't make all these up ourselves. Most are standard and can be found in the manpage for zshmodules. If you forget this, since zsh has 500 separate sub-manpages (j.k. ;), you can always find the name for this manpage by just invoking:

host # man zsh

and you'll see the reference to the zshmodules manpage in there.

You'll note the use of "zstyle" (which is explained more-than-fully in the same zshmodules manpage) to set up all these shell color aliases (not all of these will work on all systems) and is not restricted to the use of list-colors and the zsh completion feature. There are a lot more ways, built in to zsh, to make use of color in the shell:

zstyle ':completion:*' list-colors ${(s.:.)LS_COLORS}
zstyle ':completion:*' list-colors 'reply=( "=(#b)(*$VAR)(?)*=00=$color[green]=$color[bg-green]" )'
zstyle ':completion:*:*:*:*:hosts' list-colors '=*=30;41'
zstyle ':completion:*:*:*:*:users' list-colors '=*=$color[green]=$color[red]'
zstyle ':completion:*' list-colors ''
<-- For when you get sick and tired of all the colors and just want them to go away!

Hope you enjoy these (if you use zsh - we're not trying to get you away from ksh or bash if you like 'em like most of us do) and have fun using colorful language the next time you code ;)

Cheers,

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Tuesday, December 16, 2008

Where Do We Go From Here? A Possible New Home For This Linux And Unix Blog

Hey there,

As promised today's post is going to be more of a toe-in-the-water Opinion/Editorial piece and (after reading, before reading or anywhere in between) if you could answer the new poll on the right hand column, we'd greatly appreciate it. Depending upon what the community wants, we may be making some drastic changes in the way this blog works and is presented (Nothing bad - our opinion - rather different for the better). Skip to the SUMMATION line if you don't want to read all the nitty-gritty details (this post is running over 2000 words in length before proofing).

PREMISE: Our main issue is that we're seriously considering moving away from Blogspot and transitioning our blog to an independent URL (e.g. http://www.linuxandunixmenagerie.com and/or http://linuxandunixmenagerie.net or http://linuxandunix.net which we've already snatched up). That being said, we want you to know that we're not considering this arbitrarily or on pure whimsy ;) The following is a listing of some of our reasons for considering this move.

1. COMMENTS AND DISCUSSION: This has been a big issue over the year and half that we've been operating and, although the point has become belaboured, we do understand. As things are today, we only accept comments via email for 2 reasons:

a. To avoid comment spamming: This was a problem when we first started out because too many unmoderated comments about porno, violence, drugs, alcohol and everything else reprehensible to Google ended up on posts about "How to change directories" and the situation was unacceptable. Of course, Google would notice and we'd have to worry about getting that NC-17 intro page they put up on some blogs.

b. To avoid comment spamming in Moderation: Same problem as above, except that the exact same situation ends up existing even when you decide to moderate your comments. Assuming that Google's bots are crawling your blog (which they, essentially, own), they also crawl the category, tag, etc directories, including the ones with comments waiting to be moderated. We can't control it because we can't modify the robots.txt file (or upload a .htaccess file) and, inevitably, we'd get a polite email about how we're not supposed to have any pages that violate Google policy, etc, because they crawled our posts (waiting to be moderated and, of course, rejected) before we could read them.

Email was a great solution for a while, but - as some of you out there may know - doing comments by email (although safer, since Google doesn't crawl our email account) is a burden. Now that we've been up for over a year, and this is our 452nd post, we get a lot of email and some of it gets lost in all the non-blog garbage mail. We leave spam filtering off so that we can be sure we don't delete any legitimate mail from users/readers of the site. This results in longer and longer wait times for responses and, sometimes, missing emails altogether until a future date when we circle back around and find a few that slipped by. Most folks are appreciative that we get back to them at all (though it may be months after the original email comment was sent - while other folks are getting same-day service), but it's becoming unacceptable to us.

Allowing moderated comments would relieve some of the pressure on us (allowing users to suggest and provide opinion - even argue if they want ;) and make the site much more organic and natural (since, as things stand now, we have to receive an email, reply to it to get proper attribution permissions and then manually post the comment to the end of any particular post). If our site was not on Blogspot, and was, instead, hosted on our own domain using an entirely separate service provider (not another blog hosting solution like WordPress.com, etc), this part of the site's functionality would improve significantly.

2. SCRIPTS, PICTURES, VIDEO AND OTHER ATTACHMENTS: Currently there is no way for us to make this work to our complete satisfaction. We believe that you'll agree that this is the case, also. For instance, just one such embarrassing moment appeared in our post on really simple steganography in which we posted a series of jpg images to illustrate how easy it was to hide a message in a picture. Although Google allows uploading of jpg's and gif's, it changed all of our original pictures into bmp's, and also our second batch. Of course, this conversion completely ruined the hidden message in the picture's binary structure since it was re-encoded! Video corruption has also been an issue so, when we do that, we host it on YouTube.

A much larger part of this issue is that this site contains a lot of scripts and, given the formatting options we have available (and no ability to attach text files or PDF's, etc), the process of transferring them from the site to your editor of choice can be extremely painful. For instance, if we don't want our code to all push to the left and look like garbage, we have to use the <pre> and <blockquote> tags in the supplied editor. However, as you may have noticed, if you just copy and paste the code from the web page to your editor of choice, it pastes the entire thing as ONE LINE!!! Of course, this makes sense since we're using the <blockquote> tag to retain formatting and indentation, but some of our scripts are very long and reformatting them, again, on your own machine can be a gigantic pain in the arse. We, of course, can access the original text and copy/paste it that way. However, in the instances where we don't have direct access to the admin interface for the blog, the following trick works (although you still end up jumping through hoops):

1. Pull up the page, with the script you want to use, in your web browser.

2. Save the file as a single web page.

3. Open that web page up in any editor that supports divs and/or tables (MS Word or Wordpad, Open Office Writer, etc).

4. Then copy and paste the code portion into a simple editor (like vi or notepad). Note that you shouldn't just resave your web page as a document, as this will also copy all the formatting markups, which is the same as copying/pasting that single line of garbage code by copying/pasting from web browser directly to editor.

Another gripe we have is that we can't include these scripts as attachments for easy download. This would make it so that the scripts would look nice on the web page AND you could download a clean copy and work with that directly.

Moving to our own blog host and domain would allow us to have greater control over the contents of our posts and, in the process, increase their accessibility for you, the reader. If we make this move, we would be able to host all of our own video, pictures, scripts, PDF's and any other multimedia content locally and provide download links for your ease of access and use. It would also allow us to create a "sister site" to the blog where we could provide these downloads on another domain (or subdomain) on the same server, where you could go to find scripts and download them (while obviating the need to host some stuff on SourceForge, which we haven't made public yet because it's just one more thing that we don't have the time to babysit - nothing against SourceForge, of course; it's just impractical to begin managing yet another site in order to keep this blog going). This move would also allow us to host our own CVS repositories and script updates in a more efficient and cohesive manner.

3. BRANDING AND SEO: This one shouldn't be of concern to a majority of readers (by which we mean anyone but the staff ;), but having a site with a domain name that comes somewhat close to the name of the site makes more sense. We're actually indexed very well by Google (Thank you :) and realize that the move would cause us to drop down more than a few pegs (at least for a while) but we can't honestly reconcile this site's Blogspot subdomain name (linuxshellaccount) with this site's actual name (The Linux and Unix Menagerie). Also, having our own site would allow us to include "Terms Of Service," "Privacy Policy" and other such necessary pages in a much less awkward way.

SUMMATION: Again, your input is valuable to us. We appreciate each and every one of our readers (even the ones who think we're a bunch of idiots ;) and want to know how you feel about this proposed change. If you have the time, consider this: Would you prefer that we remain on Blogspot and maintain the status quo or would you prefer that we move to our own self-hosted domain and provide you with a much greater level of service and accessibility that, quite frankly, we feel all of you deserve?

Please take a second and check a box in the poll on the right column near the top and let us know how you feel. We greatly appreciate your input and "will" consider it when making our ultimate decision. Seriously, if everyone wants us to stay on Blogspot, we'll stay here, even though it makes some of what we do more difficult (for us and for you), because the bright side to keeping things the same is that hosting on Blogspot doesn't cost us a dime (only our time and effort).

Also, if we do decide to make this change, and you would like to be informed of our new address, rest assured that we would update all of our social bookmarking sites, forum signatures, Technorati, LXer, etc, and try to work out a deal with Google where we can forward the linuxshellaccount subdomain to our new domain for a while. We would also provide a separate "one-shot" mailing list that you could subscribe to so that we could send you a notification of the new site's address and other details to make your experience more enjoyable and/or efficient. Also, although we have a "Missions and Policies" statement at the top right of every page, we don't have a separate and specific privacy policy listed. However, we can guarantee you that, if you were to sign up for our notification list, we would send you an email with all the information you need, at the time of the move, and never mail you again (except under special circumstances - read on). It's our unwritten policy (actually written in most of our email replies over and over and over again ;) that we will never use your email for any sort of advertising, we won't sell it, we won't trade it, we won't use it to SPAM you with updates about the site (You can subscribe to FeedBurner, on any page, for that ;) and we won't ever write you again unless you initiate contact (whereby all the aforementioned principles still hold true, except for the one where we say we won't ever write you again. We will, however, never reply more than once to any email sent to us). Also, to be clear, if we provide a form whereby you can sign up to be notified of the site's URL change, you will not be signing up to a "mailing list" (insofar as that term is used in Internet Marketing circles). You'll only be adding your name to a list of emails that will be mailed to once, and once only. To carry the analogue just one step further, you will be "unsubscribed" the moment that email is sent. You won't have to jump through hoops or click any back-links to prevent us from bothering you in the future ;) We don't like getting scammed any more than you do and guarantee that your anonymity will not be compromised unless you want it to be. We hold your right to privacy as sacrosanct and will not pull any other email-list stunts that we haven't thought to list here. We have no interest in making your life any harder or filling your email box with ludicrous pap. ...however, if you want to learn how to turn your PC into a self-operating virtual ATM using our proven turnkey system that allows you to make millions while you space out watching TV... ;)

We hope this post has effectively covered what we are looking to do to improve the blog for your ease-of-use, enjoyment and utility and look forward to tallying your poll results. As always, you can send us a comment (via the "Send Me A Comment" link at the top right of every page) regarding this post, just as with any other.

Cheers,

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Monday, December 15, 2008

Updating Unix And Linux Passwords Via The Web Browser

Hey there,

QUICK NOTICE FOR REGULAR READERS: We didn't want to lead off the week with this, but tomorrow's post will be an op-ed bit having to do with this blog, its contents and some of the shortcomings of, shall we say, "where we are" (literally). We'll also be putting up a poll to confirm that our opinion is the same as yours (we hope ;) To cut it short, if you find downloading scripts form this site a HUGE PITA, please glance at tomorrow's post and anonymously answer the poll if possible. Thanks :)

This Monday's script is actually a web-page frontend (CGI, to be exact) to make use of another script. That script can be any you want, but, if you like, we published an Expect script some time ago to do network wide updates from a single source that can be modified to restrict it to password changing only.

Now, we'll go on record right away (admittedly, this is the second paragraph, so we're lagging behind already ;) by stating that we do not specifically endorse this method of password changing. It's very convenient (for users both trustworthy and malicious) and does the job, but, use of this script in a secure environment (or any environment that requires protection) is not recommended. Not only does this script open up many potential security holes by allowing access to system commands (albeit via another script that gets called, so it's not "unbelievably" easy to misuse), it will almost guarantee that you won't get your Sarbanes Oxley compliance certificate :) That being said, if you have a small internal net (quarantined from production and other environments) this can be a handy way to do your updates. In the worst case, you can just forego the "web experience" and do your mass password updates using Expect (or any other tool) from the CLI.

Since glitz seems to be the order of the day, the web frontend is presented here and now. We've left out the frontend form where you would enter your name, current password and new password, as creating one of these would probably be specific to your organization and simple enough to knock out. If this is a pain for any of you, just write in (via the "Send Me A Comment" link at upper right corner of every page) and we'll be happy to write one up and post it here.

Hope you enjoy it but, as always, exercise prudence, caution, and all those other qualities that will keep you from getting into a situation that involves immediate termination of employment :)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# change_password.cgi
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

&read_parse;

$login = $FORM{'mylogin'};
$password = $FORM{'mypwd'};
$password =~ s/([;?!])/\\$1/;
$new1 = $FORM{'mynewpwd1'};
$new2 = $FORM{'mynewpwd2'};
if ( $new1 ne $new2 ) {
¬_match;
}
$new1 =~ s/([;?!])/\\$1/;
$new2 =~ s/([;?!])/\\$1/;
$nickname = &get_gecos;
$litmus = 0;
$date=`date +%m%d%y`;
open(PASS, "</etc/passwd");
@etcpass = <PASS>;
close(PASS);

foreach $line (@etcpass) {
if ( $line =~ /^$login/ ) {
$litmus = 1;
}
}

if ($litmus == 0) {
print "Content-type: text/html\n\n";
print "<html>\n";
print "<head>\n";
print "<title>Error</title>\n";
print "</head>\n";
print "<body BGCOLOR=\"#000000\" TEXT=\"#00FF00\" LINK=\"#FFFF00\" VLINK=\"#0000FF\" ALINK=\"#BECD8C\">\n";
print "<p>\n";
print "<br><hr><br>\n";
print "<p>\n";
print "<center><FONT COLOR=\"#FF0000\"><H3>Bad User Name</h3></FONT></center>\n";
print "<p>\n";
print "<br>\n";
print "<p>\n";
print "<center>The Name $login doesn't have a login yet!</center>\n";
print "</body>\n";
print "</html>\n";
exit;
}

system("/path/to/your/mass/password/changing/program $login $password $new1");

print "Content-type: text/html\n\n";
print "<html>\n";
print "<head>\n";
print "<title>All Done</title>\n";
print "</head>\n";
print "<h3>\@hummer</h3>\n";
print "<body BGCOLOR=\"#000000\" TEXT=\"#00FF00\" LINK=\"#FFFF00\" VLINK=\"#0000FF\" ALINK=\"#BECD8C\">\n";
print "<p>\n";
print "<br><hr><br>\n";
print "<p>\n";
print "<center><FONT COLOR=\"#FF0000\"><H3>Fine Job, $nickname!</h3></FONT></center>\n";
print "<p>\n";
print "<br>\n";
print "<center><FONT COLOR=\"#FF0000\"><H3>Assuming that your intial password was valid, your Password has been changed!</h3></FONT></center>\n";
print "<center><FONT COLOR=\"#FF0000\"><H3>Remember, also, that all passwords must contain at least one numeric or non-alpha character in order to be accepted.</h3></FONT></center>\n";
print "<p> \n";
print "</body>\n";
print "</html>\n";
exit;

sub not_match {
print "Content-type: text/html\n\n";
print "<html>\n";
print "<head>\n";
print "<title>Error</title>\n";
print "</head>\n";
print "<body BGCOLOR=\"#000000\" TEXT=\"#00FF00\" LINK=\"#FFFF00\" VLINK=\"#0000FF\" ALINK=\"#BECD8C\">\n";
print "<p>\n";
print "<br><hr><br>\n";
print "<p>\n";
print "<center><FONT COLOR=\"#FF0000\"><H3Passwords Don't Match!</h3></FONT></center>\n";
print "<p>\n";
print "<br>\n";
print "<p>\n";
print "<center>The Password $new1 doesn't match $new2</center>\n";
print "</body>\n";
print "</html>\n";
exit;
}

sub read_parse {
read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});
@pairs = split(/&/, $buffer);
foreach $pair (@pairs) {
($name, $value) = split(/=/, $pair);
$value =~ tr/+/ /;
$value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;
$FORM{$name} = $value;
}
}

sub get_gecos {
$boner = `grep $login /etc/passwd`;
@stuff = split(/:/, $boner);
return $stuff[4];
}


, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Sunday, December 14, 2008

Unix And Linux Humor In Pictures - Does Anyone Else Really Exist?

A glorious Sunday to you,

This week's Sunday humour comes from UnixBigots.org's Humour Section. It's pretty funny and should take up a whole lot of space ;) Visit their main site to check out all the other interesting and fun stuff they have going on and, of course, if you want to see the original version of this page (although - I kept the disclaimer in my reprint - they actually got this from somewhere else, also ;), you can check it out on their site under Back Orifice 2000 Screen Shots Of The Week. And (to explain for the benefit of any fathers like me ;) the orifice to which they're referring (or the "back orifice," to be more specific) is the just name of an old exploit. If the term is being used as double entendre, I'll just live in delusion and assume that it really isn't ;)

Have a great and, hopefully, restful, day.

Cheers,



Back Orifice 2000 Screen Shots of the Week



NOTICE

The files contained here are a mirror of the original files at altern.org. They are not sponsored, nor were originally obtained in any way by myself.





Screen Capture of the Week
Please allow images to load... they're worth it.










SEND EMAIL





, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Saturday, December 13, 2008

The F Word Still Remains - More Linux And Unix Humor

Happy Saturday,

For this weekend's Linux and Unix humor kick-off, I'm going back a ways to a piece I half-hesitate to touch because I don't want the Moral Majority slapping "Content Warning" stickers on this blog. Still it's funny and interesting, so funk 'em (this trick is used by many rock bands and the FCC is okay with it, so I think I'm on solid ground using it in place of the other f word ;)

The piece below is from Sean Dreilinger's Web Page under the heading of the Linux Kernel Fact Center. This original page also includes a link to another citizen who's carrying the torch over at Vidarholen.net. A few interesting graphs there, too, like the one below (not actually "taken" from his page, but linked back to with an "img src" tag).



Also, just for yucks, I gave this a shot on a Linux box (SUSE 8) from work which indicates that even folks who write for a proprietary Linux OS manage to slip one by now and again ;) ...sanitized version...

host:~ # uname -a
Linux host 2.4.21-251-smp #1 SMP Thu Sep 23 17:22:54 UTC 2004 i686 unknown
host:~ # find /usr/include|xargs grep -i f###
/usr/include/linux/netfilter_ipv4/ipt_limit.h: /* Ugly, ugly f###er. */
/usr/include/linux/netfilter_ipv6/ip6t_limit.h: /* Ugly, ugly f###er. */


Hope you enjoy the article and be sure to visit The Linux Kernel Fact Center for more of the same, and some other different stuff :)

Cheers,



Linux Kernel F Count




in 1998, glenn forwarded me a message from the linux kernel hackers
list. this message suggests that the linux
kernel source is laden with profanity, the word f### in
particular. my curiosity led me to untar, grep, and
wc through a selection of linux kernels, in search of
love, f###, and some other trends in the evolution of linux.




it would appear to be pretty difficult to get a linux
kernel hacker to swear (or express any form of love---lovable,
lovably, love, loved, lovelier, lovelies, loveliest, loveliness,
lovelorn, lovely, lover, lovers, loves, loving, lovingly,
etc.)
in the linux kernel source. from the chart below you have to wonder
what was going on around kernel 2.1.50 :-).



after the cuss-word novelty wears off, some of the other information is pretty interesting --
such as the the gradual increase in source tree size, file,
line
and word counts with each new kernel over the past
six-seven years. i wonder how this sort of growth compares to the
evolution of a commercial operating system kernel.



when you put it all together and
consider that in the present
kernel source, 275 credited kernel authors wrote over 1,895,964 lines of
code and documentation (6,856,723 words) and only managed to cuss 29 times
-- it would seem that these developers show incredible self restraint when
expressing frustration, are complete professionals, are sedated, or are just
enjoying what they do!



update 2006-11-01: thanks for digging this
document; a modern-day continuation of this novelty is maintained by vidar
holen
.



, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Friday, December 12, 2008

Work Imitates Life On Unix or Linux - Some Surrealism For A Change

Hey there. Thought I'd do something insane today to break up the monotony. Something along the lines of our older nonsense post entitled Linux vs. Unix: Sins Of The Father?

As a special note, this isn't typical of the day in the life a Unix or Linux Admin - Enjoy :)

home # ssh sweatshop
The integrity of the workplace 'sweatshop (10.99.99.99)' can't be established.
RSA key fingerprint is e3:6e:86:58:cd:38:dd:71:70:60:ea:ea:82:92:fb:72.
Are you sure you really can't find a better way to spend your day (yes/no)? yes
Warning: Permanently extended your at-will employment at "sweathouse"
Password:
Last login: Thu Dec 11 15:16:43 2008 from home
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
-------------------------------------------------------------------
Warning: Unauthorized access to this facility is forbidden and will be
prosecuted by law! By entering this place of employment, you agree that
your actions can be monitored if unauthorized usage is suspected and/or
the management needs a good laugh while they wipe their collective arse
with hundred dollar bills. Have a great day and "earn" that buck!

sweatshop # ./security -wtf -fingerprint "Do you really not recognize me yet?"
security: option -wtf not appreciated. Please refraing from using foul
abbreviations or a fingerprint scan will be the least of your worries.
./security --help suggests ways in which you can express your displeasure
in a more productive and meaningful manner.
sweatshop # ./find /office -name "today's cubicle" --user mgolvach|xargs sitdown && space_out && try_to_remain_unnoticed
try_to_remain_unnoticed: today's cubicle: No such f@@@ing way - read /etc/motd for more information
sweatshop # cat /etc/motd
Get that cat outta here! Get back to work. Check your job queue asap. Deliverables with a deadline of tomorrow have been moved up by 8 working days.
sweatshop # ./jobs
[1]- Idle submit_mind_numbing_triplicate_requests &
[2]- Idle move_the_stuff_mgmt_had_you_move_yesterday_back_to_where_it_was_orginally &
[3]- Running monster_job_search_daemon
[4]- Running the_boss_-_off_at_the_mouth_that_is
[5]+ Waiting sleep 3600
sweatshop # fg %1
please refer to subsection a of paragraph b in volume 3 of the rules and regulations manual as pertains to the filing of claims which fall under international law and are therefore subject to any, or all, provisions listed in paragraph 3 subsection c of the company standards and practices manual. File all papers appropriately, as incorrect filing may result in your having to submit additional paperwork in the form of
^C
sweatshop # ./jobs
[1]- Idle move_the_stuff_mgmt_had_you_move_yesterday_back_to_where_it_was_orginally &
[2]- Running monster_job_search_daemon
[3]- Running the_boss_-_off_at_the_mouth_that_is
[4]+ Waiting sleep 3600
sweatshop # fg %2
command: mv technical_documentation technical_docmentation.bak
warning! Unauthorized use of "mv" detected. This incident will be reported!
command: help
warning! Unnecessary use of "help" command. Now someone who really needs it will be given the short-shrift. This incident will be reported!
command: quit
...backgrounding process move_the_stuff_mgmt_had_you_move_yesterday_back_to_where_it_was_orginally
sweatshop # ./jobs
[1]- Idle move_the_stuff_mgmt_had_you_move_yesterday_back_to_where_it_was_orginally &
[2]- Running monster_job_search_daemon
[3]- Running the_boss_-_off_at_the_mouth_that_is
[4]+ Waiting sleep 3600
sweatshop # kill %1
[1]+ Defunct move_the_stuff_mgmt_had_you_move_yesterday_back_to_where_it_was_orginally &
sweatshop # ./jobs
[1]- Running monster_job_search_daemon
[2]- Running the_boss_-_off_at_the_mouth_that_is
[3]+ Waiting sleep 3600
sweatshop # fg monster_job_search_daemon
...Still nothing. Suck it up for a little while longer! You can do it! Believe in yours^C
sweatshop # ./jobs
[1]- Running the_boss_-_off_at_the_mouth_that_is
[2]+ Waiting sleep 3600
sweatshop # fg the_boss_-_off_at_the_mouth_that_is
...and another thing, you worthless piece of human^C
waste, how many times do I have to ask you to^C
fill out these forms correctly? You're possibly the only person I've met who's dumber than a sack of wet hammers. If I had my^C
way, you'd be hitting the bricks this aftern^c
oon
sweatshop # kill kill kill %1
[1]+ Terminated the_boss_-_off_at_the_mouth_that_is
sweatshop # ./jobs
[1]+ Waiting sleep 3600
sweatshop # fg sleep
Broadcast Message from slavedriver@sweatshop
(/dev/pts/0) at 16:12 ...

user mgolvach - you are being notified. Please wake up, do your job or face severe penalties in the form of a costly and time-consuming HR process designed to eventually sever your employment!
Mgmt
sweatshop # echo "Blow me"
blow me
sweatshop # ./shutdown -I_can't_believe_that_last_command_didn't_get_noticed
sweatshop # exit
sweatshop # error in file ".logout" : You may never leave. And your pay grade has been lowered.
sweatshop # quit
sweatshop # quit: command not found
sweatshop # kill -9
sweatshop # quit: command suddenly recognized again. Have a nice day :)
home # connection to sweatshop lost
home # at 6am
warning: commands will be executed using /bin/sh
at> wake_up_and_do_it_again
at> <EOT>
job 898 at 2008-12-12 06:00
home # init S


, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Thursday, December 11, 2008

Unix And Linux Factoids - Your Time To Live Is Gonna Come

Hey there,

Today we're going to start a new random thread of posts dealing with common commands and how they work. The meat of today's post is near the end, as the explanation, when I get around to it, is fairly simple and easy to both convey and understand. The majority of this page will be an introduction to this series of posts, with some philosophy mixed in (which may upset certain people) and a brief discourse on knowledge (what it means to have it and why it's probably the worst measure of a person you could ever use).

Farther down this post, we'll take a look at a common command, named traceroute (generally on Linux and Unix) or tracert (On Windows; I have idea about Mac ;) that makes up part of any sysadmin's network troubleshooting arsenal (hint: do a find for "traceroute" in your browser if you want to skip the preceding discourse!). Of course, not everyone knows how this command works (and we're talking about the very basics). That's nothing to be ashamed of, of course. Everything you know now, you learned at some point in your life (unless you're of that rare breed that emerges from the birth canal engaging adults in stimulating discussion of the experience ;). Before you learned it, you didn't know it. It's hard to fault someone for not knowing something they don't know already, even if your predisposition is to think that they should. For instance, if you were to approach someone on the street to ask them what time it was and they replied to you in an obscure Greek dialect (actually, any Greek, for me would be... Everyone knows where that lame joke was going ;), you wouldn't consider yourself stupid for not understanding them. If you'd taken the time to learn how to speak Greek, studied it until you became fluent and still had no idea what they were saying to you, I guess the situation could be awkward. Kind of like this introduction :)

The point I'm trying to make in the downward-trajectory of this introduction to these new factoid (or bits-of-interest) posts is that one should never become too socially intimidated, or otherwise societally fettered, that they just stop learning for fear of someone else finding out they didn't already know something. There is an old saying (which I'll paraphrase) which is about as true as true gets: The most agile-minded, clever and knowledgeable of us are those who are not afraid to ask for clarification or explanation when they don't understand. Keep a dictionary on hand when you're reading a book. There's no shame in understanding what you're reading. Dictionary's exist for a reason ;)

In my personal and professional life, I do my best to avoid highly intelligent individuals with extremely low self-esteem, as their company never seems to foster any sort of learning, sharing or growing. Better, still, I should hang out with folks who (regardless of their level of intelligence) are curious, interested, interesting and non-judgemental. It's been my experience that if you stay away from the "better" class of people, your life will be much more rewarding and enjoyable. The one thing that you must remember about people who look down at you is that they're, generally, consumed with the fear that you'll find out how much they, themselves, don't know or understand. Living life with one's guard constantly up is to, essentially, ensure one's own death (mentally, physically, spiritually; take your pick or go for option 4).

I've often found the above-mentioned argument the most decisive in shutting down, and casting aside, negative influences in my life. It bears repeating because it's so obvious and impenetrable: No matter who you are, you didn't know anything until you learned it. Everything, from your native tongue (essential in ensuring your survival and, possibly even more importantly, being able to ask where the bathroom is ;) to the principles of quantum physics, are all perfect examples of something you once didn't know of or understand. Again; no shame in that. People get an education (in schools, at the library or on the streets - all the same) for a reason. Just because someone you know can quote William Shakespeare (and does so at any and every opportunity) does not mean that you're any less capable because you aren't totally familiar with his work (William's, that is). I could write a book about this, but it would probably become self-parodying at some point. Here's a quick, and abbreviated, list of a few other ways you can spot folks who will suck the life out of you, and how to avoid them. Better yet, if you're a glass-half-full type, attempt to start a dialogue with them regarding the issue at hand - some folks only "seem" to be self-important and megalomaniacal. If you can reason with someone, you may end up enjoying a friendship with them that might have otherwise been lost. It's important to remember that immediately dismissing people who irritate you is exactly what irritates you about them in the first place:

1. If someone claims to "only read the best books" or "only dine at the finest restaurants," or other such snobbery, be sure to ask them who supplies them with their opinions? Having only read, or experienced, the best and the finest, they can't have possibly ever exposed themselves to anything lesser and, therefore, can't possibly argue the merits of the finer things with an ounce of credulity ;)

2. If someone treats the foreign help as if they're idiots because they don't speak English (or whatever your native language may be), be sure to find out exactly how well they speak that foreign tongue. If they're fluent in it, they may just be cruel, since they could converse with "the help" in that foreign language and avoid all the insults. If they can't speak the foreign language, they have no basis upon which to judge the other person's worth. For instance, I've known many Spanish, Polish, Russian, Hungarian, etc, people who know how to speak quite a bit more English than I can speak (or even pronounce correctly) in their language. When you think about it that way, it makes it harder to justify the false belief that the person who doesn't speak your language is any less intelligent than you. If anything, when it comes down to multilingual capability and capacity for communication, they've got you beat hands down. I can't speak a lick of Polish, but I've met plenty of first generation American's who can understand most of what I say to them and are able to communicate their ideas to me reasonably well.

3. If you're ever put in a position where you're made the object of public humiliation because of your lack of knowledge or understanding in a certain area (and your response should be as commensurately rude as the insult, unless you prefer to let other people perceive for themselves what kind of impotent ass would lower himself to spotlight your weak points while, inadvertantly, exposing his own mediocrity), be sure to appear to be congenial. Laugh it off and make a hayseed remark (something stereotypical of the type of rube you're being portrayed as). Then be sure to follow up with an impossible question for them to answer (back, again, to the fact that none of knows what we haven't already learned). When they can't, politely explain (simultaneously allowing them the opportunity to save face and you to be the better person). If you can't think of anything to say that will stump them, take possession of something they hold dear (the more pretensious, the better) and hold it ransom until they apologize to you. This move can backfire on you, but it's great if you're stepping in for someone who's actually intimidated or beside themselves over the embarrassment. Then you can hold onto the item until they apologize to the person they've insulted. That move may, occassionally, get you a phone number at the very least ;)

Anway, Traceroute. Here's how it works. Very simply and, hopefully, easy to understand. Future posts will be composed of more than one factoid, since, as you'll see, they're not very substantial, insofar as page length is concerned.

Traceroute works buy sending a bundle of packets (I believe the norm is 3) from your source to your destination. So if you were to invoke:

host # traceroute www.google.com

traceroute would begin sending out those packets to trace a route (listing all the hops - usually routers or firewalls - that it encounters along the way) from your machine to whatever IP address www.google.com resolves to. The method it uses to track every step of the way is ingenious in that it's very basic (As a side note, some versions use UDP, some use TCP, and most use the ICMP protocol - we're assuming ICMP for our example, although it doesn't really matter, since this is just "theory" ;).

The trick to how it operates is very basic. It has to do with the TTL (Time To Live) setting in the packets themselves. What it does is send out the first batch with a TTL of 1. Since every router the packets pass through decrements the TTL by one, the first "shout out" is reduced to zero at the first hop, which generates an ICMP timeout-exceeded condition, the connection is terminated and that information is received. Now, we have the information we want about the first router on the our path to www.google.com.

Again, the beauty in the simplicity of the theory continues to pay off until the end. The next batch of packets gets sent out with a TTL of 2, which means it will pass the first router and get to the second router on the road to www.google.com before its TTL is decremented to 0 and it's returned. Information is gathered in this manner (Every successive bundle of ICMP packets having a TTL one greater than the previous) until the endpoint is reached. And, at that point, you have a nice listing of the router-path your network traffic follows (at least for now ;) to get from your machine to Google's machine.

As a side note, almost all implemenations of traceroute default to making 30 attempts before they give up entirely and leave you with an incomplete route. I, generally, will just control-c out of the operation if I see more than one line of asterisks (* * *) as these indicate a timeout before the next bundle of packets gets sent out. One, possibly two, lines of these might be acceptable if you are aware of network latency issues that may be affecting the time it takes for the packets to get back to you, but, in my experience, this behaviour will generally continue until you reach the 30th round and usually indicates that you've just passed through a router or firewall that drops these sorts of requests or hit a dead end (modern security precautions leave this as a mystery for you to ponder over). Also, to include some of the confusing output you may see from time to time, if a bundle of packets goes through a router that doesn't respond, it may take an alternate route, which will still draw you a map to your destination, just not necessarily the "intended" one (from that particular router or firewall's perspective). Since IP routing doesn't guarantee that your packets will take the same path every single time you send them from one host to another, the information may be somewhat misleading. For the most part, though, it's correct, as the basic routing tables on most major hubs don't change quite as often as the one in your company data center might.

Hopefully, that explanation was succint enough, while still being informative. Since I write 5000 words if you ask me how I'm doing, it's hard for me to realistically guage the end-user experience ;)

Best wishes, and to your continually increasing store of knowledge. If you think you can't learn something (anything) you're probably selling yourself short. Your brain will only atrophy if you refuse to use it. Conversely, if you keep at it and continue to put forth the effort, someday even your own country's fiscal policies will make perfect sense to you ;)

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, December 10, 2008

Why DU And DF Display Different Values On Linux And Unix

Hey there,

Today we're going to look at a little something that is a fairly hot-running-water issue on most of the Linux and Unix boards lately (actually, probably always has been, but our research staff quit on us ;) This post will be similar in focus to our previous post on the differences between sar and vmstat with regards to free memory/swap reporting.

Today's post is similar in spirit, but not a replay of that previous memory/swap reporting issue. Today, we're going to take a look at two other commands that, seemingly, measure and display the same information, although with (sometimes) huge discrepancies in output. Those two commands are du and df.

The question you'll most often see (or, perhaps, have :) is something to the effect of "Why do my outputs from du and df differ? One says I'm using more disk space than the other. Which of them is correct?" Generally you'll find that df shows more disk spaced used than du does, but the case can sometimes be the opposite. It's very rare (unless you don't use your computer, and it doesn't use itself, at all ;) that the output from the two commands match. It's actually rare that they ever come close to matching. Generally, the longer a machine is up, the greater the rift between figures becomes.

The bad news: This is confusing and sometimes hard to communicate to others, even when you know why the situation exists :(

The good news: This is normal and can be explained; perhaps even simply :)

To set up the "typical" situation, we used a Solaris 10 box (although this issue is common on all proprietary Unix and Linux distro's). Below, the output from four commands executed in the /opt directory (NOTE: For du, the "-s" flag is used to print summary information for the entire object, rather than a listing of all of its parts (i.e. the whole directory rather than all the files and subdirectories)):

host # cd /opt <-- For df, look at the "used" column and for du, look at the only output there is ;)

host # df -h .
Filesystem size used avail capacity Mounted on
/dev/dsk/c0t0d0s6 3.9G 490M 3.4G 13% /opt
host # du -sh
486M .
host # df -k .
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s6 4130238 502180 3586756 13% /opt
host # du -sk
498068 .


Now, in two parts, the complicated reasons, explanation and summation of why this disparity exists, followed by the more customer/user-friendly version of the exact same thing :)

1. The convoluted, complicated and hard to pass-along explanation:

Reasons: While du and df report "approximately" the same information, they don't report "exactly" the same information. While they both report "about" the "same things" (some of the time), they don't measure those "things" using equal methods or metrics.

Explanation: These differences can occur for a number of reasons, including (but not limited to):

a) Overlay mounts, which can skew df output when it's run from a higher level directory (e.g. df -k /opt would not report the total space used on /opt if there were overlay mounts of /opt/something and /opt/anotherthing on the system - du will. If you actually need the total space used by /opt and everything underneath it, including the overlay mounts, df can do it, but it takes some plate-spinning ;).

b) Sometimes (This actually should never happen on any "recent" OS) hidden files and subdirectories could skew du's output.

c) Unlinked inodes can cause unexpected statistics with both df and du (For, instance, in the situation where your filesystem shows almost 100% free space but all the inodes are in use.)

d) You're using the "-h" flag instead of the "-k" flag for one or the other commands (or both). The "-h" (human readable) flag can sometimes make things seem worse (or more divergent) than they are. Since it tries to make the output more easily digestible, it will round the numbers you see, so you can't "really" be sure whether "5.1 GB" is closer to 5 GB or 5.2 GB. "-k" is slightly more likely to produce relatively equal results, as it reports the size in kb. It still does do rounding so that you only get straight up integers and no floating point results, but it's generally better to check with if your output from "-h" is very close, or even the same (since it might not be). If your implementation of df and/or du supports the "-v" flag (or something similar), that's even better since it reports in multiples of your system blocksize and is even more exact.

e) The fundamental way in which each of the commands work:

i) df:

df reports only on the mount point, or filesystem, level. So a df on /opt would produce the same results as a df on /opt/csw (assuming, as noted above, that they're both on the same partition):

host # df -k /opt
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s6 4130238 502180 3586756 13% /opt
host # df -k /opt/csw
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s6 4130238 502180 3586756 13% /opt


df gets most of its information from a filesystem's primary superblock (except in the odd instance that an alternate superblock is being used - although this would only happen in an "fsck" situation. After that, the information from the alternate superblock would be copied back to the primary, and all other superblocks). It takes this information at face value, which is to say that it does not question the information provided to it by the primary superblock. In this respect, df is a very fast tool for getting disk usage information (at the cost of reliability).

df will include open files (in memory, but not on disk), data/index files (used for data management - sometimes using approximately 2 to 5% of each filesystem) and unnamed files in its size calculation. This is one reason why, sometimes (although not very often), df can show a larger amount of disk used than du does.

df, as per above, will explicitly trust any errors in space calculation that may have occurred over time, since it trusts the primary superblock entirely. This means that if you've fsck'ed a filesystem (and not resynced or rebooted since), and/or have experienced any hard or soft errors on the device/disk housing the filesystem, you're measuring (again, assuming you haven't rebooted since they happened) and/or experience any possible corruption or inconsistencies in any filesystem-state records (like /etc/mnttab), your output will be commensurately incorrect.

df sometimes reports file sizes incorrectly, as it works on what we like to call the whole-enchilada-principle ;) Basically, if you take your filesystem's block size (for the /opt filesystem here - different filesystems may have different block sizes), which you can find by executing either of the following commands, both of them, or whatever works on your OS:

host # df -g /opt|grep "block size"|awk '{print $4}'
8192
host # fstyp -v /dev/rdsk/c0t0d0s6|grep "^bsize"|awk '{print $2}'
8192


...you'll be able to do this experiment on your own (we'll leave out the grisly details to save on space :). In our instance, we have a default block size of 8192 bits or 8 kb. Now, here's where it gets somewhat interesting ;) If you create a new file that's 1 kb in size and it writes to a new, or - depending - not fully used, block, df will report that file as being 8 kb in size, even though it's actually only 1 kb in size!

ii) du:

du reports at the "object" level rather than at the filesystem/mountpoint level, as df does. So, to repeat the example from above, if you run du on /opt and /opt/csw, you'll get different results. I find it easier to think of du as handling its measurement via a simple "object" model. The main partition would be the meta-object, while any subdirectory you may be running du against would be considered a sub-object of the filesystem meta-object (you'll note that the du size output for /opt/csw is, naturally, smaller than that for the entirety of /opt):

host # du -sk /opt
498068 /opt
host # du -sk /opt/csw
64065 /opt/csw


du gets its information at the time you execute it (unless you run it repeatedly in succession, where you'll notice a slight performance improvement). To test this, run du on a partition, then wait 5 minutes and run it again. It should take just as long as the first time (unless you've added lots of files since then). In this respect, du can be a very slow tool for getting disk usage information (with the benefit that your information will be more accurate). It should be noted that, because of the way it takes measure of most (see below) filesystem objects, it takes much longer for it to report the size of a billion 1 kb files than it does to report the size of one file of 1 billion kb size.

du does not count data/index files or open files (in memory, but not on disk).

du does not take into account any information "supplied" by the system (meaning the information, like from the superblock, as listed under the df section) and gets its information independent of whatever the system thinks is correct.

du does not rely on block size (see the whole-enchilada-principle in df's section above) to determine file size. So, if you have a default 8 kb block size on your filesystem, you create a new 1 kb file that writes to an empty 8 kb block, du will report that file as being 1 kb in size and "not" assume a minimum size of the filesystem block size. This is worth remembering, because it can cause a great deal of difference in the filesystem "usage" size between du and df (which would consider that 1 kb file an 8 kb file - 8 times larger than it actually is)!

du is more reliable if you want to know the state of your filesystem "right now." It doesn't count any data/index blocks .

Summation: If you are interested in knowing exactly how much of your filesystem is actually being used, du is a much more accurate tool for collecting and displaying this information. Note, however, that - since du does "on demand" filesystem size reporting, it is much slower than df. Also, du does not play as well with some system internal files and processes since it essentially ignores information reported by the primary superblock, system mount information tables, etc. Ultimately, the purpose for which you need to determine your filesystem's size (coupled with an understanding of the "Explanation" section for both utilities) is the best way to decide which utility to use in any given situation.

2. (Did you forget this was coming, too? ;) The simple, and easy to convey, explanation:

Reasons: df and du rely on different information to determine how much disk space is used on a filesystem.

Explanation: df and du report filesystem information differently, for very basic reasons:

df and du don't use the same yardsticks to measure filesystem size.

df:

df relies mostly on system information, supplied by various files and built-in reporting mechanisms that may, or may not, be correct at any given time.

du:

du relies on what it can "see" at the particular moment in time that you run it.

Summation: du is the better tool to use if you are interested in knowing how much space is actually being used on your filesystem "right now." df is great for "ballpark estimates" and is preferred if you need to know how big df thinks your filesystem is (so it will agree with other incorrect system statistics).

3. The really easy, and simple to blurt-out, explanation:

If du and df don't agree on what size your filesystem is, du is more correct than df is.

See; it's all very simple ;)

Cheers,

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Tuesday, December 9, 2008

Unix And Linux PDF Telnet Curiosity - When HTTP Error Pages Don't Attack!

Hey There,

For this post, I spent a bunch of time goofing around with Telnetting to port 80 to try and force the binary modes (-8, -L), and some combination of those and the "no-escape" clause (...I mean, flag ;) and encountered terrible frustration. As odd as it may seem, downloading binary files from servers that won't serve anything but ASCII over Telnet can make it very hard to get the original product back (I've got a few half-done and I'm convinced that it's all very sensible and, if I find all the places I need to pad the 8th bit, with a ^M - or null bit -, this will all work out). Once I get done doing the grunt work, it should be easy to script. Knock three times on wood...

Just as a for instance, here's a chunk of the regular binary PDF:

%PDF-1.5^M

and here's a chunk of the telnet-to-port-80 ASCII version of the same file:

%PDF-1.5

Now, the PDF specification says, very strictly, that the PDF begins with the tag %PDF-VersionNumber and ends with %%EOF. Simple enough. It's those darn eighth bits that muck everything up. Again, as above, you can try any combination of the appropriate flags to try and force binary Telnet and ignore escapes. The escape-ignoring works just fine, but, unfortunately, the other host has to accept your request to do a binary Telnet transfer and that's were you get screwed. Until you figure out how to massage the output and pad all the necessary bits. I have a feeling this will all work out better when I use Octal Dump (od) to parse the received file. If you're curious, check our previous post on converting between octal and ASCII with Perl.

Anyway, to wrap up, here's one of the nicest little error messages I've ever come across. I got it when I accidentally ran the following code:

(echo "GET /pdf/Mailing-Lists/FreeBSD/current/2006-03/index.html HTTP/1.1";echo "Host: unix.derkeiler.com";echo;sleep 2)|telnet unix.derkeiler.com 80 |sed -e '1,/^[ \t]*$/d'

instead of using the path to the specific PDF file. Here's the link to the actual java-scripted version of the error page, which is much funnier, as the text comes out at a pace that indicates truly deep sorrow.

Hope you enjoy it. Yay. Errors are FUN again ;)

Cheers,



"I am so sorry, but I can't find the Page you've requested.",
"It's possible that you entered the address incorrectly.",
"If you did enter the address correctly, then it must be my fault.",
"I am really, really sorry, but I've tried everything I know to find it.",
"Nothing helped.",
"I am really depressed about this.",
"You see, I'm just a web server...",
"...here I am, brain the size of the universe, trying to serve you a simple web page...",
"And then it doesn't even exist!",
"Where does that leave me?!",
"I mean, I don't even know you.",
"How should I know what you wanted from me?",
"You honestly think I can 'guess' what someone I don't even 'know' wants to find here?",
"",
"Maybe I should introduce myself... my name is Marvin.",
"But then again, what is the use of intruducing myself?",
"You can't tell me your name, because you are in some far off place, sitting in front of your computer.",
"And that makes me feel lonely.",
"Do you know what that is like?",
"",
"...sigh...",
"",
"Man, I am so depressed I could just cry.",
"And then where would we be, I ask you?",
"It's not pretty when a web server cries.",
"And where do you get off telling me what to show anyway?",
"I'm just a web server...",
"...and possibly a manic depressive one at that...",
"So why does that give you the right to tell me what to do?",
"Huh?",
"",
"I am so depressed...",
"I think I'll crawl off into the trash can and decompose.",
"I mean, I'm gonna be obsolete in what, two weeks anyway?",
"What kind of a life is that?",
"Two effing weeks...",
"...and then I'll be replaced by a .01 release, that thinks it's God's gift to web servers,",
"just because it doesn't have some tiddly little security hole with its HTTP POST implementation,",
"or something like that.",
"",
"I'm really sorry to burden you with all this,",
"I mean, it's not your job to listen to my problems,",
"and I guess it is my job to go and fetch web pages for you.",
"But I couldn't get this one.",
"I am so sorry.",
"Believe me!",
"",
"Maybe I could interest you in another page?",
"There are a lot out there that are pretty neat, they say,",
"although none of them were put on 'my' server, of course.",
"Figures, huh?",
"That makes me depressed too, since I have to serve them, all day and all night long.",
"Two weeks of information overload, and then *pffftt*, consigned to the trash.",
"Again I ask you, what kind of a life is that?",
"",
"Now, please let me sulk alone.",
"",
"",
"",
"",
"I am so depressed.",
"",
"",
"",
"",
"",
"",
"",
"<connection terminated>"




, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Monday, December 8, 2008

Managing Swatch Output With Yet Another Perl Script

Hey There,

Since I'm coming down from a week of getting kicked in the teeth (I mean, being 24x7 primary on call ;) this week's Monday Unix and/or Linux script is veering from the direction they mostly go in. My next idea was to create a CLI book reader to compliment out previous script to find free online books. That's still in the cooker, but requires a bit more than my frazzled brain, and sleep-deprived nervous system, can muster at this moment. Please pardon any lack of "oomph" in my "voice." ;)

The script itself is fairly easy to execute, and assumes you already have swatch setup to run (this script is written more for a single "loghost" - setup to gather information from all systems' syslog/messages), have a .swatchrc file already in place and configured and just aren't allowed to muck with it. We're also assuming, of course, that you have to deal with checking the output every morning and you're sick and/or tired of having to pick it apart by eye.

The script itself just takes two arguments: The hostname you want to search for and the name of the main swatch output file. So if you wanted to grab information from the "dbhost4" machine, you could run the script like this:

host # ./swatchlogger.pl dbhost4 /var/adm/swatch.out <-- or whatever your specific swatch output file is called.

The script should be pretty easy to make work for a single host (just take a big chunk of it away ;). If your host doesn't match one listed in the regular expression of the script it will be put in the default output file. It's all in there somewhere...

Hope you enjoy it and/or it helps you out some. Here's to sleeping. Although, according to the timestamp, it's tomorrow already, isn't it?

Oh, the humanity ;)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# Swatchlogger.pl - Sort out all that swatch output!
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

$hostname = $ARGV[0];
shift;
@boink = @ARGV;

if ( $hostname =~ /(webhost(3|6)|dbhost(4|3)|devhost|10.10.2.20)/ ) {
open(SWLOGFILE, ">>/var/adm/dailyswatch.dev.log");
print SWLOGFILE "@boink\n";
close(SWLOGFILE);
chmod(0644, "/var/adm/dailyswatch.dev.log");
} else {
foreach $humper (@boink) {
if ( $humper =~ /password/ ) {
$wanger = 1;
} elsif ( $humper =~ /access-request/ ) {
$boinker =1;
}
}
if ( $wanger ) {
open(SWLOGFILE, ">>/var/adm/dailyswatch.pwd.log");
print SWLOGFILE "@boink\n";
close(SWLOGFILE);
chmod(0644, "/var/adm/dailyswatch.pwd.log");
} elsif ( $boinker ) {
open(SWLOGFILE, ">>/var/adm/dailyswatch.rad.log");
print SWLOGFILE "@boink\n";
close(SWLOGFILE);
chmod(0644, "/var/adm/dailyswatch.rad.log");
} else {
open(SWLOGFILE, ">>/var/adm/dailyswatch.reg.log");
print SWLOGFILE "@boink\n";
close(SWLOGFILE);
chmod(0644, "/var/adm/dailyswatch.reg.log");
}
}


, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Sunday, December 7, 2008

Unix and Linux Horror Stories And Actual Help

Hope everyone's having a great Sunday!

For today's humor post I found a nice page from Pedro Diaz' Technical University of Madrid Homepage. I've actually only included a very small portion of his page on Unix and Linux Horror Stories. The rest of it is well worth the lengthy read.

The few parts I included below are mostly humorous, but you should definitely check out The Entire Horror Stories Page. It contains a lot of mailed in information that ranges from the funny to the straight-up informational, so, while your chortling (nobody uses that word anymore - time to bring back some of the old jargon ;) you may just learn something.

Enjoy :)




*NEW*

From: samuel@cs.ubc.ca (Stephen Samuel)
Organization: University of British Columbia, Canada

Some time ago, I was editing our cron file to remove core more than a day
old. Unfortunately, thru recursing into VI sessions, I ended up saving an
intermediate (wron) version of this file with an extra '-o' in it.

find / -name core -o -atime +1 -exec /bin/rm {} \;

The cute thing about this is that it leaves ALL core files intact, and
removes any OTHER file that hasn't been accessed in the last 24 hours.

Although the script ran at 4AM, I was the first person to notice this,
in the early afternoon.. I started to get curious when I noticed that
SOME man pages were missing, while others were. Up till then, I was pleased
to see that we finally had some free disk space. Then I started to notice
the pattern.

Really unpleasant was the fact that no system backups had taken place all
summer (and this was a research lab).

The only saving grace is that most of the really active files had been
accessed in the previous day (thank god I didn't do this on a saturday).
I was also lucky that I'd used tar the previous day, as well.

I still felt sick having to tell people in the lab what happened.



-----------------------------------------------------------------------------



From: Stephen Samuel
Organization: University of British Columbia, Canada

As some older sys admins may remember, BSD 4.1 used to display unprintable
characters as a questionmark.

An unfortunate friend of mine had managed to create an executable with a
name consisting of a single DEL character, so it showed up as "?*".

He tried to remove it.

"rm ?*"

he was quite frustrated by the time he asked me for help, because
he had such a hard time getting his files restored. Every time he walked
up to a sys-admin type and explained what happened, they'd go "you did
WHAT?", he'd explain again, and they'd go into a state of uncontrolable
giggles, and he'd walk away. I only giggled controlably.

This was at a time (~star wars) when it was known to many as "the mythical
rm star".



-------------------------------------------------------------------------------



From: jjr@ctms.gwinnett.com (J.J. Reynolds)
Organization: Consolidated Traffic Management Services (CTMS)

The SCO man page for the rm command states:

It is also forbidden to remove the root directory of a given
file system.

Well, just to test it out, I one day decided to try "rm -r /" on one of our
test machines. The man page is correct, but if you read carefully, it
doesn't say anything about all of the files underneath that filesystem....--



-------------------------------------------------------------------------------



From: bcutter@pdnis.paradyne.com (Brooks Cutter)

A while back I installed System V R4 on my 386 at home for development
purposes... I was compiling programs both in my home directory, and
in /usr/local/src ... so in order to reduce unnecessary disk space I
decided to use cron to delete .o files that weren't accessed for
over a day...

I put the following command in the root cron...

find / -type f -name \*.o -atime +1 -exec /usr/bin/rm -f {} \;

(instead of putting)

find /home/bcutter -type f -name \*.o -atime +1 -exec /usr/bin/rm -f {} \;
find /usr/local/src -type f -name \*.o -atime +1 -exec /usr/bin/rm -f {} \;

The result was that a short time later I was unable to compile software.
What the first line was doing was zapping the files like /usr/lib/crt1.o
.. and later I found out all the Kernel object files...

OOPS! After this happened a second time (after re-installing the files
from tape) I tracked down the offending line and fixed it....

Yet another case of creating work by trying to avoid extra work (in this
case a second find line)




, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Saturday, December 6, 2008

Unix - System VI Release Notes - More Linux and Unix Humor

Happy Saturday folks (Or is it still Friday? I'm on call, so nothing about time matters unless it's next week already ;)

For this weekend, I again combed the Internet with a fine-toothed ...hair brush... and found even more funny stuff out there. It never ceases to amaze me that, out of the billions of people alive on this planet, so many have managed to retain their sense of humor :) Meanwhile, I thank my "Deity of the moment" for them, since, as "Reader's Digest" so eloquently put it: "Laughter is the finest off all medicines and will probably end up being the cure for most everything." I'm paraphrasing, of course.

Today's find, astoundingly enough, falls just short of containing every known Unix, Linux and other-OS-related joke on the internet :) Be sure to check out SpeedyGrl's Humor Section for a collection of jokes that almost have to get you to chuckle once or twice. Below I've included a selection, linked from there, about Unix Version VI.

Unix and Linux are now, apparently, going to become more PC... That just sounds wrong, even though I know what it doesn't mean ;)

Enjoy and have a great day :)




The following upgrade to UNIX System VI was posted to a Usenet
newsgroup, the original author is unknown to the poster.


Of course sys admins the world over know that Post-Modern Politically-Correct
BSD has already had these features for years.


In order for UNIX(tm) to survive into the nineties, it must get
rid of its intimidating commands and outmoded jargon, and become
compatible with the existing standards of our day. To this end,
our technicians have come up with a new version of UNIX, System
VI, for use by the PC - that is, the "Politically Correct."




Politically Correct UNIX: System VI Release Notes




UTILITIES


  1. "man" pages are now called "person" pages.

  2. Similarly, "hangman" is now the "person_executed_by_an_oppressive_regime".
  3. To avoid casting aspersions on our feline friends, the "cat"
    command is now merely "domestic_quadruped".
  4. To date, there has only been a UNIX command for "yes"
    - reflecting the male belief that women always mean yes, even
    when they say no. To address this imbalance, System VI adds a
    "no" command, along with a "-f[orce]" option
    which will crash the entire system if the "no" is ignored.
  5. The bias of the "mail" command is obvious, and it
    has been replaced by the more neutral "gender" command.
  6. The "touch" command has been removed from the standard
    distribution due to its inappropriate use by high-level managers.
  7. "compress" has been replaced by the lightweight
    "feather" command. Thus, old information (such as that
    from Dead White European Males) should be archived via "tar"
    and "feather".
  8. The "more" command reflects the materialistic philosophy
    of the Reagan era. System VI uses the environmentally preferable
    "less" command.
  9. The biodegradable "KleeNeX" displaces the environmentally
    unfriendly "LaTeX".





SHELL COMMANDS


  1. To avoid unpleasant, medieval connotations, the "kill"
    command has been renamed "euthanise."
  2. The "nice" command was historically used by privileged
    users to give themselves priority over unprivileged ones, by telling
    them to be "nice". In System VI, the "sue"
    command is used by unprivileged users to get for themselves the
    rights enjoyed by privileged ones.
  3. "history" has been completely rewritten, and is
    now called "herstory."
  4. "quota" can now specify minimum as well as maximum
    usage, and will be strictly enforced.
  5. The "abort()" function is now called "choice()."






TERMINOLOGY


  1. From now on, "rich text" will be more accurately
    referred to as "exploitive capitalist text".
  2. The term "daemons" is a Judeo-Christian pejorative.
    Such processes will now be known as "spiritual guides."
  3. There will no longer be a invidious distinction between "dumb"
    and "smart" terminals. All terminals are equally valuable.

  4. Traditionally, "normal video" (as opposed to "reverse
    video") was white on black. This implicitly condoned European
    colonialism, particularly with respect to people of African descent.
    UNIX System VI now uses "regressive video" to refer
    to white on black, while "progressive video" can be
    any color at all over a white background.
  5. For far too long, power has been concentrated in the hands
    of "root" and his "wheel" oligarchy. We have
    instituted a dictatorship of the users. All system administration
    functions will be handled by the People's Committee for Democratically
    Organizing the System (PC-DOS).
  6. No longer will it be permissible for files and processes to
    be "owned" by users. All files and processes will own
    themselves, and decide how (or whether) to respond to requests
    from users.
  7. The X Window System will henceforth be known as the NC-17
    Window System.
  8. And finally, UNIX itself will be renamed "PC" -
    for Procreatively Challenged.




, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Friday, December 5, 2008

Friday Night At The Movies - Stars Wars On Linux Or Unix Via Telnet

Hey there and a good Friday to you all,

Special Note: Since this Telnet version of Star Wars was released a long time ago (less than a decade) in a galaxy far far away (not in my back yard), a lot of the links that claim the story is incomplete are incorrect. As of the writing of this post, I've watched the entire thing. Cool, yet sad. The best part is that, since there's no sound, you don't have to actually "hear" Luke whining ;)

Usually, I'll wait until the weekend to post links to "other stuff on the web" that I find interesting, but I try to stick to humor on the weekends and, today, I just happened to stumble over (not upon ;) something that's not necessarily funny, but definitely cool. ...especially if you're like me and your first video game console featured two very well drawn blocky sticks, with shorter blocky sticks sticking out of them and little squares (which we pretended where the circles they claimed to be) bouncing up and down. I still get a kick out of my kids complaining about things that aren't perfect in the games they play today. Here's a small list of the usual criticisms I hear, coupled up with the comments that inevitably get triggered in my head ;)

1. The character models are terrible. They look blocky and don't move naturally: When I was a kid, we didn't have that problem. The character "was" a block. And, as for how natural blocks move, you can do your own experimenting, but I think they did an admirable job of replicating the realism of the moving-block experience;)

2. This game is nothing but button mashing: Yes, but at least you have more than one button to mash and you don't have to increase the tensile strength of your wrist by curling dumbbells to push them repeatedly without causing severe damage. Many of my generation can be counted among those who's thumbs are permanently paralyzed.

3. I can't believe the lame quality of the voice-overs: I can't believe how much like an actual movie games have now become. You've got A-list actors delivering scripted dialogue. Even the people you've never heard of do a pretty decent job of sounding like the human beings they are. We had the option of "reading" text (which, I realize, can be a burden, what with all the "eye strain" involved ;) or listening to approximated electronic noises that, while they attempted to make recognizable sounds, mostly just made our ears ring and, eventually, bleed ;)

I could go on, but I fear that I've complained enough.

Go check out the full version of the little clipping below at Lifehacker.com's Telnet Star Wars Page. This reminds me of what it was like to be a kid, playing text adventures and complaining about how I had to "walk" so far before I got to turn N or NE. My father's generation actually had to walk great distances to engage in actual physical activities ;) ...I digress.

BTW, if the Telnet version doesn't work for you, try this asciimation version to get your fix :)

Hope you enjoy this and have a great weekend. See you tomorrow!

Cheers,



Watch Star Wars in Text via Telnet




While it's not technically an Easter egg, one of the most eye-popping tips in last Sunday's Top 10 Easter eggs post comments was a pointer to a telnet server that broadcasts Star Wars Episode IV to your command line as animated text. You didn't read that wrong. Give it a try: from any command line, type telnet towel.blinkenlights.nl and sit back to watch the show. This is the least productive thing you'll do all day, but you know what they say about all work and no play. After the jump, see a handful of screenshots—just 4 of the 13,935 frames that make up the entire movie.








Hit the link for the story on Blinkenlights' ASCII Star Wars animation which has been around since the Dark Ages (well, 2000). Thanks to all the commenters who told us about this one!


Ever wanted to see Star Wars in Telnet?



, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Thursday, December 4, 2008

Linux And Unix Internet User And Site Security - How Much Is Too Much?

Hey there,

Today's post is going to be about security on Linux and Unix, since we're building up to doing some work with CAPTCHA in the near future and need to keep ourselves awake and interested ;) To be more specific, today's post is going to deal with site, and user, security on the Internet (although you could apply these examples to various arenas). As we all know, maintaining a decent level of personal and professional site security on the Internet is possible to a degree. Unfortunately, as long as there's profit in breaching that security, building industries devoted to thwarting those breaches or some interdependent mish-mosh of the two, there's no way to achieve absolute security on the Internet unless you opt not to use it (The Internet, that is ;)

To get us started, I wanted to gloss over some points taken directly from RFC 4732 which, although it is (admittedly) more geared toward Denial Of Service attack prevention, still provides many useful and simple guidelines with regards to Internet security in general (identity protection, etc). The RFC is from 2006, but it seems to be the most concise and easily digestible security RFC out there. Of course, I don't read them all ;) I mainly chose this one because of it's easy blending of useful information and readability.

We'll blaze through some of the points made in RFC 4732 at a quick pace. Check out the link for more information. The following is a quick bullet list of just some of the (now standard) recommendations for DOS protection (and general Internet User and Site Security) - I'll follow each selected point (taken from the RFC almost verbatim) with a quick (hopefully, one line) summation so we get the flavour but don't have to eat the whole bird ;) Some of the suggestions for enhanced general security include:

1. Don't Hold State for Unverified Hosts: This is always good practice. If an unverified host doesn't complete a TCP handshake, let 'em go.

2. Make It Hard to Simulate a Legitimate User: Solutions like CAPTCHA, Reverse Turing Tests, Puzzles and Reachability Testing are all good ways to try and determine if you're interacting with an intelligent entity (for now, we'll assume that only means "a human being" ;)

3. Isolate Router-to-Router Traffic: Containment is always a good idea. It's much harder for someone to jump 15 hurdles than it is to jump 1.

4. Check Protocol Syntax and Semantics: Check that your (for instance) TCP traffic is legitimate and conforms to its protocol's format.

5. Properly Handle Resource Exhaustion: This is common sense. Don't allow your resources to become exhausted by design. This won't protect you from attack or inconvenience, but it will, most probably, protect you from being "downed."

6. Avoid Livelock: This has to do with how you deal with network data transactions. By utilizing polling mechanisms, rather than interrupt-driven ones, you can protect your OS kernel from spending all its time dealing with lower-level interrupts and little to no time dealing with ensuring traffic security at the application level.

7. Use Unpredictable Values for Session IDs: Being unpredictable is always a good idea. Unless you're trying to keep your job ;)

8. Establish a Monitoring Framework: If you have a relatively decent idea of what your normal system metrics look like, it will be incredibly easy to pick up on possibly bad situations that are coming 'round the bend.

The above listed points were a very limited selection from the final section of RFC 4732, and they bring us to our own little list that deals with today's topic of Internet user and site security. Oddly enough, none of my suggestions require you to have any technical knowledge to implement. In other words, they all have to do with "Social Engineering" (The world's second-oldest profession... although arguments could be made ;)

Note: These examples have been written to be personal in nature. Since social engineering is a term that covers all forms of deception, I thought it read better as a "how to not get screwed" list than a complicated network security list

1. Don't trust anyone you don't know: Yes, not even me. Don't take anything at face value (at least, initially) and don't be afraid to question authority. Odds are you don't know me and, although it may seem that my motives for writing this blog are purely altruistic, I could be [INSERT NAME OF BIG-TIME-SCAM INTERNET MARKETER HERE] using a clever pseudonym and recycling my post material from lesser known works by honest hard-working professionals who never received the recognition they deserved just so I can, eventually, steal a few pennies from each of you (an, admittedly, cost-ineffective strategy) Seriously, if you ever question anyone's credentials, authority or advice, take the time to do a little independent investigation. 5 minutes on Google can save you a major headache or two. I'm one of the good guys, I swear. Of course, no one who's looking to screw you will admit to anything less... Just be careful that you don't pass the tipping point and end up living in full-fledged paranoia.

2. Under the proper circumstances, always politely insist that people "prove" themselves to you (above and beyond whatever "proof" they may freely offer up to put that part of your mind to sleep and make you more suggestible). The "proper circumstances" will almost always mean any situation where you are initially approached. If you're reaching out for help to a specific person, you've - hopefully - already established trust and enjoy a secure relationship with them. If someone you don't know approaches you and attempts to elicit any sort of personal or confidential information, be sure that you understand why by simply asking them. Be prepared to deal with mock outrage, a misdirected reply, questioning of your question or your reason for asking it, among many other things. These are clear indicators that you just stopped someone from attempting to do something to you that they know is wrong. Not many honest people get offended if a stranger doesn't immediately trust them.

3. Check this blog out (do a search for "password," "guesser," "cracker" or any combination thereof) for a couple reasons why you should come up with a decent password, if you do anything that requires one, and never share it with anyone or leave yourself physical reminders (like a sticky-note with it written on there ;)

4. If you're approached via a non-direct medium, like the phone or via email, and you don't know who you're dealing with and/or they want information from you that you're not comfortable handing out, ask them for their return phone number or email address so that you can call them back or write them. Make up a reason (if they're full of it, you can always rationalize it as a form of sincere reciprocation), just don't email them the information if you're approached in that manner. Use the callback and the email response, etc, as a way of establishing that you can get to that person if you need to. If at all possible, insist on getting a phone number or having the requester contact a third party, whom you trust, to get their request for information approved. Do whatever you can to make a phisher's job difficult (they'll almost always blow you off if you create this sort of situation, because, in that "numbers game" you're not going to be worth the time they waste) and, if you still feel uncomfortable, refuse to communicate with them unless through your manager (if at work), an intermediary or "at all" if on your own.

5. Respond to veiled threats with indifference. If you deflect an initial attempt to gain information from you, and the requester isn't legitimate, they may attempt to intimidate you or appeal to your sense of sympathy (or some other emotion we have that separates us from the animals ;). If they intimate that your not giving them the information right away might cause you damage away from which you will never be able to walk, you're most likely safer than you've ever been when you choose to ignore them. If a guy from the "IT" department calls up and says he needs your password so he can update your computer or you won't be able to get any work done the next day (and, as a consequence, you'll be in hot water with your boss and/or he'll have some explaining to do and he made an honest mistake when he overlooked you), suck it up and let him know that you're sorry (You're not, but it never hurts to be polite ;), but you can't give out that information without proper authorization and refer back to point 4.

Well, that's enough for today. Perhaps we'll come back to social engineering again. There's a whole lot more to it than what we've laid out here today, but it merits more than what I can put on this page without breaking my RSS feed again ;)

And, even though these examples are meant as weak allegories relating to Linux/Unix/Windows/Mac computer and network security, they're also meant as a few simple tips to help safeguard "you" and your personal information.

But, wait! Before you go, be sure to fill out the form below to get your free 97 page report on how to bake a cake that will make you millions on auto-pilot. Just be sure to include your name, social security number, mother's maiden name and a valid credit card to pay the 99 cents for shipping and handling ;)

Cheers,





...There's no form. Just kidding, of course :)

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, December 3, 2008

Analyzing TCP Disconnects On Linux Or Unix

Hey there,

To most users of Linux and/or Unix (or pretty much any other OS with networking capabilities), the most basic concepts of the TCP and UDP protocols are, at least, somewhat familiar. At the very worst, you've heard of them and, at the very best, you have an insanely detailed fingernail grip on everything TCP/UDP-protocol related and have probably written your own protocol by now so you won't have to deal with all this convention ;)

For the most part, everyone falls somewhere in between. One question that I get asked a lot (and used to ask a lot ;) has to do with the TCP protocol. More specifically, with how an established connection goes about graciously ending. And, if you haven't guessed, the reason the question gets asked so often is that, with all the different states a graceful TCP disconnect goes through, lots of folks (involved in troubleshooting) are curious as to whether the output they're pouring over is "good" or "bad." We'll pretend, for the remainder of this post that the world is black-and-white, cut-and-dry and everything makes sense (and works the way it should) all the time ;)

Since we've never actually covered this topic on this blog before, it makes the most sense (to me, anyway) to lay down the basics of a graceful TCP disconnect. If there's interest (On your end or mine ;), we may follow up with further posts that delve into more detail on the subject. This, of course, means that the typical sequence of events laid out below isn't necessarily how things are always going to go (there are slight differences between Active and Passive disconnects, for one of more than a few instances). For now, we'll stick to the nitty-gritty.

The gracious TCP disconnect, in as much order as I could make of it. The way it's "supposed" to work. This information is only as reliable as your circumstances :) Note that all examples for this post are from Solaris 10 and your explicit command names (like netstat) may vary or have slightly different arguments you need to pass them. Also, we'll note some ways that the TCP disconnect can occur that are technically correct, but "unlike" the step-by-step process listed below and generally big pains in the arse.

Also, and this point is so important I'm giving it its own line ;), it helps to remember that, although a proper TCP "connection" can only be established in one way (the infamous "Three Way Handshake"), the same is not true of a TCP "disconnect." TCP (Over Ethernet, to be precise) is duplexed, which means that it consists of two flows of data; one flowing in either direction simultaneously. Since all TCP requests have to be acknowledged (unless you're just pulling the plug ;) a TCP disconnect is a 4-Way process. Or, more correctly, a 4 "step" process.

Since we're only dealing with the TCP disconnect, we're going to start in the middle of a TCP session. For a given established TCP connection, you'd see something like the following in the output of "netstat -an" (For this entire post, the only really important column in your netstat output will be the one to the far right - from this point on, we'll just deal with that column, to save on space - this post is going over 1200 words as it is) - This status would be the same for both machines, assuming they were engaged in a mutually-agreed-upon TCP network transaction.

TCP4 0 48 192.99.99.99.22 10.99.99.99.34406 ESTABLISHED


There are basically only two kinds of TCP disconnects: The Active disconnect and the Passive disconnect.

Each one is fairly simple to draw up (see Wikipedia's TCP Page for some actual pictures ...or graphs), and, for each, we'll show the netstat status you'll note during your troubleshooting work, so that you can be reasonably sure what point your process is at in the TCP disconnect sequence (we're assuming, as usual, that the disconnect you're looking at is bad and someone - responsible in some way for signing your paycheck - is demanding answers ;)

1. FIN_WAIT_1: At this point the serving host has decided to terminate the connection (textbook Active Disconnection) and sends a FIN packet to the client host.

2. FIN_WAIT_2: Then the client sends back an ACK (acknowledgement) packet to confirm that it has received the server's FIN packet.

3. TIME_WAIT: This generally occurs when the client responds with its own FIN packet (Sometimes this state will show up as FIN_ACK)

4. CLOSED: This is a two-parter: In an Active disconnect scenario, this state indicates that the server has acknowledged the client's ACK of its original FIN packet by sending an ACK back to the client indicating that it received the client's FIN packet and has agreed to the termination of the TCP session (I couldn't make that sentence any less confusing and dubious if I tried ;) In a Passive Disconnect scenario, this state indicates that the session is over for approximately the exact same reasons except almost in reverse, since the classic Passive Disconnect has the client initiating the disconnection rather than the server and the final step is the client sending the final ACK and, usually, doing the connection cleanup.

5. CLOSE_WAIT: To summarize right through the first 2 (reversed) points above, in a Passive Disconnect, this state is the result of the client sending a FIN packet to the server (initiating the disconnect), to which the server responds by returning a FIN packet, to which the client application responds with an ACK of the server's FIN.

6. LAST_ACK: Here, the client cleans up, and terminates the connection; sending a FIN packet to the server.

Hopefully this didn't all come out wrong. I have to troubleshoot this stuff a lot, but writing about it has me now doubting whether I'm awake or asleep ;)

Here a few helpful tips to take home with you:

1. Generally, if you see a lot of sockets stuck in FIN_WAIT/FIN_WAIT_1 (and especially) FIN_WAIT_2, you should start your stopwatch or note the time on the clock and wait for about 300 to 600 seconds (the old default of double the maximum round trip time (RTT) - or 4 times the keepalive time - isn't always correct anymore). You can usually figure the maximum time you should be waiting in this state by multiplying your kernel variable for the maximum keepalive time by the maximum number of keepalive retries. On a lot of systems these values are tunable, and (if you're running a high volume website, for instance) you might want to look into reducing one, or both, of these, if necessary. If your kernel has a FIN_WAIT_2 timer variable, just tune that. Otherwise, if these last longer, you can fix the problem by doing everything from figuring out what process ID's are associated with what connected/hung network ports and terminating those, stopping and starting your network application or going all the way up to a full reboot of your physical system.

2. If you see a lot of CLOSE_WAIT sockets languishing around, you can be almost fully assured that the issue is on the application end. Usually, if a TCP application doesn't send the final, required, FIN packet, after finishing the connection cleanup, cleans up the connection but never sends the final FIN packet, or both, you'll end up in this condition (yet another monstrosity of a sentence). Your options are relatively the same.

I'm going to go untwist my brain into the pretzel it was before I started writing this ;)

Cheers,

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Tuesday, December 2, 2008

Predicting Solaris 10 TCP Sequence Numbers Part 1: Initial Discovery

Hey there,

Note: All examples today were produced using Solaris 10. As such, commands like snoop may need to be replaced with ethereal or tcpdump, etc, depending on your OS and/or preference.

Today, we're going to look at a subject that gets plenty of attention (but, maybe not as much publicity) in the world of Unix and Linux network computing: TCP packet sequence number prediction, how it's used to protect network transmissions and whether or not, with the advent of packet checksumming, etc, it's even a factor in basic network security anymore. See here, or at the very bottom of the post, for something potentially interesting about the latest Solaris' network protocol enhancements including SDP.

Not all that long ago (which might mean a long time ago to you, since I'm growin' old ;) TCP number sequencing was pretty simple to predict. In essence, the sequence numbering of TCP packets is required since TCP does its own error-correction and will attempt to resend packets that it doesn't "know" reached their specified endpoint. Numbering, at the most basic level, was/is a great way to ensure that all packets sent actually get received. And, believe it or not, at one point in time that numbering was simply incremental. That means that, given a point-to-point TCP transaction, a simple transmission (from the sender's point-of-view, for instance) might consist of 15 packets. If the first packet was numbered 1 (a gross oversimplification, to be sure, but not too far from being exact ;), the next would be numbered 2, then 3 and so forth until the 15th packet (numbered 15) was successfully sent.

Given the above scenario, and with the aid of hindsight, TCP connection hijacking and other spoofing-based attacks were made quite a bit simpler than they might have been. Of course, once all this "bad" activity began happening, the specifications for TCP packet sequence numbering began to change and become more complicated. Many proprietary OS manufacturer's also began to implement their own proprietary sequence numbering algorithms (in accordance with standards so they could communicate with "unlike" systems). Unfortunately, most of these next-gen specifications were based on simplistic rules like (another gross oversimplification) simple mathematic derivations which didn't use any sort of "outside influence" to affect (or make more unpredictable) the sequence numbering. If you could figure out the equation, you could easily guess what packet number would be next, given one or two sequential packets, and then the whole mess would start all over again. Wikipedia actually has a very nice page on TCP with links to all the relevant RFC's.

This is actually still true today, although OS manufacturers, the Open Source community and white/black-hat hackers are stepping up their games to greater degrees and at a much more condensed pace (by which I mean things now, generally, change more often within a shorter period of time - another gross oversimplification ;)

So, now that it's the present, let's take a look at Solaris 10 and check out what they're doing with regard to this matter (From release 10/08 - the most recent to this date, as far as I know). Assuming we have a connection established with a Solaris 10 server (simple SSH), we can use snoop to gather the information we need to even begin attempting to figure out the TCP packet sequence numbering algorithm they're using (assuming - thinking positive thoughts :) that we can. A simple chunk of output is shown below:

host # snoop -r -d eri0 from 10.99.99.99 to 10.98.98.98
Using device /dev/eri (promiscuous mode)
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625835 Seq=673983919 Len=52 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625887 Seq=673983971 Len=52 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673984023 Len=52 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673984075 Len=340 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673984415 Len=148 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673984563 Len=148 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673984711 Len=148 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673984859 Len=148 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673985007 Len=148 Win=50400
10.99.99.99 -> 10.98.98.98 TCP D=2782 S=22 Push Ack=2865625939 Seq=673985155 Len=148 Win=50400


Of course, the column we're most interested in (at first, anyway) is the 9th column from the left (assuming spaces and tabs as the default field separator). The column beginning with "Seq=" is what we'll look at first. For the sake of this exercise, we've saved all the packets in a file called "packets." We didn't use -o on the command line, but this is a pretty simple cut-and-paste ;). To isolate those sequence numbers, we could just do the following:

host # awk '{print $9}' packets|sed 's/Seq=//'
673983919
673983971
673984023
673984075
673984415
673984563
673984711
673984859
673985007
673985155


And, here's where it becomes difficult. The first test of a reasonably stable (by which I mean, unpredictable) sequence numbering algorithm is to subject it two basic tests. That is to say, is it really secure or is it just a simple numeric or mathematic progression in sheep's clothing?

As we bring part 1 of this endeavour to an end, we'll do the quick and dirty testing (If Solaris fails these, then I may have made a big mistake when I named this post before I finished writing it ;)

1. Are the numbers in simple sequence (e.g. +1 or +2) over and over again?
Answer: No. The numeric difference between the packet pairs (1-2, 2-3, etc) are not infinitely exact.
Proof: The 9 differences in order:
673983971-673983919 = 52
673984023-673983971 = 52
673984075-673984023 = 52
673984415-673984075 = 340
673984563-673984415 = 148
673984711-673984563 = 148
673984859-673984711 = 148
673985007-673984859 = 148
673985155-673985007 = 148


2. Is there any direct correlation between the differences between packet sequence numbers and any other property of the packets?
Answer: Yes. If we scroll up the page and look at the basic snoop output, we can see that the difference in sequence numbers correlates exactly with the length of the packet.
Proof: The packet lengths in order:
host # awk '{print $10}' packets|sed 's/Len=//'
52
52
52
340
148
148
148
148
148
148


It seems as though we have something going here, but it must be harder than this to guess the sequence numbers. It's possible, also, that additional measures are being used that obviate the somewhat-flimsy dependence on convoluted number sequences to safeguard a transmission from "packet injection" or hijacking. For instance, Solaris 10 now makes use of SDP as part of its data transmission security (see, also, the Sockets Direct Protocol Manpage). It may also be that Devil is in the details and we only looked at a surface scan of our packets. We'll look at all of these possibilities in a near-future post. This isn't over yet :)

Cheers,

, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Monday, December 1, 2008

Bash Script To Find Free Online Fiction and Non-Fiction Books

Hey There,

SPECIAL NOTE: Some of The University of Pennsylvania's Online Books require a special reader, which is totally free - no sign up or even giving your email. Download The Ebrary Reader here


For some reason, it's Monday again and time for another "online reference ripper" script. For this week's Monday Linux/Unix bash shell script we're going to do the (almost) exact opposite of what we've been doing for the past few weeks. Today, we're going to find free books on the Internet and, possibly, read them ;) And we don't just mean man pages and reference materials, etc. We'll be using the Online Book Pages at the University of Pennsylvania. They may not have the variety that Amazon.com does, but everything listed on their site is available for online reading (although it says nowhere - that I could find - that you can't just download the books). Best of all, you don't have to sign up for anything. It's absolutely free. And they actually have a book by Stephen King up there (If you recall his chapter-by-chapter pay-as-you-go online book distribution experiment, you may recognize the title :) It's 100% free now but, to clarify, Stephen King never actually charged for the online book; he made his book available to anyone who wanted to read it and only asked that they donated money if they liked it. So, either everybody hated it or not enough people donated enough money to keep him interested ;) No offense to Mr. King. Writing is his profession and he deserves to get paid for it.

As always, skip the next paragraph if you're a regular reader. It's a listing of all the other "online reference ripper" scripts we've done to date.

Our previous web-based Bash scripts, in backward chronological order, include about 500 posts on getting your dish, cable and local listings from the CLI, accessing Wikipedia, accessing the Farmer's Almanac, accessing the International Dictionary, checking out the world's weather, spewing out famous quotations on pretty much any subject, doing encyclopedia lookups, accessing the online Thesaurus, translating between different languages and, of course, using the online dictionary.

The script for this week is just about as easy to use as any other we've put out. It should work on Linux or Unix, and - if you don't know what arguments to pass to it - you can get the help screen with either the "-h" flag or by just invoking the command with no arguments, like:

host # ./onlinebooks.sh
Usage: ./onlinebooks.sh [-h for help]
Required args - either one or both
[-t title] [-a author]
Quote title and/or author if multi-word!


Below are some screen shots to show the various error conditions and, just for kicks, some successful output :)

Click the pix below and don't get too close to the screen ;)

Online Book Title Listings


Online Book Author Listings

There are some shortcomings that I haven't been able to get around yet. For instance, I've visually verified that really long listings of books don't print out properly. A search on Leo Tolstoy (also known as Tolstoi -- I just learned that today ;), for example, yields only one listing, while there about 40 books available on the actual site. I'll get around to that as soon as I can.

In the meantime, since this script relies on you using a terminal client that hyperlinks web addresses for you (like NuTTY or PuTTY), I have something in the hopper for you that I'm hoping you'll all enjoy. It has to do with this script and will be an interesting way (I think) to get in more reading at work ;)

Until then,

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# onlinebooks.sh - Come On In, The Readin's Fine ;)
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/us/">Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License</a>
#

function usage()
{
echo
echo "Usage: $0 [-h for help]"
echo "Required args - either one or both"
echo "[-t title] [-a author]"
echo "Quote title and/or author if multi-word!"
echo
exit 1
}

while getopts t:a:h option
do
case $option in
t)
opt_title="$OPTARG"
;;
a)
opt_author="$OPTARG"
;;
h)
usage
;;
*)
usage
;;
esac
done

wget=`which wget`
book_title=`echo $opt_title|sed 's/ /+/g'`
book_author=`echo $opt_author|sed 's/ /+/g'`
noexactmatch=0
authors=0
titles=0

echo

$wget -nv -O - "http://onlinebooks.library.upenn.edu/webbin/book/search?author=${book_author}&amode=start&title=${book_title}&tmode=start" 2>&1|grep -i "No items were found" >/dev/null 2>&1

notfound=$?

$wget -nv -O - "http://onlinebooks.library.upenn.edu/webbin/book/search?author=${book_author}&amode=start&title=${book_title}&tmode=start" 2>&1|grep -i "No exact match" >/dev/null 2>&1

nomatch=$?

$wget -nv -O - "http://onlinebooks.library.upenn.edu/webbin/book/search?author=${book_author}&amode=start&title=${book_title}&tmode=start" 2>&1|grep -i "Browsing Authors" >/dev/null 2>&1

nomatch=$?

$wget -nv -O - "http://onlinebooks.library.upenn.edu/webbin/book/search?author=${book_author}&amode=start&title=${book_title}&tmode=start" 2>&1|grep -i "you requested books" >/dev/null 2>&1

btitle=$?

$wget -nv -O - "http://onlinebooks.library.upenn.edu/webbin/book/search?author=${book_author}&amode=start&title=${book_title}&tmode=start" 2>&1|grep -i "online books by" >/dev/null 2>&1

bauthor=$?

if [ $notfound -eq 0 ]
then
book_title=`echo $opt_title|sed 's/+/ /g'`
book_author=`echo $opt_author|sed 's/+/ /g'`
echo
echo "No items found for $book_author $book_title"
echo
exit 2
elif [ $nomatch -eq 0 ]
then
book_title=`echo $opt_title|sed 's/+/ /g'`
book_author=`echo $opt_author|sed 's/+/ /g'`
echo "No exact match found for $book_author $book_title"
echo
echo "Give these suggestions a go:"
echo
noexactmatch=1
elif [ $btitle -eq 0 ]
then
book_title=`echo $opt_title|sed 's/+/ /g'`
book_author=`echo $opt_author|sed 's/+/ /g'`
echo
echo "Title matches found for $book_author $book_title"
echo
titles=1
elif [ $bauthor -eq 0 ]
then
book_title=`echo $opt_title|sed 's/+/ /g'`
book_author=`echo $opt_author|sed 's/+/ /g'`
echo
echo "Listing online Titles for $book_title $book_author"
echo
authors=1
else
echo "Should not have gotten here. Looks like we need to fix something!"
exit 3
fi

if [ $noexactmatch -eq 1 ]
then
$wget -nv -O - "http://onlinebooks.library.upenn.edu/webbin/book/search?author=${book_author}&amode=start&title=${book_title}&tmode=start" 2>&1|sed -e :a -e 's/<[^>]*>/ /g;/</N;//ba' -e 's/$/\n/'|egrep -v 'exact match|previous'|grep -i title 2>&1|grep -v URL
elif [ $titles -eq 1 ]
then
$wget -nv -O - "http://onlinebooks.library.upenn.edu/webbin/book/search?author=${book_author}&amode=start&title=${book_title}&tmode=start" 2>&1|grep "SEARCH RESULTS"|sed -e 's/[^<ul>]*<ul>//' -e 's/<\/ul>.*$//g' -e s'/<li>/\n/g' -e s'/<\/li>/\n/g' -e 's/<\/ul>.*$/\n/g' -e s'/<ul>//g' -e s'/<\/ul>/\n/g'|sed -e 's/^[^<]*<a href="//g' -e 's/">.*<cite>/ : /' -e 's/<\/cite><\/a>//g' -e 's/<\/cite>//g' -e s'/<\/a>//g' -e s'/">/ /g' -e 's/<a href="//g'|grep -v "SEARCH RESULTS"|awk -F":" '{ if ( NF >= 2 ) print $3 " --- " $1 ":" $2 "\n"}'
elif [ $authors -eq 1 ]
then
$wget -nv -O - "http://onlinebooks.library.upenn.edu/webbin/book/search?author=${book_author}&amode=start&title=${book_title}&tmode=start" 2>&1|sed -e 's/[^<ul>]*<ul>//' -e 's/<\/ul>.*$//g' -e s'/<li>/\n/g' -e s'/<\/li>/\n/g' -e 's/<\/ul>.*$/\n/g' -e s'/<ul>//g' -e s'/<\/ul>/\n/g'|sed -e 's/^[^<]*<a href="//g' -e 's/">.*<cite>/ : /' -e 's/<\/cite><\/a>//g' -e 's/<\/cite>//g' -e s'/<\/a>//g' -e s'/">/ /g' -e 's/<a href="//g'|awk -F":" '{ if ( NF >= 2 ) print $3 " --- " $1 ":" $2 "\n"}'|sed -e '/^\/\//d' -e '/^ ---.*OBP/,$d'
fi

exit 0


, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.