Showing posts with label sh. Show all posts
Showing posts with label sh. Show all posts

Friday, June 5, 2009

The Bourne Shell War On Terror

Hey There,

I'm now, officially, a fan of The Thin Guy's Blog over at blogs.sun.com. At least, I'm officially a fan of the post I'm highlighting here today.

I don't really get too worked up over religion, politics and/or any other subject that most people hold very strong beliefs about (What's the point in having an argument that you can't possibly win? ;), but I thought this post, titled The War On Terror, as viewed from the Bourne shell was funny enough to share and absorb any backlash (that I'm positive I'm going to get) from family and friends.

I've included the basic post below, but you should check out the official page. A lot of the comments offer suggestions for improvement or addition that are just as entertaining :)

Cheers,



The War on Terror


As viewed from the Bourne shell.


$ cd /middle_east
$ ls
Afghanistan   Iraq          Libya         Saudi_Arabia  UAE
Algeria       Israel        Morrocco      Sudan         Yemen
Bahrain       Jordan        Oman          Syria
Egypt         Kuwait        Palestine     Tunisia
Iran          Lebanon       Qatar         Turkey

$ cd Afghanistan
$ ls
bin  Taliban    
$ rm Taliban
rm: Taliban is a directory
$ cd Taliban
$ ls
soldiers
$ rm soldiers
$ cd ..
$ rmdir Taliban
rmdir: directory "Taliban": Directory not empty
$ cd Taliban
$ ls -a
.            ..           .insurgents
$ chown -R USA .*
chown: .insurgents: Not owner
$ cd ..
$ su
Password: *******
# mv Taliban /tmp
# exit
$ ls
bin
$ cd bin
$ ls
laden
$ cd ..
$ rm -r bin/laden
bin/laden: No such file or directory
$ find / -name laden
$
$ su
Password: *******
# mv bin /tmp
# exit
$ pwd
/middle_east/Afghanistan
$ cd /opt/UN
$ ln -s /Bad_Guys/Al_Qaeda /middle_east/Iraq/.
ln: cannot create /middle_east/Iraq/Al_Qaeda: Permission denied
$ su
Password:*******
# ln -s /Bad_Guys/Al_Qaeda /middle_east/Iraq/.
# cd /middle_east/Iraq/Al_Qaeda
Al_Qaeda: does not exist
# rm /middle_east/Iraq/Al_Qaeda
# mkfile 100g /middle_east/Iraq/Al_Qaeda
mkfile: No space left on device
# rm /middle_east/Iraq/Al_Qaeda
# cd /opt/Coalition/Willing
# mkfile 1b /middle_east/Iraq/Al_Qaeda
# chown -R USA:Proof /middle_east/Iraq/Al_Qaeda
#exit
$ cd /middle_east/Iraq
$ ls
saddam
$ ls
saddam
$ ls
saddam
$ ls -a
.            ..           saddam
$ find / -name [Ww][Mm][Dd]
/Korea/North/wMd
$ wall Propaganda.txt
Broadcast Message from USA (pts/1) on USS_Abraham_Lincoln Th May 1st
Mission Accomplished!
$ rm saddam
saddam: No such file or directory
$ find / -name saddam
/var/opt/dictators/spiderhole/saddam
$ wall NewsWorthy.txt
Broadcast Message from USA (pts/1) on Time.Magazine Sat Dec 13
We Got Him!
$ mv /var/opt/dictators/spiderhole/saddam /opt/jail
$ cd /opt/USA
$ cp -Rp Democracy /middle_east/Iraq
$ cd /middle_east/Iraq/Democracy
$ ./install
Install Error: Install failed.  See install_log for details.
$ more install_log
Installed failed!
Prerequisite packages missing
Conflicting package Wahhabism found in /midde_east/Saudi_Arabia
Packages Church and State must be installed separately
File System /PeakOil nearing capacity
Please read the install guide to properly plan your installation.
$


Shirt Update:  New Shirts for sale over at Giant Robot Printing.  Please read here for details.  I will be closing down the CafePress shirt store, but will leave the poster up for sale.  After working with Ken from GRP, there is no way that all that text is going to look good on the CafePress Shirt.  I fully expect to have all the prior shirts returned.


Update:  If you don't get it, go here.





, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Saturday, December 27, 2008

Unix and Linux WebTech Humor: Bourne Shell Server Pages

Happy Saturday :)

Here's a little humour, from way back when, that's as true today as it was whenever it was originally published. It's a joke (or is it?) that puts forth a bold and visionary alternative to more complicated server page technologies, like ASP, JSP, etc. As the author suggests, it may all be a bunch of sh##, but that's by design ;)

This shining example of a Linux and Unix joke that you can actually put into practice, comes from Mark Lindner's Public Domain Software Site (It's actually his software, that he developed... not a bunch of unclaimed stuff that he put together in one place ;). To check out the original version of this page (including extra specifications, quotations and a bunch of other sh## <-- All this scatological prose will tie in nicely when you read the article, I promise ;), check out the SHSP Project Home Page. For this alone, Mark deserves a contribution. Check out his main page and, if you like this sh##, consider funding his future ...stuff ;)

Enjoy and remember to be kind to animals (even other humans ;)



Bourne Shell Server Pages



Introduction



Every work of art begins with a sudden flash of inspiration. In
my case, it all started when I received the following email from
my friend, Steve:






Hey Mark:

I just came up with a totally genius idea: Bourne Shell Server
Pages.

Why should shell languages be the only ones left out of the
${YOUR_LANGUAGE_HERE} Server Pages fad?

The way I see it, the advantages of SHSPs are as follows:


  1. runs everywhere (or at least everywhere on Unix and since I consider
    Windows to be nowhere...)
  2. slo-o-ow
  3. goofy

Now most rational people would consider points 2 & 3 to be drawbacks,
but as far as I can tell they are flat-out requirements for any popular
Web-based technology.

SHSPs are the Wave of the Future, so line up and sign up today!

Tune in next week when we invent 6502 Assembler Server Pages.


—Steve





Even as I was reading this email, my mind was racing,
subconsciously piecing together the implementation details. The
Bourne shell, it turns out, is quite well suited for this type of
task...much more so, I dare say, than Java or C. The solution turned out to be both simple and elegant.



This document chronicles my foray into the not-so-fascinating
world of ${YOUR_LANGUAGE_HERE} Server Pages
technology. All of the code I developed is available for free
download, so you can use Bourne Shell Server Pages to build your
very own killer Web application.



Implementation





The basic idea behind all server page technologies is this:
rather than writing code that generates an HTML document
on-the-fly by writing it out as a series of print
statements, you start with a "skeleton" HTML document and embed
the code right inside it. Voila! Instead of having a tangled,
unreadable, unmaintainable mess of HTML embedded in source code,
you have a tangled, unreadable, unmaintainable mess of source
code embedded in HTML.



Bourne Shell Server Pages are ordinary ASCII text files, with
the special extension .shit, which denotes
"Shell-Interpreted Template." The result of invoking the page
compiler on a .shit file, is, naturally, a shell
script. (It occurred to me that this file extension might seem
objectionable to some, but since it quite accurately—if
unintentionally—conveyed my sentiments toward Web technology in
general, I decided that it should be left unchanged.)



There are three basic elements that can be embedded in HTML:
blocks of code, expressions, and include directives. Each of
these is indicated by special HTML-like tags which are processed
by the page compiler to produce an executable entity (a Java servlet in
the case of JSP, for example), which, when run, produces the
resulting HTML document. I'll discuss each of these elements in
turn.


Code Blocks

Arbitrary blocks of code can be
inserted within an HTML document. Bourne Shell Server Pages
provides the <$ ... $> tags for this
purpose. Shell code that appears between these tags will be
executed, and the output of that code will be inlined in the
HTML in its place. For example:





<html>
<body>
<h1><$ echo "Hello, world!" $></h1>
</body>
</html>



The code can of course span multiple lines, and can be of
arbitrary complexity. What's more, all of the code blocks in a
given Bourne Shell Server Page are in reality part of the same
shell script, so they share a global state: for instance, a
block can reference variables that were assigned in an earlier
block.



Expressions
Arbitrary Bourne Shell expressions
can be inserted into HTML using the <` ... `> tags. These tags serve the same purpose as
do backticks in the Bourne Shell: the expression is
evaluated and the result is inlined in its place. For example,
to set the title of the document based on the contents of the
shell variable $title, one might do this:





<title><` $title `></title>



The above example assumes that a value was assigned to the
variable $title earlier in the page, namely, within
a <$ ... $> block.



Includes
Server pages can be composed of other
server pages using the include directive. The page compiler
evaluates pages recursively, so the nesting can be arbitrarily
deep. In Bourne Shell Server Pages, the tag <^ ... ^> indicates an include. For example the tag:





<^ more.shit ^>




will be replaced with the result of evaluating the Bourne
Shell Server Page more.shit.




The Bourne Shell Server Pages page compiler (shspc.sh),
which is tasked with converting .shit files into shell
scripts, actually consists of a series of sed
commands. These commands perform the following substitutions:


  • All contiguous segments of HTML are wrapped within here
    documents
    .

  • Include directives are replaced with recursive calls to
    shspc.sh to process the included Bourne Shell Server Pages.

  • Expression tags are replaced with a call to the shell's
    eval command.



The result is a script which, when interpreted by the Bourne
Shell, produces the desired HTML page to standard output.



Most other server page technologies rely on the presence of a Web
server, which is itself a complex piece of software that often
requires a substantial amount of maintenance. Bourne Shell Server
Pages has no such requirement, as it includes its own Web server,
implemented as a small shell script (in.httpd.sh) that is launched
from inetd. While certainly not a full-featured Web server,
it does quite an admirable job of serving HTML documents, images,
and of course, Bourne Shell Server Pages.



So What About Web Services?



How does the Bourne Shell Server Pages technology fit into the
bigger picture of Web Services? It's a legitimate question. For
that matter, what the hell are "Web Services" anyway?



I've read quite a bit about Web Services, and have had some
in-depth, first-hand experience with the technologies that form
their underpinnings. To the best of my knowledge, here is an
accurate definition of the term:




Web Services  noun  A software development
meme that espouses the notion of tying together disparate software
components via a crude, non-typesafe, remote procedure call (RPC)
mechanism that consists of sending and receiving data encoded in
an excessively verbose, plaintext format (XML) over a largely
inelegant, stateless file transfer protocol (HTTP).





Whew. That doesn't sound glamorous or exciting at all. It's not
even object oriented. There must be more to this than just
inferior reinterpretations of old ideas? Sadly, there isn't.



One particularly curious aspect of Web Services is that all
communication between components must take place over TCP port
80. The other ports (all 65,534 of them) constitute a veritable
Pandora's Box of perceived dangers, horrors and evils, and so
Thou Shalt Not Bind Them. It's painfully clear that Web
Services exists along three distinct axes: a technical one, an
emotional one, and a decidedly religious one.



It turns out that aside from the fact that it employs HTTP, the
network protocol of the World Wide Web, Web Services really has
nothing to do with the Web. "HTTP Services" would probably be a
more appropriate name for the technology. (Sure, that sounds
pretty silly...but it is what it is.)



The answer to the thesis of this section is thus very simple:
Bourne Shell Server Pages doesn't fit into Web Services,
any more than does any other ${YOUR_LANGUAGE_HERE} Server
Pages technology.


That being said, Bourne Shell Server Pages is a compelling
approach to the problem of providing Web access to existing
software components, which incidentally happens to be the major
focus of Web Services. Other Web technologies require complex
software plumbing in the form of adapters, bridges, and frameworks
to expose legacy software systems to the Web. However, due to the
radically expressive and flexible nature of the Bourne Shell,
Bourne Shell Server Pages makes this problem all but disappear:
any program on the system can be invoked from within a Bourne
Shell Server Page, its output becoming immediately available on
the Web.


For example, providing a Web-based search front-end to a large
CSV-data file is almost trivial with Bourne Shell Server Pages,
since one can rely on the presence of powerful utility programs
for manipulating data, e.g.:






<table>
<tr>
<th>Last Name</th>
<th>First Name</th>
<th>Phone</th>
<th>State</th>
</tr>

<$ IFS=','; grep -i $pattern datafile.csv | \
( while read lname fname phone state;
do
$>

<tr>
<td><` $lname `></td>
<td><` $fname `></td>
<td><` $phone `></td>
<td><` $state `></td>
</tr>

<$ ;; done ) $>

</table>





The true power of Bourne Shell Server Pages stems from the fact
that any program can be executed from within a Server Page,
with no intermediate "glue" logic required. Think of SHSP as "Web
Services without all the layers upon layers of crap."



Downloads





You, too, can build exciting dynamic Web content with Bourne
Shell Server Pages.

There are no pre-requisites for deploying
the technology other than a working Bourne Shell and a decent
operating system to run it on. (Those modest requirements,
unfortunately, rule out these products.)




shspc.sh - page compiler (710 bytes)
in.httpd.sh - Web server (834 bytes)




Installation is simple. Place the above files in
/usr/local/bin, and then install this href="http">http file in /etc/xinet.d (or the
equivalent). Then send a HUP signal to xinetd.

Securing the installation is left as an exercise for the reader.



Credits



Web technology is certainly a controversial topic. Despite the
fact that building Web applications is about as enjoyable as
having one's private parts caught in a cement mixer, it continues
to be an extremely popular software development paradigm. It's
been said that although there is a certain class of problems
for which Web technology is the appropriate solution, your problem
is almost certainly not among them, no matter what your problem
is
. But one would be hard-pressed to convince the industry of
it.



As do most belief systems, the Web technology religion has its
share of rabid followers. In order to avoid their wrath, I remain,
humbly and respectfully yours, Anonymous.



Some astute and enterprising readers have graciously submitted
bugfixes and improvements for SHSP. I have included these into the
official release.




, Mike




Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, July 9, 2008

Using Traps Outside Of Shell Scripts On Unix Or Linux

Hey There,

Generally, when most folks think of the trap command, it's in the context of shell scripting. This is for good reason since it's always good practice to trap any signals that you need to handle. For instance, if you don't want your script to be crashed by someone hitting control-c (or sending a SIGINT), you can trap signal number 2 (SIGINT's corresponding numeric value) and execute some other command (or nothing) if a user tries to crash out of your script that way. It's not iron-clad/bullet-proof security, but it's good enough to protect against accidental damage or interruption. If you hit the "previous post" link a hundred times (or just check this link to our old post on trapping signals in shell and Perl ;) that should give you some idea of the advantages of trapping signals in your scripts.

Another venue, that often gets overlooked, but is just as useful (if not more so), is the use of traps in the command line interface. These can easily be introduced into a user's environment using whatever source file launches by default with the invocation of the shell (.profile for sh/ksh or .profile/.bashrc for bash, etc). Combining this functionality with security measures of a slightly lesser degree than those proposed in our post on making accounts su-only, can make controlling the average user's ability to muck with the shell that much easier to manage for the working sysadmin.

In simplistic fashion, you can test this yourself at the command line, by typing the following (this should work in sh, ksh and bash on Linux or Unix - pick a flavour:

host # trap 'echo Quit Interrupting Me!' 2

and then type a control-c character at your next shell prompt to get the following response:

host # ^C <-- This should be the control key and the "c" key pressed simultaneously. If you ever want to actually print control characters and retain their effects, check out our post on how to really write control characters.

host # Quit Interrupting Me!

host #


You can then, list out the trap you set (for kicks) and simply reset it by doing one of a few things (This is different and, in some instances, the same in sh, ksh and bash):

1. In sh: To list out the traps you've set, do:

host # trap
2: echo Quit Interrupting Me!
<-- Note that sh uses the signal "number"

and to reset them, just type:

host # trap 2
host # trap
<-- To make sure it's gone. In all shells you have to do this for each one separately. Check below for a simple way to reset them all.
host #

2. In ksh: To list out the traps you've set, do:

host # trap
trap -- 'echo Stop Interrupting Me' INT
<-- Note that ksh uses the signal's 3 letter abbreviation

and to reset them, just type:

host # trap - 2 <-- Note the additional - argument ksh requires
host # trap <-- To make sure it's gone.
host #

3. In bash: To list out the traps you've set, do:

host # trap -p
trap -- 'echo Quit Interrupting Me!' SIGINT
<-- Note that bash uses the signal definition found, most commonly, in your sys/io include file.

and to reset them just type:

host # trap - 2 <-- bash also requires the additional - argument.
host # trap
host #


On to resetting traps more efficiently!

As you've noted above, we've gone through the simple steps of setting a trap, executing it and then removing it. While this is relatively painless to do for a trap set on one signal, this can become a huge PITA if you're dealing with more than one signal (and, possibly, multiple signals from a finite pool, the size of which you're not sure, which are all possibly set with traps). Luckily, this is fairly easy to take care of.

Doing the following will wipe all of the traps you have set on all available signals in one fell swoop. Then you can walk away and get on with your life ;) Note that it's slightly different in sh/ksh and in bash.

For sh and ksh, the trap command is external, so you're restricted to it's functionality. On Solaris 8, which I'm using right now, there are no Gnu niceties. But, this is still okay. We already covered listing out all of your traps, and we can reset all of those traps for all three of the shells we're looking at today, each in one step:

1. In sh, to reset all of your traps, you'll need to do some educated guessing to figure out how many basic signals you have to work with, so that you can be sure you reset them all and don't reset way more than you need to. You can usually get this information from an include file, like /usr/include/sys/iso/signal_iso.h. Just start your range at zero and end with the highest number signal in that file. 37, for Solaris 8, is a safe number and covers more bases than most people ever cross. So, to reset the traps you have set on all 38 signals, just type:

host # trap ${seq 0 37}

The rub here is that this actually won't work on the operating system version I'm using, since it doesn't support the "seq" command or the ".." range operator. To preserve the sanctity of this noble old shell, just use a simple while loop to iterate through, like so:

host # x=0;while [ $x -lt 38 ];do trap $x;a=`expr $x + 1`;done
host # trap
<-- To double check that they're all gone.

2. In ksh, you still can't use seq on older versions of Solaris, but it will work on Linux and more recent Solaris releases, so you can cut down your "trap reset time" by doing:

host # trap - $(seq 0 37)
host # trap


3. In bash, just to sidetrack you one last time, you'll be using a shell built-in version of "trap," which has a handy flag that you can use to list out all the available signals (rather than doing it the old-fashioned way, like above):

host # trap -l
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL
5) SIGTRAP 6) SIGABRT 7) SIGEMT 8) SIGFPE
9) SIGKILL 10) SIGBUS 11) SIGSEGV 12) SIGSYS
13) SIGPIPE 14) SIGALRM 15) SIGTERM 16) SIGUSR1
17) SIGUSR2 18) SIGCHLD 19) SIGPWR 20) SIGWINCH
21) SIGURG 22) SIGIO 23) SIGSTOP 24) SIGTSTP
25) SIGCONT 26) SIGTTIN 27) SIGTTOU 28) SIGVTALRM
29) SIGPROF 30) SIGXCPU 31) SIGXFSZ 32) SIGWAITING
33) SIGLWP 34) SIGFREEZE 35) SIGTHAW 36) SIGCANCEL
37) SIGLOST


Sure, it's a bit more than you needed to know, but it tells you how many signals you have and comes in pretty handy as a quick reference :)

Now to reset all of your signal traps in bash, you can use either seq (on Linux or newer Solaris) or, even better, bash's built-in range operator (which can be used to easily copy files, as shown in our previous post regarding simplified file renaming), like this:

host # trap - {0..38}
host # trap


And you're back to square one ;)

, Mike

Friday, July 4, 2008

Basic I/O Redirection Differences In Sh/Ksh, Bash and Zsh On Linux And Unix

Hey there,

Today we're going to look at four different shells (all tests were run on a Solaris 8 box and an old RedHat Linux 9 server) and some very basic I/O redirection (which can come in handy if you're ever stuck trying to read and write files in a blind shell), to show you how differently, and commonly, they all handle the same information. The only reason I find this sort of stuff remarkable is that it's so common, sometimes its hard to wrap my head around the fact that any distinctions exists (Okay, they're not HUGE, but the differences, where there are any, are enough to make you wonder ;) For instance, in our two experiments, we're only going to find one difference each. But, since the material we're working with is so basic (not meaning "simple"), it's a bit disconcerting (...oh, the drama ;)

First of all, we'll narrow down our list of shells and consider sh and ksh to be the same, since all of my testing, with regards to these two shells, yielded exactly the same results. Note that, for all of our tests, we only tried to create two conditions in a variety of ways; the first was to create an empty file, and the second was to redirect standard error (STDERR) to a file. See what I mean? This is very familiar ground for almost all Unix or Linux admins.

Now let's take a look at a how sh/ksh, bash and zsh are the same, when it comes to basic I/O redirection.

First test: Create an empty file. The following methods worked in all 4 shells (producing the expected output from the file and "wc -l" commands of "empty file" and 0, respectively):

host # : > FILE

This creates an empty file by using the colon (:) placeholder (which, technically, is a simple notation for "true") as output to be redirected into the blank FILE

host # /bin/true >FILE

Pretty much the same as the previous example, except we're running /bin/true explicitly.

host # /bin/false >FILE

Again, the same as the previous example, except we're using /bin/false for a goof. It's another program that doesn't produce output and is mainly used to generate a desired return status.

host # cat /dev/null >FILE

Just catting nothing to the FILE, and it ends up empty, as expected.

host # <>FILE

and this works as well (specifying empty redirection of STDIN and STDOUT) on all 4 shells.

host # echo <> FILE

While this, finally, spit a line of output to the screen, it still produced an empty file in all 4 shells, even with extra commands to echo, since they all spill out to the screen instead of going into the file. Kind of a waste of resources, but interesting somewhat ;)

host # echo how are ya <> FILE
how are ya


Now's here's the only basic thing I could find where they differed, in this respect:

host # >FILE

This produces an empty file on Solaris sh, ksh and bash (generally, it's shorthand for "cat /dev/null >FILE"), but results in a perpetual hang with zsh (zsh will not infer /dev/null (or true) from a lack of argument preceding the > redirection symbol).

Second test: Redirecting STDERR to a file. The similarities.

host # truss echo hi >FILE

This produces a non-empty file (with the word "hi" in it) in all 4 shells, since truss sends its output to STDERR and we're only redirecting STDOUT. This doesn't really prove any part of the experiment, but I wanted to put it out as a baseline, so we know that the truss output is going where we expect it to before we start goofing around :)

host # truss echo hi >FILE 2>&1

This, tying STDOUT and STDERR together to put all output into the file, works on all 4 shells as well :)

But, the following command only "really" works as expected in bash, and produces remarkably different results for the other shells:

host # truss echo hi &>FILE

In bash, the &> notation is shorthand for ">FILE 2>&1" - but this doesn't work for our other shells. In Solaris sh and ksh, the command spits output to the screen, since the & activates the shells' backgrounding feature, the job completes and the output file is completely empty. On zsh, "hi" gets spit to the output terminal and only STDERR goes into the output file (???)

Remember, these have just been really basic examples. Thinking about this sort of stuff makes you realize how truly "different" different shells are. Of course, among these very basic examples, we could only find two major
differences, but that's still a good number considering how rudimentary the shell functions are. In other words, these are I/O redirection issues which you can't change, except by script-wrapping (which probably isn't worth it, since, for each difference we found at least 2 similarities between the 4 shells and both experiments only tested one property).

If you're not sleeping yet, I'll assume this article was, at least somewhat, entertaining and/or helpful ;)

Have a Happy 4th of July!

, Mike

Thursday, February 28, 2008

Finding and Reading Files In The Shell When Your System Is Hung

Howdy,

Today we're going to look at a situation that occurs probably more often than it should ;) If you've done your fair share of system administration on Linux or Unix, you've probably run into a predicament where you got paged, called, etc, to look at a machine that was hanging on the brink, only to find that (asssuming you could log in at all) it was so completely trashed that it couldn't even muster up the strength to run the simple system commands you needed to diagnose the problem before just giving up and rebooting.

Typical errors you'd get at the command line, in this sort of situation, would be similar to:

host # ls /tmp
Insufficient Memory!


or

host # cat file
Cannot fork new process!


Basically, you're stuck in a situation where you can't do anything that relies on executing anything other than the login shell that you're lucky you got in the first place ;)

The good news is that you can, quite possibly, get as much information as you need before rebooting by simply using your shell and it's built-ins. All of these examples should work for sh, ksh, bash, etc (Possibly not in csh, but, hopefully that's not your system default root shell). If you have a good idea what's wrong before your machine goes down entirely, they can even help you decide what you want to do before you boot the system back up (Check out this post for some simple tricks to figure out what commands are available to you in Solaris' PROM).

Here are the things I try to do when I find myself in that sort of situation (in no order of importance ;)

1. Move around the filesystem.

Luckily, the "cd" and "pwd" commands are built into the shell, so you can always move around your filesystem (even if you are, figuratively, in the dark) and get to the hot spots you want to check. For instance:

host # cd /var/log

will work just fine. This is the most obvious thing you can do, but cd (on its own) depends on you to know where you're going. You can't cd to a directory that doesn't exist even when things are up and running perfectly ;)

If you happen to get lost, you can figure out where you are, using the built-in "pwd," like so:

host # pwd
/var/log


2. Take a look at the contents of your filesystem.

This is actually pretty obvious, as well, once you realize how to do it. You won't be able to use "ls" any more, since that is a command that the shell invokes outside of itself, but you can always use the built-in "echo" command, like so:

host # echo *
bin opt sbin usr tmp var


Note that this output won't usually be so pretty. If there are 50 files and/or directories in the directory you're in, you'll just get 50 filenames in a row on one line. But, it's better than nothing :)

3. Capture the contents of critical files that can help you troubleshoot and/or find your root cause so this won't ever happen again (hopefully).

This last one is slightly less obvious, but can be done in a variety of ways. The first two ways are messy, but they work. Simply read your file as though you were sourcing it, using either the "source" or dot "." built-ins. The reason I don't prefer these two methods is that your screen fills up with a lot of garbage and you may hang your system by sourcing the contents of a file that contains executable statements or commands. For our example here, we'll assume a file named BOB with one line in it that says "hi":

host # source ./BOB
-bash: hi: command not found


or

host # . ./BOB
-bash: hi: command not found


You've gotten your output, but you can see where the potential problem would lie. What if "hi" was a command and your "source" directive tried to run it on your already half-dead machine? It might put it all the way down right then and there.

In these instances, I think output redirection is your best bet. You can read the contents of a file by simply executing a new file descriptor and reading from that with "echo." You can use pretty much any number that works (although, try to stay away from 0, 1 and 2 as these are your system's STDIN, STDOUT and STDERR file descriptors), you won't have to read your file through the clutter of a bunch of error messages and you will be insulating yourself from accidentally executing any commands. For instance, the following would get you much better results:

host # exec 7<BOB <--- Execute new file descriptor 7 and redirect the output of your BOB file to it.
host # while read -u7 line; do echo $line; done <--- Then, just read from the file descriptor.
hi

And that's about it. If you use all 3 of these methods in whatever combinations are necessary, you should be able to collect most of the information you need to assess your situation and/or provide root cause.

For one last example that uses all 3, this is how I would go about getting the contents of my /var/log/syslog file (I'll shorten the output to only include the relevant stuff) - Note that I'm also doing the syslog reads in a command line "for loop" because I want to get all the information I can with as little typing as possible:

host # pwd
/home/mydirectory
host # cd /var/log
host # echo *
sysylog syslog.1 syslog.2
host # for x in syslog.2 syslog.1 syslog; do exec 7<$x;while read -u7 line;do echo $line;done;done
blah....
<--- All of the sylog files' output. Notice that I did them in reverse, so that the output would be from oldest to newest. It's also a good idea (if possible) to either log your terminal session or set your terminal client's buffer to a very large number so that you can cut and paste this output into your desktop editor.

Hope this helps you out :)

Best wishes,


, Mike




Tuesday, December 11, 2007

What To Do When nohup Hangs Up Anyway - Down and Dirty

nohup is one of those commands pretty much everyone who uses Unix knows about. And for good reason. If you don't have access to a system console, or some other direct connection to a machine at work, it makes sense that you'd want to avail yourself of any utility that makes it possible for you to begin running a command through your terminal software, shut down your PC and go home. Even people who've never thought of this before will find this command almost immediately after typing something like "./sqlplus @extremelyLongDatabaseQuery" 5 minutes before quitting time ;)

nohup is the natural end-point of most searches since it's been around so long that it's actually a built-in in certain shells.

nohup stands for, literally, "no hangup." This, in layman's terms (and terms that make more grammatical sense), means that any command, run by nohup, will ignore the hangup signal (or SIGHUP) which is issued when a user disconnects from his or her pseudo-terminal (pty). Anyway; that's what it's supposed to do. In reality, I'd give it about a 50% reliability rating. One of the complaints I hear the most on the job is that "such and such" a command was run with nohup, the user disconnected and the process never completed. It, in plain fact, stopped running as soon as the user logged out; which sent the hangup signal. Yes, nohup is specifically designed to ignore that (nohup /sqlplus@extremelyLongDatabaseQuery really works in theory ;)

Alternatives to this command exist, of course. On my ratings scale, from worst to best, you've got (to my knowledge):

1. nohup <-- Included as an alternative only to hammer-home how much heartache this command causes me on a regular basis.
2. disown -h JOBID <-- This is available in bash, and exists as a way to block SIGHUP on an existing job. Your JOBID can be found, while still in your shell, by typing: jobs (If your job is listed as number 2, you can then run "disown -h %2" - In fact, you don't really need to include the "-h" flag. The outcome is pretty much the same, since disown'ing the job removes it from the job table)
3. setsid <-- Not always available. I haven't used it frequently enough to know exactly what distro's it comes standard with, but I've run it enough to know that it does run commands in a separate session (not attached to your pty) which will keep your command running after you log out.
4. GNU Screen <--- This is "almost" the best because you can run a process in one screen, disattach it, log out, and that same screen can either be accessed by you at a later point, or by any other user with access, from a different pty at any point in time before its conclusion.
5. The Down and Dirty Method <--- Explained below.

The Down and Dirty Method isn't an industry term; just what I like to call doing what most of the above mentioned programs are built to do in a way that, in my experience, is more consistently effective. In order to ensure that my programs keep running after I log off (if I happen to not have access to a console), I do the following: background the command I want to run and then execute that backgrounded command in a subshell. This probably sounds more complicated than it actually is. For example; to background a command, instead of typing:

./command

you would type

./command &

The ampersand character runs the job for you in the background. You can verify that it's still running by typing jobs at the command line.

To run a command in a subshell (in sh, ksh and bash, for example), you just need to surround it with parentheses, like so:

(./command)

You won't be able to verify that this job is running by typing jobs, but (assuming it has a long enough run-time or you're an incredibly fast typist) you can see it in the output of "ps -ef." You should note that, when you execute a command like this, the ps output will show your user id as the process owner.

Now, to make use of the Down and Dirty Method, you just need to type the following amalgam of the above concepts:

(./command &)

And that's it. You've backgrounded the command, by following it with an ampersand, and executed it in a subshell by surrounding it with parentheses. The key reason this process almost always works is the way in which the shell processes your request. First, it invokes the subshell in which to execute your command. Once in the subshell, it executes the command and processes the ampersand, which instructs it to background the process. Since the process you're executing is being run in a subshell, when the subshell backgrounds it, the process disconnects itself from your pty. You can also verify this in "ps -ef" output; noting that the user 1 (init) is listed as the processes owner. It's not going anywhere until it's finished. Unless you have the privilege to kill the process.

Hope this helps you out in some form or fashion. Even if we agree to disagree on what works best, it's always good to have alternatives :)

, Mike





Friday, December 7, 2007

The Shebang Line: An Introduction to Porting Shell to Perl

This is actually going to be, over time, a series of posts because it would be impossible (especially with my writing style) to cover every aspect of porting shell script to Perl in one posting (There may be some unusual or vague aspects that won't be covered at all, but, by that point, you will understand enough to be able to find the answers through trial and error, or judicious use of various search engines ;)

In this post we're going to cover one of the very basics. By "very basic," I mean a brief discourse on the "shebang" line. There's so much to mine in that seemingly obvious topic that it'll take far too long for me to crank out anything more today. After this, in a future post, we'll follow up with how to properly format a standard line of code, the three common variable types and how they are created and accessed in either language.

The beginning is complicated enough, when you really look at it. The "shebang" line (It's referred to as this because the exclamation point is referred as a "bang," for reasons which I don't completely understand, and the pound symbol is... I honestly can't come up with any intelligent reason for the other part of the name ;). I still call it the "pound bang" line.

In shell and in Perl, the "shebang" lines serve the same purpose. In fact, the "shebang" line is an entity unto itself. Its use extends beyond shell and/or Perl scripting and is a construct used for many purposes. The main thing the "shebang" line does is instruct the shell that you're using (you can be logged into csh, tcsh, ksh, sh, zsh or whatever shell you like to work in) to invoke a specific "command" to initiate the script with. It is customary to make this the full path to the shell in which you are writing your script, or the full path to the Perl binary, but it doesn't have to be (although, what you are asking the "shebang" to execute should be directly connected to it. No space between "#!" and "/bin/sh." Lots of shells nowadays will consume this blank space for you. Some older versions of shells are far less forgiving). For instance, our two basic examples are these:

#!/bin/sh <--- Tells the shell's interpreter to invoke "/bin/sh" in a subshell and run all the commands in the script in that subshell.
#!/usr/bin/perl <--- Tells the shell's interpreter to invoke "/usr/bin/perl" in a subshell and run all the commands in the script in that subshell.

This doesn't mean you can only use one shell or even one version of Perl within the same script, but that's outside the scope of today's post. So, to illustrate what I mean when I say the "shebang" line instructs the shell to invoke a specific "command," I mean just that. If you wrote a script, in the editor of your choice, and made the first line:

#!/bin/rm

the default action of the shell would be to delete the file and then run all the following commands (reading the script - in both shell and Perl - left to right, from top to bottom) until the script ended. If this seems counterintuitive, which it did to me at one point, that's normal. This is a good example of what goes on "under the covers" when you execute a script in your Unix or Linux shell. In the above example (as in the previous two, and all others), the entire script is read into memory before the "shebang" line is executed. That way, when your script with "#!/bin/rm" at the top is executed, the following happens:

1. The script is read into memory.
2. The script is executed.
3. The first line ("shebang") instructs the script to delete itself.
4. The rest of the script is run out from memory.
5. When the script is done, everything you instructed the script to do has been completed (provided it's at all possible in the shell you're executing it from (since the default subshell will be a subshell of the shell you're in when you execute the script) and the script is no longer there! In actuality, the script disappears before the code is executed from memory, so your script is gone, sometimes, long before your script finishes! For example, let's look at the following script:


#!/bin/rm
# /export/home/user/MyScript.sh c) 2007 MT - xyz.com
#
# Pounds are considered comment lines in both shell and Perl
# You can include interesting notes or whatever you like behind
# the pound sign and the script won't execute it. Even the "bang"
# ! has no special significance after the first line of the script
# except in special situations where you're invoking another
# subshell or another "shebang."
# Blank lines are also not acted upon in either shell or Perl.

sleep 500000000000000000


Now, if we were to run that script at the command line (after doing a chmod to 700 so that we, the user, have full read, write and execute permissions on it) we would end up sitting at a hung prompt, waiting for the sleep command to finish for a very long time. In this case that's good, because it gives us ample opportunity to login to the same machine again, on a different terminal, and have a look at what's going on:

This is an example of what our first terminal session would look like:

user# cd /export/home/user
user# ls -a
. .. .profile MyScript.sh
user# /bin/bash ./MyScript.sh
user#


Now, if we log on to the machine again, using a different terminal (or another connection, window; however you want to put it) we'd see this:

user# cd /export/home/user
user# ls -a
. .. .profile
user#


Yet - we can check that the script is still running, which it is:

user# ps -ef|grep "[M]yScript.sh
user# user 6523 6521 0 15:12:03 ? 0:14 ./MyScript.sh


And thus is all that explained. You can use the "shebang" line to run any command you want. Try it out. Be careful, because you can do some incredibly awful things if you're not careful. Luckily, the shell will protect you against doing something like this (but the way it reacts is specific to the command invoked, so some programs, like Perl, can accept a switch after the command line, like - #!/usr/bin/perl -w):

#!/bin/rm -rf /export/home/user

as it only executes the full qualified "rm" command directly attached to the "shebang." If you quote the entire command to try and get around it that way, it will try to interpret the entire line as one command and shoot you an error like:

ksh ./MyScript.sh: not found

Hopefully, this has been a somewhat informative and easy to understand introduction to the relationship between the shell and Perl. I understand that this bedrock we've discussed today isn't really specific to either of them, although it applies to both. But, as an old saying goes: It's usually best to start with the first floor (or the basement) when you're putting up a building ;)

Best Wishes,

, Mike



Thanks to Michael Pelletier, Merrimack NH for this perfect explanation of what that pound symbol means!


It's a "sharp" symbol in music notation:

http://en.wikipedia.org/wiki/Sharp_(music)

That's why the "C#" language is pronounced "C sharp," not "C plus plus plus plus:"

http://en.wikipedia.org/wiki/C_Sharp_(programming_language)

Enjoy this little tidbit of mostly useless knowledge. I pronounced it "pound bang" for maybe 20 years.