Using Mkfifo For Monitoring And Enhanced User Activity Logging

Hey there,

For today's post, if you're unfamiliar with how to execute new file descriptors and work with shell redirection, to a comfortable degree, check out our older posts on those subjects (cleverly anchor-linked into this very paragraph). They explain more basic stuff regarding those topics better than this post will and, if you want to, they'll be there for you to read for as long as I can remain within the hosting company's good graces.

One basic component of any decent security setup should always be logging. At the very least, if you're running a server that needs to be on, and servicing requests, 5 days a week (or 24x7, etc), you're probably logging sar output or iostat, vmstat, etc. You're taking some sort of metrics so that you'll be able to troubleshoot any problems that may arise in the future when your needs outgrow your capacity, or somebody just makes a mistake and runs an infinite recursive fork-and-exec loop as root.

Another thing that a lot of companies like to do is to keep tabs on their employees. Especially as it pertains to their activities on production Unix and Linux servers. This may or may not be for a bad reason. Some companies (depending on their nature) are required to monitor user activity to maintain standards compliance, while others may just do it so that if user XYZ makes a mistake and crashes the machine, they can figure out that it was he/she who did it and make sure that it doesn't happen again (hopefully, by explaining why what happened happened and considering it a lesson learned. It depends heavily on the workplace and the situation, of course)

One way to keep an eye on your users is to run kernel process accounting (pacct, which is available for Linux and Unix) and kernel auditing. Both of these probably provide the most complete picture in the event of a catastrophe, but they come at a cost because they can exact a heavy toll on your systems even when everything is fine. They can also make it so that things aren't fine any more, or expedite the non-fineness (???) of a situation if anything "bad" happens. Imagine how much harder your kernel and/or memory is being interrogated when there's a problem with it. Infinite recursion comes to mind again. The ghosts of those Fibonacci numbers just never go away ;)

In my practice, unless it's specifically requested, I prefer to log via log file. I know it seems primitive, and in a sense it is, but it's very "inexpensive" in terms of performance cost. One tool you can use to help with system auditing is called "mkfifo." It, of itself, doesn't really do anything for you, but it can be a great facilitator (especially if you use your imagination and have the latitude to experiment).

mkfifo is a program that creates a simple named pipe. You can equate it to the "|" that you use in your every day command lines, like:

host # cat FILE|grep word

although it's slightly differently. When a named pipe is created, via mkfifo (or however else you can do it), it creates a pipe "file" that remains in place until it is removed (or, in some cases, until your machine reboots, if you forget to remove it). You can create your own named pipe with mkfifo simply, as it takes very few arguments, like so:

host # mkfifo -m 777 /tmp/corncob
host # ls -l /tmp/corncob
prw-r-----   1 user    group       0 Jul  9 15:11 /tmp/corncob

That's all it takes to create the named pipe /tmp/corncob. The -m flag, which is used to set the permissions, is not necessary. Generally, if you don't include it, the default permission set for a new named pipe is whatever the default for your system would be. As another side note, you can also pass the -m flag and set alpha permissions, rather than octal, like:

host # mkfifo -m a=rwx /tmp/corncob

to create the exact same thing. You can delete the named pipe just like you delete a file. rm, and it's gone.

One thing you should note about named pipes is that they generally (so far as I've seen) are only able to fully pass one stream of input/output through themselves at a time. That is to say, if you have one process sending input to the named pipe and two process reading from it, only one of the reading processes will receive output. It should be noted, also, that, if such a situation were to exist, once the original process that was receiving output exits, the other process would begin receiving output from the named pipe (if it was still attempting to read from it). Was that a really long sentence or am I just typing fast? ;)

An example of what I mean below:

host-term1 # while :;do echo a b c d e >/tmp/corncob;sleep 15;done

host-term2 # tail -f /tmp/corncob
a b c d e
a b c d e
a b c d e
a b c d e
a b c d e
a b c d e
a b c d e
a b c d e

host-term3 # tail -f /tmp/corncob

host-term2 # ^C

host-term3 #
^C
a b c d e
a b c d e
a b c d e
a b c d e

Now, if you combine this freely available named pipe (you should probably secure it with permissions so that only users you want to be able to actually modify it have permission to remove it. Everyone you want to manage must be able to write to it and execute it.) with any number of output capturing mechanisms, you've got yourself a logger. Of course, you'll need to take care and work out how you want to parse the output coming from the named pipe because, even though it can only be read from by one process at a time, it can be written to by many and (if you have 15 people logged in, all with duplicate STDIN and STDOUT getting pushed through the same named pipe) that could get confusing for you very quickly.

My first recommendation would be, of course, to initiate an individual named pipe per user login process. This is so lightweight that generating 60 named pipes isn't going to cost you much more in overhead than generating six.

A simple thing to do is to use script to generate output and redirect it to the named pipe, like this:

host # ksh -ic "script /tmp/corncob"

using the name of the FIFO as the name of the output file for script. This is generally a bit klunky (script dumps the output in chunks and sometimes loses bits), but it does work relatively well. All you need to do is modify that slightly and put that in your users' .profile files and they'll be writing their entire sessions to /tmp/corncob (or, like I mentioned, each to a different named pipe file: /tmp/username perhaps?)

I think you can begin to see the possibilities here (and, probably some of the pitfalls - for instance, you can't use "tee" since it won't allow interactivity), but we'll continue on this topic tomorrow and look at some specific ways in which you can use named pipes, in conjunction with STDOUT and STDERR redirection, to do some serious security logging without serious stress :)

Cheers,

, Mike

Piped Variable Scoping In The Linux Or Unix Shell

Hey There,

Today we're going to look at variable scoping within a piped-while-loop in a few Linux and Unix shells. We're actually almost at this point in our series of ongoing posts regarding bash, Perl and awk porting.

Probably the most interesting thing about zsh (and shells that share its characteristic, in this sense) is that the scope of variables passed through a pipe is slightly different than in other shells, like bash and ksh (Note that not all vendor's versions are equal even if they have the same name! For instance, HP-UX's sh is the Posix shell, while Solaris' is not) I'm taking care to separate the piping construct from the "is the while loop running in a subshell?" argument, as I don't want to get too far off course. And, given this material, that can happen pretty fast.

For a very simple demonstration of whether the scoping issue is a "problem" (defining problem as either a bug or a feature ;) with the while-loop or pipes, we'll look at a very simple "scriptlet" that sticks to using a while-loop, without any piping, like this:

while true
do
        bob=joe
        echo "  $bob inside the while"
        break
done
echo "  $bob outside the while"

And we can see, easily, that the value of the "bob" variable stays the same, even after the while loop breaks, for all 3 of our test shells. If the while loop, alone, was the issue, bob shouldn't be defined when the while loop breaks:

host # zsh ./test.sh
  joe inside the while
  joe outside the while
host # ksh ./test.sh
  joe inside the while
  joe outside the while
host # bash ./test.sh
  joe inside the while
  joe outside the while

If we change this scriptlet slightly to make it "pipe" an echo to the while-loop, the behaviour changes dramatically:

echo a|while read a
do
        bob=joe
        echo "  $bob inside the while"
        break
done
echo "  $bob outside the while"

Now, if we use zsh, the value assigned to the "bob" variable inside our while loop (which has been created on the other side of the pipe) actually maintains it state when coming out of the loop, like this:

host # zsh ./test.sh
  joe inside the while
  joe outside the while

On most other shells, because of variable scope issues with the pipe, an empty value of the "bob" variable is printed after they break out of the while loop, even though it does get correctly defined within the while loop. This is because (and here's where the technicality, and subtle differences between myriad shells, usually becomes a hotbed of raging debate ;) after the pipe, the read command (as opposed to the while loop) runs in a subshell, like so:

host # bash ./test.sh
  joe inside the while
   outside the while
host # ksh ./test.sh
  joe inside the while
   outside the while

Notice, again, that the "echo $bob outside the while" statement in these two executions prints an empty variable when the value bob is declared outside the while loop, even though it is set within the while loop.

For most shells, this is easy to get around in one aspect. The main problem stems from the fact that the value is being piped to the while loop, and not a direct fault of the while loop itself. Therefore, a fix like the following should work, and does. Unfortunately, with the command-pipe (such as an echo statement), you won't be able to use a while-loop in many cases, and would have to substitute a for-loop, like so (In most shells, redirecting at the end of a while loop with << will either result in an error or clip the script at that line):

for x in 1
do
        bob=joe
        echo "  $bob inside the while"
done
echo "  $bob outside the while"

host # zsh ./test.sh
  joe inside the while
  joe outside the while
host # ksh ./test.sh
  joe inside the while
  joe outside the while
host # bash ./test.sh
  joe inside the while
  joe outside the while

This gets worse (usually hangs) if you try to get around the pipe by doing some inline subshelling with backticks, like:

while read `echo 1`

However, the following solution (awkward though it may be) does actually do the trick (substitute any other fancy i/o redirection you want, as long as you "avoid the pipe"):

exec 7<>/tmp/bob
echo -n "a" >&7
while read -r line <&7
do
        bob=joe
        echo "  $bob inside the while"
done
echo "  $bob outside the while"
exec 7<&-
exec 7>&-
rm /tmp/bob

host # zsh ./test.sh
  joe inside the while
  joe outside the while
host # ksh ./test.sh
  joe inside the while
  joe outside the while
host # bash ./test.sh
  joe inside the while
  joe outside the while

For more examples of input/output redirection, check out our older post on bash networking using file descriptors.

Now, when it comes to reading in files, the case is a bit easier to remedy. If you're in the habit of doing:

cat SOMEFILE |while read x
...

You'll run into the same scoping problem. This is also easily fixed by using i/o redirection, which would change our script to this:

while read x
do
        bob=joe
        echo "  $bob inside the while"
done < SOMEFILE
echo "  $bob outside the while"

Assuming the file SOMEFILE had one line of content, you'd get the same results as we got above with the for loop.

And that's about all there is to that (minus the highly-probably ensuing arguments ;). There are, I'm sure, a couple more ways to do this, but using the methods that fixed the "problem" of variable scope in bash and ksh is probably better practice, since zsh (and shell's that share its distinction in this case) is a rare exception to the rule (even though zsh may very well be doing things the "proper" way) and the bash/ksh fix works in zsh, while the opposite is not true.

At long last, good evening :)

, Mike

---

Thanks for this comment from Douglas Huff, which helps to clarify the underbelly of the process:

A friend of mine pointed me to this article and the
previous one in the series that you wrote [on variable scoping]...

I had two comments on these articles but you seem to have
comments disabled, so I figured I'd email them to you.

First, calling it a "scoping" issue is a bit misleading.
While technically true, understanding the underlying
reasons why this doesn't work as "expected" is key to
understanding how you can work around it in POSIX sh or in
ksh without the zsh/bash syntatical sugar for doing so.

What's going on is that a process cannot modify the
environment of it's parent.

When you do:

something | while read blah; do blah; done

What the shell is doing is first executing a subshell
(separate process) that runs the while with stdin
redirected to read from the unnamed pipe. Then in another
subshell it runs "something" with standard out redirected
to the unnamed pipe.

Knowing this it's quite easy to replicate the behaviour
from bash 2/3 and zsh in POSIX sh and ksh with a bit of
understanding of the underlying mechanics. The trick is to
keep the while inside of the original process (since it is
run by the interpretter and does not require a separate
process) and execute the other command in a subshell.
Which is exactly what the syntactical sugar does for you
behind the scenes in bash2&3/zsh.

Getting Error Values From The Middle Of A Pipe Chain In Bash

This is something very interesting I found out not too long ago, while hashing out some work with a colleague. As most administrators (or users) who do a fair amount of shell scripting know, the error status or return code (Generally referred to as "errno" in all the man pages) of a process is a fairly common method to use in determining the process flow of a script.

The one thing about the value of "errno" (or, literally, the variable "$?" in most shells) is that it's erased with each consecutive process that gets run. So if you were to run a series of command lines that echoed the return value of the grep command, the following example would be accurate (assuming the string "bob" can't be found in /home/myfile):

host # grep bob /home/myfile >/dev/null 2>&1
host # echo $?
host # 1

while this one would give you misleading information:

host # grep bob /home/myfile >/dev/null 2>&1
host # touch /home/myfile
host # echo $?
host # 0

So, on the first set of command lines, you're actually getting the return code of 1 from grep (indicating that it can't find the string "bob" in /home/myfile), while the second one gets you the return code of 0 from the touch command. "errno" always contains the return value of the last-executed command.

Which brings us around to the topic indicated in the title of this post (I promise to tie in the whole introduction about "errno" at the end; it wasn't a complete waste of your time ;). While it's easy enough to trap "errno" in any series of disconnected commands (for instance, in the second example above, if we'd echoed $? before running touch, it would have given us the correct output), I had always thought it was impossible to grab the correct value from a command in the middle of a pipe chain, like this:

host # grep bob /home/myfile 2>&1|Grep joe|xargs echo
host # echo $?
host # 0

You'll note that I purposefully capitalized the G in grep so that it would return an error code that didn't indicate success, yet - since this is a chain of commands all connected by pipes - "errno" returns the value of the xargs command, since it was the last one executed. Which means I've spent a lot of time jumping through hoops to "reword" any pipe chain so that I could extract the information I needed.

Now (and I'm almost positive this wasn't the case a few years back) the bash shell has actually taken on this predicament and come up with a nice workable solution for it(I'm waiting for it to pop up in sh and ksh, since they've been burned into my psyche over the last decade or so). In bash, if you run a series of piped-together commands, you can actually extract the value of "errno" from any command in the chain by using the shell built-in PIPESTATUS array, like so:

host # grep bob /home/myfile 2>&1|Grep joe|xargs echo
host # echo ${PIPESTATUS[@]}
host # 1 127 0

How nice is that? :) Now you can easily tell the return value of every process in a pipe-chain. The initial grep returns 1 because the string "bob" isn't in /home/myfile, the misspelled Grep returns 127 because the command can't be found and the final xargs returns 0. That solves a lot of problems and can potentially save you lines upon lines of convoluted code.

The one thing about it that can be frustrating is that it behaves in much the same way as "errno" (See, I told you I'd bring it back around ;). If you don't capture the output immediately (or dish it off into another variable), the array will zero out and contain no values as soon as you enter your next command, like so:

host # grep bob /home/myfile 2>&1|Grep joe|xargs echo
host # touch /home/myfile
host # echo ${PIPESTATUS[@]}
host # 0

At this point, after we've executed the touch command, the PIPESTATUS array has been cleared out, just like "errno" gets written over, even though we haven't executed another pipe chain. Its behaviour is basically identical. Below, we show that, once the array has been written over, its size gets reduced to 1 ( The single return value of the last executed command) and we further prove that the array really has been clipped down to one variable by attempting to print the first and second values; the second of which doesn't exist. Continued from above:

host # echo ${#PIPESTATUS[@]} <--- Here we ask bash for the size of the PIPESTATUS array
host # 1
host # echo ${PIPESTATUS[0]} <--- Here we check the first variable in the PIPESTATUS array
host # 0
host # echo ${PIPESTATUS[1]} <--- Here we check the second variable, which now doesn't exist
host #

This is easy enough to get around, however, since - just like "errno" - you can assign that array to another array before you execute another command, like so:

host # grep bob /home/myfile 2>&1|Grep joe|xargs echo
host # new_array=${PIPESTATUS[@]}
host # touch /home/myfile
host # echo ${PIPESTATUS[@]}
host # 0
host # echo ${new_array[@]}
host # 1 127 0

If you knew this already, I envy you the convenience you continue to enjoy. For the rest of us; a pleasant surprise :)

Best Wishes,

, Mike

linux unix internet technology