Hey there,
Today we're going to continue, in a sense, where we left off on yesterday's post regarding using mkfifo for user monitoring. In this post, we're going to look at that a bit more specifically, and also touch on some other ways you can do the same thing (if not better). Everything we'll be going over today should be either freeware or come with your OS (or both :)
Again, please check our older posts on executing new file descriptors and shell redirection if you need that part of the process fleshed-out more. We're going to take it as a given that you basically get how that works (This is the biggest hazard of writing a tech blog for everyone. Sometimes it's hard to explain the basics and not insult the experienced user and/or explain a more advanced concept while not seeming to talk down to the new user. Whatever I write here; no offense meant to any reader. Except a few of you. You know who you are ;)
Now, we'll get to the meat and go after logging and/or monitoring user sessions. You have various options, depending on what you want to do. Personally, I think it's cool to be able to watch someone's process in real-time, but sometimes (especially if you want to make sure you're always keeping what you're seeing) its better to dump output to files and just cat or tail those in almost-real-time. If being a few usec's behind the user you're monitoring has the power to ruin your entire day, please stop reading now. This will end badly no matter what ;)
1. Using mkfifo and enhanced profile security: This can be fun, and it has the potential for being miserable, depending on the OS you're running. For instance, Solaris' standard "script" command only has one flag (-a, to append to a previous script file) and is notorious for rarely flushing the buffer (so you'll get 50 pages of user messages at once and then nothing for 20 minutes) and, often times, will clip the information sent through a FIFO if the session ends before it flushes for the first time, leaving you with incomplete output!
The first thing you'll want to do is modify your users' .profile (or appropriate shell settings file) to be secured as if the account was su-only. If your user can modify the file after you've made your changes, you can bet they'll remove the script line as soon as possible. If you want to meet your users half-way, insist on a 750 root:root owned .profile and have the last line source in another file that they can use to make their own customizations, as deemed appropriate or possible. Here's what I'd put in somebody's .profile to create a FIFO and start script running when they login and pipe to that FIFO (Note that, although script normally makes it so you have to exit twice to log out - because it opens up in a subshell - using it in the .profile makes it so that when the user exits once, the connection is terminated!) :
tempvar=$$
user="`logname`.$tempvar"
myfifo=/tmp/$user
/usr/bin/rm -rf $myfifo
/usr/bin/bash -isc "/usr/bin/script $myfifo"
Then, as soon as the user logs in, you can have an automated process begin reading from the named pipe (/tmp/username.112327, for instance) and either begin writing it to the screen or dumping it to a file, or both. If you want to read the output to the screen you're on, just do:
host # cat /tmp/username.112327
or
host # tail -f /tmp/username.112327
to dump off the output into a log, just do:
host # cat /tmp/username.112327 >/tmp/log
to do both, just do something like:
host # cat /tmp/username.112327 2>&1|tee /var/logdir/log
For my money, like I said, you're better off just dumping to log. You won't have to worry about catching logins with another process and you can always read the log later (or watch it in real time). If you decide that's the best option, change that section of your users' .profile to read like this instead:
tempvar=$$
user="`logname`.$tempvar"
myfifo=/tmp/$user
/usr/bin/rm -rf $myfifo
/usr/bin/bash -isc "/usr/bin/script $myfifo"
cat $myfifo >/var/logdir/$user.log 2>&1 &
2. Using screen and enhanced config file security: Gnu Screen is another alternative for root (or users of similar privilege) to keep an eye on users. The rules for securing screens configuration files, are the same as those we went over regarding su-only accounts, just applied to different files. The theme is the same: Make the files usable, but not modifiable, by the user.
Screen is a bit easier to set up for ethical-hijacking. All you need to do is modify the users' "screenrc" file, make it so they can't modify it and add the lines:
multiuser on
acladd root
and, of course, set up their profiles to launch screen at login. With these conditions in place, all you need to do, as root, to watch a user session is to type the following at the command line:
host # screen -x user
If you're not sure who's on, or one user has multiple screens open:
host # screen -x
with no specific argument should let you know :) You may then, as with script, decide that you're better off dumping to a logfile and not constantly monitoring (or using resources to monitor). With screen, you don't need to jump through any gigantic hoops (making FIFO's etc, unless you want to ;) If your original entry in the users' .profile was:
screen
all you would have to do to change it to log the output to a file, as well as the screen, would be to change that to:
screen -L PickANameForALog <-- or just a number if you want to accept the default screenfile.NUMBER output
3. Wrapping up - Big differences between "script" implementations:
I just thought I'd end on this note, since it's crazy, and I can't tell you for sure what to expect from "script" between distro's. The implementation of this command is wildly different. For instance, as I mentioned above, Solaris' "script" only has a "-a" option to append to an existing script output file. BSD's "script" command (and most Linux distro's, it seems) have a lot of other useful options that you can use to make your life (and doing this sort of thing) easier. Here are the additional flags that I know of:
-c COMMAND <-- this will run a command, which would make it so you could run bash from script directly rather than having to re-invoke bash and call script from there.
-f &;lt-- This output will flush the buffer after every read/write. So, unlike the Solaris "script," you can actually get your results back in real-time. Very helpful if you need to watch a user's activity.
-q <-- Be "extremely" verbose... Just kidding ;) This tells script to not announce itself starting and stopping. This might prevent certain users from fighting you over this from day-one!
-t <-- This tells script to output timing data. It's run on an existing scriptfile and spits its output to STDERR. So, if you wanted to capture it in a file, something like this would do:
host # script -t 2>myNewTimeLog MyOriginalOutput
and then you can use that file to read your original "script" output back at relatively the same speed at which it was captured, using the "scriptreplay" command, like this:
host # scriptreplay MyNewTimeLog MyOriginalOutput
If you've got an hour's worth of output and don't want to go through the hassle of removing all the control characters, this is a great help. If you just cat the original scriptfile, it'll run through itself in under a minute and may cause seizures ;)
Now you should be all set to start monitoring users' activities and output. If your company wants to purchase a commercial product instead, you can always put that on your resume!
Hope this has been a helpful, if not enjoyable, diversion :)
Cheers,
, Mike
Friday, July 11, 2008
Using Screen, Script, Mkfifo And Redirection To Watch Or Log User Sessions
Thursday, July 10, 2008
Using Mkfifo For Monitoring And Enhanced User Activity Logging
Hey there,
For today's post, if you're unfamiliar with how to execute new file descriptors and work with shell redirection, to a comfortable degree, check out our older posts on those subjects (cleverly anchor-linked into this very paragraph). They explain more basic stuff regarding those topics better than this post will and, if you want to, they'll be there for you to read for as long as I can remain within the hosting company's good graces.
One basic component of any decent security setup should always be logging. At the very least, if you're running a server that needs to be on, and servicing requests, 5 days a week (or 24x7, etc), you're probably logging sar output or iostat, vmstat, etc. You're taking some sort of metrics so that you'll be able to troubleshoot any problems that may arise in the future when your needs outgrow your capacity, or somebody just makes a mistake and runs an infinite recursive fork-and-exec loop as root.
Another thing that a lot of companies like to do is to keep tabs on their employees. Especially as it pertains to their activities on production Unix and Linux servers. This may or may not be for a bad reason. Some companies (depending on their nature) are required to monitor user activity to maintain standards compliance, while others may just do it so that if user XYZ makes a mistake and crashes the machine, they can figure out that it was he/she who did it and make sure that it doesn't happen again (hopefully, by explaining why what happened happened and considering it a lesson learned. It depends heavily on the workplace and the situation, of course)
One way to keep an eye on your users is to run kernel process accounting (pacct, which is available for Linux and Unix) and kernel auditing. Both of these probably provide the most complete picture in the event of a catastrophe, but they come at a cost because they can exact a heavy toll on your systems even when everything is fine. They can also make it so that things aren't fine any more, or expedite the non-fineness (???) of a situation if anything "bad" happens. Imagine how much harder your kernel and/or memory is being interrogated when there's a problem with it. Infinite recursion comes to mind again. The ghosts of those Fibonacci numbers just never go away ;)
In my practice, unless it's specifically requested, I prefer to log via log file. I know it seems primitive, and in a sense it is, but it's very "inexpensive" in terms of performance cost. One tool you can use to help with system auditing is called "mkfifo." It, of itself, doesn't really do anything for you, but it can be a great facilitator (especially if you use your imagination and have the latitude to experiment).
mkfifo is a program that creates a simple named pipe. You can equate it to the "|" that you use in your every day command lines, like:
host # cat FILE|grep word
although it's slightly differently. When a named pipe is created, via mkfifo (or however else you can do it), it creates a pipe "file" that remains in place until it is removed (or, in some cases, until your machine reboots, if you forget to remove it). You can create your own named pipe with mkfifo simply, as it takes very few arguments, like so:host # mkfifo -m 777 /tmp/corncob
host # ls -l /tmp/corncob
prw-r----- 1 user group 0 Jul 9 15:11 /tmp/corncob
That's all it takes to create the named pipe /tmp/corncob. The -m flag, which is used to set the permissions, is not necessary. Generally, if you don't include it, the default permission set for a new named pipe is whatever the default for your system would be. As another side note, you can also pass the -m flag and set alpha permissions, rather than octal, like:
host # mkfifo -m a=rwx /tmp/corncob
to create the exact same thing. You can delete the named pipe just like you delete a file. rm, and it's gone.
One thing you should note about named pipes is that they generally (so far as I've seen) are only able to fully pass one stream of input/output through themselves at a time. That is to say, if you have one process sending input to the named pipe and two process reading from it, only one of the reading processes will receive output. It should be noted, also, that, if such a situation were to exist, once the original process that was receiving output exits, the other process would begin receiving output from the named pipe (if it was still attempting to read from it). Was that a really long sentence or am I just typing fast? ;)
An example of what I mean below:
host-term1 # while :;do echo a b c d e >/tmp/corncob;sleep 15;done
host-term2 # tail -f /tmp/corncob
a b c d e
a b c d e
a b c d e
a b c d e
a b c d e
a b c d e
a b c d e
a b c d e
host-term3 # tail -f /tmp/corncob
host-term2 # ^C
host-term3 #
^C
a b c d e
a b c d e
a b c d e
a b c d e
Now, if you combine this freely available named pipe (you should probably secure it with permissions so that only users you want to be able to actually modify it have permission to remove it. Everyone you want to manage must be able to write to it and execute it.) with any number of output capturing mechanisms, you've got yourself a logger. Of course, you'll need to take care and work out how you want to parse the output coming from the named pipe because, even though it can only be read from by one process at a time, it can be written to by many and (if you have 15 people logged in, all with duplicate STDIN and STDOUT getting pushed through the same named pipe) that could get confusing for you very quickly.
My first recommendation would be, of course, to initiate an individual named pipe per user login process. This is so lightweight that generating 60 named pipes isn't going to cost you much more in overhead than generating six.
A simple thing to do is to use script to generate output and redirect it to the named pipe, like this:
host # ksh -ic "script /tmp/corncob"
using the name of the FIFO as the name of the output file for script. This is generally a bit klunky (script dumps the output in chunks and sometimes loses bits), but it does work relatively well. All you need to do is modify that slightly and put that in your users' .profile files and they'll be writing their entire sessions to /tmp/corncob (or, like I mentioned, each to a different named pipe file: /tmp/username perhaps?)
I think you can begin to see the possibilities here (and, probably some of the pitfalls - for instance, you can't use "tee" since it won't allow interactivity), but we'll continue on this topic tomorrow and look at some specific ways in which you can use named pipes, in conjunction with STDOUT and STDERR redirection, to do some serious security logging without serious stress :)
Cheers,
, Mike
Posted by
Mike Golvach
at
12:42 AM
descriptor, file, kernel audit, linux, logging, mkfifo, named, pacct, pipe, redirection, shell, unix

