Showing posts with label ps. Show all posts
Showing posts with label ps. Show all posts

Monday, March 30, 2009

Idle Process Time On Linux And Unix: How To Find It Again

Hey There,

In our final installment on "finding a process's idle time on Linux or Unix," (last touched upon in our post on echo debugging) we looked at whole lot of ways one could go wrong trying to find the idle time of a process on Linux or Unix. This post is a little more upbeat ;)

All of the previous issues with "who -T" have been worked out. Basically, this means that I've gone over it every which way and could find no good reason to use it, as opposed to "w." Of course, in our particular case, we are looking, specifically, for a single process's idle time (as opposed to a user process's idle time; reported by "who -T"). And, although it's a little bit of a pain (initially), short of programming in C (accessing the pstatus struct on Solaris, to be exact - the name and location may vary from distro to distro of proprietary, or free, Unix and/or Linux), linking the pty information from ps with the "idle time" information from w, seems to be the best way to get this information. So far, it's the most efficient way I could find using simple bash scripting.

Attached to today's post is the final "blog" version of this script. It comes with a few notes (possibly of caution) and may need to be modified for your system/OS (There's the first one ;)

The script runs very simply, and you only need to supply it with a PID. You can, optionally, supply a username as a second argument:

host # ./rip 17787

if you just run it with no arguments, you'll get a usage screen, which may or may not help ;)

host # ./rip
Usage: ./rip PID [user]
User defaults to the value
of $LOGNAME if not specified


Please see our previous post on echo debugging this script for more detailed sample output.

I hope you find some good use for this script, and, without further ado, the oft-dreaded notations of explanation ;)

1. This script has been rewritten to be self-contained. Please see the bottom line for any substitute command you may want to use. Actually, making this command a variable might be a good idea. Just call me "Lazy" ;)

2. You can remove the explicit PATH definition if you like. I put it in there specifically to make sure that the "which ps" variable assignment didn't accidentally grab /usr/ucb/ps on Solaris

3. You can comment out the SIGNAL variable as well, since plain old kill is a sig TERM or 15. The only real reason to set this would be if you wanted to always run kill with a different signal (like SIGKILL,, or 9, for example)

4. I changed the minimum idle time to 30 minutes from 45 (in the previous revisions)

5. All variables appearing in this work are fictitious. Any resemblance to real variables, living or dead, is purely coincidental ;)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# rip - Kill any processes that we know have been idle for more than 30 minutes
#
# 2009 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

PATH="/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin"
procowner="rftprocowner"
prog="\./rft"
sed=`which sed`
awk=`which awk`
ps=`which ps`
grep=`which grep`
kill=`which kill`
signal="-15"

while read a b c d
do
pid=$b
pid_not_var=$(echo $pid | $grep [A-z])

if [[ ! -z $pid_not_var ]]
then
echo "pid $1 contains non-numeric characters!"
continue
fi

pid="$b"
pid_pty="$c"

if [[ -z "$pid_pty" ]]
then
echo "pid $pid is either non-existent, not owned by \"$procowner\" or not attached to a p/tty!"
continue
elif [[ "$pid_pty" = "?" || "$pid_pty" = "console" ]]
then
echo "pid $pid is not attached to a pty!" # kill OR LEAVE IT?
else
pty_num=$(echo "$pid_pty"|$sed 's/^[^\/]*\///')
fi

proc_time=$(w -sh $procowner|grep $pty_num|grep -v grep|$awk '{if ( $2 == '"$pty_num"' && NF == 4 ) print $3;else if ( $2 == '"$pty_num"' && NF == 3) print "0"}')

proc_is_num=$(echo $proc_time | $grep [A-z])
if [[ ! -z $proc_is_num ]]
then
unset proc_time
fi

ext_proc_time=$(echo $proc_is_num | $grep [A-z])

if [[ ! -z "$ext_proc_time" && -z "$proc_time" ]]
then
echo "killing $pid - $d Up Over 24 Hours: $ext_proc_time $proc_time"
### $kill $signal $pid
elif [[ "$proc_time" = "0" ]]
then
:
else
proc_idle_time=$(echo $proc_time|$grep -v "[:]")
if [[ -z $proc_idle_time || $proc_idle_time -gt 30 ]]
then
echo "killing $pid - $d Up More Than 30 Minutes: $ext_proc_time $proc_time $proc_idle_time"
### $kill $signal $pid
fi
fi
done <<< "`$ps -fu $procowner -o procowner,pid,tty,comm|$grep "$prog"|$grep -v grep`"


, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, March 25, 2009

Finding A Process's Idle Time On Linux And Unix

Hey There,

Hopefully yesterday's rant on the simplicity of complexity wasn't too much of a bitter pill. If it was, here's hoping you didn't swallow it ;)

Today, I finally found some time to make a little headway on this project (which should be a lot simpler than it is). Basically, what I'm looking to do is create a way to track specific process's idle times at any given point in time on any given Linux or Unix system. As I mentioned yesterday, there are c structures in Solaris' /proc/PID/status C data files (for one example), but that's just another thing that ended up frustrating me more. As I noted, parts of the OS that are included, should be available for use. The structure is used by the OS, in some shape or fashion, to determine idle times (as we'll see below), but no specific "tool" exists to do what i wanted. Of course, this is limited "to my knowledge." If anyone out there knows of a standard program or command that's managed to elude me, please feel free to email me and tell me all about. I promise to not get offended if you feel the need to belittle me for not having the common sense to look for it where it was at in the first place ;)

Attached to today's post is a rough-draft bash script that attempts to grab a process's idle time. It won't work in all instances, although I've tried to capture as many of those instances as possible. The one big gotcha in this whole mess is that you can't take the output of ps and directly retrieve the idle time for a process from the listing, even if you do your own formatting (I wrote this on Solaris 10 and looked at SUSE Linux 9, but found no love :) Instead, I found that I needed to run ps, extract the pty associated with the process from that (if it existed - which is an exception the script catches) and then use either who or w to retrieve the idle time associate with the pty.

See what I mean? Shouldn't it be a little bit less of a hassle than that?

Okay. I'll admit, if it was, I wouldn't be having half the fun I'm having now trying to script it all out for myself ;) So far, what I've put together works fairly well, although I'm not 100% certain that it's bullet-proof so I would recommend that you leave the "business end" of the code commented out (The stuff that performs unforgivable actions, like killing ;). I have a hard time reproducing it, but I can swear that this code will (every once in a good while) determine that a process that hasn't been idle at all (which removes a column from the "w -s" output) has been idle too long. I'm still working on that part and welcome any suggestions regarding the script, how to make it better, why I'm doing everything the wrong (and/or hard) way when I don't need to and any other constructive criticism :)

The script runs very simply, and you only need to supply it with a PID. You can, optionally supply a username as a second argument:

host # ./rip 17787

if you just run it with no arguments, you'll get a usage screen, which may or may not help ;)

host # ./rip
Usage: ./rip PID [user]
User defaults to the value
of $LOGNAME if not specified


and the following is a sample of the output you might get on a specific run. Here, I've written a command line while loop from a pipe to barbarically hammer out multiple instances at a time ;)

host # time ps -ef|grep "[b]ash"|awk '{print $2}'|while read x;do ./rip $x;done
PID 2664 is not attached to a p/tty!
-----------------------------------
PID 10700 is either non-existent, not owned by "root" or not attached to a p/tty!
-----------------------------------
PID 10855 is OK - Not Idle At All - Remove this message!
-----------------------------------
PID 23217 is OK - Not Idle At All - Remove this message!
-----------------------------------
PID 14730 is either non-existent, not owned by "root" or not attached to a p/tty!
-----------------------------------


Here's another example. This time you'll see what you get if you try to run the script specifying a user other than the user that owns the processes or, in this case, a completely bogus user. This "test" in the script really isn't necessary and I only included it as feeble attempt at damage control. Feel free to remove it if you like:

host # time ps -ef|grep "[b]ash"|awk '{print $2}'|while read x;do ./rip $x joeUser;done
ps: unknown user joeUser
PID 2664 is either non-existent, not owned by "joeUser" or not attached to a p/tty!
-----------------------------------
ps: unknown user joeUser
PID 23633 is either non-existent, not owned by "joeUser" or not attached to a p/tty!
-----------------------------------
ps: unknown user joeUser
PID 10700 is either non-existent, not owned by "joeUser" or not attached to a p/tty!
-----------------------------------
ps: unknown user joeUser
PID 10855 is either non-existent, not owned by "joeUser" or not attached to a p/tty!
-----------------------------------
ps: unknown user joeUser
PID 14730 is either non-existent, not owned by "joeUser" or not attached to a p/tty!
-----------------------------------


I hope you find some good use for this script!

NOTE: Please keep in mind the caveat noted above regarding the sometimes-false-positive I believe this script returns under certain circumstances when it decides a non-idle process (with nothing displayed in the idle column from "w -s" output) has been idle too long! It may never happen again and I may have been seeing spots. Just want to keep you in a "safe" mindset, just in case I'm not completely insane ;)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# rip - Kill any processes that we know have been idle for more than 45 minutes
#
# 2009 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if [ $# -lt 1 ]
then
echo "Usage: $0 PID [user]"
echo "User defaults to the value"
echo "of \$LOGNAME if not specified"
exit 1
fi

PID=$1
ISITAPID=$(echo $PID | grep [A-z])

if [ ! -z $ISITAPID ]
then
echo "PID $1 contains non-numeric characters!"
echo "-----------------------------------"
exit 2
fi

PID="$1"
USER=${2:-$LOGNAME}

PIDTTY=$(/usr/bin/ps -fu $USER -o pid,tty |/usr/bin/grep -w $PID|/usr/bin/grep -v grep)

#echo DEBUG::::: PIDTTY $PIDTTY

if [ -z "$PIDTTY" ]
then
echo "PID $PID is either non-existent, not owned by \"$USER\" or not attached to a p/tty!"
echo "-----------------------------------"
exit 3
else
TTYNUMBER=$(echo "$PIDTTY"|/usr/bin/sed '/TT/d'|/usr/bin/awk -F"/" '{print $2}')
fi

if [ -z "$TTYNUMBER" ]
then
echo "PID $PID is not attached to a p/tty!"
echo "-----------------------------------"
exit 4
fi

#echo DEBUG::::: W $(w -s|/usr/bin/sed 1d|/usr/bin//awk '{if ( $2 == '"$TTYNUMBER"' ) print $0}')

TIME=$(w -s|/usr/bin/sed 1d|/usr/bin/awk '{if ( $2 == '"$TTYNUMBER"' ) print $3}')

ISITANUMBER=$(echo $TIME | grep [A-z])
if [ ! -z $ISITANUMBER ]
then
unset TIME
fi

LONGTIME=$(echo $TIME | grep [A-z])

#echo DEBUG::::: LONGTIME $LONGTIME TIME $TIME

if [ -z "$LONGTIME" -a -z "$TIME" ]
then
echo "PID $PID is OK - Not Idle At All $TIME - Remove this message!"
elif [ ! -z $LONGTIME ]
then
echo "PID $PID is ancient - Idle for $TIME... Killing $PID"
# DO_WHAT_YOU_HAVE_TO_DO_TO_THE_PID_HERE
else
TIMEIDLE=$(echo $TIME|grep -v "[:]")
# echo DEBUG::::: TIME $TIME
if [ -z $TIMEIDLE ]
then
echo "PID $PID has been idle way too long - $LONGTIME $TIME so far... Killing $PID"
# DO_WHAT_YOU_HAVE_TO_DO_TO_THE_PID_HERE
elif [ $TIMEIDLE -gt 45 ]
then
echo "PID $PID has been idle too long - $TIMEIDLE minutes so far... Killing $PID"
# DO_WHAT_YOU_HAVE_TO_DO_TO_THE_PID_HERE
else
echo "PID $PID is OK - Only idle for $TIME minute(s) - Remove this message!"
fi
fi
echo "-----------------------------------"


, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, November 5, 2008

A Few Ways To Gauge Possible Memory Bottlenecks In SUSE Linux

Hey there,

Today we’re going to take a look at a few (well, maybe more than a few) ways to check your Linux box to find out if the reason it’s beginning to perform poorly is related to memory or memory-management issues. For today’s examples, in particular, we’ll be using SUSE Linux 9.x, but most of these examples translate just as easily to RedHat, Ubuntu and other widely-used distro’s, with some minor modifications.

First things second, I’d like to apologize for the lack of animation in my “voice” in this post. As you probably well know (Even if you don’t live in the USA), we’re (hopefully) going to find out who our next President is by this evening (I write these a day in advance) and I’m somewhat distracted, even though I know I shouldn't start caring until its late enough in the game for any of the numbers to make a legitimate difference. My wife is hanging on the 1% from here and 3% from there ;) I won’t reveal whom I voted for, since this blog doesn’t take any particular political stance. I’ll just let it be said that I’m really hoping my candidate comes in. It’s going to be a farce if the other contender somehow manages to steal this thing….

And, secondly, we’ll walk through the many different (and easily accessible) ways you can check up on your Linux system’s memory usage to make sure that all is well. Note that for some commands, you need root privilege to get information of any value. Most of these commands don’t require that level of access, but (as a rule) if you need to interface with the kernel (or access its symbol/memory tables, etc) you’ll probably just end up with a big old raspberry when you try to run a few of these commands. Not to worry, though: You’ll be able to get more than enough information from this mass of commands to make a reasonably accurate estimation, no matter what level of access you have.

And here’s a quick look at assessing your system’s (possible) memory problem, quickly:

1. top.

This command should be in everyone’s arsenal. If your Linux distro doesn’t include it (as part of a standard pkg) you can always build it from source. It’s been my experience that this tool comes standard with everything nowadays. Pay special attention to the physical and virtual memory sections. top should give you a fairly accurate count of free vs. used memory for both types.

2. ps –aux |grep AprocessYouSuspect

This will provide you with similar information. Pay special attention to the %MEM, VSZ and RSS columns.

3. vmstat 2

This will provide you with memory statistics at 2 second intervals (also the default refresh rate for top).

Watch the following columns (note that the “inact” and “active” columns only show up when you invoke vmstat with the “-a” flag:

swpd: To check on the amount of virtual memory in use
free: To check on the amount of free memory
buff: To check on the amount of memory being set aside for buffering
cache: To check on the amount of memory set aside for caching.
inact: To check on the amount of inactive memory
active: To check on the amount of active memory
si: To check on the amount of memory swapped in from disk
so: To check on the amount of memory swapped out to disk. <-- For more on the difference between swapping, paging, etc, check out our older posts on swapping and Paging on Linux and Unix and its inevitable follow-up

4. ps -o vsz,rss,tsiz,dsiz,majflt,minflt,pmem,cmd 9999

This will format your ps output to spit out virtually all the memory information ps can get you for a specified process ID (In this case: 9999)

5. cat /proc/9999/status

This will provide you with a lot more detailed information on the 9999 process.

6. swapon –s

This command will list out all the system swap partitions

7. free

This will show you the amount of used and free memory in terms of straight-up memory, buffers and cache

8. cat /proc/meminfo

This will show you more detailed information on the what the system thinks its memory is doing and/or how its being used.

8. sar –r

This will show you, from another perspective, memory usage defined in terms of memory, buffers and cache

9. ipcs –u

This command will list out all the shared memory status in segments, semaphores and queues.

10. ipcs –p

This will list out all shared memory used by process ID and owner.

11. Finally, check the /etc/sysctl.conf (for shared memory values,etc) and /etc/sysconfig/kernel (for tmpfs/shmfs filesystem size figures).

And, hopefully, that should be enough to get you started and/or, at least, give you some idea if your bottleneck has anything to do with a shortage (or overuse) of memory on your system.

Here’s hoping your vote counts ;)

, Mike




Olivier Berger had this excellent suggestion for a GUI tool to add!


Hi.

I'd like to suggest that you mention gmemusage, which I find a really
useful tool to visualize what's eating memory if a X display is
available.

Hope this helps.



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Friday, August 8, 2008

Using Grep To Streamline Your Shell And Command Line Scripting

Good [insert part of day here] ;)

For today's post, we're going to look at using grep to reduce script-bloat. Of course, we won't be covering every possible way this can be done, because I have yet to finish perfecting the art of using editor tricks to reinforce bad habits. The vi editor's yank (yy), put (p), double-quote (") buffers and dot (.) action repetition simplifiers make it very easy for me to write long and unwieldy code when I don't feel like thinking about how I could fit 40 lines into 3 ;) The downside is that it's a royal pain to go back and modify. Not only do I have to go back and forth 3 screens to check code above and below my "text dump," but I also have to fix the same problem 15 times if I made it in one instance of my code and then just mass-copied and slightly modified it. Of course, you can always use the substitute (s/g), etc, commands to make sweeping edits, but you get my drift. It's generally not more-or-less efficient to write unrolled loops (since that's what your shell command interpreter is going to do to your tight loops when it executes them anyway), but it can be a huge and confusing time-waster for all the humans who have to deal with it later; even for the original author.

In any event, I have managed to digress, yet again ;)

Today, we're going to look at a few ways we can use grep to clean up our shell scripting and make it look (and, most of the time, actually "be") more efficient. The solutions presented in this post can be one-upped by any number of other standard Unix/Linux commands, but (for the sake of this post) we're going to assume that "grep" is the be-all-end-all of input and output processing. Suspend your disbelief, for as long as it takes to read this post, and assume that the world is flat, grep's it and that's that ;)

On to our examples!

1. Generating line counts: This is one I see a lot (and have done a'plenty). Most Unix/Linux admins and users know of the wc command and get used to using it to determine the number of lines in a file like this:

host # wc -l FILE
17 FILE


which is perfectly acceptable. Sometimes though, since the command comes so easily to mind, it gets used inappropriately or, at least, in places where using grep, by itself, would make more sense. For instance, consider this relatively normal looking, and perfectly serviceable, line of code:

host # cat FILE|grep done|wc -l
2


And, oh yes, we're ignoring the indents that wc puts before the output for now :) This count of "how many times the word 'done' appears in FILE" can be pared down very easily, like so:

host # cat FILE|grep -c done
2


and, even better, we can use grep to get a basic "wc -l" line count on any file by using it like this:

host # grep -c . FILE
17


It's a slightly longer command than the "wc" version, but it does save you from having to process out the padded blank space if you want to assign your output to a variable, or use it "in context" within another command string as-is :)

2. Ensuring that your process checking is as accurate as possible: We actually looked at this back in October of 2007 in a post on how to keep grep out of your grep output, but the same basic principal applies here. For instance, if you've written a wrapper script that checks a process every 5 minutes and, at all times, this process must be running, you might make the core of your script (that part which you're hoping won't equal 0) something like this:

host # ps -ef|grep program|wc -l
2


but, you see the same problem you had in our first bullet, so you'd crop that back to:

host # ps -ef|grep -c program
2


But, in this case, we still have an issue, because only 1 instance of your "program" should be running and there appear to be two. Actually, if we peel back the veneer, there "is" only one instance running, but grep is ending up in the output (and, even worse, you can't depend on grep to "always" show up in its own output, which means our result may be 1 or 2 with everything being OK):

host # ps -ef|grep program
user1 2692 8747 0 13:33:52 pts/11 0:00 /bin/sh ./program
user1 3269 8747 0 13:37:05 pts/11 0:00 grep program


So, now we can use another trick to make this work correctly every time. A lot of times you may see code like:

host # ps -ef|grep program|grep -v grep|wc -l

but this is quite a bit longer, now, and we also can't use "grep -c" because it would make the following "grep -v grep" statement fail. And, since we can't count on "grep" always showing up in its own output, we couldn't predicate success or failure based on any specific number (in this case 1 or 2). Plus, if we did, this code would get even more lengthy and confusing.

Instead, we can use the shell's natural range operators "[]" to ensure that (if only 1 instance of "program" is supposed to be running) we'll always only get back the number "1" when things are going well:

host # ps -ef|grep -c "[p]rogram"
1


and that's it. The reason this works is that the shell processes the range operators (which, in our case contain a range of one letter) and pass the string "program" to grep. So, grep is scanning the ps output for "program," but the range operators still appear in the ps output (for the grep command). Therefore, it can never match its own process because ps is literally showing "[p]rogram" (with the left and right brackets ([]) being actual left and right brackets, rather than range operators which get interpreted).

Here's what the command output looks like in the working version above:

host # ps -ef|grep "[p]rogram"
user1 2692 8747 0 13:33:52 pts/11 0:00 /bin/sh ./program


and modified to look for "[g]rep" so we can see the grep and note the difference:

host # ps -ef|grep "[g]rep"
user1 5864 8747 0 13:47:16 pts/11 0:00 grep [g]rep
<-- Note that this match is matching "grep" and not "[g]rep" ;)

And that's it for today. There are actually quite a few more things you can do with grep, that get done a lot more efficiently as a result, but we'll take a look at those another day. As usual, it seems, in attempting to explain as clearly as possible, I have managed to crank out yet another tractate. It's times like these that I start to feel a bit long in the tooth. The older you get, the more outcomes you try to compensate for. That can make for some long posts. It can even make for some long endings to long posts ;)

Hopefully, it's been a relatively bearable reading :)

Cheers,

, Mike

Monday, June 9, 2008

Finding The Number Of Open File Descriptors Per Process On Linux And Unix

Hey There,

Today, we're going to take a look at a fairly simple process (no pun intended), but one that (perhaps) doesn't come up enough in our workaday environments that the answer comes to mind as obviously as it should. How does one find the number of open file descriptors being used by any given process?

The question is a bit of a trick, in and of itself, since some folks define "open file descriptors" as the number of files any given process has open at any given time. For our purposes, we'll be very strict, and make the (usually fairly large) distinction between "files open" and "open file descriptors."

Generally, the two easiest ways to find out how many "open files" a process has, at any given point in time, are to use the same utilities you'd use to find a process that's using a network port. On most Linux flavours, you can do this easily with lsof, and on most Unix flavours you can find it with a proc command, such as pfiles for Solaris.

This is where the difference in definitions makes a huge difference in outcome. Both pfiles and lsof report on information for "open files," rather than "open file descriptors," exclusively. So, if, for instance, we were running lsof on Linux against a simple shell process we might see output like this (all output dummied-up to a certain degree, to protect the innocent ;)

host # lsof -p 2034
CMD PID USER FD TYPE DEVICE SIZE NODE NAME
process 2034 user1 cwd DIR 3,5 4096 49430 /tmp/r (deleted)
process 2034 user1 rtd DIR 3,7 1024 2 /
process 2034 user1 txt REG 3,5 201840 49439 /tmp/r/process (deleted)
process 2034 user1 mem REG 3,7 340771 40255 /lib/ld-2.1.3.so
process 2034 user1 mem REG 3,7 4101836 40258 /lib/libc-2.1.3.so
process 2034 user1 0u CHR 136,9 29484 /dev/pts/9
process 2034 user1 1u CHR 136,9 29484 /dev/pts/9
process 2034 user1 2u CHR 136,9 29484 /dev/pts/9
process 2034 user1 4r CHR 5,0 29477 /dev/tty


However, if we check this same output by interrogating the /proc filesystem, we get much different results:

host # ls -l /proc/2034/fd/
total 0
lrwx------ 1 user1 user1 64 Jul 30 15:16 0 -> /dev/pts/9
lrwx------ 1 user1 user1 64 Jul 30 15:16 1 -> /dev/pts/9
lrwx------ 1 user1 user1 64 Jul 30 15:16 2 -> /dev/pts/9
lrwx------ 1 user1 user1 64 Jul 30 15:16 4 -> /dev/tty


So, we see that, although this one particular process has more than 4 "open files," it actually only has 4 "open file descriptors."

An easy way to iterate through each processes open file descriptors is to just run a simple shell loop, substituting your particular version of ps's arguments, like:

host # for x in `ps -ef| awk '{ print $2 }'`;do ls /proc/$x/fd;done

If you're only interested in the number of open file descriptors per process, you can shorten that output up even more:

host # for x in `ps -ef| awk '{ print $2 }'`;do ls /proc/$x/fd|wc -l;done

Here's to being able to over-answer that seemingly simple question in the future ;)

, Mike

Tuesday, May 13, 2008

Killing Zombie Processes In Linux And Unix

Greetings,

Today's post is going to deal with "zombie" processes. These are processes to which the definition of a process only loosely applies.

A zombie process is most often generated when a parent process loses track of its child process and that child process becomes detached. The parent process, generally running some sort of a "wait()" call to receive notification that the child process has exited, loses track of the child process and never receives that information. The child process exits normally, but the parent thinks it's still running, and thus is a zombie process born :)

There are a number of steps to take, from simplest to most obscure, to get rid of zombie processes. And then there's what to do if none of that seems to work. Here we go :)

1. First, identify the fact that you have zombie processes running on your system (you may not notice, and there's a good reason why, which we'll address near the end of this post). You can do this on most major brands of Unix and Linux by running:

host # ps -el|grep Z <--- The -l flag to ps will include the "state" column. The zombie state is represented by a capital Z.

On Solaris 9:

host # ps -el|grep Z

F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
0 Z 0 3038 1 0 0 - - 0 - ? 0:00
0 Z 0 19769 2966 0 0 - - 0 - ? 0:00


On SUSE Linux 9:

/home/ymdg001# ps -el|grep Z
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
0 Z 0 476 9874 0 0 - - 0 - ? 0:00


2. Now that you know you have zombie processes running, the first, and easiest, thing you can do to kill them is to try and "assassinate" the actual defunct (zombie) process. For the process in the SUSE example above, you can always try:

host # kill -9 476

It probably won't work, but it's worth a shot. Sometimes it does and your troubles are over just like that :)

3. Next, you should try to kill the parent process. This may or may not be possible for you. For instance, the parent process may be a process that you "need" to have running. It may also be a process that the Operating System "needs" to have running (like "init" - process 1, shown in the Solaris example above, under the PPID column).

Killing the parent process (if possible) will almost always work to get rid of a zombie process.

Please "never" try to kill init (process 1). If you're successful, your machine will go down hard and fast!

4. Assuming none of the above worked, some common wisdom says you should just give up (and for good reason, which we'll get to very soon ;). However you can try killing both the zombie process (and/or the parent process) using signals other than SIGKILL (or -9). I've seen it happen more than a few times. Different programs trap, and/or handle, different signals different ways. If your zombie doesn't go away when you execute a "kill -9" against it, try a simple "kill" (Which is, technically "kill -15" or SIGTERM). You can try to kill the process with any signal you want. I generally try signals 1 - 15 and then SIGUSR1 and SIGUSR2, just in case they're defined differently for that particular program on that particular system. You'd be surprised how many zombies you can whack with a SIGHUP or SIGINT. Sending a kill SIGCHLD or SIGCLD (Which is the same as SIGCHLD on System V) is a good one to try, as well. Sometimes your chosen method won't make "textbook sense" but it will work from time to time :)

You can find a handy list of signals to try in our old post on translating signal names to numbers and vice versa.

5. And the point I've been alluding to throughout this entire post.

What to do if your zombie process just won't die, you can't kill the parent and/or you're otherwise stuck?

The answer is: nothing.

Here's a brief explanation why: Even though zombie processes alarm most casual users of Unix and/or Linux, and they can make the process table look ugly with all those "defunct" messages scattered in between everything else, a zombie process lives up to its name in more ways than the sense defined above. It literally is like the somewhat-living dead. Although the proc table (and filesystem) have space reserved to record it, the process has already exited and is not consuming any of your system resources. It takes up none of your kernel or system space and is only a minor nuisance since "times" keeps track of its time (If you're a fly, you'll notice the 0:00 slow-down ;)

6. But WAIT!

There's more... (I'm starting to sound like a pitch man ;). Here's one last thing you can do if that ps entry for your zombie process is really bugging you: Once the zombie has totally disconnected from its parent process, you can just use the "wait" command to make it go away. For example:

host # ps -el|grep Z
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
0 Z 0 3038 1 0 0 - - 0 - ? 0:00
host # id
uid=0(root) gid=0(root)
host # wait 3038


...and when that returns (I'd recommend that you run this with "&" to background it - e.g. "wait 3038 &")

host # ps -el|grep Z
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD


It's gone :)

In any event, hopefully, after reading this, you'll no longer worry about zombies :)

Cheers,

, Mike

Wednesday, February 13, 2008

Prtdiag And "Lane Width Failed" Errors On Solaris 10

A lot of admins who work with Solaris 10 Zones may have run into this situation already, but I'm just starting to hear it now from users who like to do their own diagnostics before coming to me with a possible system issue. Don't get me wrong folks; I love it when users show this kind of initiative (as long as they don't go into the data center and start pressing buttons ;)

This problem is similar to the problem Solaris 10 has with old style /usr/ucb/ps, but it isn't quite as prevalent.

The situation that occurs is that a user, trying to gather information on the system that may, or may not be, having a hardware or software issue, runs a pretty standard command called "prtdiag," probably like this:

host # /usr/platform/`uname -i`/sbin/prtdiag

Now, in Solaris 10, this doesn't always cause an error. The reason for this, and how it differs from the old style ps error, is that it only occurs when the user runs prtdiag on Solaris 10 with Zones enabled.

Another thing that makes this unusual error so rare is that, on most Zone-enabled Solaris 10 setups, users accounts are all setup in the non-global Zone and prtdiag will only run in the global Zone. Obviously, the "rules" aren't followed all the time, insofar as system setup and access are concerned. Like they say: the customer is always right, even if he's doing something he probably shouldn't be (or something like that ;)

So far, I've only seen this error on the Mx000 Series Servers from Sun, but that's probably because we use those the most. Generally, the error will present itself in some way similar to the following (Looks scarier than it is and the actual error may vary)

host # /usr/platform/`uname -i`/sbin/prtdiag <-- Stripping the 100 lines preceding the error in my ongoing effort to fight eye-strain ;)

...
IO Lane/Frq
LSB Type LPID RvID,DvID,VnID BDF State Act, Max Name Model
--- ----- ---- ------------------ --------- ----- ----------- ------------------------ ------------------
Logical Path
------------
Getting lane width failed for path /pci@3,800000/SUNW,emlxs@0


And, again, just like our ps error on Solaris 10, running the command as root makes everything work just fine:

root@host # /usr/platform/`uname -i`/sbin/prtdiag

...
IO Lane/Frq
LSB Type LPID RvID,DvID,VnID BDF State Act, Max Name Model
--- ----- ---- ------------------ --------- ----- ----------- ------------------------ ------------------
Logical Path
------------
00 PCIe 3 1, fc21, 10af 1, 0, 0 okay 4, 4 SUNW,emlxs-pci10af,fc21 LPe110094-S
/pci@3,800000/SUNW,emlxs@0


Sun's stock answer, for now, is to change the permissions of the prtdiag command so that it runs setuid root (For those of us who are reading this and don't know what that means - A very small portion of the audience that bothered to read this far, I'm sure - when a program is setuid "username," it will run as that user - with that user's privileges - no matter what user actually executes it)

root@host # chmod 4755 /usr/platform/`uname -i`/sbin/prtdiag

or, if you prefer to change your file modes in alpha:

root@host # chmod u+s /usr/platform/`uname -i`/sbin/prtdiag

Probably the best way to work-around this, and keep with Sun's basic security requirement of not running programs like this setuid root, is to make use of a program called sudo (which comes with Solaris 10). Just include a rule like the one below, so that users can only run "/usr/platform/`uname -i`/sbin/prtdiag" straight up and can't run it with any additional flags or switches. This will allow them to get the information they want and safeguard you, the admin, against any unforeseen issues with this work-around. Example rule below:

ALL ALL = (root) /usr/platform/`uname -i`/sbin/prtdiag ""

Another promise from me that, eventually, we will get to a post devoted entirely to sudo. For now, here's a quick rundown on how this rule reads:

ALL <--- All users can use this sudo rule
ALL <--- This command can be used on any host.
= <--- The cement that connects the preceding user and host restrictions with the commands and options to follow
(root) <--- This command will be run as the user root
/usr/platform/`uname -i`/sbin/prtdiag "" <--- This is the only allowed command. This command specifies that /usr/platform/`uname -i`/sbin/prtdiag can only be run with no additional switches or flags (like "-v"). The "" (double-double quotes) indicate that no switches are allowed after the command.

Hopefully, and in all likelihood, this solution should keep everyone happy. You've kept the security flaw from being exploited, retained the original permissions on /usr/platform/`uname -i`/sbin/prtdiag and allowed users to be able to get their diagnostic output.

All that's left to do is thank your user base for helping make your job easier :)

, Mike




Saturday, December 1, 2007

Old Style ps On Solaris 10

A lot of admin's may have heard this complaint already, but I'm just starting to hear it now from users who are upset about the change in functionality of the /usr/ucb/ps command (as opposed to the standard /bin/ps command).

Up to Solaris version 9, "/usr/ucb/ps -aguxwwwww" would give you extremely detailed ps output (like the entire 2 pages of output from a java command, for instance). Theoretically, in Solaris 10, it still does. Alas, in reality, it doesn't. Here's a link to the publicly available Sun Alert Notification that explains the issue in more detail.

The reason for this stems back to a vulnerability (feature ?) that was discovered in Solaris 8 and 9, regarding /usr/ucb/ps, whereby a user could use the program to view another user's environment settings/variables if they used the "-e" switch. Patches were released for both Solaris 8 and 9, and Solaris 10 was claimed to not be affected by it.

However, all the security patches for Solaris 8 and 9 really did was strip the setUID bit on /usr/ucb/ps and Solaris 10 was released with a /usr/ucb/ps binary that came with 0555 permissions standard (users could no longer get the extended output, but root still could). This would suggest that, perhaps, Solaris 10 wasn't ever really "immune" to the vulnerability, but, more simply, came with the work-around set as the standard. A quick test shows that this is, indeed, the case. Check out the following on a simple /usr/ucb/ps test against the cron process. Note that this is run on a Solaris 10 box, with very recent patch-levels, and I've changed the permissions of /usr/ucb/ps from 0555 to 4555:

host.xyz.com # uname -r
5.10
host.xyz.com # /bin/ps -ef|grep "[2]35"
root 235 1 0 Sep 12 ? 0:07 /usr/sbin/cron
host.xyz.com # /usr/ucb/ps -eaguxwwww|grep "[2]35"
root 235 0.0 0.0 2840 1864 ? S Sep 12 0:06 /usr/sbin/cron LC_COLLATE=en_US.ISO8859-1 LC_CTYPE=en_US.ISO8859-1 LC_MESSAGES=C LC_MONETARY=en_US.ISO8859-1 LC_NUMERIC=en_US.ISO8859-1 LC_TIME=en_US.ISO8859-1 PATH=/usr/sbin:/usr/bin SMF_FMRI=svc:/system/cron:default SMF_METHOD=/lib/svc/method/svc-cron SMF_RESTARTER=svc:/system/svc/restarter:default TZ=US/Central


Looks like the problem hasn't actually been "solved."

So, there you have it; /usr/ucb/ps on Solaris 10 actually does still show regular users the environments of any user that calls a program. Of course, stripping the setUID bit makes that impossible again, but it also makes it impossible for regular users to get extended output for those long java command lines that they need to see.

There are several work-arounds for this work-around. For instance, you could try to use "pargs." It won't work without the setUID bit either, but at least it will tell you that it can't and not just shoot you truncated output with no explanation.

host.xyz.com > pargs -e 235
pargs: cannot examine 235: permission denied


Probably the best way to work-around this, and satisfy Sun's security requirements is to make use of a program called sudo (comes with Solaris 10). Just include a rule like the one below, so that users can only run "/usr/ucb/ps -aguxwwww" and can't run /usr/ucb/ps with any other command arguments, except straight-up with no arguments. They'll get the long output they need and will be prohibited from running /usr/ucb/ps with any other command line switches, like the terrible "-e." Example rule below:

ALL ALL = (root) /usr/ucb/ps -aguxwwww, /usr/ucb/ps ""

We'll go into more detail about sudo in a future post, in all probability. For now, here's a quick rundown on how this rule plays out:

ALL <--- All users can use this sudo rule
ALL <--- This command can be used on any host.
= <--- Pretty self explanatory ;)
(root) <--- These command will be run as the user root
/usr/ucb/pas -aguxwwww, /usr/ucb/ps "" <--- This is the list of allowed commands. The first command specifies that /usr/ucb/ps can only be run with the "-aguxwwww" switches. The second command (multiple commands separated by commas) specifies that the user can also run /usr/ucb/ps with no switches at all (The "" (double-double quotes) indicate that no switches are allowed after the command)

Now, everyone should be happy. You've kept the security flaw from being exploited, retained the /usr/ucb/ps permissions, satisfied the Sun Alert requirements and allowed users to be able to get their extended ouput.

If only every annoying problem were so easy solved ;)

, Mike





Tuesday, October 30, 2007

Keeping Grep Out of Your Grep Output

Hello, again,

Most folks who've worked on Linux or Unix before know that when you run a command like:

ps -ef|grep process

You end up, sometimes, with two lines of output. One with the process you're grepping for (if it exists) and one that shows your grep command executing.

Generally, most people I've known do this:

ps -ef|grep process|grep -v grep

While this is perfectly legitimate, you can actually use the shell's builtin functions to achieve the same thing with fewer keystrokes. It's look cooler too :) Do this instead:

ps -ef|grep "[p]rocess"

The big difference here is the thing we're grepping for. Note that it's in quotes and the first letter (although it could be the second, third or whichever) is in square brackets ([]). This will only produce one line of output: The line with the process you're grepping for.

The reason this works is that the shell interprets the character within the square brackets as a "range" (albeit of one character) and effectively strips the brackets when it interprets the command (it's still running your original "grep process"). And the reason it will never match itself is that, in the "ps -ef" output, your literal brackets will still be there and, of course, won't match.

Enjoy :)

, Mike