Showing posts with label idle. Show all posts
Showing posts with label idle. Show all posts

Monday, March 30, 2009

Idle Process Time On Linux And Unix: How To Find It Again

Hey There,

In our final installment on "finding a process's idle time on Linux or Unix," (last touched upon in our post on echo debugging) we looked at whole lot of ways one could go wrong trying to find the idle time of a process on Linux or Unix. This post is a little more upbeat ;)

All of the previous issues with "who -T" have been worked out. Basically, this means that I've gone over it every which way and could find no good reason to use it, as opposed to "w." Of course, in our particular case, we are looking, specifically, for a single process's idle time (as opposed to a user process's idle time; reported by "who -T"). And, although it's a little bit of a pain (initially), short of programming in C (accessing the pstatus struct on Solaris, to be exact - the name and location may vary from distro to distro of proprietary, or free, Unix and/or Linux), linking the pty information from ps with the "idle time" information from w, seems to be the best way to get this information. So far, it's the most efficient way I could find using simple bash scripting.

Attached to today's post is the final "blog" version of this script. It comes with a few notes (possibly of caution) and may need to be modified for your system/OS (There's the first one ;)

The script runs very simply, and you only need to supply it with a PID. You can, optionally, supply a username as a second argument:

host # ./rip 17787

if you just run it with no arguments, you'll get a usage screen, which may or may not help ;)

host # ./rip
Usage: ./rip PID [user]
User defaults to the value
of $LOGNAME if not specified


Please see our previous post on echo debugging this script for more detailed sample output.

I hope you find some good use for this script, and, without further ado, the oft-dreaded notations of explanation ;)

1. This script has been rewritten to be self-contained. Please see the bottom line for any substitute command you may want to use. Actually, making this command a variable might be a good idea. Just call me "Lazy" ;)

2. You can remove the explicit PATH definition if you like. I put it in there specifically to make sure that the "which ps" variable assignment didn't accidentally grab /usr/ucb/ps on Solaris

3. You can comment out the SIGNAL variable as well, since plain old kill is a sig TERM or 15. The only real reason to set this would be if you wanted to always run kill with a different signal (like SIGKILL,, or 9, for example)

4. I changed the minimum idle time to 30 minutes from 45 (in the previous revisions)

5. All variables appearing in this work are fictitious. Any resemblance to real variables, living or dead, is purely coincidental ;)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# rip - Kill any processes that we know have been idle for more than 30 minutes
#
# 2009 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

PATH="/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin"
procowner="rftprocowner"
prog="\./rft"
sed=`which sed`
awk=`which awk`
ps=`which ps`
grep=`which grep`
kill=`which kill`
signal="-15"

while read a b c d
do
pid=$b
pid_not_var=$(echo $pid | $grep [A-z])

if [[ ! -z $pid_not_var ]]
then
echo "pid $1 contains non-numeric characters!"
continue
fi

pid="$b"
pid_pty="$c"

if [[ -z "$pid_pty" ]]
then
echo "pid $pid is either non-existent, not owned by \"$procowner\" or not attached to a p/tty!"
continue
elif [[ "$pid_pty" = "?" || "$pid_pty" = "console" ]]
then
echo "pid $pid is not attached to a pty!" # kill OR LEAVE IT?
else
pty_num=$(echo "$pid_pty"|$sed 's/^[^\/]*\///')
fi

proc_time=$(w -sh $procowner|grep $pty_num|grep -v grep|$awk '{if ( $2 == '"$pty_num"' && NF == 4 ) print $3;else if ( $2 == '"$pty_num"' && NF == 3) print "0"}')

proc_is_num=$(echo $proc_time | $grep [A-z])
if [[ ! -z $proc_is_num ]]
then
unset proc_time
fi

ext_proc_time=$(echo $proc_is_num | $grep [A-z])

if [[ ! -z "$ext_proc_time" && -z "$proc_time" ]]
then
echo "killing $pid - $d Up Over 24 Hours: $ext_proc_time $proc_time"
### $kill $signal $pid
elif [[ "$proc_time" = "0" ]]
then
:
else
proc_idle_time=$(echo $proc_time|$grep -v "[:]")
if [[ -z $proc_idle_time || $proc_idle_time -gt 30 ]]
then
echo "killing $pid - $d Up More Than 30 Minutes: $ext_proc_time $proc_time $proc_idle_time"
### $kill $signal $pid
fi
fi
done <<< "`$ps -fu $procowner -o procowner,pid,tty,comm|$grep "$prog"|$grep -v grep`"


, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Thursday, March 26, 2009

Simple, But Effective. Echo Debugging On Linux And Unix

Hey there,

I took some time and did some simple "echo debugging" and found that the warning I issued about yesterday's script to find a process's idle time was completely backward. Fortunately, it turns out that my mistaken judgement meant that I had a lot less to worry about, in terms of damage control, from the flaw I perceived in my script (I'm not saying there aren't others, of course ;)

It turns out that the problem was not that the script would sometimes consider an active process, that didn't have an idle column value in the "w -s" output, to be idle and worth terminating. The actual problem was that it would consider processes that had been up for more than a day to be active and not worth terminating. This was a much better situation. At least I wouldn't be killing off active processes!

A sample of just using the DEBUG statements I put in yesterday's script pointed out the error very obviously, like so:

DEBUG::::: PIDTTY 6892 pts/119
DEBUG::::: W user1 119 2days ./program
DEBUG::::: LONGTIME TIME
PID 6892 is OK - Not Idle At All - Remove this message!
------------------------
DEBUG::::: PIDTTY 581 pts/232
DEBUG::::: W user1 232 2days ./program
DEBUG::::: LONGTIME TIME
PID 581 is OK - Not Idle At All - Remove this message!
-----------------------------------


And, yes, I felt like a complete moron when I finally took a second to actually look at the output ;) It's a amazing what a few simple echo statements in a script can tell about what problem's it has :)

From that point, I found several other issues and worked on them accordingly:

1. ISSUE WITH IDLE ALPHA DAYS NOTATION = FIX BY CHECKING FOR NON-NUMERIC TYPES

2. ISSUE WITH NO-IDLE MISSING COLUMN = FIX BY SETTING EMPTY VALUE TO NULL PADDED

3. ISSUE WITH MISSING COLUMN ERROR OUTPUT = FIX BY CHECKING COLUMN COUNT IN TIME

4. MUCH BETTER - "NOT IDLE AT ALL" EXCEPTION NEVER CAUGHT - UNNECESSARY NOW - REMOVED

5. REWORKED TIME HANDLING AND SET TO AMBIGUOUS ALPHA MATCH


Pardon my hysterical notes ;) Most of my problem stemmed from the fact that I switched from full-fledged "w" to "w -s" and made some mistakes in updating the relevant columns that I needed to assign to variables.

I should note that I also considered using "who -T" to get around the one time-stealer in this script. Although it did bring the script down to under a second (processing approximately 100 records), "who" only reports on the "user process." This is a huge consideration, since the "user process" can (and usually is) the parent process of the process you want to check the idle time on. I ultimately decided to stick with "w" since using "who" would mean I'd have to check the parent process, cross reference that with the grep output associated with the pty and then end up back at "w" again to get the process's idle time. A lot of extra work for a lot of extra uncertainty. I didn't want to end up in a situation where the "user process" was idle because the user kicked off a script that ran for 6 hours and then terminate the user's main process (which would kill the kids) based on the idle time of the user's session. Sometimes, lack of precision like that can cause you headaches you never imagined you could have ;)

As you can see below, the updates weren't all that impressive, but I did get the execution time down to 30 seconds from 2 minutes. The only way I could get it lower (that I've figure out so far ;) was to compromise the integrity of the script and remove the one awk statement that was holding it back. Notice the last step I took, just to see what would happen, that proved the awk if/else conditional in the script was responsible for a majority of the execution time:

TRIMMED CODE - REMOVED DEBUG AND UNNECESSARY ECHO STATEMENTS - USING BASH TEST AND OPERATORS
OLD SCRIPT EXECUTION TIME FOR 178 PROCS = 1m27.430s
NEW SCRIPT EXECUTION TIME FOR 179 PROCS = 0m56.517s
NEW SCRIPT EXECUTION TIME FOR 110 PROCS = 0m48.940s
SELF-CONTAINED SCRIPT EXECUTION TIME FOR 111 PROCS = 0m51.048s
ADDED TTY TO PS SCRIPT EXECUTION TIME FOR 101 PROCS = 0m29.703s
REMOVING AWK TTY STATEMENT SCRIPT EXECUTION TIME FOR 100 PROCS = 0m29.991s
ADDED ?, console and "continue" SCRIPT EXECUTION TIME FOR 102 PROCS = 0m33.018s
REMOVED W HEADING (REM SED) AND EXPLICIT USER SCRIPT EXECUTION TIME FOR 101 PROCS = 0m33.382s
TEST - HARDCODED UPTIME AND REMOVED AWK STATEMENT - SCRIPT EXECUTION TIME FOR 170 PROCS = 0m8.512s!!!!!!!!!!!


I'm going to work on it some more, because I believe it can be improved tremendously, but - to satisfy any curiosity, here's some of the mid-work that fixed that issue and made the bash script report correctly. I'll post the one with the fixes noted above (and more, I'm sure ;) once I've thoroughly tested them and removed a lot of the redundancy in this script. Redundancy really gets under my skin. I mean it; redundancy really irritates me. Plus, I don't much care for redundancy ;)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# rip - Kill any processes that we know have been idle for more than 45 minutes - v2-alpha
#
# 2009 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if [[ $# -lt 1 ]]
then
echo "Usage: $0 PID [user]"
echo "User defaults to the value"
echo "of \$LOGNAME if not specified"
exit 1
fi

PID=$1
ISITAPID=$(echo $PID | grep [A-z])

if [[ ! -z $ISITAPID ]]
then
echo "PID $1 contains non-numeric characters!"
echo "-----------------------------------"
exit 2
fi

PID="$1"
USER=${2:-$LOGNAME}

PIDTTY=$(/usr/bin/ps -fu $USER -o pid,tty |/usr/bin/grep -w $PID|/usr/bin/grep -v grep)

echo DEBUG::::: PIDTTY $PIDTTY

if [[ -z "$PIDTTY" ]]
then
echo "PID $PID is either non-existent, not owned by \"$USER\" or not attached to a p/tty!"
echo "-----------------------------------"
exit 3
else
TTYNUMBER=$(echo "$PIDTTY"|/usr/bin/sed '/TT/d'|/usr/bin/awk -F"/" '{print $2}')
fi

if [[ -z "$TTYNUMBER" ]]
then
echo "PID $PID is not attached to a p/tty!"
echo "KILL OR NOT-----------------------------------"
exit 4
fi

echo DEBUG::::: W $(w -s|/usr/bin/sed 1d|/usr/bin//awk '{if ( $2 == '"$TTYNUMBER"' ) print $0}')

TIME=$(w -s|/usr/bin/sed 1d|/usr/bin/awk '{if ( $2 == '"$TTYNUMBER"' && NF == 4 ) print $3;else if ( $2 == '"$TTYNUMBER"' && NF == 3) print "0"}')
#TIME=$(w -s|/usr/bin/sed 1d|/usr/bin/awk '{if ( $2 == '"$TTYNUMBER"' ) print $3}')
#WCOLUMNS=$(w -s|/usr/bin/sed 1d|/usr/bin/awk '{if ( NF == 4 ) print "4";else print "3"}')

ISITANUMBER=$(echo $TIME | grep [A-z])
if [[ ! -z $ISITANUMBER ]]
then
unset TIME
fi

LONGTIME=$(echo $ISITANUMBER | grep [A-z])

echo DEBUG::::: LONGTIME $LONGTIME TIME $TIME

if [[ ! -z "$LONGTIME" && -z "$TIME" ]]
then
echo "PID $PID is ancient - Idle for $LONGTIME... Killing $PID"
# KILLKILLKILL
elif [[ "$TIME" = "0" ]]
then
echo "PID $PID is OK - Not Idle At All - Remove this message!"
else
TIMEIDLE=$(echo $TIME|grep -v "[:]")
echo DEBUG::::: TIME $TIME
if [[ -z $TIMEIDLE ]]
then
echo "PID $PID has been idle way too long - $LONGTIME $TIME so far... Killing $PID"
# KILLKILLKILL
elif [[ $TIMEIDLE -gt 45 ]]
then
echo "PID $PID has been idle too long - $TIMEIDLE minutes so far... Killing $PID"
# KILLKILLKILL
else
echo "PID $PID is OK - Only idle for $TIME minute(s) - Remove this message!"
fi
fi
echo "-----------------------------------"


, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, March 25, 2009

Finding A Process's Idle Time On Linux And Unix

Hey There,

Hopefully yesterday's rant on the simplicity of complexity wasn't too much of a bitter pill. If it was, here's hoping you didn't swallow it ;)

Today, I finally found some time to make a little headway on this project (which should be a lot simpler than it is). Basically, what I'm looking to do is create a way to track specific process's idle times at any given point in time on any given Linux or Unix system. As I mentioned yesterday, there are c structures in Solaris' /proc/PID/status C data files (for one example), but that's just another thing that ended up frustrating me more. As I noted, parts of the OS that are included, should be available for use. The structure is used by the OS, in some shape or fashion, to determine idle times (as we'll see below), but no specific "tool" exists to do what i wanted. Of course, this is limited "to my knowledge." If anyone out there knows of a standard program or command that's managed to elude me, please feel free to email me and tell me all about. I promise to not get offended if you feel the need to belittle me for not having the common sense to look for it where it was at in the first place ;)

Attached to today's post is a rough-draft bash script that attempts to grab a process's idle time. It won't work in all instances, although I've tried to capture as many of those instances as possible. The one big gotcha in this whole mess is that you can't take the output of ps and directly retrieve the idle time for a process from the listing, even if you do your own formatting (I wrote this on Solaris 10 and looked at SUSE Linux 9, but found no love :) Instead, I found that I needed to run ps, extract the pty associated with the process from that (if it existed - which is an exception the script catches) and then use either who or w to retrieve the idle time associate with the pty.

See what I mean? Shouldn't it be a little bit less of a hassle than that?

Okay. I'll admit, if it was, I wouldn't be having half the fun I'm having now trying to script it all out for myself ;) So far, what I've put together works fairly well, although I'm not 100% certain that it's bullet-proof so I would recommend that you leave the "business end" of the code commented out (The stuff that performs unforgivable actions, like killing ;). I have a hard time reproducing it, but I can swear that this code will (every once in a good while) determine that a process that hasn't been idle at all (which removes a column from the "w -s" output) has been idle too long. I'm still working on that part and welcome any suggestions regarding the script, how to make it better, why I'm doing everything the wrong (and/or hard) way when I don't need to and any other constructive criticism :)

The script runs very simply, and you only need to supply it with a PID. You can, optionally supply a username as a second argument:

host # ./rip 17787

if you just run it with no arguments, you'll get a usage screen, which may or may not help ;)

host # ./rip
Usage: ./rip PID [user]
User defaults to the value
of $LOGNAME if not specified


and the following is a sample of the output you might get on a specific run. Here, I've written a command line while loop from a pipe to barbarically hammer out multiple instances at a time ;)

host # time ps -ef|grep "[b]ash"|awk '{print $2}'|while read x;do ./rip $x;done
PID 2664 is not attached to a p/tty!
-----------------------------------
PID 10700 is either non-existent, not owned by "root" or not attached to a p/tty!
-----------------------------------
PID 10855 is OK - Not Idle At All - Remove this message!
-----------------------------------
PID 23217 is OK - Not Idle At All - Remove this message!
-----------------------------------
PID 14730 is either non-existent, not owned by "root" or not attached to a p/tty!
-----------------------------------


Here's another example. This time you'll see what you get if you try to run the script specifying a user other than the user that owns the processes or, in this case, a completely bogus user. This "test" in the script really isn't necessary and I only included it as feeble attempt at damage control. Feel free to remove it if you like:

host # time ps -ef|grep "[b]ash"|awk '{print $2}'|while read x;do ./rip $x joeUser;done
ps: unknown user joeUser
PID 2664 is either non-existent, not owned by "joeUser" or not attached to a p/tty!
-----------------------------------
ps: unknown user joeUser
PID 23633 is either non-existent, not owned by "joeUser" or not attached to a p/tty!
-----------------------------------
ps: unknown user joeUser
PID 10700 is either non-existent, not owned by "joeUser" or not attached to a p/tty!
-----------------------------------
ps: unknown user joeUser
PID 10855 is either non-existent, not owned by "joeUser" or not attached to a p/tty!
-----------------------------------
ps: unknown user joeUser
PID 14730 is either non-existent, not owned by "joeUser" or not attached to a p/tty!
-----------------------------------


I hope you find some good use for this script!

NOTE: Please keep in mind the caveat noted above regarding the sometimes-false-positive I believe this script returns under certain circumstances when it decides a non-idle process (with nothing displayed in the idle column from "w -s" output) has been idle too long! It may never happen again and I may have been seeing spots. Just want to keep you in a "safe" mindset, just in case I'm not completely insane ;)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# rip - Kill any processes that we know have been idle for more than 45 minutes
#
# 2009 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if [ $# -lt 1 ]
then
echo "Usage: $0 PID [user]"
echo "User defaults to the value"
echo "of \$LOGNAME if not specified"
exit 1
fi

PID=$1
ISITAPID=$(echo $PID | grep [A-z])

if [ ! -z $ISITAPID ]
then
echo "PID $1 contains non-numeric characters!"
echo "-----------------------------------"
exit 2
fi

PID="$1"
USER=${2:-$LOGNAME}

PIDTTY=$(/usr/bin/ps -fu $USER -o pid,tty |/usr/bin/grep -w $PID|/usr/bin/grep -v grep)

#echo DEBUG::::: PIDTTY $PIDTTY

if [ -z "$PIDTTY" ]
then
echo "PID $PID is either non-existent, not owned by \"$USER\" or not attached to a p/tty!"
echo "-----------------------------------"
exit 3
else
TTYNUMBER=$(echo "$PIDTTY"|/usr/bin/sed '/TT/d'|/usr/bin/awk -F"/" '{print $2}')
fi

if [ -z "$TTYNUMBER" ]
then
echo "PID $PID is not attached to a p/tty!"
echo "-----------------------------------"
exit 4
fi

#echo DEBUG::::: W $(w -s|/usr/bin/sed 1d|/usr/bin//awk '{if ( $2 == '"$TTYNUMBER"' ) print $0}')

TIME=$(w -s|/usr/bin/sed 1d|/usr/bin/awk '{if ( $2 == '"$TTYNUMBER"' ) print $3}')

ISITANUMBER=$(echo $TIME | grep [A-z])
if [ ! -z $ISITANUMBER ]
then
unset TIME
fi

LONGTIME=$(echo $TIME | grep [A-z])

#echo DEBUG::::: LONGTIME $LONGTIME TIME $TIME

if [ -z "$LONGTIME" -a -z "$TIME" ]
then
echo "PID $PID is OK - Not Idle At All $TIME - Remove this message!"
elif [ ! -z $LONGTIME ]
then
echo "PID $PID is ancient - Idle for $TIME... Killing $PID"
# DO_WHAT_YOU_HAVE_TO_DO_TO_THE_PID_HERE
else
TIMEIDLE=$(echo $TIME|grep -v "[:]")
# echo DEBUG::::: TIME $TIME
if [ -z $TIMEIDLE ]
then
echo "PID $PID has been idle way too long - $LONGTIME $TIME so far... Killing $PID"
# DO_WHAT_YOU_HAVE_TO_DO_TO_THE_PID_HERE
elif [ $TIMEIDLE -gt 45 ]
then
echo "PID $PID has been idle too long - $TIMEIDLE minutes so far... Killing $PID"
# DO_WHAT_YOU_HAVE_TO_DO_TO_THE_PID_HERE
else
echo "PID $PID is OK - Only idle for $TIME minute(s) - Remove this message!"
fi
fi
echo "-----------------------------------"


, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.