Script To Blind Test Local Ports On Linux And Unix

Hey there,

It's been a while since we touched on network port-knocking, like we did back in December in our post on non-maliciously scanning for open network ports. And, in all that time, we've gotten sidetracked so many different ways we're just now getting around to addressing the complement to mass-querying network ports on other hosts in our network (hosts run by folks who, of course, don't mind if we query all of their machine's ports :) Today, we're going to take a look at simply and quickly getting as much dirty information about open ports on your own localhost as possible (By localhost, we mean, the server you're currently logged into and by dirty, we mean "not necessarily useful, but, probably, interesting and, maybe, useful" ;)

This intro-to-the-weekend script is pretty short and sweet. It doesn't require that you be able to do anything other than execute it (and run telnet) on your local system (from it, directly to it, which shouldn't be a security concern) and also doesn't accept any arguments (Technically, it will accept them. Then it will passively ignore them :) It can be run very simply from the command line as:

host # ./porttest.sh <-- Or whatever you decide to rename it

We've elected to have our port scan start at 80 (The traditional http server port) and grab every other (higher) listening port on the localhost and query all of them, as if they were http servers, by sending a simple HTTP/1.0 GET request. As a blanket request to any number of known, and unknown, ports, it's not always the best way to interrogate, but it does get lots of useful information from any sort of web server and a few other sorts of servers as well. The only thing you have to settle down and be comfortable with is the fact that, a lot of the time, you can find out just as much about what's running on a particular port by reading the error message you receive from a bogus query as you can from reading the result of a successful one.

As a "for instance," here's one such error message we received on a trial run:

Testing 127.0.0.1 on port 32848... HTTP/1.1 404 Not Found
server: BBC 05.20.050; com.hp.ov.ctrl.ovcd 1.5.0.0
Connection closed by foreign host.

Even though the HTTP/1.0 GET request failed, we got enough information from the service resident on that port to figure out that it is, in fact, HP OpenView!

Have fun with this one, but please quit using it if your boss, or a member of your security team, complains. It might be fun, but it's not that much fun ;)

Cheers,




This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash
#
# porttest.sh - Test those ports.  This is not a drill ;)
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
#
hostname=`hostname`
netstat -an|awk '{ if ( $NF ~ /LISTEN/ ) print $4}'|awk -F":" '{ if ( NF == 2 ) print $1 " " $2}'|while read ip port
do
        if [ $ip = "0.0.0.0" ]
        then
                ip="$hostname"
        fi
        if [ $port -gt 79 ]
        then
                echo -n "Testing $ip on port $port... "
                (echo "GET / HTTP/1.0";echo;sleep 3)|telnet $ip $port|egrep -i 'server|http'
                echo
        else
                echo "skipping port $port"
                echo
        fi
done

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Network Port Querying Script

Hey there,

The script I've put together here was originally written to meet a certain demand. That demand was actually my own, but that's beside the point ;)

This script should come in useful for you if you ever need to query a port and find out what's going on with it (like who's using it and/or what process id is associated with it). It's simple to invoke (taking only the port number as its argument) and produces information that can be a great aid in troubleshooting network connection issues.

If you refer back to this previous post you can check out a small walkthrough regarding how to query a port using lsof and/or the proc commands. This script uses lsof also, but combines it with netstat to produce output in an easy to read format, while grabbing a little more information in the process. Assuming we call it portquery, it can be invoked like this:

host # ./portquery 22 <--- Let's just see what's going on with SSH

and it will produce output for you like the following. Note that it produces a formatted output block for every single process connected to a port. On a high-traffic machine, checking SSH might produce a few pages of output. This is what it looks like when it's run:

Port 22 Information :
        Service = sshd
        PID = 469
        User = root
        Protocol = TCP
        Status = LISTEN
Port 22 Information :
        Service = sshd
        PID = 469
        User = jimmy88
        Protocol = TCP
        Status = LISTEN

...and the list goes on to print out information blocks for every PID attached to that port. This script has been a great help for me not only in that it makes a manual process automatic, but also in that it's easy for other non-admins to read.

Here's hoping you have some use for it :)

Best Wishes,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/ksh

#
# 2007 - Mike Golvach - eggi@comcast.net
# 
# Usage: portquery [port number]
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

trap 'exit' 1 2 3 9 15
if [ $# -ne 1 ]
then
 echo "Usage: $0 portNumber"
 exit 1
fi

portnumber=$1

/bin/netstat -a |grep -w "$portnumber" >/dev/null 2>&1

if [ $? -ne 0 ]
then
 echo "Nothing's listening on - or using - port $portnumber"
 exit 1
fi

/usr/local/bin/lsof 2>&1|grep -v "^lsof:"|grep -w $portnumber 2>&1|while read x
do
 portinfo=`echo $x|awk '{print $1 " " $2 " " $3 " " $4 " " $5 " " $6 " " $7 " " $8 " " $9 " " $10}'`
 echo "Port $portnumber Information :"
 echo " Service = `echo $portinfo|awk '{print $1}'`"
 echo " PID = `echo $portinfo|awk '{print $2}'`"
 echo " User = `echo $portinfo|awk '{print $3}'`"
 echo " Protocol = `echo $portinfo|awk '{print $8}'`"
 echo " Status = `echo $portinfo|awk '{print $10}'|sed 's/(//'|sed 's/)//'`"
done

, Mike

linux unix internet technology

Using Linux strace To Debug Application Issues

Today's post is a little bit of a walkthrough of using RedHat Linux strace to debug (and find the root cause of ) a system issue and a little bit of caution about how much information you should really share with application project managers if you don't want to be stuck supporting a hacked solution for longer than anyone should. As far as job security goes, it can't hurt, but certain situations really do require that folks upgrade their software to the vendor supported version, no matter how creative your solution ends up being.

Note: If you're looking for an in-depth examination of strace, this post isn't it. Not to turn any readers off, as we'll certainly be exploring that in a future post, but this is more of a walkthrough of a problem resolution involving strace rather than a dissertation on the command itself.

In this particular instance, we were working with a product (heretofore referred to as the Product or Product) that, to my knowledge, had just begun having "issues." This generally means to me that somebody did something they weren't supposed to or were trying to do something they weren't supposed to. I'm cynical, but I try to have the common courtesy to keep it to myself ;)

The Product was a client/server application running on the Java platform that had suddenly begun dropping connections from the application server to the backend database. I did a quick check of the /proc/****/status file (and, of course, gave a quick nod to "top" and "netstat -an"), confirming that they were dropping quite a few, like so:

host # cat /proc/14653/status | grep Threads
Threads: 311

host # top <-- Truncated to just show the top process, which was the "failing" one.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14653 user 16 0 1634m 1.1g 40m S 120 14.1 296:46.54 java

I found the "total time used" (listed under the TIME+ column) somewhat alarming since the process had only been up for about a half an hour! It couldn't possibly be correct.

The next thing I did, since I like to go to the guts when I get brought in on an emergency issue regarding a Product I've never used before, was to shut it down and restart it with strace; teeing the information off to a file in /tmp (since I like to watch the characters fly by), like so:

host # strace -vF ./product arguments 2>&1|tee /tmp/OUTPUT

Note that this could have also been accomplished by running the process, getting it's pid from the ps tables and adding the -p PID flag to the strace command in order to attach to the process after starting it. I prefer to start the process with strace, if I can, in case something critical happens during the initial load-up.

Long story short (It is Sunday - Our day of rest - after all ;), I found that the Product process was dumping tons of FUTEX_WAIT errors. At this point, I checked the version of the OS with uname for the only part I really cared about at that point:

x86-64

The machine was running RedHat Advanced Server 5 - 64 bit. The FUTEX_WAIT errors have to do with the implementation of Fast Userspace Mutex locking implemented in the stable version of the RedHat Linux kernel 2.6 and up. This raised a red flag, and caused me to ask the inevitable question: When did this start happening?

As it turns out, the Product had previously been hosted on another machine running RedHat Advanced Server 4, 32 bit. This made perfect sense, since the Mutex locking mechanism at the kernel level they "used" to run on was completely different, and not compatible, since the Product was built to use the old style Mutex locking mechanisms.

Futile attempts were made to add lame fixes, like:

LD_ASSUME_KERNEL=2.4.whatever (Apologies; I don't remember what it was off-hand since it didn't end up helping at all ;)

to the Product's startup scripts, but this didn't resolve the problem. The "actual" (and correct) resolution would have been to upgrade to the vendor-supported version of the Product. The manufacturer of the Product actually had a more recent version available that was built specifically to address this issue and run on RedHat AS 5 - 64 bit.

And, here's the kicker (as per the cautionary message at the beginning of this post): As some of you may know, a bug in strace (It's actually more of a "feature" since it almost "has" to happen) makes it slow down a process slightly due to all the extra work it has to do to parse all the system calls, opens, reads, writes, errors, library loads, etc.

strace slowed the process down enough that the FUTEX_WAIT errors stopped occurring and the Product began functioning normally again. We ended up adding it to the init scripts after much debate over spending a few bucks to avoid possible total confusion in the future.

In retrospect, I'm glad that I was able to help identify the issue and find the root cause. On the other hand, I wish there was some way I could have avoided identifying the problem without providing a cheap and quick "out" for the end user. I'm sure, in the future, someone else will have to work on this Product (maybe a problem with the version of Java or something more insidious) and they'll be given about as much information as I was. Documentation was laid down, but I don't expect anyone will read it. Sometimes, the solution is as bad as the problem, in a way or two ;)

Enjoy your Sunday :)

, Mike

linux unix internet technology

A Few Linux Networking Tips

Here's a little something for those of us who use Linux (The place I work uses RedHat primarily) on a day to day basis. Lots of shops these days are switching from the more expensive solutions, like those offered by Sun and HP, to cut cost of deployment and maintenance, which opens up a great pathway into the Linux administration field for folks who are eager to learn. I can tell you, after working all angles of the *nix arena for a decade or so, that there's nothing more grating than having to fill out a form and wait for someone to do something you're supposed to be being paid to be good at. I'm not going to sneeze at my paycheck, of course, but I'm worried that, if I work at too many big corporations with big contracts, my mind will atrophy and I'll die a slow miserable death long before my body gives out ;)

In this post, I just wanted to touch on some Linux networking basics. I try to write this blog for users of all skill levels and I think some of my posts assume a lot of pre-knowledge. This isn't a "for dummies" site, by any means (I've never understood how they marketed that series of books so well. I've tried flat-out telling people they're idiots for most of my natural life and it's never ended well ;). That being said, I'm hoping this blog attracts folks with a wealth of experience and is also accessible to those new to the field.

So, today, I'm going to touch on some Linux networking tips at a basic level. As with everything, over time, I'll dive into these subjects in greater detail. But for now, we'll get to the meat of the post. A lot of times, I find, it makes more sense to know what to do and understand it later, rather than the opposite. To that end (This is all pretty much RedHat specific - please don't try this on a Sun box without a healthy ego - the error messages can be blunt ;) here we go:

1. Adding a default router:

You can generally do this one of two different ways:

If the NIC is already configured and UP, all you need to do is use the route command, like so:

host # route add default gw 127.0.0.1 dev eth0

If you want to add a default route permanently, you'll just need to add this line to the existing /etc/sysconfig/network configuration file:

Note that the NETWORKING and HOSTNAME variables should already be in there (If not, assign them values of yes and "whatever your hostname is", respectively. Also, your network may not be up ;)

GATEWAY=127.0.0.1

Of course, if you prefer, you can always use the /etc/init.d/network script to bounce your NIC's and routes (You set these up in the /etc/sysconfig/network and /etc/sysconfig/network-scripts/ifcfg-eth0, etc, scripts). You can also use the service command to bring up your network, bring it down or restart it (which will, again, re-read your configuration).

2. Displaying your NIC's device driver settings:

This is most commonly done with a command called ethtool. To get your NIC's settings you could do the following:

host # ethtool -i eth0
driver: tg3
version: 3.10u6
firmware-version:
bus-info: 05:01.0

You can then use this info to help with problem solving. For instance, if eth0 isn't coming up correctly, perhaps eth0 doesn't have a proper alias setup in /etc/modules.conf, like:

alias eth0 tg3 <--- This line tells us that eth0 is actually an alias referring to the tg3 device driver.

If you look in there and it's:

alias eth0 e1000

you've found the problem right there!)

3. Display your NIC's Speed, Duplex and Negotiation settings (also with ethtool):

This one is just as simple as the command above (ethtool is much nicer than using old-style ifconfig, netstat and/or kstat - although they all have their virtues and are necessary depending on how old your Linux distro is)

host # ethtool eth0
Settings for eth0:
Cannot get device settings: Resource temporarily unavailable
Supports Wake-on: g
Wake-on: d
Current message level: 0x000000ff (255)
Link detected: no

Looks like the link's not up for that one! It's just as easy to spot if all's well (You can sometimes tell from a good distance away ;):

host # ethtool eth1
Settings for eth1:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Current message level: 0x000000ff (255)

4. Now to configure, or initialize, those settings, we'll use ethtool as well:

Assuming we found out that the device associated with eth0 didn't have the cable connected, and we've got that all set up (along with correcting the alias in /etc/modules.conf, if that was wrong), we could fix it all by doing something like this:

ethtool -s eth0 speed 1000 duplex full autoneg off

You can add as many option/value pairs as you need (e.g. speed 1000) to get the job done. Don't forget to update the /etc/sysconfig/network and /etc/sysconfig/network-scripts/ifcfg-eth0, etc, files with your settings. If you don't, you'll have to do this every time the machine reboots!

Hope this has helped you out some or, at least, helped you get started taking on the Linux world :)

Cheers,

, Mike

linux unix internet technology

Estimating Network Traffic Speed Using Regular File Utilities

Most of the time, if you want to estimate how fast your network is moving, you'll want to look at the output of tools like "netstat" and the like. Some file transfer programs, like "scp" are kind enough to print out the speed at which they're transferring a file to you while they're doing it.

But, sometimes you'll be stuck waiting on an file to make it all the way over to your machine using a protocol like "ftp." And, while it's possible to measure the speed of transfer in many different ways, I threw together this little script to let you know how fast your file is being "ftp"'ed to you by taking measure of the file being transferred, itself.

Check it out. It's kind of fun to see that there really is more than one way to skin a cat. In this case, you don't even need a skinner ;)

Note: Adjust the time variables to your liking. I wrote this for transfer of rather large files! The script, below, at the most basic level, measures the difference in the received file's size against the passing of time to determine approximately how fast your network is transferring that file.

And, yes, this is another experiment in "brute force scripting" (Getting results as fast as possible using scripting, even if it means all your t's aren't dotted ;), so you'll notice that the first output will be 0 and is accompanied by a message to the effect that measurement can't take place until more data is collected.





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/ksh

#
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

last_total=0
first_time=1

while :
do
date
echo "KB PRIOR $last_total"
total=`du -sk|awk '{print $1}'`
echo "KB ACTUAL $total"
let new_total=${total}-$last_total
last_total=$total
echo "KB DIFF $new_total"
let hourly=${new_total}*60
echo "KB HOURLY $hourly"
let pre_average=`echo 2 k ${hourly} 1048576 / p |/usr/bin/dc|/usr/bin/sed 's/\.//'`
average=`echo $pre_average|/usr/bin/sed 's/^\(.*\)\(..\)$/\1\.\2/'`
netstat -ian|grep bge3|sed -n 1p|awk '{print "Packets In: " $5 " Packets Out: " $7}'
if [ $first_time -eq 1 ]
then
echo "Average incalculable until more data received"
first_time=0
else
echo "Averaging $average Gb per hour"
fi
echo
sleep 60
done

Enjoy!

, Mike

linux unix internet technology

How to Find a Rogue Process That's Hogging a Port

Hey there,

Today's little tip can actually come in useful even if the information you're seeking isn't "mission critical" (which, by the way, ranks among one of my least favorite terms. If there's one thing positive I can say about where I work now, it's that they don't describe every problem, resolution or project as if we were engaged in war -- but that could be an entirely separate post ;).

I've actually been asked to figure out what process was running on what port more often for information's sake than to try and figure out why something was "wrong," but the same principles apply. The scenario is generally something like the following:

Internal customer Bob needs to start (or restart) an application, but it keeps crashing and getting errors about how it can't bind to a port. This port is necessarily vague, since, in my experience, it's very common to be asked to figure something out with little or no information. I consider myself lucky if I have a somewhat-specific description of the problem at the onset. As we all know, folks will sometimes just complain that "the server is broken." What does that mean? ;)

The troubleshooting process here is pretty simple and linear (perhaps more detail and information in a future post regarding similar issues, as any problem or situation can be fluid and not always follow the rules). In order to try and fix Bob's problem, we'll do the following:

1. Double check that the port (We'll use 1647 as a random example) is actually in use by running netstat.

netstat -an|grep 1647|grep LIST

you can leave out the final "grep LIST" if you just want to know if anything is going on on port 1647 at all. Generally the output to look for is in the local address column (Format is generally IP_ADDRESS:PORT - like 192.168.1.45:1647 or *:1647 - depending on your OS the colon may be a dot). Whether or not you're checking for a LISTENing process, information about a connection from your machine on any port to foreign port 1647 shouldn't concern you.

2. We're going to assume that you actually found that the port is either LISTENing, or actively connected to, on your local machine (if it isn't, your troubleshooting would likely take a much different turn at this point). Now we'll try to figure out what process is using that port.

If you have lsof installed on your machine, figuring this out is fairly simple. Just type:

lsof -i :1647

and you should get a dump of the list of processes (or single process) listening on port 1647 (Easily found under the PID column). They're probably all going to be the same, but, if not, take note of all of them.

3. Run sommething along the lines of:

ps -ef|grep PID

and Problem solved! You now know what process is listening on port 1647 and you'll probably end up having to hard kill it if Bob doesn't have any idea why it won't let go of the port using standard methods associated with whatever program is using it.

But, sometimes, the last part isn't that simple, so:

4. What's that? lsof isn't installed on your machine? My first inclination is to recommend that you download it ;) Seriously, it's a valuable tool that you'll find a million uses for. But you can find out the process ID another way, just in case you can't get your hands on it and/or time is of the essence, etc.

In this instance, and we'll just assume the worst, you can use two commands called "ptree" and "pfiles" (these are standard on Solaris in /usr/proc/bin - may be located elsewhere on your OS of choice and/or named somewhat differently). Use the following command to just grab all the information possible and weed it down to the process using port 1647:

for x in `ptree -a | grep -v ptree | awk '{print $1}'`
do
pfiles $x 2>/dev/null|grep 1647
done

and you'll get the line of output that maps your PID to your port. The above is, admittedly, somewhat messy (not really messy, but you'll end up printing a lot of blank lines ;) Feel free to tailor it to your needs and make it more general (I explicitly used port 1647, but that should also be a variable if you want to create a little script to keep in your war chest).

Run your ps, as above, and now you should know what process is hogging that port and, in the process, making Bob's life miserable. If you cleanly kill that process, Bob should have one less thing to worry about and his program should be able to bind to the now-free port :)

, Mike

linux unix internet technology

Figuring out your NIC's speed and duplex on Solaris

Here's another one admins and users have to do all time. If there's ever an issue with networking, you need to be able to confidently say that your NIC is up at 1Gb full duplex (or whatever your network admin insists).

The way to check this has changed somewhat in Solaris 10, but the old way to check is still available; although not totally reliable.

For instance, you can use "ndd" in all flavors of Solaris (at least from 2.6 up) to get information from /dev/hme (or whatever your NIC's device driver is). Generally, you would look at the speed and duplex settings using the following commands (slight variation depending on NIC's - e.g. 100Mb hme's don't have values for the 1000 Mb queries)

The following commands are pretty useful, and non-destructive, for any device driver, even though you'll get errors for all the stuff that isn't supported:

/usr/sbin/ndd -set /dev/ce instance 0
/usr/sbin/ndd -get /dev/ce adv_1000fdx_cap
/usr/sbin/ndd -get /dev/ce adv_1000hdx_cap
/usr/sbin/ndd -get /dev/ce adv_100fdx_cap
/usr/sbin/ndd -get /dev/ce adv_100hdx_cap
/usr/sbin/ndd -get /dev/ce adv_10fdx_cap
/usr/sbin/ndd -get /dev/ce adv_10hdx_cap
/usr/sbin/ndd -get /dev/ce adv_autoneg_cap

Of course, replace the "/dev/ce" with your particular driver. The only downside to this hack-and-slash method is that you may see 1's (indicating that the parameter is set) rather than 0's (indicating that the paramet is not set) in more than one place (like in adv_1000fdx_cap , adv_100hdx_cap and adv_autoneg_cap all at once ???)

The best way to do it, in my experience is to use either "netstat -k" (In Solaris up to, and including, version 9) or "kstat -p."

In Solaris 9, assuming the same NIC driver and instance "ce0," you can do the following to find out the status of your NIC:

netstat -k ce|grep 0|egrep 'link_speed|link_dupl'

On Solaris 10, you'd do this:

kstat -p|grep ce0|egrep 'speed|dupl'

Basically, the speed should be 1000 for Gb, 100 for 100Mb, etc. Your duplex is represented numerically as either 0, 1 or 2.

0 = No Duplex (or Down)
1 = Half Duplex
2 = Full Duplex

This output is great to show anyone who doubts your conviction ;)

Take care,

, Mike

linux unix internet technology