Hey there,
To most users of Linux and/or Unix (or pretty much any other OS with networking capabilities), the most basic concepts of the TCP and UDP protocols are, at least, somewhat familiar. At the very worst, you've heard of them and, at the very best, you have an insanely detailed fingernail grip on everything TCP/UDP-protocol related and have probably written your own protocol by now so you won't have to deal with all this convention ;)
For the most part, everyone falls somewhere in between. One question that I get asked a lot (and used to ask a lot ;) has to do with the TCP protocol. More specifically, with how an established connection goes about graciously ending. And, if you haven't guessed, the reason the question gets asked so often is that, with all the different states a graceful TCP disconnect goes through, lots of folks (involved in troubleshooting) are curious as to whether the output they're pouring over is "good" or "bad." We'll pretend, for the remainder of this post that the world is black-and-white, cut-and-dry and everything makes sense (and works the way it should) all the time ;)
Since we've never actually covered this topic on this blog before, it makes the most sense (to me, anyway) to lay down the basics of a graceful TCP disconnect. If there's interest (On your end or mine ;), we may follow up with further posts that delve into more detail on the subject. This, of course, means that the typical sequence of events laid out below isn't necessarily how things are always going to go (there are slight differences between Active and Passive disconnects, for one of more than a few instances). For now, we'll stick to the nitty-gritty.
The gracious TCP disconnect, in as much order as I could make of it. The way it's "supposed" to work. This information is only as reliable as your circumstances :) Note that all examples for this post are from Solaris 10 and your explicit command names (like netstat) may vary or have slightly different arguments you need to pass them. Also, we'll note some ways that the TCP disconnect can occur that are technically correct, but "unlike" the step-by-step process listed below and generally big pains in the arse.
Also, and this point is so important I'm giving it its own line ;), it helps to remember that, although a proper TCP "connection" can only be established in one way (the infamous "Three Way Handshake"), the same is not true of a TCP "disconnect." TCP (Over Ethernet, to be precise) is duplexed, which means that it consists of two flows of data; one flowing in either direction simultaneously. Since all TCP requests have to be acknowledged (unless you're just pulling the plug ;) a TCP disconnect is a 4-Way process. Or, more correctly, a 4 "step" process.
Since we're only dealing with the TCP disconnect, we're going to start in the middle of a TCP session. For a given established TCP connection, you'd see something like the following in the output of "netstat -an" (For this entire post, the only really important column in your netstat output will be the one to the far right - from this point on, we'll just deal with that column, to save on space - this post is going over 1200 words as it is) - This status would be the same for both machines, assuming they were engaged in a mutually-agreed-upon TCP network transaction.TCP4 0 48 192.99.99.99.22 10.99.99.99.34406 ESTABLISHED
There are basically only two kinds of TCP disconnects: The Active disconnect and the Passive disconnect.
Each one is fairly simple to draw up (see Wikipedia's TCP Page for some actual pictures ...or graphs), and, for each, we'll show the netstat status you'll note during your troubleshooting work, so that you can be reasonably sure what point your process is at in the TCP disconnect sequence (we're assuming, as usual, that the disconnect you're looking at is bad and someone - responsible in some way for signing your paycheck - is demanding answers ;)
1. FIN_WAIT_1: At this point the serving host has decided to terminate the connection (textbook Active Disconnection) and sends a FIN packet to the client host.
2. FIN_WAIT_2: Then the client sends back an ACK (acknowledgement) packet to confirm that it has received the server's FIN packet.
3. TIME_WAIT: This generally occurs when the client responds with its own FIN packet (Sometimes this state will show up as FIN_ACK)
4. CLOSED: This is a two-parter: In an Active disconnect scenario, this state indicates that the server has acknowledged the client's ACK of its original FIN packet by sending an ACK back to the client indicating that it received the client's FIN packet and has agreed to the termination of the TCP session (I couldn't make that sentence any less confusing and dubious if I tried ;) In a Passive Disconnect scenario, this state indicates that the session is over for approximately the exact same reasons except almost in reverse, since the classic Passive Disconnect has the client initiating the disconnection rather than the server and the final step is the client sending the final ACK and, usually, doing the connection cleanup.
5. CLOSE_WAIT: To summarize right through the first 2 (reversed) points above, in a Passive Disconnect, this state is the result of the client sending a FIN packet to the server (initiating the disconnect), to which the server responds by returning a FIN packet, to which the client application responds with an ACK of the server's FIN.
6. LAST_ACK: Here, the client cleans up, and terminates the connection; sending a FIN packet to the server.
Hopefully this didn't all come out wrong. I have to troubleshoot this stuff a lot, but writing about it has me now doubting whether I'm awake or asleep ;)
Here a few helpful tips to take home with you:
1. Generally, if you see a lot of sockets stuck in FIN_WAIT/FIN_WAIT_1 (and especially) FIN_WAIT_2, you should start your stopwatch or note the time on the clock and wait for about 300 to 600 seconds (the old default of double the maximum round trip time (RTT) - or 4 times the keepalive time - isn't always correct anymore). You can usually figure the maximum time you should be waiting in this state by multiplying your kernel variable for the maximum keepalive time by the maximum number of keepalive retries. On a lot of systems these values are tunable, and (if you're running a high volume website, for instance) you might want to look into reducing one, or both, of these, if necessary. If your kernel has a FIN_WAIT_2 timer variable, just tune that. Otherwise, if these last longer, you can fix the problem by doing everything from figuring out what process ID's are associated with what connected/hung network ports and terminating those, stopping and starting your network application or going all the way up to a full reboot of your physical system.
2. If you see a lot of CLOSE_WAIT sockets languishing around, you can be almost fully assured that the issue is on the application end. Usually, if a TCP application doesn't send the final, required, FIN packet, after finishing the connection cleanup, cleans up the connection but never sends the final FIN packet, or both, you'll end up in this condition (yet another monstrosity of a sentence). Your options are relatively the same.
I'm going to go untwist my brain into the pretzel it was before I started writing this ;)
Cheers,
, Mike
Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.
