Monday, April 21, 2008

Sar, Vmstat, Free Memory And Swap On Linux or Unix

Hey There,

Today's post will, hopefully, help out folks in the future, since I see questions about this a lot on the boards (and am happy to keep reposting my original answers :) Actually, if you have the time and any of this seems nutty, you might check out an earlier post we did on dealing with simple arithmetic in the shell or our post on floating point arithmetic and percentages if you're feeling frisky ;)

Linux is actually quite a bit better with this diagnostic issue, at least when it comes to reporting free memory, than Unix is. My heaviest bias on Unix is toward Solaris, since I use it the most. I'm not saying that the way they handle it is bad, incorrect or in any other way improper. I'm just saying that, on its face, it can be bit confusing. So we'll tackle both free memory, and free swap, reporting on vmstat and sar, one by one:

1. Free memory: Generally, like I said, on most Linux distro's, "sar -r"'s "kbmemfree" column and vmstat's "memory/free" sections use the same standard of measurement. You can easily compare the two and see that they match, since they're both reported in kilobytes.

Solaris' implementation is somewhat different, however. While vmstat's "memory/free" output is displayed in kilobytes, the output of "sar -r" is in displayed in the "number of pages" (???) Okay, so right off the bat you can be sure that the numbers won't agree. For instance, if vmstat's output showed 4840496 on Solaris, the output from "sar -r" would be 605367 (from a semi-concurrent run of both commands). Those numbers aren't even close, which is good since you'll at least notice that something is off and not necessarily think you have a huge problem ;)

So, now you'll need to calculate how many pages it takes to equal the amount of kilobytes reported by vmstat. First you'll want to run a simple command to find out the default size of memory pages on your Solaris Unix system:

host # pagesize

In this case, the page size is 8192 (which is reported in bytes), which gives us an 8 kilobyte page. Then, we'll take that number and multiply it by the number of pages being reported by "sar -r":

host # let a=605367*8 <--- Since vmstat reports in kilobytes, we want to multiply the number of pages shown in "sar -r" by the number of kilobytes in our default page (8 kilobytes, composed of 8192 bytes)

host # echo $a

Now the numbers are pretty much the same. A slight skew is probable, since we didn't run the commands at the "exact same time," memory gets thrown around between processes a lot and a kilobyte (especially these days) is a very small unit of measurement. Still, 4840496 and 4842936 are too eerily close enough for this to have been an accident ;) In short; all is well. The numbers agree.

2. Free Swap is a little bit different on most Linux distro's. In vmstat, it shows up under "memory/swpd" which really means the memory that is currently swapped out, not what's free. But that's okay, because it's reported in the same unit of measurement (kilobytes), and in the same way, that it is in "sar -r" under the "kbswpused" column.

Again, you'll need to make allowances for Solaris Unix, but (before you go cursing it up and down the figurative block) remember that the techniques used to derive the free memory above (and the almost-inverse that we're going to do to figure out if the swap numbers are okay) sometimes show up elsewhere, on other Linux and Unix distributions, so this is actually a good thing :) Really...

Anyway. When you check for how much swap you have free on Solaris, you'll notice that the numbers don't match, again! For example, if "vmstat" is showing 8993040 (in kilobytes, again) for "memory/swap," you'll note that "sar -r" is showing the much larger number, 17993811, in its "freeswap" column. Now we have a situation approximately the opposite of when we tried to make the free physical memory numbers agree.

This time the answer's a little simpler. On Solaris, "sar -r" reports free swap in blocks (which are 512 bytes, of course ;) Two of these equal 1024 bytes which is exactly the size of a kilobyte. Simple. Just divide "sar -r"'s number by 2, and you have the size in kilobytes instead of blocks, so 17993811 (divided by 2) becomes 8996905 (kilobytes), which is pretty close to the 8993040 kilobytes reported by vmstat.

Hopefully this has been more helpful than confusing :)

Best wishes,

, Mike