Tuesday, November 20, 2007

Trimming Space in /var - The Problem with Solaris' lastlog

Here's another interesting tidbit from the "terminally boring" archives of system administration ;)

A lot of times, when you get a complaint that /var, on a Solaris box, is exceeding whatever size limitation you've placed on monitoring it, your first inclination is to go and wipe out the largest (but not most necessary) files immediately and see if that takes care of the problem.

Every once in a while, if you're checking around, you may notice that /var/adm/lastlog is gigantic. Theoretically, zeroing that out (catting /dev/null into it), should take care of your disk usage problem as it seems fairly obvious. Some of us would just leave it at that. The rest of us would check "df -k /var" again and notice that the percentage of partition space used is relatively the same. That doesn't seem to make any sense.

This is where the interesting part comes in. Solaris' implementation of lastlog has an interesting bug/feature that makes it seem larger than it is; but only some of the time.

The reason for this is that, while its size remains fairly static (about 24kb maximum), lastlog always indicates its size (when using "ls -l") relative to the user account id that last logged in (after the 24 Kb maximum is reached). The equation is roughly "the user account id number" multiplied by "28 bytes." So, when root logs in with a userid of "zero" (after you've zeroed out the file), it seems to grow to a size of 28 bytes (Yes, this is the minumum - and, yes, 28 times zero should equal zero ;) However, if you do an "ls -s" (to figure out the number of blocks) and a "du -k" (to figure out the size in Kb) on /var/adm/lastlog, you'll see that it's not really taking up all that much space. Below:

$ ls -l /var/adm/lastlog
-r--r--r-- 1 root root 28 Nov 19 17:51 lastlog
$ ls -s /var/adm/lastlog
2 /var/adm/lastlog
$ du -k /var/adm/lastlog
1 /var/adm/lastlog

If a user with a userid of 6504 logs in (after zeroing out the file) the block and Kb size will show the maximum (48 and 24, respectively), but "ls -l" reports:
$ ls -l /var/adm/lastlog
-r--r--r-- 1 root root 182112 Nov 19 17:53 lastlog

Crazy, yeah? But, interesting to know, and helpful, since you can avoid this file when trying to pare down the size of the /var partition.

As a caveat, the "fake" size reported by "ls -l" is only fake when lastlog is being manipulated by the Solaris Operating System in the manner in which it was specifically designed to be manipulated. If you copy that 500Mb file (or move it, tar it, etc) it pads all the "blank" space with NULLs and you end up having a file on your hands that really "is" insanely large!

, Mike