More Quick Ways To Find CPU Bottlenecks On Linux

Hey there,

Yesterday, we took a look at some useful commands to help identify memory bottlenecks in Linux. More specifically, we were looking at SUSE 9.x. We're going to use the same Linux version today (for our examples), although - again - much of this stuff translates fairly simply to other distro's. This post will be different from yesterday's in that we'll be focusing on specific CPU-related commands you can use, in a time of crisis (or, perhaps, just a long, drawn-out eternity of soul-crushing boredom ;), to determine if the CPU(s) on your machine are at the fore of whatever problems your system is having.

On a completely unrelated note, last night's election was, indeed, historic and satisfying. Of course, I didn't get to sleep until 3am because my wife was waiting for Indiana to finish counting its votes and hoping for an Obama landslide. When I snapped out of the temporary stupor that substituted for sleep this morning, I noticed that Indiana was still a "partially yellow" state. Hopefully, they'll get the votes counted before I publish this post. If not... I'll just thank God that my wife doesn't read this blog. She's a wonderful woman, but (like most people who've known me for a long time) probably more than willing to snatch the life right out from under me ;)

And here we go. Today's hit list for CPU testing on SUSE:

1. top. This command comes in first again. Explaining it again isn't necessary. When you looked it over to find your memory bottleneck you, no doubt, noticed the %CPU column and summary at the top. About the only thing special, with regards to CPU reporting on top, is how it deals with a multiple-CPU system. You can generally flip between the regular output (All CPUs' statistics combined) and forcing it to show per-CPU stats by using the capital "I" (You may see a message indicating the "Irix mode" is either off or on. On some builds, I've seen this work but give no verbose indication of the change).

2. host # more /proc/cpuinfo

General output will look something like below. Generally, on most newer (and just slightly older) machines, you'll be dealing with CPU's that list out in /proc/cpuinfo as more than they "physically" are. That is to say that hyperthreading/multiple-core CPU's will not appear in this file as the single physical entity that they are. Of course, your situation may vary, but this file should (at the very least) give you a feel for whether you have a bad CPU problem. In a situation where you have 4 physical CPU's (hyperthreading to simulate 8 CPU's) you can get a good indication of whether the problem your facing (we'll just assume you're facing a problem ;) is of a physical nature. If 2 virtual CPU's are down (in proper sequence), you probably need some new parts :) The "physical id" line value, when compared with the "processor" line value, is usually a good indication of whether or not your system is using hyperthreading or any other virtual enhancements. Odds are, you'll probably know this information before you ever have to look at this file.

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 2
model name      : Intel(R) Xeon(TM) MP CPU 2.5GHz
stepping        : 5
cpu MHz         : 2495.259
cache size      : 512 KB
physical id     : 0
siblings        : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips        : 4980.73

3. vmstat 2 - another return visitor from yesterday's post on memory mangle-ment ;)

This will run vmstat every 2 seconds, ad infinitum. You should check the "id" (idle percentage), "us" (percentage of CPU resources dedicated to user processes) and "sy" (percentage of CPU resources dedicated to kernel processes) columns first. If the idle percentage is low, knowing whether user processes (like a program running on your system) or the kernel (basically, the operating system and all its built-in facilities) are taking up all the CPU resources can get you pointed in the right direction early on.

You'll also want to consider the "wa" (I/O wait - although this does not necessarily mean that you're experiencing CPU-related I/O wait), "in" (kernel interrupts) and "cs" (kernel context switches) columns as well. High activity in any of these columns could indicate overuse of the CPU (Note that vmstat, although it can tell you a lot about what's going on with your system, cannot pinpoint the particular application or system setting that may be causing the events it reports!)

4. If you do notice a high number of CPU interrupts in your vmstat output, be sure to check out the contents of /proc/interrupts. Check it, for instance, every 10 seconds for a few minutes. Within that amount of time, the contents of the /proc/interrupts file may point you directly to the culprit. This may not be the answer, but should provide you some relief while you find the real problem and need to verify it doubly :)

Note that, as a rule, lots of kernel interrupts and CPU context switches (especially in the thousands) are a fairly good indicator of CPU load reaching maximum capacity.

5. Check your standard log files in /var/log. If you find a ton of messages there (or even just a few), they can provide invaluable clues. Combining this additional information with the output of vmstat, top and (possibly) the contents of /proc/cpuinfo and /proc/interrupts, should paint a fairly vivid picture and allow you to assess, quickly, whether or not you need to focus more effort on reducing CPU load or, possibly, replacing a bad CPU or two.

Once again, I wish you a good night and hope this little introduction to CPU bottleneck troubleshooting has been "accessible" or, at least, somewhat helpful to you :)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

A Few Ways To Gauge Possible Memory Bottlenecks In SUSE Linux

Hey there,

Today we’re going to take a look at a few (well, maybe more than a few) ways to check your Linux box to find out if the reason it’s beginning to perform poorly is related to memory or memory-management issues. For today’s examples, in particular, we’ll be using SUSE Linux 9.x, but most of these examples translate just as easily to RedHat, Ubuntu and other widely-used distro’s, with some minor modifications.

First things second, I’d like to apologize for the lack of animation in my “voice” in this post. As you probably well know (Even if you don’t live in the USA), we’re (hopefully) going to find out who our next President is by this evening (I write these a day in advance) and I’m somewhat distracted, even though I know I shouldn't start caring until its late enough in the game for any of the numbers to make a legitimate difference. My wife is hanging on the 1% from here and 3% from there ;) I won’t reveal whom I voted for, since this blog doesn’t take any particular political stance. I’ll just let it be said that I’m really hoping my candidate comes in. It’s going to be a farce if the other contender somehow manages to steal this thing….

And, secondly, we’ll walk through the many different (and easily accessible) ways you can check up on your Linux system’s memory usage to make sure that all is well. Note that for some commands, you need root privilege to get information of any value. Most of these commands don’t require that level of access, but (as a rule) if you need to interface with the kernel (or access its symbol/memory tables, etc) you’ll probably just end up with a big old raspberry when you try to run a few of these commands. Not to worry, though: You’ll be able to get more than enough information from this mass of commands to make a reasonably accurate estimation, no matter what level of access you have.

And here’s a quick look at assessing your system’s (possible) memory problem, quickly:

1. top.

This command should be in everyone’s arsenal. If your Linux distro doesn’t include it (as part of a standard pkg) you can always build it from source. It’s been my experience that this tool comes standard with everything nowadays. Pay special attention to the physical and virtual memory sections. top should give you a fairly accurate count of free vs. used memory for both types.

2. ps –aux |grep AprocessYouSuspect

This will provide you with similar information. Pay special attention to the %MEM, VSZ and RSS columns.

3. vmstat 2

This will provide you with memory statistics at 2 second intervals (also the default refresh rate for top).

Watch the following columns (note that the “inact” and “active” columns only show up when you invoke vmstat with the “-a” flag:

swpd: To check on the amount of virtual memory in use
free: To check on the amount of free memory
buff: To check on the amount of memory being set aside for buffering
cache: To check on the amount of memory set aside for caching.
inact: To check on the amount of inactive memory
active: To check on the amount of active memory
si: To check on the amount of memory swapped in from disk
so: To check on the amount of memory swapped out to disk. <-- For more on the difference between swapping, paging, etc, check out our older posts on swapping and Paging on Linux and Unix and its inevitable follow-up

4. ps -o vsz,rss,tsiz,dsiz,majflt,minflt,pmem,cmd 9999

This will format your ps output to spit out virtually all the memory information ps can get you for a specified process ID (In this case: 9999)

5. cat /proc/9999/status

This will provide you with a lot more detailed information on the 9999 process.

6. swapon –s

This command will list out all the system swap partitions

7. free

This will show you the amount of used and free memory in terms of straight-up memory, buffers and cache

8. cat /proc/meminfo

This will show you more detailed information on the what the system thinks its memory is doing and/or how its being used.

8. sar –r

This will show you, from another perspective, memory usage defined in terms of memory, buffers and cache

9. ipcs –u

This command will list out all the shared memory status in segments, semaphores and queues.

10. ipcs –p

This will list out all shared memory used by process ID and owner.

11. Finally, check the /etc/sysctl.conf (for shared memory values,etc) and /etc/sysconfig/kernel (for tmpfs/shmfs filesystem size figures).

And, hopefully, that should be enough to get you started and/or, at least, give you some idea if your bottleneck has anything to do with a shortage (or overuse) of memory on your system.

Here’s hoping your vote counts ;)

, Mike

---

Olivier Berger had this excellent suggestion for a GUI tool to add!

---

Hi.

I'd like to suggest that you mention gmemusage, which I find a really
useful tool to visualize what's eating memory if a X display is
available.

Hope this helps.

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Linux And Unix System Security Wrap-Up - Part 4b

Hey again,,

Well, to run down the list once last time, this series of posts has gone through basic Linux and Unix installation security, even more security measures, user, group and account-based security, ensuring existing software's security and looking at additional security software. And now (finally? ;) we're ready to wrap it up by going over additional security software that you may (or may not) want to add to your list of always-installed security-enhancing programs, as well as a few extra thoughts on random things that didn't seem to fit anywhere else.

Be sure to check yesterday's system security post if you want to learn more about encap.

I'd love to be able to say that this is the end, but (like most things in life) hopefully this is just the beginning for many of you. There's always room for improvement ...I'm sounding like my Dad more and more every day ;)

Cheers,

9. Install John The Ripper:

Update /usr/dict/words (or your system's dictionary file) with a better wordfile.
Add “/usr/local/encap/jtr-VERSION/john.sh” (which you can grab from this post on password cracking software) to root's cron.

10. Install tripwire (This might be a bone of contention, since I still have a binary from when they used to give it out free on Solaris. As of my last test on Solaris 10, the last free release for Solaris still works adequately. If you want a copy of that last release, email us and we'll be happy to send it to you. If enough people are interested, we'll put it up here on the site along with instructions on how to get it up and running quickly and securely. It's always been free for Linux):

Copy installation binaries from the install cd, or the packaged version of the public domain software (hereafter referred to as PDSTW in parenthetical instructions).
Edit install.sh in the appropriate directory to set the base directory to /var/TSS (PDSTW - /var/tripwire).
If you’re unaware of the site and local key conventions used, please consult a member of your administration group before creating these (PDSTW - Unnecessary. Security is handled by permissions and ownership).
Create a policy file and configuration file from the examples provided with the distribution. Use other policy and config files for speed of application (PDSTW - Edit tw.config)
Edit your policy file (PDSTW – Edit tw.config) to remove any listed programs that no longer exist on your system.
Install policy and configuration files (PDSTW - Unnecessary).
Initialize the tripwire database (generally with "tripwire --init" You may need to specify -c with the location of the config file, as well).
Run tripwire.
Read the damage report.
Update the tripwire database, if necessary, and rerun until you’ve rectified all problems (PDSTW - Edit tw.config).
Add email functionality to your tripwire policy.
Reinstall the tripwire policy (PDSTW - Edit tw.config).
Add “/var/TSS/bin/tripwire --check --email-report >/dev/null 2>&1” to root's cron (PDSTW - add “/var/tripwire/tw-mailer”, or whatever you decide to call your script, to root cron).

11. Install Your Custom Log Management Scripts (These will, of course, vary, since you'll be putting all your log-rotating/removing scripts here. You wrote them, so, I can't name them specifically ;):

mkdir /var/logs

Copy over the logging encap package to /usr/local/encap.
Add all your log scripts to root's cron.
Add “/usr/bin/find /var/logs/ -type f –mtime +30 –exec rm {} \;” to root's cron, if you want to remove log files after they get a month (give-or-take-a-day) old.

12. Install EGD:

The entire program consists of one file: /usr/local/encap/perl-VERSION/bin/egd.pl
Create a startup script for egd and place it in /etc/init.d. Make appropriate links to it in /etc/rc2.d and /etc/rc0.d. (NOTE: This service must be initialized before SSH in the startup sequence. It should also be killed after SSH in the shutdown sequence. If you are using the OpenSSH/SSH that came with your system, this part doesn't matter)

13. Install OpenSSH (Version 5.0p1 or later):

The package you've put together should include OpenSSH, OpenSSL and zlib. Ensure all are available, or OpenSSH’s programs will not run properly
Review the documentation on creating host-keys.
Create host keys.

EX:
/usr/local/encap/openssh-5.0p1/bin/ssh-keygen -b 1024 -f /usr/local/encap/openssh-5.0p1/ssh/ssh_host_key -N ''; /usr/local/encap/openssh-5.0p1/bin/ssh-keygen -b 1024 -f /usr/local/encap/openssh-5.0p1/ssh/ssh_host_rsa_key -N '';/usr/local/encap/openssh-5.0p1/bin/ssh-keygen -d -f /usr/local/encap/openssh-5.0p1/ssh/ssh_host_dsa_key -N ''

Edit /usr/local/encap/openssh-5.0p1/etc/sshd_config. Set PermitRootLogin no, X11-Forwarding no and PrintMotd yes.
Be sure that all unnecessary r-services that can be replaced with s-services are not still functional on the machine.

14. Minimally setup ntpd or xntpd:

Create a startup script in /etc/init.d and link it to the appropriate place in /etc/rc2.d.
Add “/usr/sbin/ntpdate TIMESERVER >/dev/null 2>&1” to root's cron, unless xntpd has already been set up.

15. Install the tcp-wrappers encap package.

16. Install the wu-ftpd (proftpd, vsftpd, etc) encap package – if necessary.

Change the appropriate line in /etc/inetd.conf to use /usr/local/sbin/in.ftpd –da.
Edit /usr/local/lib/ftpd/ftpaccess to the particular machines specifications.

17. Install The Coroner’s ToolKit encap package.

18. Install the chkrootkit Root Kit and/or worm/trojan checking package of your choice (install two, they're free ;).

19. Install the Top encap package.

20. Install the environment-specific Perl package.

21. Install the lsof encap package.

22. Install the sudo encap package.

Set it up initially to allow no privileges whatsoever. Modify, per the sudo online manual pages, to suit your site requirements.

21. Build and install the Uptime (Big Brother, Nagios, cfenginge or other monitoring software) client for your server. This will most likely be provided to you in pkg format.

22. Install the NetBackup client, or whatever backup software is necessary for the environment your machine exists in and be sure to coordinate your efforts with the administrator of the backup host you will be using.

23. Build and install any additional packages required by your particular environment. Be sure to build them to encapped specifications so that you can manage all software centrally and with greater ease.

24. Remove any unnecessary packages on the machine. This may be a double or triple check for you ;)

25. Edit /etc/vfstab on Solaris (/etc/fstab on RedHat) to mount /usr read only (Only if the machine is not heavily developed upon and /usr is a separate mount point and is not being used to house any files of a volatile nature, such as if /usr/local/ does not have its own mount point).

26. Lock passwords (e.g. passwd –l accountName) for bin, sys, adm, daemon, nobody, nobody4, noaccess and any other system accounts that do not have password locking on by default. This is a double check, again, but during the course of customization things may have been added that weren't on your system initially. Best bet, since the default system accounts differ based on OS version, is to cat /etc/passwd and lock passwords for all accounts with a UID under 100, except for root, of course. On RedHat and Open Solaris, be sure to check above uid 100, too)

27. chmod 4550 /usr/bin/su – generally only on administration-team-owned or production machines – otherwise make sure permissions are 4755.

28. chown root:yourAdminGroup /usr/bin/su – generally only on administration-team-owned or production machines – otherwise make sure ownership is set root:other on Solaris or root:root on RedHat.

29. If running a web server, be sure to include the following lines (Apache specific – use equivalent for Netscape, Tomcat, JBoss, etc) to the main directory directives:

<Limit GET POST OPTIONS PROPFIND>
Order allow,deny
Allow from all
</Limit>
<Limit PUT DELETE PATCH PROPPATCH MKCOL COPY MOVE LOCK UNLOCK>
Order deny,allow
Deny from all
</Limit>
Order allow,deny
Allow from all

30. Create additional init scripts for any products that need to start up at boot time (i.e. Interbase, Uptime, Oracle, NetBackup, etc. This may be obviated by your Jumpstart or Kickstart image but may be necessary (on a per-software-package basis) if doing a manual installation).

31. Reboot the Machine over and over again, while keeping tabs on your console output, until you're satisfied that you've done your best :)

Excellent Job! :)

, Mike

Finding An "Invisible" Proc's Working Directory Without lsof On Linux Or Unix

Ahoy there,

Today, we're going to take a look at something that gets taken for granted a lot these days. lsof (a fine program, to be sure. No debate here) has become a very common staple for finding out information about processes, and where they're hanging out, on most Linux and Unix systems today. Much like the command "top," it provides a simple and robust frontend to having to do a lot of grunt-work to achieve the same results.

I find that, for the most part, lsof is used to find out where a process is, or what filesystems, etc, it's using, in order to troubleshoot issues. One of the most common is the "mysteriously full, yet empty, disk" phenomenon. Every once in a while that will turn out to be an issue where all of the inodes in a partition have been used before all of the blocks have, which produces confusing output in df, leading to the mistaken assumption that there is plenty of space left a device even when there isn't.

However, many times, that empty-yet-full disk is the victim of a process that met an untimely demise and never cleaned up a lot of temporary space in memory (or virtual disk, to split hairs). Another issue that lsof is used for is to find out which dag-nabbed process is holding onto a mount-point that claims it's in use when no one is logged on and no user processes are running that would access it (for instance, a really specific, user-defined, mountpoint like /whereILikeToPutMyStuff - Hopefully the OS isn't depending on this to be around ;) Both problems are, essentially, the same.

However, should you find yourself in a situation where lsof either doesn't come with your Operating System, and/or hasn't been installed, you can still break down these two (and I'm just limiting the post to these two particulars so I don't end up writing an embellished manpage ;) separate issues into one, and find the solution to your problems using the commonly available "pwdx" utility.

pwdx will print out the working directory of any given process (using the process ID as input) at its best. But this is enough to get you to the answer you need.

For instance, we'll take this common scenario: /tmp is reporting 100% full, but df -k shows that /tmp is only at 1% capacity (99% of it is unused). My thinking here almost immediately gravitates toward vi, or some other program that opened up a buffer in memory (using /tmp or /var/tmp), got clipped unexpectedly and never let the system know that it was done with the space it allocated for itself. This would normally not be an issue but, since your Linux or Unix machine "thinks" /tmp is full, whether or not it actually is makes no difference. It won't let you use the free space :(

This command line could be used to figure out what process was using that space in /tmp or /var/tmp:

host # ps -ef|awk '{print $2}'|xargs pwdx 2>&1|grep -iv cannot|grep /tmp
2969: /tmp

Taking it a step further (assuming we trust our own output), we could just skip right to the process in question by adding a bit more to the pipe-chain:

host # ps -ef|awk '{print $2}'|xargs pwdx 2>&1|grep -iv cannot|grep /tmp|sed 's/^\([^:]*\).*$/\1/'|xargs -n1 ps -fp
     UID   PID  PPID   C    STIME TTY         TIME CMD
    root  2969  2966   0   Mar 21 ?           0:05 /bin/vi /home/george/myHumungousFile

Since it's May already, we can fairly assume that this PID is pointing to a dead process (especially since it has no TTY associated with it), and (double-checking, just to be sure) we can probably solve our problem by killing that PID. See our previous post on killing zombie processes if it won't seem to go away and "ps -el" shows it in a Z state.

Yes, that example was pretty simplistic, but the same methodology can be used to find other programs using up other filesystems. Just like "lsof -d," you'll be able to find out what processes are using what filesystems and narrow down your list of suspects, if you don't nail the correct one right away. Since pwdx comes with your Linux or Unix OS, it's actually statistically more likely than lsof to be correct about what process is using what filesystem :)

Cheers,

, Mike

linux unix internet technology

Using Linux strace To Debug Application Issues

Today's post is a little bit of a walkthrough of using RedHat Linux strace to debug (and find the root cause of ) a system issue and a little bit of caution about how much information you should really share with application project managers if you don't want to be stuck supporting a hacked solution for longer than anyone should. As far as job security goes, it can't hurt, but certain situations really do require that folks upgrade their software to the vendor supported version, no matter how creative your solution ends up being.

Note: If you're looking for an in-depth examination of strace, this post isn't it. Not to turn any readers off, as we'll certainly be exploring that in a future post, but this is more of a walkthrough of a problem resolution involving strace rather than a dissertation on the command itself.

In this particular instance, we were working with a product (heretofore referred to as the Product or Product) that, to my knowledge, had just begun having "issues." This generally means to me that somebody did something they weren't supposed to or were trying to do something they weren't supposed to. I'm cynical, but I try to have the common courtesy to keep it to myself ;)

The Product was a client/server application running on the Java platform that had suddenly begun dropping connections from the application server to the backend database. I did a quick check of the /proc/****/status file (and, of course, gave a quick nod to "top" and "netstat -an"), confirming that they were dropping quite a few, like so:

host # cat /proc/14653/status | grep Threads
Threads: 311

host # top <-- Truncated to just show the top process, which was the "failing" one.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14653 user 16 0 1634m 1.1g 40m S 120 14.1 296:46.54 java

I found the "total time used" (listed under the TIME+ column) somewhat alarming since the process had only been up for about a half an hour! It couldn't possibly be correct.

The next thing I did, since I like to go to the guts when I get brought in on an emergency issue regarding a Product I've never used before, was to shut it down and restart it with strace; teeing the information off to a file in /tmp (since I like to watch the characters fly by), like so:

host # strace -vF ./product arguments 2>&1|tee /tmp/OUTPUT

Note that this could have also been accomplished by running the process, getting it's pid from the ps tables and adding the -p PID flag to the strace command in order to attach to the process after starting it. I prefer to start the process with strace, if I can, in case something critical happens during the initial load-up.

Long story short (It is Sunday - Our day of rest - after all ;), I found that the Product process was dumping tons of FUTEX_WAIT errors. At this point, I checked the version of the OS with uname for the only part I really cared about at that point:

x86-64

The machine was running RedHat Advanced Server 5 - 64 bit. The FUTEX_WAIT errors have to do with the implementation of Fast Userspace Mutex locking implemented in the stable version of the RedHat Linux kernel 2.6 and up. This raised a red flag, and caused me to ask the inevitable question: When did this start happening?

As it turns out, the Product had previously been hosted on another machine running RedHat Advanced Server 4, 32 bit. This made perfect sense, since the Mutex locking mechanism at the kernel level they "used" to run on was completely different, and not compatible, since the Product was built to use the old style Mutex locking mechanisms.

Futile attempts were made to add lame fixes, like:

LD_ASSUME_KERNEL=2.4.whatever (Apologies; I don't remember what it was off-hand since it didn't end up helping at all ;)

to the Product's startup scripts, but this didn't resolve the problem. The "actual" (and correct) resolution would have been to upgrade to the vendor-supported version of the Product. The manufacturer of the Product actually had a more recent version available that was built specifically to address this issue and run on RedHat AS 5 - 64 bit.

And, here's the kicker (as per the cautionary message at the beginning of this post): As some of you may know, a bug in strace (It's actually more of a "feature" since it almost "has" to happen) makes it slow down a process slightly due to all the extra work it has to do to parse all the system calls, opens, reads, writes, errors, library loads, etc.

strace slowed the process down enough that the FUTEX_WAIT errors stopped occurring and the Product began functioning normally again. We ended up adding it to the init scripts after much debate over spending a few bucks to avoid possible total confusion in the future.

In retrospect, I'm glad that I was able to help identify the issue and find the root cause. On the other hand, I wish there was some way I could have avoided identifying the problem without providing a cheap and quick "out" for the end user. I'm sure, in the future, someone else will have to work on this Product (maybe a problem with the version of Java or something more insidious) and they'll be given about as much information as I was. Documentation was laid down, but I don't expect anyone will read it. Sometimes, the solution is as bad as the problem, in a way or two ;)

Enjoy your Sunday :)

, Mike

linux unix internet technology

Cleaning Up Solaris Shared Memory

Hey There,

Today, I'm dropping down a little scriplet, and a brief line of code (if it can even be called that ;), to help out with a fairly common headache for most Solaris admins and/or users.

Generally, Solaris' shared memory allocation and cleanup functionality is relatively decent. A lot of times, you may notice that Solaris seems to be using up more than its fair share of memory which makes you worry that you don't have enough (Top and/or prstat output just don't add up).

This, however, is completely normal, and how Solaris deals with memory and swap. Solaris, once it has allocated memory for a specific program or function, will keep that memory in a pseudo-allocated state after it's done with it; the memory appears to be in use, but it actually isn't. In other words, it's keeping a seeming lock on that memory but, as soon as another process pops up that needs it, Solaris makes the memory available.

A good trick for exposing how much memory Solaris is actually using is to stress the system, like so (not always guaranteed to work):

find / -type l -name "*"

If you're dealing with Solaris' memory management model for interprocess communication and shared memory, it does the same thing.

Most of the time, you'll end up having issues because of a piece of software you're running (not blaming the software here, at all). If the software and Solaris don't play nice together, shared memory and semaphores, etc, can become "lost." By lost, I mean that they are actually still accounted for by Solaris but with no attached processes or initialization times. These chunks of IPC memory are lost to you until reboot, unless you use Solaris' IPC utilities to clean up the mess.

The following is a little bit of code to help take care of all that. It requires root privilege to run (of course) and drops shared memory segments and semaphores with no attaching processes and zeroed out start/stop times.

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl
#
# 2007 - Mike Golvach - eggi@comcast.net
#
#Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
#

$temp = $$;

system "/usr/bin/SHARED_MEM -a > /tmp/out.$temp";

open(SHARED_MEM, "</tmp/SHARED_MEM.out");
@SHARED_MEM = <SHARED_MEM>;
close(SHARED_MEM);

foreach $line (@SHARED_MEM) {
        chomp($line);
        @sem_shm = split(" ", $line);
        if ( $line =~ /^s\s/ )  {
                chomp($id = $sem_shm[1]);
                chomp($nattach = $sem_shm[8]);
                chomp($noentry = $sem_shm[9]);
                if ( $nattach == 0 ) {
                        print "Removing Semaphore ID (s) $id with NATTCH value of $nattach\n";
                        system "ipcrm -s $id";
                }       elsif ( $noentry =~ /no-entry/ )        {
                        print "Removing Semaphore ID (s) $id with start/stop-time value of $noentry\n";
                        system "ipcrm -s $id";
                }
        }       elsif ( $line =~ /^m\s/ )       {
                chomp($id = $sem_shm[1]);
                chomp($nattach = $sem_shm[8]);
                chomp($noentry = $sem_shm[13]);
                if ( $noentry =~ /no-entry/ ) {
                        print "Removing Shared Memory Segment ID (s) $id with NATTCH value of $nattach\n";
                        system "ipcrm -m $id";
                }       elsif ( $noentry =~ /no-entry/ )        {
                        print "Removing Shared Memory ID (s) $id with start/stop-time value of $noentry\n";
                        system "ipcrm -m $id";
                }
        }
}

unlink("/tmp/out.$temp");

exit;

, Mike

linux unix internet technology