Broad Usage Of ipcs And ipcrm On Linux And Unix

Hey there,

A wise man once said to me: "Mike, " he said; "If you always remember to keep your mind focused and follow the simple and logical path of progression in every area of your life, you will accomplish many great things." As I pondered his words, he added the following punctuation, which has since become a part of the definition of "me." He added: "Of course, you'll eventually die with the realization that you missed out on 99% of your life and you'll most probably be miserable and bored at the end of both the literal and figurative day. Well before you reach your expiration date your milk will have already turned into cottage cheese. Leave me now and go do something either irrational, contrary or both, but just go. And have fun!" He was probably the most upbeat nihilist I've ever met ;)

And today, to commemorate his entertaining, and often confusing, apocalyptic outlook on life, I'm going to make our post about something that has almost no relation whatsoever to any of our posts so far this week, this month or this year, and attack two commands that vary in options and output on almost every distro of Linux or Unix you'll ever pick up: ipcs (Inter-Process Communication Status) and ipcrm (Inter-Process Communication message queue, semaphore set or shared memory segment Remover). I salute, and dedicate this post to, you, Captain Trips (he never told me his real name. It was more than he could bear ;)

Hopefully this attempt to capture the essence of using ipcs and ipcrm (along with reading the man pages on your system) will make the whole experience easy to get into and fun to customize :) If you're at all interested, we did a post a while back on cleaning up Solaris shared memory that looks at a very specific use of IPC commands on a single Unix distribution.

As a special reminder, since we're covering such broad ground and trying to make this post accessible to users of the widest variety of ipcs and ipcrm versions out there, distro-specific extended options will not be covered in this post!

1. ipcs:

Basic Usage: This command is used to identify currently active (or hung in stasis) message queues, semaphore sets and/or shared memory segments. It best serves as a way to interpret possible system performance problems (commonly referred to as sluggishness ;) and can be very useful. You'll generally want to call it in one of four or five ways (for the most generic purposes):

host # ipcs -q <-- To query the message queue
host # ipcs -m <-- To query shared memory segments
host # ipcs -s <-- To query semaphore sets
host # ipcs -a <-- To query all three (like -qms)
host # ipcs <-- To query all three the same as with "-a". Use this if you're not sure if you have the "-a" option. It should be safer.

General Listing Information by column (Note that these may appear in different forms on your machine. For instance, the type (T) might be listed in a banner across the top of the output and columns like ID might be invocation specific, like shmid for "ipcs -m," etc):

Note: To avoid carpal tunnel syndrome, I'll be referring, from this point on, to messages queues as MQ's, shared memory segments as SHM's and Semaphore Sets as SEM's.

Columns generally shared amongst all object types:

T - The type of listing: q for MQ's, m for SHM's and s for SEM's
ID - The ID number of an MQ, SHM or SEM
KEY - The hexadecimal or, more often, octal key used to identify an MQ, SHM or SEM
MODE - The permissions of the particular MQ, SHM or SEM. These work almost exactly like regular file permissions, with a few exceptions:

1. The first two columns may be either of the following (or they may be backward). Some options may not exist on your version:

Usually column 1:

S : an ipcs object waiting for a msgsnd() call
- : the exact opposite of S if in the same column

Usually column 2:

R : an ipcs object waiting on a msgrcv() call
C : an SHM that will be cleared on next SHM attach operation
D : an SHM that has been removed but won't disappear until the last process attached to it detaches
- : the opposite of any or all of the above, or (in other words) none of the above

Columns 3 through 9: Owner, Group and Other permissions. These may consist of r (read), w (write) and a (alter) permissions. the "-" character indicates that the permission is not granted. Every permissions set (from left to right: Owner, Group and Other) is made of up three columns (r, w and a, with optional "-"'s in any column):

Ex:
SDrwarw-r--

Columns specific to MQ's:

CREATOR: The username or UID that created the listed MQ
CGROUP: The group of the MQ's CREATOR
CBYTES: The number of bytes outstanding in the MQ
QNUM: The number of messages outstanding in the MQ
QBYTES: The maximum number of bytes allowed in messages outstanding in the MQ
LSPID: The last PID to execute a msgsnd call to the MQ
LRPID: The last PID to receive a msgrcv call from the MQ
STIME: The last time a msgsnd call was sent to the MQ
RTIME: The last time a msgrcv call was received from the MQ

Columns specific to SHM's:

CREATOR / SID: The username or UID that created the listed SHM
CGROUP: The group of the SHM's CREATOR
NATTACH: The number of processes attached to the SHM
SEGSZ: The size of the SHM
CPID: The PID of the SHM CREATOR
LPID: The last PID to attach to, or detach from, the SHM
ATIME: The time the last attach operation was executed on the SHM
DTIME: The time the last detach operation was executed on the SHM

Columns specific to SEM's:

CREATOR: The username or UID that created the listed SEM
CGROUP: The group of the SEM's CREATOR
NSEMS: The number of semaphores in the SEM
OTIME: The time the last semaphore operation was completed on the SEM

Other Column headers you may see, that can be easily matched up with their equals in the lists above:

Additional possible MQ columns:

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status

Additional possible SHM columns:

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status

Additional possible SEM columns:

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

2. ipcrm:

Basic Usage: This command is used to remove currently active (or hung in stasis) MQ's, SHM's and/or SEM's. In a perfect world, you'd never have to run this command, but (if you work with IPC a lot) you'll probably use it quite a bit (by which I'm, in no way, implying that this world is anything less than perfect ;) You'll generally want to call it in one of three to six ways (I've found that quite a few distro's do not allow removal by "key," as illustrated below):

These three work on every version I've tested:

host # ipcrm -q MQ_ID <-- To remove an MQ by ID
host # ipcrm -m SHM_ID <-- To remove an SHM by ID
host # ipcrm -s SEM_ID <-- To remove a SEM by ID

These three work on most Linux distro's, but not most Unix that I've tested:

host # ipcrm -Q MQ_KEY <-- To remove an MQ by hexadecimal or Octal KEY
host # ipcrm -M SHM_KEY <-- To remove an SHM by hexadecimal or Octal KEY
host # ipcrm -S SEM_KEY <-- To remove a SEM by hexadecimal or Octal KEY

And that, my friends, concludes the short version of this attempt to reasonably cover every possible widely-portable ipcs and ipcrm option (with result-layouts explained). As I mentioned at the beginning, be sure to check the man pages for your Unix or Linux distro's specific implementation of ipcs and ipcrm. You're liable to find a lot more options and/or how some options vary slightly from what has been posted here (For instance, Solaris 10 - with all the fancy zone functionality - has quite a few options most other OS's don't ;)

Enjoy and have a safe and peaceful morning, afternoon, evening and night :)

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

A Few Ways To Gauge Possible Memory Bottlenecks In SUSE Linux

Hey there,

Today we’re going to take a look at a few (well, maybe more than a few) ways to check your Linux box to find out if the reason it’s beginning to perform poorly is related to memory or memory-management issues. For today’s examples, in particular, we’ll be using SUSE Linux 9.x, but most of these examples translate just as easily to RedHat, Ubuntu and other widely-used distro’s, with some minor modifications.

First things second, I’d like to apologize for the lack of animation in my “voice” in this post. As you probably well know (Even if you don’t live in the USA), we’re (hopefully) going to find out who our next President is by this evening (I write these a day in advance) and I’m somewhat distracted, even though I know I shouldn't start caring until its late enough in the game for any of the numbers to make a legitimate difference. My wife is hanging on the 1% from here and 3% from there ;) I won’t reveal whom I voted for, since this blog doesn’t take any particular political stance. I’ll just let it be said that I’m really hoping my candidate comes in. It’s going to be a farce if the other contender somehow manages to steal this thing….

And, secondly, we’ll walk through the many different (and easily accessible) ways you can check up on your Linux system’s memory usage to make sure that all is well. Note that for some commands, you need root privilege to get information of any value. Most of these commands don’t require that level of access, but (as a rule) if you need to interface with the kernel (or access its symbol/memory tables, etc) you’ll probably just end up with a big old raspberry when you try to run a few of these commands. Not to worry, though: You’ll be able to get more than enough information from this mass of commands to make a reasonably accurate estimation, no matter what level of access you have.

And here’s a quick look at assessing your system’s (possible) memory problem, quickly:

1. top.

This command should be in everyone’s arsenal. If your Linux distro doesn’t include it (as part of a standard pkg) you can always build it from source. It’s been my experience that this tool comes standard with everything nowadays. Pay special attention to the physical and virtual memory sections. top should give you a fairly accurate count of free vs. used memory for both types.

2. ps –aux |grep AprocessYouSuspect

This will provide you with similar information. Pay special attention to the %MEM, VSZ and RSS columns.

3. vmstat 2

This will provide you with memory statistics at 2 second intervals (also the default refresh rate for top).

Watch the following columns (note that the “inact” and “active” columns only show up when you invoke vmstat with the “-a” flag:

swpd: To check on the amount of virtual memory in use
free: To check on the amount of free memory
buff: To check on the amount of memory being set aside for buffering
cache: To check on the amount of memory set aside for caching.
inact: To check on the amount of inactive memory
active: To check on the amount of active memory
si: To check on the amount of memory swapped in from disk
so: To check on the amount of memory swapped out to disk. <-- For more on the difference between swapping, paging, etc, check out our older posts on swapping and Paging on Linux and Unix and its inevitable follow-up

4. ps -o vsz,rss,tsiz,dsiz,majflt,minflt,pmem,cmd 9999

This will format your ps output to spit out virtually all the memory information ps can get you for a specified process ID (In this case: 9999)

5. cat /proc/9999/status

This will provide you with a lot more detailed information on the 9999 process.

6. swapon –s

This command will list out all the system swap partitions

7. free

This will show you the amount of used and free memory in terms of straight-up memory, buffers and cache

8. cat /proc/meminfo

This will show you more detailed information on the what the system thinks its memory is doing and/or how its being used.

8. sar –r

This will show you, from another perspective, memory usage defined in terms of memory, buffers and cache

9. ipcs –u

This command will list out all the shared memory status in segments, semaphores and queues.

10. ipcs –p

This will list out all shared memory used by process ID and owner.

11. Finally, check the /etc/sysctl.conf (for shared memory values,etc) and /etc/sysconfig/kernel (for tmpfs/shmfs filesystem size figures).

And, hopefully, that should be enough to get you started and/or, at least, give you some idea if your bottleneck has anything to do with a shortage (or overuse) of memory on your system.

Here’s hoping your vote counts ;)

, Mike

---

Olivier Berger had this excellent suggestion for a GUI tool to add!

---

Hi.

I'd like to suggest that you mention gmemusage, which I find a really
useful tool to visualize what's eating memory if a X display is
available.

Hope this helps.

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Cleaning Up Solaris Shared Memory

Hey There,

Today, I'm dropping down a little scriplet, and a brief line of code (if it can even be called that ;), to help out with a fairly common headache for most Solaris admins and/or users.

Generally, Solaris' shared memory allocation and cleanup functionality is relatively decent. A lot of times, you may notice that Solaris seems to be using up more than its fair share of memory which makes you worry that you don't have enough (Top and/or prstat output just don't add up).

This, however, is completely normal, and how Solaris deals with memory and swap. Solaris, once it has allocated memory for a specific program or function, will keep that memory in a pseudo-allocated state after it's done with it; the memory appears to be in use, but it actually isn't. In other words, it's keeping a seeming lock on that memory but, as soon as another process pops up that needs it, Solaris makes the memory available.

A good trick for exposing how much memory Solaris is actually using is to stress the system, like so (not always guaranteed to work):

find / -type l -name "*"

If you're dealing with Solaris' memory management model for interprocess communication and shared memory, it does the same thing.

Most of the time, you'll end up having issues because of a piece of software you're running (not blaming the software here, at all). If the software and Solaris don't play nice together, shared memory and semaphores, etc, can become "lost." By lost, I mean that they are actually still accounted for by Solaris but with no attached processes or initialization times. These chunks of IPC memory are lost to you until reboot, unless you use Solaris' IPC utilities to clean up the mess.

The following is a little bit of code to help take care of all that. It requires root privilege to run (of course) and drops shared memory segments and semaphores with no attaching processes and zeroed out start/stop times.

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl
#
# 2007 - Mike Golvach - eggi@comcast.net
#
#Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
#

$temp = $$;

system "/usr/bin/SHARED_MEM -a > /tmp/out.$temp";

open(SHARED_MEM, "</tmp/SHARED_MEM.out");
@SHARED_MEM = <SHARED_MEM>;
close(SHARED_MEM);

foreach $line (@SHARED_MEM) {
        chomp($line);
        @sem_shm = split(" ", $line);
        if ( $line =~ /^s\s/ )  {
                chomp($id = $sem_shm[1]);
                chomp($nattach = $sem_shm[8]);
                chomp($noentry = $sem_shm[9]);
                if ( $nattach == 0 ) {
                        print "Removing Semaphore ID (s) $id with NATTCH value of $nattach\n";
                        system "ipcrm -s $id";
                }       elsif ( $noentry =~ /no-entry/ )        {
                        print "Removing Semaphore ID (s) $id with start/stop-time value of $noentry\n";
                        system "ipcrm -s $id";
                }
        }       elsif ( $line =~ /^m\s/ )       {
                chomp($id = $sem_shm[1]);
                chomp($nattach = $sem_shm[8]);
                chomp($noentry = $sem_shm[13]);
                if ( $noentry =~ /no-entry/ ) {
                        print "Removing Shared Memory Segment ID (s) $id with NATTCH value of $nattach\n";
                        system "ipcrm -m $id";
                }       elsif ( $noentry =~ /no-entry/ )        {
                        print "Removing Shared Memory ID (s) $id with start/stop-time value of $noentry\n";
                        system "ipcrm -m $id";
                }
        }
}

unlink("/tmp/out.$temp");

exit;

, Mike

linux unix internet technology