A Few Ways To Gauge Possible Memory Bottlenecks In SUSE Linux

Hey there,

Today we’re going to take a look at a few (well, maybe more than a few) ways to check your Linux box to find out if the reason it’s beginning to perform poorly is related to memory or memory-management issues. For today’s examples, in particular, we’ll be using SUSE Linux 9.x, but most of these examples translate just as easily to RedHat, Ubuntu and other widely-used distro’s, with some minor modifications.

First things second, I’d like to apologize for the lack of animation in my “voice” in this post. As you probably well know (Even if you don’t live in the USA), we’re (hopefully) going to find out who our next President is by this evening (I write these a day in advance) and I’m somewhat distracted, even though I know I shouldn't start caring until its late enough in the game for any of the numbers to make a legitimate difference. My wife is hanging on the 1% from here and 3% from there ;) I won’t reveal whom I voted for, since this blog doesn’t take any particular political stance. I’ll just let it be said that I’m really hoping my candidate comes in. It’s going to be a farce if the other contender somehow manages to steal this thing….

And, secondly, we’ll walk through the many different (and easily accessible) ways you can check up on your Linux system’s memory usage to make sure that all is well. Note that for some commands, you need root privilege to get information of any value. Most of these commands don’t require that level of access, but (as a rule) if you need to interface with the kernel (or access its symbol/memory tables, etc) you’ll probably just end up with a big old raspberry when you try to run a few of these commands. Not to worry, though: You’ll be able to get more than enough information from this mass of commands to make a reasonably accurate estimation, no matter what level of access you have.

And here’s a quick look at assessing your system’s (possible) memory problem, quickly:

1. top.

This command should be in everyone’s arsenal. If your Linux distro doesn’t include it (as part of a standard pkg) you can always build it from source. It’s been my experience that this tool comes standard with everything nowadays. Pay special attention to the physical and virtual memory sections. top should give you a fairly accurate count of free vs. used memory for both types.

2. ps –aux |grep AprocessYouSuspect

This will provide you with similar information. Pay special attention to the %MEM, VSZ and RSS columns.

3. vmstat 2

This will provide you with memory statistics at 2 second intervals (also the default refresh rate for top).

Watch the following columns (note that the “inact” and “active” columns only show up when you invoke vmstat with the “-a” flag:

swpd: To check on the amount of virtual memory in use
free: To check on the amount of free memory
buff: To check on the amount of memory being set aside for buffering
cache: To check on the amount of memory set aside for caching.
inact: To check on the amount of inactive memory
active: To check on the amount of active memory
si: To check on the amount of memory swapped in from disk
so: To check on the amount of memory swapped out to disk. <-- For more on the difference between swapping, paging, etc, check out our older posts on swapping and Paging on Linux and Unix and its inevitable follow-up

4. ps -o vsz,rss,tsiz,dsiz,majflt,minflt,pmem,cmd 9999

This will format your ps output to spit out virtually all the memory information ps can get you for a specified process ID (In this case: 9999)

5. cat /proc/9999/status

This will provide you with a lot more detailed information on the 9999 process.

6. swapon –s

This command will list out all the system swap partitions

7. free

This will show you the amount of used and free memory in terms of straight-up memory, buffers and cache

8. cat /proc/meminfo

This will show you more detailed information on the what the system thinks its memory is doing and/or how its being used.

8. sar –r

This will show you, from another perspective, memory usage defined in terms of memory, buffers and cache

9. ipcs –u

This command will list out all the shared memory status in segments, semaphores and queues.

10. ipcs –p

This will list out all shared memory used by process ID and owner.

11. Finally, check the /etc/sysctl.conf (for shared memory values,etc) and /etc/sysconfig/kernel (for tmpfs/shmfs filesystem size figures).

And, hopefully, that should be enough to get you started and/or, at least, give you some idea if your bottleneck has anything to do with a shortage (or overuse) of memory on your system.

Here’s hoping your vote counts ;)

, Mike

---

Olivier Berger had this excellent suggestion for a GUI tool to add!

---

Hi.

I'd like to suggest that you mention gmemusage, which I find a really
useful tool to visualize what's eating memory if a X display is
available.

Hope this helps.

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Sar, Vmstat, Free Memory And Swap On Linux or Unix

Hey There,

Today's post will, hopefully, help out folks in the future, since I see questions about this a lot on the boards (and am happy to keep reposting my original answers :) Actually, if you have the time and any of this seems nutty, you might check out an earlier post we did on dealing with simple arithmetic in the shell or our post on floating point arithmetic and percentages if you're feeling frisky ;)

Linux is actually quite a bit better with this diagnostic issue, at least when it comes to reporting free memory, than Unix is. My heaviest bias on Unix is toward Solaris, since I use it the most. I'm not saying that the way they handle it is bad, incorrect or in any other way improper. I'm just saying that, on its face, it can be bit confusing. So we'll tackle both free memory, and free swap, reporting on vmstat and sar, one by one:

1. Free memory: Generally, like I said, on most Linux distro's, "sar -r"'s "kbmemfree" column and vmstat's "memory/free" sections use the same standard of measurement. You can easily compare the two and see that they match, since they're both reported in kilobytes.

Solaris' implementation is somewhat different, however. While vmstat's "memory/free" output is displayed in kilobytes, the output of "sar -r" is in displayed in the "number of pages" (???) Okay, so right off the bat you can be sure that the numbers won't agree. For instance, if vmstat's output showed 4840496 on Solaris, the output from "sar -r" would be 605367 (from a semi-concurrent run of both commands). Those numbers aren't even close, which is good since you'll at least notice that something is off and not necessarily think you have a huge problem ;)

So, now you'll need to calculate how many pages it takes to equal the amount of kilobytes reported by vmstat. First you'll want to run a simple command to find out the default size of memory pages on your Solaris Unix system:

host # pagesize
8192

In this case, the page size is 8192 (which is reported in bytes), which gives us an 8 kilobyte page. Then, we'll take that number and multiply it by the number of pages being reported by "sar -r":

host # let a=605367*8 <--- Since vmstat reports in kilobytes, we want to multiply the number of pages shown in "sar -r" by the number of kilobytes in our default page (8 kilobytes, composed of 8192 bytes)

host # echo $a
4842936

Now the numbers are pretty much the same. A slight skew is probable, since we didn't run the commands at the "exact same time," memory gets thrown around between processes a lot and a kilobyte (especially these days) is a very small unit of measurement. Still, 4840496 and 4842936 are too eerily close enough for this to have been an accident ;) In short; all is well. The numbers agree.

2. Free Swap is a little bit different on most Linux distro's. In vmstat, it shows up under "memory/swpd" which really means the memory that is currently swapped out, not what's free. But that's okay, because it's reported in the same unit of measurement (kilobytes), and in the same way, that it is in "sar -r" under the "kbswpused" column.

Again, you'll need to make allowances for Solaris Unix, but (before you go cursing it up and down the figurative block) remember that the techniques used to derive the free memory above (and the almost-inverse that we're going to do to figure out if the swap numbers are okay) sometimes show up elsewhere, on other Linux and Unix distributions, so this is actually a good thing :) Really...

Anyway. When you check for how much swap you have free on Solaris, you'll notice that the numbers don't match, again! For example, if "vmstat" is showing 8993040 (in kilobytes, again) for "memory/swap," you'll note that "sar -r" is showing the much larger number, 17993811, in its "freeswap" column. Now we have a situation approximately the opposite of when we tried to make the free physical memory numbers agree.

This time the answer's a little simpler. On Solaris, "sar -r" reports free swap in blocks (which are 512 bytes, of course ;) Two of these equal 1024 bytes which is exactly the size of a kilobyte. Simple. Just divide "sar -r"'s number by 2, and you have the size in kilobytes instead of blocks, so 17993811 (divided by 2) becomes 8996905 (kilobytes), which is pretty close to the 8993040 kilobytes reported by vmstat.

Hopefully this has been more helpful than confusing :)

Best wishes,

, Mike

linux unix internet technology

Troubleshooting To Find The Bottleneck On Unix and Linux

Hey there,

Today, we're going to follow up on yesterday's post regarding the definitions of swapping and paging on Linux and Unix, as well as our humble follow up post on clarification of the definitions of paging and swapping with the a tutorial on basic troubleshooting. Today we'll get our primary examples from Solaris Unix and point up the differences, where they exist, in extracting the same information from Linux. Our only really unique bent is that we'll be coming at the issue by considering paging and swapping activity and going from there, assuming no knowledge of what could possibly be causing the problem. All we know is we have a server that's "really slow," which is "not good" ;)

The first thing we'll do is hop on the machine and take a look at vmstat. You'll note that, in each example, I'm zeroing out the values we don't need to look at to make the individual examples easier to read.

This is what we see:

host # vmstat 1 5 <--- We're running vmstat to give us ouput every 1 second 5 times. We've removed the first line from the output below, because it is always a "summary line" (averaging all recorded activity since the last reboot) and can sometimes be misleading.

 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr s6 sd sd --   in   sy   cs us sy id
 - - - - - s u m m a r y l i n e - r e m o v e d - on - p u r p o s e - - - - -
 67 0 0  0     0     0   0  0  0  0  0  0  0  0  0  0    0    0    0  0  0  3
 56 0 0  0     0     0   0  0  0  0  0  0  0  0  0  0    0    0    0  0  0  9
 57 0 0  0     0     0   0  0  0  0  0  0  0  0  0  0    0    0    0  0  0  12
 64 0 0  0     0     0   0  0  0  0  0  0  0  0  0  0    0    0    0  0  0  6

This condition would indicate that we've probably got a problem with our CPU (Remember all the zeroed out values are assumed to be "normal" for this server). The "run queue" is very high (the first column "r"), which indicates that there are an average of approximately 59 processes waiting for CPU execution time at any given second. We couple this with the fact that the CPU "idle time" (the last column "id") is very low, along with the fact that there's no indication of any paging or swapping activity at all (which will almost never happen, really), and it becomes fairly obvious that the bottleneck lies with the CPU.

No special options to vmstat are required to see this information on Linux, but the "id" column is generally second in from the right.

Now, if we change this output slightly, the bottleneck most probably changes to our system's memory:

 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr s6 sd sd --   in   sy   cs us sy id
 - - - - - s u m m a r y l i n e - r e m o v e d - on - p u r p o s e - - - - -
 67 0 0  0     0     12  0  0  0  0  0  13 0  0  0  0    0    0    0  0  0  77
 56 0 0  0     0     15  0  0  0  0  0  12 0  0  0  0    0    0    0  0  0  67
 57 0 0  0     0     10  0  0  0  0  0  14 0  0  0  0    0    0    0  0  0  78
 64 0 0  0     0     8   0  0  0  0  0  11 0  0  0  0    0    0    0  0  0  85

Notice that in this next example, pretty much everything is the same, but the "scan rate" (the "sr" column) and "page reclaim" (the "re" column) values have increased dramatically. Generally, numbers like the ones I'm posting here wouldn't make my pulse change, but, for the sake of argument, we'll assume that the "scan rate" and "page reclaim" rate have been flatlining at 0 ever since this server launched. An increase in the "scan rate" indicates that the system is paging more heavily; that is, it's spending a lot of CPU cycles trying to manage writing from memory to disk and from disk to memory. One might assume that this situation would indicate a problem with the CPU, but the "idle time" doesn't agree with that assumption. Also, it helps to keep in mind that paging occurs more frequently when the system runs out of real physical memory to read from, and write to, and has to revert to using "disk based" virtual memory, which it interacts much more slowly with. This is bolstered by the additional "page reclaim" activity that is going on. Adding physical memory to this server will probably fix the bottleneck.

On Linux, in order to grab comparative "page reclaim" and "scan rate" values, you'll need to take a look at /proc/vmstat, like so:

host # cat /proc/vmstat|grep pgscan
pgscan_kswapd_high <--- These statistics are for the generic "scan rate"
pgscan_kswapd_normal
pgscan_kswapd_dma32
pgscan_kswapd_dma
pgscan_direct_high <--- This is were the generic "page reclaim" statistics begin
pgscan_direct_normal
pgscan_direct_dma32
pgscan_direct_dma

You can also check, specifically, for pages reclaimed by "inode stealing."

host # cat /proc/vmstat|grep pginodesteal

There also may be other variations, depending on the flavor of Linux you're running. Catting /proc/vmstat and doing a:

host # cat /proc/vmstat|egrep 'scan|steal'

should get you most, if not all, of them.

And, in our final permutation, the bottleneck becomes the disk:

kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr s6 sd sd --   in   sy   cs us sy id
 - - - - - s u m m a r y l i n e - r e m o v e d - on - p u r p o s e - - - - -
 0 67 0  0     0     0   0  0  0  0  0  0  33 0  0  0    0    0    0  0  0  77
 0 56 0  0     0     0   0  0  0  0  0  0  98 0  0  0    0    0    0  0  0  67
 0 57 0  0     0     0   0  0  0  0  0  0  89 0  0  0    0    0    0  0  0  78
 0 64 0  0     0     0   0  0  0  0  0  0  94 0  0  0    0    0    0  0  0  85

Again, these numbers aren't crazy, but now we've got an entirely different situation on our hands. We can see, from this example that our only disk (the "s6" column) is now starting to do some heavy writing (yes, I know the numbers aren't really that big. This is all relative ;) This might not be an issue, but the server's "slow." We couple this with a drastic increase in processes that are blocked waiting for I/O (the "b" colomn), along with a lack of significant swapping and/or paging activity, and the disk begins to look like the favorite. The "b" column can generally be considered a very vague indicator of the actual issue, since it reports on processes blocked waiting for resources no matter what they are (the CPU could be slow, the memory could be bad; even the network could be down). However, when we combine this with the fact that disk read/write activity has increased, the disk becomes our most probable bottleneck on the system.

On Linux, using either "vmstat -d" or "vmstat -p ARGUMENT" (with ARGUMENT being a specific partition) will get you the disk statistics.

These examples have been fairly stark. Always keep in mind that, outside of this vacuum, it's always good practice to keep regular tabs on your servers. For instance, you might run vmstat a few times a day (or a few times an hour) and record the results. When you get into habits like that, you're much better prepared when a problem does arise, as you'll have a good "baseline" to refer to when dealing with abnormal behaviour on any of your systems. Everyone will notice the biggest number in the vmstat output when they're frantically trying to figure out what's wrong, but (armed with your "baseline" knowledge) you'll know if that number has always been gigantic, in which case you can ignore it and go on to fix the actual problem :)

Hopefully this little exercise has been helpful to you; if even in the tiniest way :)

Cheers,

, Mike

linux unix internet technology

Further Dissection Of Paging And Swapping On Linux And Unix

Hey again,

Believe it or not, I actually got a few emails about our previous post on paging and swapping in Linux or Unix because it wasn't specific enough ;) While I can certainly understand the frustration, I was hoping to explain the main differences as completely and concisely as possible. I seem to have faltered a bit on each front: I glossed over two specifics which, I agree, deserve some attention and I wrote yet another novel ;)

With that in mind (and with a prayer that my fingers won't type any more than they have to ;) I'd like to address, and/or clarify, the level of depth I didn't descend to in my last post on the difference between paging and swapping and write about the difference between paging and swapping (The redundancy was intentional and any resulting confusion is expected, given the topic at hand and my writing style ;)

As I mentioned previously, the terms paging and swapping are used almost interchangeably these days. Some industry manuals will actually talk about "swapping out pages" which seems to be contradictory and, theoretically, impossible if swapping and paging are two separate concepts with distinct and unique definitions. This is where language and implied meaning become a barrier to actual definition. And, all the more reason to clarify this one last bit of the puzzle.

And here they come. The extra clarifications...

1.  Difference in resident virtual memory management with paging and swapping.

When a system swaps a program, or process, it guarantees that it is resident (on disk or in memory) before it schedules it for execution, and will often hold onto the mapped resources reserved for that process from the time the process requests them until it notifies the scheduler that it is complete. When a system pages during the execution of a program, or process, there isn't any such direct correlation. You don't necessarily know (without specifically checking) how much of a process's virtual memory is resident or whether the process is entirely able to be scheduled for execution at the time paging begins. Pages can be selectively grabbed from a process (out of mapped physical memory) and never returned, unless re-requested by the process when it looks for the memory, can't find it and generates a page fault.

Phew... That's one down. Hopefully this isn't just becoming more confusing :)

2.  More specific definition of paging and swapping with regard to page ins/outs and swap ins/outs.

In this instance, swapping specifically refers only to the transfer of memory pages from physical memory to dedicated swap devices or swap disk (on most systems this is now referred to as swapfs - or a unique swap filesystem) and vice versa. Paging, on the other hand, refers to the transfer of memory pages from physical memory to disk (regular disk or swap disk) and vice versa. So, really, the major difference is that swapping is limited to only transferring memory pages back and forth between the physical memory and a dedicated swap device or filesystem, while paging can transfer between physical memory and any sort of disk device.

Hopefully, we've reached a sufficient amount of explanation at this point, and this thing won't turn into the monster I'd hoped it wouldn't become ;)

Thank you, everyone who wrote in, for your helpful input. As many folks have also noted, we don't have comments set up on this blog (because of issues with "comment spam" and not wanting to get shut down). If you ever want to leave a comment, or an objection, we welcome you to email us directly at our most often-check email address or, if you have a lot to say (or want to attach video, etc), sign up (for free) on our sister Linux and Unix Menagerie Forum, and we'll get your remarks there, as well. We do our best to reply, personally, to everyone who takes the time to write us. So far, I think we're still batting a thousand in that regard ;)

Best wishes,

, Mike

linux unix internet technology

Swapping Or Paging On Linux And Unix?

Howdy,

Here's a question that gets asked a lot, and has a relatively simple answer to go with it: On Unix and/or Linux, what's the difference between paging and swapping?

It's a relevant question, given that the terms are used almost interchangeably these days. Even in most Linux or Unix monitoring commands, the issue can become confused. Consider our previous posts on free memory graphing on Unix and graphing out paging statistics on Linux. They're both showing approximately the same thing, but one of them is using the terminology in a not-totally-correct sense.

The good news is you only need to understand one thing about each (which is also a common thread) in order to understand what the terms "really" mean. This can be a great help when you're trying to determine the cause of a system issue, like a big slow-down. Of course, since the terms are mixed up a lot, it's a good rule of thumb to assume that any problem with "paging" or "swapping" may be a problem with either. Depending upon who's asking, they could mean one thing or the other. As in public speaking, it's always a good idea to know your audience ;)

The main difference between paging and swapping (on both Linux and Unix; all flavors, as far as I know) is this:

1. Swapping: This occurs when an entire process ( sometimes consisting of multiple parts like a read-only text segment, writable data segment and, more often nowadays, writable stack segment ) gets transferred to disk from physical memory or is read back into physical memory from the disk.

2. Paging: This occurs when part of a process ( a page, or a segment, of a process ) gets transferred to disk from physical memory or is read back into physical memory from disk. Paging also requires a MMU (Memory Management Unit) and a CPU capable of handling requests from it. This is just a side note, and slightly outside the scope of the definition. It really doesn't even make a difference any more since I haven't seen an OS without paging capability in years, and most dedicated Unix/Linux servers have had the latent capability for even longer.

Tomorrow, we'll begin looking at a real-life examples of determining a system issue highlighted by excessive paging (or is it swapping?). For today, we'll keep it abstract.

To wrap up, on today's system's (The year now being 2008 - Just dating this in case it gets read 2 years from now and I'm totally off-base by then ;) there's almost no such thing as swapping. Paging occurs normally and, if you do see actual heavy swapping, it's generally an indication of a problem with memory or disk (Except in situations where you have large applications - like an Oracle database, for instance - that hoard lots of Virtual Memory Address (VMA) space and cause the system to swap naturally). In somewhat contrast, if your system is paging heavily, but not swapping, your issue is most likely with CPU or memory. Memory is often mistakenly assumed to be the culprit in most situations because both swapping and paging involve writing to, and reading from, memory. However, it should always be taken into account what other component of the OS is doing the work to make that activity possible, or maybe even necessary.

One last thing to remember is that either of these situations ( excessive swapping, excessive paging or both ) could be indicators of either memory, CPU or disk issues. They could also point to a problem with your network subsystem or any number of things. The generic explanations/answers in the previous paragraph assume a relative norm. In reality, you have to look at the situation in the context of the problem you're facing on the system that's having the issue and work from there.

We'll run down some quick and easy real-life troubleshooting starting tomorrow.

Until then, best wishes :)

, Mike

linux unix internet technology

Shell Script To Report Linux Server Hardware Information

Please click above for a slightly larger view of the beginning of the output today's script provides :)

Hey There,

Well, I guess it's about time we starting putting some more shell scripts out there. The last 3 or 4 posts have all been how-to's (except the last one, which I suppose you could trim all the surrounding text and make a script out of ;) and it's high time to start hitting the shell again.

Today's offering is something we cooked up to tiptoe the fine-line between producing what a manager wants to see and what an administrator wants to see in a quick system profile. This has been tested on RedHat Linux and SUSE (only up to release 9.x). The only major difference is some extra output in the "SERVER - MEMORY" section (mostly when run on x86_64 architecture machines) that some of you may find useful.

If you're interested in something more basic, or generic, check out our previous posts on gathering system information on Solaris and gathering system information on RedHat Linux.

This is a pretty straightforward shell script offering that basically parses the output of the hwinfo command. We run it in "--short" mode for most options, but leave it long for parts where the shortening process removed vital information (Like the brand name of the server). It's formatted loosely, but is fairly easy to read. One of the things I like most about it (and the main reason I started writing it in the first place) is that it highlights the Manufacturer, Model and Serial number of the machine your Linux OS is running on. This generally isn't an issue when you're, say, running Solaris on your Sun box ;) Then, of course, I couldn't get away from putting in all the basic information about CPU, Memory, Disks, etc.

If you want to know more about your system than this little shell script will show you, the hwinfo command has a variety of options I chose not to include (Neither my manager nor I want to know about every little "debug" detail of the PCI controller unless we have to ;), but you can access just about any hardware related information using that command. Just run it as:

host # hwinfo --help

Assuming, of course, that you've run this script already as:

host # ./server_info.sh

and found it lacking.

If hwinfo isn't available on your machine (Oh, yes. Be sure you're "root" when you run this or you might not have the access required to pull some of the information hwinfo tries to get for you!), there are a number of other options available to you, both on SUSE, RedHat and different flavors of Linux. Off the top of my head, you can always give these commands a shot (assuming they exist ;) --> kudzu, lspci, lsusb, dmidecode and a great project (which even has a GUI now) called lshw. You should check that out if you or your manager dig this little shell script :)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# server_info.sh - display server hardware info
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
# 

hwinfo="/usr/sbin/hwinfo --short"
hostname=`hostname`
separator="----------------------------------------"
echo $separator
echo "System Information For $hostname"
echo $separator
echo $separator
echo SERVER - MEMORY
echo $separator
/usr/sbin/hwinfo --bios|egrep 'OEM id:|Product id:|CPUs|Product:|Serial:|Physical Memory Array:|Max. Size:|Memory Device:|Location:|Size:|Speed:|Location:'|sed -e 's/"//g' -e '/^ *Speed: */s/Memory Device:/\n  Memory Device:/' -e 's/\(Max. Speed:\)/CPU \1 MHz/' -e 's/\(Current Speed\)/CPU \1 MHz/'
echo $separator
echo SMP
echo $separator
$hwinfo --smp
echo $separator
echo CPU
echo $separator
$hwinfo --cpu
echo $separator
echo CD_ROM
echo $separator
/usr/sbin/hwinfo --cdrom|egrep '24:|Device File:|Driver:'|awk -F":" '{ if ( $1 ~ /[0-9][0-9]*/ ) print $0; else print "  " $2}'|sed -e 's/^.*[0-9] //' -e 's/ //' -e 's/"//g'
echo $separator
echo DISK
echo $separator
$hwinfo --disk
echo $separator
echo PARTITION
echo $separator
$hwinfo --partition
echo $separator
echo NETWORK
echo $separator
$hwinfo --network
echo $separator
echo NETCARD
echo $separator
$hwinfo --netcard
echo $separator

, Mike

linux unix internet technology

Linux Perl Script To Graph Out Paging, Memory and CPU

Hey there,

Yesterday's post has finally arrived. I've converted as much as I could from our previous posts into this one comprehensive script for RedHat Linux. I'm not going to overstuff this post with pictures, since the results you'll get here will look exactly like the ones in our previous posts for CPU, Paging and Memory graphing, except our Perl script today will create all 3 at once!

You may notice that the disk buffer reads and writes Graph has not been included here. This is on purpose and due to the fact that I couldn't find much more than opinion about what constituted the equivalent of Solaris cache reads/writes on Linux. The native sar didn't produce the exact same type of statistics, so I didn't want to throw out some garbage just for the sake of filling the page. As we look into this further, it seems that an entire post on how to determine and track Linux's buffer cache would be helpful, informative and quite lengthy all on it's own. I look forward to scripting that out in the near future :)

Today's script was put through its paces on a machine running RedHat AS2.1 (Pensacola) with sar from the sysstat RPM version 5.0.5-12.21as. I mention these specifics because its been my experience, in the past (when RedHat was free) that the column order would occasionally change when the sysstat RPM was updated. With that in mind, I've commented this script with the header notation for the column that I'm grabbing at each point in the script. This way, if it changes for you, all you'll need to do is change the array number to the one matching the data column we're graphing to keep this script pertinent.

These column notation comments (and all the other lines that varied enough from the Solaris script to necessitate changing) are noted in the script with:

# <-----

Again, this script can be run at any time, simply by calling it, since it interprets the sar data and creates the GIF Graphs when you run it, like so:

./linuxcpupagememgraph.pl <--- Worst script name ever? ;)

Over the weekend, we'll post on using these sorts of scripts in CGI and the security implications that go along with automating processes through the web browser (convenient, but sometimes dangerous!).

I hope you find this script helpful and flexible enough to use across however many different versions or flavors of Linux you run at your site.

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# linuxcpupagememgraph.pl
# Graph CPU, Paging and Memory
# statistics at will!
# Written specifically for Linux
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

&graphsarB; # sar -g on Solaris # <-----
&graphsarr;
&graphsaru;

sub graphsarB { # <-----
use GIFgraph::bars;
open(SARG, "/usr/bin/sar -B|"); # <-----
@sarg = <SARG>;
close(SARG);
$my_graph = new GIFgraph::bars(800,600);
$my_graph->set( 
 x_label => 'Time',
 y1_label => 'Number',
 y2_label => 'Number',
 title => 'Paging Activity by Server - host',
 two_axes => 1,
 y_min_value => 0,
 y1_max_value => 20,
 y2_max_value => 20,
 y_tick_number => 5,
 long_ticks => 1,
 x_ticks => 0,
 legend_marker_width => 24,
 line_width => 10,
 bar_spacing => 0,
 gifx => 800,
 gify => 600,
 transparent => 1,
 dclrs => [ qw( red green blue black) ],
);
$my_graph->set_legend( qw( Pages_Paged_Out Minor_Page_Faults-Buffer_Access Major_Page_Faults ) ); # <-----
$a = $b = $c = $d = 0;
foreach $line (@sarg) {
 next if ( ($line !~ /:/) || ($line =~ /\//));
 @line=split(" ", $line);
 if ( ( $a % 12 ) != 0 ) {
  $pandata0[$a] = undef;
 } else {
  $line[0] =~ s/:00$//;
  $pandata0[$a] = "$line[0] $line[1]"; # <-----
 }
 if ( $line[3] == 0 ) { # <-----
  $pandata1[$b] = 0;
 } else {
  $pandata1[$b] = $line[3]; # <----- pgpgout/s
 }
 if ( $line[3] == 0 ) { # <-----
  $pandata2[$c] = 0;
 } else {
  $pandata2[$c] = $line[4]; # <----- fault/s
 }
 if ( $line[3] == 0 ) { # <-----
  $pandata3[$d] = 0;
 } else {
  $pandata3[$d] = $line[5]; # <----- majflt/s
 }
 $a++;
 $b++;
 $c++;
 $d++;
}
if ( ! $c ) {
 @pandata0[0] = `date "+%H:%M"`;
 @pandata1[0] = 0;
 @pandata2[0] = 0;
 @pandata3[0] = 0;
}
@data = (\@pandata0, \@pandata1, \@pandata2, \@pandata3);
$my_graph->plot_to_gif( "/your/data/dir/lsarB.gif", \@data ); # <-----
}

sub graphsarr {
use GIFgraph::lines;
open(SARR, "/usr/bin/sar -r|"); # <-----
@sarr = <SARR>;
close(SARR);
$my_graph = new GIFgraph::lines(800,600);
$my_graph->set( 
 x_label => 'Time',
 y1_label => 'MegaBytes',
 y2_label => 'MegaBytes',
 title => 'Free Memory By Server - host',
 y_min_value => 0,
 y_tick_number => 5,
 long_ticks => 1,
 x_ticks => 0,
 legend_marker_width => 24,
 line_width => 10,
 bar_spacing => 0,
 gifx => 800,
 gify => 600,
 transparent => 1,
 dclrs => [ qw( red green blue ) ],
);
$my_graph->set_legend( qw( Real_Memory_Available Swap_Memory_Available ) ); # <-----
$a = $b = $c = 0;
foreach $line (@sarr) {
 next if ( ($line !~ /:/) || ($line =~ /\//));
 @line=split(" ", $line);
 if ( ( $a % 12 ) != 0 ) {
  $pandata0[$a] = undef;
 } else {
  $line[0] =~ s/:00$//;
  $pandata0[$a] = "$line[0] $line[1]"; # <-----
 }
 $pandata1[$b] = ($line[2]*8)/1000; # <----- kbmemfree
 $pandata2[$c] = ($line[7]/2)/1000; # <----- kbswpfree
 $a++;
 $b++;
 $c++;
}
if ( ! $c ) {
 @pandata0[0] = `date "+%H:%M"`;
 @pandata1[1] = 0;
 @pandata2[2] = 0;
}
@data = (\@pandata0, \@pandata1, \@pandata2);
$my_graph->plot_to_gif( "/your/data/dir/lsarr.gif", \@data ); # <-----
}

sub graphsaru {
use GIFgraph::lines;
open(SARU, "/usr/bin/sar -u|"); # <-----
@saru = <SARU>;
close(SARU);
$my_graph = new GIFgraph::lines(800,600);
$my_graph->set( 
 x_label => 'Time',
 y1_label => 'Percentage',
 y2_label => 'Percentage',
 title => 'CPU Usage By Server - host',
 y1_max_value => 100,
 y2_max_value => 100,
 y_min_value => 0,
 y_tick_number => 5,
 long_ticks => 1,
 x_ticks => 0,
 legend_marker_width => 24,
 line_width => 10,
 bar_spacing => 0,
 gifx => 800,
 gify => 600,
 transparent => 1,
 dclrs => [ qw( red green ) ],
);
$my_graph->set_legend( qw( CPU_Utilization ) );
$a = $b = 0;
foreach $line (@saru) {
 next if ( ($line !~ /:/) || ($line =~ /\//));
 @line=split(" ", $line);
 if ( ( $a % 12 ) != 0 ) {
  $pandata0[$a] = undef;
 } else {
  $line[0] =~ s/:00$//;
  $pandata0[$a] = "$line[0] $line[1]"; # <-----
 }
 $pandata1[$b] = 100-$line[7]; # <----- %idle
 $a++;
 $b++;
}
if ( ! $c ) {
 @pandata0[0] = `date "+%H:%M"`;
 @pandata1[1] = 0;
}
@data = (\@pandata0, \@pandata1);
$my_graph->plot_to_gif( "/your/data/dir/lsaru.gif", \@data ); # <-----
}

, Mike

linux unix internet technology

Free Memory Graph Generation With Perl And Sar

Note: Click the above picture to see it at 800x600.

Hey There,

For today's post, we're continuing from yesterday, but moving on to scripting out a graphological representation of the free memory usage on a server. Over the next few days I'll be posting the only other two variations I find that people really want to see all the time (and by people, I mean my managers). Then we'll go over setting them all up to run interactively in CGI so "people" will never bother you again. At least, not for that ;)

We'll also take a quick look, in retrospect, on how to "fix" all four scripts to run using Linux sar from the sysstat RPM. Maybe we'll just make this a graph generation script week. It won't compete with the Oscars, but it might resemble the closest thing we've ever come to a theme amongst a group of consecutive posts on this blog ;)

The script below is, again, written in Perl and will, run on Solaris Unix and RedHat Linux. The script was written specifically to run on Solaris and the output from "sar -r" on RedHat is in slightly different order (so the array elements I use to seed the Graph values in this script won't translate without some modification).

You can run it easily by just invoking it (as it is, it should be run on the host it's measuring, since it runs sar at the time you invoke it):

./sarmemgraph.pl

Note that there are some significant differences in this graph, compared to yesterday's post, including the use of multiple y values. Yay :)

Again, click on the picture above to see it at its actual resolution and enjoy :)

Enjoy,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# sarmemgraph.pl
# Graph Free Memory in Pictorial Form
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/us/">Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License</a>
#

use GIFgraph::lines;
open(SARR, "/usr/sbin/sar -r|");
@sarr = <SARR>;
close(SARR);
$my_graph = new GIFgraph::lines(800,600);
$my_graph->set( 
 x_label => 'Time',
 y1_label => 'MegaBytes',
 y2_label => 'MegaBytes',
 title => 'Free Memory By Server - host',
 y_min_value => 0,
 y_tick_number => 5,
 long_ticks => 1,
 x_ticks => 0,
 legend_marker_width => 24,
 line_width => 10,
 bar_spacing => 0,
 gifx => 800,
 gify => 600,
 transparent => 1,
 dclrs => [ qw( red green blue ) ],
);
$my_graph->set_legend( qw( Real_Memory_Pages_Available Swap_Memory_Available ) );
$a = $b = $c = 0;
foreach $line (@sarr) {
 next if ( ($line !~ /:/) || ($line =~ /\//));
 @line=split(" ", $line);
 if ( ( $a % 12 ) != 0 ) {
  $pandata0[$a] = undef;
 } else {
  $line[0] =~ s/:00$//;
  $pandata0[$a] = $line[0];
 }
 $pandata1[$b] = ($line[1]*8)/1000;
 $pandata2[$c] = ($line[2]/2)/1000;
 $a++;
 $b++;
 $c++;
}
if ( ! $c ) {
 @pandata0[0] = `date "+%H:%M"`;
 @pandata1[1] = 0;
 @pandata2[2] = 0;
}
@data = (\@pandata0, \@pandata1, \@pandata2);
$my_graph->plot_to_gif( "/my/stats/directory/sarr.gif", \@data );

, Mike

linux unix internet technology

Cleaning Up Solaris Shared Memory

Hey There,

Today, I'm dropping down a little scriplet, and a brief line of code (if it can even be called that ;), to help out with a fairly common headache for most Solaris admins and/or users.

Generally, Solaris' shared memory allocation and cleanup functionality is relatively decent. A lot of times, you may notice that Solaris seems to be using up more than its fair share of memory which makes you worry that you don't have enough (Top and/or prstat output just don't add up).

This, however, is completely normal, and how Solaris deals with memory and swap. Solaris, once it has allocated memory for a specific program or function, will keep that memory in a pseudo-allocated state after it's done with it; the memory appears to be in use, but it actually isn't. In other words, it's keeping a seeming lock on that memory but, as soon as another process pops up that needs it, Solaris makes the memory available.

A good trick for exposing how much memory Solaris is actually using is to stress the system, like so (not always guaranteed to work):

find / -type l -name "*"

If you're dealing with Solaris' memory management model for interprocess communication and shared memory, it does the same thing.

Most of the time, you'll end up having issues because of a piece of software you're running (not blaming the software here, at all). If the software and Solaris don't play nice together, shared memory and semaphores, etc, can become "lost." By lost, I mean that they are actually still accounted for by Solaris but with no attached processes or initialization times. These chunks of IPC memory are lost to you until reboot, unless you use Solaris' IPC utilities to clean up the mess.

The following is a little bit of code to help take care of all that. It requires root privilege to run (of course) and drops shared memory segments and semaphores with no attaching processes and zeroed out start/stop times.

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl
#
# 2007 - Mike Golvach - eggi@comcast.net
#
#Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
#

$temp = $$;

system "/usr/bin/SHARED_MEM -a > /tmp/out.$temp";

open(SHARED_MEM, "</tmp/SHARED_MEM.out");
@SHARED_MEM = <SHARED_MEM>;
close(SHARED_MEM);

foreach $line (@SHARED_MEM) {
        chomp($line);
        @sem_shm = split(" ", $line);
        if ( $line =~ /^s\s/ )  {
                chomp($id = $sem_shm[1]);
                chomp($nattach = $sem_shm[8]);
                chomp($noentry = $sem_shm[9]);
                if ( $nattach == 0 ) {
                        print "Removing Semaphore ID (s) $id with NATTCH value of $nattach\n";
                        system "ipcrm -s $id";
                }       elsif ( $noentry =~ /no-entry/ )        {
                        print "Removing Semaphore ID (s) $id with start/stop-time value of $noentry\n";
                        system "ipcrm -s $id";
                }
        }       elsif ( $line =~ /^m\s/ )       {
                chomp($id = $sem_shm[1]);
                chomp($nattach = $sem_shm[8]);
                chomp($noentry = $sem_shm[13]);
                if ( $noentry =~ /no-entry/ ) {
                        print "Removing Shared Memory Segment ID (s) $id with NATTCH value of $nattach\n";
                        system "ipcrm -m $id";
                }       elsif ( $noentry =~ /no-entry/ )        {
                        print "Removing Shared Memory ID (s) $id with start/stop-time value of $noentry\n";
                        system "ipcrm -m $id";
                }
        }
}

unlink("/tmp/out.$temp");

exit;

, Mike

linux unix internet technology

The Shebang Line: An Introduction to Porting Shell to Perl

This is actually going to be, over time, a series of posts because it would be impossible (especially with my writing style) to cover every aspect of porting shell script to Perl in one posting (There may be some unusual or vague aspects that won't be covered at all, but, by that point, you will understand enough to be able to find the answers through trial and error, or judicious use of various search engines ;)

In this post we're going to cover one of the very basics. By "very basic," I mean a brief discourse on the "shebang" line. There's so much to mine in that seemingly obvious topic that it'll take far too long for me to crank out anything more today. After this, in a future post, we'll follow up with how to properly format a standard line of code, the three common variable types and how they are created and accessed in either language.

The beginning is complicated enough, when you really look at it. The "shebang" line (It's referred to as this because the exclamation point is referred as a "bang," for reasons which I don't completely understand, and the pound symbol is... I honestly can't come up with any intelligent reason for the other part of the name ;). I still call it the "pound bang" line.

In shell and in Perl, the "shebang" lines serve the same purpose. In fact, the "shebang" line is an entity unto itself. Its use extends beyond shell and/or Perl scripting and is a construct used for many purposes. The main thing the "shebang" line does is instruct the shell that you're using (you can be logged into csh, tcsh, ksh, sh, zsh or whatever shell you like to work in) to invoke a specific "command" to initiate the script with. It is customary to make this the full path to the shell in which you are writing your script, or the full path to the Perl binary, but it doesn't have to be (although, what you are asking the "shebang" to execute should be directly connected to it. No space between "#!" and "/bin/sh." Lots of shells nowadays will consume this blank space for you. Some older versions of shells are far less forgiving). For instance, our two basic examples are these:

#!/bin/sh <--- Tells the shell's interpreter to invoke "/bin/sh" in a subshell and run all the commands in the script in that subshell.
#!/usr/bin/perl <--- Tells the shell's interpreter to invoke "/usr/bin/perl" in a subshell and run all the commands in the script in that subshell.

This doesn't mean you can only use one shell or even one version of Perl within the same script, but that's outside the scope of today's post. So, to illustrate what I mean when I say the "shebang" line instructs the shell to invoke a specific "command," I mean just that. If you wrote a script, in the editor of your choice, and made the first line:

#!/bin/rm

the default action of the shell would be to delete the file and then run all the following commands (reading the script - in both shell and Perl - left to right, from top to bottom) until the script ended. If this seems counterintuitive, which it did to me at one point, that's normal. This is a good example of what goes on "under the covers" when you execute a script in your Unix or Linux shell. In the above example (as in the previous two, and all others), the entire script is read into memory before the "shebang" line is executed. That way, when your script with "#!/bin/rm" at the top is executed, the following happens:

1. The script is read into memory.
2. The script is executed.
3. The first line ("shebang") instructs the script to delete itself.
4. The rest of the script is run out from memory.
5. When the script is done, everything you instructed the script to do has been completed (provided it's at all possible in the shell you're executing it from (since the default subshell will be a subshell of the shell you're in when you execute the script) and the script is no longer there! In actuality, the script disappears before the code is executed from memory, so your script is gone, sometimes, long before your script finishes! For example, let's look at the following script:


#!/bin/rm
# /export/home/user/MyScript.sh c) 2007 MT - xyz.com
#
# Pounds are considered comment lines in both shell and Perl
# You can include interesting notes or whatever you like behind
# the pound sign and the script won't execute it. Even the "bang"
# ! has no special significance after the first line of the script
# except in special situations where you're invoking another
# subshell or another "shebang."
# Blank lines are also not acted upon in either shell or Perl.

sleep 500000000000000000

Now, if we were to run that script at the command line (after doing a chmod to 700 so that we, the user, have full read, write and execute permissions on it) we would end up sitting at a hung prompt, waiting for the sleep command to finish for a very long time. In this case that's good, because it gives us ample opportunity to login to the same machine again, on a different terminal, and have a look at what's going on:

This is an example of what our first terminal session would look like:

user# cd /export/home/user
user# ls -a
. .. .profile MyScript.sh
user# /bin/bash ./MyScript.sh
user#

Now, if we log on to the machine again, using a different terminal (or another connection, window; however you want to put it) we'd see this:

user# cd /export/home/user
user# ls -a
. .. .profile
user#

Yet - we can check that the script is still running, which it is:

user# ps -ef|grep "[M]yScript.sh
user# user 6523 6521 0 15:12:03 ? 0:14 ./MyScript.sh

And thus is all that explained. You can use the "shebang" line to run any command you want. Try it out. Be careful, because you can do some incredibly awful things if you're not careful. Luckily, the shell will protect you against doing something like this (but the way it reacts is specific to the command invoked, so some programs, like Perl, can accept a switch after the command line, like - #!/usr/bin/perl -w):

#!/bin/rm -rf /export/home/user

as it only executes the full qualified "rm" command directly attached to the "shebang." If you quote the entire command to try and get around it that way, it will try to interpret the entire line as one command and shoot you an error like:

ksh ./MyScript.sh: not found

Hopefully, this has been a somewhat informative and easy to understand introduction to the relationship between the shell and Perl. I understand that this bedrock we've discussed today isn't really specific to either of them, although it applies to both. But, as an old saying goes: It's usually best to start with the first floor (or the basement) when you're putting up a building ;)

Best Wishes,

, Mike

---

Thanks to Michael Pelletier, Merrimack NH for this perfect explanation of what that pound symbol means!

---

It's a "sharp" symbol in music notation:

http://en.wikipedia.org/wiki/Sharp_(music)

That's why the "C#" language is pronounced "C sharp," not "C plus plus plus plus:"

http://en.wikipedia.org/wiki/C_Sharp_(programming_language)

Enjoy this little tidbit of mostly useless knowledge. I pronounced it "pound bang" for maybe 20 years.

---

linux unix internet technology