Hey there,
Today, we're going to follow up on yesterday's post regarding the definitions of swapping and paging on Linux and Unix, as well as our humble follow up post on clarification of the definitions of paging and swapping with the a tutorial on basic troubleshooting. Today we'll get our primary examples from Solaris Unix and point up the differences, where they exist, in extracting the same information from Linux. Our only really unique bent is that we'll be coming at the issue by considering paging and swapping activity and going from there, assuming no knowledge of what could possibly be causing the problem. All we know is we have a server that's "really slow," which is "not good" ;)
The first thing we'll do is hop on the machine and take a look at vmstat. You'll note that, in each example, I'm zeroing out the values we don't need to look at to make the individual examples easier to read.
This is what we see:
host # vmstat 1 5 <--- We're running vmstat to give us ouput every 1 second 5 times. We've removed the first line from the output below, because it is always a "summary line" (averaging all recorded activity since the last reboot) and can sometimes be misleading. kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s6 sd sd -- in sy cs us sy id
- - - - - s u m m a r y l i n e - r e m o v e d - on - p u r p o s e - - - - -
67 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3
56 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9
57 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 12
64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6
This condition would indicate that we've probably got a problem with our CPU (Remember all the zeroed out values are assumed to be "normal" for this server). The "run queue" is very high (the first column "r"), which indicates that there are an average of approximately 59 processes waiting for CPU execution time at any given second. We couple this with the fact that the CPU "idle time" (the last column "id") is very low, along with the fact that there's no indication of any paging or swapping activity at all (which will almost never happen, really), and it becomes fairly obvious that the bottleneck lies with the CPU.
No special options to vmstat are required to see this information on Linux, but the "id" column is generally second in from the right.
Now, if we change this output slightly, the bottleneck most probably changes to our system's memory: kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s6 sd sd -- in sy cs us sy id
- - - - - s u m m a r y l i n e - r e m o v e d - on - p u r p o s e - - - - -
67 0 0 0 0 12 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 77
56 0 0 0 0 15 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 67
57 0 0 0 0 10 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 78
64 0 0 0 0 8 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 85
Notice that in this next example, pretty much everything is the same, but the "scan rate" (the "sr" column) and "page reclaim" (the "re" column) values have increased dramatically. Generally, numbers like the ones I'm posting here wouldn't make my pulse change, but, for the sake of argument, we'll assume that the "scan rate" and "page reclaim" rate have been flatlining at 0 ever since this server launched. An increase in the "scan rate" indicates that the system is paging more heavily; that is, it's spending a lot of CPU cycles trying to manage writing from memory to disk and from disk to memory. One might assume that this situation would indicate a problem with the CPU, but the "idle time" doesn't agree with that assumption. Also, it helps to keep in mind that paging occurs more frequently when the system runs out of real physical memory to read from, and write to, and has to revert to using "disk based" virtual memory, which it interacts much more slowly with. This is bolstered by the additional "page reclaim" activity that is going on. Adding physical memory to this server will probably fix the bottleneck.
On Linux, in order to grab comparative "page reclaim" and "scan rate" values, you'll need to take a look at /proc/vmstat, like so:
host # cat /proc/vmstat|grep pgscan
pgscan_kswapd_high <--- These statistics are for the generic "scan rate"
pgscan_kswapd_normal
pgscan_kswapd_dma32
pgscan_kswapd_dma
pgscan_direct_high <--- This is were the generic "page reclaim" statistics begin
pgscan_direct_normal
pgscan_direct_dma32
pgscan_direct_dma
You can also check, specifically, for pages reclaimed by "inode stealing."
host # cat /proc/vmstat|grep pginodesteal
There also may be other variations, depending on the flavor of Linux you're running. Catting /proc/vmstat and doing a:
host # cat /proc/vmstat|egrep 'scan|steal'
should get you most, if not all, of them.
And, in our final permutation, the bottleneck becomes the disk:kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s6 sd sd -- in sy cs us sy id
- - - - - s u m m a r y l i n e - r e m o v e d - on - p u r p o s e - - - - -
0 67 0 0 0 0 0 0 0 0 0 0 33 0 0 0 0 0 0 0 0 77
0 56 0 0 0 0 0 0 0 0 0 0 98 0 0 0 0 0 0 0 0 67
0 57 0 0 0 0 0 0 0 0 0 0 89 0 0 0 0 0 0 0 0 78
0 64 0 0 0 0 0 0 0 0 0 0 94 0 0 0 0 0 0 0 0 85
Again, these numbers aren't crazy, but now we've got an entirely different situation on our hands. We can see, from this example that our only disk (the "s6" column) is now starting to do some heavy writing (yes, I know the numbers aren't really that big. This is all relative ;) This might not be an issue, but the server's "slow." We couple this with a drastic increase in processes that are blocked waiting for I/O (the "b" colomn), along with a lack of significant swapping and/or paging activity, and the disk begins to look like the favorite. The "b" column can generally be considered a very vague indicator of the actual issue, since it reports on processes blocked waiting for resources no matter what they are (the CPU could be slow, the memory could be bad; even the network could be down). However, when we combine this with the fact that disk read/write activity has increased, the disk becomes our most probable bottleneck on the system.
On Linux, using either "vmstat -d" or "vmstat -p ARGUMENT" (with ARGUMENT being a specific partition) will get you the disk statistics.
These examples have been fairly stark. Always keep in mind that, outside of this vacuum, it's always good practice to keep regular tabs on your servers. For instance, you might run vmstat a few times a day (or a few times an hour) and record the results. When you get into habits like that, you're much better prepared when a problem does arise, as you'll have a good "baseline" to refer to when dealing with abnormal behaviour on any of your systems. Everyone will notice the biggest number in the vmstat output when they're frantically trying to figure out what's wrong, but (armed with your "baseline" knowledge) you'll know if that number has always been gigantic, in which case you can ignore it and go on to fix the actual problem :)
Hopefully this little exercise has been helpful to you; if even in the tiniest way :)
Cheers,
, Mike
linux unix internet technology
Sunday, April 6, 2008
Troubleshooting To Find The Bottleneck On Unix and Linux
Saturday, April 5, 2008
Further Dissection Of Paging And Swapping On Linux And Unix
Hey again,
Believe it or not, I actually got a few emails about our previous post on paging and swapping in Linux or Unix because it wasn't specific enough ;) While I can certainly understand the frustration, I was hoping to explain the main differences as completely and concisely as possible. I seem to have faltered a bit on each front: I glossed over two specifics which, I agree, deserve some attention and I wrote yet another novel ;)
With that in mind (and with a prayer that my fingers won't type any more than they have to ;) I'd like to address, and/or clarify, the level of depth I didn't descend to in my last post on the difference between paging and swapping and write about the difference between paging and swapping (The redundancy was intentional and any resulting confusion is expected, given the topic at hand and my writing style ;)
As I mentioned previously, the terms paging and swapping are used almost interchangeably these days. Some industry manuals will actually talk about "swapping out pages" which seems to be contradictory and, theoretically, impossible if swapping and paging are two separate concepts with distinct and unique definitions. This is where language and implied meaning become a barrier to actual definition. And, all the more reason to clarify this one last bit of the puzzle.
And here they come. The extra clarifications...
When a system swaps a program, or process, it guarantees that it is resident (on disk or in memory) before it schedules it for execution, and will often hold onto the mapped resources reserved for that process from the time the process requests them until it notifies the scheduler that it is complete. When a system pages during the execution of a program, or process, there isn't any such direct correlation. You don't necessarily know (without specifically checking) how much of a process's virtual memory is resident or whether the process is entirely able to be scheduled for execution at the time paging begins. Pages can be selectively grabbed from a process (out of mapped physical memory) and never returned, unless re-requested by the process when it looks for the memory, can't find it and generates a page fault.1. Difference in resident virtual memory management with paging and swapping.
Phew... That's one down. Hopefully this isn't just becoming more confusing :)
In this instance, swapping specifically refers only to the transfer of memory pages from physical memory to dedicated swap devices or swap disk (on most systems this is now referred to as swapfs - or a unique swap filesystem) and vice versa. Paging, on the other hand, refers to the transfer of memory pages from physical memory to disk (regular disk or swap disk) and vice versa. So, really, the major difference is that swapping is limited to only transferring memory pages back and forth between the physical memory and a dedicated swap device or filesystem, while paging can transfer between physical memory and any sort of disk device.2. More specific definition of paging and swapping with regard to page ins/outs and swap ins/outs.
Hopefully, we've reached a sufficient amount of explanation at this point, and this thing won't turn into the monster I'd hoped it wouldn't become ;)
Thank you, everyone who wrote in, for your helpful input. As many folks have also noted, we don't have comments set up on this blog (because of issues with "comment spam" and not wanting to get shut down). If you ever want to leave a comment, or an objection, we welcome you to email us directly at our most often-check email address or, if you have a lot to say (or want to attach video, etc), sign up (for free) on our sister Linux and Unix Menagerie Forum, and we'll get your remarks there, as well. We do our best to reply, personally, to everyone who takes the time to write us. So far, I think we're still batting a thousand in that regard ;)
Best wishes,
, Mike
linux unix internet technology
Friday, April 4, 2008
Swapping Or Paging On Linux And Unix?
Howdy,
Here's a question that gets asked a lot, and has a relatively simple answer to go with it: On Unix and/or Linux, what's the difference between paging and swapping?
It's a relevant question, given that the terms are used almost interchangeably these days. Even in most Linux or Unix monitoring commands, the issue can become confused. Consider our previous posts on free memory graphing on Unix and graphing out paging statistics on Linux. They're both showing approximately the same thing, but one of them is using the terminology in a not-totally-correct sense.
The good news is you only need to understand one thing about each (which is also a common thread) in order to understand what the terms "really" mean. This can be a great help when you're trying to determine the cause of a system issue, like a big slow-down. Of course, since the terms are mixed up a lot, it's a good rule of thumb to assume that any problem with "paging" or "swapping" may be a problem with either. Depending upon who's asking, they could mean one thing or the other. As in public speaking, it's always a good idea to know your audience ;)
The main difference between paging and swapping (on both Linux and Unix; all flavors, as far as I know) is this:
1. Swapping: This occurs when an entire process ( sometimes consisting of multiple parts like a read-only text segment, writable data segment and, more often nowadays, writable stack segment ) gets transferred to disk from physical memory or is read back into physical memory from the disk.
2. Paging: This occurs when part of a process ( a page, or a segment, of a process ) gets transferred to disk from physical memory or is read back into physical memory from disk. Paging also requires a MMU (Memory Management Unit) and a CPU capable of handling requests from it. This is just a side note, and slightly outside the scope of the definition. It really doesn't even make a difference any more since I haven't seen an OS without paging capability in years, and most dedicated Unix/Linux servers have had the latent capability for even longer.
Tomorrow, we'll begin looking at a real-life examples of determining a system issue highlighted by excessive paging (or is it swapping?). For today, we'll keep it abstract.
To wrap up, on today's system's (The year now being 2008 - Just dating this in case it gets read 2 years from now and I'm totally off-base by then ;) there's almost no such thing as swapping. Paging occurs normally and, if you do see actual heavy swapping, it's generally an indication of a problem with memory or disk (Except in situations where you have large applications - like an Oracle database, for instance - that hoard lots of Virtual Memory Address (VMA) space and cause the system to swap naturally). In somewhat contrast, if your system is paging heavily, but not swapping, your issue is most likely with CPU or memory. Memory is often mistakenly assumed to be the culprit in most situations because both swapping and paging involve writing to, and reading from, memory. However, it should always be taken into account what other component of the OS is doing the work to make that activity possible, or maybe even necessary.
One last thing to remember is that either of these situations ( excessive swapping, excessive paging or both ) could be indicators of either memory, CPU or disk issues. They could also point to a problem with your network subsystem or any number of things. The generic explanations/answers in the previous paragraph assume a relative norm. In reality, you have to look at the situation in the context of the problem you're facing on the system that's having the issue and work from there.
We'll run down some quick and easy real-life troubleshooting starting tomorrow.
Until then, best wishes :)
, Mike
linux unix internet technology
