Controlling Group Access To Directories Recursively And Automatically Using Simple FACL's On Linux

Hey there,

Today we're going to take a look at using simple FACL's (file access control lists) to allow additional access restrictions (for one or more groups other than the directory owner's group) to a directory, while ensuring that the change propagates automatically to all subdirectories and files within all or any. Our examples for today should work on most distro's of Linux (and most proprietary Unix distro's), although the actual commands needed may be different ("man getfacl" or "man setfacl" should sort you out). For today's examples, all commands were run on RedHat Enterprise Server 5.2 (Tikanga - A Mãori word idiomatically meaning "The Mãori Way" or, more sanctimoniously, "correct," "right" or "true" - all absolutes ;)

The situation for today's example is pretty simple, although highly useful (of course, those aren't necessarily opposites ;). You have a directory that is owned by the user "user1" and the group "group1." A user name "user2", in the group "group2" needs to have access to your directory and also needs to be able to add new directories and/or files within that directory. To top it off, when either "user1" or "user2" creates a file or directory within the directory to be shared, that directory and/or file must be accessible to both "user1" and "user2" and owned by the group "group1." Simple enough :)

Of course, the easiest way to do this would be to add "user2" to "group1" and be done with it. However, since that might possibly afford "user2" more privilege than is actually needed (he/she may be able to access other files/directories that are owned by "group1"), a more restrictive solution is required. This is where simple FACL's come into play.

First, we'll take a quick look at the directory in question, with ls -l:

host # ls -ld shared_dir
drwxrwx---  2 user1  group1  512 May 25 13:24 shared_dir

Once we're done, it will look almost exactly the same, except for the "+" symbol at the end of the alpha permissions display (This is an additional field, as opposed to setuid, setgid and sticky bit which replace the final alpha permissions field with a substitute character) As a point of note; some distributions include the "+" symbol at the end of "every" file, since all files/directories (on a filesystem that supports FACL's) have a default FACL. This seem counter-intuitive to me, since it makes it harder to eyeball which files/directories you've mucked with on purpose ;)

host # ls -ld shared_dir
drwxrwx---+ 2 user1  group1  512 May 25 13:24 shared_dir

Now, in order to make the whole concept of FACL's more accessible, it should be noted (at least, these days) that "all" files have FACL's (see slight caveat above), even if you don't explicitly set them. These FACL's mirror the existing file/directory permissions by default and you'll only the see the "+" symbol on files/directories where the default FACL has been modified (which is what we're going to do).

For instance, the FACL for our original shared_dir, looks like this before we even do anything, and can be obtained with the "getfacl" command (consider owner: user1, group:group1 and permissions of 770 - or drwxrwx---):

host # getfacl shared_dir
# file: shared_dir
# owner: user1
# group: group1

user::rwx
group::rwx
mask::rwx
other::---

All pound (#) comments are not acted upon, just like in most scripting languages. You can remove them, but they'll get automatically recreated every time you run a FACL command.

So, for this directory, we want to allow the group "group2" recursive read, write and execute permissions (read permissions, so they can see the contents of the directory or read any files in it, write permissions, so they can create new files/directories within the directory and/or write to files in it, and execute permissions, so that they can cd into the directory and/or execute files within it)

Luckily for us, this is actually fairly simple. All we need to do is to make use of the "setfacl" command.

host # setfacl -m g:group2:rwx shared_dir

This will change the FACL settings for the shared_dir directory, but won't do "exactly" what we want:

host # getfacl shared_dir
# file: shared_dir
# owner: user1
# group: group1

user::rwx
group::rwx
group:group2:rwx
mask::rwx
other::---

While doing this will allow "group2" read, write and execute access to the directory "shared_dir", the extended access will begin and end with the "shared_dir" directory, and any files/directories created by any user in "group2" will still be owned by "group2" when they are written to this directory.

If we want to satisfy the one remaining requirement (allowing "group2" to create new files/directories within "shared_dir" and have them all belong to "group1", with the unique FACL applied to each and every one), we need to set "default FACL's". Again, this is very simply done, and very easy to manage using the setfacl command Our first command will be to remove the rule we added with our last command. Then we'll set default FACL's for everyone and for the specific users and groups:

host # setfacl -x g:group2:rwx shared_dir
host # setfacl -d -m u:user1:rwx shared_dir
host # setfacl -d -m g:group1:rwx shared_dir
host # setfacl -d -m u:user2:rwx shared_dir
host # setfacl -d -m g:group2:rwx shared_dir

If you use the form above (which is, admittedly, more strict than it needs be; but I'm a control freak ;) and don't see the extra "default" FACL entries for the default user and group that own the file/directory, you can set those like this (also using setfacl):

host # setfacl -m d:u:rwx shared_dir
host # setfacl -m d:g:rwx shared_dir
host # setfacl -m d:o:--- shared_dir

These three entries should be put into your new FACL for the shared_dir by default, when you type the highly-specific FACL commands above, but if they're not, you'll want to add them. They are the key to propagating correct permissions on all new files/directories created within that directory.

Now typing "getfacl" should show you a dramatically different output:

host # getfacl shared_dir
# file: shared_dir
# owner: user1
# group: group1

user::rwx
group::rwx
mask::rwx
other::---
default:user::rwx
default:user:user1:rwx
default:user:user2:rwx
default:group::rwx
default:group:group1:rwx
default:group:group2:rwx
default:mask::rwx
default:other::---

As I mentioned, you could leave out all the commands pertaining to the original owner (and group) of the file, but I like to be as specific as possible, just in case. Now, any file/directory that either user1, user2, or members of group1 or group2 create in this directory will have 770 permissions (rwxrwx---+) and their group will be set to "group1." You can also easily tell that your settings have worked by looking for the "+" symbol at the end of newly created files, which indicates that the FACL has been applied to your new file (although with a slightly different output for files than for directories):

host # who am i
user2
host # touch shared_dir/test
host # ls -l shared_dir
total 0
drwxrwx---+ 2 user2 group1 96 May  1 14:01 test

host # getfacl shared_dir/test
# file: test
# owner: user2
# group: group1

user::rw-
user:user1:rwx
user:user2:rwx
group::rwx
group:group1:rwx
group:group2:rwx
mask::rwx
other::---

And that's that :) If you still don't get the group propagation that you're expecting, it may not be supported on your OS. In that case, you can add this additional command (on top of the FACL's) to ensure that the correct group "sticks":

host # chmod g+s shared_dir

And you should be all set :)

Cheers,

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Simple Code To Implement C's stat() function On Linux And Unix

Hey There,

Today's post loosely ties back to our post last week on Perl stat() basics for Linux and Unix, although this post is not a continuation of that one. Just trying to keep it confusing ;)

I've often been bashed for my C programming skills (the "bash" in that sentence wasn't meant to be a pun), and, although I've come to expect the vicious backlash, I still like to come back to C every once in a while to look at some of the things you can do with it and to keep me awake ;) That being said, any competent C programmer out there who finds fault with this post's attached C-code is, most probably, correct. This could be written better, have more robust error-checking, implement measures to prevent buffer over-runs, under-runs, and a thousand other exploits. That being said; it works, which is miracle enough for me :) Feel free to modify it in any way, shape or form you prefer. Everything I put on this blog is considered at-your-own-risk material. There's not all that much to worry about, though, if you just use this code for yourself to test and play with. Just secure it with simple ownership and file permission settings and delete the binary when you're done.

It's been a long time since I've had to write this, but typing it will be faster than trying to find another post with C code in it ;) If you want to compile the attached code, you can do the following (should work with most any C compiler, I'm just using gcc as an example since that's what I have installed on my machine):

host # gcc -o filestat filestat.c

and the program is ready to run. That part's pretty simple, too. You just execute it and follow it with one or more file names (they can be "any" type of file: file, directory, link, socket, etc). If you want to see the usage information, just run it without any arguments, like so:

host # ./filestat
Usage: ./filestat file1 file2 fileN...

The most interesting part for me was figuring out how to decipher the "file mode" information. If you check your system's "stat" manpage, you'll see that there are actually a good deal of macros already written-in that make printing out plain English file types and modes much simpler, but I decided to keep it cryptic. I take my modes like I take my permissions; in octal ;)

Here's a sample run of the program - just using one file to make it simple (it should be noted, also, that you can only run this command against file types that you have permission to read. If you run this as your regular user on a file owned by root with 700 permissions, you'll get the "File or Directory Not Found!" error, which is, technically, incorrect. My God, am I a lazy C programmer. No wonder I get beat up so bad every time I goof around with it on here ;)

NOTE: This program interacts with the shell you're using, so if, for instance, you can access a file named "myfile" by typing "my*", the same globbing rules will be taken care of, applied to the filename and expanded by the shell before this program processes it!

host # ./filestat tct-1.18
****************************************
Checking tct-1.18
****************************************
tct-1.18 stat values are:
****************************************
File Mode: 40700
Inode Number: 17540
Device ID: 35651599
Raw Device ID: 0
Number Of Links: 12
User ID: 0
Group ID: 0
Bytes Size: 1024
Access Time: Mon, March 02, 2009 - 14:15:31
Modification Time: Mon, April 09, 2007 - 08:36:09
Status Change Time: Mon, March 02, 2009 - 15:47:38
Block Size: 8192
Number Of Blocks: 2
File System Type: ufs
****************************************

Most of the above is self-explanatory. The few things to note (if you just "really" don't want to read the manpage ;) are that the "Raw Device ID" is almost always zero unless you're dealing with a disk, or other file type, that supports raw access. From my own testing, you can definitely get back a "Raw Device ID" from any disk (/dev/rdsk/c0t0d0s0, /dev/dsk/c0t0d0s0, /dev/hda1, etc). The "Bytes Size" is simply the Size in Bytes. I left it the way it is because it reminds me that I need to eat less ;)

Lastly, you may have noted that the "File Mode" (listed first in the output, since I just went through the "stat" struct and used each value in order) seems a bit off. The last 3 digits are probably familiar to most Linux and Unix users, as they represent the file permissions. By the same virtue, the first 4 (including those original 3), probably seem familiar as well, since that 4th-left bit is used in octal permissions settings to represent setuid, setgid, and sticky-bit, etc, special permissions. So, for the above "File Mode" of 40700, you know that the file permissions are "0700" (no setuid, setgid or sticky-bit - read/write/execute for the user and no permissions for the group and world).

The only question, once you've got the right side of the "File Mode" down is, what do the variable number of numerals to the left of those stand for? The answer is pretty simple and (as I noted above) there are standard macros that you can use in your C code to test the mode and print it out so that it's more human-readable. Basically, they determine the type of file you're interrogating. Since our one number to the left of the four permission bits is "4," we know that the file we're accessing is a directory. You can see this in the table below (shamelessly lifted from the system header file - /usr/include/sys/stat.h ;) <-- This is just a subset of all available macros. There's a lot more information in that header file than I want to jam into this post... It's getting long enough now. Especially bad if you're an "F" reader ;)

As one last interesting side-note, the "File Mode" for the gzipped tarball, that was extracted to create this directory, is "100700" which gives us the same permissions, but a "10" on the left-hand "file type" side. You can also see this showing up below as a "regular" file.

#define S_IFMT  0170000  /* type of file mask */
#define S_IFIFO  0010000  /* named pipe (fifo) */
#define S_IFCHR  0020000  /* character special */
#define S_IFDIR  0040000  /* directory */
#define S_IFBLK  0060000  /* block special */
#define S_IFREG  0100000  /* regular */
#define S_IFDB  0110000  /* record access file */
#define S_IFLNK  0120000  /* symbolic link */
#define S_IFSOCK 0140000  /* socket */
#define S_IFWHT  0160000  /* whiteout */
#define S_ISVTX  0001000  /* save swapped text even after use */
#define S_IRWXU         00700   /* read, write, execute: owner */
#define S_IRUSR         00400   /* read permission: owner */
#define S_IWUSR         00200   /* write permission: owner */
#define S_IXUSR         00100   /* execute permission: owner */
#define S_IRWXG         00070   /* read, write, execute: group */
#define S_IRGRP         00040   /* read permission: group */
#define S_IWGRP         00020   /* write permission: group */
#define S_IXGRP         00010   /* execute permission: group */
#define S_IRWXO         00007   /* read, write, execute: other */
#define S_IROTH         00004   /* read permission: other */
#define S_IWOTH         00002   /* write permission: other */
#define S_IXOTH         00001   /* execute permission: other */

Hope you enjoy the attached C code and get some use out of it (or your own Frankenstein'ed version of it ;)

Cheers,




This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <dirent.h>
#include <pwd.h>
#include <grp.h>
#include <time.h>
#include <locale.h>
#include <langinfo.h>
#include <fcntl.h>

/*********************************************
filestat.c - Would you like fries with that?
2009 - Mike Golvach - eggi@comcast.net
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
*********************************************/
int main (int argc, char *argv[])
{
        if (argc <= 1)  {
                printf("Usage: %s file1 file2 fileN...\n", argv[0]);
                exit(1);
        }
        struct stat dirfilebuf;
        int arguments, dirfilestate;
        const char *dirfilename;
        char *nicetime, *errormessage, *divider;

        dirfilename=(char*)(malloc(sizeof(char)));
        nicetime=(char*)(malloc(sizeof(char)));
        errormessage=(char*)(malloc(sizeof(char)));
        divider=(char*)(malloc(sizeof(char)));

        errormessage="File or Directory Not Found!";
        divider="****************************************";

        for (arguments = 1; arguments < argc; arguments++)      {
                dirfilename = argv[arguments];
                printf("%s\nChecking %s\n%s\n", divider,dirfilename,divider);
                dirfilestate = stat(dirfilename, &dirfilebuf);
                if ( dirfilestate == -1 )       {
                        printf("%s - Quitting!\n", errormessage);
                        exit(1);
                }       else{
                        printf("%s stat values are:\n%s\n", dirfilename,divider);
                        printf("File Mode: %lo\n", dirfilebuf.st_mode);
                        printf("Inode Number: %-10d\n", dirfilebuf.st_ino);
                        printf("Device ID: %d\n", dirfilebuf.st_dev);
                        printf("Raw Device ID: %d\n", dirfilebuf.st_rdev);
                        printf("Number Of Links: %d\n", dirfilebuf.st_nlink);
                        printf("User ID: %-8d\n", dirfilebuf.st_uid);
                        printf("Group ID: %-8d\n", dirfilebuf.st_gid);
                        printf("Bytes Size: %jd\n", (intmax_t)dirfilebuf.st_size);
                        strftime(nicetime,34,"%a, %B %d, %Y - %H:%M:%S", localtime(&dirfilebuf.st_atime));
                        printf("Access Time: %s\n", nicetime);
                        strftime(nicetime,34,"%a, %B %d, %Y - %H:%M:%S", localtime(&dirfilebuf.st_mtime));
                        printf("Modification Time: %s\n", nicetime);
                        strftime(nicetime,34,"%a, %B %d, %Y - %H:%M:%S", localtime(&dirfilebuf.st_ctime));
                        printf("Status Change Time: %s\n", nicetime);
                        printf("Block Size: %d\n", dirfilebuf.st_blksize);
                        printf("Number Of Blocks: %d\n", dirfilebuf.st_blocks);
                        printf("File System Type: %s\n", dirfilebuf.st_fstype);
                        printf("%s\n", divider);
                }
        }
}

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Back To Basics: Getting File Information Using Perl's Stat Function

Hey there,

Today we're going take a (somewhat) beginner's look at extracting lots of useful information from files on Linux or Unix using Perl. More specifically, we'll be using Perl's stat() function. If you're a regular reader, you've probably noticed that we've strayed quite a bit from our original motto (which is screaming in the meta-description tag of every page ;) which is to keep up a blog that appeals to Linux and Unix users and/or admins of any skill level. Lately, we kind of feel like we've neglected the folks newer to the Linux/Unix world. Perhaps we're wrong. We're obviously confused. Perhaps, the next time we start a blog we'll be bit more specific about the subject matter. That being said, we've made our beds and have resigned to lie in them, since the floor is getting uncomfortable again ;)

Today we're going to take a brief introductory look at Perl's stat() function and how you can use it to extract lots of useful information from files, directories, etc (actually, everything in Linux/Unix is a "file" of a certain type). It's incredibly easy to use and, even if you've never used Perl before, very easy to implement. If you're still stuck at the Shebang line in point of reference to Perl/shell scripting, check out that post, as we hope it provides a good base for one of the things you're most likely to find in a Perl or shell script. It also goes beyond just scripting and attacks some other basic concepts.

Here's a very simple Perl script (which, as demonstrated, can be written on the command line as well) that will let you know the user id and group id associated with a particular file. We won't be exploring stat() completely in this post. just enough to get you started and comfortable (you could get the same information by running "ls -l" instead of just "ls," but that would defeat the purpose, right? ;):

host # ls
file.txt
host # perl -e '$file="file.txt";($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime,$blksize,$blocks)=stat($file);print "$file : User: $uid - Group: $gid\n"';
file.txt : User: 1000 - Group: 513

Your output may vary depending on what flavour of Linux or Unix you're using. In script format, this command line would look this way (not really all the much different, but, perhaps, easier to read):

#!/usr/bin/perl

$file="file.txt";
($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime,$blksize,$blocks)=stat($file);
print "$file : User: $uid - Group: $gid\n"';

When you run this (after giving it execute permission for the user - you :) - ) you get the same result (assuming you called it file.pl):

host# ./file.pl
file.txt : User: 1000 - Group: 513

Granted, today's examples were rudimentary for anyone who's had experience with Perl's functionality and the Unix/Linux shell, but, in this on and off series, we're going to take it a bit slow (we'll probably build on this post every two or three other posts). This is in no way an attempt to dumb-down the blog, just to get back to basics. Think of it as a bracing slap in the face we're giving ourselves to remind us that this blog isn't all about higher concepts.

All of us here, at one point, didn't know what Linux or Unix was, much less how to make any use of it. This interweaving series is for those of you who are just getting started (just like everyone in the business did at some point :)

We look forward to getting more into it. Perhaps the learning curve will be surprisingly fast. At the rate our children are picking up on computers, perhaps the transfer of knowledge won't be quite as difficult as it was in our day. Who can say? We don't know, but we're positive that, as soon as they can, they will ;)

In our next post in this series, we'll look at the components of the Perl stat() function and the rest of the information you can gather from it. Hopefully this post has been easily digestible enough to act as an introduction. If not, please contact us (via the link at the right hand top of every page) and let us know. We write a lot for ourselves (especially Mike, who thinks everything that makes him laugh is funny to everyone else in the entire universe ;), but are committed to providing useful information for you :)

Cheers,

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Why DU And DF Display Different Values On Linux And Unix

Hey there,

Today we're going to look at a little something that is a fairly hot-running-water issue on most of the Linux and Unix boards lately (actually, probably always has been, but our research staff quit on us ;) This post will be similar in focus to our previous post on the differences between sar and vmstat with regards to free memory/swap reporting.

Today's post is similar in spirit, but not a replay of that previous memory/swap reporting issue. Today, we're going to take a look at two other commands that, seemingly, measure and display the same information, although with (sometimes) huge discrepancies in output. Those two commands are du and df.

The question you'll most often see (or, perhaps, have :) is something to the effect of "Why do my outputs from du and df differ? One says I'm using more disk space than the other. Which of them is correct?" Generally you'll find that df shows more disk spaced used than du does, but the case can sometimes be the opposite. It's very rare (unless you don't use your computer, and it doesn't use itself, at all ;) that the output from the two commands match. It's actually rare that they ever come close to matching. Generally, the longer a machine is up, the greater the rift between figures becomes.

The bad news: This is confusing and sometimes hard to communicate to others, even when you know why the situation exists :(

The good news: This is normal and can be explained; perhaps even simply :)

To set up the "typical" situation, we used a Solaris 10 box (although this issue is common on all proprietary Unix and Linux distro's). Below, the output from four commands executed in the /opt directory (NOTE: For du, the "-s" flag is used to print summary information for the entire object, rather than a listing of all of its parts (i.e. the whole directory rather than all the files and subdirectories)):

host # cd /opt <-- For df, look at the "used" column and for du, look at the only output there is ;)

host # df -h .
Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c0t0d0s6      3.9G   490M   3.4G    13%    /opt
host # du -sh
 486M   .
host # df -k .
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0t0d0s6    4130238  502180 3586756    13%    /opt
host # du -sk
498068  .

Now, in two parts, the complicated reasons, explanation and summation of why this disparity exists, followed by the more customer/user-friendly version of the exact same thing :)

1. The convoluted, complicated and hard to pass-along explanation:

Reasons: While du and df report "approximately" the same information, they don't report "exactly" the same information. While they both report "about" the "same things" (some of the time), they don't measure those "things" using equal methods or metrics.

Explanation: These differences can occur for a number of reasons, including (but not limited to):

a) Overlay mounts, which can skew df output when it's run from a higher level directory (e.g. df -k /opt would not report the total space used on /opt if there were overlay mounts of /opt/something and /opt/anotherthing on the system - du will. If you actually need the total space used by /opt and everything underneath it, including the overlay mounts, df can do it, but it takes some plate-spinning ;).

b) Sometimes (This actually should never happen on any "recent" OS) hidden files and subdirectories could skew du's output.

c) Unlinked inodes can cause unexpected statistics with both df and du (For, instance, in the situation where your filesystem shows almost 100% free space but all the inodes are in use.)

d) You're using the "-h" flag instead of the "-k" flag for one or the other commands (or both). The "-h" (human readable) flag can sometimes make things seem worse (or more divergent) than they are. Since it tries to make the output more easily digestible, it will round the numbers you see, so you can't "really" be sure whether "5.1 GB" is closer to 5 GB or 5.2 GB. "-k" is slightly more likely to produce relatively equal results, as it reports the size in kb. It still does do rounding so that you only get straight up integers and no floating point results, but it's generally better to check with if your output from "-h" is very close, or even the same (since it might not be). If your implementation of df and/or du supports the "-v" flag (or something similar), that's even better since it reports in multiples of your system blocksize and is even more exact.

e) The fundamental way in which each of the commands work:

i) df:

df reports only on the mount point, or filesystem, level. So a df on /opt would produce the same results as a df on /opt/csw (assuming, as noted above, that they're both on the same partition):

host # df -k /opt
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0t0d0s6    4130238  502180 3586756    13%    /opt
host # df -k /opt/csw
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0t0d0s6    4130238  502180 3586756    13%    /opt

df gets most of its information from a filesystem's primary superblock (except in the odd instance that an alternate superblock is being used - although this would only happen in an "fsck" situation. After that, the information from the alternate superblock would be copied back to the primary, and all other superblocks). It takes this information at face value, which is to say that it does not question the information provided to it by the primary superblock. In this respect, df is a very fast tool for getting disk usage information (at the cost of reliability).

df will include open files (in memory, but not on disk), data/index files (used for data management - sometimes using approximately 2 to 5% of each filesystem) and unnamed files in its size calculation. This is one reason why, sometimes (although not very often), df can show a larger amount of disk used than du does.

df, as per above, will explicitly trust any errors in space calculation that may have occurred over time, since it trusts the primary superblock entirely. This means that if you've fsck'ed a filesystem (and not resynced or rebooted since), and/or have experienced any hard or soft errors on the device/disk housing the filesystem, you're measuring (again, assuming you haven't rebooted since they happened) and/or experience any possible corruption or inconsistencies in any filesystem-state records (like /etc/mnttab), your output will be commensurately incorrect.

df sometimes reports file sizes incorrectly, as it works on what we like to call the whole-enchilada-principle ;) Basically, if you take your filesystem's block size (for the /opt filesystem here - different filesystems may have different block sizes), which you can find by executing either of the following commands, both of them, or whatever works on your OS:

host # df -g /opt|grep "block size"|awk '{print $4}'
8192
host # fstyp -v /dev/rdsk/c0t0d0s6|grep "^bsize"|awk '{print $2}'
8192

...you'll be able to do this experiment on your own (we'll leave out the grisly details to save on space :). In our instance, we have a default block size of 8192 bits or 8 kb. Now, here's where it gets somewhat interesting ;) If you create a new file that's 1 kb in size and it writes to a new, or - depending - not fully used, block, df will report that file as being 8 kb in size, even though it's actually only 1 kb in size!

ii) du:

du reports at the "object" level rather than at the filesystem/mountpoint level, as df does. So, to repeat the example from above, if you run du on /opt and /opt/csw, you'll get different results. I find it easier to think of du as handling its measurement via a simple "object" model. The main partition would be the meta-object, while any subdirectory you may be running du against would be considered a sub-object of the filesystem meta-object (you'll note that the du size output for /opt/csw is, naturally, smaller than that for the entirety of /opt):

host # du -sk /opt
498068  /opt
host # du -sk /opt/csw
64065   /opt/csw

du gets its information at the time you execute it (unless you run it repeatedly in succession, where you'll notice a slight performance improvement). To test this, run du on a partition, then wait 5 minutes and run it again. It should take just as long as the first time (unless you've added lots of files since then). In this respect, du can be a very slow tool for getting disk usage information (with the benefit that your information will be more accurate). It should be noted that, because of the way it takes measure of most (see below) filesystem objects, it takes much longer for it to report the size of a billion 1 kb files than it does to report the size of one file of 1 billion kb size.

du does not count data/index files or open files (in memory, but not on disk).

du does not take into account any information "supplied" by the system (meaning the information, like from the superblock, as listed under the df section) and gets its information independent of whatever the system thinks is correct.

du does not rely on block size (see the whole-enchilada-principle in df's section above) to determine file size. So, if you have a default 8 kb block size on your filesystem, you create a new 1 kb file that writes to an empty 8 kb block, du will report that file as being 1 kb in size and "not" assume a minimum size of the filesystem block size. This is worth remembering, because it can cause a great deal of difference in the filesystem "usage" size between du and df (which would consider that 1 kb file an 8 kb file - 8 times larger than it actually is)!

du is more reliable if you want to know the state of your filesystem "right now." It doesn't count any data/index blocks .

Summation: If you are interested in knowing exactly how much of your filesystem is actually being used, du is a much more accurate tool for collecting and displaying this information. Note, however, that - since du does "on demand" filesystem size reporting, it is much slower than df. Also, du does not play as well with some system internal files and processes since it essentially ignores information reported by the primary superblock, system mount information tables, etc. Ultimately, the purpose for which you need to determine your filesystem's size (coupled with an understanding of the "Explanation" section for both utilities) is the best way to decide which utility to use in any given situation.

2. (Did you forget this was coming, too? ;) The simple, and easy to convey, explanation:

Reasons: df and du rely on different information to determine how much disk space is used on a filesystem.

Explanation: df and du report filesystem information differently, for very basic reasons:

df and du don't use the same yardsticks to measure filesystem size.

df:

df relies mostly on system information, supplied by various files and built-in reporting mechanisms that may, or may not, be correct at any given time.

du:

du relies on what it can "see" at the particular moment in time that you run it.

Summation: du is the better tool to use if you are interested in knowing how much space is actually being used on your filesystem "right now." df is great for "ballpark estimates" and is preferred if you need to know how big df thinks your filesystem is (so it will agree with other incorrect system statistics).

3. The really easy, and simple to blurt-out, explanation:

If du and df don't agree on what size your filesystem is, du is more correct than df is.

See; it's all very simple ;)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Recovering Deleted Files By Inode Number In Linux And Unix

Hey there,

It should be noted, at the outset, that this post is limited in its scope. We're going to be looking at one particular way in which you can recover an accidentally deleted file on Linux or Unix ( Tested on RHEL3 and Solaris 8 ). If you ever want to scour a hard drive that you need to get lots of information back from, then (assuming that you quarantined it immediately upon noticing this and haven't written to it since) you should check out The Coroner's Toolkit. Specifically, you'll want to look at the "ils" or "icat" programs, and most probably the "grave-robber" application to recover as much of everything as possible.

In our post, we're going to look at one condition under which Unix and Linux operating systems can actually hold onto files, even after they've been deleted, so that you can recover them. Of course, you have to realize you've deleted something you wanted to keep (in most cases) immediately, and a fairly specific set of circumstances has to be in play.

Note that all demonstrations have been tested on Solaris and RedHat. We're using RedHat's output, but the differences between the two were minor enough that they didn't bear repeating.

First, we'll look at the controlled scenario: In this case, we're going to create a file. Then we're going to delete it, knowing that we want to retrieve it afterward. How many times does that happen "by accident"? ;)

host # echo "hi there" >>FILE
host # ls -lai             
total 8
 57794 drwxr-s---  2 user group 4096 Aug 11 12:36 .
851795 drwxr-s---  3 user group 4096 Aug 11 12:31 ..
 57795 -rw-r-----  1 user group    9 Aug 11 12:36 FILE

Now that the file's created, we can cat it by referencing the inode number (gotten with "ls -lai" above) very easily

host # find . -inum 57795 -exec cat {} \;
hi there

Then we'll set ourselves up so that we can delete the file and still be able to get it back. In order for us to be able to retrieve the deleted file later, we'll need to associate it with a filehandle. One easy way to do that is to run a tail (or similar command) on it):

host # tail -f FILE &
[1] 4741

and then we'll delete it:

host # rm FILE
host # ls -lai
total 8
 57794 drwxr-s---  2 user group 4096 Aug 11 12:41 .
851795 drwxr-s---  3 user group 4096 Aug 11 12:31 ..

So, now it's gone. But, and this is the only thing that's saving us a whole lot of headache, the file is still open in memory since we still have a "tail -f" job running in the background. This means that the tail command still has a filehandle open for our file FILE. Of course, we can't refer to it by that name anymore. One interesting thing to note about an inode is that it contains virtually all the information you ever wanted to know about your file... except it's name! :)

Therefore, the following query with lsof fails to produce results:

host # lsof|grep FILE

IMPORTANT NOTE: If you have followed this process closely and either wrote down or remembered the "inode number" of the FILE file before we deleted it, you can skip all of this lsof stuff. Jump straight to the next IMPORTANT NOTE in this post :)

A quick look at what pseudo-terminal we're using, coupled with the knowledge that we ran "tail -f" on the file, makes for a pretty tidy grep string (of course, you don't need to have this much information. You can do this without of a filter and just have more lsof output to muck through):

host # tty
/dev/pts/0
host # lsof|grep pts/0|grep tail
tail       4741   user    0u      CHR  136,0                 2 /dev/pts/0
tail       4741   user    1u      CHR  136,0                 2 /dev/pts/0
tail       4741   user    2u      CHR  136,0                 2 /dev/pts/0

Now we know the PID (second column in from the left) of the process that still has the file open and we can use lsof to drill down even further, using the output of the pwd command to whittle down the output:

host # pwd
/home/users/user
host # lsof -p 4741|grep home/users/user
COMMAND  PID  USER   FD   TYPE DEVICE    SIZE    NODE NAME
tail    4741  user  cwd    DIR   0,22    4096   57794 /home/users/user (host:/remote/users)
tail    4741  user    3r   REG   0,22       9   57795 /home/users/user/FILE (host:/remote/users)

IMPORTANT NOTE: Welcome back, if you already knew the inode number, and "on we go" to everyone! One interesting thing to note about the lsof output is that there is only a "read" filehandle open for FILE. This is normal, since we're doing a "tail -f" and there doesn't need to be a write or read/write file descriptor active.

Now we can verify that the our "file" still exists by accessing it via the inode number:

host # find . -inum 57795 -exec cat {} \;
hi there

And, it's still there :) In order to preserve it (since it the inode-fd connection will be severed as soon as "tail -f" quits), we'll use similar "find" syntax to copy the inode to a filename and verify it, like so:

host # find . -inum 57795 -exec cp {} FILE.recovered \;
host # ls -lai
total 8
 57794 drwxr-s---  2 user  group 4096 Aug 11 12:42 .
851795 drwxr-s---  3 user  group 4096 Aug 11 12:31 ..
 57796 -rw-r-----  1 user  group    9 Aug 11 12:42 FILE.recovered
host # cat FILE.recovered
hi there

Now we can quit our backgrounded "tail -f" job and not worry about it.

host # kill %%
host # 
[1] + Terminated           tail -f FILE

Like I said, a neat trick, but will only work if you get lucky. Pardon the double-cliche, but keeping calm when you realize that you may have royally screwed the pooch can go a long way toward keeping you from having to take the long way home (isolating the disk, scouring it with forensics tools, perhaps grepping the raw filesystem and manually reconstructing, etc).

The one thing to remember is that, if you delete a file on Linux or Unix, as long as one other process, that was using it when it existed, is still up and running after you delete it, there's a possibility that you can get your file back fairly quickly (actual results may vary. The possible scenario's are many and varied).

And, to give us a fair sendoff, here's a quick look at what you can expect if you delete a file and no process is attached to it and/or has a filehandle open against it (If you don't like watching train-wrecks, then I'll bid a good evening to you right here and now - cheers :)

host # ls -lai
total 8
 57794 drwxr-s---  2 user  group 4096 Aug 11 12:43 .
851795 drwxr-s---  3 user  group 4096 Aug 11 12:31 ..
 57796 -rw-r-----  1 user  group    9 Aug 11 12:42 FILE
host # find . -inum 57796 -exec cat {} \;
hi there
host # rm FILE
host # find . -inum 57796 -exec cat {} \;
host # tty
/dev/pts/0
host # lsof|grep pts/0
ksh        3907   user    1u      CHR  136,0                 2 /dev/pts/0
ksh        3907   user    2u      CHR  136,0                 2 /dev/pts/0
ksh        3907   user   11u      CHR  136,0                 2 /dev/pts/0
lsof       5062   user    0u      CHR  136,0                 2 /dev/pts/0
lsof       5062   user    2u      CHR  136,0                 2 /dev/pts/0
grep       5063   user    2u      CHR  136,0                 2 /dev/pts/0
grep       5064   user    1u      CHR  136,0                 2 /dev/pts/0
grep       5064   user    2u      CHR  136,0                 2 /dev/pts/0
host # lsof|grep FILE
host # lsof|grep 57796
host # ls -lai
total 8
 57794 drwxr-s---  2 user  group 4096 Aug 11 12:44 .
851795 drwxr-s---  3 user  group 4096 Aug 11 12:31 ..

...bummer.

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Using Mkfifo For Monitoring And Enhanced User Activity Logging

Hey there,

For today's post, if you're unfamiliar with how to execute new file descriptors and work with shell redirection, to a comfortable degree, check out our older posts on those subjects (cleverly anchor-linked into this very paragraph). They explain more basic stuff regarding those topics better than this post will and, if you want to, they'll be there for you to read for as long as I can remain within the hosting company's good graces.

One basic component of any decent security setup should always be logging. At the very least, if you're running a server that needs to be on, and servicing requests, 5 days a week (or 24x7, etc), you're probably logging sar output or iostat, vmstat, etc. You're taking some sort of metrics so that you'll be able to troubleshoot any problems that may arise in the future when your needs outgrow your capacity, or somebody just makes a mistake and runs an infinite recursive fork-and-exec loop as root.

Another thing that a lot of companies like to do is to keep tabs on their employees. Especially as it pertains to their activities on production Unix and Linux servers. This may or may not be for a bad reason. Some companies (depending on their nature) are required to monitor user activity to maintain standards compliance, while others may just do it so that if user XYZ makes a mistake and crashes the machine, they can figure out that it was he/she who did it and make sure that it doesn't happen again (hopefully, by explaining why what happened happened and considering it a lesson learned. It depends heavily on the workplace and the situation, of course)

One way to keep an eye on your users is to run kernel process accounting (pacct, which is available for Linux and Unix) and kernel auditing. Both of these probably provide the most complete picture in the event of a catastrophe, but they come at a cost because they can exact a heavy toll on your systems even when everything is fine. They can also make it so that things aren't fine any more, or expedite the non-fineness (???) of a situation if anything "bad" happens. Imagine how much harder your kernel and/or memory is being interrogated when there's a problem with it. Infinite recursion comes to mind again. The ghosts of those Fibonacci numbers just never go away ;)

In my practice, unless it's specifically requested, I prefer to log via log file. I know it seems primitive, and in a sense it is, but it's very "inexpensive" in terms of performance cost. One tool you can use to help with system auditing is called "mkfifo." It, of itself, doesn't really do anything for you, but it can be a great facilitator (especially if you use your imagination and have the latitude to experiment).

mkfifo is a program that creates a simple named pipe. You can equate it to the "|" that you use in your every day command lines, like:

host # cat FILE|grep word

although it's slightly differently. When a named pipe is created, via mkfifo (or however else you can do it), it creates a pipe "file" that remains in place until it is removed (or, in some cases, until your machine reboots, if you forget to remove it). You can create your own named pipe with mkfifo simply, as it takes very few arguments, like so:

host # mkfifo -m 777 /tmp/corncob
host # ls -l /tmp/corncob
prw-r-----   1 user    group       0 Jul  9 15:11 /tmp/corncob

That's all it takes to create the named pipe /tmp/corncob. The -m flag, which is used to set the permissions, is not necessary. Generally, if you don't include it, the default permission set for a new named pipe is whatever the default for your system would be. As another side note, you can also pass the -m flag and set alpha permissions, rather than octal, like:

host # mkfifo -m a=rwx /tmp/corncob

to create the exact same thing. You can delete the named pipe just like you delete a file. rm, and it's gone.

One thing you should note about named pipes is that they generally (so far as I've seen) are only able to fully pass one stream of input/output through themselves at a time. That is to say, if you have one process sending input to the named pipe and two process reading from it, only one of the reading processes will receive output. It should be noted, also, that, if such a situation were to exist, once the original process that was receiving output exits, the other process would begin receiving output from the named pipe (if it was still attempting to read from it). Was that a really long sentence or am I just typing fast? ;)

An example of what I mean below:

host-term1 # while :;do echo a b c d e >/tmp/corncob;sleep 15;done

host-term2 # tail -f /tmp/corncob
a b c d e
a b c d e
a b c d e
a b c d e
a b c d e
a b c d e
a b c d e
a b c d e

host-term3 # tail -f /tmp/corncob

host-term2 # ^C

host-term3 #
^C
a b c d e
a b c d e
a b c d e
a b c d e

Now, if you combine this freely available named pipe (you should probably secure it with permissions so that only users you want to be able to actually modify it have permission to remove it. Everyone you want to manage must be able to write to it and execute it.) with any number of output capturing mechanisms, you've got yourself a logger. Of course, you'll need to take care and work out how you want to parse the output coming from the named pipe because, even though it can only be read from by one process at a time, it can be written to by many and (if you have 15 people logged in, all with duplicate STDIN and STDOUT getting pushed through the same named pipe) that could get confusing for you very quickly.

My first recommendation would be, of course, to initiate an individual named pipe per user login process. This is so lightweight that generating 60 named pipes isn't going to cost you much more in overhead than generating six.

A simple thing to do is to use script to generate output and redirect it to the named pipe, like this:

host # ksh -ic "script /tmp/corncob"

using the name of the FIFO as the name of the output file for script. This is generally a bit klunky (script dumps the output in chunks and sometimes loses bits), but it does work relatively well. All you need to do is modify that slightly and put that in your users' .profile files and they'll be writing their entire sessions to /tmp/corncob (or, like I mentioned, each to a different named pipe file: /tmp/username perhaps?)

I think you can begin to see the possibilities here (and, probably some of the pitfalls - for instance, you can't use "tee" since it won't allow interactivity), but we'll continue on this topic tomorrow and look at some specific ways in which you can use named pipes, in conjunction with STDOUT and STDERR redirection, to do some serious security logging without serious stress :)

Cheers,

, Mike

Finding The Number Of Open File Descriptors Per Process On Linux And Unix

Hey There,

Today, we're going to take a look at a fairly simple process (no pun intended), but one that (perhaps) doesn't come up enough in our workaday environments that the answer comes to mind as obviously as it should. How does one find the number of open file descriptors being used by any given process?

The question is a bit of a trick, in and of itself, since some folks define "open file descriptors" as the number of files any given process has open at any given time. For our purposes, we'll be very strict, and make the (usually fairly large) distinction between "files open" and "open file descriptors."

Generally, the two easiest ways to find out how many "open files" a process has, at any given point in time, are to use the same utilities you'd use to find a process that's using a network port. On most Linux flavours, you can do this easily with lsof, and on most Unix flavours you can find it with a proc command, such as pfiles for Solaris.

This is where the difference in definitions makes a huge difference in outcome. Both pfiles and lsof report on information for "open files," rather than "open file descriptors," exclusively. So, if, for instance, we were running lsof on Linux against a simple shell process we might see output like this (all output dummied-up to a certain degree, to protect the innocent ;)

host # lsof -p 2034
CMD      PID USER   FD  TYPE DEVICE    SIZE  NODE NAME
process  2034  user1  cwd   DIR    3,5    4096 49430 /tmp/r (deleted)
process  2034  user1  rtd   DIR    3,7    1024     2 /
process  2034  user1  txt   REG    3,5  201840 49439 /tmp/r/process (deleted)
process  2034  user1  mem   REG    3,7  340771 40255 /lib/ld-2.1.3.so
process  2034  user1  mem   REG    3,7 4101836 40258 /lib/libc-2.1.3.so
process  2034  user1    0u  CHR  136,9         29484 /dev/pts/9
process  2034  user1    1u  CHR  136,9         29484 /dev/pts/9
process  2034  user1    2u  CHR  136,9         29484 /dev/pts/9
process  2034  user1    4r  CHR    5,0         29477 /dev/tty

However, if we check this same output by interrogating the /proc filesystem, we get much different results:

host # ls -l /proc/2034/fd/
total 0
lrwx------    1 user1   user1         64 Jul 30 15:16 0 -> /dev/pts/9
lrwx------    1 user1   user1         64 Jul 30 15:16 1 -> /dev/pts/9
lrwx------    1 user1   user1         64 Jul 30 15:16 2 -> /dev/pts/9
lrwx------    1 user1   user1         64 Jul 30 15:16 4 -> /dev/tty

So, we see that, although this one particular process has more than 4 "open files," it actually only has 4 "open file descriptors."

An easy way to iterate through each processes open file descriptors is to just run a simple shell loop, substituting your particular version of ps's arguments, like:

host # for x in `ps -ef| awk '{ print $2 }'`;do ls /proc/$x/fd;done

If you're only interested in the number of open file descriptors per process, you can shorten that output up even more:

host # for x in `ps -ef| awk '{ print $2 }'`;do ls /proc/$x/fd|wc -l;done

Here's to being able to over-answer that seemingly simple question in the future ;)

, Mike

Enumerating Files In The Linux or Unix Shell - More Improvements

Hey There,

Today, we're going to take a little more time and devote it to the folks on all the boards and networking sites who've made excellent suggestions for improvement on some of the shell/Perl scripts and other posts we've put out here over time. As was mentioned the first time we decided to start posting these suggestions and improvements, rather than update old posts that never get any attention, we'll be putting the good stuff out here with updated timestamps so that anyone who reads this blog doesn't ever have to wonder if an issue they find has been addressed. And, also, of course, as a thanks for some great tips for refinement.

Today we're going to look at a couple of improvements and refinements to the shell one-liner to enumerate file types, suggested by folks via email and on the boards over at linuxtoday.com, LXer.com, fsdaily.com, and many other venues.

Again, since our policy regarding privacy is to regard everyone's privacy as equal and well deserved, we will only be referring to the folks who contributed by the nicknames and/or screen-names they used in "talkbacks" which are already posted on the internet. And, then, only if it's relevant and unavoidable.

The suggestions for change to this one-liner were excellent, many, and more brief. They also indirectly pointed out the fact that, when I wrote it, I was obviously obsessing over awk ;) The original one liner was this:

find . -print|xargs file|awk '{$1="";x[$0]++;}END{for(y in x)printf("%d\t%s\n",x[y],y);}'|sort -nr

To this, it was first suggested that names with spaces in them wouldn't work. This is absolutely true, and can be countered using a variation on "xargs" in the command line, like this:

find . -print|xargs -Ivar file "var"|awk '{$1="";x[$0]++;}END{for(y in x)printf("%d\t%s\n",x[y],y);}'|sort -nr

This effectively "double quotes" the arguments passed to xargs, but, while I was thinking about that I realized that there would also be additional work you'd need to do for single quotes/apostrophes, etc, to keep them from screwing up the command chain, as well. It was beginning to seem more and more like a solution that could definitely use some re-tooling.

So, naturally, one suggestion I received was to do it without using xargs. Good deal: One less hassle, to my way of thinking, and removing a whole lot of issues that didn't have to exist. The difference here is the use of the -exec flag with the find command, rather than piping to xargs:

find . -print -exec file {} \;|awk '{$1="";x[$0]++;}END{for(y in x)printf("%d\t%s\n",x[y],y);}'|sort -nr

The next suggestion I received was to do it without using awk, but keeping xargs. This has the advantage of removing one additional external command (standard though it may be) from the process. And removing awk can make things a lot less confusing for most folks (myself included, which is probably why I used it originally. Not out of a twisted desire to cause myself grief, but to try and get more comfortable with it ;) That suggestion looked like this:

find . -print | xargs file -b | sort | uniq -c | sort -nr

But, this took us back to the xargs quoting and space-in-filename issue. This is the final suggestion that came from that community debate, which I think is probably the best (as in most succinct and utile) since it does it without awk, addresses the issues with xargs and can handle all the issues raised above:

find . -print0 | xargs -0r file -b | sort | uniq -c | sort -nr

If you want to check out this interaction, to gain some more insight into the thought behind each version, you can find it here on linuxtoday.com (at least for a while, assuming it will get moved eventually).

And, once again, a huge "Thank you" to anyone and everyone who's helpful criticism proved that, not only is there more than one way to skin a cat, there are far more efficient ways to do it than you or I may imagine in one sitting (but, please, don't skin any cats ;)

There's probably someone out there who knows a way to do it even better. Which is, of course, the beauty of Linux and Unix and why I enjoy working in the shell. It can be as simple or as complicated as you need it to be, and is flexible enough to allow users' the creativity to determine the path and the outcome of virtually everything that can be accomplished using either OS (or both :)

Have a great morning/day/afternoon/evening,

, Mike

Further Dissection Of Paging And Swapping On Linux And Unix

Hey again,

Believe it or not, I actually got a few emails about our previous post on paging and swapping in Linux or Unix because it wasn't specific enough ;) While I can certainly understand the frustration, I was hoping to explain the main differences as completely and concisely as possible. I seem to have faltered a bit on each front: I glossed over two specifics which, I agree, deserve some attention and I wrote yet another novel ;)

With that in mind (and with a prayer that my fingers won't type any more than they have to ;) I'd like to address, and/or clarify, the level of depth I didn't descend to in my last post on the difference between paging and swapping and write about the difference between paging and swapping (The redundancy was intentional and any resulting confusion is expected, given the topic at hand and my writing style ;)

As I mentioned previously, the terms paging and swapping are used almost interchangeably these days. Some industry manuals will actually talk about "swapping out pages" which seems to be contradictory and, theoretically, impossible if swapping and paging are two separate concepts with distinct and unique definitions. This is where language and implied meaning become a barrier to actual definition. And, all the more reason to clarify this one last bit of the puzzle.

And here they come. The extra clarifications...

1.  Difference in resident virtual memory management with paging and swapping.

When a system swaps a program, or process, it guarantees that it is resident (on disk or in memory) before it schedules it for execution, and will often hold onto the mapped resources reserved for that process from the time the process requests them until it notifies the scheduler that it is complete. When a system pages during the execution of a program, or process, there isn't any such direct correlation. You don't necessarily know (without specifically checking) how much of a process's virtual memory is resident or whether the process is entirely able to be scheduled for execution at the time paging begins. Pages can be selectively grabbed from a process (out of mapped physical memory) and never returned, unless re-requested by the process when it looks for the memory, can't find it and generates a page fault.

Phew... That's one down. Hopefully this isn't just becoming more confusing :)

2.  More specific definition of paging and swapping with regard to page ins/outs and swap ins/outs.

In this instance, swapping specifically refers only to the transfer of memory pages from physical memory to dedicated swap devices or swap disk (on most systems this is now referred to as swapfs - or a unique swap filesystem) and vice versa. Paging, on the other hand, refers to the transfer of memory pages from physical memory to disk (regular disk or swap disk) and vice versa. So, really, the major difference is that swapping is limited to only transferring memory pages back and forth between the physical memory and a dedicated swap device or filesystem, while paging can transfer between physical memory and any sort of disk device.

Hopefully, we've reached a sufficient amount of explanation at this point, and this thing won't turn into the monster I'd hoped it wouldn't become ;)

Thank you, everyone who wrote in, for your helpful input. As many folks have also noted, we don't have comments set up on this blog (because of issues with "comment spam" and not wanting to get shut down). If you ever want to leave a comment, or an objection, we welcome you to email us directly at our most often-check email address or, if you have a lot to say (or want to attach video, etc), sign up (for free) on our sister Linux and Unix Menagerie Forum, and we'll get your remarks there, as well. We do our best to reply, personally, to everyone who takes the time to write us. So far, I think we're still batting a thousand in that regard ;)

Best wishes,

, Mike

linux unix internet technology

Linux Update Patch For Expect User Removal Script

Hey There,

Today we have the "Linux Additions" patch to follow up on yesterday's post on removing old users network wide.

Note that this patch was created with "diff -c," so I've been sure to specify both the file I want to patch and the patch file in my arguments to "patch" (whereas I didn't need to in our post on easily patching multiple files). The exact command line to create the patch was:

host # diff -c eraser eraser.new > eraser.patch

All you'll need to do to update yesterday's script to include today's modifications is put the patch file (tagged on to the end of this post) in the same directory as the original script (or wherever you want, assuming you know how this all works and my explaining it is trivial ;) and name it something like "eraser.patch" - Then do the following:

host # ls
. .. eraser eraser.patch
host # cp eraser eraser.bak <--- Just in case something goes wrong, I'm always for making backups :)

As I noted before, you'll want to be specific in your arguments to "patch" when you run this, or be careful of "patch"'s default behaviour and don't choose to do a reverse patching, like this (BTW, even though my new script "eraser.new" is listed in the patch file, you don't actually need to have it for all of this to work):

host # patch -p0 -i eraser.patch
Hmm... Looks like a new-style context diff to me...
The text leading up to this was:
--------------------------
|*** eraser Sat Mar 29 18:57:29 2008
|--- eraser.new Sat Mar 29 18:57:50 2008
--------------------------
Patching file eraser using Plan A...
Reversed (or previously applied) patch detected! Assume -R? [y]
^C

Instead, patch it like this, so you can do it hands off:

host # patch -p0 eraser eraser.patch
Hmm... Looks like a new-style context diff to me...
The text leading up to this was:
--------------------------
|*** eraser Sat Mar 29 18:57:29 2008
|--- eraser.new Sat Mar 29 18:57:50 2008
--------------------------
Patching file eraser using Plan A...
Hunk #1 succeeded at 3.
Hunk #2 succeeded at 36.
Hunk #3 succeeded at 72.
Hunk #4 succeeded at 145.
Hunk #5 succeeded at 235.
Hunk #6 succeeded at 267.
done

And "diff -c" will show that your "eraser" Expect script is now different than the original that you copied off before:

host # diff -c eraser eraser.bak
*** eraser Sat Mar 29 19:02:21 2008
--- eraser.bak Sat Mar 29 18:57:50 2008
***************
...

And, of course, you can just check your "eraser" file and see the changes. If you need to back out the patch, simply do the "reverse patch" operation, like so:

host # patch -R -p0 eraser eraser.patch
Hmm... Looks like a new-style context diff to me...
The text leading up to this was:
--------------------------
|*** eraser Sat Mar 29 18:57:29 2008
|--- eraser.new Sat Mar 29 18:57:50 2008
--------------------------
Patching file eraser using Plan A...
Hunk #1 succeeded at 3.
Hunk #2 succeeded at 36.
Hunk #3 succeeded at 70.
Hunk #4 succeeded at 130.
Hunk #5 succeeded at 186.
Hunk #6 succeeded at 217.
done

And then do a cursory check to make sure the unpatched file matches the original that you copied off before:

host # diff -c eraser eraser.bak <--- You'll get no output if the files are the same. That's good :)

Enjoy and have a restful Sunday :)





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

*** eraser        Sat Mar 29 18:57:29 2008
--- eraser.new  Sat Mar 29 18:57:50 2008
***************
*** 3,9 ****
  log_user 0

  #########################################################################
! # eraser - root out ex-employees across network
  # 2008 - Mike Golvach - eggi@comcast.net
  # <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/us/">Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License</a>
  #
--- 3,9 ----
  log_user 0

  #########################################################################
! # eraser - root out ex-employees across network - with Linux procs added
  # 2008 - Mike Golvach - eggi@comcast.net
  # <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/us/">Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License</a>
  #
***************
*** 36,41 ****
--- 36,43 ----
    puts " sun1 sun2"
    puts "HP Hosts:"
    puts " hp1 hp2"
+   puts "LINUX Hosts:"
+   puts " linux1 linux2"
    puts "#############################################################################"
    puts "FLAGS:"
    puts " -h: Show this message."
***************
*** 70,75 ****
--- 72,90 ----
   expect "word: " {send "$qpass\r"}
  }

+ proc linux_login {name tprompt login lpass} {
+  send_user "$name\n"
+
+  set timeout 3
+  set hostname $name
+  set prompt $tprompt
+  set qlogin $login
+  set qpass $lpass
+
+  expect "ogin: " {send "$qlogin\r"}
+  expect "word: " {send "$qpass\r"}
+ }
+
  proc hp_login {tprompt login lpass} {

   set timeout 12
***************
*** 130,135 ****
--- 145,184 ----
   }
   return "$hostname $insidepwd $insidealias"
  }
+
+ proc linux_q {tprompt login lpass userlogin host} {
+  set timeout 5
+  set log $login
+  set opass $lpass
+  set prompt $tprompt
+  set ulogin $userlogin
+  set hostname $host
+  set insidealias none
+  set insidepwd none
+
+  send_user "Scanning $hostname...  "
+
+         expect -re $tprompt {send "grep $ulogin /etc/passwd\r"}
+  expect -re $tprompt
+         send "echo $?\r"
+         expect -re "\r\n(.*)\r\n"
+         set returnval $expect_out(1,string)
+         if { $returnval == 0 }  {
+   set insidepwd 1
+         }
+         expect -re $tprompt {send "grep $ulogin /etc/mail/aliases\r"}
+  expect -re $tprompt
+         send "echo $?\r"
+         expect -re "\r\n(.*)\r\n"
+         set returnval $expect_out(1,string)
+         if { $returnval == 0 }  {
+   set insidealias 1
+         }
+         expect -re $tprompt {
+   send "exit\r"
+  }
+  return "$hostname $insidepwd $insidealias"
+ }

  proc sco_q {tprompt login lpass userlogin host} {
   set timeout 5
***************
*** 186,191 ****
--- 235,241 ----
  set sun [list sun1 sun2]
  set sco [list sco1 sco2]
  set hp [list hp1 hp2]
+ set linux [list linux1 linux2]

  set userlogin [lindex $argv 0]
  set login [lindex $argv 1]
***************
*** 217,222 ****
--- 267,278 ----
   spawn telnet $host
   hp_login $tprompt $login $lpass
   lappend finalists [sunhp_q $tprompt $login $lpass $userlogin $host]
+ }
+
+ foreach host $linux {
+  spawn telnet $host
+  linux_login $tprompt $login $lpass
+  lappend finalists [linux_q $tprompt $login $lpass $userlogin $host]
  }

  foreach host $sco {

, Mike

linux unix internet technology

Easy Multiple File Patching And Patch Removal On Linux And Unix

Hello again,

Today we're going to continue with yesterday's post on manually mass patching files on Linux and Unix, but come at it from a different, and much simpler, angle.

First things first; you can delete the ed patch files from yesterday :) When you look at them, you can see that using "diff -e" to create an ed patch file creates a patch that contains absolutely no information that would allow you to reverse a change that you made, as opposed to a patch file made from diff straight-up. The small comparison below points this up more clearly:

host #diff -e tmpdir1/file1 tmpdir2/file1
5c
BASEDIR="/usr/binky"
.
host # diff tmpdir1/file1 tmpdir2/file1
5c5
< HOMEDIR="/usr/binky"
---
> BASEDIR="/usr/binky"

The ed patch file type that we used yesterday, although simpler to understand, doesn't make it possible to reverse your changes without keeping backup copies of your files.

The good news is that patch files created with diff can still be used to patch massive amounts of files and recover them even if the original, unpatched, files get lost or deleted. The method used to obtain these results is simpler than what we walked through yesterday and only requires the use of the "diff" and "patch" commands. Both of these commands should come standard with your OS. They've been on Linux for a long time and have been on Solaris Unix since, at least, release 2.6 (probably earlier).

So, let's get started patching lots of files and then backing out those patches. Since yesterday's post was so long (and this one has the same potential), I'm going to use only 2 files as the base number of files (although the number of files can be however large you want) and grep out the relevant information for display purposes. Hopefully it will save us all some eye strain ;)

The first thing we'll want to do is copy all the scripts we want to patch into a new directory to work on them. We'll also put the new scripts (that we'll need to create our diff patches) in yet another directory. We'll never work directly on the scripts in their native directory until we're sure they're patched correctly. This isn't absolutely necessary, but is generally good practice. No sense in letting a simple mistake cost you any more time, or grief, than it has to. Again, the only thing that is different between our existing scripts and the new scripts is that the HOMEDIR variable has been changed to BASEDIR.

host # cp scriptdir/* tmpdir1/

<--- This could be any number of files.

The following is our setup, with only one difference between the files, as noted above:

host # ls tmpdir1
.  ..  file1  file2
host # ls tmpdir2
.  ..  file1  file2
host # grep DIR tmpdir1/*
tmpdir1/file1:HOMEDIR="/usr/binky"
tmpdir1/file2:HOMEDIR="/usr/binky"
host # grep DIR tmpdir2/*
tmpdir2/file1:BASEDIR="/usr/binky"
tmpdir2/file2:BASEDIR="/usr/binky"

Next, we'll use diff to create a patch file. Although, rather than just doing diff straight-up, we're going to run it in "contextual" mode. When you invoke "diff -c" it creates a contextual diff which, literally, means that it puts the diff output in context (so you can see the lines before and after the lines that differ). The main reason I like to use this option is that the output works with "patch" when patching multiple files from multiple directories. The output from a standard diff of multiple files in multiple directories doesn't work well for this (mostly because it puts all the file names on one line and "patch" attempts to find a file named "diff tmpdir1/file tmpdir2/file" - literally. And that file can never exist (I hope ;)

Now we'll create the multiple file patch and examine its contents, created at the directory level directly above tmpdir1 and tmpdir2, so we can get all the files in both directories (Note that only the lines beginning with the exclamation point (!) in the diff output are different. The lines above and below only serve to showcase the line in its context within the file):

host # diff -c tmpdir1 tmpdir2  >patchfile.patch
host # cat patchfile.patch
diff -c tmpdir1/file1 vtmpdir2/file1
*** tmpdir1/file1     Wed Mar 26 14:36:05 2008
--- tmpdir2/file1     Wed Mar 26 14:54:10 2008
***************
*** 2,8 ****

  COMMAND_ARGS="-d --takeforeverandaday"

! HOMEDIR="/usr/binky"

  case $x in

--- 2,8 ----

  COMMAND_ARGS="-d --takeforeverandaday"

! BASEDIR="/usr/binky"

  case $x in

diff -c tmpdir1/file2 vtmpdir2/file2
*** tmpdir1/file2     Wed Mar 26 14:36:05 2008
--- tmpdir2/file2     Wed Mar 26 14:54:10 2008
***************
*** 2,8 ****

  COMMAND_ARGS="-d --takeforeverandaday"

! HOMEDIR="/usr/binky"

  case $x in

--- 2,8 ----

  COMMAND_ARGS="-d --takeforeverandaday"

! BASEDIR="/usr/binky"

  case $x in

Now we're ready to patch all of the files in tmpdir1 at once, using the simple form of the command (patch -i PATCHFILE), and receive an error for doing so. Note that we're running this from the directory above tmpdir1 and tmpdir2; exactly where we were when we used "diff -c" to create the patch file:

host # patch -i patchfile.patch
can't find file to patch at input line 4
Perhaps you should have used the -p or --strip option?
The text leading up to this was:
--------------------------
|diff -c tmpdir1/file1 tmpdir2/file1
|*** tmpdir1/file1    Wed Mar 26 14:36:05 2008
|--- tmpdir2/file1    Wed Mar 26 14:54:10 2008
--------------------------
File to patch: ^C 

<--- Type the control (ctl) key + C, or any other escape/control key combination to break out of this prompt

This error killed us before it actually made any changes because of the way "patch" works. When run without any options (other than -i, which we used to indicate the name of our patch file), "patch" parses the patch file and strips the file names down to the base, in much the same way the "basename" command does. So, even though the file name in the patch file is "tmpdir/file1," the "patch" program is looking for a file named "file1" and it's looking for it in the directory we're in which, unfortunately, isn't where the file is.

Luckily, this little setback is easy to remedy. Using the -p option to "patch" we can instruct "patch" how to interpret the file names in our patch file. As we noted, when the option isn't present, "patch" reverts to "basename" type behaviour. If we used -p1, we would be instructing "patch" to remove the leading slash (/) from the file name. We're going to use -p0, which instructs "patch" to not interpret the file name and just take it as it is (In this case "tmpdir/file1," which is relative, but just fine considering where we're running the command from).

host # patch -p0 -i patchfile.patch
patching file tmpdir1/file1
patching file tmpdir1/file2

Success! Now, let's check that the patch actually took:

host # grep DIR tmpdir1/*
tmpdir1/file1:BASEDIR="/usr/binky"
tmpdir1/file2:BASEDIR="/usr/binky"

Excellent! HOMEDIR is now BASEDIR. Alas, as I intimated in our post yesterday on mass file updating, our boss has, only minutes later, decided that the BASEDIR variable really should be HOMEDIR after all. He's not going to explain why, we just need to switch everything back now ;)

And this is where our method of execution really pays off. Assuming we held on to that patch file, we can now use "patch" to put everything back the way it was in short order, by simply adding the -R (to reverse the patch operations) to the command line and running it again, like so:

host # patch -p0 -R -i patchfile.patch
patching file tmpdir1/file1
patching file tmpdir1/file2

And, then, just to verify that the patch differences have been removed:

host # grep DIR tmpdir1/*
tmpdir1/file1:HOMEDIR="/usr/binky"
tmpdir1/file2:HOMEDIR="/usr/binky"

And we're all set :) Hopefully this follow-up tutorial was easy enough to follow, and you can find some good use for it in your work routine. BTW, don't forget to copy the changed scripts back to the real script directory, but only after copying that directory off somewhere else, again. But only if you're as paranoid as I am ;)

Best Wishes,

, Mike

linux unix internet technology