Making Use Of Lazarus' Web Interface On Linux Or Unix

Hey There,

Today's post is a follow up (and, in many respects, a point of closure) to our previous posts on using TCT's unrm to recover lost data and using TCT's Lazarus to make data recovery using TCT's unrm a bit simpler (both posts are filled with sentences even more convoluted than that one ;)

As we noted in those previous posts, Lazarus comes with the option to restore from tagged blocks using an HTML interface (Please see the preceding two posts for any back-explanation. It took two posts to cover all the details, and this post won't benefit from being 6 screen length's longer ;). At first this may seem trivial. In point of fact, it is to a certain degree. Everything that's done for you via the web browser, you could do yourself. However, this might be a bear of a wheel to reinvent ;)

We're using the same recovered data blocks that we used in our first two posts (so everything will tie together nicely if you need to refer back to them) to showcase what the HTML output from Lazarus can produce. Also, one should note that producing the HTML output (in addition to the tagged blocks) didn't seem to add any noticeable time to Lazarus' block-tagging process (which takes so freakin' long, you'll walk away eventually ;) and can be added by simply inserting the "-h" flag into your command line. So:

host # ./lazarus /usr/local/recovery/the_found_file_I_hope

becomes

host # ./lazarus -h /usr/local/recovery/the_found_file_I_hope

NOTE: Although (since we didn't specify on the command line) the default directory for block restoration is under TCT's main directory (the one with bin, etc in it - see the previous post), the main HTML files get created in the directory you named on the command line! In our case: /usr/local/recovery

Below, we're going to walk through a few different ways you can get to your recovered data more easily (or browse through all the stuff you thought you'd gotten rid of or never knew was there ;) using the HTML files created by Lazarus. The first thing to note (which is very easy to see when you do an "ls" on /usr/local/recovery) is that Lazarus creates three main HTML files:

host # ls /usr/local/recovery
the_found_file_I_hope             the_found_file_I_hope.html
the_found_file_I_hope.frame.html  the_found_file_I_hope.menu.html

You'll want to open the "frame" HTML page so that the upper menu and main screen are both visible. Every single link in the main screen is already created (assuming you've let Lazarus finish ;) in TCT's main directory under the "www" directory. Each block has a frame and a menu HTML page. The main page (when you select a particular hyperlink to a block) is actually just the block itself viewed through a browser. This works out very nicely in some ways, as we'll see below. Enough with the explaining. Let's go :)

The first picture in our gallery is the main screen you'd see (the_found_file_I_hope.frame.html), as brought up on a local browser. The cartoonish arrows and big letters were drawn in by yours truly using The Gimp's line tool. The same with the blanked out parts. It's a much better tool than we make it seem. Thankfully, we don't work in graphic design ;)

Click any of the below Pictures to return them to their original gigantic size :)



The next two pictures demonstrate what executable data looks like. It's not very impressive in the browser, but you'll note that the menu allows you to follow blocks of a similar type, so you can just hop from one to the next and note those numbers so you'll have an easier go of recreating a binary executable from its individual blocks (if it spans more than one) when you get back to the command line. Of course, you can do that just as simply by grepping out the block numbers and types from the available files in the blocks directory. Purists may note that one of these "exe" blocks is an "elf" block. I have no idea how Lazarus makes the distinction, but I'm not going to complain about it, either... The next stuff is cooler - I promise :)




This next picture shows how a recovered HTML block would look (in HTML, but not parsed as such):



And this picture shows us how a "program" (What it says in the menu) looks like. Every one we found before boredom set in was actually a shell (or someother kind of) script, which would account for the "exe," "elf," and generic "binary" types being represented.



And, neatest of all, most of the pictures we found came up just like this one; no guessing involved (unless your browser's MIME types or associations are set up non-standardly). We have no idea who this picture belonged to. Probably the last guy who used this machine:



The moral of the story? I don't suppose there is one, since this is just a demonstration of easy data recovery, but, if this were a morality play, I suppose the moral would be: Be sure to scrub your disk before you leave it behind. Some goofballs may come around later and find stuff you didn't want them to ;)

Hope you enjoyed the slide-show!

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Why DU And DF Display Different Values On Linux And Unix

Hey there,

Today we're going to look at a little something that is a fairly hot-running-water issue on most of the Linux and Unix boards lately (actually, probably always has been, but our research staff quit on us ;) This post will be similar in focus to our previous post on the differences between sar and vmstat with regards to free memory/swap reporting.

Today's post is similar in spirit, but not a replay of that previous memory/swap reporting issue. Today, we're going to take a look at two other commands that, seemingly, measure and display the same information, although with (sometimes) huge discrepancies in output. Those two commands are du and df.

The question you'll most often see (or, perhaps, have :) is something to the effect of "Why do my outputs from du and df differ? One says I'm using more disk space than the other. Which of them is correct?" Generally you'll find that df shows more disk spaced used than du does, but the case can sometimes be the opposite. It's very rare (unless you don't use your computer, and it doesn't use itself, at all ;) that the output from the two commands match. It's actually rare that they ever come close to matching. Generally, the longer a machine is up, the greater the rift between figures becomes.

The bad news: This is confusing and sometimes hard to communicate to others, even when you know why the situation exists :(

The good news: This is normal and can be explained; perhaps even simply :)

To set up the "typical" situation, we used a Solaris 10 box (although this issue is common on all proprietary Unix and Linux distro's). Below, the output from four commands executed in the /opt directory (NOTE: For du, the "-s" flag is used to print summary information for the entire object, rather than a listing of all of its parts (i.e. the whole directory rather than all the files and subdirectories)):

host # cd /opt <-- For df, look at the "used" column and for du, look at the only output there is ;)

host # df -h .
Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c0t0d0s6      3.9G   490M   3.4G    13%    /opt
host # du -sh
 486M   .
host # df -k .
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0t0d0s6    4130238  502180 3586756    13%    /opt
host # du -sk
498068  .

Now, in two parts, the complicated reasons, explanation and summation of why this disparity exists, followed by the more customer/user-friendly version of the exact same thing :)

1. The convoluted, complicated and hard to pass-along explanation:

Reasons: While du and df report "approximately" the same information, they don't report "exactly" the same information. While they both report "about" the "same things" (some of the time), they don't measure those "things" using equal methods or metrics.

Explanation: These differences can occur for a number of reasons, including (but not limited to):

a) Overlay mounts, which can skew df output when it's run from a higher level directory (e.g. df -k /opt would not report the total space used on /opt if there were overlay mounts of /opt/something and /opt/anotherthing on the system - du will. If you actually need the total space used by /opt and everything underneath it, including the overlay mounts, df can do it, but it takes some plate-spinning ;).

b) Sometimes (This actually should never happen on any "recent" OS) hidden files and subdirectories could skew du's output.

c) Unlinked inodes can cause unexpected statistics with both df and du (For, instance, in the situation where your filesystem shows almost 100% free space but all the inodes are in use.)

d) You're using the "-h" flag instead of the "-k" flag for one or the other commands (or both). The "-h" (human readable) flag can sometimes make things seem worse (or more divergent) than they are. Since it tries to make the output more easily digestible, it will round the numbers you see, so you can't "really" be sure whether "5.1 GB" is closer to 5 GB or 5.2 GB. "-k" is slightly more likely to produce relatively equal results, as it reports the size in kb. It still does do rounding so that you only get straight up integers and no floating point results, but it's generally better to check with if your output from "-h" is very close, or even the same (since it might not be). If your implementation of df and/or du supports the "-v" flag (or something similar), that's even better since it reports in multiples of your system blocksize and is even more exact.

e) The fundamental way in which each of the commands work:

i) df:

df reports only on the mount point, or filesystem, level. So a df on /opt would produce the same results as a df on /opt/csw (assuming, as noted above, that they're both on the same partition):

host # df -k /opt
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0t0d0s6    4130238  502180 3586756    13%    /opt
host # df -k /opt/csw
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0t0d0s6    4130238  502180 3586756    13%    /opt

df gets most of its information from a filesystem's primary superblock (except in the odd instance that an alternate superblock is being used - although this would only happen in an "fsck" situation. After that, the information from the alternate superblock would be copied back to the primary, and all other superblocks). It takes this information at face value, which is to say that it does not question the information provided to it by the primary superblock. In this respect, df is a very fast tool for getting disk usage information (at the cost of reliability).

df will include open files (in memory, but not on disk), data/index files (used for data management - sometimes using approximately 2 to 5% of each filesystem) and unnamed files in its size calculation. This is one reason why, sometimes (although not very often), df can show a larger amount of disk used than du does.

df, as per above, will explicitly trust any errors in space calculation that may have occurred over time, since it trusts the primary superblock entirely. This means that if you've fsck'ed a filesystem (and not resynced or rebooted since), and/or have experienced any hard or soft errors on the device/disk housing the filesystem, you're measuring (again, assuming you haven't rebooted since they happened) and/or experience any possible corruption or inconsistencies in any filesystem-state records (like /etc/mnttab), your output will be commensurately incorrect.

df sometimes reports file sizes incorrectly, as it works on what we like to call the whole-enchilada-principle ;) Basically, if you take your filesystem's block size (for the /opt filesystem here - different filesystems may have different block sizes), which you can find by executing either of the following commands, both of them, or whatever works on your OS:

host # df -g /opt|grep "block size"|awk '{print $4}'
8192
host # fstyp -v /dev/rdsk/c0t0d0s6|grep "^bsize"|awk '{print $2}'
8192

...you'll be able to do this experiment on your own (we'll leave out the grisly details to save on space :). In our instance, we have a default block size of 8192 bits or 8 kb. Now, here's where it gets somewhat interesting ;) If you create a new file that's 1 kb in size and it writes to a new, or - depending - not fully used, block, df will report that file as being 8 kb in size, even though it's actually only 1 kb in size!

ii) du:

du reports at the "object" level rather than at the filesystem/mountpoint level, as df does. So, to repeat the example from above, if you run du on /opt and /opt/csw, you'll get different results. I find it easier to think of du as handling its measurement via a simple "object" model. The main partition would be the meta-object, while any subdirectory you may be running du against would be considered a sub-object of the filesystem meta-object (you'll note that the du size output for /opt/csw is, naturally, smaller than that for the entirety of /opt):

host # du -sk /opt
498068  /opt
host # du -sk /opt/csw
64065   /opt/csw

du gets its information at the time you execute it (unless you run it repeatedly in succession, where you'll notice a slight performance improvement). To test this, run du on a partition, then wait 5 minutes and run it again. It should take just as long as the first time (unless you've added lots of files since then). In this respect, du can be a very slow tool for getting disk usage information (with the benefit that your information will be more accurate). It should be noted that, because of the way it takes measure of most (see below) filesystem objects, it takes much longer for it to report the size of a billion 1 kb files than it does to report the size of one file of 1 billion kb size.

du does not count data/index files or open files (in memory, but not on disk).

du does not take into account any information "supplied" by the system (meaning the information, like from the superblock, as listed under the df section) and gets its information independent of whatever the system thinks is correct.

du does not rely on block size (see the whole-enchilada-principle in df's section above) to determine file size. So, if you have a default 8 kb block size on your filesystem, you create a new 1 kb file that writes to an empty 8 kb block, du will report that file as being 1 kb in size and "not" assume a minimum size of the filesystem block size. This is worth remembering, because it can cause a great deal of difference in the filesystem "usage" size between du and df (which would consider that 1 kb file an 8 kb file - 8 times larger than it actually is)!

du is more reliable if you want to know the state of your filesystem "right now." It doesn't count any data/index blocks .

Summation: If you are interested in knowing exactly how much of your filesystem is actually being used, du is a much more accurate tool for collecting and displaying this information. Note, however, that - since du does "on demand" filesystem size reporting, it is much slower than df. Also, du does not play as well with some system internal files and processes since it essentially ignores information reported by the primary superblock, system mount information tables, etc. Ultimately, the purpose for which you need to determine your filesystem's size (coupled with an understanding of the "Explanation" section for both utilities) is the best way to decide which utility to use in any given situation.

2. (Did you forget this was coming, too? ;) The simple, and easy to convey, explanation:

Reasons: df and du rely on different information to determine how much disk space is used on a filesystem.

Explanation: df and du report filesystem information differently, for very basic reasons:

df and du don't use the same yardsticks to measure filesystem size.

df:

df relies mostly on system information, supplied by various files and built-in reporting mechanisms that may, or may not, be correct at any given time.

du:

du relies on what it can "see" at the particular moment in time that you run it.

Summation: du is the better tool to use if you are interested in knowing how much space is actually being used on your filesystem "right now." df is great for "ballpark estimates" and is preferred if you need to know how big df thinks your filesystem is (so it will agree with other incorrect system statistics).

3. The really easy, and simple to blurt-out, explanation:

If du and df don't agree on what size your filesystem is, du is more correct than df is.

See; it's all very simple ;)

Cheers,

, Mike

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Securing All Of A Host's Network Programs At Once Using Extended ACL's

At most shops where I've worked, a lot of time and effort is put into securing this and that network service from certain users. Mostly, it's done on a per process basis. Like segregating Telnet or RSH, since those are considered insecure. But, for the most part, I find, what security folk really want to do, when locking down a box, is to prevent regular users from running any network commands, and only allow certain people in certain groups to avail themselves of those services. This can be likened to most firewall setups: Block everything and allow what you need to, as opposed to allowing everything and blocking what you find out is dangerous (which can be disastrous)

This is actually very simple to accomplish on both Linux and Solaris by using simple group permissions and File ACL's (or facl's). In this post we'll walk, step by step, through setting up a box with network services totally locked down for any users that we don't explicitly want to use them. Here we go:

First, you'll want to work on the group angle. All users are already members of a group (or groups) and these groups are used to limit (or augment) their access to certain areas of the OS and certain programs. For our purposes here, we'll set up an additional group. We'll call it noudptcp for kicks (since we're not going to let anyone who has this group as their primary group use any udp or tcp-based programs). We'll set it up just like any other group, like so:

bash@host # groupadd -g 999 noudptcp

Next, we'll visit the two critical files that we'll need to change in order to pull this all together in the next few steps: /dev/udp and /dev/tcp. They should look like this by default:

bash@host # ls -lL /dev/udp /dev/tcp
crw-rw-rw- 1 root sys 41, 0 Oct 2 2006 /dev/udp
crw-rw-rw- 1 root sys 42, 0 Oct 2 2006 /dev/tcp

Note that both of these character special files (device drivers) are readable and writable by everyone. This is one of the reasons any user can use Telnet, RSH, SSH and many other network-based programs on your system. Here, we'll run the getfacl command to view the ACL for the /dev/tcp device driver file and take a look at it, like so:

bash@host # getfacl /dev/tcp

# file: /dev/tcp
# owner: root
# group: sys
user::rw-
group::rw- #effective:rw-
mask:rw-
other:rw-

Then, we'll augment the ACL on both files (they're essentially the same) my favorite way. I like to just take the output of getfacl, modify it in a file, and then feed that file to setfacl (Note that I'm doing this on Solaris and that I will point out the differences for Linux along the way - for instance, the Linux getfacl output will look slightly different than this, but you can still dump it to a file and modify it), like so:

bash@host # cat FACL_FILE <--- We'll assume I already edited it, because that's so hard to emulate using a simple keyboard ;) -- Note also that I've removed the comments, as they don't get applied by setfacl and may be confusing.

user::rw-
group::rw- #effective:rw-
group:noudptcp:--- #effective:---
mask:rw-
other:rw-

Now, we feed this file to setfacl (Note that we only added the one extra line to set permissions for the noudptcp group) and the ACL will be updated so that members of the noudptcp group won't have any access to either /dev/tcp or /dev/udp (In Linux, use -M instead of -f):

bash@host # setfacl -f FACL_FILE /dev/tcp
bash@host # setfacl -f FACL_FILE /dev/udp

And now we can use getfacl to show the new permissions. In the "ls -lL" output, you'll notice the "+" at the end of the file listing, indicating that the default ACL has been modified, and the output of the getfacl command will be the same as our FACL_FILE, with the comments added by default:

bash@host # ls -lL /dev/udp /dev/tcp
crw-rw-rw-+ 1 root sys 41, 0 Oct 2 2006 /dev/udp
crw-rw-rw-+ 1 root sys 42, 0 Oct 2 2006 /dev/tcp
bash@host # getfacl /dev/tcp

# file: /dev/tcp
# owner: root
# group: sys
user::rw-
group::rw- #effective:rw-
group:noudptcp:--- #effective:---
mask:rw-
other:rw-

Finally, all we have to do is add any users we deem necessary to the noudptcp group and they will not be able to access any network services! Note that this will not prevent them from connecting "to" the box; it will just prevent them from connecting to another box from your box, or using ping, or any program that needs to access /dev/tcp or /dev/udp.

For instance, a few select lines of truss output show what happens when a user in the noudptcp group attempts to SSH off of the box:

open("/dev/udp", O_RDONLY) Err#13 EACCES
so_socket(2, 2, 0, "", 1) Err#13 EACCES
write(2, 0xFFBEF058, 27) = 27
s o c k e t : P e r m i s s i o n d e n i e d\r\n

Congratulations! Now everyone you don't want to be able to use network services on your box is no longer a threat. At least, not in that way ;) And, even better, they can still use everything else that they would normally be allowed to!

Cheers,

, Mike

linux unix internet technology