Showing posts with label lazarus. Show all posts
Showing posts with label lazarus. Show all posts

Tuesday, June 16, 2009

Making Use Of Lazarus' Web Interface On Linux Or Unix

Hey There,

Today's post is a follow up (and, in many respects, a point of closure) to our previous posts on using TCT's unrm to recover lost data and using TCT's Lazarus to make data recovery using TCT's unrm a bit simpler (both posts are filled with sentences even more convoluted than that one ;)

As we noted in those previous posts, Lazarus comes with the option to restore from tagged blocks using an HTML interface (Please see the preceding two posts for any back-explanation. It took two posts to cover all the details, and this post won't benefit from being 6 screen length's longer ;). At first this may seem trivial. In point of fact, it is to a certain degree. Everything that's done for you via the web browser, you could do yourself. However, this might be a bear of a wheel to reinvent ;)

We're using the same recovered data blocks that we used in our first two posts (so everything will tie together nicely if you need to refer back to them) to showcase what the HTML output from Lazarus can produce. Also, one should note that producing the HTML output (in addition to the tagged blocks) didn't seem to add any noticeable time to Lazarus' block-tagging process (which takes so freakin' long, you'll walk away eventually ;) and can be added by simply inserting the "-h" flag into your command line. So:

host # ./lazarus /usr/local/recovery/the_found_file_I_hope

becomes

host # ./lazarus -h /usr/local/recovery/the_found_file_I_hope

NOTE: Although (since we didn't specify on the command line) the default directory for block restoration is under TCT's main directory (the one with bin, etc in it - see the previous post), the main HTML files get created in the directory you named on the command line! In our case: /usr/local/recovery

Below, we're going to walk through a few different ways you can get to your recovered data more easily (or browse through all the stuff you thought you'd gotten rid of or never knew was there ;) using the HTML files created by Lazarus. The first thing to note (which is very easy to see when you do an "ls" on /usr/local/recovery) is that Lazarus creates three main HTML files:

host # ls /usr/local/recovery
the_found_file_I_hope the_found_file_I_hope.html
the_found_file_I_hope.frame.html the_found_file_I_hope.menu.html


You'll want to open the "frame" HTML page so that the upper menu and main screen are both visible. Every single link in the main screen is already created (assuming you've let Lazarus finish ;) in TCT's main directory under the "www" directory. Each block has a frame and a menu HTML page. The main page (when you select a particular hyperlink to a block) is actually just the block itself viewed through a browser. This works out very nicely in some ways, as we'll see below. Enough with the explaining. Let's go :)

The first picture in our gallery is the main screen you'd see (the_found_file_I_hope.frame.html), as brought up on a local browser. The cartoonish arrows and big letters were drawn in by yours truly using The Gimp's line tool. The same with the blanked out parts. It's a much better tool than we make it seem. Thankfully, we don't work in graphic design ;)

Click any of the below Pictures to return them to their original gigantic size :)

The Kids Menu

The next two pictures demonstrate what executable data looks like. It's not very impressive in the browser, but you'll note that the menu allows you to follow blocks of a similar type, so you can just hop from one to the next and note those numbers so you'll have an easier go of recreating a binary executable from its individual blocks (if it spans more than one) when you get back to the command line. Of course, you can do that just as simply by grepping out the block numbers and types from the available files in the blocks directory. Purists may note that one of these "exe" blocks is an "elf" block. I have no idea how Lazarus makes the distinction, but I'm not going to complain about it, either... The next stuff is cooler - I promise :)

Binary makes for good reading
Keep the naughty bits coming

This next picture shows how a recovered HTML block would look (in HTML, but not parsed as such):

html as plaintext: A study in regression

And this picture shows us how a "program" (What it says in the menu) looks like. Every one we found before boredom set in was actually a shell (or someother kind of) script, which would account for the "exe," "elf," and generic "binary" types being represented.

This script was written to distract you

And, neatest of all, most of the pictures we found came up just like this one; no guessing involved (unless your browser's MIME types or associations are set up non-standardly). We have no idea who this picture belonged to. Probably the last guy who used this machine:

Objects in this diagram may be closer than they appear

The moral of the story? I don't suppose there is one, since this is just a demonstration of easy data recovery, but, if this were a morality play, I suppose the moral would be: Be sure to scrub your disk before you leave it behind. Some goofballs may come around later and find stuff you didn't want them to ;)

Hope you enjoyed the slide-show!

Cheers,

, Mike


Banner for College Student-Oriented Sites (728 version 1)



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, May 20, 2009

Using TCT To Recover Lost Data On Linux Or Unix - Part Two

Hey there,

Today's post is a follow up to yesterday's post (conspicuously titled Recovering Lost Data On Linux or Unix Using TCT, or something like that ;). Please refer to that post for the 3 or 4 paragraphs of over-explanation of some of the minutiae which may or may not be helpful to you :)

Today, we'll move on to the second (easier, but more time-consuming) method of recovering your deleted data (on any Linux or Unix system) using The Coroner's Toolkit (TCT). Today, we'll be using lazarus to make file recovery somewhat simpler. Again, please see yesterday's post on Recovering Lost Data if you're recovering simple text files and/or just want to use unrm and be done with it.

The situation today, will be the same as yesterday.

THE SITUATION: You've created a text file, containing valuable information that you couldn't commit to memory, using your favorite text editor and saved it. Then, an hour or so later, you accidentally deleted it, realizing that you'd completely screwed up just seconds after pressing the enter key. Deja Vu? ;)

host # cat /usr/THE_ALMOST_LOST_FILE
we'll just put some
semi-random text in
here to see if we can
find this later with
grep. For simplicity's
sake, we'll include the
word semi-unusual so that we
have something in this
file that probably won't
be in any other files
host # rm /usr/THE_ALMOST_LOST_FILE
host # cat /usr/THE_ALMOST_LOST_FILE
cat: cannot open /usr/THE_ALMOST_LOST_FILE


Now, since we've already created our recovery area (required to be on a separate partition than the one on which we lost the data) and have run unrm to create that one gigantic file composed of all the free blocks on the partition where we accidentally deleted our file, we're ready to make the process of data recovery simpler using lazarus (Longest sentence ever? Maybe not ;)

Lazarus is a simple tool to run, but it does come with a few caveats:

1. Unlike unrm(not a double-negative ;), which only requires that you have 100% of the free space available on the partition where you deleted your file, available on your recovery partition, lazarus requires you to have 220% of that space available. These sentences are "killing" me! Apologies for any confusion caused by their Byzantine structure ;)

2. Lazarus depends on the output from unrm, so you'll need to run that first. Technically you can run lazarus against any file, but your results may be less than satisfactory.

3. Lazarus picks apart the gigantic block-file created by unrm and separates it into individual files and tags those files as being of a certain type (text, audio, HTML, C code, etc). This makes it take a very very long time to complete execution. Depending upon the amount of free space on the partition on which you deleted the file you wish to recover, you may be waiting days (literally) for lazarus to complete its work!

Below is a listing of the common file types lazarus will recognize. It will assign the corresponding letters to the files it cranks out. As an example, if it finds a block that's composed of "unresolved text" it will save it as: BLOCKNUMBER.TYPE.txt (All files will have the extension .txt, although lazarus does allow you to produce HTML output instead, with the -h flag). So, if it recovered block number 3714, which happened to be tar/cpio, etc file, it would name it: 3714.a.txt

THE LAZARUS BLOCK FILE OUTPUT LEGEND:

A "." represents unrecognized binary blocks of data.
type value color meaning

t 777777 gray unresolved text
f ff0000 bright red (alarm) sniffer stuff
m 0066ff blue mail
q 6633ff pale blue mailq files
s 6699ff purply emacs/lisp
p cc6666 greenish program file
c 336666 green C code
h ff99ff light purple HTML
w cc3333 reddish password file
l cc9900 light brown log file

Binary files are represented by:

type value color meaning
o bbbbbb light grey null block
r 000000 black removed block
x 000000 black binary exe
e d9d9i9 gold ELF
i 238e68 greenish JPG/GIF
a d19275 black cpio/tar/etc
z 336633 greenish compressed
! 000000 black audio


When you run lazarus, it can be as simple as just invoking the command (assuming you've run unrm already). Using our example from yesterday's post on Data Recovery, we could invoke it using the method below (and then kick back and wait and wait and wait...):

NOTE: The files, etc, used in these examples are the same as in yesterday's post. Please refer back to that post (hyperlinked to, a few times, above) if you have any questions about the command lines below that don't require further explanation in this context). The file entitled "the_found_file_I_hope" is the gigantic block file we created with unrm yesterday.

NOTE: Unless you run lazarus with the -D option, it will create the "blocks" subdirectory in TCT's base directory!

First, we'll check again to see if text from our deleted file even exists in our unrm block recovery file:

host # du -sh /usr/local/recovery/the_found_file_I_hope
1.7G /usr/local/recovery/the_found_file_I_hope
host # egrep -il 'unusual|later|random' /usr/local/recovery/the_found_file_I_hope
/usr/local/recovery/the_found_file_I_hope


Since we've confirmed that our deleted file is in there (your test may need to be more exact. This experiment has the luxury of being controlled and isn't subject to the normal laws of break-neck work stress ;) we'll go ahead and kick off lazarus:

NOTE: While the display below will seem cool for about 10 minutes, tops, you'll eventually have to walk away from your terminal (unless you can bring it with you into the bathroom ;) - The process run time for combing 1.7GB of data took about 17 hours on a middle-weight desktop server. I can't say for sure, because I left it to finish-up while I went on living my life ;) You'll also find that the output to your screen will be long and, probably, useless to you. Generally, it's not worth watching unless you're really jaded, tired and/or bored ;)

host # ./lazarus /usr/local/recovery/the_found_file_I_hope
....t........tt.....pp.......tt....t.....pp....pp...tt...t...pp..pp........c...
......!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!t.ttt...
....ttt.t.t.aaaaaaatt...tt.t...t....ttt...tt..t...t.tt...t....ttt.t...t.tt..tt.
.tt...tt..t.tt..tt.t...t.tt..tt...tt.t..t.t.....t.ttt.....t.......t...tt......t
t..tt.tt..t..tt..tt..tt..t..t...tt..tt..t...tt...........xxxxxxxxxxxt......aaaa
ahhh..ttt..ttt....ttt..t.t..t..t..t.t.t...t........aaaaaaaaat.t.t.....gggttt.tt
t.ttt.ttt.ttt.ttt.ttt..aaaaaaaaaaaaaaaaaat.t.aaaaaaaaacc..t........aaaaaattt.tt
t.ttt.ttt.ttt.ttt.ttt........t...t......xxxxxxxaaaeeeeee!!!ttt.....t...eeeeeeee
...


As noted above, since we ran lazarus with no command line arguments, the "blocks" subdirectory (which it creates by default) will be located in the base directory in which TCT resides (as opposed to the directory in which you invoked lazarus). In our case, for the sake of argument, we have the binary in /var/tmp/tct-1.18/bin and the "blocks" directory will get automatically created in /var/tmp/tct-1.18.

host # cd /var/tmp/tct-1.18
host # ls
Beware LICENSE TODO blocks help-recovering-file patchlevel
CHANGES MANIFEST TODO.before-next-release conf help-when-broken-into quick-start
COPYRIGHT Makefile additional-resources docs lazarus reconfig
Date OS-NOTES bibliography etc lib src
INSTALL README.FIRST bin extras man www
host # cd blocks
host # ls -1|wc -l
10163
host # ls
1...txt 116389.t.txt 138610.t.txt 162373...txt 174689...txt 178282.t.txt 191521...txt 19517.t.txt 213764.t.txt 38401.p.txt 6250.x.txt
...
116375...txt 13861...txt 162371.t.txt 174682.t.txt 178281...txt 191514.t.txt 195169...txt 213759...txt 38377.t.txt 62006.x.txt


You'll note that the contents of the "blocks" directory are truncated in the output above. This directory (once lazarus has completed its run) is filled with one file per character that you saw in the output during its run. In the interest of keeping this post under 300 pages, the middle was clipped ;)

Now, as we did yesterday, we can use grep to very simply discover which of the 10,163 recovered block files in the "blocks" directory most probably contains our deleted file:

host # grep -l semi-random *
16.t.txt


And, now we can (hopefully) verify that our file is actually in there. As with the output from yesterday's post (just using unrm), this file will probably contain plenty of garbage surrounding the simple text. In this case (since I don't believe in revisionist-history, I'm not going to delete the previous sentence ;) it turns out that my assumption was wrong and we have one very clean copy of our deleted text file :)

home # cat 16.t.txt
we'll just put some
semi-random text in
here to see if we can
find this later with
grep. For simplicity's
sake, we'll include the
word semi-unusual so that we
have something in this
file that probably won't
be in any other files
host #


And, that's all there is to it :) As mentioned previously, this process can take lots and lots of time, although it makes the data discovery much easier in the end; especially if you're dealing with binary data or once-contiguous blocks that are now all still available for recovery, but scattered about. If you use the -h flag to have lazarus output HTML data for you, it will create a mini-website in the "blocks" directory that can make it much easier to piece together binary data, like a picture or an audio file.

Have fun waiting, and here's to your success in recovering your lost data!

Cheers,

, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Tuesday, May 19, 2009

Recovering Lost Data On Linux Or Unix Using The Coroner's Toolkit (TCT)

Hey there,

Finally, as promised, I'm getting around to finishing this post on recovering lost data on Linux and Unix. I'm far too lazy to look through the insane drivel I posted while I had the flu, but I'm pretty sure this was supposed to go out last week. Perhaps it's all for the best ;)

This is going to be a two-parter, since it seems so incredibly convoluted to explain ;)

The method we'll be using for this walkthrough requires a few tools, plenty of disk space and a good deal of patience (most likely). The two main tools are found in The Coroner's Toolkit (TCT), which you can download for free from the preceding hyperlink. The two tools you'll need from this set of computer forensic tools are:

1. unrm: This command basically creates one gigantic file composed of all the free blocks on the partition you want to do your data recovery from (not to be confused with the partition on which you lost your data).
2. lazarus: This command takes the output of unrm and breaks it up into files (by block), which can make it simpler for you to find your lost data and restore it.

It's recommended, when using unrm, that you mount the partition from which you want to recover data, in read-only mode. If this is impossible, that's not a big deal. It's imperative that you create your unrm recovery file on a "different" partition than the one from which you will be recovering. You will also need to make sure that this separate partition has about 220% of the free space available on the partition from which you want to recover, available for use.

The breakdown on the 220% worth of disk space is pretty simple. On your recovery partition, you'll need 100% of the available free space on the partition from which you want to recover your lost data, just to copy all those free blocks over. Makes sense. Now, the other 120% is, actually, optional and dependant on whether you make use of lazarus or not. If you don't need it (you'll see if you do or don't as we move along), then you'll only need to have that 100% available on your recovery partition. If you do use lazarus, then you'll need an additional 100% for the duplicate blocks that lazarus will create from your unrm output, plus about an additional 20% for overhead (since lazarus will be tagging each block as containing data of a certain type, like audio, picture, text, etc)

From you, as noted above, we'll just need patience. You may get the desired response from this process immediately, but probably not. To prepare recovery from about 2GB of free space, expect unrm to take a minutes or so (only just slightly longer than doing a "cp"). To comb that free space with lazarus, expect to wait (assuming you sit through the entire thing), at least 6 or 7 hours. In all honesty, I left the process running overnight and found it finished when I returned. It had been running for approximately 7 hours when I left it, and "du" indicated that is wasn't even halfway done recreating the individual blocks.

So, for today, we're going to set up the situation and do things the easy way. That's right; for all the explanation above, we will only be using unrm today. Tomorrow, we'll get into lazarus, with a pointer back here to the heavy explanation of the whole process. I'm just trying to save virtual trees. ...or my sanity. ...we'll see ;)

THE SITUATION: You've created a text file, containing valuable information that you couldn't commit to memory, using your favorite text editor and saved it. Then, an hour or so later, you accidentally deleted it, realizing that you'd completely screwed up just seconds after pressing the enter key. Anybody "not" know that feeling? ;)

host # cat /usr/THE_ALMOST_LOST_FILE
we'll just put some
semi-random text in
here to see if we can
find this later with
grep. For simplicity's
sake, we'll include the
word semi-unusual so that we
have something in this
file that probably won't
be in any other files
host # rm /usr/THE_ALMOST_LOST_FILE
host # cat /usr/THE_ALMOST_LOST_FILE
cat: cannot open /usr/THE_ALMOST_LOST_FILE


So, you're basically screwed (although the contents of your file would be much more compelling than those above ;)

The first thing we'll want to do is create a recovery area on a separate partition that has free space in the amount of 220% of the free space on the partition where we deleted our file (explanation above). We'll only be using 100% today (or would that be approximately 45% of the 220%?). Basically, we'll just need to have the same amount of free space on our recovery partition as exists on the partition we accidentally deleted our file on. We'll just assume that you can't unmount the old partition and mount it read-only (generally the case if you're working on a live system that other people are using ;) For this case, /usr/local looks like a good fit to recover data from /usr on.

host # df -h /usr /usr/local
Filesystem size used avail capacity Mounted on
/dev/dsk/c0t0d0s3 5.0G 3.4G 1.6G 68% /usr
/dev/dsk/c0t0d0s5 49G 2.0G 47G 5% /usr/local

host # mkdir /usr/local/recovery
host # ./unrm /dev/dsk/c0t0d0s3 >/usr/local/recovery/the_found_file_I_hope
./unrm: impossible: unallocated meta block 4228499!!
.....


You'll see a ton of these "unallocated meta block" messages. Ignore them. They're perfectly harmless :)

When the unrm process is complete you will have created one gigantic file, equal (sometimes larger) to the size of the free space available on the partition where you deleted your file accidentally, with the name that you redirected unrm's output to. Let's check out that file:

host # ls /usr/local/recovery/the_found_file_I_hope
/usr/local/recovery/the_found_file_I_hope
host # file /usr/local/recovery/the_found_file_I_hope
/usr/local/recovery/the_found_file_I_hope: data
host # wc -l /usr/local/recovery/the_found_file_I_hope
10481696 /usr/local/recovery/the_found_file_I_hope
host # du -sh /usr/local/recovery/the_found_file_I_hope
1.7G /usr/local/recovery/the_found_file_I_hope
host # egrep -il 'semi-unusual|later|random' /usr/local/recovery/the_found_file_I_hope
/usr/local/recovery/the_found_file_I_hope


And it appears that our file is actually in there :)

Now, since we're not going to use lazarus today, to chunk that huge file up into easy to digest blocks (one, or more, of which would be the text file we deleted), we'll need to get the info back the "old-school" way. This is where lazarus really comes in handy, because, in order for us to extract the file we want, we need to extract it "very specifically" from that huge recovery file. This is somewhat trivial with our text file (that we remember the entire contents of) but can be very tricky with binary data!

Even with text, getting it back from the one big file is going to be messy; especially at the beginning - for instance, we can't grep (using Solaris' grep, anyway) anything from the first line. I found it in line 5 of the huge file unrm created. To demonstrate what I mean, let take a look at line 5 (line 1 of our original text) and line 6 (line 2 or our original text).

Line 2 of our original file is normal looking:

host # grep -n semi-random the_found_file_I_hope
6:semi-random text in
host # sed -n 6p the_found_file_I_hope
semi-random text in


But check out line 1! :

host # grep -n some the_found_file_I_hope 2>&1|head -1
12:have something in this


It didn't get caught (???) Of course, grep has problems with it because the line actually looks like this in the recovery file:

host # sed -n 5p the_found_file_I_hope
l¨oÿÿþp-B
8B0
BBÀÀÜ
LgØRytyt
Xyy
^yy
f}}
Co G
t££2£à£à}¥ ¥ O§p§pÄ §§oÿÿö¨Awe'll just put some


Ugly ;) Usually (mostly when you get directly on top of trying to recover your data) your deleted data will be in this gigantic block-file in sequence. If you wait too long, and you're still lucky enough to get most of it, you may find that your data is out-of-sequence. In our case, we can retrieve our file by simply printing out the lines, one by one, using a blanket "sed -n" print statement:

host # sed -n 5,14p the_found_file_I_hope
l¨oÿÿþp-B
8B0
BBÀÀÜ
LgØRytyt
Xyy
^yy
f}}
Co G
t££2£à£à}¥ ¥ O§p§pÄ §§oÿÿö¨Awe'll just put some
semi-random text in§o+
here to see if we can
find this later with
grep. For simplicity's
sake, we'll include the
word semi-unusual so that we
have something in this
file that probably won't
be in any other files


Sure, it's not perfect (a little garbage to edit out), but it's better than nothing ;)

Tomorrow, we'll take a look at how lazarus can make this process much simpler. ...and take much much longer ;)

Cheers,

, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.