Hey there,
It should be noted, at the outset, that this post is limited in its scope. We're going to be looking at one particular way in which you can recover an accidentally deleted file on Linux or Unix ( Tested on RHEL3 and Solaris 8 ). If you ever want to scour a hard drive that you need to get lots of information back from, then (assuming that you quarantined it immediately upon noticing this and haven't written to it since) you should check out The Coroner's Toolkit. Specifically, you'll want to look at the "ils" or "icat" programs, and most probably the "grave-robber" application to recover as much of everything as possible.
In our post, we're going to look at one condition under which Unix and Linux operating systems can actually hold onto files, even after they've been deleted, so that you can recover them. Of course, you have to realize you've deleted something you wanted to keep (in most cases) immediately, and a fairly specific set of circumstances has to be in play.
Note that all demonstrations have been tested on Solaris and RedHat. We're using RedHat's output, but the differences between the two were minor enough that they didn't bear repeating.
First, we'll look at the controlled scenario: In this case, we're going to create a file. Then we're going to delete it, knowing that we want to retrieve it afterward. How many times does that happen "by accident"? ;)host # echo "hi there" >>FILE
host # ls -lai
total 8
57794 drwxr-s--- 2 user group 4096 Aug 11 12:36 .
851795 drwxr-s--- 3 user group 4096 Aug 11 12:31 ..
57795 -rw-r----- 1 user group 9 Aug 11 12:36 FILE
Now that the file's created, we can cat it by referencing the inode number (gotten with "ls -lai" above) very easily
host # find . -inum 57795 -exec cat {} \;
hi there
Then we'll set ourselves up so that we can delete the file and still be able to get it back. In order for us to be able to retrieve the deleted file later, we'll need to associate it with a filehandle. One easy way to do that is to run a tail (or similar command) on it):
host # tail -f FILE &
[1] 4741
and then we'll delete it:host # rm FILE
host # ls -lai
total 8
57794 drwxr-s--- 2 user group 4096 Aug 11 12:41 .
851795 drwxr-s--- 3 user group 4096 Aug 11 12:31 ..
So, now it's gone. But, and this is the only thing that's saving us a whole lot of headache, the file is still open in memory since we still have a "tail -f" job running in the background. This means that the tail command still has a filehandle open for our file FILE. Of course, we can't refer to it by that name anymore. One interesting thing to note about an inode is that it contains virtually all the information you ever wanted to know about your file... except it's name! :)
Therefore, the following query with lsof fails to produce results:
host # lsof|grep FILE
IMPORTANT NOTE: If you have followed this process closely and either wrote down or remembered the "inode number" of the FILE file before we deleted it, you can skip all of this lsof stuff. Jump straight to the next IMPORTANT NOTE in this post :)
A quick look at what pseudo-terminal we're using, coupled with the knowledge that we ran "tail -f" on the file, makes for a pretty tidy grep string (of course, you don't need to have this much information. You can do this without of a filter and just have more lsof output to muck through):host # tty
/dev/pts/0
host # lsof|grep pts/0|grep tail
tail 4741 user 0u CHR 136,0 2 /dev/pts/0
tail 4741 user 1u CHR 136,0 2 /dev/pts/0
tail 4741 user 2u CHR 136,0 2 /dev/pts/0
Now we know the PID (second column in from the left) of the process that still has the file open and we can use lsof to drill down even further, using the output of the pwd command to whittle down the output:host # pwd
/home/users/user
host # lsof -p 4741|grep home/users/user
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
tail 4741 user cwd DIR 0,22 4096 57794 /home/users/user (host:/remote/users)
tail 4741 user 3r REG 0,22 9 57795 /home/users/user/FILE (host:/remote/users)
IMPORTANT NOTE: Welcome back, if you already knew the inode number, and "on we go" to everyone! One interesting thing to note about the lsof output is that there is only a "read" filehandle open for FILE. This is normal, since we're doing a "tail -f" and there doesn't need to be a write or read/write file descriptor active.
Now we can verify that the our "file" still exists by accessing it via the inode number:
host # find . -inum 57795 -exec cat {} \;
hi there
And, it's still there :) In order to preserve it (since it the inode-fd connection will be severed as soon as "tail -f" quits), we'll use similar "find" syntax to copy the inode to a filename and verify it, like so:host # find . -inum 57795 -exec cp {} FILE.recovered \;
host # ls -lai
total 8
57794 drwxr-s--- 2 user group 4096 Aug 11 12:42 .
851795 drwxr-s--- 3 user group 4096 Aug 11 12:31 ..
57796 -rw-r----- 1 user group 9 Aug 11 12:42 FILE.recovered
host # cat FILE.recovered
hi there
Now we can quit our backgrounded "tail -f" job and not worry about it.host # kill %%
host #
[1] + Terminated tail -f FILE
Like I said, a neat trick, but will only work if you get lucky. Pardon the double-cliche, but keeping calm when you realize that you may have royally screwed the pooch can go a long way toward keeping you from having to take the long way home (isolating the disk, scouring it with forensics tools, perhaps grepping the raw filesystem and manually reconstructing, etc).
The one thing to remember is that, if you delete a file on Linux or Unix, as long as one other process, that was using it when it existed, is still up and running after you delete it, there's a possibility that you can get your file back fairly quickly (actual results may vary. The possible scenario's are many and varied).
And, to give us a fair sendoff, here's a quick look at what you can expect if you delete a file and no process is attached to it and/or has a filehandle open against it (If you don't like watching train-wrecks, then I'll bid a good evening to you right here and now - cheers :)host # ls -lai
total 8
57794 drwxr-s--- 2 user group 4096 Aug 11 12:43 .
851795 drwxr-s--- 3 user group 4096 Aug 11 12:31 ..
57796 -rw-r----- 1 user group 9 Aug 11 12:42 FILE
host # find . -inum 57796 -exec cat {} \;
hi there
host # rm FILE
host # find . -inum 57796 -exec cat {} \;
host # tty
/dev/pts/0
host # lsof|grep pts/0
ksh 3907 user 1u CHR 136,0 2 /dev/pts/0
ksh 3907 user 2u CHR 136,0 2 /dev/pts/0
ksh 3907 user 11u CHR 136,0 2 /dev/pts/0
lsof 5062 user 0u CHR 136,0 2 /dev/pts/0
lsof 5062 user 2u CHR 136,0 2 /dev/pts/0
grep 5063 user 2u CHR 136,0 2 /dev/pts/0
grep 5064 user 1u CHR 136,0 2 /dev/pts/0
grep 5064 user 2u CHR 136,0 2 /dev/pts/0
host # lsof|grep FILE
host # lsof|grep 57796
host # ls -lai
total 8
57794 drwxr-s--- 2 user group 4096 Aug 11 12:44 .
851795 drwxr-s--- 3 user group 4096 Aug 11 12:31 ..
...bummer.
, Mike
Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.
