Enumerating Files In The Linux or Unix Shell - More Improvements

Hey There,

Today, we're going to take a little more time and devote it to the folks on all the boards and networking sites who've made excellent suggestions for improvement on some of the shell/Perl scripts and other posts we've put out here over time. As was mentioned the first time we decided to start posting these suggestions and improvements, rather than update old posts that never get any attention, we'll be putting the good stuff out here with updated timestamps so that anyone who reads this blog doesn't ever have to wonder if an issue they find has been addressed. And, also, of course, as a thanks for some great tips for refinement.

Today we're going to look at a couple of improvements and refinements to the shell one-liner to enumerate file types, suggested by folks via email and on the boards over at linuxtoday.com, LXer.com, fsdaily.com, and many other venues.

Again, since our policy regarding privacy is to regard everyone's privacy as equal and well deserved, we will only be referring to the folks who contributed by the nicknames and/or screen-names they used in "talkbacks" which are already posted on the internet. And, then, only if it's relevant and unavoidable.

The suggestions for change to this one-liner were excellent, many, and more brief. They also indirectly pointed out the fact that, when I wrote it, I was obviously obsessing over awk ;) The original one liner was this:

find . -print|xargs file|awk '{$1="";x[$0]++;}END{for(y in x)printf("%d\t%s\n",x[y],y);}'|sort -nr

To this, it was first suggested that names with spaces in them wouldn't work. This is absolutely true, and can be countered using a variation on "xargs" in the command line, like this:

find . -print|xargs -Ivar file "var"|awk '{$1="";x[$0]++;}END{for(y in x)printf("%d\t%s\n",x[y],y);}'|sort -nr

This effectively "double quotes" the arguments passed to xargs, but, while I was thinking about that I realized that there would also be additional work you'd need to do for single quotes/apostrophes, etc, to keep them from screwing up the command chain, as well. It was beginning to seem more and more like a solution that could definitely use some re-tooling.

So, naturally, one suggestion I received was to do it without using xargs. Good deal: One less hassle, to my way of thinking, and removing a whole lot of issues that didn't have to exist. The difference here is the use of the -exec flag with the find command, rather than piping to xargs:

find . -print -exec file {} \;|awk '{$1="";x[$0]++;}END{for(y in x)printf("%d\t%s\n",x[y],y);}'|sort -nr

The next suggestion I received was to do it without using awk, but keeping xargs. This has the advantage of removing one additional external command (standard though it may be) from the process. And removing awk can make things a lot less confusing for most folks (myself included, which is probably why I used it originally. Not out of a twisted desire to cause myself grief, but to try and get more comfortable with it ;) That suggestion looked like this:

find . -print | xargs file -b | sort | uniq -c | sort -nr

But, this took us back to the xargs quoting and space-in-filename issue. This is the final suggestion that came from that community debate, which I think is probably the best (as in most succinct and utile) since it does it without awk, addresses the issues with xargs and can handle all the issues raised above:

find . -print0 | xargs -0r file -b | sort | uniq -c | sort -nr

If you want to check out this interaction, to gain some more insight into the thought behind each version, you can find it here on linuxtoday.com (at least for a while, assuming it will get moved eventually).

And, once again, a huge "Thank you" to anyone and everyone who's helpful criticism proved that, not only is there more than one way to skin a cat, there are far more efficient ways to do it than you or I may imagine in one sitting (but, please, don't skin any cats ;)

There's probably someone out there who knows a way to do it even better. Which is, of course, the beauty of Linux and Unix and why I enjoy working in the shell. It can be as simple or as complicated as you need it to be, and is flexible enough to allow users' the creativity to determine the path and the outcome of virtually everything that can be accomplished using either OS (or both :)

Have a great morning/day/afternoon/evening,

, Mike

Simple Shell One-Liner To Enumerate File Types In Linux and Unix

Hey there,

Lately, we've been focusing a lot on Perl "one liners," from mass file time syncing to name and IP resolution and I thought it was only fair that we should write a post about a shell "one liner" for once. After all, most standard Unix and Linux shells are perfect for that purpose :)

Here's a very quick way to take an inventory of all the different file types (including directories, sockets, named pipes, etc) that exist in a given directory tree and provide a tally of each file type. I'm not entirely sure that I care if I have 15 ASCII text files and 2 Perl scripts in my current working directory, but this little piece of code must be able to help someone somewhere accomplish something, even if it is only to use as a smaller part of a larger organism ;)

This works in pretty much any standard shell on every flavour of Linux and/or Unix I've tested (ash, sh, bash, jsh, ksh, zsh, even csh and tcsh, which is huge for me since I never use those shells. One day, soon, I will redouble my efforts and just learn how to use them well, which, hopefully, won't result in my writing a whole bunch of posts about "cool" stuff that everyone has known about for the past few decades ;).

This one-liner could actually be written as a script, to make it more readable, like this:

#!/bin/sh
find . -print | xargs -I var file "var"|
awk '{
        $1="";
        x[$0]++;
} 
END {
        for (y in x) printf("%d\t%s\n", x[y], y);
}' | sort -nr

But, since I'm a big fan of brevity (which is about as far away from obvious as possible if you consider my writing style ;), I would run it like this:

host # find . -print|xargs file|awk '{$1="";x[$0]++;}END{for(y in x)printf("%d\t%s\n",x[y],y);}'|sort -nr

In my terminal, that all comes out on one line :) And here's the sort of output you can expect:

host # find . -print|xargs file|awk '{$1="";x[$0]++;}END{for(y in x)printf("%d\t%s\n",x[y],y);}'|sort -nr 
23       Bourne-Again shell script text
11       ASCII text
10       perl script text
5        pkg Datastream (SVR4)
4        directory
4        ASCII English text
2        UTF-8 Unicode English text, with overstriking
2        Bourne shell script text
1        RPM v3 bin i386 m4-1.4.10-1

and, just to confirm the file count, we'll change the command slightly (to total up everything, instead of being particular -- another "one-liner" that's, admittedly, completely unnecessary unless you're obsessive/compulsive like me ;) and compare that with a straight-up find:

host # find . -print|xargs file|awk 'BEGIN{x=0}{x++}END{print x}'
62
host # find .|wc -l
62

Good news; everything appears to be in order! Hope this helps you out in some way, shape or form :)

Cheers,

, Mike

Getting Error Values From The Middle Of A Pipe Chain In Bash

This is something very interesting I found out not too long ago, while hashing out some work with a colleague. As most administrators (or users) who do a fair amount of shell scripting know, the error status or return code (Generally referred to as "errno" in all the man pages) of a process is a fairly common method to use in determining the process flow of a script.

The one thing about the value of "errno" (or, literally, the variable "$?" in most shells) is that it's erased with each consecutive process that gets run. So if you were to run a series of command lines that echoed the return value of the grep command, the following example would be accurate (assuming the string "bob" can't be found in /home/myfile):

host # grep bob /home/myfile >/dev/null 2>&1
host # echo $?
host # 1

while this one would give you misleading information:

host # grep bob /home/myfile >/dev/null 2>&1
host # touch /home/myfile
host # echo $?
host # 0

So, on the first set of command lines, you're actually getting the return code of 1 from grep (indicating that it can't find the string "bob" in /home/myfile), while the second one gets you the return code of 0 from the touch command. "errno" always contains the return value of the last-executed command.

Which brings us around to the topic indicated in the title of this post (I promise to tie in the whole introduction about "errno" at the end; it wasn't a complete waste of your time ;). While it's easy enough to trap "errno" in any series of disconnected commands (for instance, in the second example above, if we'd echoed $? before running touch, it would have given us the correct output), I had always thought it was impossible to grab the correct value from a command in the middle of a pipe chain, like this:

host # grep bob /home/myfile 2>&1|Grep joe|xargs echo
host # echo $?
host # 0

You'll note that I purposefully capitalized the G in grep so that it would return an error code that didn't indicate success, yet - since this is a chain of commands all connected by pipes - "errno" returns the value of the xargs command, since it was the last one executed. Which means I've spent a lot of time jumping through hoops to "reword" any pipe chain so that I could extract the information I needed.

Now (and I'm almost positive this wasn't the case a few years back) the bash shell has actually taken on this predicament and come up with a nice workable solution for it(I'm waiting for it to pop up in sh and ksh, since they've been burned into my psyche over the last decade or so). In bash, if you run a series of piped-together commands, you can actually extract the value of "errno" from any command in the chain by using the shell built-in PIPESTATUS array, like so:

host # grep bob /home/myfile 2>&1|Grep joe|xargs echo
host # echo ${PIPESTATUS[@]}
host # 1 127 0

How nice is that? :) Now you can easily tell the return value of every process in a pipe-chain. The initial grep returns 1 because the string "bob" isn't in /home/myfile, the misspelled Grep returns 127 because the command can't be found and the final xargs returns 0. That solves a lot of problems and can potentially save you lines upon lines of convoluted code.

The one thing about it that can be frustrating is that it behaves in much the same way as "errno" (See, I told you I'd bring it back around ;). If you don't capture the output immediately (or dish it off into another variable), the array will zero out and contain no values as soon as you enter your next command, like so:

host # grep bob /home/myfile 2>&1|Grep joe|xargs echo
host # touch /home/myfile
host # echo ${PIPESTATUS[@]}
host # 0

At this point, after we've executed the touch command, the PIPESTATUS array has been cleared out, just like "errno" gets written over, even though we haven't executed another pipe chain. Its behaviour is basically identical. Below, we show that, once the array has been written over, its size gets reduced to 1 ( The single return value of the last executed command) and we further prove that the array really has been clipped down to one variable by attempting to print the first and second values; the second of which doesn't exist. Continued from above:

host # echo ${#PIPESTATUS[@]} <--- Here we ask bash for the size of the PIPESTATUS array
host # 1
host # echo ${PIPESTATUS[0]} <--- Here we check the first variable in the PIPESTATUS array
host # 0
host # echo ${PIPESTATUS[1]} <--- Here we check the second variable, which now doesn't exist
host #

This is easy enough to get around, however, since - just like "errno" - you can assign that array to another array before you execute another command, like so:

host # grep bob /home/myfile 2>&1|Grep joe|xargs echo
host # new_array=${PIPESTATUS[@]}
host # touch /home/myfile
host # echo ${PIPESTATUS[@]}
host # 0
host # echo ${new_array[@]}
host # 1 127 0

If you knew this already, I envy you the convenience you continue to enjoy. For the rest of us; a pleasant surprise :)

Best Wishes,

, Mike

linux unix internet technology

Using find and xargs to locate Windows Files

A lot of times, when you're asked to find something on a machine, and you only have a moderate idea of what you're specifically looking for, you'll use the obvious command: find. find is a great command to use because you can use wildcards in your expression argument. So, if you know that you're looking for something like "theWordiestScriptEver," and you have no idea where it's located on your box, you could find it by typing just this:

find / -iname "*word*" -print

This will find every file on the system (even on non-local mounts if you have them set up) and only print the results for files with the word "word" in the name. Note that the "-iname" option matches without regards to case, so h and H both match. This option isn't available in all versions of find. If you don't have this option available to you, you'll get an error when you run the above line (just use "-name" instead). The standard Solaris find does not do "case insensitive" pattern matching, so your best bet is to find the smallest substring that you're sure of the case on, or use another attribute to search for the file (like -user for the userid or -atime for the last access time). Alternatively, you could spend hours stringing together a bunch of "or" conditions for every conceivable combination of upper and lower case letters in your expression.

Now suppose you needed to perform an action on a file you found. You could use find,s built-in exec function, like so:

find / -iname "*word*" -print -exec grep spider {} \;

This will perform the command "grep spider" on all files that match the expression. Which brings us around to the next predicament. What do you do if you have to try and find something simple, have no idea where it is on your box "and" that box hosts file systems that Windows users are allowed to write files to. The above example should work just fine on those. My own advice is, if you can get away with just using find, do so, since it handles all of the rogue characters, tabs and spaces in Windows files on its own.

Now, if you have to do something much more complicated (or convoluted), you'll want to pipe to a program like xargs, which is where all those funny Windows file names and characters (some of which are special to your shell) start to cause issues. Again, this would return ok:

# find . -name "*word*" -print
./word - file's
./word file
./word & file's
./word file's

But this will become an issue if you pipe it to xargs, as shown below:

# find . -name "*word*" -print|xargs ls
xargs: Missing quote: files

Ouch! xargs doesn't deal with those spaces, tabs and special characters very well. You can fix the space/tab problem very simply by using xarg's "named variable" option. Normally, xargs acts on the input it receives (thus: "xargs ls," above, is processing ls on each file name find sends it), but you can alter how it deals with that data in a simple way (at least as far as the spacing issue is concerned). Example below:

# find . -name "*file" -print|xargs -ivar ls "var"
./word file
# find . -name "*word*" -print|xargs -ivar ls "var"
xargs: Missing quote: ./word - files

But, in the second invocation above, you see that it still can't handle the "shell special" characters, like "'" or """ <--- Double quote - so it's time to step it up. I prefer to just sanitize everything that's not kosher, even though I know I don't technically have to avoid the ---> \ / : * ? "< > <--- characters, since Windows won't allow them as parts of file names. It seems easier just to react on anything that isn't a letter or number and pass it along with enough escapes (back slashes) so that xargs can parse it correctly, and get you back good information. Here's how to do that; using sed (and a little grep, to keep it neat), also:

# find . -name "*word*" 2>&1|grep -iv denied|sed "s/\([^A-Za-z0-9]\)/\\\\\1/g"|xargs -ivar ls "var"
./word - file's
./word file
./word & file's
./word file's

And now you can use find, combined with xargs, on all the files you have permission to see, no matter what goofy characters are in them :)

, Mike

linux unix internet technology