Command Line Veritas NetBackup Activity Monitor For Linux And Unix

Hey there,

Many of you who use FeedBurner (or whatever smaller, productive, company Google's bought out this week ;) to get blog posts by email have probably already gotten this post even before I finished writing it. I made the cardinal sin of hitting the enter key while I wrote this post online :P I apologize for your (quite understandable) previous confusion ;)

Today's post will be short and sweet, since it's getting late in the day and I'm still working and I don't think I get to sleep in tomorrow either... With that in mind, I tried to make this post heavier on content and have been working on it (on and off) all day. Again, I also apologize for the quality of the video attached to this post, which shows the output of the script. It's very very slow and does not accurately reflect the refresh rate of the PuTTY terminal I was running the program on ;) I had to download 3 or 4 different video processing "portable apps" so that I could try and put something together. I'm not "allowed" to install anything anywhere except for my desktop (and I'm not really sure if that's 100% kosher, which is why those programs are no longer installed) and I had to take what I could get.

Today's Bash script is my first attempt at rewriting one of those oldies-but-goodies that got away. A few years ago, I wrote a TUI Activity Monitor for Veritas NetBackup (In Perl, not like this one in Bash), since that was the only thing I ever used the GUI for. Somehow, over the years, I misplaced the CD (or maybe floppy ;) that I had it saved on and I've finally come to peace with the fact that I'll probably never ever find it again.

This script is much less glitzy than the old one, but it's a good starting point, and I hope a few people (aside from me ;) find it useful. It's a very simple program to run, since it only takes one optional variable (The refresh rate in seconds - defaulted at 10) and, as yet, does not have any special functionality built in. And, yes, you should read that as "The output formatting is God Awful!" :)

Check out the video below to watch a few backup jobs, and a catalog job, run in slooooooooooooooooooo-mo ;) Then grab the script and put it to good use. If you can make it better (I know I can, so there must be thousands of you out there who can, too :), feel free to improve upon it before I get around to that. I try to stay away from exporting Java GUI's over X-Windows as often as I possibly can (especially when working on systems overseas or across-country), and I've been meaning to rewrite this script for a long long time.

If this video makes you feel drowsy, download it and play it at double speed in the player of your choice. For some reason, it recorded at half speed and I ran out of time to try and figure out why ;)



And here's the script. Enjoy it and abuse it in good health :)





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# nbuvtam - v1 of the netbackup vt activity monitor
#
# 2009 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

trap 'exit 0' 2
perl=`which perl`

if [ ! -z $1 ]
then
        refresh=$1
else
        refresh=10
fi

function local_time()
{
        regular_time=$(perl -e "print scalar(localtime($1))")
        echo -n "$regular_time"
}


jobtype_array=(backup archive restore verify duplicate import catalogue_backup vault label erase tape_request clean format_tape physical_inventory qualification)
state_array=(queued active wait_for_retry done)
operation_array=(mount position connect write vault_initialize vault_duplication vault_duplication_complete vault_catalog_backup vault_eject report duplicate import verify restore catalog_backup vault label erase)
subtype_array=(immediate scheduled user_initiated quick_erase long_erase)
schedule_type_array=(full incr user_backup user_archive cumulative_incr)

tput clear
OLDIFS=$IFS
IFS=","
while :
do
        tput clear
        echo "======================================================================="
        echo "THE TOTALLY LAME VERITAS NETBACKUP COMMAND LINE ACTIVITY MONITOR"
        echo "V0.0000000001-alpha - Ctl-C to Quit"
        echo "======================================================================="
        echo "jobid type state state_details status policy schedule client media_server start_time elapsed_time end_time storage_unit attempt operation kilobytes files pathname percent_complete job_pid owner master"
        echo "======================================================================="
        echo "======================================================================="
        while read jobid jobtype state status policy schedule client server started elapsed ended storage_unit attempt operation kilobytes number_of_files path_last_written percent jobpid owner subtype classtype scheduled_type priority group master_server retention_unit retention_period compression kb_last_written files_last_written file_list_count files the_rest
        do
                start_time=$(local_time $started)
                end_time=$(local_time $ended)
                elapsed=$(echo $elapsed|sed 's/^0*//g')
                hours=$((elapsed / 3600))
                elapsed=$((elapsed % 3600))
                minutes=$((elapsed / 60))
                elapsed=$((elapsed % 60))
                if [[ $hours -lt 10 ]]
                then
                        hours="0$hours"
                fi
                if [[ $minutes -lt 10 ]]
                then
                        minutes="0$minutes"
                fi
                if [[ $elapsed -lt 10 ]]
                then
                        elapsed="0$elapsed"
                fi
                echo "$jobid ${jobtype_array[${jobtype}]} ${state_array[${state}]} ${operation_array[${operation}]} $policy ${schedule_type_array[${schedule}]} $client $server $start_time $elapsed_time $end_time $storage_unit $attempt $operation $kilobytes $number_of_files $path_last_written ${percent}% $jobpid $owner $master_server"
                echo "---------------------------------------------------------------"
        done <<< "`/usr/openv/netbackup/bin/admincmd/bpdbjobs -all_columns|head -5`"
        sleep $refresh
done

, Mike

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Online Encyclopedia Shell Script For Linux And Unix

Hey There,

Today's Linux/Unix bash shell script is a follow up (and the finishing touch, I hope ;) to a whole slew of scripts we've written to mine the knowledge on tap at reference.com. If you missed any of the others, you can still find them in our older bash script posts to access the online Thesaurus and, of course, the online dictionary. This time we're breaking the chains and losing the albatross by finally attacking (with some level of precision) the Online Encyclopedia and producing a shell script does it as much justice as we could muster.

Note added after initial post: Forgot to include the Online Language Translation Script. It fits in with this bunch fairly well :)

One of the major differences you'll note between this script and the others is that we've included a "pager" variable. If you've ever checked out the reference.com Online Encyclopedia you're aware that the average return on a query is substantially larger than what you get back from a typical dictionary definition or thesaurus synonym query (not to mention the size of your terminal window). The screen shots we have for you, below, were the result of painstaking effort trying to find a word or phrase that didn't have an exceedingly long and thorough write-up. We chose "/usr/bin/more" as the pager for this script, but it can be easily changed to "/usr/bin/less," or whatever your favorite pager program happens to be.

After toying with this project for a while, trying to figure out why some pages worked, some didn't and where in the good Lord's name any consistent marker could be found to create consistent output, I've actually kind of gotten into using it. Now, if I'm at my terminal "working my fingers to the bone ;) " I can just type:

host # ./ency.sh engelbert humperdinck

and try to figure out what my generation "doesn't" see in him ;) ...apologies to die-hard fans.

Like our previous scripts, this script uses wget and sed. This script was also an attempt at "brute force scripting" (just writing it as we think it), but it ended up being so aggravating trying to find the right mix of regular expressions to get back decent results for the broadest amount of queries, that I think the method can, now, only be referred to as "blunt force scripting" (that's our new term for scripting as if you'd just been hit in the head with a brick ;). There are a few places the script could be tightened-up, but, to be perfectly honest with you, we're scared to death of changing anything about it right now. Maybe later, when we've steeled our nerves ;)

Below are a few screen shots of the script's output. The first is for an actual "one-pager" on a query for a pretty decent novel named Glamorama and the other is what you can expect to see when your query returns the equivalent of 15 leatherbound volumes of text on a query for Robert DeNiro. The last, just shows common error messages we generated on purpose.

Possibly interesting fact: If anyone out there is having an 80's moment and vaguely recalls Toni Basil ("oh, Mickey, you're so fine, you're so fine, you blow my mind..." why do I know the lyrics? ;), you'd be surprised to know that even "her" entry spanned multiple pages. I was actually surprised to find that she's also an actress and starred with Jack Nicholson in Five Easy Pieces under the very same name.

Click on the captures below to see them all in IMAX:







As one last bit of explanation. There are 2 specific error conditions that we ended up throwing in here. One is pretty obvious, involving No Results being found. The other had me going for a little while, since I was positive that a query on "linux" must have some return, since a query for "unix" did. It turns out that, when the Online Encyclopedia returns way too many results (I believe around 752 for "linux") it will state that it found a whole ton of results and is only going to show you one. The glitch in the program here is that, at some large number, it says that but then doesn't return an actual entry. I ended up noticing this when I finally broke down and parsed the HTML return and saw nothing, followed by a visit to the website via browser where I noticed the condition. You can see what I'm talking about right here, unless they've fixed it already (or it only happens intermittently). In any event, we wrote a check so that, if there are a large number of results, and it does return one, you'll still get it, but you'll get a self-explanatory error message if you hit this bug. Not that it needs to be said, but every query returns more than one result (insofar as our field-testing has gone), and the default behaviour of the site is to only return the top "one" and include links to the others.

Hope you enjoy this script, and can find some use for it. Any suggestions for improvement would be greatly appreciated. Be forewarned; there is the possibility that some of your queries may come back with extra parts added on that we didn't catch (usually tacked on to the end... phew), which could also make the output look horrible. This is the best we've got "for now" :)

I'm just glad somebody already figured out how to order Pizza online so I don't have that hanging over my head ;)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# ency.sh - More than you ever wanted to know about anything
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if [ $# -lt 1 ]
then
        echo "Usage: $0 Your Encyclopedia Terms"
        echo "May be two or more words separated by spaces"
        echo  "but only one definition per execution."
        echo "Ex: $0 money"
        echo "Ex: $0 the beatles"
        exit 1
fi

args="$@"
wget=/usr/bin/wget
pager=/usr/bin/more

if [ $# -gt 1 ]
then
        args=`echo $args|sed 's/ /%20/g'`
fi

echo

$wget -nv -O - http://www.reference.com/search?q="$args" 2>&1|grep -i "No results found" >/dev/null 2>&1

anygood=$?

$wget -nv -O - http://www.reference.com/search?q="$args" 2>&1|grep -i "<cite>" >/dev/null 2>&1

toomuch=$?

if [ $anygood -eq 0 ]
then
        args=`echo $args|sed 's/%20/ /g'`
        echo "No results found for $args"
        exit 2
fi

if [ $toomuch -ne 0 ]
then
        args=`echo $args|sed 's/%20/ /g'`
        echo "Too many results returned for $args"
        echo "Try doing a broader query - For Ex:"
        echo "$0 linux = Too many results"
        echo "$0 linux os = Information!"
        echo "Double check at www.reference.com"
        echo "to see the difference"
        exit 3
fi

$wget -nv -O - http://www.reference.com/search?q="$args" 2>&1|sed '1,/Cite This Source/d'|sed '/External links\|See also\|More from Wikipedia/,$d'|sed -e :a -e 's/<[^>]*>/ /g;/</N;//ba' -e 's/$/\n/'|$pager

exit 0

, Mike

---

Douglas Taylor submitted these enhancements to make the output look much nicer. Thanks, Douglas!

---

1) Run the output through fmt before outputting. It will wrap the paragraph text fairly nicely, although it kind of destroys tabular data, and the indenting can get weird, or

2) Use lynx with the -dump option instead of wget. I replaced the final wget command with

lynx -dump http://www.reference.com/search?q="$arg" |sed '1,/Cite This Source/d'|sed '/External links\|See also\|More from Wikipedia/,$d'|sed 's/\[[0-9]*\]//g'|$pager

and the result looks fairly nice.

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Using Grep To Streamline Your Shell And Command Line Scripting

Good [insert part of day here] ;)

For today's post, we're going to look at using grep to reduce script-bloat. Of course, we won't be covering every possible way this can be done, because I have yet to finish perfecting the art of using editor tricks to reinforce bad habits. The vi editor's yank (yy), put (p), double-quote (") buffers and dot (.) action repetition simplifiers make it very easy for me to write long and unwieldy code when I don't feel like thinking about how I could fit 40 lines into 3 ;) The downside is that it's a royal pain to go back and modify. Not only do I have to go back and forth 3 screens to check code above and below my "text dump," but I also have to fix the same problem 15 times if I made it in one instance of my code and then just mass-copied and slightly modified it. Of course, you can always use the substitute (s/g), etc, commands to make sweeping edits, but you get my drift. It's generally not more-or-less efficient to write unrolled loops (since that's what your shell command interpreter is going to do to your tight loops when it executes them anyway), but it can be a huge and confusing time-waster for all the humans who have to deal with it later; even for the original author.

In any event, I have managed to digress, yet again ;)

Today, we're going to look at a few ways we can use grep to clean up our shell scripting and make it look (and, most of the time, actually "be") more efficient. The solutions presented in this post can be one-upped by any number of other standard Unix/Linux commands, but (for the sake of this post) we're going to assume that "grep" is the be-all-end-all of input and output processing. Suspend your disbelief, for as long as it takes to read this post, and assume that the world is flat, grep's it and that's that ;)

On to our examples!

1. Generating line counts: This is one I see a lot (and have done a'plenty). Most Unix/Linux admins and users know of the wc command and get used to using it to determine the number of lines in a file like this:

host # wc -l FILE
      17 FILE

which is perfectly acceptable. Sometimes though, since the command comes so easily to mind, it gets used inappropriately or, at least, in places where using grep, by itself, would make more sense. For instance, consider this relatively normal looking, and perfectly serviceable, line of code:

host # cat FILE|grep done|wc -l
       2

And, oh yes, we're ignoring the indents that wc puts before the output for now :) This count of "how many times the word 'done' appears in FILE" can be pared down very easily, like so:

host # cat FILE|grep -c done
2

and, even better, we can use grep to get a basic "wc -l" line count on any file by using it like this:

host # grep -c . FILE
17

It's a slightly longer command than the "wc" version, but it does save you from having to process out the padded blank space if you want to assign your output to a variable, or use it "in context" within another command string as-is :)

2. Ensuring that your process checking is as accurate as possible: We actually looked at this back in October of 2007 in a post on how to keep grep out of your grep output, but the same basic principal applies here. For instance, if you've written a wrapper script that checks a process every 5 minutes and, at all times, this process must be running, you might make the core of your script (that part which you're hoping won't equal 0) something like this:

host # ps -ef|grep program|wc -l
       2

but, you see the same problem you had in our first bullet, so you'd crop that back to:

host # ps -ef|grep -c program
2

But, in this case, we still have an issue, because only 1 instance of your "program" should be running and there appear to be two. Actually, if we peel back the veneer, there "is" only one instance running, but grep is ending up in the output (and, even worse, you can't depend on grep to "always" show up in its own output, which means our result may be 1 or 2 with everything being OK):

host # ps -ef|grep program
user1  2692  8747  0 13:33:52 pts/11   0:00 /bin/sh ./program
user1  3269  8747  0 13:37:05 pts/11   0:00 grep program

So, now we can use another trick to make this work correctly every time. A lot of times you may see code like:

host # ps -ef|grep program|grep -v grep|wc -l

but this is quite a bit longer, now, and we also can't use "grep -c" because it would make the following "grep -v grep" statement fail. And, since we can't count on "grep" always showing up in its own output, we couldn't predicate success or failure based on any specific number (in this case 1 or 2). Plus, if we did, this code would get even more lengthy and confusing.

Instead, we can use the shell's natural range operators "[]" to ensure that (if only 1 instance of "program" is supposed to be running) we'll always only get back the number "1" when things are going well:

host # ps -ef|grep -c "[p]rogram"
1

and that's it. The reason this works is that the shell processes the range operators (which, in our case contain a range of one letter) and pass the string "program" to grep. So, grep is scanning the ps output for "program," but the range operators still appear in the ps output (for the grep command). Therefore, it can never match its own process because ps is literally showing "[p]rogram" (with the left and right brackets ([]) being actual left and right brackets, rather than range operators which get interpreted).

Here's what the command output looks like in the working version above:

host # ps -ef|grep "[p]rogram"
user1  2692  8747  0 13:33:52 pts/11   0:00 /bin/sh ./program

and modified to look for "[g]rep" so we can see the grep and note the difference:

host # ps -ef|grep "[g]rep"
user1  5864  8747  0 13:47:16 pts/11   0:00 grep [g]rep

<-- Note that this match is matching "grep" and not "[g]rep" ;)

And that's it for today. There are actually quite a few more things you can do with grep, that get done a lot more efficiently as a result, but we'll take a look at those another day. As usual, it seems, in attempting to explain as clearly as possible, I have managed to crank out yet another tractate. It's times like these that I start to feel a bit long in the tooth. The older you get, the more outcomes you try to compensate for. That can make for some long posts. It can even make for some long endings to long posts ;)

Hopefully, it's been a relatively bearable reading :)

Cheers,

, Mike

More Unix And Linux Humor

Hey there,

For this "Lazy Sunday" post, I'm putting some more funny stuff I've found while crusing the web, regarding our two favourite flavours of OS. Of course, all the content has been sanitized, so I don't lose my "non-adult" status on this blog. If you want to check out the original sources, please stop by jeffcovey.net and jokes2go.com to have a look around.

Hope you find some of this stuff funny. Hilarious works too ;) Everyone needs a little time off, which is why I'm posting some goofy stuff over the weekend (Really, on the weekends readership goes way down, anyway, since I still haven't figured out how to get people interested in reading about work on their days off ;) Please note that the "command line" humor in the last section was done in a few different shells (a lot of it in csh, which won't reproduce in any other shell except tcsh, possibly)

Since there's so much space being used in this post, here's a little roadmap. Jokes 1 and 2 are song parodies, joke 3 is a clever paragraph and everything after that (All considered Joke 4 ;) are command line funnies.

Tomorrow, we'll get back to the "business" of Unix and Linux.

Cheers, and enjoy!

1. 50 ways to leave your editor

From: tbpic1@aurora.cc.monash.edu.au (Tim Pickett)
Subject: Re: Simon and Garfunkel (was Re: This is a test

umisef@yoyo.cc.monash.edu.au (Bernd Meyer) writes:
>So let's go for "there must be 50 ways to leave your editor" :-)

All right, you've got a deal. Memories of trying to quit an editor in
VMS that I didn't understand . . . (A bad song this for parodying. It
makes it sound like I can't make words scan, even though I'm referring
to the sheet music in front of me.)

The problem is all inside your ed(1), she sed(1) to me,
But the answer is emacs if you like C-x C-c,
I'm here to help you if you're struggling to use free(),
There must be fifty ways to leave your editor.

She said, `I hope your print job doesn't get dequeued,
I hope that you've stocked up lots of vending machine food.
But I'll repeat myself - have you gone and colon-q'd?
There must be fifty ways to leave your editor.
Fifty ways to leave your editor.'

Just press control-K, Ray,
Try double-Z, Fred,
You're still in insert, Bert,
So hit control-C.
Press meta-x, Lex,
Then type, `exit-quit-leave'
Or else just suspend, friend,
and kill(1) it with glee.

She said, `It grieves me now to stty(1) sane,
I wish there was somthing I could do to see your prompt again,'
I said, "I appreciate that,' and started to complain
About the fifty ways.

She said, `Why don't we both just sleep(1) on it tonight?
I'm sure in the morning you'll be sick of reading Byte.'
And then she warned me not to be a power-cycling neophyte,
There must be fifty ways to leave your editor.
Fifty ways to leave your editor.

(Repeat chorus)

2. Addicted to vi

Title : Addicted To Vi
Original : Addicted To Love
Group : Robert Palmer
Author : Chuck Musciano <chuck@trantor.harris-atd.com>
Intro : After thinking about that poor wretch who has become
addicted to vi, I was inspired to compose the following
ditty, sung to the tune of "Addicted To Love" by Robert
Palmer. As you sing this, it may help the effect to imagine
a dozen women, all of whom resemble Bill Joy, dressed in
black and dancing sinuously.
Song :


Addicted To Vi
(with apologies to Robert Palmer)

You press the keys with no effect,
Your mode is not correct.
The screen blurs, your fingers shake;
You forgot to press escape.
Can't insert, can't delete,
Cursor keys won't repeat.
You try to quit, but can't leave,
An extra "bang" is all you need.

You think it's neat to type an "a" or an "i"--
Oh yeah?
You won't look at emacs, no you'd just rather die
You know you're gonna have to face it;
You're addicted to vi!

You edit files one at a time;
That doesn't seem too out of line?
You don't think of keys to bind--
A meta key would blow your mind.
H, J, K, L? You're not annoyed?
Expressions must be a Joy!
Just press "f", or is it "t"?
Maybe "n", or just "g"?

Oh--You think it's neat to type an "a" or an "i"--
Oh yeah?
You won't look at emacs, no you'd just rather die
You know you're gonna have to face it;
You're addicted to vi!

Might as well face it,
You're addicted to vi!
You press the keys without effect,
Your life is now a wreck.
What a waste! Such a shame!
And all you have is vi to blame.

Oh--You think it's neat to type an "a" or an "i"--
Oh yeah?
You won't look at emacs, no you'd just rather die
You know you're gonna have to face it;
You're addicted to vi!

Might as well face it,
You're addicted to vi!

3. Santa Claus Learns Unix

better !pout !cry
better watchout
lpr why
santa claus <north pole >town
cat /etc/passwd >list
ncheck list
ncheck list
cat list | grep naughty >nogiftlist
cat list | grep nice >giftlist
santa claus <north pole > town

who | grep sleeping
who | grep awake
who | grep bad || good
for (goodness sake) {be good}

echo "Oh,"

better !pout !cry
better watchout
lpr why
santa claus <north pole >town

4. Command Line Jokes.

1. % make love
Make: Don't know how to make love. Stop.

2. % got a light?
No match.

3. % sleep with me
bad character

4. % man: Why did you get a divorce?
man:: Too many arguments.

5. % rm God
rm: God nonexistent

6. % make 'heads or tails of all this'
Make: Don't know how to make heads or tails of all this. Stop.

7. % make sense
Make: Don't know how to make sense. Stop.

8. % make mistake
Make: Don't know how to make mistake. Stop.

9. % make bottle.open
Make: Don't know how to make bottle.open. Stop.

10. % \(-
(-: Command not found.

11. % rm -i God
rm: remove God? y
% ls God
God not found

% make light
Make: Don't know how to make light. Stop.

12. % date me
You are not superuser: date not set
Thu Aug 25 15:52:30 PDT 1988

13. % man rear
No manual entry for rear.

14. % If I had a ) for every dollar Reagan spent, what would I have?
Too many )'s.

15. % * How would you describe Bill Clinton
*: Ambiguous.

16. % %Vice-President
%Vice-President: No such job.

17. % ls Meese-Ethics
Meese-Ethics not found

18. % "How would you rate Reagan's senility?
Unmatched ".

19. % [Where is Jimmy Hoffa?
Missing ].

20. % ^How did the^sex change operation go?
Modifier failed.

21. % cp /dev/null sex;chmod 000 sex
% more sex
sex: Permission denied

% strip show
strip: show: Permission denied

22. % who is my match?
No match.

23. % set i="DemocraticPlatform";mkdir $i;chmod 000 $i;ls $i
DemocraticPlatform unreadable

24. % awk "Polly, the ship is sinking"
awk: syntax error near line 1
awk: bailing out near line

25. % %blow
%blow: No such job.

26. % 'thou shalt not commit adultery'
thou shalt not commit adultery: Command not found.

27. $ test my argument
test: too many arguments

28. $ "Amelia Earhart"
Amelia Earhart: not found

29. $ PATH=pretending! /usr/ucb/which sense
no sense in pretending!

30. $ man -kisses dog
dog: nothing appropriate

, Mike

Variable MultiLine Spacing With Sed On Linux Or Unix

Hey There,

Today's post is going to be on a relatively basic subject that can elude a lot of folks who use sed casually. I, myself, for years, only used it to print out matched patterns or do pattern substitutions, but it's capable of much more than that. I don't think it's quite up there with awk (insofar as the programming functionality goes), but it does have lots of neat features (like letting you switch between the current and hold buffer, for example) that can sometimes make it the preferred tool for the job. Especially if you're like me and you're not attached to any one language or interpreter. My basic philosophy is: Whatever can get it done the fastest and most efficiently. I think it works nicely, as philosophies go, since I also compulsively automate everything ;)

For this post, we'll focus on "line spacing" (If anyone calls it something else, I'm referring to the number of spaces between lines). This is one thing that sed can do relatively simply and which is requested quite often from end users. The following tips should work for both Unix and Gnu/Linux sed.

The great thing about this feature is that it's so easy to use that, if you haven't used it already, you'll be amazed at the simplicity :)

Say, for instance, you had a file named FILE with the contents:

abcdef
ghijkl

this is all you would have to do with sed to make it double-spaced (Note that I'm leaving the output redirection to another file, since it's not really germane to the explanation and you probably already know how to redirect the output of one process to another. And, since this blog is for users of all skill levels, check out any of these old posts on using sed in a wide variety of situations if you don't. See below for more information on this feature:

host # sed G FILE
abcdef

ghijkl

# host

Pretty simple, yeah? :) The "G" argument to sed simply appends whatever follows it to the pattern space that gets printed out. Since we didn't have anything to print out, it prints a space by default.

Now, if you want to triple space, it's just a little more complicated, but not much. Here's how you'd do that

host # sed 'G;G' FILE
abcdef


ghijkl


#host

Note that we had to put the arguments in single quotes, since there are (of course) more than one ;) We could have written the last command as:

host # sed 'G' FILE

and gotten the same result as without the quotes, since there was only one argument to sed. If you want to quadruple-space, just use 3 G's, etc. That's about as simple as it gets on a single command line without control characters :)

It's also interesting to note (and overlooked in this old post on using Perl like sed) that you can use "r" and "w" to read from, and write to, files so you don't need to do any redirection outside of your one-liner.

For instance, if you wanted to read information in from another file (FILE2 with contents "xyz") and double space, you could type:

host # sed 'G;r FILE2' <FILE
abcdef

xyz
ghijkl

xyz
host #

Not exactly the same, but pretty useful in different situations.

Also, you can write out to a file, using w, so you don't have to do output redirection after-the-fact, like this (we'll overwrite the FILE2 file with the contents of FILE):

host # sed 'G;w FILE2' <FILE
abcdef

ghijkl

host # cat FILE2
abcdef

ghijkl

host #

Hope you enjoyed today's post and it gets you a little more interested in sed. We'll be looking at in the future, of course, as there are a lot more things you can do with it (especially if you escape the command-line construct)

Cheers,


, Mike

Fight Phishing From The Unix Or Linux Command Line

Hey There,

This weekend, we're going to look at a little something (actually a HUGE something) that we all seem to have to deal with now (through email, IM, etc). It's called phishing and, for lack of a better explanation, it's a fraudulent way for a malicious individual, or group, to get username and password information from you using deceptive practices (was that redundant? ;) It's most commonly used to get information regarding a person's online bank accounts, credit card/mortgage management information, etc. In other words, stuff that could really screw up their lives if someone else had it.

First off, I'd like to be clear that I am "not against SPAM." As annoying as it may be, I don't begrudge anyone their right to send me a sales pitch, even if that product is a complete scam and I didn't ask to be pitched. I've been throwing stuff out that I get in my regular postal mail since I've been able to throw stuff out and I'm not sure if I've ever double-opted-in (the ethical way) or just had my address sold to some random company when it comes to internet SPAM.

Phishing, on the other hand, upsets me to no end. I do begrudge everyone their right to try and, literally, steal from others, using the internet. I feel the same way about the folks who used to steal carbons from credit cards (when that trick used to work... sadly, to a degree, it still does) and that other, more honest, faction of individuals who walk right up to the counter, point a gun in a bank teller's face and demand everyone else's hard earned cash. I'm not that this sort of activity should be encouraged, or glorified, in any way. I'm just saying it, at least, shows some character. If you're going to steal, and that's your business, why not come right out and say it? Only cowards pretend they're not pointing that figurative gun in your face and giving you a good mugging.

I can, literally, feel my email box filling with requests for password-resets at my bank of choice. Funny thing is, a lot of times, these clowns don't even bother to verify your personal information before they try and scam you. IMPORTANT NOTICE TO PHISHER'S WORLDWIDE: I do not have an account at the Federated Bank Of Hindustan. And, even if I did, it's been so long since I've attempted to speak Urdu that I can't possibly remember my clever username ;)

If you have the time and inclination to hurt these phishers, check out sites like The Anti-Phishing Working Group and PhishTank, and read up on how this whole operation works at sites like Security Focus. PhishTank actually even has a plug in for FireFox named Site Checker. Here's another site that provides a great list of possible phishing programs that may be running on your machine. The advantages that these organizations bring is that they don't merely do "anti-spam" work (which is, by all means, necessary and laudable). They actually collect data on phishing schemes, get to these people backward through their own chains of deceit and, if possible, have them put in cell block A (or wherever it is that they can legally squirrel away thieves and murderers - preferably together).

Unfortunately, though the calls-for-action are many, you should not expect any automated anti-phishing toolbars, etc, for your web browser. There's a very good reason why they're slow in coming out. Phishing is a lot like "social engineering," whereby a stranger attempts to extract personal information from you under a false pretense. Using this as an example, a stranger may also innocently request information you're not comfortable with giving under a legitimate pretense (?) or it all may be a big misunderstanding. This is why most phishing, SPAM and "social engineering" confidence violations are handled by manual submission or, at least, with some degree of user interaction. If you set Internet Explorer on auto-pilot to trash every site it thought was a phishing site, I'd bet about 50% of legitimately required bank (or other e-commerce) authorization, or confirmation, emails would end up getting skunked and ruin your web experience (and a few reputations in the process) very very quickly.

To wrap this all up (although it's certainly not the only solution), I prefer to use PhishTank since they have a very easy email submission process (You must register with them to send phishing complaints, and I encourage you to investigate their legitimacy on your own; but I like 'em a lot) which works whether or not you're sending them phishing information from your PC email client or from your Linux or Unix box (or any box that you may not have actually "received" the message on -- more below).

This is where the Linux and Unix command line comes into play. While you can easily forward an email from your mail client of choice to phish@phishtank.com (Not a mailto link on purpose), you will almost always lose some vital information in the forwarding. Instead I prefer to do the following:

1. From my email software of choice, view the entire email, including headers <-- Aside from the phishing URL, this is the most important part!

2. Log on to my Linux box (like I'm not already ;) and use any editor, like vi to paste the "entire contents" of the above output into a file.

3. Then, I send email to PhishTank using the customized email address that they've provided me. Their customized email address, for a user to send reports in from anywhere, is of the following format: phish-YOURACCOUNTNAME.YOUR_RANDOMLY_GENERATED_PHISHTANK_KEY@phishtank.com. Note that your randomly generated key will remain the same unless you want to change it. You can use any email client you like to send them the email, then. Assuming you put in it a text file called WTF, you might use mailx to send it, like so (Note that no Subject line is actually necessary):

host # mailx -s "Phishing Abuse Report" phish-YOURACCOUNTNAME.YOUR_RANDOMLY_GENERATED_PHISHTANK_KEY@phishtank.com < WTF

And, just like that, you've launched an investigation and, hopefully, put some purse-snatcher out of business.

And, lest it be said that I never said it, always double check any emails you receive from PhishTank, any bank, any service, even from me :) It never hurts to take a second and make sure you're not being scammed. The alternative can be devastating!

Best wishes,

, Mike

More Fun With Security Through Obfuscation On Linux And Unix

Hey There,

A few months ago, we ran a post on security through obfuscation, regarding ways to make your code less comprehensible to folks who didn't understand it as well as you do. While, at the core, this does nothing to actually "secure" your code from anyone with the patience to unravel it, it can be useful for keeping folks who know "just enough to do a lot of damage" from messing with your scripts ;) And, on another level, it can be a useful stalling technique, since you'll be making anyone work to figure out how your script/code works before they can change it without risking destroying it.

While I'm sure there are many more points to cover (and, hopefully, we'll cover them all eventually ;), today's addition to the arsenal is pretty obvious: Messing with your variable names :)

Most decent and responsible scripters will try to have their variable names make sense (if for no one's benefit than for their own - writing a confusing script isn't going to be worth it if even you can't figure it out later ;). So, when they use a variable to use as a counter, they might name it $counter, or something like that. @array_of_files is another such descriptive name. Very helpful later on when you need to modify your own stuff, and good practice so that, if you ever leave a place of employment, you won't be screwing your friends and co-workers who will, inevitably, get stuck maintaining your code.

However, assuming you do something as simple as keeping a separate file with a simple legend, like:

VAR ORIG
a file
b count
...

and so forth, naming your variables as arcanely as possible is a great way to make long, convoluted, stream-of-consciousness scripts as close to incomprehensible as possible ;) Having this sort of legend available can make it easy for you to do variable replacement within the "confusing" version of the script to make it easily understandable once again.

For instance, this Perl script to reverse all lines in a file on Linux or Unix can be rewritten to be even more confusing that it may already be.

Following is a working, but awful looking, revision of the original code, modified to include meaningless variables (scalar, array and filehandle), no indentation, no spaces between lines and no comments (except at the top). I've opted not to include a legend, since we have the original code hosted on this blog (linked to above) and you won't need to go through the hassle of reversing it.

In a future post, we'll look at creating such a legend for any script and backward-applying changes to your code (even though it would be much easier to do with "patch" ;)

Enjoy :)





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if ( $#ARGV != 0 )      {
exit(1);
}
$a=$ARGV[0];
open(A, "<$a");
@a = <A>;
close(A);
foreach $b (@a)      {
chomp($b);
@b = split(//, $b);
$c = @b;
undef(@c);
while ( $c > 0 )   {
push(@c, $b[$c-1]);
$c--;
}
$e = join(/ /, @c);
unshift(@f, "$e");
}
open(B, ">$a.reverse");
foreach $d (@f)    {
print B "$d\n";
}
close(B);
exit(0);

, Mike

Using Perl On Linux To Do Mass Synchronization Of File Time Stamps

Hey There,

Today we're going to take a look at another quick and simple Perl command line execution statement that you can use to save yourself lots of time ( If it really does equal money, this post is going to be a lot more valuable than I originally thought ;)

In a previous post, we looked at using Perl to figure out how old all our files really are. This post is a little twist on that; and it's a lot less complicated. It will run on whatever version of Perl you have (unless it's ancient ;) and produced equal results on all tested Linux and Unix flavours I could get my hands on.

Today we're going to be using Perl's stat and utime functions (similar to the way we did in our previous Perl post), but instead of using them do determine the ages of all of our files, we're going to use them make all of our files conform to the specific time and date of one representative file. This trick can come in really handy if you wanted to make a system backup consisting only of files that, let's say, are a certain number of days old, and only some of the files that you know you need to backup are slightly older or newer.

The trick is very simple. All you need to do is find one specific file that has a time stamp you want to mimic, and then apply its time stamp attributes to the group of files that you want to modify the time stamp on. So (for a simple example) if you had a directory consisting of 5 files, like this:

host # ls
file1 file2 file3 file4 file5

and you needed them all to share the same time stamp as the file "file1," I'd first suggest that you take some form of online-backup, using tar or cpio (assuming that you may need to reverse the process later):

host # tar cpf /tmp/backout.tar file1 file2 file3 file4 file5

or

host # tar cpf /tmp/backout.tar f* <--- I rarely glob this globally, but, for our example, all the files we want to back up start with "f" and there aren't any others. Worst case, on a tar backup, you can just write over it if you accidentally tar-copy something you didn't want to.

Now, let's take one more look at those files. We'll use "ls -l" to show the time stamp at this time:

host # ls -l
total 4
-rw-r--r--  1 eggi  newpeople  90 May 13 22:27 file1
-rw-r--r--  1 eggi  newpeople   0 May 25 14:25 file2
-rw-r--r--  1 eggi  newpeople   0 May 25 14:25 file3
-rw-r--r--  1 eggi  newpeople   0 May 25 14:25 file4
-rw-r--r--  1 eggi  newpeople   0 May 25 14:25 file5

Now, well make them all have the same time stamp.

We need files 2 through 5 to match the time stamp of file1, as this is how the system is going to determine what to back up later in the day (fill in your own hypothetical situation here ;)

And here's all we have to do to make all the files have the same time stamp as file1 :)

host # perl -e '$x=utime ((stat($ARGV[0]))[8,9], @ARGV);print $x' file1 file[2345]

And we should be all set!

host # ls -l
total 4
-rw-r--r--  1 eggi  newpeople  90 May 13 22:27 file1
-rw-r--r--  1 eggi  newpeople   0 May 13 22:27 file2
-rw-r--r--  1 eggi  newpeople   0 May 13 22:27 file3
-rw-r--r--  1 eggi  newpeople   0 May 13 22:27 file4
-rw-r--r--  1 eggi  newpeople   0 May 13 22:27 file5

Now all the files have the same time stamp! You'll note, in the command above, that we only modifed the atime and mtime of the files. This is sufficient because Perl's utime function automatically changes the ctime (or inode change time) to the current date and time. You can include the utime in this Perl command line execution statement (no error will be generated), but it's not necessary.

If you ever do need to copy the ctime, just add the 10th index of the array returned by stat to your command line, like this:

host # perl -e '$x=utime ((stat($ARGV[0]))[8,9,10], @ARGV);print $x' file1 file[2345]

And, of course, you can use this same process on directories as well, without a direct correlation (e.g. copying file timestamps to directories, vice versa and any combination between file types that support time stamps will work :)

Here's hoping this helps you out (at least, a little ;) This simple command can be a very convenient way for you to make massive changes by applying the time stamp of any one file to as many other files as you like. And, no matter how many files you need to do this for, as long as you can wait a few moments for ingenuity to strike, the command line shouldn't get too much longer :)

Cheers,

, Mike