The Linux and Unix Menagerie: automate

Good Day Good People :)

In previous posts we've looked at various aspects of the diff command, including augmenting it to generate unique content only and work with users, groups and permissions.

Today, I thought we'd look at diff's usage in a, generally, manual process and peek at the first step toward automation. Everything in this post can be scripted out to provide more granular control, and done much more easily. I thought doing it in a laborious tutorial style to start out with would be best to get the major parts of the process laid out simply and, hopefully, written in such a way that they're simply understood. After reading that last sentence, I'm wishing myself lots of luck ;)

For purposes of our example today, we're going to assume that we have a directory in which we keep 3 executable files. All of these files are approximately the same, because they all do approximately the same thing, just for different programs. For simplicity, they've been named: file1, file2 and file3, as shown in this directory listing:

host # ls
.  ..  file1  file2  file3

All of their contents are also almost totally identical:

host # cat file1 file2 file3
#!/bin/bash
# Start File1 Process

COMMAND_ARGS="-d --takeforeverandaday"

HOMEDIR="/usr/binky"

#!/bin/bash
# Start File2 Process

COMMAND_ARGS="-d --takeforeverandaday"

HOMEDIR="/usr/binky"

#!/bin/bash
# Start File3 Process

COMMAND_ARGS="-d --takeforeverandaday"

HOMEDIR="/usr/binky"

For today (and this would work if we had 5000 files, but this post is going to be long enough ;) all we need to do is change the HOMEDIR standard variable in all of our scripts to BASEDIR. I'm not exactly sure why. Someone with more authority than me or you decided it would be a good idea. They will probably change their mind and have us switch it back next week ;)

The first thing we should do is create a backup directory and do our work there.

host # mkdir ../u
host # cp file* ../u
host # cd ../u
host # ls
. .. file1 file2 file3

Then we'll just use a simple while loop and feed each file to sed and make the substitution, which we'll dump into another file called FILENAME.diff (So, file1 will get the substitution made on it by sed, and that difference will be put into a new file called file1.diff). Within that same loop, we'll be using the actual command "diff -e" to create an "ed" (A very basic Unix and Linux editor) patch file. Note that, for file1, this new file will be called file1.patch and that we echo a "w" and a "q" (on separate lines on purpose) into the bottom of the patch file.

host # ls -1d *|while read x;do sed s'/HOMEDIR=\"$\/usr\/binky$\"/BASEDIR=\"\1\"/' $x >$x.diff;diff -e $x $x.diff >$x.patch;echo w >>$x.patch;echo q >>$x.patch;done

We could have deleted our file*.diff files above, as well, but I thought you might like to take a look at them. These serve as proof that our sed command actually worked :) HOMEDIR is now BASEDIR!

host # cat file1.diff file2.diff file3.diff
#!/bin/bash
# Start File1 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

#!/bin/bash
# Start File2 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

#!/bin/bash
# Start File3 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

Now, let's delete the *.diff files and take a look at the patch files. These are all considered "ed" scripts. This means that we can feed them to the "ed" Linux/Unix editor and have it execute commands for us automatically. This will come in handy in a second :)

host # rm *.diff
host # cat file1.patch file2.patch file3.patch
5c <--- These first three
BASEDIR="/usr/binky" <--- lines were created
. <--- by "diff -e"
w <--- We manually added
q <--- these last two!
5c
BASEDIR="/usr/binky"
.
w
q
5c
BASEDIR="/usr/binky"
.
w
q

The next step in the process is to apply those patches. We're going to be doing this all in our test directory, so there's no need to tar up all the patch files for a move to our "live" script directory

host # ls
. file1 file2 file3
.. file1.patch file2.patch file3.patch

So, now we're ready to test our mass change and pray all goes well. Note that, since we're still in our backup directory, we can do lots of damage to the scripts in here and it shouldn't make a difference to anyone. Only we will have to live with the shame ;)

Now, we'll do another fancy command line and pipe the output of "ls -1d *", grepping out all the patch files and then using "ed" to patch each file in the directory. Notice that the syntax for patching a file with an "ed" diff script is "ed FILENAME <FILENAME.patch" - This would normally hang and wait for us to send it a control-D or some other signal, but, since we echoed the "w" and "q" characters into the patch files, "ed" knows to write (w) and quit (q) - At the end of the command pipe, we'll purposefully not delete all the patch files. You can remove them if you like, but we'll be looking at using them to undo changes in a future post!

host # ls -1d *|grep -v patch|while read x;do ed $x <$x.patch;done
107
107
107
107
107
107
host # ls
. file1 file2 file3
.. file1.patch file2.patch file3.patch

Okay, we're all done and now we can check the results. Looks like everything in our (potentially huge amount of) files got patched appropriately :) This shouldn't be a surprise since we checked our sed output already, but you never know.

host # cat file1 file2 file3

#!/bin/bash
# Start File1 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

#!/bin/bash
# Start File2 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

#!/bin/bash
# Start File3 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

In a future post we'll look at a script that will allow us to do what we've just done today and make life a little easier when doing mass updates or patching :)

Cheers,

, Mike

linux unix internet technology

Hey again,

A great tool to use when you want to do a little automation in your scripts is a little thing commonly referred to as a "Here Document." I still don't know why, but it hasn't ever mattered ;)

Of course, the "Here Document" can't automate everything (SSH, Telnet or basically anything that requires an interactive terminal login -- There's an excellent tool for that called Expect - but that's for another day)

The most basic use of a "Here Document" is to help automate FTP'ing of files.

The following scriptlet illustrates this as such:

This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/ksh

#
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

file=$1

# Now let's ftp that file to our host

# Begin "Here Document"
/usr/bin/ftp <<- END
open ftphost.xyz.com
user bjohnson
BJOHNSON
cd uploads
put $file
bye
END

# End of "Here Document"

exit

Notice that the structure of the "Here Document" is fairly simple You have the program you want to invoke on the first line.

That's followed by either <<- KEYWORD or << KEYWORD. The main difference is that the <<- Form will strip leading tabs (not spaces, though) from the lines between the beginning and ending KEYWORD. This option basically arose so that users could write "Here Document"s and make them more easily readable.

The KEYWORD is also important. The only real things that are important about it are that it be the same at the beginning and end. The KEYWORD is a placeholder, telling your "Here Document" what to look for so it knows when it's done :) Another, less conspicuous thing is that it shouldn't be the name of any other variable in the script. General usage dictates all uppercase and, more often than not, the keyword EOF. Ultimately, it's up to you.

Downsides to doing FTP this way are that you have to write the password in plain-text (BJOHNSON above) and, even though this is more of a downside to "Here Document"s altogether, you cannot comment within the KEYWORD brackets.

You can do a lot more with here documents. Pretty much any command you can pipe input to can be used. Have fun trying it out!

, Mike

linux unix internet technology