Manual Mass File Patching On Linux Or Unix

Good Day Good People :)

In previous posts we've looked at various aspects of the diff command, including augmenting it to generate unique content only and work with users, groups and permissions.

Today, I thought we'd look at diff's usage in a, generally, manual process and peek at the first step toward automation. Everything in this post can be scripted out to provide more granular control, and done much more easily. I thought doing it in a laborious tutorial style to start out with would be best to get the major parts of the process laid out simply and, hopefully, written in such a way that they're simply understood. After reading that last sentence, I'm wishing myself lots of luck ;)

For purposes of our example today, we're going to assume that we have a directory in which we keep 3 executable files. All of these files are approximately the same, because they all do approximately the same thing, just for different programs. For simplicity, they've been named: file1, file2 and file3, as shown in this directory listing:

host # ls
.  ..  file1  file2  file3

All of their contents are also almost totally identical:

host # cat file1 file2 file3
#!/bin/bash
# Start File1 Process

COMMAND_ARGS="-d --takeforeverandaday"

HOMEDIR="/usr/binky"

#!/bin/bash
# Start File2 Process

COMMAND_ARGS="-d --takeforeverandaday"

HOMEDIR="/usr/binky"

#!/bin/bash
# Start File3 Process

COMMAND_ARGS="-d --takeforeverandaday"

HOMEDIR="/usr/binky"

For today (and this would work if we had 5000 files, but this post is going to be long enough ;) all we need to do is change the HOMEDIR standard variable in all of our scripts to BASEDIR. I'm not exactly sure why. Someone with more authority than me or you decided it would be a good idea. They will probably change their mind and have us switch it back next week ;)

The first thing we should do is create a backup directory and do our work there.

host # mkdir ../u
host # cp file* ../u
host # cd ../u
host # ls
.  ..  file1  file2  file3

Then we'll just use a simple while loop and feed each file to sed and make the substitution, which we'll dump into another file called FILENAME.diff (So, file1 will get the substitution made on it by sed, and that difference will be put into a new file called file1.diff). Within that same loop, we'll be using the actual command "diff -e" to create an "ed" (A very basic Unix and Linux editor) patch file. Note that, for file1, this new file will be called file1.patch and that we echo a "w" and a "q" (on separate lines on purpose) into the bottom of the patch file.

host # ls -1d *|while read x;do sed s'/HOMEDIR=\"\(\/usr\/binky\)\"/BASEDIR=\"\1\"/' $x >$x.diff;diff -e $x $x.diff >$x.patch;echo w >>$x.patch;echo q >>$x.patch;done

We could have deleted our file*.diff files above, as well, but I thought you might like to take a look at them. These serve as proof that our sed command actually worked :) HOMEDIR is now BASEDIR!

host # cat file1.diff file2.diff file3.diff
#!/bin/bash
# Start File1 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

#!/bin/bash
# Start File2 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

#!/bin/bash
# Start File3 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

Now, let's delete the *.diff files and take a look at the patch files. These are all considered "ed" scripts. This means that we can feed them to the "ed" Linux/Unix editor and have it execute commands for us automatically. This will come in handy in a second :)

host # rm *.diff
host # cat file1.patch file2.patch file3.patch
5c                    <--- These first three
BASEDIR="/usr/binky"  <--- lines were created
.                     <--- by "diff -e"
w                     <--- We manually added
q                     <--- these last two!
5c
BASEDIR="/usr/binky"
.
w
q
5c
BASEDIR="/usr/binky"
.
w
q

The next step in the process is to apply those patches. We're going to be doing this all in our test directory, so there's no need to tar up all the patch files for a move to our "live" script directory

host # ls
.   file1        file2        file3
..  file1.patch  file2.patch  file3.patch

So, now we're ready to test our mass change and pray all goes well. Note that, since we're still in our backup directory, we can do lots of damage to the scripts in here and it shouldn't make a difference to anyone. Only we will have to live with the shame ;)

Now, we'll do another fancy command line and pipe the output of "ls -1d *", grepping out all the patch files and then using "ed" to patch each file in the directory. Notice that the syntax for patching a file with an "ed" diff script is "ed FILENAME <FILENAME.patch" - This would normally hang and wait for us to send it a control-D or some other signal, but, since we echoed the "w" and "q" characters into the patch files, "ed" knows to write (w) and quit (q) - At the end of the command pipe, we'll purposefully not delete all the patch files. You can remove them if you like, but we'll be looking at using them to undo changes in a future post!

host # ls -1d *|grep -v patch|while read x;do ed $x <$x.patch;done
107
107
107
107
107
107
host # ls
.   file1        file2        file3
..  file1.patch  file2.patch  file3.patch

Okay, we're all done and now we can check the results. Looks like everything in our (potentially huge amount of) files got patched appropriately :) This shouldn't be a surprise since we checked our sed output already, but you never know.

host # cat file1 file2 file3

#!/bin/bash
# Start File1 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

#!/bin/bash
# Start File2 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

#!/bin/bash
# Start File3 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

In a future post we'll look at a script that will allow us to do what we've just done today and make life a little easier when doing mass updates or patching :)

Cheers,

, Mike

linux unix internet technology