Minor Fix Update - Script To Unpack Solaris Datastream Pkg Files Without Sun's Pkg Utils

Hello again,

Updated 4/4/08 to output pkgproto results to a file named prototype. Originally had it writing to a file named pkgproto. Typo corrected :)

It's been a while since I started looking for a way to finish up our series of articles on RPM and pkg ripping and creating. We started out with creating our own RPM's and did pretty much everything I could think of to those poor packages, all the way through creating RPM's from already installed RPM's and converting RPM's to Solaris pkg's ;)

The only project I had left over (from that bunch) was how to extract the binary contents of Solaris Unix datastream pkg files.

Unfortunately, unlike the "rpm2cpio" command that Solaris includes, they have no regular program for converting a datastream package to an unpackable file (like a cpio or tar), unless you count all the basic pkg commands. What I was looking for was a way to extract a datastream pkg file, and get all the binary contents, without actually installing it. I got no love from Solaris, but I got some free time for myself and eventually figured it out ;)

I've included a script, below, to rip apart a Solaris datastream pkg and create a subdirectory (named the name of the pkg) into which I dump the pkginfo, pkgmap, pkgproto (which I derive from the pkgmap) and all the binary files that the pkg contains. The script, itself, is far from perfect, but if you check it out you can see, fairly easily, how you can manually pull apart a Solaris datastream pkg file and get to the binary contents that you really want (At least, I'm assuming you do - just like me ;)

Note that the version of cpio that comes with Solaris (in the SUNWcsu coreutils package) will complain about garbage bits in your cpio header, when you've clipped off the top of the pkg file. There are really only 2 garbage bits you have to delete (which you can do with vi), but I prefer to use the Gnu version of cpio which will "magically" ignore garbage bits and extract the cpio file no matter what. This version of cpio comes standard on Linux and can be downloaded (in Solaris datastream pkg format) from sunfreeware.com or (in source format, with links to all sorts of OS ports) Gnu's CPIO Download site - just in case you don't want to have to rip apart your Solaris pkg's on Linux (or don't have both OS's at your workplace).

Another strange thing is that Sun's version of cpio won't make directories, by default, that don't exist when you unpack a cpio archive (???) All I'm really saying is: Get the Gnu version - It's free and it'll save you many a gray hair ;)

I've tested today's script on multiple packages from sunfreeware.com and had great success with it working right the first time (assuming that I have Gnu's cpio). The script has broken on a few custom pkg's I've put together myself and can use a lot of work in the "additional" areas (like ensuring pre and post-install scripts get accounted for, depend files get tagged, checkinstall programs are properly extracted, etc). Still, it's a good beginning :)

Hopefully this will be of help to you, if you've been looking all over for something like this, and, almost definitely, can be the starting point for a fully functional Solaris datastream pkg ripper :)

If you want to do this manually, in short terms, just vi the pkg file you're interested in and delete everything up to the final instance of "pkginfo" in the file (after the final "pkgmap" line) and save the rest into another file, which you can manipulate with cpio. Who'd have thought it would end up being that simple? :)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# rip_pkg - rip apart Solaris pkg's and get the binaries out
#
# 2008 - Mike Golvach - eggi@comcast.net
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if ( $#ARGV != 0 ) {
 print "Usage: $0 pkg_file\n";
 exit(1);
}

$pkg_name = $ARGV[0];

# check and make sure it's a SVR4 DataStream Pkg

chomp($simple_header_check=`grep -i "package datastream" $pkg_name >/dev/null 2>&1;echo \$?`);

if ( $simple_header_check != 0 ) {
 print "This doesn't appear to be a valid SVR4 DataStream pkg file.  Exiting...\n";
 exit;
}

# Memory Hogging Here - Until This Cold Quits Me And I Can Think My Way Out Of This ;)

open(PKG, "<$pkg_name");
@pkg_file = <PKG>;
close(PKG);

# Split out header information and kludgey cpio file with binary contents

$header_end = 0;
$cpio_start = 0;
open (P_HEADERS, ">$pkg_name.headers");
foreach $line (@pkg_file)   {
 if ( $header_end == 0 && $line =~ /TRAILER/ && $line =~ /pkginfo/ )  {
  print P_HEADERS $line;
  $header_end = 1;
 } elsif ( $header_end == 1 && $line =~ /pkginfo/ ) {
  print P_HEADERS $line;
  close(P_HEADERS);
  $cpio_start = 1;
  open(P_CPIO, ">$pkg_name.cpio");
 } elsif ( $cpio_start == 0 ) {
  print P_HEADERS $line;
 } elsif ( $cpio_start == 1 ) {
  print P_CPIO $line;
 } else {
  print STDERR "WTF?\n";
 }
}
close(P_CPIO);

open(P_HEADERS, "<$pkg_name.headers");
@p_headers = <P_HEADERS>;
close(P_HEADERS);

$output_dir = 0;
$pkginfo_start = 0;
$pkginfo_seek = 0;
$pkgmap_start = 0;
foreach $p_head (@p_headers)  {
 if ( $output_dir eq 1 && $pkginfo_seek == 0 )  {
  $output_dir = $p_head;
  $output_dir =~ s/^(\w+)\W*.*$/$1/;
  chomp($output_dir);
  $pkginfo_seek = 1;
 } elsif ( $output_dir eq 0 && $p_head =~ /datastream/i ) {
  $output_dir = 1;
 } elsif ( $output_dir eq 0 && $pkginfo_start == 0 ) {
  next;
 } elsif ( $output_dir ne 0 && $p_head =~ /pkgmap/ ) {
  $pkgmap_start = 1;
  $pkginfo_start = 0;
 } elsif ( $pkgmap_start == 1 && $p_head =~ /pkginfo/ ) {
  chomp($p_head);
  push(@pkgmap, $p_head);
  last;
 } elsif ( $output_dir ne 0 && $p_head =~ /pkginfo.*PKG=/ ) {
  $pkginfo_start = 1;
 } elsif ( $output_dir ne 0 && $pkgmap_start == 1 ) {
  chomp($p_head);
  push(@pkgmap, $p_head);
 } elsif ( $output_dir ne 0 && $pkginfo_start == 1 ) {
  chomp($p_head);
  push(@pkginfo, $p_head);
 } else {
  print "WTF NOTHING MATCHED\n OD $output_dir PST $pkginfo_start PS $pkginfo_seek PMST $pkgmap_start\n$p_head\n";
 }
}

mkdir("$output_dir");
chdir("$output_dir");
open(PKGINFO_OUT, ">>pkginfo");
foreach $info (@pkginfo) {
 print PKGINFO_OUT "$info\n";
}
open(PKGMAP_OUT, ">>pkgmap");
foreach $map (@pkgmap) {
 print PKGMAP_OUT "$map\n";
}
open(PROTO_OUT, ">>prototype");
foreach $proto (@pkgmap) {
 @proto = split(" ", $proto);
 print PROTO_OUT "$proto[1] $proto[2] $proto[3] $proto[4] $proto[5] $proto[6]\n";
}
system("cpio -iv <../$pkg_name.cpio >/dev/null 2>&1");
chdir("reloc");
system("tar cpf - *|(cd ../;tar xpf -)");
chdir("../");
system("rm -r reloc");
chdir("../");
unlink "$pkg_name.headers";
unlink "$pkg_name.cpio";

, Mike

linux unix internet technology

Linux Update Patch For Expect User Removal Script

Hey There,

Today we have the "Linux Additions" patch to follow up on yesterday's post on removing old users network wide.

Note that this patch was created with "diff -c," so I've been sure to specify both the file I want to patch and the patch file in my arguments to "patch" (whereas I didn't need to in our post on easily patching multiple files). The exact command line to create the patch was:

host # diff -c eraser eraser.new > eraser.patch

All you'll need to do to update yesterday's script to include today's modifications is put the patch file (tagged on to the end of this post) in the same directory as the original script (or wherever you want, assuming you know how this all works and my explaining it is trivial ;) and name it something like "eraser.patch" - Then do the following:

host # ls
. .. eraser eraser.patch
host # cp eraser eraser.bak <--- Just in case something goes wrong, I'm always for making backups :)

As I noted before, you'll want to be specific in your arguments to "patch" when you run this, or be careful of "patch"'s default behaviour and don't choose to do a reverse patching, like this (BTW, even though my new script "eraser.new" is listed in the patch file, you don't actually need to have it for all of this to work):

host # patch -p0 -i eraser.patch
Hmm... Looks like a new-style context diff to me...
The text leading up to this was:
--------------------------
|*** eraser Sat Mar 29 18:57:29 2008
|--- eraser.new Sat Mar 29 18:57:50 2008
--------------------------
Patching file eraser using Plan A...
Reversed (or previously applied) patch detected! Assume -R? [y]
^C

Instead, patch it like this, so you can do it hands off:

host # patch -p0 eraser eraser.patch
Hmm... Looks like a new-style context diff to me...
The text leading up to this was:
--------------------------
|*** eraser Sat Mar 29 18:57:29 2008
|--- eraser.new Sat Mar 29 18:57:50 2008
--------------------------
Patching file eraser using Plan A...
Hunk #1 succeeded at 3.
Hunk #2 succeeded at 36.
Hunk #3 succeeded at 72.
Hunk #4 succeeded at 145.
Hunk #5 succeeded at 235.
Hunk #6 succeeded at 267.
done

And "diff -c" will show that your "eraser" Expect script is now different than the original that you copied off before:

host # diff -c eraser eraser.bak
*** eraser Sat Mar 29 19:02:21 2008
--- eraser.bak Sat Mar 29 18:57:50 2008
***************
...

And, of course, you can just check your "eraser" file and see the changes. If you need to back out the patch, simply do the "reverse patch" operation, like so:

host # patch -R -p0 eraser eraser.patch
Hmm... Looks like a new-style context diff to me...
The text leading up to this was:
--------------------------
|*** eraser Sat Mar 29 18:57:29 2008
|--- eraser.new Sat Mar 29 18:57:50 2008
--------------------------
Patching file eraser using Plan A...
Hunk #1 succeeded at 3.
Hunk #2 succeeded at 36.
Hunk #3 succeeded at 70.
Hunk #4 succeeded at 130.
Hunk #5 succeeded at 186.
Hunk #6 succeeded at 217.
done

And then do a cursory check to make sure the unpatched file matches the original that you copied off before:

host # diff -c eraser eraser.bak <--- You'll get no output if the files are the same. That's good :)

Enjoy and have a restful Sunday :)





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

*** eraser        Sat Mar 29 18:57:29 2008
--- eraser.new  Sat Mar 29 18:57:50 2008
***************
*** 3,9 ****
  log_user 0

  #########################################################################
! # eraser - root out ex-employees across network
  # 2008 - Mike Golvach - eggi@comcast.net
  # <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/us/">Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License</a>
  #
--- 3,9 ----
  log_user 0

  #########################################################################
! # eraser - root out ex-employees across network - with Linux procs added
  # 2008 - Mike Golvach - eggi@comcast.net
  # <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/us/">Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License</a>
  #
***************
*** 36,41 ****
--- 36,43 ----
    puts " sun1 sun2"
    puts "HP Hosts:"
    puts " hp1 hp2"
+   puts "LINUX Hosts:"
+   puts " linux1 linux2"
    puts "#############################################################################"
    puts "FLAGS:"
    puts " -h: Show this message."
***************
*** 70,75 ****
--- 72,90 ----
   expect "word: " {send "$qpass\r"}
  }

+ proc linux_login {name tprompt login lpass} {
+  send_user "$name\n"
+
+  set timeout 3
+  set hostname $name
+  set prompt $tprompt
+  set qlogin $login
+  set qpass $lpass
+
+  expect "ogin: " {send "$qlogin\r"}
+  expect "word: " {send "$qpass\r"}
+ }
+
  proc hp_login {tprompt login lpass} {

   set timeout 12
***************
*** 130,135 ****
--- 145,184 ----
   }
   return "$hostname $insidepwd $insidealias"
  }
+
+ proc linux_q {tprompt login lpass userlogin host} {
+  set timeout 5
+  set log $login
+  set opass $lpass
+  set prompt $tprompt
+  set ulogin $userlogin
+  set hostname $host
+  set insidealias none
+  set insidepwd none
+
+  send_user "Scanning $hostname...  "
+
+         expect -re $tprompt {send "grep $ulogin /etc/passwd\r"}
+  expect -re $tprompt
+         send "echo $?\r"
+         expect -re "\r\n(.*)\r\n"
+         set returnval $expect_out(1,string)
+         if { $returnval == 0 }  {
+   set insidepwd 1
+         }
+         expect -re $tprompt {send "grep $ulogin /etc/mail/aliases\r"}
+  expect -re $tprompt
+         send "echo $?\r"
+         expect -re "\r\n(.*)\r\n"
+         set returnval $expect_out(1,string)
+         if { $returnval == 0 }  {
+   set insidealias 1
+         }
+         expect -re $tprompt {
+   send "exit\r"
+  }
+  return "$hostname $insidepwd $insidealias"
+ }

  proc sco_q {tprompt login lpass userlogin host} {
   set timeout 5
***************
*** 186,191 ****
--- 235,241 ----
  set sun [list sun1 sun2]
  set sco [list sco1 sco2]
  set hp [list hp1 hp2]
+ set linux [list linux1 linux2]

  set userlogin [lindex $argv 0]
  set login [lindex $argv 1]
***************
*** 217,222 ****
--- 267,278 ----
   spawn telnet $host
   hp_login $tprompt $login $lpass
   lappend finalists [sunhp_q $tprompt $login $lpass $userlogin $host]
+ }
+
+ foreach host $linux {
+  spawn telnet $host
+  linux_login $tprompt $login $lpass
+  lappend finalists [linux_q $tprompt $login $lpass $userlogin $host]
  }

  foreach host $sco {

, Mike

linux unix internet technology

Removing Old Users Network Wide Using Expect

Hey there,

Here comes another Expect monster-sized weekend script for you ;)

Today's script is a twist on a previous script we posted to do network wide updates. The essence is still the same. This should basically save you the hassle of having to log in to a million different servers to find and remove users who you know shouldn't have accounts on your boxes.

Again, you can also delete the "Weed out the undesirables pronto" section if you don't want to minimally secure the script from unauthorized usage by doing a logname check.

We've left out the login functionality for Linux this time, but we'll add it tomorrow as a "diff -c" patch file and add it to this file with "patch" to give the week an appropriate finish :)

There's a section I wrote specifically for an Informix sql database, that you can modify (or remove) if you like, to include an example of how to check for users (or anything, really) in a database. If you don't want or need that functionality, just delete the "$userlogin mail server sql user report:" section. All the sections are highlighted with tons of pound signs (#)

Once you edit this Expect script to suit your needs, you can run it easily, like so (with -h if you want to see the help screen):

Ex:
host # ./eraser

or

host # ./eraser -h

See you tomorrow with the Linux additions. Have a great weekend :)





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/local/bin/expect

log_user 0 

#########################################################################
# eraser - root out ex-employees across network
# 2008 - Mike Golvach - eggi@comcast.net
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
# 
# Usage: eraser <-h> [User Login] [Your Login]
#########################################################################

#########################################################################
# Weed out the undesirables pronto
#########################################################################

set gauntlet [exec logname]
if { [string compare $gauntlet "user1"] != 0 && [string compare $gauntlet "user2"] != 0 }   {
 puts "Sorry, $gauntlet. You may not use eraser."
 exit
}           

#########################################################################
# Go for the help short-circuit
#########################################################################

foreach member $argv {
 if { $member == "-h" } {
  puts "#############################################################################"
  puts "USAGE:"
  puts " $argv0 <-h> \[User Login\] \[Your Login\]"
  puts "#############################################################################"
  puts "SCO Hosts:"
  puts " sco1 sco2"
  puts "SUN Hosts:"
  puts " sun1 sun2"
  puts "HP Hosts:"
  puts " hp1 hp2"
  puts "#############################################################################"
  puts "FLAGS:"
  puts " -h: Show this message."
  puts " User Login: Theirs."
  puts " Your Login: Yours."
  puts "#############################################################################"
  puts "A FEW VALID COMMAND LINE EXPRESSIONS:"
  puts " $argv0 -h"
  puts "  \(The -h command invalidates all others\)"
  puts " $argv0 userlogin yourlogin"
  puts "  \(The Regular Way\)"
  puts "#############################################################################"
  exit
 } elseif { [string match -* $member] == 1 } {
  puts "Usage: $argv0 <-h> \[User Login\] \[Your Login\]"
  exit
 }
}

#########################################################################
# Generic login procs divided by architecture
#########################################################################

proc sun_login {tprompt login lpass} {

 set timeout 3
 set prompt $tprompt
 set qlogin $login
 set qpass $lpass

 expect "ogin: " {send "$qlogin\r"}
 expect "word: " {send "$qpass\r"}
}

proc hp_login {tprompt login lpass} {

 set timeout 12
 set prompt $tprompt
 set qlogin $login
 set qpass $lpass

 expect "ogin: " {send "$qlogin\r"}
        expect "word:" {send "$qpass\r"}
}

proc sco_login {tprompt login lpass} {

 set timeout 6
 set prompt $tprompt
 set qlogin $login
 set qpass $lpass

 expect "ogin: " {send "$qlogin\r"}
        expect "word:" {send "$qpass\r"}
        expect -re $tprompt {send "\r"}
}

#########################################################################
# passwd and mail aliases groper procs by architecture
#########################################################################

proc sunhp_q {tprompt login lpass userlogin host} {
 set timeout 5
 set log $login
 set opass $lpass
 set prompt $tprompt
 set ulogin $userlogin
 set hostname $host
 set insidealias none
 set insidepwd none

 send_user "Scanning $hostname...  "

        expect -re $tprompt {send "grep $ulogin /etc/passwd\r"}
 expect -re $tprompt
        send "echo $?\r"
        expect -re "\r\n(.*)\r\n"
        set returnval $expect_out(1,string)
        if { $returnval == 0 }  {
  set insidepwd 1
        }
        expect -re $tprompt {send "grep $ulogin /etc/mail/aliases\r"}
 expect -re $tprompt
        send "echo $?\r"
        expect -re "\r\n(.*)\r\n"
        set returnval $expect_out(1,string)
        if { $returnval == 0 }  {
  set insidealias 1
        }
        expect -re $tprompt {
  send "exit\r"
 }
 return "$hostname $insidepwd $insidealias"
}
 
proc sco_q {tprompt login lpass userlogin host} {
 set timeout 5
 set log $login
 set opass $lpass
 set prompt $tprompt
 set ulogin $userlogin
 set hostname $host
 set insidealias none
 set insidepwd none

 send_user "Scanning $hostname...  "

 expect -re $tprompt {send "\r"}
        expect -re $tprompt {send "grep $ulogin /etc/passwd\r"}
 expect -re $tprompt
        send "echo $?\r"
        expect -re "\r\n(.*)\r\n"
        set returnval $expect_out(1,string)
        if { $returnval == 0 }  {
  set insidepwd 1
        }
        expect -re $tprompt {send "grep $ulogin /usr/mmdf/table/alias.user\r"}
 expect -re $tprompt
        send "echo $?\r"
        expect -re "\r\n(.*)\r\n"
        set returnval $expect_out(1,string)
        if { $returnval == 0 }  {
  set insidealias 1
        }
       expect -re $tprompt {
  send "exit\r"
 }
 return "$hostname $insidepwd $insidealias"
}

#########################################################################
# The main process loop
#########################################################################

#########################################################################
# Check command line args
#########################################################################

if { [llength $argv] < 2 } {
 puts "Usage: $argv0 <-h> \[User Login\] \[Your Login\]"
 exit
}

#########################################################################
# Set list of servers, passwords and other globals
#########################################################################

set sun [list sun1 sun2]
set sco [list sco1 sco2]
set hp [list hp1 hp2]

set userlogin [lindex $argv 0]
set login [lindex $argv 1]
set tprompt "(%|#|\\\/|\\\$|\\\/|\\\/\\\/#|mgolvach|\\\)|\\\>|\\\}) ?$"

#########################################################################
# Grab user password.
#########################################################################

send_user "Enter your login password : "
stty -echo
expect -re "(.*)\n"
stty echo
set lpass $expect_out(1,string)

send_user "\n"

#########################################################################
# Do It To It.
#########################################################################

foreach host $sun { 
 spawn telnet $host
 sun_login $tprompt $login $lpass
 lappend finalists [sunhp_q $tprompt $login $lpass $userlogin $host]
}

foreach host $hp {
 spawn telnet $host
 hp_login $tprompt $login $lpass
 lappend finalists [sunhp_q $tprompt $login $lpass $userlogin $host]
}

foreach host $sco {
 spawn telnet $host
 sco_login $tprompt $login $lpass
 lappend finalists [sco_q $tprompt $login $lpass $userlogin $host]
}

puts ""
puts "#################################################################"
puts "$userlogin is a valid user on the following machines:"
puts "#################################################################"
set counter 0
foreach item $finalists {
 if { [lindex $item 1] == 1 } {
  incr counter
  puts [lindex $item 0]
 }
}
if { $counter == 0 } {
 puts "No valid accounts found for $userlogin"
}
puts "#################################################################"
puts "$userlogin has mail alias entries on the following machines:"
puts "#################################################################"
set counter 0
foreach item $finalists {
 if { [lindex $item 2] == 1 } {
  incr counter
  puts [lindex $item 0]
 }
}
if { $counter == 0 } {
 puts "No alias file entries found for $userlogin"
}
puts "#################################################################"
puts "$userlogin mail server sql user report:"
puts "#################################################################"
set counter 0
set squealer [open "|isql userdb 2>/dev/null >/tmp/tattle1.tmp" w]
puts $squealer "select * from mail_users where mail_user_id = \"$userlogin\""
close $squealer
set stooly [open "/tmp/tattle1.tmp"]
while {[gets $stooly buffer]!=-1} {
 if { [string match *$userlogin* $buffer] == 1 } {
  puts "Mail Dir : $buffer"
  incr counter
 }  
}
if { $counter == 0 } {
 puts "No Mail Dir entries found for $userlogin"
}
system "rm /tmp/tattle1.tmp"
puts "#################################################################"
puts "$userlogin excess files and directories found on mail server"
puts "#################################################################"
set counter 0
if { [file exists "/var/mail/$userlogin"] == 1 } {
 incr counter
 puts "File: /var/mail/$userlogin"
}
if { [file isdirectory "/var/imsp/user/$userlogin"] == 1 } {
 incr counter
 puts "Directory: /var/imsp/user/$userlogin"
}
if { $counter == 0 } {
 puts "No excess mail server files found for $userlogin"
}
puts "#################################################################"
puts "DON'T FORGET TO:"
puts " 1. Do this".
puts " 2. Do that."
puts " 3. Do everything else."
puts "#################################################################"

exit

, Mike

linux unix internet technology

Easy Multiple File Patching And Patch Removal On Linux And Unix

Hello again,

Today we're going to continue with yesterday's post on manually mass patching files on Linux and Unix, but come at it from a different, and much simpler, angle.

First things first; you can delete the ed patch files from yesterday :) When you look at them, you can see that using "diff -e" to create an ed patch file creates a patch that contains absolutely no information that would allow you to reverse a change that you made, as opposed to a patch file made from diff straight-up. The small comparison below points this up more clearly:

host #diff -e tmpdir1/file1 tmpdir2/file1
5c
BASEDIR="/usr/binky"
.
host # diff tmpdir1/file1 tmpdir2/file1
5c5
< HOMEDIR="/usr/binky"
---
> BASEDIR="/usr/binky"

The ed patch file type that we used yesterday, although simpler to understand, doesn't make it possible to reverse your changes without keeping backup copies of your files.

The good news is that patch files created with diff can still be used to patch massive amounts of files and recover them even if the original, unpatched, files get lost or deleted. The method used to obtain these results is simpler than what we walked through yesterday and only requires the use of the "diff" and "patch" commands. Both of these commands should come standard with your OS. They've been on Linux for a long time and have been on Solaris Unix since, at least, release 2.6 (probably earlier).

So, let's get started patching lots of files and then backing out those patches. Since yesterday's post was so long (and this one has the same potential), I'm going to use only 2 files as the base number of files (although the number of files can be however large you want) and grep out the relevant information for display purposes. Hopefully it will save us all some eye strain ;)

The first thing we'll want to do is copy all the scripts we want to patch into a new directory to work on them. We'll also put the new scripts (that we'll need to create our diff patches) in yet another directory. We'll never work directly on the scripts in their native directory until we're sure they're patched correctly. This isn't absolutely necessary, but is generally good practice. No sense in letting a simple mistake cost you any more time, or grief, than it has to. Again, the only thing that is different between our existing scripts and the new scripts is that the HOMEDIR variable has been changed to BASEDIR.

host # cp scriptdir/* tmpdir1/

<--- This could be any number of files.

The following is our setup, with only one difference between the files, as noted above:

host # ls tmpdir1
.  ..  file1  file2
host # ls tmpdir2
.  ..  file1  file2
host # grep DIR tmpdir1/*
tmpdir1/file1:HOMEDIR="/usr/binky"
tmpdir1/file2:HOMEDIR="/usr/binky"
host # grep DIR tmpdir2/*
tmpdir2/file1:BASEDIR="/usr/binky"
tmpdir2/file2:BASEDIR="/usr/binky"

Next, we'll use diff to create a patch file. Although, rather than just doing diff straight-up, we're going to run it in "contextual" mode. When you invoke "diff -c" it creates a contextual diff which, literally, means that it puts the diff output in context (so you can see the lines before and after the lines that differ). The main reason I like to use this option is that the output works with "patch" when patching multiple files from multiple directories. The output from a standard diff of multiple files in multiple directories doesn't work well for this (mostly because it puts all the file names on one line and "patch" attempts to find a file named "diff tmpdir1/file tmpdir2/file" - literally. And that file can never exist (I hope ;)

Now we'll create the multiple file patch and examine its contents, created at the directory level directly above tmpdir1 and tmpdir2, so we can get all the files in both directories (Note that only the lines beginning with the exclamation point (!) in the diff output are different. The lines above and below only serve to showcase the line in its context within the file):

host # diff -c tmpdir1 tmpdir2  >patchfile.patch
host # cat patchfile.patch
diff -c tmpdir1/file1 vtmpdir2/file1
*** tmpdir1/file1     Wed Mar 26 14:36:05 2008
--- tmpdir2/file1     Wed Mar 26 14:54:10 2008
***************
*** 2,8 ****

  COMMAND_ARGS="-d --takeforeverandaday"

! HOMEDIR="/usr/binky"

  case $x in

--- 2,8 ----

  COMMAND_ARGS="-d --takeforeverandaday"

! BASEDIR="/usr/binky"

  case $x in

diff -c tmpdir1/file2 vtmpdir2/file2
*** tmpdir1/file2     Wed Mar 26 14:36:05 2008
--- tmpdir2/file2     Wed Mar 26 14:54:10 2008
***************
*** 2,8 ****

  COMMAND_ARGS="-d --takeforeverandaday"

! HOMEDIR="/usr/binky"

  case $x in

--- 2,8 ----

  COMMAND_ARGS="-d --takeforeverandaday"

! BASEDIR="/usr/binky"

  case $x in

Now we're ready to patch all of the files in tmpdir1 at once, using the simple form of the command (patch -i PATCHFILE), and receive an error for doing so. Note that we're running this from the directory above tmpdir1 and tmpdir2; exactly where we were when we used "diff -c" to create the patch file:

host # patch -i patchfile.patch
can't find file to patch at input line 4
Perhaps you should have used the -p or --strip option?
The text leading up to this was:
--------------------------
|diff -c tmpdir1/file1 tmpdir2/file1
|*** tmpdir1/file1    Wed Mar 26 14:36:05 2008
|--- tmpdir2/file1    Wed Mar 26 14:54:10 2008
--------------------------
File to patch: ^C 

<--- Type the control (ctl) key + C, or any other escape/control key combination to break out of this prompt

This error killed us before it actually made any changes because of the way "patch" works. When run without any options (other than -i, which we used to indicate the name of our patch file), "patch" parses the patch file and strips the file names down to the base, in much the same way the "basename" command does. So, even though the file name in the patch file is "tmpdir/file1," the "patch" program is looking for a file named "file1" and it's looking for it in the directory we're in which, unfortunately, isn't where the file is.

Luckily, this little setback is easy to remedy. Using the -p option to "patch" we can instruct "patch" how to interpret the file names in our patch file. As we noted, when the option isn't present, "patch" reverts to "basename" type behaviour. If we used -p1, we would be instructing "patch" to remove the leading slash (/) from the file name. We're going to use -p0, which instructs "patch" to not interpret the file name and just take it as it is (In this case "tmpdir/file1," which is relative, but just fine considering where we're running the command from).

host # patch -p0 -i patchfile.patch
patching file tmpdir1/file1
patching file tmpdir1/file2

Success! Now, let's check that the patch actually took:

host # grep DIR tmpdir1/*
tmpdir1/file1:BASEDIR="/usr/binky"
tmpdir1/file2:BASEDIR="/usr/binky"

Excellent! HOMEDIR is now BASEDIR. Alas, as I intimated in our post yesterday on mass file updating, our boss has, only minutes later, decided that the BASEDIR variable really should be HOMEDIR after all. He's not going to explain why, we just need to switch everything back now ;)

And this is where our method of execution really pays off. Assuming we held on to that patch file, we can now use "patch" to put everything back the way it was in short order, by simply adding the -R (to reverse the patch operations) to the command line and running it again, like so:

host # patch -p0 -R -i patchfile.patch
patching file tmpdir1/file1
patching file tmpdir1/file2

And, then, just to verify that the patch differences have been removed:

host # grep DIR tmpdir1/*
tmpdir1/file1:HOMEDIR="/usr/binky"
tmpdir1/file2:HOMEDIR="/usr/binky"

And we're all set :) Hopefully this follow-up tutorial was easy enough to follow, and you can find some good use for it in your work routine. BTW, don't forget to copy the changed scripts back to the real script directory, but only after copying that directory off somewhere else, again. But only if you're as paranoid as I am ;)

Best Wishes,

, Mike

linux unix internet technology

Manual Mass File Patching On Linux Or Unix

Good Day Good People :)

In previous posts we've looked at various aspects of the diff command, including augmenting it to generate unique content only and work with users, groups and permissions.

Today, I thought we'd look at diff's usage in a, generally, manual process and peek at the first step toward automation. Everything in this post can be scripted out to provide more granular control, and done much more easily. I thought doing it in a laborious tutorial style to start out with would be best to get the major parts of the process laid out simply and, hopefully, written in such a way that they're simply understood. After reading that last sentence, I'm wishing myself lots of luck ;)

For purposes of our example today, we're going to assume that we have a directory in which we keep 3 executable files. All of these files are approximately the same, because they all do approximately the same thing, just for different programs. For simplicity, they've been named: file1, file2 and file3, as shown in this directory listing:

host # ls
.  ..  file1  file2  file3

All of their contents are also almost totally identical:

host # cat file1 file2 file3
#!/bin/bash
# Start File1 Process

COMMAND_ARGS="-d --takeforeverandaday"

HOMEDIR="/usr/binky"

#!/bin/bash
# Start File2 Process

COMMAND_ARGS="-d --takeforeverandaday"

HOMEDIR="/usr/binky"

#!/bin/bash
# Start File3 Process

COMMAND_ARGS="-d --takeforeverandaday"

HOMEDIR="/usr/binky"

For today (and this would work if we had 5000 files, but this post is going to be long enough ;) all we need to do is change the HOMEDIR standard variable in all of our scripts to BASEDIR. I'm not exactly sure why. Someone with more authority than me or you decided it would be a good idea. They will probably change their mind and have us switch it back next week ;)

The first thing we should do is create a backup directory and do our work there.

host # mkdir ../u
host # cp file* ../u
host # cd ../u
host # ls
.  ..  file1  file2  file3

Then we'll just use a simple while loop and feed each file to sed and make the substitution, which we'll dump into another file called FILENAME.diff (So, file1 will get the substitution made on it by sed, and that difference will be put into a new file called file1.diff). Within that same loop, we'll be using the actual command "diff -e" to create an "ed" (A very basic Unix and Linux editor) patch file. Note that, for file1, this new file will be called file1.patch and that we echo a "w" and a "q" (on separate lines on purpose) into the bottom of the patch file.

host # ls -1d *|while read x;do sed s'/HOMEDIR=\"\(\/usr\/binky\)\"/BASEDIR=\"\1\"/' $x >$x.diff;diff -e $x $x.diff >$x.patch;echo w >>$x.patch;echo q >>$x.patch;done

We could have deleted our file*.diff files above, as well, but I thought you might like to take a look at them. These serve as proof that our sed command actually worked :) HOMEDIR is now BASEDIR!

host # cat file1.diff file2.diff file3.diff
#!/bin/bash
# Start File1 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

#!/bin/bash
# Start File2 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

#!/bin/bash
# Start File3 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

Now, let's delete the *.diff files and take a look at the patch files. These are all considered "ed" scripts. This means that we can feed them to the "ed" Linux/Unix editor and have it execute commands for us automatically. This will come in handy in a second :)

host # rm *.diff
host # cat file1.patch file2.patch file3.patch
5c                    <--- These first three
BASEDIR="/usr/binky"  <--- lines were created
.                     <--- by "diff -e"
w                     <--- We manually added
q                     <--- these last two!
5c
BASEDIR="/usr/binky"
.
w
q
5c
BASEDIR="/usr/binky"
.
w
q

The next step in the process is to apply those patches. We're going to be doing this all in our test directory, so there's no need to tar up all the patch files for a move to our "live" script directory

host # ls
.   file1        file2        file3
..  file1.patch  file2.patch  file3.patch

So, now we're ready to test our mass change and pray all goes well. Note that, since we're still in our backup directory, we can do lots of damage to the scripts in here and it shouldn't make a difference to anyone. Only we will have to live with the shame ;)

Now, we'll do another fancy command line and pipe the output of "ls -1d *", grepping out all the patch files and then using "ed" to patch each file in the directory. Notice that the syntax for patching a file with an "ed" diff script is "ed FILENAME <FILENAME.patch" - This would normally hang and wait for us to send it a control-D or some other signal, but, since we echoed the "w" and "q" characters into the patch files, "ed" knows to write (w) and quit (q) - At the end of the command pipe, we'll purposefully not delete all the patch files. You can remove them if you like, but we'll be looking at using them to undo changes in a future post!

host # ls -1d *|grep -v patch|while read x;do ed $x <$x.patch;done
107
107
107
107
107
107
host # ls
.   file1        file2        file3
..  file1.patch  file2.patch  file3.patch

Okay, we're all done and now we can check the results. Looks like everything in our (potentially huge amount of) files got patched appropriately :) This shouldn't be a surprise since we checked our sed output already, but you never know.

host # cat file1 file2 file3

#!/bin/bash
# Start File1 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

#!/bin/bash
# Start File2 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

#!/bin/bash
# Start File3 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

In a future post we'll look at a script that will allow us to do what we've just done today and make life a little easier when doing mass updates or patching :)

Cheers,

, Mike

linux unix internet technology

Using Color In The Linux Or Unix Shell

Click the image above to see all the pretty colors bigger ;)

Hey There,

Today's we're going to go off on a tangent and take a look at color and the Linux and/or Unix console. I first noticed the use of color in the shell while using Linux a long long time ago, and it seemed really cool back then. Every now and again it bugs me, but I must admit that it does come in handy for a lot of things; editing code or simple file type identification are two great examples.

The script attached to this post was written mostly to illustrate the concept that you can see in the picture attached to this text. More to the point, I wrote it so that we could see the colors I'm writing about :) Admittedly, I used every possible combination of color and modification for completeness, which has the nasty side effect of making some of the output invisible (You can't see the black foreground on the black background unless reverse video or bolding are turned on), but that's the nature of the beast. You probably won't ever need or want to print black on black, unless you're writing a menuing system for someone you can't stand ;)

One thing to note is that the colors and modifications in this script are limited to the ECMA-compliant codes. Linux and the Bash shell, for example, have a lot more color options, but I wanted to keep this post as fairly balanced as possible. These codes should work on almost any Unix or Linux console and most SSH or Telnet shell clients today probably support them as well. Still, not everything will work everywhere. For instance, the blinking text code does not work on my PuTTY terminal, although it does work on my SecureCRT. Since I'm not paying for the color, I guess I can't complain ;)

Getting these colors to show on your terminal is actually fairly simple. First, you just need to know what all the codes stand for. Below is a quick listing. The 30's are foregrounds, the 40's are backgrounds and the remaining set are modifications (any and all of which can be combined, if you so choose):

# 30 black foreground
# 31 red foreground
# 32 green foreground
# 33 brown foreground
# 34 blue foreground
# 35 purple foreground
# 36 light blue foreground
# 37 gray foreground
#
# 40 black background
# 41 red background
# 42 green background
# 43 brown background
# 44 blue background
# 45 purple background
# 46 light blue background
# 47 white background
#
# 1 turn on bolding
# 22 turn off bolding
# 5 turn on blinking
# 25 turn off blinking
# 7 turn on reverse video
# 27 turn off reverse video
# 0 reset everything to default

Now, if you wanted to print a color to your screen, all you need to do is use the echo command (Note that -e tells echo to interpret the backslashed characters, so "\033[30m" doesn't show up literally). For instance, to print "HI" in red font, you could type:

host # echo -e "\033[31mHI"

The "m" character following the number code 31 (Indicating that you want to print in red) completes the instantiation of the color. Without it, your text would remain whatever color it already was. The incorrect echo statement may also have other bizarre side effects, like repositioning your cursor or resetting your terminal. The final "m" is very important :)

You can also combine different colors, as well as modifications, by using the semicolon. Below, the first line writes "HI" in light blue on a green background (Uggh) and the second does the same in BOLD :)

host # echo -e "\033[36;42mHI"
host # echo -e "\033[36;42;1mHI"

One thing to note is that all of these commands would leave your terminal stuck in whatever state you instructed on the command line. That is, if you used the color commands to type with a green background, your terminal's background would remain green after you typed the line. In order to return everything to normal, you can use the modification number "0" - This will reset your terminal to the default color settings. If you only want to print one line in red font, the line shown below would work better for you than the original (the first example, above)

host # echo -e "\033[31mHI\033[0m"

Note, also, that (for the color code commands) there can't be any spaces. "\033[31m" has to be one unit. The text you put in between this command and the closing "\033[0m" (should you elect to go back to the way things were ;) can have spaces anywhere, just like any other block of text.

Hope you have some fun with shell colors. Enjoy the script. If nothing else, it should serve as a good reminder of the syntax and a good indicator of the breadth of options ECMA compliant colors give you. Linux, as I mentioned, has even more variety, but using those specific codes can make your scripts less portable. Good or bad? You decide ;)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

# colors.sh - Print out all the different ECMA color-mod combos
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

declare -a color_array
declare -a alphabet_soup

color_array=( '30;40;1' '30;40;5' '30;40;7' '30;40;22' '30;40;25' '30;40;27' '30;40' '30;41;1' '30;41;5' '30;41;7' '30;41;22' '30;41;25' '30;41;27' '30;41' '30;42;1' '30;42;5' '30;42;7' '30;42;22' '30;42;25' '30;42;27' '30;42' '30;43;1' '30;43;5' '30;43;7' '30;43;22' '30;43;25' '30;43;27' '30;43' '30;44;1' '30;44;5' '30;44;7' '30;44;22' '30;44;25' '30;44;27' '30;44' '30;45;1' '30;45;5' '30;45;7' '30;45;22' '30;45;25' '30;45;27' '30;45' '30;46;1' '30;46;5' '30;46;7' '30;46;22' '30;46;25' '30;46;27' '30;46' '30;47;1' '30;47;5' '30;47;7' '30;47;22' '30;47;25' '30;47;27' '30;47' '30;1' '30;5' '30;7' '30;22' '30;25' '30;27' '30' '31;40;1' '31;40;5' '31;40;7' '31;40;22' '31;40;25' '31;40;27' '31;40' '31;41;1' '31;41;5' '31;41;7' '31;41;22' '31;41;25' '31;41;27' '31;41' '31;42;1' '31;42;5' '31;42;7' '31;42;22' '31;42;25' '31;42;27' '31;42' '31;43;1' '31;43;5' '31;43;7' '31;43;22' '31;43;25' '31;43;27' '31;43' '31;44;1' '31;44;5' '31;44;7' '31;44;22' '31;44;25' '31;44;27' '31;44' '31;45;1' '31;45;5' '31;45;7' '31;45;22' '31;45;25' '31;45;27' '31;45' '31;46;1' '31;46;5' '31;46;7' '31;46;22' '31;46;25' '31;46;27' '31;46' '31;47;1' '31;47;5' '31;47;7' '31;47;22' '31;47;25' '31;47;27' '31;47' '31;1' '31;5' '31;7' '31;22' '31;25' '31;27' '31' '32;40;1' '32;40;5' '32;40;7' '32;40;22' '32;40;25' '32;40;27' '32;40' '32;41;1' '32;41;5' '32;41;7' '32;41;22' '32;41;25' '32;41;27' '32;41' '32;42;1' '32;42;5' '32;42;7' '32;42;22' '32;42;25' '32;42;27' '32;42' '32;43;1' '32;43;5' '32;43;7' '32;43;22' '32;43;25' '32;43;27' '32;43' '32;44;1' '32;44;5' '32;44;7' '32;44;22' '32;44;25' '32;44;27' '32;44' '32;45;1' '32;45;5' '32;45;7' '32;45;22' '32;45;25' '32;45;27' '32;45' '32;46;1' '32;46;5' '32;46;7' '32;46;22' '32;46;25' '32;46;27' '32;46' '32;47;1' '32;47;5' '32;47;7' '32;47;22' '32;47;25' '32;47;27' '32;47' '32;1' '32;5' '32;7' '32;22' '32;25' '32;27' '32' '33;40;1' '33;40;5' '33;40;7' '33;40;22' '33;40;25' '33;40;27' '33;40' '33;41;1' '33;41;5' '33;41;7' '33;41;22' '33;41;25' '33;41;27' '33;41' '33;42;1' '33;42;5' '33;42;7' '33;42;22' '33;42;25' '33;42;27' '33;42' '33;43;1' '33;43;5' '33;43;7' '33;43;22' '33;43;25' '33;43;27' '33;43' '33;44;1' '33;44;5' '33;44;7' '33;44;22' '33;44;25' '33;44;27' '33;44' '33;45;1' '33;45;5' '33;45;7' '33;45;22' '33;45;25' '33;45;27' '33;45' '33;46;1' '33;46;5' '33;46;7' '33;46;22' '33;46;25' '33;46;27' '33;46' '33;47;1' '33;47;5' '33;47;7' '33;47;22' '33;47;25' '33;47;27' '33;47' '33;1' '33;5' '33;7' '33;22' '33;25' '33;27' '33' '34;40;1' '34;40;5' '34;40;7' '34;40;22' '34;40;25' '34;40;27' '34;40' '34;41;1' '34;41;5' '34;41;7' '34;41;22' '34;41;25' '34;41;27' '34;41' '34;42;1' '34;42;5' '34;42;7' '34;42;22' '34;42;25' '34;42;27' '34;42' '34;43;1' '34;43;5' '34;43;7' '34;43;22' '34;43;25' '34;43;27' '34;43' '34;44;1' '34;44;5' '34;44;7' '34;44;22' '34;44;25' '34;44;27' '34;44' '34;45;1' '34;45;5' '34;45;7' '34;45;22' '34;45;25' '34;45;27' '34;45' '34;46;1' '34;46;5' '34;46;7' '34;46;22' '34;46;25' '34;46;27' '34;46' '34;47;1' '34;47;5' '34;47;7' '34;47;22' '34;47;25' '34;47;27' '34;47' '34;1' '34;5' '34;7' '34;22' '34;25' '34;27' '34' '35;40;1' '35;40;5' '35;40;7' '35;40;22' '35;40;25' '35;40;27' '35;40' '35;41;1' '35;41;5' '35;41;7' '35;41;22' '35;41;25' '35;41;27' '35;41' '35;42;1' '35;42;5' '35;42;7' '35;42;22' '35;42;25' '35;42;27' '35;42' '35;43;1' '35;43;5' '35;43;7' '35;43;22' '35;43;25' '35;43;27' '35;43' '35;44;1' '35;44;5' '35;44;7' '35;44;22' '35;44;25' '35;44;27' '35;44' '35;45;1' '35;45;5' '35;45;7' '35;45;22' '35;45;25' '35;45;27' '35;45' '35;46;1' '35;46;5' '35;46;7' '35;46;22' '35;46;25' '35;46;27' '35;46' '35;47;1' '35;47;5' '35;47;7' '35;47;22' '35;47;25' '35;47;27' '35;47' '35;1' '35;5' '35;7' '35;22' '35;25' '35;27' '35' '36;40;1' '36;40;5' '36;40;7' '36;40;22' '36;40;25' '36;40;27' '36;40' '36;41;1' '36;41;5' '36;41;7' '36;41;22' '36;41;25' '36;41;27' '36;41' '36;42;1' '36;42;5' '36;42;7' '36;42;22' '36;42;25' '36;42;27' '36;42' '36;43;1' '36;43;5' '36;43;7' '36;43;22' '36;43;25' '36;43;27' '36;43' '36;44;1' '36;44;5' '36;44;7' '36;44;22' '36;44;25' '36;44;27' '36;44' '36;45;1' '36;45;5' '36;45;7' '36;45;22' '36;45;25' '36;45;27' '36;45' '36;46;1' '36;46;5' '36;46;7' '36;46;22' '36;46;25' '36;46;27' '36;46' '36;47;1' '36;47;5' '36;47;7' '36;47;22' '36;47;25' '36;47;27' '36;47' '36;1' '36;5' '36;7' '36;22' '36;25' '36;27' '36' '37;40;1' '37;40;5' '37;40;7' '37;40;22' '37;40;25' '37;40;27' '37;40' '37;41;1' '37;41;5' '37;41;7' '37;41;22' '37;41;25' '37;41;27' '37;41' '37;42;1' '37;42;5' '37;42;7' '37;42;22' '37;42;25' '37;42;27' '37;42' '37;43;1' '37;43;5' '37;43;7' '37;43;22' '37;43;25' '37;43;27' '37;43' '37;44;1' '37;44;5' '37;44;7' '37;44;22' '37;44;25' '37;44;27' '37;44' '37;45;1' '37;45;5' '37;45;7' '37;45;22' '37;45;25' '37;45;27' '37;45' '37;46;1' '37;46;5' '37;46;7' '37;46;22' '37;46;25' '37;46;27' '37;46' '37;47;1' '37;47;5' '37;47;7' '37;47;22' '37;47;25' '37;47;27' '37;47' '37;1' '37;5' '37;7' '37;22' '37;25' '37;27' '37' '40;1' '40;5' '40;7' '40;22' '40;25' '40;27' '40' '41;1' '41;5' '41;7' '41;22' '41;25' '41;27' '41' '42;1' '42;5' '42;7' '42;22' '42;25' '42;27' '42' '43;1' '43;5' '43;7' '43;22' '43;25' '43;27' '43' '44;1' '44;5' '44;7' '44;22' '44;25' '44;27' '44' '45;1' '45;5' '45;7' '45;22' '45;25' '45;27' '45' '46;1' '46;5' '46;7' '46;22' '46;25' '46;27' '46' '47;1' '47;5' '47;7' '47;22' '47;25' '47;27' '47' '1' '5' '7' '22' '25' '27' )

alphabet_soup=( ABC DEF GHI JKL MNO PQR STU VWX YYZ )

color_array_count=${#color_array[@]}
alphabet_soup_count=${#alphabet_soup[@]}

echo "Number of color code combinations in array = $color_array_count"

alpha_count=0
for ((x=0; x<$color_array_count; x++))
do
 if [ $alpha_count -gt 9 ]
 then
  alpha_count=0
 fi 
 echo -ne "\033[${color_array[${x}]}m${alphabet_soup[${alpha_count}]}\033[0m"
 ((alpha_count++))
done
echo

, Mike

linux unix internet technology

Creating Linux RPM's From Already Installed Packages

Click on the image for a higher resolution version of the output shown above!

Hey there,

Today's post is a follow up to, or the complement of, a post we did a little while ago on creating Solaris pkg files from already installed packages. Yes, there was a reason today's title sounded familiar ;)

Actually, I've been meaning to follow up on the Linux counterpart of this issue for a week or so now, but got caught up trying out various things. For instance, one of the more unusual (and interesting things) I ran across was a Perl project called alien that endeavours to pretty much translate between almost all package formats (dpkg, rpm, pkg, etc), but it didn't really address this specific issue. Still, pretty neat and in the experimental stage.

I also followed a lot of trial and error using rpmrebuild and standard rpm with the --repackage option, but found that both of these methods required the additional execution of at least one thing I didn't need to do to achieve my goal. For instance, using rpm with the --repackage option requires that you be upgrading, re-installing, deleting, etc, the RPM in question. All I want to do is create a package from existing source, and I'd rather not have to upgrade, re-install or delete the software on a live system to do it :)

Luckily, in the end, it boiled down to a fairly simple process. A lot of confusion came from various interpretations of the rpmbuild "spec file" and how, and why, it should be used. Many of the ideas I got that led me to an eventual solution came off of message boards rather than official sources. To put it somewhat humorously, a lot of those guys may not know what they're talking about, but they sure do know what they're doing ;) Seriously, if it weren't for the folks who strayed from the info-docs, this solution might have taken even longer to come around to.

You can call today's command (which I've name "rpm_remk" to remind me of my similar Solaris pkg project), very simply from the command line, like so:

host # ./rpm_remk mutt <--- If, for example, you wanted to create an RPM of the installed version of "mutt" on your system.

You can be as specific or general as RPM will let you when you use this command. For instance, on my box, I only have one instance of mutt:

host # rpm -qa|grep mutt
mutt-1.5.6i-64.9

So that simple command line suffices. For another program like, say, xaw3d, I would have to be more specific, because the underlying RPM system wouldn't know which package I was referring to. For example:

host # rpm -qa|grep xaw3d
xaw3d-1.5E-216.3
xaw3d-32bit-9-200407011229

Shows two different selections. RPM generally only picks the first one, like this:

host # rpm -q xaw3d
xaw3d-1.5E-216.3

So, if we ran the command with only "xaw3d" as the argument, we'd create that 1.5E RPM, which would be great unless we wanted the 32bit version ;) Specifying "xaw3d-32bit-9-200407011229" as the argument would guarantee that the intended RPM got built!

Here's hoping this script helps you out and makes it a little easier for you to retrieve some of those old installed software packages for which the original RPM's have long since disappeared :)





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

# rpm_remk - Create new RPM's from installed RPM's
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

trap 'rm -rf $temp_var;exit 1' 1 2 3 9 15
if [ $# -ne 1 ]
then
        echo "Usage: $0 PackageName"
        exit 1
fi

existing_rpm=$1
starting_point=`pwd`
temp_var=$$

mkdir ${starting_point}/RPMS
mkdir ${starting_point}/RPMS/noarch
mkdir ${starting_point}/RPMS/`uname -i`
mkdir $temp_var
cd $temp_var

if [ `expr match \`rpm -q --queryformat "%{INSTALLPREFIX}\n" $existing_rpm\` '(none)'` -ne 0 ]
then
        Prefix="/"
else
        Prefix=`rpm -q --queryformat "Prefix: %{INSTALLPREFIX}\n" $existing_rpm`
fi

echo "%define _topdir $starting_point" >>$existing_rpm.spec
rpm -q --queryformat "Summary: %{SUMMARY}\n" $existing_rpm >>$existing_rpm.spec
rpm -q --queryformat "Name: %{NAME}\n" $existing_rpm >>$existing_rpm.spec
rpm -q --queryformat "Version: %{VERSION}\n" $existing_rpm >>$existing_rpm.spec
rpm -q --queryformat "Release: %{RELEASE}\n" $existing_rpm >>$existing_rpm.spec
rpm -q --queryformat "Copyright: %{COPYRIGHT}\n" $existing_rpm >>$existing_rpm.spec
rpm -q --queryformat "Group: %{GROUP}\n" $existing_rpm >>$existing_rpm.spec
echo "Prefix: $Prefix" >>$existing_rpm.spec
description=`rpm -q --queryformat "%{DESCRIPTION}\n" $existing_rpm`
echo "%description" >>$existing_rpm.spec
echo "$description" >>$existing_rpm.spec
echo "%files" >>$existing_rpm.spec
rpm -ql $existing_rpm|while read x
do
        echo "%{prefix}$x" >>$existing_rpm.spec
done

rpmbuild -bb $existing_rpm.spec 2>&1|grep -v twice
cd $starting_point
mv -f ${starting_point}/RPMS/*/* .
rm -rf $temp_var ${starting_point}/RPMS

, Mike

linux unix internet technology

Doing Network Wide Updates Using Expect

Hey there,

It's been a while since we took a look at Expect on this blog, so I'm throwing a monster out there today ;)

Today's script is a bit long-in-the-tooth, and may be complicated looking, but it's basically a simple script to enable you to do updates across your network all at once; only having to login once (or twice, but all at the same time ;)

I put a small section in the beginning that checks whether or not a user invoking the script is allowed to use it (although this script doesn't allow anyone to do anything that they don't have the privilege or access and knowledge to do already). It's a simple check against an exec of the logname command, to make sure that the user's logname is among the list of lognames allowed to use the script. This whole section can be removed without affecting the script's functionality. Simply delete the section with the header: "Weed out the undesirables pronto" and it'll no longer be an issue.

I also added separate login functions for Solaris, Linux, HP-UX and SCO Unix for variety (Note that you may need to tweak these Expect routines, in case the regular expression matching doesn't suit your flavor). You may not need all of these functions, and you may only need one. In any case, you can either call the script specifically to use it the way in which you desire, or edit it so that it will only run the way you desire. If you call it with the "-h" flag it should give you more help than you could ever possibly want ;)

Ex:
host # ./update.exp -h

Hope you enjoy this and find it useful. Good Times! :)





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/local/bin/expect

#########################################################################
# update.exp - do mass updates across network
# without lifing a finger
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#
# Usage: update.exp <-h> <-d> 
# [ +host1 +hostn | (+sco &| +sun &| +hp &| +linux) | +all ]
# [ login ] [ passwd | <command> <args> ]
#########################################################################

#########################################################################
# Weed out the undesirables pronto
#########################################################################

set gauntlet [exec logname]
if { [string compare $gauntlet "user1"] != 0 && [string compare $gauntlet "user2"] != 0 && [string compare $gauntlet "user3"] != 0 }   {
 puts "Sorry, $gauntlet. You may not use update.exp."
 exit
}           

#########################################################################
# Go for the help short-circuit
#########################################################################

foreach member $argv {
 if { $member == "-h" } {
  puts "#############################################################################"
  puts "USAGE:"
  puts " update.exp <-h> <-d>"
  puts " \[ +host1 +hostn | (+sco &| +sun &| +hp &| +linux) | +all \]"
  puts " \[ login \] \[ passwd | <command> <args> \]"
  puts "#############################################################################"
  puts "SCO Hosts:"
  puts " sco1 sco2 sco3"
  puts "SUN Hosts:"
  puts " sun1 sun2 sun3"
  puts "HP Hosts:"
  puts " hp1 hp2 hp3"
  puts "LINUX Hosts:"
  puts " linux1 linux2 linux3"
  puts "#############################################################################"
  puts "FLAGS:"
  puts " -h: Show this message."
  puts " -d: Force extremely verbose output during operation."
  puts " +host\[1-n\]: The plus symbol followed by any vaild hostname."
  puts " +sco: Process all sco hosts."
  puts " +sun: Process all sun hosts."
  puts " +hp: Process all hp hosts."
  puts " +linux: Process all linux hosts."
  puts " +all: Process all hosts."
  puts " login: Your own."
  puts " passwd: Change password on specified machines."
  puts " command: Any architecture-supported command, with optional arguments."
  puts "#############################################################################"
  puts "A FEW VALID COMMAND LINE EXPRESSIONS:"
  puts " update.exp -h"
  puts "  \(The -h command invalidates all others\)"
  puts " update.exp -d *****"
  puts "  \(The -d command may be used in conjunction with any other\)"
  puts " update.exp +host1 +host2 login command args"
  puts "  \(Multiple hosts may be named on command line\)"
  puts " update.exp +host1 +sun login command args"
  puts "  \(Specific host may be coupled with groups\)"
  puts " update.exp +sun +hp login command"
  puts "  \(Groups may be combined.  Args to command are trivial\)"
  puts " update.exp +all login passwd"
  puts "  \(+all obviates any other hosts named, but may be combined for"
  puts "  a specific purpose\)"
  puts "#############################################################################"
  exit
 }
}

#########################################################################
# Hide the gory details unless they want blood
#########################################################################

if { [llength $argv] == 0 } {
 set debug 0
} else {
 foreach entrail $argv {
  if { $entrail == "-d" } {
   set debug 1
   break
  } else {
   set debug 0
  }
 }
}

if { $debug } {
 log_user 1
} else {
 log_user 0
}

#########################################################################
# Generic login procs divided by architecture
#########################################################################

proc sun_login {name tprompt login lpass} {
 send_user "$name\n"

 set timeout 3
 set hostname $name
 set prompt $tprompt
 set qlogin $login
 set qpass $lpass

 expect "ogin: " {send "$qlogin\r"}
 expect "word: " {send "$qpass\r"}
}

proc linux_login {name tprompt login lpass} {
 send_user "$name\n"

 set timeout 3
 set hostname $name
 set prompt $tprompt
 set qlogin $login
 set qpass $lpass

 expect "ogin: " {send "$qlogin\r"}
 expect "word: " {send "$qpass\r"}
}

proc hp_login {name tprompt login lpass} {
 send_user "$name\n"

 set timeout 12
 set hostname $name
 set prompt $tprompt
 set qlogin $login
 set qpass $lpass

 expect "ogin: " {send "$qlogin\r"}
        expect "word:" {send "$qpass\r"}
}

proc sco_login {name tprompt login lpass} {
 send_user "$name\n"

 set timeout 6
 set hostname $name
 set prompt $tprompt
 set qlogin $login
 set qpass $lpass

 expect "ogin: " {send "$qlogin\r"}
        expect "word:" {send "$qpass\r"}
        expect -re $tprompt {send "\r"}
}

#########################################################################
# Command and user passwd procs by architecture
#########################################################################

proc sunhp_q {command arglist tprompt debug login lpass} {
 set timeout 5
 set log $login
 set opass $lpass
 set argvec $arglist
 set commvec $command
 set prompt $tprompt
 set db $debug

 if { $command == "passwd" } {
  set username $log
  set oldpass $opass
  set newpass $argvec

  expect -re $prompt {send "$commvec $username\r"}
  expect -re "word: ?$" {send "$oldpass\r"}
  expect -re "New password: ?$" {send "$newpass\r"}
  expect {
   -re "New password: ?$" {
    send "\r"
    send_user "Password not accepted!\n"
    expect -re "New password: ?$" {send "\r"}
    expect -re "New password: ?$" {send "\r"}
   }
   -re "word: ?$" {send "$newpass\r"}
  }
  expect -re $prompt {send "exit\r"}
 } else {
  if { $argvec != 0 } {
          expect -re $tprompt {send "$commvec $argvec\r"}
  } else {
          expect -re $tprompt {send "$commvec\r"}
  }
  log_user 1
         expect -re $tprompt {
   if { $db } {
    log_user 1
   } else {
    log_user 0
   }
   send "exit\r"
  }
 }
}
 
proc linux_q {command arglist tprompt debug login lpass} {
 set timeout 5
 set log $login
 set opass $lpass
 set argvec $arglist
 set commvec $command
 set prompt $tprompt
 set db $debug

 if { $command == "passwd" }     {
  set username $log
  set oldpass $opass
  set newpass $argvec
 
  expect -re $prompt {send "$commvec\r"}
  expect -re "word: ?$" {send "$oldpass\r"}
  expect -re "assword: ?$" {send "$newpass\r"}
  expect  {
   -re "BAD.*$" {
   send "\r"
   send_user "Password not accepted!\n"
   expect -re "BAD.*$" {send "\r"}
   expect -re "BAD.*$" {send "\r"}
   }
   -re "word: ?$" {send "$newpass\r"}
  }
  expect -re $prompt {send "exit\r"}
 }       else    {
  if { $argvec != 0 }     {
   expect -re $tprompt {send "$commvec $argvec\r"}
  }       else    {
   expect -re $tprompt {send "$commvec\r"}
  }
  log_user 1
  expect -re $tprompt {
   if { $db }      {
    log_user 1
   }       else    {
    log_user 0
   }
   send "exit\r"
  }
 }
}      

proc sco_q {command arglist tprompt debug login lpass} {
 set timeout 5
 set log $login
 set opass $lpass
 set argvec $arglist
 set commvec $command
 set prompt $tprompt
 set db $debug

 if { $db } {
  log_user 1
 } else {
  log_user 0
 }
 if { $command == "passwd" } {
  set username $log
  set oldpass $opass
  set newpass $argvec

  expect -re $prompt {send "$commvec $username\r"}
  expect -re "word: ?$" {send "$oldpass\r"}
  expect -re "1\\\): ?$" {send "\r"}
  expect -re "New password: ?$" {send "$newpass\r"}
  expect {
   -re "New password: ?$" {
    send "$newpass\r"
    expect -re "New password: ?$" {send "$newpass\r"}
    send_user "Password not accepted!\n"
   }
   -re "word: ?$" {send "$newpass\r"}
  }
  expect -re $prompt {send "exit\r"}
 } else {
         expect -re $tprompt {send "\r"}
         expect -re $tprompt {
   if { $argvec != 0 } {
    send "$commvec $argvec\r"
   } else {
           send "$commvec\r"
   }
   log_user 1
  }
         expect -re $tprompt {
   if { $db } {
    log_user 1
   } else {
    log_user 0
   }
   send "exit\r"
  }
 }
}

#########################################################################
# The main process loop
#########################################################################

#########################################################################
# Check command line args
#########################################################################

if { [llength $argv] < 3 } {
 puts "Usage: $argv0 <-h> <-d> \\ "
 puts "\t\[+host1 +hostn | \(+sun &| +sco &| +hp &| +linux\) | +all\] \\ "
 puts "\t\[login\] \[rpasswd | passwd | <command> <args>\]"
 exit
}

set gotserv 0
set gotlog 0

foreach name $argv {
 if { [string match "+*" $name ] } {
  set gotserv 1
 } elseif { [string match "+*" $name ] && $gotserv } {
  continue
 } elseif { $gotserv } {
  set gotlog 1
 }
}

if { [expr $gotlog+$gotserv] != 2 } {
 puts "Usage: $argv0 <-h> <-d> \\ "
 puts "\t\[+host1 +hostn | \(+sun &| +sco &| +hp &| +linux\) | +all\] \\ "
 puts "\t\[login\] \[rpasswd | passwd| <command> <args>\]"
 exit
}

#########################################################################
# Set list of servers, passwords and other globals
#########################################################################

set sun [list sun1 sun2 sun3]
set sco [list sco1 sco2 sco3]
set hp [list hp1 hp2 hp3]
set linux [list linux1 linux2 linux3]

list servers
list arglist
set allservers 0
set sunservers 0
set linuxservers 0
set scoservers 0
set hpservers 0
set needargs 0
set gotargs 0
set verify 0
set tprompt "(%|#|\\\/|\\\$|\\\/|\\\/\\\/#|user1|user2|user3|\\\)|\\\>|\\\}) ?$"

#########################################################################
# Parse arg vector and delegate procs
#########################################################################

foreach name $argv {
 if { [string match "+*" $name] == 1 } {
  if { [string match "+all" $name] == 1 } {
   set allservers 1
  } elseif { [string match "+sun" $name] == 1 } {
   set sunservers 1
  } elseif { [string match "+linux" $name] == 1 } {
   set linuxservers 1
  } elseif { [string match "+sco" $name] == 1 } {
   set scoservers 1
  } elseif { [string match "+hp" $name] == 1 } {
   set hpservers 1
  } else {
   lappend servers [string range $name 1 [expr [string length $name]-1]]
  }
 } else {
  if { $needargs == 1 } {
   lappend arglist $name
   set gotargs 1
  } elseif { $verify == 1 } {
   set command $name
   set needargs 1
  } elseif { $name == "-d" } {
   continue
  } else {
   set login $name
   incr verify
  }
 }
}

set ssshservers [expr $sunservers + $linuxservers + $scoservers + $hpservers]

if { $gotargs == 0 } {
 set arglist 0
}

#########################################################################
# Grab user password and new password if necessary.
#########################################################################

send_user "Enter your login password : "
stty -echo
expect -re "(.*)\n"
stty echo
set lpass $expect_out(1,string)

send_user "\n"

if { $command == "passwd" } {
 send_user "Enter your new password : "
 stty -echo
 expect -re "(.*)\n"
 stty echo
 set arglist $expect_out(1,string)
}

send_user "\n"

#########################################################################
# Loop until all conditions are satisfied
#########################################################################

for {} {1} {} {
 if {$allservers} {
  foreach host $sun {
   spawn telnet $host
   sun_login $host $tprompt $login $lpass
   sunhp_q $command $arglist $tprompt $debug $login $lpass
  }
  foreach host $linux {
   spawn telnet $host
   linux_login $host $tprompt $login $lpass
   linux_q $command $arglist $tprompt $debug $login $lpass
  }
  foreach host $hp {
   spawn telnet $host
   hp_login $host $tprompt $login $lpass
   sunhp_q $command $arglist $tprompt $debug $login $lpass
  }
  foreach host $sco {
   spawn telnet $host
   sco_login $host $tprompt $login $lpass
   sco_q $command $arglist $tprompt $debug $login $lpass
  }
  break
 } elseif {$sunservers} {
  foreach host $sun {
   spawn telnet $host
   sun_login $host $tprompt $login $lpass
   sunhp_q $command $arglist $tprompt $debug $login $lpass
  }
  if { $ssshservers > 1 } {
   set sunservers 0
   incr ssshservers -1
   continue
  }
  break
 } elseif {$linuxservers} {
  foreach host $linux {
   spawn telnet $host
   linux_login $host $tprompt $login $lpass
   linux_q $command $arglist $tprompt $debug $login $lpass
  }
  if { $ssshservers > 1 } {
   set linuxservers 0
   incr ssshservers -1
   continue
  }
  break
 } elseif {$scoservers} {
  foreach host $sco {
   spawn telnet $host
   sco_login $host $tprompt $login $lpass
   sco_q $command $arglist $tprompt $debug $login $lpass
  }
  if { $ssshservers > 1 } {
   set scoservers 0
   incr ssshservers -1
   continue
  }
  break
 } elseif {$hpservers} {
  foreach host $hp {
   spawn telnet $host
   hp_login $host $tprompt $login $lpass
   sunhp_q $command $arglist $tprompt $debug $login $lpass
  }
  if { $ssshservers > 1 } {
   set hpservers 0
   incr ssshservers -1
   continue
  }
  break
 } else {
  foreach host $servers {
   if { [lsearch $sun $host] >= 0 } {
    spawn telnet $host
    sun_login $host $tprompt $login $lpass
    sunhp_q $command $arglist $tprompt $debug $login $lpass
   } elseif { [lsearch $hp $host] >= 0 }      {
    spawn telnet $host
    hp_login $host $tprompt $login $lpass
    sunhp_q $command $arglist $tprompt $debug $login $lpass
   } elseif { [lsearch $linux $host] >= 0 }      {
    spawn telnet $host
    linux_login $host $tprompt $login $lpass
    linux_q $command $arglist $tprompt $debug $login $lpass
   } elseif { [lsearch $sco $host] >= 0 }      {
    spawn telnet $host
    sco_login $host $tprompt $login $lpass
    sco_q $command $arglist $tprompt $debug $login $lpass
   }
  }
  if { $ssshservers > 1 } {
   incr ssshservers -1
   continue
  }
  break
 }
}

exit

, Mike

linux unix internet technology

Apt and Yum Command Cross Reference

Hey there,

Today, we're following up on yesterday's post of our common PKG and RPM command cross reference. Hopefully, this will enhance our previous post by providing the equivalent apt and yum commands for the Solaris Unix pkg and RedHat Linux RPM commands (where they exist) that we went through yesterday..

The format for the list below, will be the same as our PKG and RPM reference and we'll present the actions in the exact same order, so you can put the posts side by side and translate between all four package manipulation methods at a glance. Again, the format is:

What We Want To Do: APT command - YUM command

<--- with PKG being the package name.

Enjoy your Sunday and hope this helps you out. If you know of any way to do a lot of these things with apt or yum that I don't, please feel free to email me and I'll be glad to update this page :)

Show package info: apt-cache show PKG - yum info PKG
Install package: apt-get install PKG - yum install PKG
Remove package: apt-get remove PKG - yum remove PKG
List the package that owns a particular FILE: Can't be done to my knowledge - yum provides PKG
List all files in an already installed package: Can't be done to my knowledge - Can't be done to my knowledge
List all files in an uninstalled package FILE: Can't be done to my knowledge - Can't be done to my knowledge
List all packages installed on your system: Can't be done to my knowledge - yum list installed
Check that all files from a package are installed correctly: Can't be done to my knowledge - Can't be done to my knowledge
Check that all files from all packages are installed correctly: Can't be done to my knowledge - Can't be done to my knowledge
Check that a package FILE is OK before installing: Can't be done to my knowledge - Can't be done to my knowledge
Fix an altered package installation:     Can't be done to my knowledge   -   Can't be done to my knowledge

Cheers,

, Mike

linux unix internet technology




, Mike

linux unix internet technology