Minor Fix Update - Script To Unpack Solaris Datastream Pkg Files Without Sun's Pkg Utils

Hello again,

Updated 4/4/08 to output pkgproto results to a file named prototype. Originally had it writing to a file named pkgproto. Typo corrected :)

It's been a while since I started looking for a way to finish up our series of articles on RPM and pkg ripping and creating. We started out with creating our own RPM's and did pretty much everything I could think of to those poor packages, all the way through creating RPM's from already installed RPM's and converting RPM's to Solaris pkg's ;)

The only project I had left over (from that bunch) was how to extract the binary contents of Solaris Unix datastream pkg files.

Unfortunately, unlike the "rpm2cpio" command that Solaris includes, they have no regular program for converting a datastream package to an unpackable file (like a cpio or tar), unless you count all the basic pkg commands. What I was looking for was a way to extract a datastream pkg file, and get all the binary contents, without actually installing it. I got no love from Solaris, but I got some free time for myself and eventually figured it out ;)

I've included a script, below, to rip apart a Solaris datastream pkg and create a subdirectory (named the name of the pkg) into which I dump the pkginfo, pkgmap, pkgproto (which I derive from the pkgmap) and all the binary files that the pkg contains. The script, itself, is far from perfect, but if you check it out you can see, fairly easily, how you can manually pull apart a Solaris datastream pkg file and get to the binary contents that you really want (At least, I'm assuming you do - just like me ;)

Note that the version of cpio that comes with Solaris (in the SUNWcsu coreutils package) will complain about garbage bits in your cpio header, when you've clipped off the top of the pkg file. There are really only 2 garbage bits you have to delete (which you can do with vi), but I prefer to use the Gnu version of cpio which will "magically" ignore garbage bits and extract the cpio file no matter what. This version of cpio comes standard on Linux and can be downloaded (in Solaris datastream pkg format) from sunfreeware.com or (in source format, with links to all sorts of OS ports) Gnu's CPIO Download site - just in case you don't want to have to rip apart your Solaris pkg's on Linux (or don't have both OS's at your workplace).

Another strange thing is that Sun's version of cpio won't make directories, by default, that don't exist when you unpack a cpio archive (???) All I'm really saying is: Get the Gnu version - It's free and it'll save you many a gray hair ;)

I've tested today's script on multiple packages from sunfreeware.com and had great success with it working right the first time (assuming that I have Gnu's cpio). The script has broken on a few custom pkg's I've put together myself and can use a lot of work in the "additional" areas (like ensuring pre and post-install scripts get accounted for, depend files get tagged, checkinstall programs are properly extracted, etc). Still, it's a good beginning :)

Hopefully this will be of help to you, if you've been looking all over for something like this, and, almost definitely, can be the starting point for a fully functional Solaris datastream pkg ripper :)

If you want to do this manually, in short terms, just vi the pkg file you're interested in and delete everything up to the final instance of "pkginfo" in the file (after the final "pkgmap" line) and save the rest into another file, which you can manipulate with cpio. Who'd have thought it would end up being that simple? :)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# rip_pkg - rip apart Solaris pkg's and get the binaries out
#
# 2008 - Mike Golvach - eggi@comcast.net
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if ( $#ARGV != 0 ) {
 print "Usage: $0 pkg_file\n";
 exit(1);
}

$pkg_name = $ARGV[0];

# check and make sure it's a SVR4 DataStream Pkg

chomp($simple_header_check=`grep -i "package datastream" $pkg_name >/dev/null 2>&1;echo \$?`);

if ( $simple_header_check != 0 ) {
 print "This doesn't appear to be a valid SVR4 DataStream pkg file.  Exiting...\n";
 exit;
}

# Memory Hogging Here - Until This Cold Quits Me And I Can Think My Way Out Of This ;)

open(PKG, "<$pkg_name");
@pkg_file = <PKG>;
close(PKG);

# Split out header information and kludgey cpio file with binary contents

$header_end = 0;
$cpio_start = 0;
open (P_HEADERS, ">$pkg_name.headers");
foreach $line (@pkg_file)   {
 if ( $header_end == 0 && $line =~ /TRAILER/ && $line =~ /pkginfo/ )  {
  print P_HEADERS $line;
  $header_end = 1;
 } elsif ( $header_end == 1 && $line =~ /pkginfo/ ) {
  print P_HEADERS $line;
  close(P_HEADERS);
  $cpio_start = 1;
  open(P_CPIO, ">$pkg_name.cpio");
 } elsif ( $cpio_start == 0 ) {
  print P_HEADERS $line;
 } elsif ( $cpio_start == 1 ) {
  print P_CPIO $line;
 } else {
  print STDERR "WTF?\n";
 }
}
close(P_CPIO);

open(P_HEADERS, "<$pkg_name.headers");
@p_headers = <P_HEADERS>;
close(P_HEADERS);

$output_dir = 0;
$pkginfo_start = 0;
$pkginfo_seek = 0;
$pkgmap_start = 0;
foreach $p_head (@p_headers)  {
 if ( $output_dir eq 1 && $pkginfo_seek == 0 )  {
  $output_dir = $p_head;
  $output_dir =~ s/^(\w+)\W*.*$/$1/;
  chomp($output_dir);
  $pkginfo_seek = 1;
 } elsif ( $output_dir eq 0 && $p_head =~ /datastream/i ) {
  $output_dir = 1;
 } elsif ( $output_dir eq 0 && $pkginfo_start == 0 ) {
  next;
 } elsif ( $output_dir ne 0 && $p_head =~ /pkgmap/ ) {
  $pkgmap_start = 1;
  $pkginfo_start = 0;
 } elsif ( $pkgmap_start == 1 && $p_head =~ /pkginfo/ ) {
  chomp($p_head);
  push(@pkgmap, $p_head);
  last;
 } elsif ( $output_dir ne 0 && $p_head =~ /pkginfo.*PKG=/ ) {
  $pkginfo_start = 1;
 } elsif ( $output_dir ne 0 && $pkgmap_start == 1 ) {
  chomp($p_head);
  push(@pkgmap, $p_head);
 } elsif ( $output_dir ne 0 && $pkginfo_start == 1 ) {
  chomp($p_head);
  push(@pkginfo, $p_head);
 } else {
  print "WTF NOTHING MATCHED\n OD $output_dir PST $pkginfo_start PS $pkginfo_seek PMST $pkgmap_start\n$p_head\n";
 }
}

mkdir("$output_dir");
chdir("$output_dir");
open(PKGINFO_OUT, ">>pkginfo");
foreach $info (@pkginfo) {
 print PKGINFO_OUT "$info\n";
}
open(PKGMAP_OUT, ">>pkgmap");
foreach $map (@pkgmap) {
 print PKGMAP_OUT "$map\n";
}
open(PROTO_OUT, ">>prototype");
foreach $proto (@pkgmap) {
 @proto = split(" ", $proto);
 print PROTO_OUT "$proto[1] $proto[2] $proto[3] $proto[4] $proto[5] $proto[6]\n";
}
system("cpio -iv <../$pkg_name.cpio >/dev/null 2>&1");
chdir("reloc");
system("tar cpf - *|(cd ../;tar xpf -)");
chdir("../");
system("rm -r reloc");
chdir("../");
unlink "$pkg_name.headers";
unlink "$pkg_name.cpio";

, Mike

linux unix internet technology

Converting Linux RPM's Into Solaris Pkg Files

Hey there,

As this week wraps up, I thought I'd put out the last few things I've been toying around with. Today, we've got a Perl script for you that will take an RPM (From Linux) and convert it to a Solaris datastream pkg file. Of course, we've got the opposite (just like we have the opposite of our post on creating Solaris pkg files from already installed content, which we'll post in the next few days. That might be a more useful script than this one (you might actually "need" to recreate a Linux RPM from what's on your box more often), but I thought this script was kind of cool :)

Basically, you can take any Linux RPM (I tested against RedHat AS and SUSE 9), feed it to this script, like so:

host # ./rpm2pkg PROGRAM-3.2-1.rpm

and end up with your own valid Solaris pkg file named:

PROGRAM-3.2-1.pkg

My thought was that this might be useful, since one of the commands used inside it (rpm2cpio) is already included in Solaris 9 and 10. For architecture-independent, and binary compatible, programs, there obviously exists a need to convert Linux RPM's to a cpio archive that can then be extracted to the local filesystem on a Solaris machine. In my mind, the logical next step would be to skip ahead and create a valid Solaris datastream pkg file from that cpio output. This way, the process could be completed once and then distributed easily as a single pkg installation file to all of your Solaris servers:)

The script works, as mentioned above, by using rpm2cpio to extract the Linux RPM's contents and then using the "strings" command on the actual RPM to extract all the header information that we require to seed the "pkginfo" file. The prototype file is the easiest necessary pkgmk file to create since you just have to use find and pkgproto on the extracted rpm2cpio output.

If you want to be able to tweak this more to your liking, there's a lot more information about the Solaris pkg making process in our previous posts on building Solaris pkg files quickly and, more theoretically, what you need to know to create your own Solaris pkg files.

Enjoy the script and have fun trying to get Linux binaries to run on your Solaris box (Hint: You're success rate will be much greater on Solaris 10, since they're finally buying into "open source" :)

Best wishes,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# rpm2pkg - creating Solaris pkg files from Linux rpm's
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

$rpm=$ARGV[0];
$tmp_dir="dir.$$";
$orig_dir=`pwd`;

mkdir("$tmp_dir");
chdir("$tmp_dir");
system("rpm2cpio ../$rpm|cpio -dim");
$proto_list=`find . -print|pkgproto|grep -v prototype`;
open(TMP_FILE, ">>prototype");
print TMP_FILE "i pkginfo\n";
print TMP_FILE $proto_list;
close(TMP_FILE);
@pkg_info=`strings ../$rpm`;
$count = 0;
foreach (@pkg_info)     {
        if ( $_ =~ /\# \@\(\#\)BegWS/ ) {
                push(@rpm_info, $_);
                $count++;
        }       elsif ( $_ =~ /\# \@\(\#\)EndWS/ )      {
                last;
        }       elsif ( $count > 0 )    {
                push(@rpm_info, $_);
        }
}
$uname_s = `uname -s`;
$uname_r = `uname -r`;
$uname_p = `uname -p`;
$pkg_name = $rpm;
$pkg_name =~ s/\.rpm//;
chomp $uname_s;
chomp $uname_r;
chomp $uname_p;
chomp $rpm_info[1];
chomp $rpm_info[2];
chomp $rpm_info[5];
chomp $rpm_info[12];
open(PKGINFO, ">>pkginfo");
print PKGINFO "SUNW_PRODNAME=\"$uname_s\"\n";
print PKGINFO "SUNW_PRODVERS=\"$uname_r\"\n";
print PKGINFO "SUNW_PKGTYPE=\"usr\"\n";
print PKGINFO "PKG=\"$pkg_name\"\n";
print PKGINFO "NAME=\"$rpm_info[2]\"\n";
print PKGINFO "VERSION=\"$rpm_info[5]\"\n";
print PKGINFO "VENDOR=\"$rpm_info[1]\"\n";
print PKGINFO "ARCH=\"$uname_p\"\n";
print PKGINFO "EMAIL=\"me@xyz.com\"\n";
print PKGINFO "CATEGORY=\"application\"\n";
print PKGINFO "BASEDIR=/\n";
print PKGINFO "DESC=\"$rpm_info[12]\"\n";
print PKGINFO "PSTAMP=\"Your Name Here\"\n";
print PKGINFO "CLASSES=\"none\"\n";
close(PKGINFO);
system("pkgmk -o -b `pwd` -d /tmp");
system("pkgtrans -o -s /tmp `pwd`/${pkg_name}.pkg $pkg_name");
system("mv ${pkg_name}.pkg ../");
system("cd ../;pwd;rm -r $tmp_dir");

, Mike

linux unix internet technology

CC FLAG UPDATE - Building Solaris Packages Quickly - An Old Post Revisited

Hey There,

Updated Some More - Thank you for your emails :) I've updated this once more to hard-code the CC=gcc flag on the configure line. This db package's configure script appears not to pickup gcc more often than it does, so, unless you're using cc, it's best just to force gcc.

Now that we've just gotten through a marathon "how to" on creating Linux RPMS, I'd thought I'd look back at the old post I referenced at the beginning of the week regarding making your own Solaris packages.

That post was, and still is, theoretically sound. Although, looking back at it, I feel that it really lacked the kind of example work-up that would have made the instructions come back and hit home. So with reference to my original (admittedly dry) instruction on how to make your own Solaris pkg files, here's a quick example of the actual process of creating a package from compiling your software to the finished product. Hopefully this will not only flesh-out the original post, but also show how much simpler this is to do in reality than it is in theory :)

And, in an effort to make up for lost time and dispel any possibility of confusion, we're going to use a real freely-available source gzip that anyone can grab for free and walk through this on their own Solaris machine step-by-step, cut-and-paste. Here are the basic requirements in order to reproduce this on your own without having to do any extra work outside of the instructions posted:

1. Have gcc and make installed on your Solaris System.
2. Run Solaris on Sparc (not x_86 or Intel - although this should still work).
3. If possible, use the 64 bit version (As long as the "isainfo" or "isalist" commands return "sparcv9" somewhere in their output, you should be in the same boat I am as I do this compile and package creation. Again, this probably won't be an issue if you use 32 bit.

And that should be it. Now we just need to download the Berkeley DB 4.6.21 source gzipped tarball (available by clicking the name of the product on this line or here ;)

And off we go! I'm going to basically walk through this by example (what I'll actually type during the process) and leave out the tons of output from "configure" and "make," etc, so this doesn't get too hard to read. Hopefully, it will be painfully easy to understand :)

Step 1. Unpack and build your software (grab your Berkeley DB source now if you haven't already!

SPECIAL NOTE: This post has been updated to reflect the need to run configure from the build_unix subdirectory under the db-4.6.21 directory. You may also need to set CC=gcc as an option on your configure command line, but db should pick it up as long as it comes first in your PATH. Apologies for the typo.

host # mkdir /tmp/build
host # cp db-4.6.21.tar.gz /tmp/build/.
host # cd /tmp/build
host # gzip -d -c db-4.6.21.tar.gz|tar xvpf -
... <--- This will be shorthand for obligatory command output that doesn't make a difference to us here.
host # cd db-4.6.21
host # cd build_unix
host # ../dist/configure CC=gcc --prefix=/usr/local/db-4.6.21
...
host # make
...
host # make docdir=/usr/local/db-4.6.21/doc/4.6.21 install
...
host # cd /tmp/build
host # rm -r db-4.6.21

And we're done with the compile :)

Step 2. Create the Solaris pkg file from our installation!

host # cd /usr/local/db-4.6.21
host # find . -print|pkgproto >prototype
host # echo "i pkginfo" >>TEMPFILE
host # echo "i checkinstall" >>TEMPFILE
host # cat prototype >>TEMPFILE
host # mv TEMPFILE prototype
host # vi checkinstall <--- For the two files we need to edit, I'll just indent and put the contents of each below the lines where I use "vi" (my favorite editor). You can edit the file with any editor you want :)

#!/bin/sh

 platform_should_be="sparc"
 platform=`uname -p`
 if [ ${platform} != ${platform_should_be} ]
 then
  echo "PROGRAM can only be installed on ${platform_should_be}"
  exit 1
 fi
 exit

host # vi pkginfo

SUNW_PRODNAME="SunOS"
 SUNW_PRODVERS="5.9"
 SUNW_PKGTYPE="usr"
 PKG="MTdb"
 NAME="Berkely DB-4.6.21"
 VERSION="4.6.21"
 VENDOR="Open Source"
 ARCH="sparc"
 EMAIL="eggi@comcast.net"
 CATEGORY="application"
 BASEDIR=/usr/local/db-4.6.21
 DESC="http://linuxshellaccount.blogspot.com DB-4.6.21"
 PSTAMP="The Linux And Unix Menagerie"
 CLASSES="none"

host # pkgmk -b `pwd` -d /tmp
...
host # pkgtrans -s /tmp /tmp/build/MTdb.pkg MTdb
...
host # rm -r /tmp/MTdb

And now you can move the /tmp/build/MTdb.pkg file anywhere on the file system you want. You should be able to utilize all the applicable "pkginfo" command options against it and it should install as easily (assuming you left it in /tmp/build) as this (of course, it won't be any harder if you install it from anywhere else ;)

host # pkgadd -d /tmp/build/MTdb.pkg

also

host # pkgrm MTdb <--- If you ever want to delete it from the package repository

Hope you enjoyed that whirlwind tour of a Solaris package (pkg) build in almost-real-time. Hopefully, also, it helps complement our previous post on creating Solaris packages by giving you a solid working example to compare against the theory!

Cheers,


, Mike

linux unix internet technology

Making Your Own Solaris Packages!

While there are more than a few ways to pack up and distribute software distro's you've either downloaded and custom-compiled, or built yourself from scratch, if you work on a lot of Solaris boxes, it's nice to be able to produce legitimate "pkg" files to showcase your efforts.

Today, we'll look at how easily this can be done (although, of course, many other methods are much much less drawn-ouot). After reading through this post, you should be able to create your own "pkg" files and even (we'll look at this in some detail in a future post) script out and automate the process with a little ingenuity :)

For the purposes of this post, we'll assume that we're going to build the latest version of PROGRAM (pardon my lack of originality in naming ;)

1. First, ensure that you have the disk space and compile your program with the "--prefix" suffix (pretty much standard now with all publicly available source builds):

cd /tmp/PROGRAM_SOURCE
./configure --prefix=/usr/local/builds/PROGRAM
make
make install

2. Now you should have your program built and installed under /usr/local/builds/PROGRAM, with all the subdirectories beginning there (e.g. /usr/local/builds/PROGRAM/bin, /usr/local/builds/PROGRAM/sbin, etc). Make sure that all the files have the ownership and permissions that you will want them to have when your "pkg" file is complete!

3. Now we'll create the "prototype" file. Add all necessary links (like usr=/usr) for all subdirectories that you want your package to unpack into. We're going to assume that your package is going to unpack into the root directory (/) like most Solaris "pkg" files:

cd /usr/local/builds/PROGRAM
find . -print|pkgproto usr=/usr bin=/bin sbin=/sbin etc=/etc >prototype

4. Then, add pointers to the prototype file. You'll "need" one to the "pkginfo" file that we'll create soon, and you can also create entries for "preinstall," "postinstall" and "checkinstall" files, although they're not completely necessary. "checkinstall" is a good file to have if you want to be able to ensure that your package won't install if it is somehow corrupt or not being installed on a proper system.
Note that the "checkinstall" script is run by the user "nobody," and runs automatically, so your source needs to be accessible for checks by that user in order for it to work correctly. "preinstall" and "postinstall" are run as "root" and prompt the user to run them. You can also add a line to reference a script to check dependencies within the "prototype" file of the form "depend=/path/to/depend/script." Add the following to the top of the /usr/local/builds/PROGRAM/prototype file:

i pkginfo
i checkinstall

5. Now create a "pkginfo" file with the basics:

SUNW_PRODNAME="SunOS"
SUNW_PRODVERS="5.10"
SUNW_PKGTYPE="usr"
PKG="PROGRAM"
NAME="The PROGRAM Progam"
VERSION="1.0"
ARCH="sparc"
VENDOR="XYZ, Inc."
EMAIL="insertyouremailhere"
CATEGORY="system"
ISTATES="S 1 2 3"
RSTATES="S 1 2 3"

Lots of these aren't necessary, and are self explanatory. ISTATES are the acceptable run levels in which the package can be installed. RSTATES are the acceptable run levels in which it can be removed.

6. And create a simple "checkinstall" file as well (both the "pkginfo" and "checkinstall" files need to be in the same directory as the "prototype" file and the software - in our case /usr/local/builds/PROGRAM):

#!/bin/sh

# Sample checkinstall script

platform_should_be="sparc"
platform=`uname -p`
if [ ${platform} != ${platform_should_be} ]
then
echo "PROGRAM can only be installed on ${platform_should_be}"
exit 1
fi
exit 0

7. Now, we're finally ready to create the package with "pkgmk" and "pkgtrans":

cd /usr/local/builds/PROGRAM <--- just in case we wandered off
pkgmk -b `pwd` -d /tmp <--- "-b" is your source base diretory and "-d" indicates the "device" (in this case, the /tmp directory) into which you want to build the actual "pkg" file.
pkgtrans -s /tmp /usr/local/builds/PROGRAM.pkg PROGRAM.pkg <--- The "-s" indicates that you want to translate the PROGRAM.pkg file in /tmp in datastream format as /usr/local/builds/PROGRAM.pkg

Now you're all set to grab the PROGRAM.pkg file from the /usr/local/builds directory and install it with "pkgadd" on any Solaris system you want to (depending, of course, on the restrictions you put in place in your "checkinstall" - and possibly "depend" - script)!

, Mike

linux unix internet technology