Simple Cumulative Math Using Awk On Unix or Linux

Hey There,

I'm off to start a new work assignment today (after sleeping in late for a week), so, as you're reading this I may be trying not to fall asleep on the train or wishing I had slept more and/or was dead ;) In any event, this week, I thought we'd start getting back to that interstitial series on porting between shell, awk and Perl on Linux or Unix and, by way of getting back into it, make this post (at least a little bit) resemble the last post, which dealt with simple arithmetic.

One of the questions I see floating around the boards a lot has to do with using awk to calculate cumulative (or incremental) sums within a file. So, for a simple example, if a user had a file called "numbers" which contained the following:

host #cat numbers
a 4
b 2
c 8
d 57
e 12
f 8967
g 3
h 58
i 3

they'd want awk to read through that file, and give the total result of all the numbers in column 2. For kicks, they may also want to print each line in the file as it gets processed (at least until they're satisifed that it's doing what they want. After that it gets to be a consumption of screen space for no reason)

Our attached shell script (which only runs awk inside it and accepts a filename as an argument) does just that. Note that it doesn't do any error checking, and assumes you know that your input file should be of the form: one pair per line of "name" [space or tab] "number," like above. It can be run very simply from the command line, and should provide output, like so:

host # ./sum.sh numbers
a numvalue: 4
b numvalue: 2
c numvalue: 8
d numvalue: 57
e numvalue: 12
f numvalue: 8967
g numvalue: 3
h numvalue: 58
i numvalue: 3
Sum total: 9114

Of course, this script can also be written as a one-liner, as long as you redirect the input to it that it requires, like so:

host # awk 'BEGIN {name="testcase";oldname="testcase"}{name=$1;if (name != oldname || oldname != "testcase") print name " numvalue: " $2; sum=0;finalsum += $2;oldname = name} END {print "Sum total: " finalsum}' numbers
a numvalue: 4
b numvalue: 2
c numvalue: 8
d numvalue: 57
e numvalue: 12
f numvalue: 8967
g numvalue: 3
h numvalue: 58
i numvalue: 3
Sum total: 9114

Basically, all we're doing is going through each line of the file and adding the numeric value of each line to the numeric value of the total until the input file ends. Of course, you can modify this slightly to make it do any sort of arithmetic operation you require. Multiplication, division, subtraction, etc. At least, as much as awk will allow ;)

Hope you're enjoying this more than I'm probably enjoying my morning. And, in both cases, there's probably a better, and more efficient way to do this ;) Here's to that!

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

#
# sum.sh
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

file=$1
awk 'BEGIN {name="testcase"
        oldname="testcase"}
        {name=$1
        if (name != oldname || oldname != "testcase") {
                print name " numvalue: " $2
                sum=0}
        finalsum += $2
        oldname = name}
        END {print "Sum total: " finalsum}' $file

, Mike

Fast Perl HTML POD Creation On Linux and Unix

Hey There,

For our "Lazy Sunday" Post this week, we're going to look at some more, hopefully, useful and easy html creation, like we've done many times in the past with posts on listing out Linux or Unix crontabs and a few others. Today's Bash shell script will produce an index.html file (and a whole lot of others) that should end up as your one-stop-shop for the documentation that came with your system's Perl distribution. It's nothing fancy (although you can make it that way if you want) but it should serve its purpose well enough right out-of-the-box.

Of course, you should edit the POD_HTML_DIR variable, at the least, so that the script can find your Perl modules and use pod2html to convert them. Other than that, running it straight-up, like so:

host # ./pods.sh

should produce output like below, in the following two pictures (click on them to make them larger if you need to :)




Hope you enjoy the script and it brings you some convenience! Note that I'm 100% aware that there is no error handling done for files that can't be created (i.e. when pod2html fails, a blank html page is hyperlinked to rather than no hyperlink, or html, being created), but that should make the whole experience more educational if you want to add that. You can refer to our earlier posts on Bash's implementation of errno and the Bash-specific PIPESTATUS variable to help handle those errors and do what you want with them ;)

Happy Sunday,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# pods.sh - make perdocs into one html repository
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

trap 'cd $PWD;rm -rf $POD_HTML_DIR;exit 2' 1 2 3 15

PWD=`pwd`
PERL_MANPAGE_BASE=/usr/lib/perl5/5.8
POD_HTML_DIR=podhtml
INDEX_PAGE=index.html

if [ -d $POD_HTML_DIR ]
then
        rm -rf $POD_HTML_DIR
fi

mkdir podhtml
echo -n "Working on it..."

find $PERL_MANPAGE_BASE -name "*.pm"|while read line do
do
        echo -n "."
        output=`echo $line|sed 's/^.*\/\(.*\)$/\1/'`
        pod2html $line >podhtml/${output}.html 2>/dev/null;
done

cd $POD_HTML_DIR

echo "<html><head><title>Perl Module Document Repository</title></head><body>" >$INDEX_PAGE
ls |grep html|while read line
do
        echo "<a href=\"$line\">$line</a><br>" >>$INDEX_PAGE
done

echo "</body></html>" >> $INDEX_PAGE
echo "done"

, Mike

Fight Phishing From The Unix Or Linux Command Line

Hey There,

This weekend, we're going to look at a little something (actually a HUGE something) that we all seem to have to deal with now (through email, IM, etc). It's called phishing and, for lack of a better explanation, it's a fraudulent way for a malicious individual, or group, to get username and password information from you using deceptive practices (was that redundant? ;) It's most commonly used to get information regarding a person's online bank accounts, credit card/mortgage management information, etc. In other words, stuff that could really screw up their lives if someone else had it.

First off, I'd like to be clear that I am "not against SPAM." As annoying as it may be, I don't begrudge anyone their right to send me a sales pitch, even if that product is a complete scam and I didn't ask to be pitched. I've been throwing stuff out that I get in my regular postal mail since I've been able to throw stuff out and I'm not sure if I've ever double-opted-in (the ethical way) or just had my address sold to some random company when it comes to internet SPAM.

Phishing, on the other hand, upsets me to no end. I do begrudge everyone their right to try and, literally, steal from others, using the internet. I feel the same way about the folks who used to steal carbons from credit cards (when that trick used to work... sadly, to a degree, it still does) and that other, more honest, faction of individuals who walk right up to the counter, point a gun in a bank teller's face and demand everyone else's hard earned cash. I'm not that this sort of activity should be encouraged, or glorified, in any way. I'm just saying it, at least, shows some character. If you're going to steal, and that's your business, why not come right out and say it? Only cowards pretend they're not pointing that figurative gun in your face and giving you a good mugging.

I can, literally, feel my email box filling with requests for password-resets at my bank of choice. Funny thing is, a lot of times, these clowns don't even bother to verify your personal information before they try and scam you. IMPORTANT NOTICE TO PHISHER'S WORLDWIDE: I do not have an account at the Federated Bank Of Hindustan. And, even if I did, it's been so long since I've attempted to speak Urdu that I can't possibly remember my clever username ;)

If you have the time and inclination to hurt these phishers, check out sites like The Anti-Phishing Working Group and PhishTank, and read up on how this whole operation works at sites like Security Focus. PhishTank actually even has a plug in for FireFox named Site Checker. Here's another site that provides a great list of possible phishing programs that may be running on your machine. The advantages that these organizations bring is that they don't merely do "anti-spam" work (which is, by all means, necessary and laudable). They actually collect data on phishing schemes, get to these people backward through their own chains of deceit and, if possible, have them put in cell block A (or wherever it is that they can legally squirrel away thieves and murderers - preferably together).

Unfortunately, though the calls-for-action are many, you should not expect any automated anti-phishing toolbars, etc, for your web browser. There's a very good reason why they're slow in coming out. Phishing is a lot like "social engineering," whereby a stranger attempts to extract personal information from you under a false pretense. Using this as an example, a stranger may also innocently request information you're not comfortable with giving under a legitimate pretense (?) or it all may be a big misunderstanding. This is why most phishing, SPAM and "social engineering" confidence violations are handled by manual submission or, at least, with some degree of user interaction. If you set Internet Explorer on auto-pilot to trash every site it thought was a phishing site, I'd bet about 50% of legitimately required bank (or other e-commerce) authorization, or confirmation, emails would end up getting skunked and ruin your web experience (and a few reputations in the process) very very quickly.

To wrap this all up (although it's certainly not the only solution), I prefer to use PhishTank since they have a very easy email submission process (You must register with them to send phishing complaints, and I encourage you to investigate their legitimacy on your own; but I like 'em a lot) which works whether or not you're sending them phishing information from your PC email client or from your Linux or Unix box (or any box that you may not have actually "received" the message on -- more below).

This is where the Linux and Unix command line comes into play. While you can easily forward an email from your mail client of choice to phish@phishtank.com (Not a mailto link on purpose), you will almost always lose some vital information in the forwarding. Instead I prefer to do the following:

1. From my email software of choice, view the entire email, including headers <-- Aside from the phishing URL, this is the most important part!

2. Log on to my Linux box (like I'm not already ;) and use any editor, like vi to paste the "entire contents" of the above output into a file.

3. Then, I send email to PhishTank using the customized email address that they've provided me. Their customized email address, for a user to send reports in from anywhere, is of the following format: phish-YOURACCOUNTNAME.YOUR_RANDOMLY_GENERATED_PHISHTANK_KEY@phishtank.com. Note that your randomly generated key will remain the same unless you want to change it. You can use any email client you like to send them the email, then. Assuming you put in it a text file called WTF, you might use mailx to send it, like so (Note that no Subject line is actually necessary):

host # mailx -s "Phishing Abuse Report" phish-YOURACCOUNTNAME.YOUR_RANDOMLY_GENERATED_PHISHTANK_KEY@phishtank.com < WTF

And, just like that, you've launched an investigation and, hopefully, put some purse-snatcher out of business.

And, lest it be said that I never said it, always double check any emails you receive from PhishTank, any bank, any service, even from me :) It never hurts to take a second and make sure you're not being scammed. The alternative can be devastating!

Best wishes,

, Mike

SunCluster Quick Command Reference

Hey There,

Well, sitting back and reflecting today, I realized that we've done our fair share of posts on cheat-sheets and such (Our LVM command reference, for example). However, when it comes to clustering systems, we've probably been unfair in our bias toward Veritas Clustering as opposed to SunCluster, for which we've only put up, I believe, one post with our Simple SunCluster Monitoring Script (Thankfully linked back to from The SunCluster Wiki). What's a poor blog to do? ;)

With that in mind, today we're going to lay down a quick command reference for SunCluster. Hopefully you'll find it useful and enjoy it :) Of course, it assumes that you know, mostly, how to use SunCluster (and are familiar with basic clustering concepts), but it can be used as an attractive wallpaper, if nothing else ;)

All of this information, and much more, can be found at Sun's Online Documentation Center for SunCluster.

Enjoy,

Quorum Commands:

host # clquorum add device - Add a SCSI Quorum Device
host # clquorum add -t netapp_nas -p filer =nasdevicename,lun_id =IDnumdevice Nasdevice - Add a NAS Quorum Device
host # clquorum add -t quorumserver -p qshost =IPaddress, port =portnumber quorumservername - Add a Quorum Server
host # clquorum remove device - Remove a Quorum Device

Resource Type Commands:

host # clresourcetype register type - Register a Resource Type
host # clresourcetype unregister type - Remove a Resource Type

Resource Group Commands:

host # clresourcegroup create group - Create a Failover Resource Group
host # clresourcegroup create -S group - Create a Scalable Resource Group
host # clresourcegroup online + - Bring Online All Resource Groups
host # clresourcegroup delete group - Delete a Resource Group
host # clresourcegroup delete -F group - Delete a Resource Group and All of Its Resources
host # clresourcegroup switch -n nodename group - Switch the Current Primary Node of a Resource Group
host # clresourcegroup unmanage group - Move a Resource Group Into the UNMANAGED State
host # clresourcegroup suspend group - Suspend Automatic Recovery of a Resource Group
host # clresourcegroup resume group - Resume Automatic Recovery of a Resource Group
host # clresourcegroup set -p Failback=true + name=value - Change a Resource Group Property
host # clresourcegroup add-node -n nodename group - Add a Node To a Resource Group
host # clresourcegroup remove-node -n nodename group - Remove a Node From a Resource Group

Resource Administration Commands:

host # clreslogicalhostname create -g group lh-resource - Create a Logical Hostname Resource
host # clressharedaddress create -g group sa-resource - Create a Shared Address Resource
host # clresource create -g group -t type resource - Create a Resource
host # clresource delete resource - Remove a Resource
host # clresource disable resource - Disable a Resource
host # clresource set -t type -p name=value + - Change a Single-Value Resource Property
host # clresource set -p name+=value resource - Add a Value to a List of Property Values; Existing values in the list are unchanged.
host # clresource create -t HAStoragePlus -g group -p FileSystemMountPoints=mount-point-list -p Affinityon=true rs-hasp - Create an HAStorage Plus Resource
host # clresource clear -f STOP_FAILED resource - Clear the STOP_FAILED Error Flag on a Resource

Device Administration Commands:

host # cldevicegroup create -t vxvm -n node-list -p failback=true vxdevgrp - Add a VxVM Device Group
host # cldevicegroup delete devgrp - Remove a Device Group
host # cldevicegroup switch -n nodename devgrp - Switch a Device Group to a New Node
host # cldevicegroup offline devgrp - Bring Offline a Device Group
host # cldevice refresh diskname - Update Device IDs for the Cluster

Additional Administration and Monitoring Commands:

To add a Node to Cluster:

From the node to be added, which has access:

host # clnode add -c clustername -n nodename -e endpoint1, endpoint2 - Use this only if the node has access to the cluster

To remove a Node From the Cluster:

From the node to be removed, which is in noncluster
mode and has access:

host # clnode remove - Use this only if the node has access to the cluster

host # clnode evacuate nodename - Switch All Resource Groups and Device Groups Off of a Node
host # clinterconnect disable nodename:endpoint - Manage the Interconnect Interfaces
host # clinterconnect enable nodename:endpoint - To disable a cable so that maintenance can be performed, then enable the same cable afterward.
host # cluster status - Display the Status of All Cluster Components
host # command status - Display the Status of One Type of Cluster Component
host # cluster show - Display the Complete Cluster Configuration
host # command show Component - Display the Configuration of One Type of Cluster
host # command list - List One Type of Cluster Component
host # clnode show-rev -v - Display Sun Cluster Release and Version Information

To list the software versionson the current node.

host # clnode show | grep nodename - Map Node ID to Node Name
host # cltelemetryattribute enable -t disk rbyte.rate wbyte.rate read.rate write.rate - Enable Disk Attribute Monitoring on All Cluster Disks
host # cltelemetryattribute disable -t disk rbyte.rate wbyte.rate read.rate write.rate - Disable Disk Attribute Monitoring on All Cluster Disks

SHUTTING DOWN AND BOOTING A CLUSTER

To shut Down the Entire Cluster:

host # cluster shutdown

From one node:

host # clnode evacuate
host # shutdown - Shut Down a Single Node

To boot a Single Node from the OBP:

ok> boot

To reboot a Node Into Noncluster Mode from the OBP:

ok> boot -x

, Mike

Mirroring Your Boot Disk Using Veritas On Sun Without Encapsulation

Hey There,

Today, we're going to take a look at a using Veritas to mirror your boot disks, without using encapsulation. There's raging debate about whether or not Veritas root disk encapsulation is bad (some folks are great with it because it takes care of everything, while some folks hate it because it's very hard to deal with when it doesn't take care of everything ;)

The following procedure was introduced quite a while ago by a pair of gentleman named Gene Trantham and John S. Howard in an article they wrote for Sun Blueprints Online (A site run by Sun that highlights best-practices for many, many things Solaris). I believe it's now simply referred to as the "Trantham/Howard Method."

Although I've used this method many times in my career, there's no point in re-writing the instructions (although I did, to a certain degree, to make them more concise and accessible). I did, however, want to have this information included on my blog, to complement (and complete) our earlier posts on patching your Veritas root disk, Veritas Volume Group setup, etc. The full document can be read here in the August 2000 Online Edition of Sun Blueprints.

So, without further ado, and, hopefully you'll find this helpful as well, the "Trantham/Howard Method" for mirroring boot disks using Veritas on Sun Solaris (somewhat modified and truncated):

Enjoy,

1. Install the basic Veritas Volume Manager Software.

2. Install any required Veritas licenses.

3. Use vxinstall to encapsulate the root disk c0t0d0 (Or whatever cxtxdx disk name accurately reflects your setup). Choose not to encapsulate the root disk and don't mirror it. For this post's sake, name it rootdisk Once the root disk is encapsulated, simply reboot and check "vxprint -ht" - You should note that only your rootdisk (and its Veritas name) is listed out.

4. Initialize the root mirror disk c0t1d0 (again, substitute with your own specific controller-target-disk name).

host # /usr/lib/vxvm/bin/vxdisksetup -i c0t1d0
host # vxdg adddisk rootmirror=c0t1d0

5. Manually attach the mirror plexes to the root volumes with "vxrootmir" (manually mirror the swapvol, usr and var volumes, if necessary, as well):

host # /usr/lib/vxvm/bin/vxrootmir rootmirror
host # vxassist mirror swapvol alloc=rootmirror
host # vxassist mirror usr alloc=rootmirror
host # vxassist mirror var alloc=rootmirror

While you wait for this to finish, you can check on the progress with the vxtask command.

6. Once your root disk is mirrored, disassociate the original plexes on c0t0d0, remove them, and remove the encapsulated rootdisk from the rootdg diskgroup:

host # vxplex dis rootvol-01 swapvol-01 var-01 usr-01
host # vxdg rmdisk rootdisk

Now your operating system is entirely running off of the c0t1d0 disk.
At this point, the operating system is based just on the initialized disk c0t1d0, which you can see by looking at the output of "vxprint -ht" again.

7. Now initialize, as opposed to encapsulating, c0t0d0:

host # /usr/lib/vxvm/bin/vxdisksetup -i c0t0d0
host # vxdg adddisk rootdisk=c0t0d0

8. Now, mirror everything from the rootmirror disk back to the rootdisk:

host # /usr/lib/vxvm/bin/vxrootmir rootdisk
host # vxassist mirror swapvol alloc=rootdisk
host # vxassist mirror var alloc=rootdisk
host # vxassist mirror usr alloc=rootdisk

Again, you can use the vxtask command to view the progress of this command, which may take some time to complete. Also, this process removes the "rootdisk-B0" subdisk. This is used by Veritas to protect the boot block, but isn't necessary when you're not encapsulating your disks.

9. Using the output of "vxprint -ht", create disk slices corresponding to the Veritas subdisks. The usage for vxmksdpart command is:

vxmksdpart [-f] [-g diskgroup] subdisk sliceno [tag flags]

You won't need to run through this for the root slice since vxrootmir has already done this for you. You will need to do this for the other slices, though. But, don't use vxmksdpart just yet.

10. On the "new" root disk, first create the slices for the partitions you're going to mirror:

host # /usr/lib/vxvm/bin/vxmksdpart rootdisk-02 1 0x03 0x01
host # /usr/lib/vxvm/bin/vxmksdpart rootmirror-02 1 0x03 0x01

and so on for each required partition (again, you can get this output from "vxprint -ht"). The "sliceno" part of the command syntax is very important here!

host # /usr/lib/vxvm/bin/vxmksdpart rootdisk-03 5 0x07 0x00
host # /usr/lib/vxvm/bin/vxmksdpart rootmirror-03 5 0x07 0x00
...

11. Among other things (like backing up /etc/vfstab, etc) be sure to capture the current disk partition information for both disks, before rebooting, using vxprtvtoc:

host # /usr/lib/vxvm/bin/vxprtvtoc -f /tmp/disk1 /dev/rdsk/c0t0d0s2
host # /usr/lib/vxvm/bin/vxprtvtoc -f /tmp/disk2 /dev/rdsk/c0t1d0s2
host # cat /tmp/disk1 /tmp/disk2 > /etc/vtoc

12. Update your system dump configuration to use the swap partition on the primary disk:

host # dumpadm -d /dev/dsk/c0t0d0s1
Dump content: kernel pages
Dump device: /dev/dsk/c0t0d0s1 (swap)
Savecore directory: /var/crash/host
Savecore enabled: yes

13. In the OBP, be sure to setup both disks using devalias (name them disk and rootdisk, if you want, or whatever makes the most sense to you) and set your configuration to have the boot-device set up to make it easy for your system to fail over if the primary disk goes bad:

ok> setenv boot-device disk, rootdisk, net

14. Be sure to test booting from the OBP for both disk and rootdisk. Assuming that goes well, you're all set :)

, Mike

Simple Perl Script To Ease Console Server Use On Linux And Unix

Howdy,

Well, it's been a good long time (almost a week) since I've posted anything but numbered lists of, hopefully, useful information. It's about time for a little scripting. So, just to maintain my sanity (if not your own - assuming you follow this blog a bit, for which I thank you), today I'm putting up a little Perl script I wrote to maintain connections to a simple Cyclades console server. It can be re-worked fairly easily to connect to multiple console servers of any kind.

Technically, this script could be used for any server which accepts connections over telnet (Which this script is set up to use, even though I'm pretty sure I fought the good fight arguing against it). For more information on why it's a bad idea to use telnet for anything you don't have to, check out our old posts on using tcpdump on Linux or snoop on Solaris to easily grab user passwords. It should also be noted that I wrote this for a non-primary employer (contract). The IRS can rest assured that they'll get their cut of my money at the end of the quarter, but I can't, of course, reveal the identity of the client at this point in time (and, for ethical reasons, I won't later, either, since, given the setup, it would be like setting up a turkey-shoot ;)

The script can be run simply, like most of the scripts we put out here, from the command line. It only requires that you supply a hostname and will spit out help information if you just run it with no arguments, like so:

host # ./dconsole.pl
Usage: ./dconsole.pl [-h|-l|hostname]

This Perl script is fairly limited for a reason, but it's very easy to expand upon and only requires minor adjustment to set it up to use SSH. Since, of course, SSH would require some sort of key-or-host-based authentication method set up before this script would work, be sure to do that before you just run this out-of-the-box. Believe it or not, we've got a post on distributing and setting up SSH keys network-wide on this site, as well ;)

Sometimes I feel like I'm beginning to write in a mobius strip ;)

Anyway, enjoy, and best wishes,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# dconsole.pl - Connect to your console server or modify to connect to a whole bunch!
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if ( $#ARGV == -1 || $#ARGV > 0 ) {
 print "Usage: $0 [-h|-l|hostname]\n";
 exit(1);
}

# This is a simple array to list out all hosts connected to the console server

@hosts = qw(host1 host2 host3);

if ( $ARGV[0] =~ /^-h$/ ) {
 print "\n";
 print "dconsole.pl - Mike Golvach - eggi\@comcast.net\n";
 print "\n";
 print "-h : This piddly screen.\n";
 print "-l : List hosts available.\n";
 print "hostname : host to connect to.\n";
 print "\n";
 exit;
} elsif ( $ARGV[0] =~ /^-l$/ ) {
 print "\n";
 print "@hosts";
 print "\n";
 exit;
}

$goodtogo = 0;
for $hostname (@hosts) {
 if ( $ARGV[0] eq $hostname ) {
  $goodtogo = 1;
 }
}

if ( ! $goodtogo ) {
 print "\n";
 print "$ARGV[0]?  Never heard of it.\n";
 print "\n";
 exit(1);
}

# This is a simple hash to associate connected hosts to their ports on the console server

%whatitis = qw(host1 2 host2 3 host3 4);

$conport = $whatitis{$ARGV[0]};
print "\n"; 
print "Opening $ARGV[0]...\n";
print "\n";
system("/usr/bin/telnet THECONSOLESERVER $conport");
exit;

, Mike

Linux And Unix System Security Wrap-Up - Part 4b

Hey again,,

Well, to run down the list once last time, this series of posts has gone through basic Linux and Unix installation security, even more security measures, user, group and account-based security, ensuring existing software's security and looking at additional security software. And now (finally? ;) we're ready to wrap it up by going over additional security software that you may (or may not) want to add to your list of always-installed security-enhancing programs, as well as a few extra thoughts on random things that didn't seem to fit anywhere else.

Be sure to check yesterday's system security post if you want to learn more about encap.

I'd love to be able to say that this is the end, but (like most things in life) hopefully this is just the beginning for many of you. There's always room for improvement ...I'm sounding like my Dad more and more every day ;)

Cheers,

9. Install John The Ripper:

Update /usr/dict/words (or your system's dictionary file) with a better wordfile.
Add “/usr/local/encap/jtr-VERSION/john.sh” (which you can grab from this post on password cracking software) to root's cron.

10. Install tripwire (This might be a bone of contention, since I still have a binary from when they used to give it out free on Solaris. As of my last test on Solaris 10, the last free release for Solaris still works adequately. If you want a copy of that last release, email us and we'll be happy to send it to you. If enough people are interested, we'll put it up here on the site along with instructions on how to get it up and running quickly and securely. It's always been free for Linux):

Copy installation binaries from the install cd, or the packaged version of the public domain software (hereafter referred to as PDSTW in parenthetical instructions).
Edit install.sh in the appropriate directory to set the base directory to /var/TSS (PDSTW - /var/tripwire).
If you’re unaware of the site and local key conventions used, please consult a member of your administration group before creating these (PDSTW - Unnecessary. Security is handled by permissions and ownership).
Create a policy file and configuration file from the examples provided with the distribution. Use other policy and config files for speed of application (PDSTW - Edit tw.config)
Edit your policy file (PDSTW – Edit tw.config) to remove any listed programs that no longer exist on your system.
Install policy and configuration files (PDSTW - Unnecessary).
Initialize the tripwire database (generally with "tripwire --init" You may need to specify -c with the location of the config file, as well).
Run tripwire.
Read the damage report.
Update the tripwire database, if necessary, and rerun until you’ve rectified all problems (PDSTW - Edit tw.config).
Add email functionality to your tripwire policy.
Reinstall the tripwire policy (PDSTW - Edit tw.config).
Add “/var/TSS/bin/tripwire --check --email-report >/dev/null 2>&1” to root's cron (PDSTW - add “/var/tripwire/tw-mailer”, or whatever you decide to call your script, to root cron).

11. Install Your Custom Log Management Scripts (These will, of course, vary, since you'll be putting all your log-rotating/removing scripts here. You wrote them, so, I can't name them specifically ;):

mkdir /var/logs

Copy over the logging encap package to /usr/local/encap.
Add all your log scripts to root's cron.
Add “/usr/bin/find /var/logs/ -type f –mtime +30 –exec rm {} \;” to root's cron, if you want to remove log files after they get a month (give-or-take-a-day) old.

12. Install EGD:

The entire program consists of one file: /usr/local/encap/perl-VERSION/bin/egd.pl
Create a startup script for egd and place it in /etc/init.d. Make appropriate links to it in /etc/rc2.d and /etc/rc0.d. (NOTE: This service must be initialized before SSH in the startup sequence. It should also be killed after SSH in the shutdown sequence. If you are using the OpenSSH/SSH that came with your system, this part doesn't matter)

13. Install OpenSSH (Version 5.0p1 or later):

The package you've put together should include OpenSSH, OpenSSL and zlib. Ensure all are available, or OpenSSH’s programs will not run properly
Review the documentation on creating host-keys.
Create host keys.

EX:
/usr/local/encap/openssh-5.0p1/bin/ssh-keygen -b 1024 -f /usr/local/encap/openssh-5.0p1/ssh/ssh_host_key -N ''; /usr/local/encap/openssh-5.0p1/bin/ssh-keygen -b 1024 -f /usr/local/encap/openssh-5.0p1/ssh/ssh_host_rsa_key -N '';/usr/local/encap/openssh-5.0p1/bin/ssh-keygen -d -f /usr/local/encap/openssh-5.0p1/ssh/ssh_host_dsa_key -N ''

Edit /usr/local/encap/openssh-5.0p1/etc/sshd_config. Set PermitRootLogin no, X11-Forwarding no and PrintMotd yes.
Be sure that all unnecessary r-services that can be replaced with s-services are not still functional on the machine.

14. Minimally setup ntpd or xntpd:

Create a startup script in /etc/init.d and link it to the appropriate place in /etc/rc2.d.
Add “/usr/sbin/ntpdate TIMESERVER >/dev/null 2>&1” to root's cron, unless xntpd has already been set up.

15. Install the tcp-wrappers encap package.

16. Install the wu-ftpd (proftpd, vsftpd, etc) encap package – if necessary.

Change the appropriate line in /etc/inetd.conf to use /usr/local/sbin/in.ftpd –da.
Edit /usr/local/lib/ftpd/ftpaccess to the particular machines specifications.

17. Install The Coroner’s ToolKit encap package.

18. Install the chkrootkit Root Kit and/or worm/trojan checking package of your choice (install two, they're free ;).

19. Install the Top encap package.

20. Install the environment-specific Perl package.

21. Install the lsof encap package.

22. Install the sudo encap package.

Set it up initially to allow no privileges whatsoever. Modify, per the sudo online manual pages, to suit your site requirements.

21. Build and install the Uptime (Big Brother, Nagios, cfenginge or other monitoring software) client for your server. This will most likely be provided to you in pkg format.

22. Install the NetBackup client, or whatever backup software is necessary for the environment your machine exists in and be sure to coordinate your efforts with the administrator of the backup host you will be using.

23. Build and install any additional packages required by your particular environment. Be sure to build them to encapped specifications so that you can manage all software centrally and with greater ease.

24. Remove any unnecessary packages on the machine. This may be a double or triple check for you ;)

25. Edit /etc/vfstab on Solaris (/etc/fstab on RedHat) to mount /usr read only (Only if the machine is not heavily developed upon and /usr is a separate mount point and is not being used to house any files of a volatile nature, such as if /usr/local/ does not have its own mount point).

26. Lock passwords (e.g. passwd –l accountName) for bin, sys, adm, daemon, nobody, nobody4, noaccess and any other system accounts that do not have password locking on by default. This is a double check, again, but during the course of customization things may have been added that weren't on your system initially. Best bet, since the default system accounts differ based on OS version, is to cat /etc/passwd and lock passwords for all accounts with a UID under 100, except for root, of course. On RedHat and Open Solaris, be sure to check above uid 100, too)

27. chmod 4550 /usr/bin/su – generally only on administration-team-owned or production machines – otherwise make sure permissions are 4755.

28. chown root:yourAdminGroup /usr/bin/su – generally only on administration-team-owned or production machines – otherwise make sure ownership is set root:other on Solaris or root:root on RedHat.

29. If running a web server, be sure to include the following lines (Apache specific – use equivalent for Netscape, Tomcat, JBoss, etc) to the main directory directives:

<Limit GET POST OPTIONS PROPFIND>
Order allow,deny
Allow from all
</Limit>
<Limit PUT DELETE PATCH PROPPATCH MKCOL COPY MOVE LOCK UNLOCK>
Order deny,allow
Deny from all
</Limit>
Order allow,deny
Allow from all

30. Create additional init scripts for any products that need to start up at boot time (i.e. Interbase, Uptime, Oracle, NetBackup, etc. This may be obviated by your Jumpstart or Kickstart image but may be necessary (on a per-software-package basis) if doing a manual installation).

31. Reboot the Machine over and over again, while keeping tabs on your console output, until you're satisfied that you've done your best :)

Excellent Job! :)

, Mike

Securing Your Linux Or Unix System - Part 4a

Hey again,,

Well, so far in this series we've hit upon basic Linux and Unix installation security, even more security measures, user, group and account-based security and ensuring existing software's security. Today, we're going to continue by going over some additional security software that you may (or may not) want to add to your list of always-installed security-enhancing programs. This post, like the basic Linux and Unix installation security post, is going to end up being a two-(long)-parter. If only I planned my writing before I sat down and wrote these little novels... ;)

Cheers,

OS Installation Guidelines, Part 4: Additional Software Installation

Note: Our assumption here is that you've built your additional custom software to live in /usr/local/encap/PRODUCT-VERSION and packaged your it using a program called encap, originally developed at UIUC by a former manager of mine, who's name I will not mention because I haven't seen him in a long time and I don't publish anything personal about anyone else but me on this blog (unless they want me to ;) If you're reading this, you know who you are and my hat's still off to you, sir. Thanks for politely kicking me in the arse until I got my act together ;) I'm actually super-old-school on this, install all my custom software in /usr/local/encap, prefer version 1.2 (encap.pl) and haven't ever had the need to complicate my life with any of the newer versions. They should all be available for download from the above link, including instructions on proper usage.

Apologies for getting misty up there ;) Aside from our assumption, which will explain some of the instructions, any (or all) of this can be handled by Solaris pkg's or RedHat RPM's (rolling your own is my preference if I go down that road, but, these days, the products that come out are usually good enough if you're okay with "default build options"). The relative pro's and con's of using encap or epkg vs. regular packages are listed at the encap FAQ.

Note, also, that any packages for which there are no additional instructions do not require any additional setup, necessarily.

1. Clean up the /usr/local area as well as possible:

Remove all compilers (preferably, build the software on a different machine, or just clean up when you're done) if the machine is on our network’s perimeter. Be sure to check if they were installed, using pkginfo on Solaris or rpm -qa on RedHat, so you can remove them that way.

Remove any unnecessary programs, or rebuild them in encapped form before deleting them. If you do build new packages in encapped form, follow the same process as building any encapped software and place it in the /usr/local/encap directory.

2. Install encap to place software –

mkdir –p /usr/local/encap

3. assuming that you've built everything on another server, unpack your compiled software bundle into /usr/local/encap. A good collection of packages might look something like this, and then some (Note that a lot of these programs are included with Solaris 10, Open Solaris and RedHat, but have been included for folks still using older OS versions):

encap-1.2.
cops.
tcp-wrappers – tcpd.
chrootuid.
skey.
lsof.
gzip-gunzip.
John The Ripper.
Log management scripts.
Perl – Statically stripped in production environments.
egd – Entropy Gathering Daemon.
Open-SSH (Compiled to use EGD).
wu-ftpd or similar ftp server (if necessary).
The Coroner’s ToolKit (TCT).
top.
ntpd/xntpd
sudo
chkrootkit-latest-version (Or latest revision)

4. Create a file called /usr/local/encap/encap.exclude, which should include the lines:

cops
jtr
tct
tcp-wrappers
chkrootkit-latest-version

5. Run:

/usr/local/encap/encap-1.2/bin/encap.pl /usr/local/encap /usr/local

NOTE: You really don't need to run this command until you have successfully installed all of the packages per the instructions following for each. If you do, you will need to re-run it once you’ve completed your installations to ensure that all the links get made (or removed) as necessary.

6. Install cops:

mkdir /usr/local/encap/cops-VERSION

Copy your entire cops distribution to this directory.
Edit /usr/local/encap/cops-VERSION/cops and ensure that the secure directory is set to /usr/local/encap/cops-VERSION, the MMAIL variable is set to YES, the ONLY_DIFF variable is set to NO, the RUN_SUID variable is set to YES and the secure user is set to yourAdminGroup@yourhost.com.
Edit the cops executable script. Look for this section and add the noted line if it does not exist already:

if $TEST ! -s $RESULT".FILT" ; then
$RM -f $RESULT
$RM -f $RESULT".FILT" <--- Add this line so your directory doesn't fill up with useless files.
exit 0
fi

Run cops.
Fix reported problems
Run cops again.
Repeat until you’re left with a report you can live with.
Run /usr/local/encap/cops/cfm (a script you can grab from this previous post on how to create a cops filter file).
Add “/usr/local/encap/cops-VERSION/cops –f /usr/local/encap/cops-VERSION/cops_filter” to root's cron.

7. Install the gzip-gunzip encap package.

8. Install S/key (configure only if not a heavy-user machine or if the machine acts as a login gateway) – (Check out the s/key entry in Wikipedia for lots of links to downloads and explanations of this one-time-password system. OPIE, SecuRemote, and others are all equally worthwhile but s/key is about the cheapest you can get (Free!!! :)

Run “keyinit username” for all user accounts except root and system accounts.
Change all user account login shells to /usr/local/bin/keysh.

to be continued...

, Mike

Linux and Unix Software Security Basics - Part 3

Howdy,

Continuing along, following our first 3 posts on basic Linux and Unix installation security, even more security measures and user, group and account-based security, we're ready to move on to the next section. Today, we'll be focusing more specifically on locking down and/or removing existing software on your newly-installed system.

Again, I hope this series of posts is able to help out, if just a little bit, and I welcome any suggestions for improvement or expansion to include other OS's :)

See you tomorrow for the wrap-up!

Cheers,

OS Installation Guidelines, Part 3: Existing Software Modifications

1. chmod o-w /var/adm/utmp, /var/adm/wtmp, /var/adm/wtmpx, /var/adm/utmpx - any and/or all on Solaris. On RedHat, these files should be in /var/log. If only one of the four files exists, you're okay.

2. touch /var/adm/loginlog.

3. Add “/usr/lib/acct/accton /var/adm/pacct” to /etc/init.d/syslog, if possible considering the caveats in our previous posts. Again, this is only necessary on Solaris. The psacct RPM for RedHat should take care of this for you.

4. chmod 0 /usr/bin/at or /usr/sbin/at.

5. chmod 0 /usr/bin/finger or /usr/sbin/finger.

6. rm /etc/acct/holidays (If this generates errors, just create a file with one line, indicating what year it is).

7. chmod o-rwx /etc/security. On RedHat, or Solaris with PAM enabled, limit access to /etc/pam.d, as well, if possible.

IMPORTANT NOTE: For points 8 through 15, the file locations are general and your particular release of either Solaris of RedHat may place the binaries elsewhere, but you can generally find them in one of the directories we go through in these steps. It should also be noted that, removing the pkg's or RPM's associated with these files should have, theoretically, removed them already, but I don't believe in things working the way they should (a hazard of my profession ;) Obviously, if you're using lp or CUPS, you won't want to delete any of the print commands. For all of these steps, my recommendation is to always ask if you're unsure. If I started a new job and removed all the NIS commands on a box and then found out that's what they used for their directory service... ouch!

8. In /usr/bin, rm if necessary and possible (ask first, if unsure):

cu
disable
enable
tip
uucp
uuencode
uudecode
uuglist
uulog
uuname
uupick
uustat
uuto
uux
cancel
lp
lpget
lpset
lpstat
rsh <--- The remote shell as opposed to restricted sh!
rdate
rdist
rcp
remsh
rlogin
rup
ruptime
rusers
rwho
yppasswd
nispasswd

9. In /etc, rm if possible (ask first, if unsure):

–r lp
–r uucp

10. In /usr/lib, rm if possible (ask first, if unsure):

accept
–r lp
print
reject
lpadmin
lpfilter
lpforms
lpsched
lpshut
lpmove
lpsystem
lpusers
–r uucp
–r nfs
–r netsvc
rsh

11. In /usr/sbin, rm if possible (ask first, if unsure):

lpmove
lpadmin
lpfilter
lpforms
lpshut
lpsystem
lpusers
accept
reject
sadmind

12. For /usr/sbin/in.*, remove all except the following files, if possible (ask first, if unsure):

in.ftpd
in.telnetd
in.rarpd

13. In /usr/sbin/static (on Solaris only), rm if possible (ask first, if unsure):

rcp

14. In /usr/ucb (also only on Solaris), rm if possible (ask first, if unsure):

lpc
lpq
lpr
lprm
lptest
rcp
rdate
rdist
rlogin
rsh
ruptime
rusage
rwho

15. In /var, rm if possible (ask first, if unsure):

–r lp
–r uucp
–r spool/lp
–r spool/uucp
-r spool/print

16. Comment out all of /etc/rpc except the rpcbind line. NOTE: If using Solstice Disk Suite (Volume Manager) software, do not comment out the following lines (on Solaris 9 leave all lines uncommented near the bottom of the file, beginning with the line for metad and ending with the last line of the file). As always, be sure not to comment out any services you really need:

rpc.metad
metad
rpc.metamhd
metamhd

17. chmod –R 0 , if possible, /usr/openwin/bin/* on Solaris and /usr/X11R6 on RedHat.

18. chmod –R 0 , if possible, /usr/dt/bin/* on Solaris. Additional locations of X-Windows binary files might be in many different locations on RedHat. Check out the man pages for your system and Gnome, KDE, etc, to be sure that you're disabling all access to the GUI.

19. All device files have should have good permissions at first. Do not change them unless it's really necessary or you're looking to do enhanced network security of the order we went over in an older post on securing all network programs at once using extended ACL's. That goes a bit beyond the scope of this series ;)

to be continued...

, Mike

Security Basics For Linux and Unix Installations - Part 2

Hey there,

Following up on our first two posts on basic Linux and Unix installation security and even more installation security measures, we're going to move on to the next section. Today, we'll be focusing more specifically on user, and group, accounts, and modifications that we want to make in that arena.

Again, this series of posts is based mostly on Solaris Unix and RedHat Linux, so it can't possibly apply to everything. I welcome any comments regarding it, especially as they pertain to additional security and other Operating Systems. Just to be clear, also, we won't be touching on more advanced stuff like setting up firewalls, etc. The summary of this 5-part post should end up being a bullet list that you can run through and take care of without having to go too crazy ;)

And, of course, my suggestion, to everyone who has the rights and means, is to script out as much of this method as possible. However you end up securifying your system, once you have your routine down, having Jumpstart or Kickstart do it for you from then on out is much less of a hassle (usually :)

See you tomorrow for section 3!

Cheers,

OS Installation Guidelines, Part 2: Users and Accounts

1. Add accounts for all necessary users. Require valid reasons for all accounts created.

2. You can ignore this step if you're using Solaris or RedHat's OpenSSH pkg or rpm, since it will be complete already: Be sure to add a user account for sshd. This will be required for our OpenSSH to run correctly. Should be as simple as

host # useradd –m –d /users/sshd sshd
host # passwd –l sshd <--- (The passwd –l just locks the password so no one can login to the account. This is okay, since the sshd account is only necessary for privilege separation.

IMPORTANT NOTE FOR POINTS 3 THROUGH 6 REGARDING USER REMOVAL: Please note that for point three, we specify "userdel -r," because these users have home directories that are NOT "/" - If there is any doubt in your mind that an account you want to remove has the root filesystem as its home directory, use "userdel" without the -r, as use of that flag may effectively destroy your system. An easy way to find out what a user's home directory is involves simply grepping the username out of /etc/passwd and checking out the second to last field.

3. userdel –r lp, uucp, nuucp, some, all or more if possible (the nobody account should be removed, too, unless it’s used by a web server or some other application). Note that your list will be larger on RedHat since they generally have a user for every service and you should remove every user for every service you're not using if it wasn't removed by rpm -e. Also, on Redhat, be sure to check for matching entries for the user in /etc/group

4. userdel nobody (This should not be removed if it’s used by a web server or some other application)

5. userdel smtp (Non-existent in Solaris9).

6. userdel mail (Non-existent in Solaris9).

7. chmod 700 all user directories.

8. Comment out gsscred and nfsfind from root cron on Solaris. Delete sys, lp, adm, root.au, uucp and all unnecessary crontabs, if possible.

9. Add “0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/lib/sa/sa1” and “59 23 * * * /usr/lib/sa/sa2 -b -g -r –u” to root's cron. On RedHat you'll need to have the sysstat RPM installed in order to use sar. If it "is" installed on Redhat, check out /etc/cron.d to make sure that it is being run in cron and modify the setup per above, to increase or decrease the number of times it runs, if you need to.

10. Create the directory /var/adm/sa (/var/log/sa on RedHat) if it doesn't exist, and chown it to root:yourAdministratorsGroup. Then chmod the directory to 660.

11. Add “/usr/lib/acct/ckpacct” to root's cron on Solaris, if the machine does not have high user activity and has adequate disk space for logging. Remove any other /usr/lib/acct lines in the Solaris root crontab. For RedHat users, make sure that you have the psacct RPM installed. The default settings should be okay and you can use chkconfig or ntsysv to ensure that process accounting is set to run at boot time.

12. Remove user directories for users who no longer have accounts on the machine.

13. Remove crontab entries in /usr/spool/cron/crontabs (Solaris) or /var/spool/cron (RedHat) for users that no longer have accounts on the machine.

14. Add empty .rhosts, .netrc and .exrc files in all user home directories.

15. chmod 0 and chown root all user .rhosts, .netrc and .exrc files.

16. userdel (–r, if their home directory isn’t "/" ) all users, aside from root, with uid 0.

17. Change the group for any user (other than root) that has a gid of 0, if possible.

18. Be sure to include a umask of 022 in any user’s .profile if their account will require their home directory to be somewhat publicly available. Examples of such an accounts include development or group-based accounts. These kind of accounts are used to ease the ability of a group to work together. Of course, the standard 027 umask would make this impossible. Please use discretion and do this only for accounts where the need is real.

19. For all generic accounts, make them su-only. You can find detailed instructions on how to do this, including some pitfalls and stumbling blocks you may encounter, in our post on making generic accounts su only. As luck would have it, it's still aptly named ;)

to be continued...

, Mike

Installation Security Baselines For Linux and Unix - 1b

Hey there,

Following up on yesterday's beginning of our five part post series on basic Linux and Unix installation security, we're going to complete the initial installation guidelines today. I'll dispense with the fancy rhetoric, and if necessary, apologize for the length of yesterday's post (and today's ;). Although this list is, without a doubt in my mind, incomplete in many ways, I hope that some of you out there find it helpful. If even one suggestion gives you a boost, all this typing is worth it :)

I'll see you on the other side, or when part 2 on user account security comes out tomorrow. Hopefully, given the general connotation of the phrase I just used, it will be the latter ;)

Cheers,

For the initial "default value steps," there are many different default script locations (as well as /etc/xinetd.d), where you can set many of these - through step 25 - on Redhat. I think the most important thing to note is that we're hammering down the umask setting and default paths wherever possible.

20. Set the following values in /etc/default/login (leave all else):

PATH=/bin:/usr/bin
SUPATH=/bin:/usr/bin:/sbin:/usr/sbin
TIMEOUT=300
UMASK=027

21. Set the following value in /etc/default/passwd (check your password policy for setting expiration, etc):

PASSLENGTH=8

22. Set the following values in /etc/default/su (leave all else):

CONSOLE=/dev/console
PATH=/usr/bin:/bin
SUPATH=/usr/bin:/bin:/usr/sbin:/sbin

23. Set the following value in /etc/default/sys-suspend (leave all else):

PERMS=-

24. Create /etc/default/telnetd with the following contents, if this is a box that you own:

BANNER=”\nThis computer is the property of YOURCOMPANYNAME HERE.\nUnauthorized use is expressly forbidden. Violators\nwill be prosecuted to the full extent of the law.\n\n”
UMASK=027

25. Create /etc/default/ftpd with the following contents:

BANNER=”Unauthorized access prohibited”. This will basically be obviated by wu-ftpd (and other, more secure ftp servers), but should be kept in case of any situation which requires you to revert back to the original in.ftpd.
UMASK=027

26. Create /etc/issue (along with /etc/release, /etc/redhat.release, /etc/issue.net and any other issue/release files you may have) with the following contents, if this is a box that you own. The return before the closing double quote is necessary on most systems for the output to format correctly and not bleed into whatever comes next:

“This computer is the property of YOURCOMPANYNAME.
Unauthorized access is expressly forbidden. Violators
will be prosecuted to the full extent of the law.
“

27. cp /etc/issue /etc/motd.

28. Run /etc/security/bsmconv on Solaris. To achieve the same effect on RedHat, add the line "audit=1" to your grub.conf or supply it to grub on the command line. Also, for Redhat, this makes step 29 unnecessary)

29. Add all user accounts on Solaris to /etc/security/audit_user with attributes username:lo:no (NOTE – Please be sure to add and remove users from this file henceforth whenever adding or deleting an account on the system!!!)

30. If possible, within system’s application, add empty .rhosts and hosts.equiv in / and /etc respectively, and chmod 0 and chown root the .rhosts and hosts.equiv files.

31. If possible, within system’s application, add empty .netrc and .exrc files in /. chmod 0 and chown root both.

32. If possible, within system’s application, add empty netrc and exrc files in /etc. chmod 0 and chown root both.

33. Edit /etc/init.d/inetsvc to bring up inetd with –st flags (Use -sd for xinetd). Also, add the following code snippet to the end of inetd's, or xinetd's, init script and disable the specific system auditing user's crontab if possible (On RedHat this may end up in the /etc/cron.* directories). Note that for RedHat, we should use /var/log rather than /var/adm:

if [ ! -d "/var/adm/sa" ]
then
        mkdir /var/adm/sa
fi
chown root:itopsgroup /var/adm/sa
chmod 644 /var/adm/sa/*

34. Uncomment “enable-cache hosts no” line in /etc/nscd.conf. Add “enable-cache passwd no” and “enable-cache group no”, also.

35. If not running Solaris 7 or higher, uncomment savecore in /etc/init.d/sysetup. For RedHat, ensure that you have the diskdumputils RPM installed.

36. Create /etc/ftpusers. Include all accounts except administrators, if possible. Definitely include all generic accounts. Otherwise, it should include root and all system accounts. This will basically be obviated by wu-ftpd (and other more advanced ftp software), but should be kept in case of any situation which requires us to revert back to the original in.ftpd.

37. Create /etc/hosts.allow and /etc/hosts.deny files. If possible, limit the hosts that are allowed to connect by subnet or domain (By IP would be great, but probably impractical). Otherwise, create placeholder files to generate tcpd logging (e.g. ALL:ALL). NOTE: Remove X-11 Forwarding if it is present and you can afford to.

38. Set the following parameters in /etc/profile or /etc/bashrc (add or mix around to suit your needs). Many directories listed are specific to Solaris. Your best judgment on what should be in the default PATH, MANPATH and LD_LIBRARY_PATH variables should be enough to guide you on this part. Everyone's setup is probably different:

PATH=$PATH:/bin:/usr/bin:/usr/local/bin:/etc/vx/bin
umask 027
MANPATH=$MANPATH:/usr/man:/usr/local/man:/usr/share/man:/usr/openwin/man:/usr/dt/man:/opt/VRTSvxvm/man
LD_LIBRARY_PATH=/lib:/usr/lib:/usr/local/lib:/usr/share/lib:/usr/openwin/lib:/usr/dt/lib
EDITOR=vi
export PATH MANPATH LD_LIBRARY_PATH EDITOR

39. Add administrators group (groupadd –g 666 itopsgroup).

40. Add administrators to the administrators' group.

41. Add additional interface files, if needed, in /etc (e.g. /etc/hostname.hme1 on Solaris or /etc/sysconfig/network-scripts/ifcfg-NICNAME on RedHat).

42. Add two lines containing “set noexec_user_stack=1” and “set noexec_user_stack_log=1” to /etc/system on systems up to and including Solaris version 9. You can still add this in 10, but in later versions of 9, and in 10, this isn't necessary any more. For RedHat, consider using something like execShield or your alternate preferred method to prevent buffer overflow attacks system-wide.

43. Set hostname to the simple, unqualified hostname, rather than the FQDN, if possible.

to be continued...

, Mike

Basic Installation Security Guidelines For Linux and Unix - Part 1a

Hey there,

Today we're back at the concept-well, and starting out a five part post on setting up a Unix or Linux system securely. This should be an interesting series of posts since I (like most folks, probably) haven't worked with every flavour of Linux or Unix out there, so this set of guidelines is more of a "rough draft" based on experience with Solaris and RedHat servers than an attempt to encompass all areas of system installation, setup and security on every possible OS. Doing so would either be incredibly vague or unbelievably presumptuous. Probably, both ;)

With that in mind, we'll get going with part 1a. Setting up the basic system and keeping it clean from the start. If you're a big fan of numbered lists, you're gonna love this post ;) Hopefully, this post (and the following to come) will generate feedback from users of other systems - or these same systems running different versions - and become a nice broad-but-useful guide to system setup and security!

Best wishes,

OS Installation Guidelines, Part 1a: Basic Setup and System Security Steps (1 of 2... as in too many to fit on one reasonably sized page ;)

1. Create a list of functions the machine will be expected to perform and the services required to fulfill those needs. Also create a map of partitions you expect to use on the disks you have available.

2. Install the basic Operating System. NOTE: If you will be using disk management software (e.g. Solstice Disk Suite, Volume Manger, LVM, Veritas Volume Manager, etc), be sure to read the documentation for these packages before doing your OS install. Most of these types of packages require free partitions to store their disk management data.

3. Download the latest Recommended Patch bundle from sunsolve.sun.com for Solaris machines. Also download the Security Patch bundle if this exists separately. Install both and reboot the machine. For Redhat, do a complete up2date run-and-reboot cycle as many times as necessary.

4. Comment out the following files in /etc/init.d (Some filenames may vary or not exist, depending on what OS you're installing. For an easy way to comment out base init.d files, assuming they're linked to run level directories, check out our old post on editing startup scripts with the vi editor. Don't let the Solaris brand name in the post put you off. It works for any OS on which you can run vi) - Note, also, that this list is pulled from a Solaris box and your security standards may require removing even more. As a general rule, if you don't need it, disable it! RedHat users can do this with chkconfig or ntsysv more easily. Solaris 10 user can make use of the svcs and svcadm commands):

a. acct ( since we prefer to control the process accounting ourselves).
b. asppp ( if we won’t be hooking up a modem to do ppp dialups to the machine).
c. autofs (if automounting not necessary)
d. autoinstall (if machine is not a bootp client)
e. cachefs.daemon (if you will not be using nfs’s cachefs subsystem)
f. cachefs.root (if you will not be using nfs’s cachefs subsystem)
g. cacheos (if you will not be using nfs’s cachefs subsystem)
h. cacheos.finish (if you will not be using nfs’s cachefs subsystem)
i. dhcp (if machine is not a dhcp server or client)
j. dtlogin (if software in use does not require X-windows default login - Removing this will not remove X-windows functionality)
k. init.dmi (if machine will not be availing itself of snmp management)
l. init.snmpdx (if machine will not be availing itself of snmp management)
m. initpcmcia ( if the machine has no pcmcia devices).
n. lp (if print services are not required)
o. mkdtab ( this script runs after installation and is never used again).
p. nfs.client (if the nfs client subsystem will not be required)
q. nfs.server (if the nfs server subsystem will not be required)
r. pcmcia (if machine has no pcmcia devices)
s. power (if machine doesn’t use individual ups power management)
t. rpc (if machine will not be using rpc services) (If the machine will be using rpc services, comment out all of the start section of the script, except for the rpcbind initialization. All other parts of this section initiate nis services, which aren't necessary if you don't use nis :)
u. sendmail (if machine will not need to be a mail receiving agent.)
v. sshd (if you'll be installing you own ssh to maintain flexibility to be more up to date with any security releases).
w. spc ( if print services are not required).
x. uucp ( if you won’t be making use of the uucp subsystem).
y. xntpd (if machine will not serve as a time server or be doing persistent ntp scanning)
z. PRESERVE (this invokes /usr/lib/expreserve, which has a known buffer-overflow bug).
aa. README (read it first, if you like).

5. rm /etc/init.d/*.old

6. Check in /etc/rc0.d, /etc/rc1.d, /etc/rc2.d and /etc/rc3.d (or your system specific run level directories) for any old links to nonexistent files. Delete these.

7. Check in the rc directories, also, for files beginning with X or lowercase s or k. Delete these files as well.

8. If sshd is running, stop it. If you don't plan on using Sun's or RedHat's packaged ssh, we'll be replacing this later. You can leave it if you like.

9. Run /etc/init.d/openssh (or /etc/init.d/sshd) start to start OpenSSH.

10. Connect to the machine via ssh to continue executing the steps in this document. This way you won’t have to worry about poor console terminal emulation creating any accidental errors in the files you will be editing.

11. After installation, ensure that all unnecessary processes have been disabled, using a simple scan of ps output, and disable any that are running and not necessary (sendmail, for instance, is a program that is notorious, on Solaris, for being reactivated as a mail sending agent after patching and/or updating!)

12. Next, be sure to check the network interface. Many of the Sun machines come with all the network ndd settings enabled, which can cause problems with some routers, which are pinned to certain speeds and don’t auto-negotiate. RedHat user's should have been asked about what network settings they wanted to enforce during installation. If not, ethtool can be used to force the required speed/duplex setting Do the following to test:

i. ftp, or scp a file from somewhere, or to somewhere over the network

ii. Run netstat –ian and check the Collisions column. This should still be at 0 for the network interface. If it isn’t, you’ll need to augment the /etc/init.d/nddconfig script, use nddconfig or (for RedHat) use ethtool (and double check the /etc/sysconfig/network and /etc/sysconfig/network-scripts/ifcfg-NICNAME files) to suit the machine and/or environment.

iii. Should you need to fix the interface, so that you get zero collisions (which collisions should, theoretically be impossible on a packet-switched network) and ensure maximum speed without any dropped packets, refer to your system's, or department's documentation. There are really no generalities that can be put to paper here. In my experience, Solaris Gig NIC's usually require auto-negotiation at the switch (for Cisco's) and adv_autoneg_cap set to 1, with all other configurable parameters set to 0, in order for them to work, while old hme interfaces require that the switch be pinned to 100mb and only the adv_100fdx_cap be set to 1 in order for them to function properly. Cut and paste the contents into /etc/init.d/nddconfig after the case/esac clause and before the exit command to ensure the configuration happens at boot time. For RedHat, update the /etc/sysconfig/network and /etc/sysconfig/network-scripts/ifcfg-NICNAME files if necessary). If hme0 gives you a really big headache on Solaris, check our older post on disabling network devices at the PROM level.

13. Comment out every unnecessary line in /etc/inetd.conf (or delete or set to "off" all unnecessary files in /etc/xinetd.d), particularly the unnecessary r-services, and ensure that all services, where applicable, are managed through tcpd – ensure that telnetd is enabled as a backup during install (but, only if you plan on using a custom SSH later, and then just in case)

Ex: ftp with: /usr/local/encap/tcp-wrappers/bin/tcpd and /usr/sbin/in.ftpd –d as the file and args.

14. Add loghost to /etc/syslog.conf: (NOTE: It is imperative that a tab, and not a space, separate the list of facilities-severities and the hostname of the remote logging host or target file!) Also Note that, at this point, if you do NOT have a syslogging server dedicated and set up, you do NOT have to complete this step (not at all, if you don't plan on using one):

*.err;*.emerg;*.crit;*.alert;mail.debug;kern.notice;auth.notice;auth.info @loghost.xyz.com
*.err;kern.debug;daemon.notice;mail.crit;auth.notice;auth.info /var/adm/messages

Also be sure to change any references to /var/console to /var/adm/console on Solaris or /dev/console on RedHat.

15. Configure sendmail to use its relay properly, if you are going to use it to send mail out. If the machine is an email server; that’s another story. NOTE: If running sendmail v8.8.8 or earlier, be sure to reverse the ruleset for S94 to ensure proper masquerading of envelope addresses from the host, if you have that feature enabled!

16. Set up networking and name services minimally:

a. Create /etc/hosts with loopback, host, logging host , default router, nameserver, and mail host information if applicable. (Note: When entering host information, ensure that the first entry in the host field is the fully qualified domain name. This is a pet peeve of mine. You can do the simple hostname first and then the FQDN. Preferably, list them both for each IP.)

b. Create and populate /etc/defaultrouter or set the DEFAULT_GATEWAY value in either of the /etc/sysconfig/network and /etc/sysconfig/network-scripts/ifcfg-NICNAME files on RedHat.

c. Edit /etc/nsswitch.conf to use "dns [ NOTFOUND="continue"]" for hosts. If you don't set NOTFOUND, the default value is "return" and this can cause issues with telnet and ftp hostname lookups, etc.

d. Create and populate /etc/resolv.conf.

17. If host is multi-homed, disable IP-forwarding by creating /etc/notrouter on Solaris. Do "echo 0 > /proc/sys/net/ipv4/ip_forward" on RedHat to disable packet routing/forwarding.

18. If possible, trim /etc/mail/aliases to include at least (and, maybe, at most) the following (be sure to execute newaliases if any changes are made):

Postmaster: root
MAILER-DAEMON: root
nobody: /dev/null
root: itopsgroup@xyz.com
itopsgroup: itopsgroup@xyz.com

19. Create /etc/shells with these contents (remove any shells you don't want to be run on your system and add any, that I forgot to include, that you do!):

/sbin/sh
/bin/ash
/bin/sh
/bin/csh
/bin/ksh
/usr/bin/ksh
/bin/false
/bin/bash
/bin/zsh
/usr/local/bin/keysh (Only include if using keysh on system)

To be continued...

, Mike

Pinging And Checking Port Status With Perl CGI On Linux And Unix

Greetings,

To change things up a bit today, we're going to go back to some Perl scripting. It seems like it's been a while, but that may just be my distorted sense of space and time. In any event, since this blog covers many many things related to Linux and Unix (which my probable life-span wouldn't allow me to script out or write about ;), today seems a good a day as any to get back to putting out some script work.

Today's Perl script is very simply written and, although a bit lengthy, fairly limited in what it does. Of course, it should also be fairly simple to expand upon and make do much more work than would normally be commensurate with a general breakdown of the keystroke/output-usefulness ratio.

This script closely echoes previous scripts we put out to check on web server status and check on network server port-health insofar as the end result is concerned. It should run fairly simply, too (you'll probably just need to change the target host, target port and, possibly, the location of the ping command, and its arguments, to suit your taste - or have those all fed to the script from the command line using the @ARGV array):

host # ./portpinger.pl

This version, however, is a bit more complex (or convoluted, depending on how you look at it ;) to highlight a few other concept-based posts that we've put out in the interim. For instance, this Perl script (while it's not absolutely necessary, given the abundance of variable names we could have used to get-around) makes use of variable scoping within subroutines. This is something, actually, that we're building toward in our ever-expanding series on porting code between shell, Perl and awk. And the final thing we highlight, somewhat, in this particular script (that my green-screen-addled brain can still discern ;) is signal trapping and handling with Perl.

Whether or not you have any use for it, I hope you can find something in its over-production that sparks some interest or gets you thinking more about the many different ways you can use Perl to do many different things. Basically, this script does a ping, a port check and then puts up a CGI web page. But, sometimes, the lessons (good or bad) are found more in the context than in the message :)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# portpinger.pl - Ping a Port and Check Another One Just for kicks.
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

use Socket;
use CGI;

$socketpinger = new CGI;
$pingee = "www.xyz.com";
system("ping $pingee 1 1 2>/dev/null 1>/dev/null");
$pingyn = $? >> 8;

print $socketpinger->header();
print "<html><head><title>Caps Checker</title></head>\n";
print "<body>\n";
if ( $pingyn ) {
print "<center><h3>Result of ping to $pingee:</3></center> <center><h2>N
o Answer</h2></center>\n";
} else {
print "<center><h3>Result of ping to $pingee:</h3></center> <center><h2>
$pingee is alive!</h2></center>\n";
}

print "<center><h3>Result of tcp connect to port 443:</h3></center>\n";
&checkhost($pingee,443);

print $socketpinger->end_html();
sub Timer { die "Alarm Clock\n"; }

sub checkhost {
local($host,$port) = @_;
local($t,$cnt,@var,$ip,$down);
undef @fdata;
$AF_INET=2; $PF_INET=$AF_INET; $SOCK_STREAM=1; $IPPROTO_TCP=6;
$sockaddr = 'S n a4 x8';
($t,$t,$t,$t,@var) = gethostbyname($host);
$ip = $var[0];

$down = 0;
$this = pack($sockaddr, $AF_INET, 0, "\0\0\0\0");
$serveraddr = pack($sockaddr, $AF_INET, $port, $ip);
eval 'socket(RS, $PF_INET, $SOCK_STREAM, $IPPROTO_TCP)|| die "socket: $!"';
if ($@) {
$SOCK_STREAM=2;
socket(RS, $PF_INET, $SOCK_STREAM, $IPPROTO_TCP) || die print "socket: $
!";
}
bind(RS, $this) || ($down = 1);
if ($down) {
print "<center><h2>$host at port $port is down.</h2></center>\n";
shutdown(RS,2);
close(RS);
return;
}
$SIG{'ALRM'} = 'Timer';
eval {
alarm(5);
connect(RS, $serveraddr) || die ($down = 1);
};
alarm(0);
if ($down || $@ =~ /Alarm Clock/) {
print "<center><h2>$host at port $port is down.</h2></center>\n";
shutdown(RS,2);
close(RS);
return;
}
$up[$num] = 1;
print "<center><h2>Connection to $host at port $port successful.</h2></cente
r>\n";
shutdown(RS,2);
close(RS);
}

, Mike