Manually Editing The Main VCS Configuration File The Right Way

Hey again,

If you're like me and work on all sorts of things in the course of your sysadmin duties, working with Veritas Cluster Server has its pros and cons. On the one hand, it's great once it's all set up and working correctly, and troubleshooting it is fairly easy if you're working off of a solid foundation. On the other hand, making simple additions to the configuration file can be a lengthy and confusing ordeal.

Here's a little trick I picked up along the way, in my experience working with VCS. It's not exactly kosher (if you like to follow rules to the letter), but it works and it's 99% legitimate :)

Let's say, for instance, you wanted to add an NFS mount resource to your VCS cluster configuration. For ease of example, we'll assume you either know the syntax of the main configuration file (main.cf) or can find another cluster setup somewhere else on your network that you can "borrow" from (always more convenient ;) and modify.

Note that this entire process will keep VCS happy and won't cause any sort of split-brain condition that might result from your just manually editing the configuration file and rebooting your server (If things are working correctly, VCS will probably just overwrite your changes if you do this, anyway)

The first thing you'll want to do is to open the main.cf file for editing in memory. You can do this with the haconf command, like so:

host # haconf -makerw

Now, copy off ( just in case ) and open the main.cf file (Generally located in the /etc/VRTSvcs/conf/config directory) in your favorite editor and paste that other configuration addition to the bottom of your config file. We'll assume it's something simple, like this (Note that I haven't included the entire entry for brevity's sake and that you'll also not want to include the "dependency tree" lines. The ones that start with "//" - These will be created for you during this process):

group nfs_sg (
 SystemList = { host1 = 0, host2 = 1 }
 AutoStartList = { host1, host2 }
 )

 DiskGroup nfsdg (
  DiskGroup = nfsdg
  )
...

Now, you'll want to modify the parts that need modifying to suit your particular setup and move your augmented main.cf to another directory:

host # cp /etc/VRTSvcs/conf/config/main.cf /temporary/directory/main.cf

and verify that it's okay (Technically, you could still do this while it's in the regular directory) by running hacf:

host # hacf -verify /temporary/directory <--- This command will assume that there is a main.cf file in the directory you specify

Then, and this is the part that takes the most work out of your job, use hacf to create a commands file (These are commands you would normally have to type at the command line. There are so many of them involved in almost any addition, that I always update this way, just so there's no way I can forget one!):

host # hacf -cftocmd /temporary/directory

This will create a file called main.cmd in the /temporary/directory. This is chock full of commands that you would normally have to execute in order to augment your VCS configuration.

After backing up the main.cmd file in /etc/VRTSvcs/conf/config to, say, main.cmd.bak (Note: Never use the suffix .previous as this is the VCS default and your backup may get overwritten if you run subsequent haconf commands!), move your new main.cmd command into the VCS configuration directory, like so:

host # cp /temporary/directory/main.cmd /etc/VRTSvcs/conf/config/main.cmd

Now, instead of using the "hacf -cmdtocf" command (which should copy the main.cmd file to your configuration, but doesn't always work correctly), do this instead: Simply execute the main.cmd file you just created:

host # sh /etc/VRTSvcs/conf/config/main.cmd

Note: You're going to get a million errors flying across the screen. These are all warning you that you are attempting to add or modify a resource that already exists or has the property you're attempting to change it to. Don't worry about these at all. The errors result in the commands not being executed. Running the entire main.cmd file, and just ignoring those errors, ensures that every "new" command you need to enter will be entered :)

Now that you're all done "running all those commands" you can close the main.cf file in memory the proper way with haconf again, like this:

host # haconf -dump -makero

Now your new resource will have been added to the main configuration file, VCS will be happy and propagate the change to all the nodes in your cluster and you should be able to begin monitoring your new resource and, possibly, troubleshooting any errors that occur if you got any little part of your new entry wrong. Fun ;)

Best wishes,

, Mike

linux unix internet technology

Bit Shifting In Perl To Get Errno Values From The System Function

Hey there,

My two main reasons for putting together this little Perl script are:

1. To account for the unlikely event when mail queue monitoring commands don't work correctly, or you suspect that your system is so munged up that your OS commands aren't giving you correct information. Most Linux and Unix setups are far more robust than Windows, etc, and can take a severe beating before they go overboard. But, just like anything else, if pushed way too hard, they'll all eventually crack. Common symptoms of a total meltdown include obviously incorrect memory, swap and cpu statistics.

2. To demonstrate how to get back the actual errno result from a system command in Perl without using backticks.

This Perl script can, basically, be used on any directory and I set it up so that it takes a directory name as its only argument (so that it will work with any mail software no matter where it keeps its mail queue). You can call it simply, like this:

host # ./mailkue.sh /var/spool/clientmqueue <--- Substitute your mail queue location

We're making use of Perl's stat() and localtime() functions to parse a time-sorted directory listing. This should help you determine if there really are tons of old expired emails in your mail queue that you may or may not want to delete.

Also, probably more notably, we're using Perl's bit shift operator to extract the correct errno value from the system function. Normally, the system function only returns a scalar value indicating whether or not it executed the subshell sucessfully. If you run a program within the system function, like so:

host # $return = system("touch a");

The return value from that system command would be the numerical value of the success or failure of system's ability to invoke the shell.

If you did the same thing with backticks, your return value would be the output from that command. So you could do something as easy as this to get the real errno value:

host # $return = `touch a;echo $?`;

If you're not interested in the output, as we're clearly not in the above examples, but just want the proper return value from the command executed by the system function (as opposed to the return value of the system function... getting confusing and convoluted, here ;), you can get that by changing the above example, like so:

host # system("touch a");
host # $return = $? >> 8

And you'll know whether or not your touch was successful, not just if the system function worked. And it's guaranteed to work on Linux, Unix or any *nix system that Perl will run on.

Of course, backticks are generally simpler to use, but, under certain circumstances the above trick can come in very handy. Here's hoping it helps you out with your Perl scripting in the future!

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# mailkue.sh - check your mail
# queue - or anywhere else :)
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#   

if ( $#ARGV < 0 )       {
        print "Usage: $0 directory\n";
        exit(1);
}

%months = qw(0 Jan 1 Feb 2 Mar 3 Apr 4 May 5 Jun 6 Jul 7 Aug 8 Sep 9 Oct 10 Nov 11 Dec);
chdir $ARGV[0];
system("ls -t >/dev/null 2>&1");
$status = $? >> 8;
if ( ! $status ) {
 @nfiles = `ls -t`;
 foreach $nfile (@nfiles) {
  chomp($nfile);
  $count = 1;
  print "FILE: $nfile\n";
  @files = `/usr/bin/find $ARGV[0]/$nfile -print|xargs -I {} ls -d1 "{}"`;
  foreach $file (@files) { 
   chomp($file);
   ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime,$blksize,$blocks) = stat($file);
   ($csec,$cmin,$chour,$cmday,$cmon,$cyear,$cwday,$cyday,$cisdst) = localtime($mtime);
   $year = $cyear + 1900;
   printf("%-2d:  %-45s\t%10d  %-3s %2d, %-4d\n", $count, $file, $size, $months{$cmon}, $cmday, $year);
   $count++;
  }
 }  
} else {
 print "No mail in queue\n";
}

, Mike

linux unix internet technology

Summary Post Regarding Sun 6800/6900 Server Component Management

Hey There,

I'm putting this "lazy Sunday" post out to tie up all of our previous posts regarding dealing with board, power, and component management on Sun 6800/6900 Servers.

This post truly deserves the designation of a "lazy Sunday" post because it takes care of two things at once:

1. It lays down a roadmap of all of our previous posts on this topic, in order, in one easy to find location.
2. It allows me to enjoy my Sunday (insofar as that's possible, after working a weekend Friday/night/Saturday shift. We all have to sleep sometime, and I have a Sunday very-early-morning job still coming up ;)

Initially, take a look at our post on shutting down domains on 6800/6900 servers. It's important to read this one first, as the next two entries, although more specific, assume some of this information.

Then, follow up with our post on installing system boards on 6800/6900 servers.

Finally, hit yesterday's post on moving boards between domains on 6800/6900 servers and everything's wrapped up with a bow :)

Hope you're all enjoying a restful Sunday :)

, Mike

linux unix internet technology

Moving Boards Between Domains On Sun 6800/6900 Servers

Hey There,

For some light weekend fare. I thought I'd put together this quick little tutorial on the simple task of moving boards (CPU, IO) between domains on Sun 6800/6900 servers. For our purposes, we'll assume that the boards are already installed on the physical system and that we want to move one CPU and one I/O board from Domain C to Domain A.

The first thing you want to do is connect to the System Controller and then attach to the Platform Shell. This is basically necessary since you'll be working on two separate Domains. The Platform Shell will allow you to connect to each and go back and forth if you need to.

System Controller 'host-sc0':

    Type  0  for Platform Shell

    Type  1  for domain A console
    Type  2  for domain B console
    Type  3  for domain C console
    Type  4  for domain D console

    Input: 0

Platform Shell

host-sc0:SC>

The first thing you'll want to do is show the board assignments, to make sure you're moving the right devices. You can do this simply with the showboards command below (output trimmed):

host-sc0:SC> showboards

Slot     Pwr Component Type                 State      Status     Domain
----     --- --------------                 -----      ------     ------
/N0/SB0  Off CPU Board                      Assigned   Not tested C
/N0/SB1  Off CPU Board                      Assigned   Not tested C
/N0/SB2  Off CPU Board                      Assigned   Not tested A
/N0/SB3  Off CPU Board                      Assigned   Not tested A
/N0/SB4  Off CPU Board                      Assigned   Not tested C
/N0/SB5  Off CPU Board                      Assigned   Not tested C
/N0/IB6  Off PCI I/O Board                  Assigned   Not tested C
/N0/IB7  Off PCI I/O Board                  Assigned   Not tested C
/N0/IB8  Off PCI I/O Board                  Assigned   Not tested C
/N0/IB9  Off PCI I/O Board                  Assigned   Not tested A

Now assuming we wanted to move IO Board 7 and CPU Board 1 from Domain C to Domain A, you can do so, also simply, using the deleteboard and addboard commands:

host-sc0:SC> deleteboard IB7
host-sc0:SC> deleteboard SB1
host-sc0:SC> addboard -d A IB7
host-sc0:SC> addboard -d A SB1

Then just double check :)

host-sc0:SC> showboards

Slot     Pwr Component Type                 State      Status     Domain
----     --- --------------                 -----      ------     ------
/N0/SB0  Off CPU Board                      Assigned   Not tested C
/N0/SB1  Off CPU Board                      Assigned   Not tested A
/N0/SB2  Off CPU Board                      Assigned   Not tested A
/N0/SB3  Off CPU Board                      Assigned   Not tested A
/N0/SB4  Off CPU Board                      Assigned   Not tested C
/N0/SB5  Off CPU Board                      Assigned   Not tested C
/N0/IB6  Off PCI I/O Board                  Assigned   Not tested C
/N0/IB7  Off PCI I/O Board                  Assigned   Not tested A
/N0/IB8  Off PCI I/O Board                  Assigned   Not tested C
IB9      Off PCI I/O Board                  Available  Not tested A

And repeat as necessary. When you boot up the separate domains (The logical machines) you'll enjoy the benefit of the new hardware. Easy peasy :)

Cheers,

, Mike

linux unix internet technology

Simple Linux Mods For Shell Script To Disable User Accounts

Hey there,

Yesterday, we posted a huge chunk of code getting the latest version our script to disable or delete user accounts. So, in the interest of keeping this page from becoming way too long, I'm just going to post the "diff" today. I quote the term diff, because I'm using it very loosely ;)

For the astute reader, I also noted a few missing single ticks on two lines in yesterday's script and have modified that post. Strangely enough, after repeated testing to make sure I was still all there, Solaris Unix doesn't seem to care (???) RedHat Linux picked it up right away though!

The modifications necessary to make the manual part of this version of our user disabling/deleting shell script are actually quite refreshing. While Solaris still insists that you unpack the utmp struct in /var/adm/wtmpx in order to get any year information from login monitoring commands, Linux provides this information for you with the "lastlog" command.

Yes, the modification is really that simple :)

So, without further ado, I've laid out the chunks of old code from yesterday, followed by the uncommented code that you should replace it with when running this on Linux. I put the two different places you'll need to make a change, in order, from top to bottom, as that seems normal to me ;)

You'll notice that the script's output is slightly different (and actually more conversational in tone), but, for our purposes, it doesn't need to be modified. We're still at the stage where we want to do a visual double-check.

Enjoy,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

*** This section was in yesterday's script, but I've
*** changed the RedHat variable to be the lastlog
*** program instead of the /var/log/wtmp file

# $wtmpx_file_loc = "/var/adm/wtmpx"; # For Solaris
$wtmpx_file_loc = "/usr/bin/lastlog"; # For RedHat

*** The only other difference is a big chunk of code, but
*** actually a much simpler solution to implement in Linux

*** The Old Solaris Code - You should delete, or comment
* out this entire section and replace it with the new
* code below on Linux systems
*
* $template = "A32 A4 A32 l s s2 x2 l2 l x20 s A257 x";
* $recordsize = length(pack($template,(  )));
* open(WTMP,$wtmpx_file_loc) or die "Unable to open wtmpx:$!\n";
* while (read(WTMP,$record,$recordsize)) {
* if ( $wtmpx_counter == 1 )     {
*        print ".";
*        $wtmpx_counter = 500;
*        $wtmpx_total++;
* }
*        ($ut_user,$ut_id,$ut_line,$ut_pid,$ut_type,$ut_e_termination,
*        $ut_e_exit,$tv_sec,$tv_usec,$ut_session,$ut_syslen,$ut_host)=
*        unpack($template,$record);
*        push (@wtmpx_uv, "$ut_user ".scalar localtime($tv_sec));
*        $wtmpx_counter--;
*        $wtmpx_total++;
* }
*
*** End of the old Solaris code you should comment out or delete

* The substitute code for your Linux Shell Script
*
open(WTMP, "lastlog|") or die "can't open $wtmpx_file_loc: $!";
@rhwtmp = <WTMP>;
close(WTMP);
foreach $rhwtmp (@rhwtmp)       {
        if ( $wtmpx_counter == 1 )      {
                print ".";
                $wtmpx_counter = 500;
                $wtmpx_total++;
        }
        push (@wtmpx_uv, $rhwtmp);
        $wtmpx_counter--;
        $wtmpx_total++;
}
*
* End of the substitute code - a lot easier to understand.

, Mike

linux unix internet technology

Shell Script To Disable and Delete User Accounts - Follow Up

Hey there,

This is a long overdue follow up to an older post where we began walking through the process of developing a decent and useful shell script, from scratch, to identify user accounts to disable or terminate based on the amount of time passed since they last logged in. Our goal was to make it simple to use and workable on RedHat Linux and Solaris Unix.

The rules are still the same: 45 days without a login equals account password locking and 90 days without a login equals account deletion. All actions (the password locking and account deletion) are still being kept manual, since the script isn't finished yet. You could argue that, no matter how good the script gets, that kind of stuff should always be kept manual, just in case. Don't want to delete the wrong person's account ;)

In this update, we've integrated another cross-check, utilizing Perl to get relevant information from the system wtmpx file (Referenced in greater detail in this post on taking a look inside wtmpx). We'll need to use this to provide a comparison against the modification time checks we're already doing on the users' home directories.

Important Note: Since this script is for both Linux and Unix, I made the location of the wtmpx file a variable, since it resides in /var/adm on Solaris and /var/log on RedHat Linux. The variable (comment out the one you need to) $wtmpx_file_loc has been added to address that issue. There are still other issues with Linux's implementation of wtmp that we will look at tomorrow (So that this script will run correctly on that OS. Please, no complaints until then ;)

Neither check is guaranteed to give a correct result, but utilizing both of them will quickly point out accounts flagged for deletion or lockdown that actually shouldn't be! In certain circumstances, both checks will fail to give entirely definitive results (for instance, if a user's home directory is over 90 days old and he/she doesn't have any entries in wtmpx, it's possible that we'd be deleting or locking the account in error, but much less likely).

Note that this is still a work in progress since, right now, we've got it printing out information to the screen for manual verification. In our next update, we'll take the wtmpx output (crucial to use Perl for this because we need the "year" and commands like "last" and "logins" don't provide that information) and convert it, so that we can have the "time-since-last-login" it provides to directly cross-reference against the filesystem checks inside the script; thus removing that manual part of the verification.

We've also made the user notifications a bit more warm and cuddly. Always in the best interest of everyone to be courteous when we can :)


Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# accountminder.pl
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

$hostname=`hostname`;
$wtmpx_counter = 500;
$wtmpx_total = 0;
$wtmpx_file_loc = "/var/adm/wtmpx"; # For Solaris
# $wtmpx_file_loc = "/var/log/wtmp"; # For RedHat
print "PROCESSING WTMPX FILE FOR CROSS-CHECKING: ";
$| = 1;
$template = "A32 A4 A32 l s s2 x2 l2 l x20 s A257 x";
$recordsize = length(pack($template,(  )));
open(WTMP,$wtmpx_file_loc) or die "Unable to open wtmpx:$!\n";
while (read(WTMP,$record,$recordsize)) {
if ( $wtmpx_counter == 1 ) {
 print ".";
 $wtmpx_counter = 500;
 $wtmpx_total++;
}
 ($ut_user,$ut_id,$ut_line,$ut_pid,$ut_type,$ut_e_termination,
 $ut_e_exit,$tv_sec,$tv_usec,$ut_session,$ut_syslen,$ut_host)=
 unpack($template,$record);
 push (@wtmpx_uv, "$ut_user ".scalar localtime($tv_sec));
 $wtmpx_counter--;
 $wtmpx_total++;
}
close(WTMP);
print ".\n";
print "PROCESSED $wtmpx_total ENTRIES\n";

$today=time();

@no_shells=`egrep -i 'nologin|false' /etc/passwd|awk -F":" '{print \$1}'`;
@in_admin=`grep -w 99999 /etc/passwd|awk -F":" '{print \$1}'`;
@user_names=`awk -F":" '{print \$1}' /etc/passwd`;
open(LOCKEDTEXT, "<locked.txt");
@locked_text = <LOCKEDTEXT>;
close(LOCKEDTEST);
print "\nRemoving LOCKED ACCOUNTS ";
foreach $locked_user (@user_names) {
 print ".";
 chomp($locked_user);
 if ( grep /$locked_user/,@locked_text ) {
  push(@tmp_user_names,$locked_user);
  print "$locked_user ";
 }
}
print "\n";
print "\nRemoving NO_SHELL ACCOUNTS ";
foreach $locked_user (@user_names) {
 print ".";
 chomp($locked_user);
 if ( grep /$locked_user/,@no_shells ) {
  push(@tmp_user_names,$locked_user);
  print "$locked_user ";
 }
}
print "\n";
print "\nRemoving ADMINISTRATIVE ACCOUNTS ";
foreach $locked_user (@user_names) {
 print ".";
 chomp($locked_user);
 if ( grep /$locked_user/,@in_admin ) {
  push(@tmp_user_names,$locked_user);
  print "$locked_user ";
 }
}
print "\n\n";
@total = @both = @unique_names = ();
%count = ();
foreach $possible_dupe (@user_names, @tmp_user_names) {
 $count{$possible_dupe}++
}
foreach $possible_dupe (keys %count) {
 push @total, $possible_dupe;
 push @{ $count{$possible_dupe} > 1 ? \@both : \@unique_names }, $possible_dupe;
}
@user_names = @unique_names;
foreach $diruser (@user_names) {
 $filer = `grep $diruser /etc/passwd|awk -F":" '{print \$6}'`;
 push(@user_dirs,$filer);
}
print "FIRST RUN - Checking Home Directories...\n\n";
foreach $history_file (@user_dirs)      {
        chomp($history_file);
 chomp($user_name = $user_names[$counter]);
        print "${user_name}: ";
 $counter++;
        if ( -d "$history_file" )   {
                @file_dates=stat("$history_file");
                $file_date=$file_dates[9];
        }   else    {
                $file_date=0;
        }
        $difference=$today - $file_date;
        if ( $difference >= 3888000 && $difference < 7776000 )  {
#                print "45 days old or older, but under 90\n";
#                print "passwd -l would be executed\n";
          $| = 1;
                print "Lock Em Up!\n";
  print "Verifying Last Login: ";
  $wtmpx_user = grep(/$user_name/, @wtmpx_uv);
  if ( $wtmpx_user == 0 ) {
   print "No Entries in $wtmpx_file_loc\n";
  } else {
   @wtmpx_user = grep(/$user_name/, @wtmpx_uv);
   $verified = $wtmpx_user[$#wtmpx_user];
   print "$verified\n";
  }
  push(@broken, $user_name);
  open(SENDMAIL, "|/usr/lib/sendmail -t");
  print SENDMAIL "From: System Administration\n";
  print SENDMAIL "Reply-To: sec_account\@xyz.com\n";
  print SENDMAIL "To: $user_name@\xyz.com\n";
  print SENDMAIL "Subject: $user_name Account LockDown Notice\n";
  print SENDMAIL "$user_name,\n";
  print SENDMAIL "\n";
  print SENDMAIL "\tWe found that you haven't logged into your account on unix server $hostname within the last 45 days.\n";
  print SENDMAIL "\n";
  print SENDMAIL "\tYour account has been locked.  If you still require this access, please have your unix password reset.\n";
  print SENDMAIL "\n";
  print SENDMAIL "\tIf your account remains dormant for 90 days, your account will be deleted.\n";
  print SENDMAIL "\n";
  print SENDMAIL "Thanks,\n";
  print SENDMAIL "\tSystem Administration\n";
  close(SENDMAIL);
        }       elsif ( $difference >= 7776000 )        {
  push(@buckwheats, $user_name);
  $| = 1;
#  print "User needs to be deleted - ";
#  print "90 days old or older\n";
  print "Delete Em!\n";
  print "Verifying Last Login: ";
  $wtmpx_user = grep(/$user_name/, @wtmpx_uv);
  if ( $wtmpx_user == 0 ) {
   print "No Entries in $wtmpx_file_loc\n";
  } else {
   @wtmpx_user = grep(/$user_name/, @wtmpx_uv);
   $verified = $wtmpx_user[$#wtmpx_user];
   print "$verified\n";
  }
  open(SENDMAIL, "|/usr/lib/sendmail -t");
  print SENDMAIL "From: System Administration\n";
  print SENDMAIL "Reply-To: sec_account\@xyz.com\n";
  print SENDMAIL "To: $user_name@\xyz.com\n";
  print SENDMAIL "Subject: $user_name Account Termination Notice\n";
  print SENDMAIL "$user_name,\n";
  print SENDMAIL "\n";
  print SENDMAIL "\tWe found that you haven't logged into your account on unix server $hostname within the last 90 days.\n";
  print SENDMAIL "\n";
  print SENDMAIL "\tYour account has been terminated.  If you still require this access, please send email to the system admin team.\n";
  print SENDMAIL "\n";
  print SENDMAIL "Thanks,\n";
  print SENDMAIL "\tSystem Administration\n";
  close(SENDMAIL);
        }       else    {
                print "OK\n";
        }
}

, Mike

linux unix internet technology

C Program For Linux Or Unix To Unravel Symlinks

Good morning/afternoon/evening :)

Today, I put together a quick C program (Tested to compile with gcc on Linux and Unix - Note that your "include" files might need to be slightly modified depending on where they are and what flavor of Unix or Linux you're using) to make it easier to solve a common issue.

Depending upon how things are setup where you work and/or play, this c program may be immensely helpful for you. The opposite, of course, may be true ;)

This program makes use of the c "realpath" function, which (put simply) will give you the "actual" absolute path or location of any file or directory name that you feed it. This is only highly useful, again, if you work somewhere were people go nuts using symlinks all over the place on your hosts' filesystems; leaving you with tons of relative path file locations.

You can compile this code simply with gcc, by just putting the contents into a file (I'll call it realpath.c) and running a simple compile command, like so:

host # gcc -o realpath realpath.c

this will output a binary called "realpath" that you can use like this (We'll show, below, the two ways feeding this program an existing file can turn out):

host # ./realpath /usr/local/bin/program
/usr/local/bin/program is the real pathname.

host # ./realpath /usr/local/bin/otherprogram
/usr/local/bin/program is a linked pathname for /apps/local/ops/sbin/otherprogram

using this simple program can save you considerable heartburn under the right circumstances. Here's to your not having to find yourself in the "right circumstances" too often ;)

Best Wishes,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

# include <errno.h>
# include <stdio.h>
# include <stdlib.h>
# include <string.h>
# include <sys/param.h>
# include <sys/stat.h>
# include <sys/types.h>

/* realpath - weed out those symlinks

 2008 - Mike Golvach - eggi@comcast.net

 Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
*/

main(int argc, char **argv)
{

char *real;
const char bone[1024];
int diff, bob;
struct stat ruthere;

if ( argc != 2 ) {
 printf("Usage: %s [pathname in question]\n", argv[0]);
 exit(1);
 }

if ((realpath(argv[1], real)) < 0) {
 printf("%s : File Not Found\n", argv[1]);
 exit(1);
 }

if ((stat(argv[1], &ruthere)) < 0 ) {
 bob = errno;
 if ( bob == 2 ) {
  printf("%s : File Not Found\n", argv[1]);
  exit(1);
  } else if ( bob == 13 || bob == 20 ) {
  printf("\n");
  printf("You can't access this file!\n");
  printf("Results may be incorrect...\n");
  } 
 }

if ((diff = strcmp(argv[1], real)) == 0) {
 printf("\n");
 printf("%s is the real pathname.\n", argv[1]);
 } else {
 printf("\n");
 printf("%s is a linked pathname for %s\n", argv[1], real);
 }

}

, Mike

linux unix internet technology

Shell Script To Augment Antivirus Software On Unix and Linux Mail Servers

Hello again,

Today's offering is, as usual, borne of personal experience. I first found this method of helping combat virus emails back in the "I Love You" days, when vbscript was automatically opened by all MS products and allowed sufficient access to the Windows system Registry to enable anyone with moderate scripting abilities (or the manual dexterity to cut and paste) to completely destroy other users' systems via embedded (or even attached) scripts. Thank God they've gone out of their way over the past decade to make sure that won't ever happen again (heavy sarcasm - I'm not sure what the proper emoticon for that would be ;)

This shell script is a bit of a "shell" in and of itself, since it reflects a specific setup and probably won't exactly match anyone else's without a bit of modifying. It's one way I found to help out a flood of virus emails on any Linux or Unix host that acts as a mail server (sendmail, qmail, what have you). If you still store your email locally on a Linux or Unix host and have your users access that via POP of IMAP, this can help you clean out virii without having to go near anyone's personal computer.

This script should, optimally, be run as a cron job (once a minute perhaps), but if the situation is desperate, it can also be run in a command line "while loop" to make sure it never takes a break. That can be accomplished like so (We'd also, under this circumstance, recommend running it so that it won't stop running if you get disconnected from your terminal - You can use nohup or check this previous post regarding what to do if nohup hangs up anyway for alternatives, like running the command backgrounded in a subshell) :

host # while :; do /full/path/to/viruscleaner.sh;sleep 5;done

The mechanics of the script are fairly crude, as this is not meant to be a substitute for adequate virus protection on Windows machines, nor is it meant to outperform existing Linux and Unix antivirus products. This shell script is presented as an augmentation to the above mentioned safeguards and can help make them more efficient by taking some of the work away from them.

As an example, one of the simple things this script does to help combat virii waiting in your mail queue is to simply change extensions on attachments to mails that match your search criteria. Rather than delete, or quarantine, a potentially viral email, simply changing the extension of the attachment (From, say, .vbs to .whyInTheWorldWouldYouEverOpenThis ;) can make a huge difference. When someone receives an email with an unknown file type extension attached, they will (at the very least) be prompted by Windows as to whether or not they wish to open it. If you name your new extensions ridiculously enough, they should also be forced to pick a program to open the attachment with. You have every right to be completely baffled if one of your users goes through the double hassle of opening up an email attachment called ClickHere.whyInTheWorldWouldYouEverOpenThis ;) That's why another thing the script tries to do is rewrite vbs code so it won't execute properly.

Extensible possibilities for this script are pretty much endless, since you can do anything to the contents of a mail file on a Linux or Unix system. For instance, if the attachment was uuencoded, you could change the name and then randomly substitute characters in the uuencoded attachment so that it would never be able to be decoded.

Hope you all have fun with this and it saves you some sweat :)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

#
# viruscleaner.sh
# Check for viral attachments in
# your email work directory
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

cd /var/mail
while :
do
 for x in `ls -1d *`
 do
 egrep -il "Subject: Resume"\|"Subject: I *LOVE *YOU"\|"Subject: fw"\|"Subject: Susitikim"\|"Subject: This *Is *Funny"\|"Subject: Mother"\|"Subject: joke" $x >/dev/null 2>&1
 if [ $? -eq 0 ]
 then
  /usr/local/bin/perl -p -i -e 's/HKEY.*/ifthenfor else #/' $x
  /usr/local/bin/perl -p -i -e 's/begin.*//' $x
  /usr/local/bin/perl -p -i -e 's/(ap\.wri|c.Copy|regget|regcreate|On Error).*//i' $x
  /usr/local/bin/perl -p -i -e 's/set (fso|file|wscr|dir).*//i' $x
  /usr/local/bin/perl -p -i -e 's/\(\)/####/' $x
  /usr/local/bin/perl -p -i -e 's/sub /end sub/' $x
  /usr/local/bin/perl -p -i -e 's/vbs/bad/' $x
  /usr/local/bin/perl -p -i -e 's/Subject: I *LOVE *YOU/Subject: Cleaned_ILOVEYOU/i' $x
  /usr/local/bin/perl -p -i -e 's/Subject: fw/Subject: Cleaned_fw/i' $x
  /usr/local/bin/perl -p -i -e 's/Subject: Sus/Subject: Cleaned_Sus/i' $x
  /usr/local/bin/perl -p -i -e 's/Subject: This *Is *Funny/Subject: Cleaned_This Is Funny/i' $x
  /usr/local/bin/perl -p -i -e 's/Subject: Mother/Subject: Cleaned_Mother/i' $x
  /usr/local/bin/perl -p -i -e 's/Subject: joke/Subject: Cleaned_joke/i' $x
  /usr/local/bin/perl -p -i -e 's/Subject: Resume/Subject: Cleaned_Resume/i' $x
 fi
 done
 sleep 60
done

, Mike

linux unix internet technology

Perl Script To Mail Users JTR Password Warnings

Hey there,

Today's post harkens back to an earlier post regarding a popular password cracking program called John The Ripper (JTR).

This Perl script we've put together for today will run equally well on Unix or Linux and takes the output file of a JTR run as its input. Then, it mails every account with a cracked password a little reminder note. The default filename JTR uses to save the passwords its cracked is called john.pot, so I'll use that in our example. You can, theoretically, change that output file name (which would change this script's input file name) to whatever you want.

You'd run the following script like so:

./pwwarn.pl <john.pot

You'll note that we're not reading the ARGV array to get the argument for the file name in the script, but rather reading from the standard input file descriptor (STDIN or <>). This should make it easier to integrate into any larger Perl script you've already written.

Hopefully this will help save you some administrative hassle. Also, hopefully, the automated password generation used to print the "threat password" (as in: You really don't want us to change it to this, do you? ;) will work to bring people over to your side of the security fence.

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

# pwwarn.pl
# Take output from JTR and send reminders via
# email to folks with really bad passwords
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

$tripwire = 0;
while ( <> ) {
 if ( $_ =~ /^Loaded/ ) {
  $tripwire = 1;
 }  
 if ( $_ =~ /^Subject/ ) {
  $hostname = $_;
  chomp($hostname);
  $hostname =~ s/.*Cracked on (.*)/$1/;
 }  
 next if ( ! $tripwire );
 next if ( $_ =~ /^Loaded/ );
 next if ( $_ =~ /^guess/ );
 next if ( $_ =~ /^ * *$/ );
 push(@passwd, $_);
}

foreach $entry (@passwd) {
 if ( $entry =~ /\(/ ) {
  $entry =~ s/\(//;
  $entry =~ s/\)//;
  @entrees = split(/\s+/,$entry);
  $entries{$entrees[1]} = $entrees[0];
 }
}
foreach $key ( sort {$a <=> $b} keys %entries ) {

 $random = rand('netstat -a'*10000);
 $count=0;
 $newpassword = "";
 while ( $count < 8 ) {
  $tester = rand('netstat -a')*100;
  if ( $tester < 33 ) {
   $tester += 33;
  } elsif ( $tester > 93 ) {
   $tester -= 7;
  }
  $newpassletter = sprintf("%c", $tester);
  $newpassword .= $newpassletter;
  $count++;
 }
 $newpassword =~ s/ ?//g;
 
 open(MAILER, "|/usr/lib/sendmail -t ${key}\@xyz.com");
 select MAILER; 
 print MAILER "Subject: Cracked Password\n";
 print MAILER "From: sysadmin\@xyz.com\n";
 print MAILER "Reply-To: sysadmin\@xyz.com\n";
 print MAILER "This is an automatically generated notice.\n";
 print MAILER "Do not reply to this letter.\n";
 print MAILER "Replies will not be answered.\n";
 print MAILER "\n";
 print MAILER "The password for your account on $hostname was\n";
 printf("cracked in %2.2f seconds.\n", $random);
 print MAILER " username: $key\n";
 print MAILER " password: $entries{$key}\n";
 print MAILER "\n";
 print MAILER " In the best-case scenario, a malicious user\n";
 print MAILER "could have logged in as you and performed operations\n";
 print MAILER "as you.  In the worst-case scenario, someone could\n";
 print MAILER "have logged in as you and used your account to wage\n";
 print MAILER "an attack against this machine or other machines on \n";
 print MAILER "the network.\n";
 print MAILER " Please change your password immediately.  After a\n";
 print MAILER "certain amount of warnings, your password will be changed\n";
 print MAILER "for you to something ugly like:\n";
 print MAILER "\n";
 print MAILER " $newpassword\n";
 print MAILER "\n";
 print MAILER " If you want to change it, and do not know how, please\n";
 print MAILER "contact help\@xyz.com\n";
 print MAILER "\n";
 print MAILER " Remember, the security of the entire network depends on the\n";
 print MAILER "security of its parts.\n";
 print MAILER "\n";
 print MAILER " Thank you,\n";
 print MAILER "  System Administration\n";
 print MAILER "\n";
}

, Mike

linux unix internet technology

Regarding "Creative Commons" And Scripts And Programs Posted On This Site

In response to the unprecedented number of email's we've received regarding this blog's content, we decided to devote an entire post to answer the most common question regarding the Creative Commons licensing used on this site. Here comes the well intentioned explanation ;)

Any, and every, single script posted on this blog is meant to be useful for Linux and Unix users and administrators.

With regards to the Creative Commons licensing used, we are merely (as openly as possible) attempting to maintain and protect some intellectual property rights over the (sometimes small) fruits of our labor.

The licensing is in no way meant to discourage your use of these scripts. You may modify them to suit your needs as necessary (In fact, you probably have to most of the time, right? ;)

The non-commercial aspect of the license refers only to marketing/sales of these scripts. If you find something you like here, modify it and sell it as your own (For Profit sales. Not to be confused with pitching), you would be in violation of the license. If you find something you like here, modify it and use it to make your work life at a commercial corporation easier, that is perfectly okay and we're glad we could help you out.

Once you've modified the script for yourself, or the business at which you work, if your employer attempts to claim it as their intellectual property the water gets rocky, but you still don't have anything to worry about.

Creative Commons licensing works on good faith (with some solid legal backing), just like Gnu Public Licensing does. If big corporations get involved and try to steal profit from folks like us (and you) by hijacking an idea and re-branding it, the license provides us all some measure of protection.

In summary, feel free to modify and distribute the scripts you find on this site. All the license asks is that you include proper attribution to the script writer(s), share it with the same license and not use it for profit.

Under our interpretation of this license, non-commercial is equivalent to non-profit. These scripts are all meant for use, and we suspect that a lot of that use would be on the computers of commercial corporations. You (and your employer) may make use of them (and modify them) for free, as long as the very generous attribution portion of the license is honored and they are not used for profit (unless you want to share the profit with us. Then we'll be happy to work something out ;)

Our hope is that this middle-ground of licensing (one of the main reasons we support Creative Commons) will protect us all from the businessmen who have demonstrated a history of illegally and/or unethically profiting from the work, and inspiration, of others.

If you have any other questions regarding the licensing of scripts on this site, or any doubts or questions, please direct email to: eggi@comcast.net. He'll be happy to read however many thousands of emails you want to write him on the subject ;)

Please visit the Creative Commons official site for more detailed information on the legal specifics regarding their outstanding work.

Keep on creatively solving your world's problems and/or kicking its butt :)

Best wishes to you all,

, Mike

linux unix internet technology

Shell Script To Fake Linux Locate On Solaris

Hey There,

For this lazy weekend, I thought I'd put together a little shell script to address an issue I run into commonly enough.

If you've used RedHat Linux (or pretty much any Linux distro nowadays), you may have come to enjoy the convenience of the "locate" and "slocate" commands. Basically, all the command really does, in one mode, is create a database of every directory and file on a system (dated right then and there) and, in the most common usage mode, search that database file.

Adhering to these basic concepts, I wrote up a quick script to kind-of do the same thing on Solaris (As per above, this should work on pretty much any Unix system, as well).

Basically, you'll invoke it once to create file and directory databases (using simple find and ls commands), and run the generated scripts to search those text databases. This script has only 3 options:

f - to make a file-listing text database and create a built in script to emulate searching for files.
d - to make a directory-listing text database and create a built in script to emulate searching for directories.
nodat - to not create the additional tree.dat file that is only a directory database; no built-in script.

Depending upon what you want, if you were to run:

host # ./gfind f d nodat

you'd end up with two files, which you could invoke like this:

host # ./filefind FILENAME
host # ./treefind FILENAME

and these two commands would list out all the files matching your query (grep is used under the covers so you can use shortened versions of your desired search string to get back more results. It works just like locate/slocate in that when you run it (just like updatedb for locate/slocate), the results you get are what was on the system at the time you ran it.

If you opt to not include the "nodat" argument on your command line (only valid with the "d" option), the script will also create a totally separate directory database that you can run whatever crazy grep-like commands you want to against it.

Hope it helps you out to a good degree :)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/ksh
#
# gfind - file and directory search program producer and
# directory tree mapper
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

trap 'echo "";echo "Cleaning up and bailing out!!!";rm -f treefind filefind tree.dat johno tempmap maptemp newmap oldmap;exit' 1 2 3 15

rm -f treefind filefind tree.dat johno tempmap maptemp newmap oldmap

function make_treeexec
{
    print "#!/bin/ksh" >> treefind
    print "" >> treefind
    print "for x" >> treefind
    print "do" >> treefind
    print "grep \$x treefind" >> treefind
    print "done" >> treefind
    print "exit" >> treefind
    print "" >> treefind
    chmod 777 treefind
}

function make_fileexec
{
    print "#!/bin/ksh" >> filefind
    print "" >> filefind
    print "for x" >> filefind
    print "do" >> filefind
    print "grep \$x filefind" >> filefind
    print "done" >> filefind
    print "exit" >> filefind
    print "" >> filefind
    chmod 777 filefind
}

function make_filefind
{
    find / -print -depth 2>/dev/null >> johno
    sed '/^\proc/d' johno >> filefind
    rm johno
}

function make_treefind
{
    ls -R / 2>/dev/null >> tempmap
    sed -n '/\//p' tempmap >> maptemp
    sed -e '/^\/dos/d' -e '/^\/proc/d' maptemp >> treefind
    rm tempmap 
}

function make_dirtree
{
    awk '{if ($0 ~ /^\/[^\/]*\/[^\/]*$/)
            {print "        "$0}
        else if ($0 ~ /^\/[^\/]*\/[^\/]*\/[^\/]*$/)
            {print "                "$0}
        else if ($0 ~ /^\/[^\/]*\/[^\/]*\/[^\/]*\/[^\/]*$/)
            {print "                        "$0}
        else if ($0 ~ /^\/[^\/]*\/[^\/]*\/[^\/]*\/[^\/]*\/[^\/]*$/)
            {print "                                "$0}
        else if ($0 ~ /^\/[^\/]*$/)
            {print $0}
        else
        {print "                                        "$0}
    }' maptemp >> newmap
    sed 's/^        \/[^\/]*/       /' newmap >> oldmap
    sed 's/^                \/[^\/]*\/[^\/]*/               /' oldmap > newmap
    sed 's/^                        \/[^\/]*\/[^\/]*\/[^\/]*/                       /' newmap > oldmap
    sed 's/^                                \/[^\/]*\/[^\/]*\/[^\/]*\/[^\/]*/                               /' oldmap > newmap
    sed 's/^                                        \/[^\/]*\/[^\/]*\/[^\/]*\/[^\/]*\/[^\/]*/                                       /' newmap > oldmap
    rm maptemp newmap
    mv oldmap tree.dat
}

if [ $# -lt 1 ] || [ $# -gt 3 ]
then
    echo "Must specify at least one option!!!!"
    echo ".. and no more than three..."
    echo "Usage: gfind ( d ) ( f ) ( nodat )"
    echo "Option \"nodat\" only valid with \"d\""
    exit 1
fi
if [ $# -eq 3 ]
then
    if [[ $1 = d && $2 = f && $3 = nodat ]] || [[ $1 = f && $2 = d && $3 = nodat ]] || [[ $1 = d && $2 = nodat && $3 = f ]] || [[ $1 = f && $2 = nodat && $3 = d ]] || [[ $1 = nodat && $2 = d && $3 = f ]] || [[ $1 = nodat && $2 = f && $3 = d ]]
    then
        make_fileexec
        make_filefind
        make_treeexec
        make_treefind
        rm -f maptemp
        echo "File \"treefind\" created successfully!"
        echo "File \"filefind\" created successfully!"
        exit
    else
        echo "Hmm... Something wasn't right there"
        echo "Your only options are \"d\" \"f\" or"
        echo "\"nodat\", which is only good with \"d\""
        exit 1
    fi
fi
if [ $# -eq 2 ]
then
    if [[ $1 = d && $2 = f ]] || [[ $1 = f && $2 = d ]]
    then
        make_fileexec
        make_filefind
        make_treeexec
        make_treefind
        make_dirtree
        echo "File \"treefind\" created successfully!"
        echo "File \"filefind\" created successfully!"
        echo "File \"tree.dat\" created successfully!"
        exit
    elif [[ $1 = d && $2 = nodat ]] || [[ $1 = nodat && $2 = d ]]
    then
        make_treeexec
        make_treefind
        rm -f maptemp
        echo "File \"treefind\" created successfully!"
        exit
    else
        echo "Ooops, you can only possibly combine"
        echo "\"d\" and \"f\" or \"d\" and \"nodat\""
        echo "with only two options taken!!!"
        exit 1
    fi
fi
if [ $# -eq 1 ]
then
    if [[ $1 = d ]]
    then
        make_treeexec
        make_treefind
        make_dirtree
        echo "File \"treefind\" created successfully!"
        echo "File \"tree.dat\" created successfully!"
        exit
    elif [[ $1 = f ]]
    then
        make_fileexec
        make_filefind
        echo "File \"filefind\" created successfully!"
        exit
    elif [[ $1 = nodat ]]
    then
        echo "Option \"nodat\" only works with option \"d\"!"
        exit 1
    else
        echo "I Don't know that option!!! Try \"d\" or \"f\""
        exit 1
    fi
fi

, Mike

linux unix internet technology

Solaris Shell Script To Gather Basic System Info

Wrapping up this work week, it occured to me that I'd spent all my time on Linux and none on Unix (A bad thing? Judgement call ;)

In any event, I thought it would be somewhat equalizing if I put together one of my basic shell scripts, specifically for Solaris, that you can be run on any Unix system (although you'll need root privilege to ensure that all of these commands give you the information you want) and print up a nicely formatted listing of system information.

When run in an SSH shell loop, it can be a quick and tidy way to put together a system inventory to keep the big dogs happy ;) You can run it simply by calling it on the system you're running it on, like so:

./solinfo.sh

Cheers; enjoy your weekend :)





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

#
# solinfo.sh
#
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

echo "hostname:  \c"
/usr/bin/hostname
echo

echo "model:   \c"
/usr/bin/uname -mi
echo

echo "cpu count: \c"
/usr/bin/dmesg|grep cpu|sed 's/.*\(cpu.*\)/\1/'|awk -F: '{print $1}'|sort -u|wc -l
echo

echo "disks online: \c"
echo "^D"|format 2>/dev/null|grep ".\. "|wc -l
echo

echo "disk types:"
echo
echo "^D"|format 2>/dev/null|grep ".\. "
echo

echo "dns name and aliases:"
echo
nslookup `hostname`|grep Name;nslookup `hostname`|sed -n '/Alias/,$p'
echo

echo "Interfaces:"
echo
netstat -in|grep -v Name|grep -v lo0|awk '{print "Name: " $1 " : IP: " $4}'
echo

echo "Access Restrictions:"
echo
if [ -f /etc/hosts.allow ]
then
 cat /etc/hosts.allow
else
 echo "No host based access restrictions in place"
fi
echo
echo "OS Release: \c"
uname -r

, Mike

linux unix internet technology

Linux Shell Script To Gather Basic System Info

Today, I thought we'd go for some lighter fare, so I put together another one of my simple shell scripts, specifically for Linux, that you can run on any system (although you do need some elevated privilege to get the correct information for a few of the commands) and print up a nicely formatted listing of system information (I tried to pick the stuff superiors tend to want to know).

If you run this in an SSH shell loop, it might just be a handy way to slap together a little system inventory to pacify the big guys ;) You can run it simply by calling it on the system you're running it on, like so:

./sysinfo.sh

Anyway, enjoy and best wishes. The work week is almost over :)





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# sysinfo.sh
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

echo -n "hostname:  "
/bin/hostname
echo

echo -n "model:   "
/bin/uname -pm
echo

echo -n "cpu count: "
/bin/dmesg|grep CPU|awk -F: '{print $1}'|sort -u|wc -l
echo

echo -n "disks online: "
fdisk -l|awk '{print $1}'|grep \/dev\/|sed 's/[1234567890]//'|sort -u|wc -l
echo

echo "disk types:"
echo
fdisk -l|awk '{print $1}'|grep \/dev\/|sed 's/[1234567890]//'|sed 's/\/dev\///'|sort -u|xargs -iboink grep "^boink" /var/log/dmesg
echo

echo "dns name and aliases:"
echo
nslookup `hostname`|grep Name;nslookup `hostname`|sed -n '/Alias/,$p'
echo

echo "Interfaces:"
echo
netstat -in|grep -v Kernel|grep -v Iface|grep -v lo|awk '{print $1}'|xargs -iboink cat /etc/sysconfig/network-scripts/ifcfg-boink|sed -n 1,2p
echo

echo "Access Restrictions:"
echo
if [ -f /etc/hosts.allow ]
then
 cat /etc/hosts.allow
else
 echo "No host based access restrictions in place"
fi
echo
echo -n "OS Release: "
uname -r

, Mike

linux unix internet technology

Recovering failed RAID disks on Linux

Hey there once more,

As I mentioned in yesterday's post regarding creating RAID disk sets we'll be looking at how to deal with RAID disk failures on Linux in today's post. Once we're done here, I think we can call it a wrap on this subject for a while (Are those sighs of relief I'm hearing? ;) At least until we come up with some good shell scripts to expedite some of this stuff.

Disk failure, on the hardware side, is too broad to cover in any great detail here, but the following basic steps should be followed (of course, as noted, your setup may require otherwise). The scenario here is that one of your disks has just gone "bad." It's beyond recovery.

1. If the disk is hot-swappable, simply remove it. If it isn't, you'll need to schedule downtime and remove the disk then. If the failed disk is your boot disk, you'll already have your downtime ;)

2. Replace the failed disk and restart your machine (and skip to step 6) if your failed disk isn't the boot disk.

3. If your failed disk is the boot disk, you'll next want to boot off of CD (We're using RedHat Linux AS), mount the root filesystem under a temporary mountpoint, and do the following:

host # mkdir /tmp/recovery
host # mount /dev/hda0 /tmp/recovery
host # chroot /tmp/recovery
host # grub --batch (This may take a while as grub probes and tries to guess where all of your drives are)

4. Once grub is finished probing, do the following at the "grub>" prompt:

grub> root (hd0,0)
root (hd0,0)
Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd0)
setup (hd0)
...
Running "install /grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/grub/stage2
/grub/grub.conf"... succeeded
grub> exit

5. Now take a second and verify that all is well while still running off of the CD, like so:

host # cat /boot/grub/device.map:
(fd0) /dev/fd0
(hd0) /dev/hda

host # df -k (should show the /dev/md0 mount)
host # umount /tmp/recovery
host # reboot (Be sure to set the grub device map for hd0 to /dev/hdc if /dev/hda has gone bye-bye)

6. Now that you have the disk physically replaced and you've booted back up, check the content of /proc/mdstat, like so:

host # cat /proc/mdstat

Personalities : [raid1]
read_ahead 1024 sectors
...
md0 : active raid1 hda1[0]
     307328 blocks [2/1] [U_]
...
unused devices: <none>

Again, the [2/1] listing indicates that one part of the mirror is not active in this two-way mirror set!

7. If this has been driving you crazy from the first post, we're now going to recreate the mirror so it's not incomplete (apologies for any stress-related reading injuries. Sometimes I go a long way to make a point ;). The first thing we'll do is repartition the disk, again, with fdisk (instructions in our previous post regarding creating RAID disk sets ) and we should end up with our partition table looking exactly the same:

host # fdisk -l /dev/hda (The partition table should look almost identical for /dev/hdc)

Disk /dev/hda: 16 heads, 63 sectors, 77520 cylinders 
Units = cylinders of 1008 * 512 bytes 

   Device Boot    Start       End    Blocks   Id  System 
/dev/hda1   *         1      6095   3071848+  fd  Linux raid autodetect 
/dev/hda2          6096     67047  30719808   fd  Linux raid autodetect 
/dev/hda3         67048     73142   3071880   fd  Linux raid autodetect
/dev/hda4         75175     77206   1024096+  82  Linux swap

8. As a matter of course, just to be sure, check /etc/raidtab to see what devices need to be mirrored with what devices:

host # cat /etc/raidtab

raiddev             /dev/md0
raid-level                  1
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
   device          /dev/hda1
   raid-disk     0
   device          /dev/hdc1
   raid-disk     1
...

9. Now, we just need to add back all the partitions we lost:

host # raidhotadd -a /dev/md0 /dev/hda1
...

And keep tabs on the process by checking on /proc/mdstat:

host # cat /proc/mdstat

Personalities : [raid1]
read_ahead 1024 sectors
...
md0 : active raid1 hda1[2] hdc1[1] 
      30716160 blocks [2/1] [_U] 
      [===========>..........]  recovery = 45.9% (34790000/61432320) finish=98.7min speed=8452K/sec
unused devices: <none>

Once the RAID sync is done, you should be good to go! I'd reboot one more time, just to be sure. Especially if you were forced to take downtime in the first place :P

Take care,

, Mike

linux unix internet technology

Creating RAID disk sets on Linux

Today's post is a somewhat-continuation of yesterday's post on RAID disk monitoring. We looked at monitoring then, and (the reason I call this a somewhat-continuation) now we're going to take a step back and look at setting up those RAID disk sets (a simple mirror set in this example)in the first place ;) Note that all the examples I'm using today are taken from RedHat Linux AS and can be run in bash or any native shell. Scripting some of this out is near impossible, but someday soon I may give it a go.

In any event, once we've covered all this ground, we'll definitely look at how a handy shell script or two can save a whole lot of time on these processes :) So, off we go:

The first thing you'll want to do is to create your partitions on each physical disk, using the fdisk utility. I personally prefer it from the command line, so if you use Disk Druid or any other GUI interface, it should be easy enough to translate this to that so I don't have to over-explain.

The first thing we'll do is to create the disk partitions and assign them to an appropriate RAID device. The steps should roughly be the same as these:

1. Invoke fdisk at that command line.
2. Select "n" to add a new partition (anything from a slice of disk for mirroring two disks, to an entire disk for higher level RAID and mega-storage).
3. Once added, select "t" to change the partition flag and select "Linux Raid Autodetect" (selection "fd" on my console, possibly different on yours).
4. Select "a" to add a bootable flag to the partition, if necessary.
5. Save the new disk layout, and quit, by selecting "w"
6. Repeat as necessary

It's important, since we're doing this from the command line down-and-dirty, that we make sure that all partitions on both disks are "exactly" the same size. Unless your secondary mirror disk is larger, in which case you just need to make sure that you only use partitions larger than the originals to mirror to (It only makes sense. You can't copy a larger filesystem to a smaller one without losing data - The RAID utilities will discourage you from doing so, anyway).

Now, you'll need to put filesystems on all of those partitions. Generally, you should be able to take care of this using mkfs:

host # mkfs -t ext3 /dev/hda1

In the bizarre event that this fails, or you're using an older or different OS than I am, this should also work:

host # mkfs -t ext2 -j /dev/hda1

Now we'll use set up our /etc/raidtab file, with the following (assuming the same for all slices):

host # raiddev /dev/md0
        raid-level      linear
        nr-raid-disks   2
        nr-spare-disks  0
        chunk-size      64
        persistent-superblock 1
        device          /dev/hda1
        raid-disk       0
        device          /dev/hdc1
        raid-disk       1
...

Then we'll initialize the RAID device, like so:

host # mkraid /dev/md0

Once again, per yesterday's post on RAID disk monitoring, we can check on the RAID device's progress by simply doing the following:

host # cat /proc/mdstat

and we'll know instantly if the RAID device is in a good or bad state, and how far along it is if it's still synching. You can use the above command to check status whenever you make a change to your RAID configuration for any valid RAID md devices.

If you ever want, or need, to remove any RAID partitions, you can do that like this (In this example, the hda1 slice is having issues on the md0 device):

host # raidsetfaulty /dev/md0 /dev/hda1
host # raidhotremove /dev/md0 /dev/hda1

Sometimes, as a failsafe (this depends on your situation and what device you're removing) you may want to re-install grub on your disk device, like so:

host # grub-install /dev/hda

And, finally for today, if you do end up having to remove a RAID mirror partition and it can be replaced inline (A whole disk in itself, for instance - possibly just a partition on a disk if bizarre filesystem corruption was the only issue and you've recreated the filesystem), do the following:

1. Add the new RAID devices as before (create, set filesystem type, assign same partition names if possible, etc) and check the status of the RAID groups. [2/2] means that the RAID group is ok, [2/1] means that the RAID group needs to be resynced (you'll note from yesterday that the RAID group we're working on is in a [2/1] state, with only one active mirror. If you noted that this could be, and should be, fixed - as well as the fact that this failure would drop that RAID partition because of this condition - you're sprinting ahead of my tutorial skills ;)

host # cat /proc/mdstat 
Personalities : [raid1] 
read_ahead 1024 sectors 
md0 : active raid1 hda1[0] 
      30716160 blocks [2/1] [U_] 
...
unused devices: <none>

2. Determine your RAID group and mirrors from /etc/raidtab:

host # cat /etc/raidtab 
...
raiddev             /dev/md0 
raid-level                  1 
nr-raid-disks               2 
chunk-size                  64k 
persistent-superblock       1 
nr-spare-disks              0 
    device          /dev/hda1 
    raid-disk     0 
    device          /dev/hdc1 
    raid-disk     1 
...

3. Now, you'll want to resync the RAID mirror sets that got goofed up when we had to replace the RAID parititions:

host # raidhotadd /dev/md0 /dev/hda1

4. Assuming you've lost more than 1 mirror, just repeat the procedure above as many times as necessary.

5. Lastly, assuming all went well, kick back and enjoy the view ;) - Note that Linux will normally queue resyncs and do them one at a time, so don't panic if you only see one out of 5 disk groups being worked on when you check this out!

host # cat /proc/mdstat  
...     
md3 : active raid1 hda1[2] hdc1[1] 
      30716160 blocks [2/1] [_U] 
      [===========>..........]  recovery = 45.9% (34790000/61432320) finish=98.7min speed=8452K/sec 
...      
unused devices: <none>

And that's it for today. In another near future post (probably tomorrow) we'll look at the fun involved with RAID disk failure ;)

Best Wishes,

A Few Linux RAID Disk Monitoring Tips

Hey there,

For today's post, I thought I'd put together a few Linux RAID disk monitoring tips. This list isn't meant to be too specific, but more of a catch-all of stuff you probably have to do over and over again. But sometimes not often enough that it burns into your brain ;)

At the very basic level, you can manage your disk partitions using fdisk, much like in Windows, although completely different (type "m" to get a menu listing of available commands once you bring this little utility up). You can use the "-l" option to just print out the partition menu and not use it in interactive mode:

host # fdisk -l /dev/hda 

Disk /dev/hda: 16 heads, 63 sectors, 77520 cylinders 
Units = cylinders of 1008 * 512 bytes 

   Device Boot    Start       End    Blocks   Id  System 
/dev/hda1   *         1      6095   3071848+  fd  Linux raid autodetect 
/dev/hda2          6096     67047  30719808   fd  Linux raid autodetect 
/dev/hda3         67048     73142   3071880   fd  Linux raid autodetect
/dev/hda4         75175     77206   1024096+  82  Linux swap

If you want to display the status of your RAID devices, much the way metastat does on Solaris, it's much simpler. One of the great things about Linux is the way they've taken the /proc filesystem and turned into a really useful tool.

Entering the following will display the information you need. Note that this is a two way mirror with only one active mirror (I take the examples I can find on our network ;). Ideally, you'd want to have a [2/2] configuration for better HA.

host # cat /proc/mdstat

Personalities : [raid1]
read_ahead 1024 sectors
md2 : active raid1 hda3[0]
     62912 blocks [2/1] [U_]

md1 : active raid1 hda2[0]
     153152 blocks [2/1] [U_]

md0 : active raid1 hda1[0]
     307328 blocks [2/1] [U_]

If you're syncing up your RAID devices and want to check on the progress, just use the same command, like so:

host # cat /proc/mdstat 
Personalities : [raid1] 
read_ahead 1024 sectors 
md2 : active raid1 hdc3[1] 
      1020032 blocks [2/1] [_U] 
      
md0 : active raid1 hdc1[1] 
      3068288 blocks [2/1] [_U]
      
md1 : active raid1 hda2[2] hdc2[1] 
      3068288 blocks [2/1] [_U] 
      [==>..................]  recovery = 13.7% (894298/6136576) finish=10.2min speed=7833K/sec 
unused devices: 

Now, assuming all that is taken care of and you want to check out your RAID configuration, you'll just need to read your RAID configuration table (/etc/raidtab):

host # cat /etc/raidtab

raiddev             /dev/md0
raid-level                  1
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
   device          /dev/hda1
   raid-disk     0
   device          /dev/hdc1
   raid-disk     1
raiddev             /dev/md1
raid-level                  1
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
   device          /dev/hda2
   raid-disk     0
   device          /dev/hdc2
   raid-disk     1
raiddev             /dev/md2
raid-level                  1
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
   device          /dev/hda3
   raid-disk     0
   device          /dev/hdc3
   raid-disk     1

If you want to get more detailed with your analysis, there are (of course) built in commands to enable you to do so. lsraid is one of my favorites :)

To display a short listing of the md0 device, type the following:

# lsraid -A -a /dev/md0
[dev 9, 0] /dev/md0 8F88ACB0.7D06B4C4.FA677344.C3448700 online
[dev 3, 3] /dev/hda1 8F88ACB0.7D06B4C4.FA677344.C3448700 good
[dev 22, 3] /dev/hdc1 8F88ACB0.7D06B4C4.FA677344.C3448700 good

If you'd like to display a short listing of the RAID array that the disk hda1 belongs to (essentially the same output in this case), just type:

# lsraid -A -d /dev/hda1
[dev 9, 0] /dev/md0 8F88ACB0.7D06B4C4.FA677344.C3448700 online
[dev 3, 3] /dev/hda1 8F88ACB0.7D06B4C4.FA677344.C3448700 good
[dev 22, 3] /dev/hdc1 8F88ACB0.7D06B4C4.FA677344.C3448700 good

lsraid also comes in handy if you want to list out any faulty devices. This is something to consider putting in cron, or running regularly, so that you get notified of any failures in a timely fashion. No sense in having RAID's HA setup if you wait for all the components to fail before you replace any of them ;)

For a short listing, go with:

# lsraid -A -f -a /dev/md0
[dev 9, 0] /dev/md0 8F88ACB0.7D06B4C4.FA677344.C3448700 online

If you want, or need, all the gruesome details, lsraid will be happy to oblige. Just invoke it like this:

lsraid -D -l -a /dev/md0 
[dev 3, 3] /dev/hda1: 
        md version              = 0.90.0 
        superblock uuid         = 8F88ACB0.7D06B4C4.FA677344.C3448700 
        md minor number         = 0 
        created                 = 1169715493 (Fri Jan 25 08:58:13 2007) 
        last updated            = 1169933481 (Sun Jan 27 21:31:21 2007) 
        raid level              = 1 
        chunk size              = 64 KB 
        apparent disk size      = 3068288 KB 
        disks in array          = 2 
        required disks          = 2 
        active disks            = 2 
        working disks           = 2 
        failed disks            = 0 
        spare disks             = 0 
        position in disk list   = 0 
        position in md device   = 0 
        state                   = good 

[dev 22, 3] /dev/hdc1: 
...

In a near future post (maybe tomorrow), we'll follow up with some tips on setting up your RAID arrays in Linux and walk through that process together.

Hope this helps you out some in the meantime :)

Cheers,

, Mike

linux unix internet technology

Articles Up To Date On Linux Unix Local Hosting

Hey There,

I've finished re-tooling the last of my currently active articles on Linux, Unix, shell scripting, etc, and uploaded them to local storage.

As I mention, whenever I do this, I like to bring my articles in locally so they won't disappear when some editor somewhere finds a fault he/she neglected to harang me about earlier ;)

For today, I've posted both versions of an article on ways to beat certification tests :) There's not much more I can say about this. Even if you could write a shell script to automate the process, it might be quite a feat getting it to run on the PC's at the testing center. If you figure out how to do that discreetly, please email me. You'll get full credit for the idea, of course :)

Here they are - the two separate versions that may or may not be out there right now:

Beating Certification Tests - version 1

Beating Ceritification Tests - version 2

Hope you enjoy reading at least one of them and have a relaxing Sunday!


, Mike

linux unix internet technology

Local Unix Linux Article Repository Updated!

Happy weekend,

As you may or may not know, I'm in the habit of submitting articles on Linux, Unix, shell scripting, etc, to article sites every now and again. Mostly to promote this blog, which shouldn't be a problem if you're reading this ;)

Now that we've got another socked away, I'm bringing it in-house. I do this for 2 reasons:

1. I have no idea when my online articles are going to be randomly deleted

2. Generally, I have to "edit for content" no matter where I submit. This is okay, although I must admit that it burns a little when I have to pour over an article to remove html tags and relevant links (against policy) only to have my words draped with 100's of advertisements for miracle cream ;)

For today, I've posted both versions of an article on ways to trick JumpStart into working across subnets without following the rules :) In a future post, we'll go into how to script out this kind of setup, so you can do it on the fly. Sometimes, when the seconds count, executing a shell script is a lot more comforting than methodically following a long set of rules ;)

Here they are - the two separate versions (Posting both because the two separate article sites mangled them in two separate ways):

Jumpstarting Across Subnets - version 1

Jumpstarting Across Subnets - version 2

Hope you enjoy reading one of them and have a great Saturday!

, Mike

linux unix internet technology