Simple SCP Utility For Network Wide File Distribution

Hey again,

This little shell script is a somewhat follow-up to an older post we did on creating an SSH command line runner. It's basically the same concept, except this time we're looking specifically at copying files all over the network, using SCP, rather than executing commands all over the network using SSH (One might argue that, under the covers, they're the same thing. But one might also argue that each has an advantage over the other, depending on what it's being used for and there's no argument that neither takes the exact same command line options :)

One other thing to note, before you begin using this script, is that it assumes that you have passwordless key-based SSH authentication set up on all the hosts you're going to be copying to. If you haven't got that set up already, check out our post on setting up your SSH keys network wide quickly and easily.

This script is fairly simple to run and should work on most flavors of Linux and Unix. If any modification is necessary it should be minor, and center around the line where we actually invoke SCP. It takes two arguments (where we're copying from and where we're copying to) and can be run simply, from the command line, like so:

host # ./scopy /users/bob/file /users/bob/file

Of course, you may note that the above example is incredibly specific. You don't need to include the directory if you don't want to (I just usually do so to be sure things end up where they're supposed to). You could easily do this, as well (assuming that your home directory isn't in the same location on every box on your network):

host # ./scopy /users/bob/file ~/bob/file

or any number of variations. Have fun with it :) The only thing it looks for outside of itself, is a host file, which I've cleverly named "hostfile" ;) This is completely arbitrary and can be changed, as well. The format I've chosen is "name : IP Address," like this:

host1 : 192.168.0.10

with one entry per line. Again, with some simple modification, this can be changed easily to suit your taste.

Hope you enjoy it and it helps you out some :)

Cheers,





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/ksh

#
# scopy - scp files from one location to 
#   all of your network hosts
#
# 2008 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

trap 'rm tmpfile.$pid;echo "Caught Signal.  Cleaning Up And Quitting...";exit 3"' 1 2 3 9 15

pid=$$

if [ $# -ne 2 ] 
then
 echo "Usage: $0 fromDirFile ToDirFile"
 exit 1
fi

if [ -f hostfile ]
then
 hostfile="hostfile"
elif [ -f /users/bob/data/hostfile ]
then
 hostfile="/users/bob/data/hostfile"
else
 echo "Can't find hostfile.  No hosts to copy to.  Out..."
 exit 2
fi

from=$1
to=$2
squashed_command=`echo $1 $2|/bin/sed -e 's/ * *//g' -e 's/|//' -e 's/\///g'`

print "" >>tmpfile.$pid
print "Report Output for \"scp $1 $2\"" >>tmpfile.$pid
print "______________________________" >>tmpfile.$pid
print "" >>tmpfile.$pid

cat $hostfile|while read name colon address
do
 if [ $name == "#" ]
 then
  :
 else
  print "\nCopying \"$1 to $2\" on \c" >>tmpfile.$pid
  print "$name... " >>tmpfile.$pid
  print "$name... \"scp $1 $2\"... "
  /usr/bin/scp -r $1 ${address}:$2 2>/dev/null|tee -a tmpfile.$pid
 fi
done 

mv tmpfile.$pid OUTPUT.${squashed_command}.$pid

, Mike

linux unix internet technology

A Quick Guide To Setting Up SSH Keys Network Wide

Today, in honor of the spirit of the day, I thought I'd write a little something for which you might, eventually, give many thanks :)

As we mentioned in yesterday's post, once you have SSH keys set up network-wide, so that you can passwordlessly login as yourself on all the machines you administrate, our "scmd" script can be a blessing; especially if you're ever stuck with the seemingly impossible task of retrieving some obscure bit of information off of every single box!

I won't lie to you; this is the worst part of the process. In order to get your keys initially set up, you'll have to put forth a good deal of effort (Your misery will be commensurate with the size of your network). But, once you're done, you'll be on easy street.

For our purposes, we're going to assume that you're using Solaris' (now) standard implementation of SSH (Basically OpenSSH) and craft our instructions using those assumptions. Depending on what kind of SSH client/server you're using, there may be differences in the names of certain files (e.g. authorized_keys2 might be authorized_keys -- The man pages are your friends :) We're also assuming that you actually already have user accounts on all the machines you're required to do work on.

So, to get the worst part over with, simply do the following (cut it up into chunks if it becomes frustrating. Depending on the size of your network, this is quite possible):

1. Create a file with the names or IP addresses of all of the machines you need to work on. For our purposes, format that file with one hostname/IP per line. We'll assume it's named "HostFile"

2. If you haven't done so already, generate your personal ssh-keys on the one machine that you're going to use as your central hub for administrating all the others, like so:

ssh-keygen -t dsa
(Answer yes to the default file locations -- /your/home/directory/.ssh/id_dsa and /your/home/directory/.ssh/id_dsa.pub)
(Also, just hit return both times you're prompted to enter a password. While this is just slightly less secure than entering the double-confirm password, it will completely defeat the purpose of setting this easy-administration up)

2. Run the following from your home directory (This is the first really long and trying part, as you'll have to interact manually):

for x in `cat HostFile`;do ssh -n $x "mkdir -m 700 .ssh";done
(Actuall SSH command may vary depending on your distro. I'm using the "-n" flag to avoid having SSH break out of the "for loop")

Now, if we're starting from scratch, you'll need to answer "yes" for each machine's initial prompt (it will ask you if you trust the host and want to accept and save the key) and enter your password.

3. When that's finally over, cd into your home directory's .ssh directory and run the following on the command line ("HostFile" is still in your home directory, so we're using a relative path to it here):

for x in `cat ../HostFile`;do scp id_dsa.pub $x:/your/home/dir/.ssh/authorized_keys2
(Again, you'll need to enter your password for every machine. Note that if you run a variety of servers, or build standards have changed over time so that your home directory isn't always in the exact same place on every box, you can use "~.ssh/authorized_keys2" as a substitute for "/your/home/dir/.ssh/authorized_keys2)"

4. And, now, you're all done, except to make sure that it actually worked! My favorite way to do this is to just run the same command again. Overwriting your authorized_keys2 file on all the remote hosts won't hurt, and - this time - you should be just kicking back, watching scp's progress meters fly by as you try not to fall asleep ;) Again the "~.ssh/authorized_keys2" substitution is perfectly valid and useful:

for x in `cat ../HostFile`;do scp id_dsa.pub $x:/your/home/dir/.ssh/authorized_keys2

For all the machines that still ask you for your password, take note of those hosts and troubleshoot as necessary. For the most part, you shouldn't have to worry about running into too many of those.

And from here on out, you can use the "scmd" command we put in our pre-Thanksgiving post, or your own custom scripts, to effortlessly run any command (as yourself, of course) on all your machines, by typing a single line on your hub computer.

Happy Thanksgiving :)

, Mike

linux unix internet technology

Estimating Network Traffic Speed Using Regular File Utilities

Most of the time, if you want to estimate how fast your network is moving, you'll want to look at the output of tools like "netstat" and the like. Some file transfer programs, like "scp" are kind enough to print out the speed at which they're transferring a file to you while they're doing it.

But, sometimes you'll be stuck waiting on an file to make it all the way over to your machine using a protocol like "ftp." And, while it's possible to measure the speed of transfer in many different ways, I threw together this little script to let you know how fast your file is being "ftp"'ed to you by taking measure of the file being transferred, itself.

Check it out. It's kind of fun to see that there really is more than one way to skin a cat. In this case, you don't even need a skinner ;)

Note: Adjust the time variables to your liking. I wrote this for transfer of rather large files! The script, below, at the most basic level, measures the difference in the received file's size against the passing of time to determine approximately how fast your network is transferring that file.

And, yes, this is another experiment in "brute force scripting" (Getting results as fast as possible using scripting, even if it means all your t's aren't dotted ;), so you'll notice that the first output will be 0 and is accompanied by a message to the effect that measurement can't take place until more data is collected.





This work is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

#!/bin/ksh

#
# 2007 - Mike Golvach - eggi@comcast.net
#
# Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License
#

last_total=0
first_time=1

while :
do
date
echo "KB PRIOR $last_total"
total=`du -sk|awk '{print $1}'`
echo "KB ACTUAL $total"
let new_total=${total}-$last_total
last_total=$total
echo "KB DIFF $new_total"
let hourly=${new_total}*60
echo "KB HOURLY $hourly"
let pre_average=`echo 2 k ${hourly} 1048576 / p |/usr/bin/dc|/usr/bin/sed 's/\.//'`
average=`echo $pre_average|/usr/bin/sed 's/^\(.*\)\(..\)$/\1\.\2/'`
netstat -ian|grep bge3|sed -n 1p|awk '{print "Packets In: " $5 " Packets Out: " $7}'
if [ $first_time -eq 1 ]
then
echo "Average incalculable until more data received"
first_time=0
else
echo "Averaging $average Gb per hour"
fi
echo
sleep 60
done

Enjoy!

, Mike

linux unix internet technology

Setting Basic File Permissions in a Non-Interactive SFTP/SCP Shell

Hey There,

I'm sure you've probably had this happen once and again. A user moves files to a box regularly, but can't be specific about how, and the files always have permissions he/she didn't expect. For instance, they may be 640 instead of the 644 expected, so the files aren't world-readable like he/she wants.

A cursory check of /etc/profile, /etc/.login, /etc/default/login, /users/home/directory/.profile, etc all show a umask of 022. Meaning all new files should be created as 644. Yet, they're still coming up 640.

In these sorts of cases, you're probably dealing with someone who's moving files using a transfer mechanism that doesn't open up an interactive shell. Common utilities like SCP and SFTP do this by design.

The most assured way to fix this problem quickly and simply (and, through security-by-ignorance on a per-user basis ;) is to just find your sshd_config file (usually in /etc, /usr/local/etc, /usr/local/openssh/etc or wherever your configuration places them) and check for a setting called:

PermitUserEnvironment no

All you need to do is change that "no" to a "yes" and then do the following, to easily change the permissions for that user's upcoming transfers.

1. Kill -1 (Kill -HUP) the sshd process. You can find the PID of the sshd process using ps -ef or looking in a state file, like /var/run/sshd.pid. Be sure to check /var/adm/messages, or wherever you log your ssh activity to, to be sure that the server restarts, as it needs to in order to re-read it's configuration file. Of course, if it crashes, you won't need to read a file to find out!

2. Create a file for the user (or su to the user and create the file as the user) in his/her .ssh directory in his/her home directory called, simply, "environment"

touch /users/home/directory/.ssh/environment
chmod 640 /users/home/directory/.ssh/environment

3. Then add one line to that file and your SCP/SFTP permissons problems should disappear.

umask=002

Of course, 002 is a pretty generous umask, but you can set this to whatever is appropriate for your needs.

Now you have control of new file permissions in both interactive and non-interactive shells!

, Mike

linux unix internet technology