Howto Script a Secure Copy
or
scripting scp and sftp network file copies
Today I needed to set up a scheduled file copy between two Linux servers. Due to security requirements, I could only use ssh. Here's how I got on.
Before going any further I would like to state that this is, as usual, written so that I can come back to it in 6 months and remind myself how this works. It is therefore written for someone of my Linux abilities, which is to say familiar with Linux, but not a guru. However, if you don't understand at least the basics of TCP/IP networking, Linux system administration, ssh, and public/private key pairs, then this might not be much use to you as understanding of that stuff is implicit in the notes below.
OK, firstly, let me set the scene by saying that:
- the copy had to be a pull operation (internal server pulling from DMZ server)
- the target (DMZ) server only allows SSH protocol 2
- root logins over SSH are disabled
So, the difficulty is that there is no way to script an SSH copy and provide a user password. This script is scheduled, so it must
not require any interaction.
The solution is to use a public-private ssh key pair for authentication of the ssh session (this stuff rocks!).
OK, so first I created a user on each server (source and target) with the same username. This is a standard Linux user, as it is only to be used for the purposes of this copy operation. The only caveat is that on the target (DMZ) server, the user has to have read access to the files you want to copy.
Next, I created the ssh key pair on the
internal server. The deal is that this server will establish the connection, so it holds the private key. We give the public key to the target server. That way, only our internal server can establish an authenticated session to the target server using the keys (i.e. without a password).
To create the ssh keypair, we log in as our user account created earlier and use the following command:
ssh-keygen -t dsaNote that this can take a minute on a slow server. When prompted for a password, just hit enter - we are setting a blank password. (If we do not set a blank password, then our scripted copy will require interaction to type the password). You can also specify rsa instead of dsa for the key type for SSH2 if you like (and if you're a nutter and not worried about security, rsa1 is the type for SSH1, but I would not use SSH1 on any production systems as it is insecure).
This creates two files in
~/.ssh:
id_dsaand
id_dsa.pubIn a fairly self-explanatory naming convention, id_dsa is our private key and id_dsa.pub is our public key. For obvious reasons, do
not ever export the private key or make it available over the network in any way, as it can be used to access the target server without a password.
Next, we need to set up the target (DMZ) server to allow ssh connections authenticated by our public key. We do this by logging in as our new user on the target server and editing
~/.ssh/authorized_keys. Note that if this file does not exist, you can just create it.
We then set the file up with our new public key by
appending our public key to whatever is there already.
Now we test the connection from the internal server to the target (DMZ) server, by issuing the following command on the internal server (logged in as our new user):
sftp targetservere.g.
sftp webserveror
sftp 192.168.0.1If all is well, you will be straight into an sftp session without being prompted for authentication. This means that the session has been authenticated by the certificate key pair.
Now we can do some scripting!
The command I used today is as follows:
scp -2 -r user@targetserver:/directory/subdirectory /home/user/subdirectory/Note that this uses scp (secure copy) rather than sftp, as it's a bit less cumbersome for this type of operation. The -2 option forces SSH2, and the -r makes the copy recursive.
So, finally I just chucked this command in a script that also rotates the last three copies on the internal server (so we have the last three days copies on disk) and set it to run in the new user's crontab (sending output to
/dev/null, of course).
In summary, this is a nice way of securely harvesting data from a DMZ server to an internal host. The only
slight, super-paranoid security concern is that the public key on the target (DMZ) server contains the FQDN hostname of the internal server. This is not an attack vector, but if the DMZ server was compromised, it could give an attacker useful information about the internal DNS naming structure.
Note that experimentation suggested that removing the FQDN name (an indeed the user name) from the public key on the target server still allows connection,
but I haven't researched this, and it may well be the case that this would allow connection from any host (or any user) using the private key, rather than just from the internal server (and specified user) so I have not made this change on my production server.
