Probelm getting a large compressed tar file over ssh.

swares has asked for the wisdom of the Perl Monks concerning the following question:

This code seems to work but breaks on large files due to out of memory issues. I would think the file needs to be written out to file periodicly instead of cached in memory until the xfer completes or fails but I do not know quite how to go about it. We would use this to make backups of directories on remote machines with out creating a tar ball on the remote machine.

Does anyone know how to make it work with very large files?

use Net::SSH::Perl
# vars to set: $user, $password, $host, $local_fqn, $rmt_dir
open STDOUT, ">$local_fqn" or die "Can't redirect stdout";
select STDOUT; $| = 1;         # make unbuffered
$cmd="tar -czf - $rmt_dir";
my $ssh = Net::SSH::Perl->new($host);
$ssh->login($user, $password);
my($stdout, $stderr, $exit) = $ssh->cmd($cmd);
close STDOUT;
[download]

Comment on Probelm getting a large compressed tar file over ssh. Download Code

Replies are listed 'Best First'.
Re: Probelm getting a large compressed tar file over ssh. by Corion (Patriarch) on Nov 30, 2005 at 08:13 UTC
You have an error in your script - the command you're sending to the remote machine is not interpolated. You likely want: `$cmd="tar -czf - $rmt_dir";` [download] Depending on how dire your error handling situation is, avoiding Perl might be the most convenient way to resolve your memory issues: `#!/usr/bin/sh ssh -c "tar -czf - $rmt_dir" >backup.tar.gz` [download] This will run the tar command remotely and output the created .tar.gz file directly to STDOUT, and on the local end will write the output directly into a file instead of buffering it locally in memory. You will need some error checking afterwards though.	[reply] [d/l] [select]
Re^2: Probelm getting a large compressed tar file over ssh. by swares (Monk) on Nov 30, 2005 at 09:09 UTC
I used to do this as you sugested but that method requires a trusted host type relationship which I do not have in this environment. I have to conect via ssh to a number of different systems using username / password authentication.	[reply]
Re^3: Probelm getting a large compressed tar file over ssh. by Corion (Patriarch) on Nov 30, 2005 at 09:25 UTC
You don't really need a trusted host relationship, you just need passwordless keys. If passwordless keys are impossible, then you will have to use the interactive method. Maybe you can go forward by hacking Net::Ssh::Perl to redirect the STDOUT part of the connection. Looking into the source of Net::SSH::Perl::SSH1, there seem to be handlers like `SSH_SMSG_STDOUT_DATA` and replacing the default handler with something that doesn't accumulate the string might help: `# Original code sub cmd { ... unless ($ssh->handler_for(SSH_SMSG_STDOUT_DATA)) { $ssh->register_handler(SSH_SMSG_STDOUT_DATA, sub { $ssh->{_cmd_stdout} .= $_[1]->get_str }); } ... }` [download] I would try to supply my own callback like this: `my $ssh = Net::SSH::Perl->new(...); open my $outfh, ">", $filename or die "Couldn't create '$filename' : $!"; binmode $outfh; $ssh->register_handler('stdout', sub { print $outfh $_[1]; });` [download]	[reply] [d/l] [select]
Re^4: Probelm getting a large compressed tar file over ssh. by 0xbeef (Hermit) on Nov 30, 2005 at 12:21 UTC
Re^2: Probelm getting a large compressed tar file over ssh. by science_gone_bad (Beadle) on Nov 30, 2005 at 19:40 UTC
I was following this thread, and I never saw anything about the GNU tar portion. I've had issues in the past where GNU tar chokes if the tar file created exceeds 2GB in size. I don't know if this has been fixed, but I was seeing this issue as late as Sept of 2004. Maybe...	[reply]