NET::FTP ASCII mode not translating end-of-lines

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I am using net::ftp in a perl script running on Windows 2000 copying files from the Windows 2000 machine to a variety of non-Windows target machines. When I put a file to these machines, the files do not get converted from the Windows end-of-line convention (cr-lf) to the target convention, thus rendering the files useless. The target systems are OpenVMS, Unix and Linux. I am setting ascii with no affect. Tracing with debug gives me:

Net::FTP=GLOB(0x1d3ccb4)<<< 220 FTP Service Ready
Net::FTP=GLOB(0x1d3ccb4)>>> user foo
Net::FTP=GLOB(0x1d3ccb4)<<< 331 User name foo received, please send pa
+ssword
Net::FTP=GLOB(0x1d3ccb4)>>> PASS ....
Net::FTP=GLOB(0x1d3ccb4)<<< 230 foo logged in, directory DKA300:[foo]
Net::FTP=GLOB(0x1d3ccb4)>>> TYPE A
Net::FTP=GLOB(0x1d3ccb4)<<< 200 Command OK
Net::FTP=GLOB(0x1d3ccb4)>>> CWD [foo.install]
Net::FTP=GLOB(0x1d3ccb4)<<< 250 Working directory changed to "DKA300:[
+foo.INSTALL]"
---   starting ftp of 402 files
Net::FTP=GLOB(0x1d3ccb4)>>> PORT x,x,x,x,x,x
Net::FTP=GLOB(0x1d3ccb4)<<< 200 PORT Command OK.
Net::FTP=GLOB(0x1d3ccb4)>>> STOR cnxsockh.h
Net::FTP=GLOB(0x1d3ccb4)<<< 150 ASCII transfer started for DKA300:[foo
+.INSTALL]CNXSOCKH.H;
Net::FTP=GLOB(0x1d3ccb4)<<< 226 File transfer completed ok
[download]

Extensive searching of the Internet does not show anyone with a similar problem, but I can't believe I am unique. This problem does not occur if I use the native Windows 2000 ftp client. I really don't want to do that, though, as net::ftp gives me a far more consistent connection in every respect other than this end-of-line conversion problem. Thank you very much for your prompt assistance. Marc

Comment on NET::FTP ASCII mode not translating end-of-lines Download Code

Replies are listed 'Best First'.
Re: NET::FTP ASCII mode not translating end-of-lines by Thelonius (Priest) on Mar 06, 2003 at 15:05 UTC
I have confirmed that this is a bug in Perl 5.8.0. I have submitted it as http://bugs6.perl.org/rt2/Ticket/Display.html?id=21473. As a workaround you can change `sysread` to `read` at line 729 of /Perl/lib/Net/FTP.pm. Receiving in ASCII mode is also broken. A workaround for this is to change lines 481/482 to `my $written = print $loc $buf; unless($written)` [download]	[reply] [d/l] [select]
Re: Re: NET::FTP ASCII mode not translating end-of-lines by mgibian (Acolyte) on Mar 06, 2003 at 17:23 UTC
The read workaround did indeed resolve my problem. Thank you VERY much for your help with this.	[reply]
Re: NET::FTP ASCII mode not translating end-of-lines by Thelonius (Priest) on Mar 04, 2003 at 02:54 UTC
I haven't had any problem with this and I've done it frequently. I just tried it from Windows XP to HP-UX and it worked fine. If you are using cygwin Perl and the file is on a binmode mount, then a file with CR-LF will be sent with CR-CR-LF. What version of Perl are you using? I could also imagine silly things, such as: `open IN, "<cnxsockh.h"; binmode IN; $ftp->send(\*IN, "cnxsockh.h");` [download] but I really doubt you would have done something like that.	[reply] [d/l]
Re: Re: NET::FTP ASCII mode not translating end-of-lines by mgibian (Acolyte) on Mar 04, 2003 at 20:47 UTC
Here is the full version info for the Perl I'm running on my Windows 2000 machine: Q:\>perl -V Summary of my perl5 (revision 5 version 8 subversion 0) configuration: Platform: osname=MSWin32, osvers=4.0, archname=MSWin32-x86-multi-thread uname='' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef usethreads=undef use5005threads=undef useithreads=define usemultip +licity=define useperlio=define d_sfio=undef uselargefiles=undef usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cl', ccflags ='-nologo -Gf -W3 -MD -DNDEBUG -O1 -DWIN32 -D_CON +SOLE -DNO_STRICT -DPERL_IMPLICIT_CONTEXT -DPERL_ IMPLICIT_SYS -DUSE_PERLIO -DPERL_MSVCRT_READFIX', optimize='-MD -DNDEBUG -O1', cppflags='-DWIN32' ccversion='', gccversion='', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=10 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', + lseeksize=4 alignbytes=8, prototype=define Linker and Libraries: ld='link', ldflags ='-nologo -nodefaultlib -release -libpath:"c:\ +perl\lib\CORE" -machine:x86' libpth=C:\lang\VC98\lib libs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib + comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib wsock32.lib mpr.lib winmm.lib ver +sion.lib odbc32.lib odbccp32.lib msvcrt.lib perllibs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool +.lib comdlg32.lib advapi32.lib shell32.lib ole32. lib oleaut32.lib netapi32.lib uuid.lib wsock32.lib mpr.lib winmm.lib + version.lib odbc32.lib odbccp32.lib msvcrt.lib libc=msvcrt.lib, so=dll, useshrplib=yes, libperl=perl58.lib gnulibc_version='undef' Dynamic Linking: dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -release -l +ibpath:"c:\perl\lib\CORE" -machine:x86' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY USE_ITHREADS PERL_IMPLICIT_CONTEX +T PERL_IMPLICIT_SYS Built under MSWin32 Compiled at Nov 14 2002 12:27:02 @INC: c:/perl/lib c:/perl/site/lib . Q:\> [download] What I don't understand is why net::ftp gives me the behavior of preserving the silly windows -CR-LF while using the native Windows 2000 ftp client does the appropriate translation? I expected both facilities to exhibit the same behavior.	[reply] [d/l]
Re: NET::FTP ASCII mode not translating end-of-lines by graff (Chancellor) on Mar 04, 2003 at 03:20 UTC
I tried fetching/putting a windows-style text file (between a solaris server and my local linux box) using ascii mode with both Net::FTP and the command-line ftp utility. They both preserved the "\r\n" line terminations in the text file. I noticed that the alleged "ascii" mode transfer actually causes more bytes to be transmitted -- a remote file with 20162 bytes (including 278 \r's and 279 \n's) reports 20441 bytes sent, whereas a binary transfer reports exactly 20162 bytes. Alas, both methods leave me with a 20162-byte file (i.e. no difference in the local copy). Anyway, it might not be the fault of the perl module. I saw nothing in Net::FTP that would allow sending/getting data using a scalar variable as if it were a local file -- this would be a neat feature if it were available, and would give you a work-around for this problem. I thought about trying to use a file handle instead of a file name, given that the file is set to ":crlf" with binmode, but that didn't seem to have any effect on the ftp transfer either. You might consider using Archive::Tar to create a tar file for each distinct type of recipient host -- this allows you to control the line-term behavior of files being assembled into the tar set, and you have just one file to push to each receiver in binary mode (could even have Archive::Tar create the file in compressed form). But it seems a shame to add this much overhead before and after the transfer -- create the tar file, delete it when the transfer's done, unpack then delete the tar file at each recipient host. The only other recourse would be to create an alternative form of Net::FTP (a derived class module, or simply a replacement) where you redefine the internal "_store_cmd()" method, so that it does the right thing instead of (or supplementing) the current "sysread()" call on the local file being sent (or maybe even support the use of a scalar that holds file content rather than using a file name or file handle).	[reply]
Re: Re: NET::FTP ASCII mode not translating end-of-lines by Thelonius (Priest) on Mar 04, 2003 at 04:34 UTC
I tried fetching/putting a windows-style text file (between a solaris server and my local linux box) using ascii mode with both Net::FTP and the command-line ftp utility. They both preserved the "\r\n" line terminations in the text file. I noticed that the alleged "ascii" mode transfer actually causes more bytes to be transmitted -- a remote file with 20162 bytes (including 278 \r's and 279 \n's) reports 20441 bytes sent, whereas a binary transfer reports exactly 20162 bytes. Alas, both methods leave me with a 20162-byte file (i.e. no difference in the local copy). Anyway, it might not be the fault of the perl module. This is the expected behavior. The sending system in an FTP ASCII transfer takes whatever the local notion of a line is and sends it as a series of characters terminated with CR-LF "\r\n". So if you are sending from a Unix system, a file that starts of "abc\ndef\r\nghi\n" will be sent as "abc\r\ndef\r\r\nghi\r\n". Any \r that is already in the text file is treated as just another character. The receiving system in an FTP ASCII transfer translates the CR-LF pairs back to whatever the local notion of a line is. So on another Unix system, you'll get back the original file. On Windows, though, it's different. A file that starts "abc\r\ndef\r\n" will be sent just like that in ASCII mode. A receiving Unix FTP process (in ASCII mode) will convert it to "abc\ndef\n". As I said above, if he is running Cygwin Perl instead of ActivePerl, and the file is on a binmode mount, then the file will be treated just like a Unix file.	[reply]