Re: One Liner to strip crlf

Hmmm ... it feels wrong to me, and here's why:

If you can't figure out a one-liner for that, it may be that it'll be too hard to remember and/or alter. One liners are great for trivial transformations, but if they're complex enough to be worth asking about, it's probably better to write a script for.
If you're using a one-liner as a filter, then you still need to process the entire file. If you're trying to avoid the unneeded search and replace to save time, then you'd probably want to save all the I/O time as well, as that's likely going to be significant as well.
If speed is *that* important, you should probably write a quick C program specially built for the job.

Just for grins, I built a reasonably large file (465MB) and tried several filters on it:

$ # Do nothing but count lines
$ time perl -i -pe '++$cnt; END {print STDERR $cnt}' floop.cr
10000000
real    0m2.641s
user    0m2.218s
sys     0m0.375s

$ # Your original filter
$ time perl -i -pe 's/\r//g; ++$cnt; END {print STDERR $cnt}' floop.cr
10000000
real    0m6.298s
user    0m5.703s
sys     0m0.421s

$ # Don't do it globally, end at the first one
$ time perl -i -pe 's/\r//; ++$cnt; END {print STDERR $cnt}' floop.cr
10000000
real    0m3.439s
user    0m2.937s
sys     0m0.390s

$ # Do it only at the end of the line
$ time perl -i -pe 's/\r$//; ++$cnt; END {print STDERR $cnt}' floop.cr
10000000
real    0m3.188s
user    0m2.781s
sys     0m0.359s
[download]

So you can gain a bit of performance by tweaking your regular expression a bit. After I did so, the search and replace overhead was roughly 20% of the entire runtime. So you can't really get a big win here. Or, if 20% is enough time to be significant, then I'd suggest changing your processing so that rather than using a filter, you instead write a small perl script that would simply check the first line of the file. If it has "\r" then filter it, otherwise process using the original file. That way could could save nearly all of the I/O time when you don't have a "\r" in the file.

...roboticus

When your only tool is a hammer, all problems look like your thumb.

Comment on Re: One Liner to strip crlf Download Code

Replies are listed 'Best First'.
Re^2: One Liner to strip crlf by toolic (Bishop) on Sep 04, 2014 at 16:19 UTC
If you can't figure out a one-liner for that, it may be that it'll be too hard to remember and/or alter. One liners are great for trivial transformations, but if they're complex enough to be worth asking about, it's probably better to write a script for. ++ My exceptions to this rule are: Golf fun If the slightly complex one-liner is captured in a file, like Makefiles, unix aliases, etc. That being said, if I do want a one-liner and I'm struggling with it, I usually resort to creating a throw-away script first, then converting it to a one-liner.	[reply]
Re^2: One Liner to strip crlf by dirtdog (Monk) on Sep 04, 2014 at 16:10 UTC
Good point Roboticus...I'll just run the one liner on the entire file as follows: `perl -i -pe 's/\r// if /\r$/' <file>` [download] The time savings is probably negligable as you demonstrated. thanks for the help	[reply] [d/l]