I recall that there is a way to tell Perl to not look at the new line character as the end of a record, so that the whole file could be globbed into one string for processing, then written back out easily. But I can't remember where I read that, or what it was called. Does anyone know what I'm talking about?

Once you have your slurped input, if you need to output as paragraphs, Text::Wrap may be helpful.

Update: If you don't want to muck with $/, you could use the slurp function from File::Slurp.

See perlrun, there "paragraph mode". The boundary between paragraphs in that mode may comprise several blank lines.

Given a file named "text" as follows

this is the first paragraph.
It has three lines.
Of which this one is the third.

A blank line denotes a paragraph. This one
has got two lines only.



The third paragraph is preceded with several
 blank lines, and it has itself
  three,
   no, wait, four lines.

Last paragraph. All paragraphs  should be shown
    each as
one line running "perl -p00 -le 's/\n/ /gs;s/\s+/ /g;'"
on that file, with multiple blanks condensed into one.
[download]

the snippet

perl -p00 -le 's/\n/ /gs;s/\s+/ /g;' text
[download]

does what you want. If you want inplace-edit (see perlrun again), say

perl -p00 -i.bak -le 's/\n/ /gs;s/\s+/ /g;' text
[download]

to have the file backed up with the suffix .bak as text.bak

You can provide multiple files on the command line; each will be processed in turn (and backed up, if requested).

Thank you to all you kind Perl Monks!! All of the above are great solutions. The INPUT_RECORD_SEPARATOR was what was trying to recall, but you've all offered good solutions. I've only had a chance to try the last solution, and it works like a charm. I didn't realize I could do search and replaces direct from the command line like that. That's awesome! Thanks again!

For anyone else that needs to cleanup poorly formatted ebooks in text files, here's what I came up with from the help I received above:


perl -p00 -i.bak -le "s/-\n//gs;s/([^!\?\.\"\'\`])\n/\1 /gs;" myfile.txt


This will remove newline characters at the end of lines that do NOT end in a period, question mark, exclamation point or some form of quote. However, prior to removing those newline characters, it removes any newline character preceded by a hyphen as well as removing the hyphen.

It works great. Thanks Perl Monks!

You're talking about undefined $/. Reading a whole file at once is called "slurping the file" or "slurping the file's content".

my $file; { local $/; $file = <>; }
$file =~ s/(?<!\n)\n(?!\n)//g
print $file;
[download]

Usage for above:

perl script.pl infile > outfile

perl -i.bak script.pl file   # in-place
[download]

As a one-liner:

perl -0777pe's/(?<!\n)\n(?!\n)//g' infile > outfile

perl -i.bak -0777pe's/(?<!\n)\n(?!\n)//g' file
[download]

You are thinking of 'file slurping' - which is done by modifying the $/ perl line separator (use $/ = undef; to slurp in one go), or using a module like File::Slurp.

Then you can use a regex to remove single newline characters, but leave alone double ones using a lookahead assertion, or something simple like:

use strict;
use warnings;

my $line = "foo\nbar\n\nbaz";
print "Before :\n \'$line\'\n";
#Before :
#'foo
#bar
#
#baz'

$line =~ s/\n([^\n])/$1/;
print "after : \'$line\'\n";
#after :
#'foobar
#
#baz'
[download]

HTH!