Concatenate or Join?

perldaemon has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I've always believed that joining a lot of strings together would be faster than concatenating them.

The idea is that concatenation creates a new copy of the string for each concatenation whereas join could, well join them, and only make one copy.

I created two test programs and was surprised by the result. Can anyone shed some light on this.

The only real difference between the programs is that one collects the chars in an array and then joins, the other concatenates them onto a string.

t_concat:

#!/usr/bin/perl
#t_concat

my $file = $ARGV[0];
local ($/) = undef;
open FH, $file or die;
my $data = <FH>;
close FH;

my $len = length($data);
my $result;
my $c;

for($i = 0; $i < $len; $i++){
    $c = substr $data, $i, 1;
    #concatenate onto result.
    $result .= $c;
}

print $result;
[download]

t_join:

#!/usr/bin/perl
#t_join
$file = $ARGV[0];
local ($/) = undef;
open FH, $file or die;
my $data = <FH>;
close FH;

my $len = length($data);
my @result;
$#result = $len;
my $c;

for($i = 0; $i < $len; $i++){
    $c = substr $data, $i, 1;
    #put the string in an array for later join.
    $result[$i] = $c;
}

print join('', @result);
[download]

These results where consistent over 30 runs. Any wisdom?
t_join test.data > /dev/null 0.21s user 0.02s system 99% cpu 0.240 total
t_concat test.data > /dev/null 0.11s user 0.00s system 98% cpu 0.112 total

Comment on Concatenate or Join? Select or Download Code

Replies are listed 'Best First'.
Re: Concatenate or Join? by dave_the_m (Monsignor) on Nov 12, 2012 at 22:52 UTC
You are not comparing concat against join. You are comparing concat against creating and repeatedly extending an array and joining. With a level playing field, join is quicker: `use Benchmark; my @a = ('x') x 1_000_000; timethese(100, { 'concat' => sub { my $s; $s .= $_ for @a }, 'join' => sub { my $s = join '', @a }, },); __END__ concat: 11 wallclock secs (10.86 usr + 0.01 sys = 10.87 CPU) @ 9 +.20/s (n=100) join: 4 wallclock secs ( 4.01 usr + 0.00 sys = 4.01 CPU) @ 24 +.94/s (n=100)` [download] Dave.	[reply] [d/l]
Re^2: Concatenate or Join? by perldaemon (Novice) on Nov 13, 2012 at 00:43 UTC
Thanks Dave for the input. I was comparing a situation where I can either fill an array as I parse through data and then later join, or just concatenate as I go. I agree that if the array is already created then it is certainly faster to join. But if you have to build the array just so you can join the elements later, it looks like concatenation is faster. Or am I missing something?	[reply]
Re^3: Concatenate or Join? by dave_the_m (Monsignor) on Nov 13, 2012 at 09:11 UTC
In that case, doing the concat (and thus skipping creating the array) could be faster, but would probably depend on such factors as the pattern of string lengths, and whether you could presize the string. Dave.	[reply]
Re: Concatenate or Join? by LanX (Saint) on Nov 12, 2012 at 20:33 UTC
Simple, concat doesn't create a new copy, it's just enlarging the old one as long as the allocated buffer allows to. I think you are confusing Perl with languages like JS, Java or Python where (the memory representation) of strings is immutable, i.e. can only be created, read or deleted but never updated. see also http://stackoverflow.com/questions/51185/are-javascript-strings-immutable-do-i-need-a-string-builder-in-javascript Cheers Rolf	[reply]
Re: Concatenate or Join? by johngg (Canon) on Nov 12, 2012 at 20:38 UTC
Reducing the join and concatenation to a simple example without any file i/o, I don't get the same result when Benchmarking. `use strict; use warnings; use Benchmark qw{ cmpthese }; my @arr = ( q{abc} ) x 10000; cmpthese( -5, { concat => sub { my $ret; $ret .= $_ for @arr; return $ret; }, join => sub { my $ret = join q{}, @arr; return $ret; }, } );` [download] $ ./spw1003500 Rate concat join concat 994/s -- -70% join 3310/s 233% -- $ [download] Perhaps something else is skewing your timing. Cheers, JohnGG	[reply] [d/l] [select]
Re^2: Concatenate or Join? by tobyink (Canon) on Nov 12, 2012 at 20:54 UTC
It depends what you're doing really. In your example, join beats concat, but that's because you are looping through an array. Here though, concatenation beats it (albeit only by about 10%): `use strict; use warnings; use Benchmark qw{ cmpthese }; sub foo { 'abc' }; sub bar { 'def' }; cmpthese( -3, { concat => sub { my $ret = foo().bar(); return $ret; }, join => sub { my $ret = join q{}, foo(), bar(); return $ret; }, });` [download] And here (thanks to the Perl compiler being so good at constant folding) concatenation is more than twice as fast as join: `use strict; use warnings; use Benchmark qw{ cmpthese }; sub foo () { 'abc' }; sub bar () { 'def' }; cmpthese( -3, { concat => sub { my $ret = foo . bar; return $ret; }, join => sub { my $ret = join q{}, foo, bar; return $ret; }, });` [download] `perl -E'sub Monkey::do{say$_,for@_,do{($monkey=[caller(0)]->[3])=~s{::}{ }and$monkey}}"Monkey say"->Monkey::do'`	[reply] [d/l] [select]