Here's a version that only requires a single pass over the array. The end result is that the 5th element of each segment is a ref to the next one. The last one doesn't reference the next or we'd get a circular structure.

use warnings;
use strict;
my @seg = (
    [ 1, 1, 2, 1 ],
    [ 5, 5, 1, 5 ],
    [ 2, 3, 5, 3 ],
    [ 5, 3, 5, 5 ],
    [ 2, 1, 2, 3 ],
    [ 1, 5, 1, 1 ]
);

use Data::Dumper qw(Dumper);
my (%have, %want);
while (my $seg = shift @seg)
{
  my $have = join(",", @{$seg}[0,1]);
  my $want = join(",", @{$seg}[2,3]);

  print "we have $have\n";
  if (my $wanted = delete $want{$have})
  {
    print "something wants it\n";
    $wanted->[4] = $seg;
  }
  else
  {   
    print "nothing wants it yet\n";
    $have{$have} = $seg;
  }
   
  last if @seg == 0; # don't want to make a circular reference

  print "we want $want\n";
  if (my $had = delete $have{$want})
  {
    print "found it\n";
    $seg->[4] = $had;  
  }
  else
  {   
    print "didn't find it, asking for it\n";
    $want{$want} = $seg;
  }
   
}  
$Data::Dumper::Indent = 0;
print Dumper values %have;
[download]

the end result is

[1,1,2,1,[2,1,2,3,[2,3,5,3,[5,3,5,5,[5,5,1,5,[1,5,1,1]]]]]]
[download]

It's glorious day in Dublin so I'm off to the pub now :-)

UpdateI suppose it does require a second pass to produce output in the format requested...