(Better typography in the node above makes the problem more clear, but use of  <code> ... </code> tags would have been better yet. Please see Markup in the Monastery and Writeup Formatting Tips.) Here's one approach (needs Perl version 5.10+ for \K and //):

c:\@Work\Perl\monks>perl -wMstrict -le
"use 5.010;
 ;;
 my $s = '(12,4666,{xyx,yyy,ab1{1213,34343},3433,xy1[{3232,989},{8737,
+933}],3343,pq1[{1212,989}],98383},8383,xty)';
 print qq{'$s'};
 ;;
 my $curly_grp  = qr{ { \d+ (?: , \d+)* } }xms;
 my $curly_grps = qr{ $curly_grp (?: , $curly_grp)* }xms;
 my $string = qr{ [[:alpha:]] [[:alnum:]]+ }xms;
 ;;
 $s =~ s{
   $string    \K ($curly_grp)
   |
   $string \[ \K ($curly_grps) (?= \])
   }
   { (my $one = ($1 // $2)) =~ s{,}{|}xmsg;  $one; }xmsge;
 print qq{'$s'};
"
'(12,4666,{xyx,yyy,ab1{1213,34343},3433,xy1[{3232,989},{8737,933}],334
+3,pq1[{1212,989}],98383},8383,xty)'
'(12,4666,{xyx,yyy,ab1{1213|34343},3433,xy1[{3232|989}|{8737|933}],334
+3,pq1[{1212|989}],98383},8383,xty)'
[download]

(Beware possible wraparound!) (Update: Also be aware that this approach only works for one single level of nesting! I think a regex multi-level nesting solution is possible, but perhaps a bit tricky.)

Update: The substitution can be made a bit simpler with the  (?|alternations) extension of 5.10+ (which causes both captures to go to $1), and the  s///r or  tr///r (new /r modifier) extension of 5.14+. E.g. (tested):

$s =~ s{
  (?| $string    \K ($curly_grp)
      |
      $string \[ \K ($curly_grps) (?= \])
      )
  }
  { $1 =~ tr{,}{|}r }xmsge;
[download]