(Better typography in the node above makes the problem more clear, but use of  <code> ... </code> tags would have been better yet. Please see Markup in the Monastery and Writeup Formatting Tips.) Here's one approach (needs Perl version 5.10+ for \K and //):

c:\@Work\Perl\monks>perl -wMstrict -le
"use 5.010;
 ;;
 my $s = '(12,4666,{xyx,yyy,ab1{1213,34343},3433,xy1[{3232,989},{8737,
+933}],3343,pq1[{1212,989}],98383},8383,xty)';
 print qq{'$s'};
 ;;
 my $curly_grp  = qr{ { \d+ (?: , \d+)* } }xms;
 my $curly_grps = qr{ $curly_grp (?: , $curly_grp)* }xms;
 my $string = qr{ [[:alpha:]] [[:alnum:]]+ }xms;
 ;;
 $s =~ s{
   $string    \K ($curly_grp)
   |
   $string \[ \K ($curly_grps) (?= \])
   }
   { (my $one = ($1 // $2)) =~ s{,}{|}xmsg;  $one; }xmsge;
 print qq{'$s'};
"
'(12,4666,{xyx,yyy,ab1{1213,34343},3433,xy1[{3232,989},{8737,933}],334
+3,pq1[{1212,989}],98383},8383,xty)'
'(12,4666,{xyx,yyy,ab1{1213|34343},3433,xy1[{3232|989}|{8737|933}],334
+3,pq1[{1212|989}],98383},8383,xty)'
[download]

Update: The substitution can be made a bit simpler with the  (?|alternations) extension of 5.10+ (which causes both captures to go to $1), and the  s///r or  tr///r (new /r modifier) extension of 5.14+. E.g. (tested):

$s =~ s{
  (?| $string    \K ($curly_grp)
      |
      $string \[ \K ($curly_grps) (?= \])
      )
  }
  { $1 =~ tr{,}{|}r }xmsge;
[download]

I tried the below expression. It worked for few cases and failed for other cases. Dont know the reason. First expression is to replace comma between <string>{ and } and other to replace comma between <string>[{ and }]

perl -pe 's/,(?=[^\{\}]*\d\b*\{*\})/|/g;s/,(?=[^\[\]]*\[*\{*\}*\])/|/g
+;'
[download]

You still haven't explained what "failed in other cases" means.

It's hard to read in your one-liner (have you tried adding the /x flag to your regexp which allows you to write it over multiple lines, with comments?), but in your second statement it sure seems like you are replacing anything that matches ,(?=[^\[\]]*\[*\{*\}*\]) with | ... you probably need to learn about extracting matches, too.

Remember: Ne dederis in spiritu molere illegitimi!

Many thanks for your inputs and advice. I dont have much expertise on the Perl Regular Expression. I need to do this using substitute operator, So I am looking for help. Let me put my requirement in a simple way.

(1,2,4,{8,9,xy{4343,0943},pq[{434,5454}]},4,6)
[download]

Replacement #1

Need to replace commas in the pattern xy{4343,0943} with pipe to get xy{4343|0943}

Replacement #2

Need to replace commas in the pattern pq[{434,5454}] with pipe to get pq[{434|5454}]

Please suggest