I can't reproduce the slowdown you report. Adding a little crude timing code to your script

my $start = times;
my $config = XMLin($data);
my $stop = times;
print "$start:$stop: ",$stop - $start;
print "config read\n";
[download]

It seems to show that it takes just over twice as long to run under the debugger as not

P:\test>junk
0.39:1.381: 0.991
config read


P:\test>perl -d junk.pl8

Loading DB routines from perl5db.pl version 1.19
Editor support available.

Enter h or `h h' for help, or `perldoc perldebug' for more help.

main::(junk.pl8:5):     my $data;
main::(junk.pl8:6):     {
  DB<1> c 14
1.792:3.945: 2.153
main::(junk.pl8:14):    print "config read\n";
  DB<2>
[download]

which seems reasonable and nowhere near the 30 seconds you are seeing.

Perhaps you have some PERL5DB_OPTS set that could account for it?