I would expect the Build command to run perl in a subprocess to run each test. So the Build process having the debugger active would have no bearing on whether or not the debugger was active in the other process.

Either use the debugger to find the point in Build where it starts the other Perl and have that invocation add "-d" or, more simply, add "-d" to the "#!" line of your test script and just run Build normally.