Turns out the code was calling Test::More using eval. One I used a normal use statement the error went away.