Don't restart Apache. Stop it and start it.

If you restart it, you really signal the running Apache to recycle---it won't know anything about the environment of your shell.

But I'd really recommend to use a "startup.pl" module that "use lib"s the required path(s).

PerlModule lib/perl/startup.pl

if I remember correctly...