Learn C. If you're serious about doing something combining 3D and physics then a knowledge of C will be indispensible. Even if you end up using Perl for a lot you'll almost certainly be making calls into C libraries for stuff like SDL, OpenGL, whatever physics library you choose, etc. A working knowledge of C will make all of this much, much easier.

Also, learn matrix math, if you don't know it already. There are some reasonably good "math for 3d programming" books out there. Without a complete understanding of matrix operations, OpenGL will be very confusing.

-sam