Depending on the complexity of your game, perl could be fast enough. But if you're aiming for fast paced game with loads of special effects, some other tool might be a better choice. C/C++ seems to be the most common which means there is a lot of tutorials and communities to learn from. It also has quite steep learning curve and lot of things that modern languages handle automatically has to be done manually.

Another option would be Java, but there you would sacrifice quite some CPU power. I'm well aware that it's possible to develope good commercial games with Java, but I wouldn't start learning it just because of that. Especially since you already know perl and have some experience on that. I would leverage that experience and choose perl, if the final choice would be between Java and perl.

If you want to do some 2D and C/C++, I would recommend Kyra Sprite Library (http://www.grinninglizard.com/kyra/) It's rather neat sprite system and has somewhat active community (at least back when I was still working with it). It's build on top of SDL. SDL would be my choice over Allegro (no real reason though, I just liked it better).

For 3D my choice would also be OpenGL. Not because of the larger audience (which is nice too, of course), but because I found it easier to use than DirectX. But my knowledge of DirectX is over 6 years old now, so lots of things must have changed there too.

So yeah, lots of ifs and buts and nothing definetive. But hopefully that gives you some ideas what options there are.