If you want to do graphics and sounds, and remain anything resembling portable, the SDL is your best bet.

If you want to do 3D graphics, and remain anything resembling portable, the OpenGL implementations are your best bet.

However, the Perl bindings for both have some issues.

There was a fork of the Perl-SDL effort, which also includes some framework packages to facilitate game development.

I am also working on a library of framework packages, but they're not geared to the same kind of game development that everyone finds in game development books.

Game development has never been a high-level pursuit. Expect to dig in and learn a lot about the lowest levels of the application. Write your own helper classes which will facilitate your game concept.