You might have more luck with Loki's SDL, especially as there are Perl bindings available from the CPAN.

(Now under Unix, writing to devices is a lot easier as they appear to be files in the filesystem, but writing actual drivers for speed-intensive hardware like a 3d card in Perl strikes me as a sign of too much free time. :)

Update: merlyn has just pointed out the shiny new Inline module, which may get you closer to writing drivers in Perl.