It would let you put perl code into your HTML files,

Where do you get that from? You can of course already do this today (assuming you use a module like Template::Toolkit or Mason), but that has nothing to do with the Apache module backend, it's a language feature. I don't believe mod_perlite has anything to do with templating.

One of the reasons (I think, this is far from my main area of expertise) that mod_php offers better performance than CGI is the fact that things Perl has as modules (e.g. database drivers) are compiled into mod_php itself (that also makes mod_php a pain to install on less popular platforms, but that's not my point). So one thing that may improve performance would be to load modules at Apache startup time (which, as perrin pointed out, you can do in mod_perl and FastCGI), but I don't see that as being a major design goal of mod_perlite. And since their approach is to reinitialise the perl interpreter on every request they would conceivably also have to reload modules, thus removing much of the benefit.