If you are using https, then there is no problem. All you would need to do is setup a program to decrypt your files on the server, then send it out to the browser with https. You still would have to limit access with a login, where you could use GPG keys auth, or just a plain password authentication, but you say you have that covered.

On the server, the first method to decrypt which comes to mind is a GPG pipe. See GnuPG tie to gpg binary gives broken pipe error in CGI output and he gives a working code example.