I exclude access to all the subdirectories of cgi-bin with a .htaccess file.

Forget to create that file or forget to enable .htaccess files and you are screwed.

Moving libraries (and configuration) out of the webserver's document root and outside any aliased directory not only completely avoids those errors, but is also faster. The webserver does not have to parse .htaccess files. Quoting https://httpd.apache.org/docs/current/howto/htaccess.html#when (Apache v2.4 at time of writing):

In general, you should only use .htaccess files when you don't have access to the main server configuration file. There is, for example, a common misconception that user authentication should always be done in .htaccess files, and, in more recent years, another misconception that mod_rewrite directives must go in .htaccess files. This is simply not the case. You can put user authentication configurations in the main server configuration, and this is, in fact, the preferred way to do things. Likewise, mod_rewrite directives work better, in many respects, in the main server configuration.

[...]

However, in general, use of .htaccess files should be avoided when possible. Any configuration that you would consider putting in a .htaccess file, can just as effectively be made in a <Directory> section in your main server configuration file.

There are two main reasons to avoid the use of .htaccess files.

The first of these is performance. When AllowOverride is set to allow the use of .htaccess files, httpd will look in every directory for .htaccess files. Thus, permitting .htaccess files causes a performance hit, whether or not you actually even use them! Also, the .htaccess file is loaded every time a document is requested.

Further note that httpd must look for .htaccess files in all higher-level directories, in order to have a full complement of directives that it must apply. (See section on how directives are applied.) Thus, if a file is requested out of a directory /www/htdocs/example, httpd must look for the following files:

• /.htaccess
• /www/.htaccess
• /www/htdocs/.htaccess
• /www/htdocs/example/.htaccess

And so, for each file access out of that directory, there are 4 additional file-system accesses, even if none of those files are present. [...]

[...]

In the case of RewriteRule directives, in .htaccess context these regular expressions must be re-compiled with every request to the directory, whereas in main server configuration context they are compiled once and cached. Additionally, the rules themselves are more complicated, as one must work around the restrictions that come with per-directory context and mod_rewrite. [...]

Update:

I usually place only a minimal CGI below document root or in an aliased directory, and have all remaining code and configuration elsewhere, inaccessable to HTTP clients. Something like this (untested):

#!/usr/bin/perl -T

use strict;
use warnings;
use CGI::Carp qw( fatalsToBrowser ); # <-- not always
use lib '/opt/my-app/lib';
use My::App;

My::App->run();
[download]

All remaining code is in My::App or loaded by My::App. A welcome side-effect is that almost all errors occur in modules loaded after CGI::Carp is active, and so I get reasonable error messages in the browser during development.

Alexander

Just found some numbers on .htaccess by nginx:

https://www.nginx.com/resources/wiki/start/topics/examples/likeapache-htaccess/

This may look like comparing Apache vs. nginx, but it is not. It is comparing .htaccess enabled (the Apache case) vs. .htaccess disabled / not implemented (the nginx case). Apache with AllowOverride none should be the same as nginx.

To explain: Each FS stat and each FS read is "expensive" and should be avoided where possible for a fast webserver. .htaccess forces more FS stats and more FS reads, and so the numbers for .htaccess enabled (the Apache case) are way higher than for .htaccess disabled / not implemented (the nginx case).

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

In general, you should only use .htaccess files when you don't have access to the main server configuration file.

This is on shared hosting so I don't have access to the server configuration file!

In general, you should only use .htaccess files when you don't have access to the main server configuration file.

This is on shared hosting so I don't have access to the server configuration file!

That's no reason to stop reading.

Depending on how your shared hosting is set up, there may still be a way to place files outside the reach of the webserver. My shared hoster gives me some filespace somewhere on a unix system, and has a configuration panel that allows to assign any directory of that filespace to any domain hosted there. So I made a few subdirectories and used a subdirectory per domain. Other subdirectories are still unassigned, can not be accessed via HTTP, and thus are perfect for libraries and configuration files.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)