Actually, Apache has two ways of doing it:

ScriptAlias /cgi-bin/ "/path/to/cgi-bin/"
[download]

will cause all files accessed via /cgi-bin/ in the URI to be executed as CGI processes, as the OP theorized.

AddHandler cgi-script .cgi
[download]

will cause all files whose names end with .cgi to be executed as CGI processes, regardless of their location. This could easily be modified to cause CGI execution of, say .pl files if a server admin so desired.