The most obvious one is an intermittent restriction of some O/S level resource: open files, available RAM, etc.

Once again hippo you seem to have identified the issue when everything tells me it cannot be that!
Note to self - always listen to hippo's wisdom regardless of how I am interpreting the evidence differently.

Digging deep into cPanel where I've never needed to go before, I have found a graph of CPU usage over the last 24 hours. It shows an error state several times during that time period. The last time roughly corresponds with the last 500 error I got from the site...

So, perhaps the reason I don't see errors on other sites is because this site is doing something much more CPU intensive than the other sites. Some of the other sites get much more traffic. Therefore I have been thinking what is different on this site to any of the other:

1. Perl is in taint mode
2. The database has tables with spatial data types
3. The file structure is laid out differently so modules are above the webroot
4. Scripts are called by the Apache default handler
5. AJAX is used to deliver (sometimes large) JSON files to the browser for Google/Bing maps APIs

Other than that, there are no major differences that I can think of. There are less scripts than most other sites and they are laid out better as they were planned out more thoroughly before any code was written. Nothing has been written that I would expect to be especially resource intensive - there are no big sorts or especially complex maths.

It has to be something that is common to all the scripts as the 500 errors are not limited to always the same script. So modules or something external like the database or the connection to Facebook.

you might consider this as yet another reason to make the move to a VM/VPS instead

Yes indeed. This site is a commercial venture that is starting to bring in revenue. I see it as the generator of the funds to enable me to upgrade web services to use across all the businesses and sites. Everything is a bit crazy right now and having time to properly move forward with a hosting upgrade would be wonderful.