Well, I've tried switching from epoll to poll explicitly in Danga::Socket, and I'm still getting the load issues, only (obviously) this time strace shows (square brackets removed because I can't work out formatting):

poll({fd=10, events=POLLIN}, {fd=9, events=POLLIN}, {fd=11, events=POL
+LIN|POLLERR|POLLHUP|POLLNVAL}, {fd=12, events=POLLIN|POLLERR|POLLHUP|
+POLLNVAL}, {fd=15, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=14, e
+vents=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=18, events=POLLIN|POLLERR
+|POLLHUP|POLLNVAL}, {fd=13, events=POLLIN|POLLOUT|POLLERR|POLLHUP|POL
+LNVAL, revents=POLLOUT}, {fd=16, events=POLLIN|POLLERR|POLLHUP|POLLNV
+AL}, {fd=19, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, 10, -1) = 1
poll({fd=10, events=POLLIN}, {fd=9, events=POLLIN}, {fd=11, events=POL
+LIN|POLLERR|POLLHUP|POLLNVAL}, {fd=12, events=POLLIN|POLLERR|POLLHUP|
+POLLNVAL}, {fd=15, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=14, e
+vents=POLLIN|POLLERR|POLLHUP|POLLNVAL}, {fd=18, events=POLLIN|POLLERR
+|POLLHUP|POLLNVAL}, {fd=13, events=POLLIN|POLLOUT|POLLERR|POLLHUP|POL
+LNVAL, revents=POLLOUT}, {fd=16, events=POLLIN|POLLERR|POLLHUP|POLLNV
+AL}, {fd=19, events=POLLIN|POLLERR|POLLHUP|POLLNVAL}, 10, -1) = 1
[download]

etc. Oh funky. Oh well - at least that's one part of the equation marked off...

Time to start chasing the SSL options....