Re^6: IO::Socket::SSL / Net::SSLeay inefficient in non-blocking mode ?

Ok so I did a lot of tests, placing debug timers in various xs and Perl code places and using NYTProf. I think there are two problems:

the Perl code which is executed every time the IO::Socket::SSL::sysread() function is called is not optimized well enough to be executed tens of thousands of times per second or more (mainly when non-blocking mode is used)
there are too many SSLeay functions which need to be called from Perl code every time an unproductive read is performed (Net::SSLeay::read() + loop calling both Net::SSLeay::get_error() and Net::SSLeay::ERR_clear_error())

I managed to workaround the first problem by re-implementing the SSL read logic directly in my Perl code, without calling any IO::Socket::SSL function, just by calling the SSLeay xs functions directly. This allowed me to halve the average execution time of the unproductive reads. Performances still aren't satisfying though, due to the second problem.

I guess the only way to address the second problem would be to re-implement the IO::Socket::SSL::_generic_read() function (or maybe directly the IO::Socket::SSL::sysread() function ?) in xs instead of Perl. It seems it's not possible to call that many external functions from Perl in a loop being executed tens of thousands times per second, without the overhead starting to show and impacting the performances. If the main SSL read function, handling both the read and the error management (updating IO::Socket::SSL internals), was the only one which had to be called from Perl, I think we could get acceptable performance.

Some profiling data before and after optimization:

Before optimization:
Flame graph
Top subroutines
After optimization (bypassing IO::Socket::SSL):
Flame graph
Top subroutines

Comment on Re^6: IO::Socket::SSL / Net::SSLeay inefficient in non-blocking mode ?

Replies are listed 'Best First'.
Re^7: IO::Socket::SSL / Net::SSLeay inefficient in non-blocking mode ? by NERDVANA (Priest) on Jun 27, 2022 at 18:06 UTC
I think you make a pretty good case here for adding a new helper method to SSLeay that would offload some of the work of IO::Socket::SSL into XS. Hopefully the maintainers are intrigued by the problem. In the meantime, assuming you have work to get on with, your application should probably just use SSLeay directly :-) Maybe just have it assume that a undef read is an unproductive read for the first N times it happens, then check the error codes if it continues failing? I'd also like to point out that the flame graph looks like you could save about 10% overhead by calling `$socket->sysread(...)` instead of `sysread($socket, ...)` which is why I suggested that earlier. It's unfortunate that perl's global functions add that much overhead when operating on a user object, but a good incentive to move to OO-style syntax on handles.	[reply] [d/l] [select]
Re^8: IO::Socket::SSL / Net::SSLeay inefficient in non-blocking mode ? by Yaribz (Beadle) on Jun 27, 2022 at 20:06 UTC
your application should probably just use SSLeay directly :-) Maybe just have it assume that a undef read is an unproductive read for the first N times it happens, then check the error codes if it continues failing? Yes, that's exactly what I ended up doing as final optimization in my non-blocking SSL test client :) I was a bit afraid of memory leak due to error data not being pop'ed from the SSLeay structure after they occur, but apparently there's no memory leak. It's still not perfect though, the CPU usage is still much higher than in blocking mode, but at least I no longer have a core with 100% usage during non-blocking 1Gbps SSL transfers. Now I just need to integrate and test this code into my actual application, which is another story... I'd also like to point out that the flame graph looks like you could save about 10% overhead by calling $socket->sysread(...) instead of sysread($socket, ...) which is why I suggested that earlier. It's unfortunate that perl's global functions add that much overhead when operating on a user object, but a good incentive to move to OO-style syntax on handles. That's true, however in my case most of the CPU time was actually wasted in IO::Socket::SSL::sysread(), that's why I didn't notice significant improvements with this change (I still had a CPU core used at 100%).	[reply]