From the documentation, it looks like that is what you need to do. I would try putting the poll in a separate thread, and passing in the ssh2 filehandle's fileno.

The docs say:

poll ( timeout, arrayref of hashes )
Pass in a timeout in milliseconds and an arrayref of hashes with the following keys:

handle
May be a Net::SSH2::Channel or Net::SSH2::Listener object, integer file descriptor, or perl file handle.

events
Requested events. Combination of LIBSSH2_POLLFD_* constants (with the POLL prefix stripped if present), or an arrayref of the names ('in', 'hup' etc.).

On linux, if the 'in' event fires, and there is no data to be read, then it can be interpreted that the connection is lost. See Net::SSH2 odd polling behavior for example. Also see Net::SSH2 Interactive command example for a similar problem.