If you already have a copy of the server cert and the CA chain that signed it (public keys all), you simply need to verify on the client:

$socket = IO::Socket::SSL->new (
   PeerAddr   => $ip,
   PeerPort   => $port,
   Proto      => 'tcp',
   SSL_version   => 'SSLv3',
   SSL_verify_mode => 0x02
);
[download]

The above will fail if the client does not already have a copy of the server's public key in whatever directory your openssl.cnf file says they should reside. If you need to do additional verifications on your own, use SSL_verify_callback to specify your own additional code.

If you were looking for some way of setting up a server without having it even send a public key, you're no longer talking SSL. Consider the Crypt:: modules in that case ...

Updated: It would have been polite for you to mention that your update occurred after I had posted this ...