Security is not an absolute. If the security of this server is important, you should implement multiple layers of protection. This might include any or all of: TCP wrappers, disabling an ssh server, local packet filtering (iptables/ipfw), a firewall protecting the network segment, and armed guards patrolling the facility. If one layer of security fails, you are (hopefully) protected by the other layers. Take a holistic approach. Remember that your application isn't secure if the host isn't secure.