I see you are using IP_FREEBIND, but I believe you really need to be using IP_TRANSPARENT (which implies FREEBIND, but is a superset)
You might want to post the iptables commands you have setup. I think the connect will fail if the underlying iptables routing options are not in place.
When dealing with netlink/iptables, never expect any help from the error codes. They are almost always misleading and unhelpful. I speak from experience