I think there are some design problems here. First, why would you use the most complicated model, threads instead of separate processes for each port?

This requirement: I have to use UDP and can't afford to miss anything is incompatible with UDP. UDP is a "connectionless", non-error checked protocol. Sure there is going to be data loss - this is inevitable when transferring lots of data. One of my colleagues measured this on one network at 10% packet loss. With UDP there is no "end to end" error checking or retransmission - this can be like shouting into a crowded sports arena.

Update: I see that BrowserUk hit the send button on a better response than mine. His questions are excellent. Independent of these questions, with UDP some amount of data loss is going to happen and this fundamental requirement of "can't miss anything" cannot be satisfied no matter what when using UDP as the base protocol. Just cannot happen.