If I understood you correctly, I think you need to implement the concept of a session which is created/used/closed.

Maybe it is sufficient to use the methods $packet->source_ip() and $packet->source_port() of Net::RTP::Packet to compose a unique session ID? If that isn't unique, you need to further inspect the payload.

At the point where you have a session ID, you can demultiplex each stream into a proper file or internal data structure.