I don't know if you can do this from IPC::Shareable, but you probably need to do something like this:

# Deallocate the shared memory segment when exiting program.
  shmctl ($segment_id, IPC_RMID, 0);
  $sem->remove;
[download]

update
From the IPC::Shareable docs, it says for the ->destroy method: destroy
If set to a true value, the shared memory segment underlying the data
binding will be removed when the process calling tie() exits
(gracefully)3. Use this option with care. In particular you should
not use this option in a program that will fork after binding the
data. On the other hand, shared memory is a finite resource and should
be released if it is not needed. The default is false