How to slim it down ?

Try stripping the symbols from it:

strip -s your.so
[download]

Often, but not always, perl is built with a cflags setting that ensures that the shared object is automatically stripped.

Cheers,
Rob