Should I learn about any other tools?

You haven't mentioned which tools you know already, so maybe I'm not telling you anything new :)

Anyhow, vmstat would certainly be useful for figuring out if the box is paging (option -p will give details on paging activity). iostat might also be useful (for checking disk I/O, etc.)  To get an idea of what the progam is actually doing (systemcall-wise), I'd use truss first. Dtrace is definitely nice, too, but it might take quite a bit of reading to get going...

Generally, I'd second what stiller said: get more physical RAM, if possible. If your program does need more memory than available (and this is something you can't change), it might well turn out that no amount of system analysing will actually help to make things faster (at least not significantly). Sure, you may have learned a lot in the end, but it's not too unlikely that the program will still be slow... :)