To get around this problem, I started building up a library of non-work related code that I've done at home on my own time as a sort of portfolio I could show to other potential employers (should the need arise). The only problem with this approach is that it takes time to develop anything worthwhile

In your case you are able to do this. Problems occur where people sign-on with companies that stipulate that we own everything:

'...all code, ideas and design at work and home become the property of the company...' .

Does everyone get a legal opinion on the contracts or new jobs they sign up for?

This is a disadvantage of working for propriortry software companies that stupulate such rules. Of course on the other hand IP is exactly what companies see as a $$$ value in software and the reason they hire in the first place. I guess your solution works for you.

Another way could be with your specific domain knowledge create a CPAN module and become the code owner with GPL over the code. Then negotiate a license over the code for commercial use and enhancements.