Without seeing your code, it will be very hard to suggest things on how to make it do what you want.
You have discarded all the obvious things that would make it easier, because you say that you really need this.
Ideally, you show us some minimal code that reproduces the problem so that we can run it ourselves. For example, the following could be a start:
#!perl
use strict;
use warnings;
my $memory_eaten = 8 * 1024 * 1024 * 1024; # 8GB, adjust to fit
my %memory_eater = (
foo => scalar( " " x $memory_eaten ),
);
my $cmd = "foo bar";
system($cmd) == 0
or die "Couldn't launch '$cmd': $!/$?";
Updated: Actually make the hash eat memory by creating a loong string