It looks like you've already got the the optional arguments handled correctly in your subroutine, but you're passing it a hash reference where it expects a plain list. Change your subroutine invocation to:

StartCapture( 
  twigs   => {filter => $filterTwig,  capture => $captureTwig},
  hashes  => {filter => \%filterHash,  capture => \%captureHash},
  files   => {filter => 'filter.txt', capture => 'capture.txt'}, 
  results => 'c:/temp.txt' 
            );
[download]

Notice that I also changed %captureHash and %filterHash to \%captureHash and \%filterHash so they are passed as references instead of flattening into lists.

To access nested elements, you can do something like $args{twigs}{filter}. For more, see perlreftut and perldsc.