It turns out that making this broadcast properly over multiple images was impressively hard.
which had a hardcoded limitation that it would only work on a 1-dim ndarray (fixed by imitating PDL::VectorValued's strategy of having a "how many set for each vector" output ndarray).
Then actually using that with minmaximum also posed a problem, since the number of values per image might vary (in other words, the outputs might be ragged) - fixed that by having which fill the unset index-values with -1, and then setting that as the badvalue for the output ndarray, so minmaximum would ignore them.
After that, the rest was easy! The new PDL::Image2D::crop works on a mask, so the above snippet would call
$x1x2y1y2 = ($img->slice('(3)') != 0)->crop;