Unless your problems have extra constraints, then refusing to swap more than one at a time will allow you to get stuck in a local minima, and never find the true solution.

For example, if you have a capacity/goal of 10, and your current solution is [6], with leftovers of [1,2,3,4] No single swap will improve your score, but trading your single value for all the leftovers would get you a perfect score.

In a similar way, [5,2] with [4,4] left over, and a max and goal of 8. Swapping the 5 for a 4 makes it worse. Swapping the 2 for a 4 puts you over the max. N-to-1 swaps and 1-to-N swaps also fail. The only way to get the optimal solution without making the intermediate solution worse is a 2-for-2 swap.