I don't think it's swapping. The process takes about 70 MB and there are still at least 100 MB left. The $N=500_000 process with sort takes less memory than the $N=5_000_000 process with no sort, but the latter is faster.