$ NUM_THREADS=24 NUM_MAPS=96 ./llil4tkh2 big{1,2,3}.txt | cksum llil4tkh2 (fixed string length=12) start sharding managed by the application use OpenMP use boost sort get properties 0.446 secs 23.645 mil QPS hashDBMs to vector 0.354 secs vector stable sort 0.081 secs write stdout 0.210 secs total time 1.092 secs count lines 10545600 count unique 10367603 2956888413 93308427 # Results for 26 big files: $ NUM_THREADS=24 NUM_MAPS=96 ./llil4tkh2 in/biga* | cksum llil4tkh2 (fixed string length=12) start sharding managed by the application use OpenMP use boost sort get properties 3.507 secs 26.051 mil QPS hashDBMs to vector 1.777 secs vector stable sort 0.665 secs write stdout 1.532 secs total time 7.483 secs count lines 91395200 count unique 79120065 2005669956 712080585 $ NUM_THREADS=48 NUM_MAPS=128 ./llil4tkh2 in/biga* | cksum llil4tkh2 (fixed string length=12) start sharding managed by the application use OpenMP use boost sort get properties 2.335 secs 39.141 mil QPS hashDBMs to vector 1.410 secs vector stable sort 0.677 secs write stdout 1.555 secs total time 5.979 secs count lines 91395200 count unique 79120065 2005669956 712080585 # One billion+ lines (312 big files) $ NUM_THREADS=48 NUM_MAPS=128 ./llil4tkh2 \ in/biga* in/biga* in/biga* in/biga* in/biga* in/biga* \ in/biga* in/biga* in/biga* in/biga* in/biga* in/biga* \ | cksum llil4tkh2 (fixed string length=12) start sharding managed by the application use OpenMP use boost sort get properties 24.295 secs 45.143 mil QPS hashDBMs to vector 1.410 secs vector stable sort 0.644 secs write stdout 1.439 secs total time 27.790 secs count lines 1096742400 count unique 79120065 3625599930 791200650