$ NUM_THREADS=24 NUM_MAPS=96 ./llil4tkh big{1,2,3}.txt | cksum llil4tkh (fixed string length=12) start sharding managed by the tkrzw::ShardDBM library use OpenMP use boost sort get properties 0.564 secs 18.698 mil QPS shardDBM to vector 0.352 secs vector stable sort 0.078 secs write stdout 0.206 secs total time 1.202 secs count lines 10545600 count unique 10367603 2956888413 93308427 # Results for 26 big files: $ NUM_THREADS=24 NUM_MAPS=96 ./llil4tkh in/biga* | cksum llil4tkh (fixed string length=12) start sharding managed by the tkrzw::ShardDBM library use OpenMP use boost sort get properties 4.355 secs 20.986 mil QPS shardDBM to vector 1.789 secs vector stable sort 0.667 secs write stdout 1.577 secs total time 8.389 secs count lines 91395200 count unique 79120065 2005669956 712080585 $ NUM_THREADS=48 NUM_MAPS=128 ./llil4tkh in/biga* | cksum llil4tkh (fixed string length=12) start sharding managed by the tkrzw::ShardDBM library use OpenMP use boost sort get properties 2.858 secs 31.979 mil QPS shardDBM to vector 1.412 secs vector stable sort 0.663 secs write stdout 1.553 secs total time 6.488 secs count lines 91395200 count unique 79120065 2005669956 712080585 # One billion+ lines (312 big files) $ NUM_THREADS=48 NUM_MAPS=128 ./llil4tkh \ in/biga* in/biga* in/biga* in/biga* in/biga* in/biga* \ in/biga* in/biga* in/biga* in/biga* in/biga* in/biga* \ | cksum llil4tkh (fixed string length=12) start sharding managed by the tkrzw::ShardDBM library use OpenMP use boost sort get properties 28.506 secs 38.474 mil QPS shardDBM to vector 1.456 secs vector stable sort 0.645 secs write stdout 1.453 secs total time 32.062 secs count lines 1096742400 count unique 79120065 3625599930 791200650