More impressive analysis from our mysterious anonymonk.

I noticed that new C++ code, supposed to be faster, is actually slower (compared to llil2grt) with "long" dataset from two "official" datasets used in this thread

Good catch! I was so hyper-focused on trying to beat the lowest time for the simple "big1.txt big2.txt big3.txt" test case that most folks were using, I didn't even test the "long1.txt long2.txt long3.txt" test case. :( With a few seconds thought it seems obvious that very long words will favour the hash-based approach over the vector-based one. I'm pleased in a way because my hash-based solution is so much simpler and clearer than my vector-based one ... plus it encourages us to explore different approaches.

I'm happy for folks to continue to claim the fastest time for the short string big1.txt big2.txt big3.txt test case. That's like the 100 metre sprint. We should add a marathon world record for fastest long1.txt long2.txt long3.txt long4.txt long5.txt long6.txt test case. And maybe a 1500 metre world record for big1.txt big2.txt big3.txt long1.txt long2.txt long3.txt. :)

Update: Timings for these three Olympic Events

Run under Ubuntu on my modest laptop against the code here: Interim version of llil3grt.cpp and llil4vec-tbb.cpp.

llil4vec easily won the 100 metre sprint; llil3grt easily won the marathon; llil4vec narrowly won the decider, the 1500 metres.

--------------------- 100 metre sprint -------------------------------
+----

$ ./llil3grt big1.txt big2.txt big3.txt >grt.tmp
llil3grt start
get_properties      CPU time : 2.63733 secs
emplace set sort    CPU time : 1.1302 secs
write stdout        CPU time : 1.17949 secs
total               CPU time : 4.94712 secs

$ NUM_THREADS=6 ./llil4vec-tbb big1.txt big2.txt big3.txt >4vec.tmp
llil4vec-tbb (fixed string length=6) start
use TBB
get properties      time : 0.395106 secs
sort properties     time : 0.325554 secs
emplace set sort    time : 0.701055 secs
write stdout        time : 0.541473 secs
total               time : 1.96334 secs

--------------------------------- 1500 metre -------------------------
+----

$ ./llil3grt big1.txt big2.txt big3.txt long1.txt long2.txt long3.txt 
+ >big-long-grt.tmp
llil3grt start
get_properties      CPU time : 3.13399 secs
emplace set sort    CPU time : 1.0814 secs
write stdout        CPU time : 1.30358 secs
total               CPU time : 5.51907 secs

$ NUM_THREADS=6 ./llil4vec-tbb big1.txt big2.txt big3.txt long1.txt lo
+ng2.txt long3.txt >big-long-4vec.tmp
llil4vec-tbb start
use TBB
get properties      time : 1.05054 secs
sort properties     time : 1.13865 secs
emplace set sort    time : 1.16597 secs
write stdout        time : 0.449603 secs
total               time : 3.80495 secs

---------------------------------- marathon --------------------------
+----

$ ./llil3grt long1.txt long2.txt long3.txt long4.txt long5.txt long6.t
+xt >long-long-3grt.tmp
llil3grt start
get_properties      CPU time : 0.846717 secs
emplace set sort    CPU time : 0.005067 secs
write stdout        CPU time : 0.002561 secs
total               CPU time : 0.854441 secs

$ NUM_THREADS=6 ./llil4vec-tbb long1.txt long2.txt long3.txt long4.txt
+ long5.txt long6.txt >long-long-4vec.tmp
llil4vec-tbb start
use TBB
get properties      time : 2.36002 secs
sort properties     time : 1.28398 secs
emplace set sort    time : 0.118658 secs
write stdout        time : 0.001721 secs
total               time : 3.76457 secs

---------------------------------------------
[download]