Recently, I tried NVIDIA's nvc++ compiler and noticed a compile time regression (> 40 seconds). That's a long time. I reached out to NVIDIA. Though, no resolution from their side. Some time later, I saw another regression upgrading GCC from version 13 to 14. It seems that std::mutex regressed.
That spawned a chain-reaction. I logged-appended the March and April 2024 events to my summary page.
A spinlock mutex class resolved both issues, further improving performance. I reached out to Greg, author of the phmap C++ library. The LLiL challenge is interesting. He shared tips plus the string_cnt struct, accommodating fixed-length and variable-length keys.
Clang performance regression due to GCC 14
phmap: Ability to reset the inner sub map
string_cnt struct commit by Greg
eyepopslikeamosquito's llil3vec.cpp inspiration (scroll down the page), for making the vector version llil4vec fast (though gobbles memory for unlimited length strings), is the reason efforts to making the map variants catch up, while keeping memory consumption low. There are several map variants. The llil4hmap, llil4emh, and llil4umap demonstrations compute the key hash_value once only, and stores it with with the key.
llil4map sub maps managed by the C++ library, using phmap::parallel_flat_hash_map
llil4map2 memory efficient version, vector of pointers to the map key-value pairs
llil4hmap sub maps managed by the application, using phmap::flat_hash_map
llil4emh sub maps managed by the application, using emhash7::HashMap
llil4umap sub maps managed by the application, using std::unordered_map
Greg Popovitch added a clever new type string_cnt, supporting fixed-length and long strings without recompiling. See enhanced llil4map found in his examples folder, also memory efficient. Note: I beautified the include directives in my demonstrations, matching your version for consistency.
I glanced through the long thread by eyepopslikeamosquito. At the time, we were thrilled completing in less than 10 seconds. More so below 6 seconds. Fast forward to early 2024, the llil4map demonstration completes in 0.3 seconds, due to linear scaling capabilities (all levels parallel). The llil4vec example may run faster, but stop improving beyond more CPU cores.
We reached the sub-second territory, processing three input files.
Limit 12 CPU Cores
$ NUM_THREADS=12 ./llil4map big{1,2,3}.txt | cksum llil4map (fixed string length=12) start use OpenMP use boost sort get properties 0.311 secs map to vector 0.055 secs vector stable sort 0.095 secs write stdout 0.036 secs total time 0.500 secs count lines 10545600 count unique 10367603 2956888413 93308427 $ NUM_THREADS=12 ./llil4vec big{1,2,3}.txt | cksum llil4vec (fixed string length=12) start use OpenMP use boost sort get properties 0.170 secs sort properties 0.061 secs vector reduce 0.017 secs vector stable sort 0.065 secs write stdout 0.047 secs total time 0.362 secs count lines 10545600 count unique 10367603 2956888413 93308427
No Limit
$ ./llil4map big{1,2,3}.txt | cksum llil4map (fixed string length=12) start use OpenMP use boost sort get properties 0.101 secs map to vector 0.052 secs vector stable sort 0.115 secs write stdout 0.029 secs total time 0.298 secs count lines 10545600 count unique 10367603 2956888413 93308427 $ ./llil4vec big{1,2,3}.txt | cksum llil4vec (fixed string length=12) start use OpenMP use boost sort get properties 0.203 secs sort properties 0.088 secs vector reduce 0.024 secs vector stable sort 0.103 secs write stdout 0.029 secs total time 0.449 secs count lines 10545600 count unique 10367603 2956888413 93308427
Many thanks eyepopslikeamosquito for being there. I promise no more messages for a while. You inspired C++ in me. Next is Taichi Lang.
Greg Popovitch shared a tip for releasing memory immediately, during "map to vector". Thank you. That was helpful also, for llil4umap.
Blessings and grace,
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re: [OT] The Long List is Long resurrected
by eyepopslikeamosquito (Archbishop) on Apr 07, 2024 at 12:04 UTC | |
by marioroy (Prior) on Apr 08, 2024 at 00:46 UTC | |
|
Re: [OT] The Long List is Long resurrected
by marioroy (Prior) on Apr 16, 2024 at 04:51 UTC | |
by marioroy (Prior) on Apr 23, 2024 at 14:56 UTC | |
by marioroy (Prior) on Jul 26, 2024 at 21:17 UTC | |
|
Re: [OT] The Long List is Long resurrected
by cavac (Prior) on Aug 09, 2024 at 14:17 UTC | |
by marioroy (Prior) on Aug 09, 2024 at 21:36 UTC |