Re: SHA-256? What do you all think of this?

I suspect it's going to be strongly I/O bound, with the sha calculation not adding much time, but you could use a thread pool and hand off the read data blocks to a thread for the SHA calculation/file update steps...

It might be worth profiling it first (or just adding some printf's with timestamps) to see if anything but the reads (or possibly the sha calculation, if that's what takes all the time) takes significant time, though.

Given how large your reads are, you're not going to want more than a few blocks in progress at a time, though, or you'll be soaking up too much RAM.

Mike

Comment on Re: SHA-256? What do you all think of this?

Replies are listed 'Best First'.
Re^2: SHA-256? What do you all think of this? by locked_user erichansen1836 (Sexton) on May 23, 2019 at 18:14 UTC
Okay I'll try that and see if it speeds things up a bit/byte. “The world went and got itself in a big damn hurry.” -- "The SHA-Shank Redemption" #-- Output snippet: 1\|100000000\|d04e015247bf7b97c7687a48ef450c4155947b5772b10f45255b5d6 +cdd3c1e57 2\|100000000\|b36e8bd3719da62b29fb662f4ad0b7ceacdf0216f0459161bd55547 +e2d08f708 3\|100000000\|22c8d79642a7489885546dce83fe79f5d93027205de254f42a3269e +3e72ed0e1 4\|100000000\|38dffbb0997badf1ffcfda22dfa10402609982ba037ad00d293e0fb +9d218b1ce 5\|100000000\|bb0be91d5a969f41d54caf23de5c4d9595dae526f27029d7de8b76f +7cb814207 6\|100000000\|3f633fb8533b98a88912ecb5538e9cadd388668df9be2cf9bbb55ed +d3fe77d89 7\|100000000\|4a5c6f0ddd358163fff067e58eac88cf8dc3979a104f7b10a45f2fa +face1b5a7 8\|100000000\|03ea2a1990d9ebf76219f998437f8d125a96da592b41729f2189e80 +c93b68d2c 9\|100000000\|909899bd2583f374fbd9afb3cbf0b6295fd6be18ad06ddf03a203fc +df4b4d4b3 10\|100000000\|61f975350639d5282c3ea05e31b2cfc54c3014efcd2493cf238db3b +6515d5b5e 11\|100000000\|50101cd70c068b421d73b85c54c2d51962cf1257ffa4d156ee24106 +c6bcfd198 12\|100000000\|d03abf48f5a1fce2343b96f15b1974e4c840fc24330a91c896df2d0 +8ef483124 13\|100000000\|5cf64d35570d0d9304d9b42ba2f5c5d134305da521217b4ce235631 +87b7dc296 14\|100000000\|c180d599cd64077550c026a77a6be517be9313b1680331bca5bcfa6 +aa021ec0f 15\|100000000\|1c41af9bf0f5b997553fb523c3a7c13ae28a428ccac931895acc35e +68e979dd5 [download]

Replies are listed 'Best First'.

Re^2: SHA-256? What do you all think of this?
by locked_user erichansen1836 (Sexton) on May 23, 2019 at 18:14 UTC

Okay I'll try that and see if it speeds things up a bit/byte.

“The world went and got itself in a big damn hurry.” -- "The SHA-Shank Redemption"

#-- Output snippet:

   1|100000000|d04e015247bf7b97c7687a48ef450c4155947b5772b10f45255b5d6
+cdd3c1e57
   2|100000000|b36e8bd3719da62b29fb662f4ad0b7ceacdf0216f0459161bd55547
+e2d08f708
   3|100000000|22c8d79642a7489885546dce83fe79f5d93027205de254f42a3269e
+3e72ed0e1
   4|100000000|38dffbb0997badf1ffcfda22dfa10402609982ba037ad00d293e0fb
+9d218b1ce
   5|100000000|bb0be91d5a969f41d54caf23de5c4d9595dae526f27029d7de8b76f
+7cb814207
   6|100000000|3f633fb8533b98a88912ecb5538e9cadd388668df9be2cf9bbb55ed
+d3fe77d89
   7|100000000|4a5c6f0ddd358163fff067e58eac88cf8dc3979a104f7b10a45f2fa
+face1b5a7
   8|100000000|03ea2a1990d9ebf76219f998437f8d125a96da592b41729f2189e80
+c93b68d2c
   9|100000000|909899bd2583f374fbd9afb3cbf0b6295fd6be18ad06ddf03a203fc
+df4b4d4b3
  10|100000000|61f975350639d5282c3ea05e31b2cfc54c3014efcd2493cf238db3b
+6515d5b5e
  11|100000000|50101cd70c068b421d73b85c54c2d51962cf1257ffa4d156ee24106
+c6bcfd198
  12|100000000|d03abf48f5a1fce2343b96f15b1974e4c840fc24330a91c896df2d0
+8ef483124
  13|100000000|5cf64d35570d0d9304d9b42ba2f5c5d134305da521217b4ce235631
+87b7dc296
  14|100000000|c180d599cd64077550c026a77a6be517be9313b1680331bca5bcfa6
+aa021ec0f
  15|100000000|1c41af9bf0f5b997553fb523c3a7c13ae28a428ccac931895acc35e
+68e979dd5
[download]