Re^3: script optmization

Time to run against a 200 MB file. That is the OP's input file and appending the same 884,873 times to make a 200 MB file.

serial: 12.557 seconds

slurped: 1.644 seconds 7.6 x

parallel: 0.531 seconds 23.6 x

Comment on Re^3: script optmization

Replies are listed 'Best First'.
Re^4: script optmization by Anonymous Monk on May 14, 2017 at 23:27 UTC
For memory constraint systems, one may run using a smaller size for chunk_size. In this case 4 MB will do just fine. For this demonstration, a smaller chunk size value decreases the wait time for workers to read again. It completes in 0.476 seconds for 26.4 x performance increase. `chunk_size => '4m',` [download] The slurped example is likely fast enough. Parallel is nice if you want that. But, serial doesn't take that long either. All completed in less than 20 seconds.	[reply] [d/l]

Replies are listed 'Best First'.

Re^4: script optmization
by Anonymous Monk on May 14, 2017 at 23:27 UTC

For memory constraint systems, one may run using a smaller size for chunk_size. In this case 4 MB will do just fine. For this demonstration, a smaller chunk size value decreases the wait time for workers to read again. It completes in 0.476 seconds for 26.4 x performance increase.

chunk_size => '4m',
[download]

The slurped example is likely fast enough. Parallel is nice if you want that. But, serial doesn't take that long either. All completed in less than 20 seconds.

[reply]
[d/l]