in reply to Re: Writing multiple Excel::Writer::XLSX worksheets in parallel (3rd and final attempts)
in thread Writing multiple Excel::Writer::XLSX worksheets in parallel (3rd and final attempts)
Double zipping xlsx files is not necessary. The following override is the minimum needed for the demonstration above. The _prepare_sst_string_data may be omitted when calling $wb->set_optimization().
{ no warnings 'redefine'; sub Excel::Writer::XLSX::Workbook::_store_workbook { my $self = shift; my $tempdir = File::Temp->newdir( DIR => $self->{_tempdir} ); my $packager = Excel::Writer::XLSX::Package::Packager->new(); # Convert the SST strings data structure. $self->_prepare_sst_string_data(); # Package the workbook. $packager->_add_workbook( $self ); $packager->_set_package_dir( $tempdir ); $packager->_create_package(); # Free up the Packager object. $packager = undef; } }
The time for 60 million cells reduced by 16.5 seconds. Thus, reaching 232k cells per second (2.5x for multi-core).
$ time perl write_60mil_parallel.pl AMS ---- DONE. EMEA ---- DONE. APJ ---- DONE. Node List is Done. real 4m18.715s user 11m15.291s sys 0m 3.766s
For comparison, the time is shy of 11 minutes for 1 core at 92k cells per second.
$ time perl write_60mil_serial.pl AMS ---- DONE. APJ ---- DONE. EMEA ---- DONE. Node List is Done. real 10m54.015s user 10m51.045s sys 0m 3.224s
Kind regards, Mario
|
|---|