prowler has asked for the wisdom of the Perl Monks concerning the following question:

Greetings

I'm currently having some problems generating a large excel sheet using Spreadsheet::WriteExcel, Spreadsheet::WriteExcel::Big is not being any use to me at all.

With the testing data I am using, the file is generated with no obvious problems at the 17985 row mark, but if 17986 rows of data are used, the file is sent (file is generated to STDOUT by an Apache CGI script) as a zero byte file.

Each row is 14 columns wide (the result of a fairly ugly query), the boundary region falls far short of the 65536 row count for excel sheets, the largest working files are not too far shy of the 7MB limit I have seen associated with excel files, and there doesn't seem to be any problems with the actual data that is being used (no weird characters, the break line changes depending on previous lines, ie if I throw away the first line of data, the break zone moves down), so I assume the problem is related to the 7MB limit.

Unfortunately Spreadsheet::WriteExcel::Big isn't playing nicely - it won't generate a usable file with even the first 10 rows of the 17000+ rows of data that were successfully used with standard Spreadsheet::WriteExcel. I have upgraded from 2.11 to 2.15 of S::WE to try and eliminate any bugs that may have been present and subsequently fixed, but the problem still persists.

The relevant code fragment is:

require Spreadsheet::WriteExcel::Big; #1A print $foo->header(-type => 'application/vnd.ms-excel', -attachment => + 'export.xls', -expires => 'now'); binmode(STDOUT); my $export_sheet = Spreadsheet::WriteExcel::Big->new(\*STDOUT) or warn +("Unable to create new Excel document"), do_empty(), exit; #1B my $sheet = $export_sheet->add_worksheet('Support Export'); ##spreadsheet header row and formatting elided## my $row_num=1; my $stop = 17986; #2 $stop = 10; #3 my $count = 0; #$results is an arrayref of hashrefs from a db query foreach(@$results) { $count++; last if $count > $stop; $sheet->write_row($row_num++, 0, [$_->{issue_id}, $_->{issue_name}, $ +_->{contact_name}, $_->{company_name}, $_->{staff_name}, $_->{product +_name}, $_->{issue_status}, $_->{issue_submitted}, $_->{issue_opened} +, $_->{issue_resolved}, $_->{issue_closed}, $_->{billable}, $_->{non_ +billable}, $_->{issue_resolution}],); } $export_sheet->close;

As it is, the generated file is zero bytes. If the ::Big is removed from lines 1A and 1B this will generate a small version of the file, as expected. Removing line 3 will result in the file being zero bytes again. Altering line 2 to 17985 leads to a file of size 6.81 MB (7,143,936 bytes). (these steps are performed in that order)

Q 1)Am I missing something obvious with ::Big? I am following the instructions as I understand them.

Q 2)If, as I suspect, this is simply due to the file not generating once it hits the magic threshold, is there an easy way to handle this? If necessary I could get away with truncating (but I would obviously have to know where to truncate), and giving a message to use more explicit search criteria, but giving the full data set would be preferable.

EDIT: solution found (thanks jmcnamara)
It seems that when the documentation says that OLE::Storage_Lite is a requirement, you should actually pay attention.

Thanks,

Prowler
 - Spelling is a demanding task that requies you full attention.

Replies are listed 'Best First'.
Re: Spreadsheet::WriteExcel and large data sets
by jmcnamara (Monsignor) on Sep 29, 2005 at 15:34 UTC

    There shouldn't be any (practical) upper limit on the size of file that you can create with Spreadsheet::WriteExcel::Big.

    I've generated 50MB files and I know users who have created 100MB+ files.

    I'd recommend a few things. The first is to check that you have OLE::Storage_Lite installed (it is required by ::Big but is an optional install) and that it is the latest version, 0.14.

    The second is to remove Apache from the equation and see if the results are the same. For example, could you be hitting a timeout in Apache?

    Also (perhaps this should have been the first) check your error logs and add 'fatalsToBrowser' if you are using CGI:

    use CGI::Carp 'fatalsToBrowser';

    --
    John.