From a quick look at your code, my first guess is that the Apache process is running out of memory, because of the way you're getting data from the database. The selectall-arrayref call is going to suck up lots of memory doing the data retrieval that way.

A better method would be to get the data, and then print it, just one row at a time.

Thanks Alex

Alas, though plausible at first, it turns out that that wasn't it. I modified the script in question to get the data one row at a time, and the same limit is hit. The file actually received is precisely the same that I get when I use selectall_arrayref.

In retrospect, if apache is able to use all free memory on the server, there should be no such limit. After all, there are several GB RAM available, and even a ridiculously large recordset from the DB would only be a few MB. We're talking a 64 bit Windows server here, with a 64 bit build of Apache's web server, and a 64 bit build of MySQL. If any of those server products have trouble with even a few MB of data, then there is a serious problem with it. And the perl we're using (v 5.16) is a 64 bit build, so there ought not be a problem there. But that constraint is so consistent, it must be a constraint imposed somewhere: but where do I look?

Thanks again.

Ted

but where do I look?

In the apache error log. If there's nothing there, start peppering your script with debugging statements until you track it down.

Also look in the apache access log which will show the total size of the response body. It may be the case that the web server is sending the data fine, but something else in the network (or on the Windows box itself) is cutting it off.

Just a side note: you are generating both invalid and probably even unparseable CSV. In the header line, you are including spaces after the comma, causing the header fields to be seen as e.g.  Amount ("<space>Amount"). I cannot believe that to be intentional. Inside the loop you just spit out all fields joined on ,. Please try to think what will happen if any of the dat contains a comma. Having amounts makes that highly likely. The US uses $ 2,000,000.00 other countries might use € 2.000.000,00. Try to think about your data consumer. Please use a CSV generator like Text::CSV_XS which makes your code not only correct, but also removes the need for extra undefined checks.

print $cgi->header (
    -type       => "text/csv",
    -charset    => "utf-8",
    -attachment => "data.csv",
    );
my $csv = Text::CSV_XS->new ({ binary => 1, eol => "\r\n" });
$csv->print (*STDOUT, [ "Platform ID", "Processor ID",
    "Transaction ID", "Date", "Settlement Date",
    "Transaction type", "Status", "Amount", "First Name",
    "Last Name", "Address1", "Address(cont.)", "City",
    "State or Province", "Postal Code", "Country", "Phone",
    "Email", "Processor response code", "MSC response code",
    "IP Address", "CC Number", "Descriptor" ]);
foreach my $v (@a) {
    my ($m_id, @r) = @$v;
    $r[6] = $r[6] ? "succeeded" : "failed";
    $csv->print (*STDOUT, [ @r, $descriptors{$m_id}{$r[0]} ]);
    }
$dbh->disconnect;
[download]

Thanks for this.

Point noted. In this particular case, the data is not formatted. Rather, it is a straight dump of data from the DB. But, I will have to take a look at that, to ensure that the CSV file MySQL produces is valid. The amount field won't be a problem as it is an unformatted decimal number, but one of the address fields may well contain commas. IIRC, MySQL allows one to add a clause that encloses all fields in single or double quotes, which would prevent problems if one field or another includes commas as a part fo the data.

Thanks

Ted

How bit is the file, how many bytes total?

Why are you duplicating this array? my @a = @$a_ref;

whatever limits you're hitting, they're made by your operating system + webserver