Re^3: Need help in reading csv files >1GB for converting into xlsx files in Solaris 10

Hi! Thanks for your reply ! I have tried this code with my csv file of the size 271MB contains 179K records !! and i got the output like this ! </p

>perl new_cmp.pl
            (warning: too few iterations for a reliable count)
Use of uninitialized value in split at new_cmp.pl line 14, <$fh> line 
+408638.
Use of uninitialized value in split at new_cmp.pl line 14, <$fh> line 
+579912.
Use of uninitialized value in split at new_cmp.pl line 14, <$fh> line 
+751186.
Use of uninitialized value in split at new_cmp.pl line 14, <$fh> line 
+922460.
        s/iter                  c_pp                 perl             
+   c_xs  cbxs
c_pp      45.6                    --                 -85%             
+   -97% -100%
perl      6.71                  579%                   --             
+   -77% -100%
c_xs      1.54                 2853%                 335%             
+     -- -100%
cbxs 2.50e-016 18241099999999997952% 2684799999999998464% 617799999999
+998208%    --
[download]

I dont understand why it is showing lines like 408K - 579K - 751K - 922K ! since my csv file contained the records of 179K and the columns are about #135! :( Please tell me how can i get through this and made it to be done ! Thanks !

Comment on Re^3: Need help in reading csv files >1GB for converting into xlsx files in Solaris 10 - Perl v-5.8.4 Download Code

Replies are listed 'Best First'.
Re^4: Need help in reading csv files >1GB for converting into xlsx files in Solaris 10 - Perl v-5.8.4 by Tux (Canon) on Feb 12, 2015 at 11:10 UTC
Hard to tell from what we see here, but I can only guess that your bench has one or more errors, as cbxs just cannot be that fast (compared to c_xs). So if you want us to give you a better answer than any wild guess possible, you'll need to post your code and your data. update: the high line numbers are because I forgot to close `$fh` at the end of each test. With the close added, here are the new compares: $ perl5.8.4 test.pl Rate c_pp c_xs cbxs perl c_pp 9.63e-02/s -- -94% -97% -98% c_xs 1.65/s 1616% -- -42% -57% cbxs 2.84/s 2845% 72% -- -27% perl 3.88/s 3932% 135% 37% -- $ perl5.20.1 test.pl Use of uninitialized value in split at test.pl line 13, <$fh> line 200 +000. Use of uninitialized value in split at test.pl line 13, <$fh> line 200 +000. Use of uninitialized value in split at test.pl line 13, <$fh> line 200 +000. Use of uninitialized value in split at test.pl line 13, <$fh> line 200 +000. Rate c_pp c_xs cbxs perl c_pp 8.65e-02/s -- -96% -97% -98% c_xs 2.07/s 2295% -- -27% -53% cbxs 2.84/s 3179% 37% -- -35% perl 4.40/s 4980% 112% 55% -- [download] Enjoy, Have FUN! H.Merijn	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^4: Need help in reading csv files >1GB for converting into xlsx files in Solaris 10 - Perl v-5.8.4
by Tux (Canon) on Feb 12, 2015 at 11:10 UTC

Hard to tell from what we see here, but I can only guess that your bench has one or more errors, as cbxs just cannot be that fast (compared to c_xs). So if you want us to give you a better answer than any wild guess possible, you'll need to post your code and your data.

update: the high line numbers are because I forgot to close $fh at the end of each test. With the close added, here are the new compares:

$ perl5.8.4 test.pl
           Rate  c_pp  c_xs  cbxs  perl
c_pp 9.63e-02/s    --  -94%  -97%  -98%
c_xs     1.65/s 1616%    --  -42%  -57%
cbxs     2.84/s 2845%   72%    --  -27%
perl     3.88/s 3932%  135%   37%    --
$ perl5.20.1 test.pl
Use of uninitialized value in split at test.pl line 13, <$fh> line 200
+000.
Use of uninitialized value in split at test.pl line 13, <$fh> line 200
+000.
Use of uninitialized value in split at test.pl line 13, <$fh> line 200
+000.
Use of uninitialized value in split at test.pl line 13, <$fh> line 200
+000.
           Rate  c_pp  c_xs  cbxs  perl
c_pp 8.65e-02/s    --  -96%  -97%  -98%
c_xs     2.07/s 2295%    --  -27%  -53%
cbxs     2.84/s 3179%   37%    --  -35%
perl     4.40/s 4980%  112%   55%    --
[download]

Enjoy, Have FUN! H.Merijn

[reply]
[d/l]
[select]