Re^2: How does the while works in case of Filehandle when reading a gigantic file in Perl

@perlholic has posted my code, and I have checked the timings for each step separately but all are linear. The problem lies with 'while' because for every new line the time requirements go up

Comment on Re^2: How does the while works in case of Filehandle when reading a gigantic file in Perl

Replies are listed 'Best First'.
Re^3: How does the while works in case of Filehandle when reading a gigantic file in Perl by Corion (Patriarch) on Feb 06, 2015 at 12:43 UTC
There is nothing particular with a `while(<>){` loop that would make the time go up linearly. Maybe some of your processing is consuming memory or accumulating data in an array that gets larger and larger without ever getting cleared. We will need to see more accurate code than the reduced version that was posted. If the time taken per line gets larger and larger, maybe you can post a small/short XML example and some code more to the point so we can try to replicate the problem? Please also make sure that the problem appears with the code you post. If you do other work, like for example, inserting the data into a database instead of writing it to a file, that could get slower with each new row that gets added.	[reply] [d/l]

Replies are listed 'Best First'.

Re^3: How does the while works in case of Filehandle when reading a gigantic file in Perl
by Corion (Patriarch) on Feb 06, 2015 at 12:43 UTC

There is nothing particular with a while(<>){ loop that would make the time go up linearly. Maybe some of your processing is consuming memory or accumulating data in an array that gets larger and larger without ever getting cleared. We will need to see more accurate code than the reduced version that was posted.

If the time taken per line gets larger and larger, maybe you can post a small/short XML example and some code more to the point so we can try to replicate the problem? Please also make sure that the problem appears with the code you post.

If you do other work, like for example, inserting the data into a database instead of writing it to a file, that could get slower with each new row that gets added.

[reply]
[d/l]