536711168 is pretty close to 512M, which (IIRC) is some default memory resource limit on FreeBSD.

What do ulimit -a and limits show?

What changed between when this last worked and when it broke? The answer is not "nothing."

Hi planetscape, the script runs through thousands of directories finding any images and stores the paths inside an array.

It then runs through this huge array and, for each image, extracts the IPTC information and writes it to a text file (one image per line).

The script breaks when it's performing this second step, and the text file it was writing reaches a size of 151 MB by the point it breaks.

Nothing has changed with the script itself, but the image archive (directories) has grown in size.

How about not storing all the paths into this huge array but processing each path immediately?

If this is not possible because you need to do some sorting, how about presorting and storing the paths into (for example) 5-10 files and then reading each file separately.

Some questions:

Why are you storing the paths in an array? Why not process the files as you come across them, or at least use a hash to avoid tons of data repetition?
Also, how much are you burning to load the image files? Are some of your images in the 100's of Megs range? Could you seek instead of slurp?

If you can show some code it would pre-answer a lot of questions like these.

1. I think you need to change the extension of ur image file to either JPG for processing the file or

2. possibly since it is going out of memory then try using less expensive methods to read the file in the directory i.e. rather than reading the entire file into an array read it line by line. I am not sure how it works with image file but I am sure there will be some methods to do that.

3. empty the arrays or hash or the any data structure whichever you are using after using it rather than over witing it.

Good Luck !!

1. I understand what desemondo (Friar) is saying but, my intention was that we can reduce the time for doing the manipulations with the file by converting into JPG and see if it is working fine or not. 2. Empty the data structures means that in case you are doing the push or shift operations then it will take a lot of time as compared to using the same in smaller parts.

" I think you need to change the extension of ur image file to either JPG for processing the file "
Why do you think that? Isn't a file just a file, even on Windows? All the extension does is allow windows to know what viewer/editor/or program to associate and/or launch when you double click that file? I don't see how the extension could affect reading the file...

"empty the arrays or hash or the any data structure whichever you are using after using it rather than over witing it."
Is this what you mean by that?

my @images = qw(file1.png file2.png file3.png);
@images = ();
@images = qw(file4.png file5.png file6.png);
[download]

If so, why is that better than this?

my @images = qw(file1.png file2.png file3.png);
@images = qw(file4.png file5.png file6.png);
[download]


Update:
Perhaps you meant that you need to empty your data structure periodically if you are pushing or unshifting onto it, as it would continue to grow otherwise?