Problem with Tie::File

psandie has asked for the wisdom of the Perl Monks concerning the following question:

I'm using ActiveState Perl 5.8.8 for MSWin32-x86-multi-thread,and appear to be having problems with Tie::File. I am using the module as part of some code which compares two csv files. Both files are preprocessed prior to the comparision, which is ultimately just a file diff. I tie both files and process the arrays as required, then write out processed copies of the original files from the tied data - and diff these. The problem I have is that for relatively small files (a few 100kB) this is fine, but when the files are larger (a few MB) the processed files appear to have been only partially processed. i.e. the code that processes the arrays from the tied files has executed, but the untied file appears to have been only partially processed.

Comment on Problem with Tie::File

Replies are listed 'Best First'.
Re: Problem with Tie::File by matze77 (Friar) on Feb 09, 2009 at 11:07 UTC
Just to be sure: You verified that the files have been completely processed and it is not taking very long caused by the size? Linking on PerlMonks Writeup Formatting Tips MH	[reply]
Re^2: Problem with Tie::File by psandie (Initiate) on Feb 09, 2009 at 11:17 UTC
Yep the arrays are definitely fully processed - I put debug in to make sure. It does take a long time and uses up a lot of memory and processing power. I even switched autodefer off but that hasn't helped.	[reply]
Re: Problem with Tie::File by jdporter (Paladin) on Feb 09, 2009 at 13:31 UTC
Show some code. The minimal complete program necessary to reproduce the problem. Thanks.	[reply]
Re^2: Problem with Tie::File by psandie (Initiate) on Feb 09, 2009 at 13:44 UTC
Unfortunately that's not so easy. The processing of the arrays is fairly extensive (runs to hundreds of lines). I had hoped what I'd put might just ring a bell with somone who'd seen similar problems. The fact that it works with small files but not large ones perplexes me. It may be that I'm running into memory problems I guess. I don't know the inner workings of the Tie code but I suppose if I tie large files I'm ultimately using a lot of memory.	[reply]
Re^3: Problem with Tie::File by jdporter (Paladin) on Feb 09, 2009 at 14:05 UTC
I'm not asking you to post your production code, unaltered. You can replace your "processing" (whatever that is; you don't say) with some kind of no-op. That's why I said "minimal complete". Strip it down. Please. I'd like to try to help. Also, please make sure you have read and understand the Tie::File Caveats. Between the mind which plans and the hands which build, there must be a mediator... and this mediator must be the heart.	[reply]
Re^4: Problem with Tie::File by psandie (Initiate) on Feb 09, 2009 at 15:27 UTC
Re^5: Problem with Tie::File by jdporter (Paladin) on Feb 11, 2009 at 12:23 UTC
Some notes below your chosen depth have not been shown here
Re^3: Problem with Tie::File by ww (Archbishop) on Feb 09, 2009 at 14:16 UTC
Super_Search could have helped you here: http://search.cpan.org/~timb/Devel-NYTProf-2.07/lib/Devel/NYTProf.pm Profiling your code. Trying to profile my app's memory usage Re: Profiling your code.	[reply]
Re^3: Problem with Tie::File by matze77 (Friar) on Feb 11, 2009 at 09:49 UTC
Maybe if the files are larger there are "conditions" that apply which are not foreseen and the code "breakes (the programs dies e.g.)" at these larger files, but i am not experienced enough to give you detailed hints what to search for, as already mentioned maybe memory. Perl is only limited by your (computing) ressources there is no "hard" limit which is set by Perl ... Perhaps printing debugging lines could help to see where it is broken or printing the contents of variables/arrays into a log-file but i really dont know the best practices on this ... hth MH	[reply]