Re^2: Using ramfs to process files

Sometimes when I post, it makes sense in my head.

My files are gzipped binary files. I have to send them through another program to get them into the text format. The issue is that after processing they are very large and it is easier to break my task into mutiple scripts. Keeping the file in memory is cumbersome.

I haven't had time to write something to process the binary format directly, but that is probably next on the list.

My question was more about possible problems with using ramfs.

Comment on Re^2: Using ramfs to process files

Replies are listed 'Best First'.
Re^3: Using ramfs to process files by Tanktalus (Canon) on Feb 03, 2005 at 18:04 UTC
I'm going to pretend for a second that you're in control of the conversion program, or that the conversion program is a well-thought-out program. In the ideal world, you'd simply extend the pipe: `open my $fh, "gunzip -c $filename \| convert_to_text - \|";` [download] Thus, gunzip would uncompress as much as it could before getting blocked on a pipe to convert_to_text which would read the binary from stdin, and write the text to stdout, both of which would block when empty/full, respectively, and your script could deal with the text on the way through. Back to reality ... in the more common case, I see nothing wrong with ramfs - as long as it works and doesn't corrupt your data, same as any other filesystem. You just have to be careful that you're not wasting ram on stuff that could make your system faster in other ways. For example, if that ram were instead used for physical filesystem caching, you'd get almost the same effect, and ram that wasn't in use for caching (vs ramfs) could still be used for your programs.	[reply] [d/l]

Replies are listed 'Best First'.

Re^3: Using ramfs to process files
by Tanktalus (Canon) on Feb 03, 2005 at 18:04 UTC

I'm going to pretend for a second that you're in control of the conversion program, or that the conversion program is a well-thought-out program.

In the ideal world, you'd simply extend the pipe:

  open my $fh, "gunzip -c $filename | convert_to_text - |";
[download]

Thus, gunzip would uncompress as much as it could before getting blocked on a pipe to convert_to_text which would read the binary from stdin, and write the text to stdout, both of which would block when empty/full, respectively, and your script could deal with the text on the way through.

Back to reality ... in the more common case, I see nothing wrong with ramfs - as long as it works and doesn't corrupt your data, same as any other filesystem. You just have to be careful that you're not wasting ram on stuff that could make your system faster in other ways. For example, if that ram were instead used for physical filesystem caching, you'd get almost the same effect, and ram that wasn't in use for caching (vs ramfs) could still be used for your programs.

[reply]
[d/l]