Updating File from Other Files

artist has asked for the wisdom of the Perl Monks concerning the following question:

I have a large destination file which often needs to be updated from the data from other source files. Source files changes with external processes. There could be few source files changes out of the lot. (ex.. 3 out of 200) Changes in source files does not necessarily happens at the end of the file. How do I update the destination file from the source files without concating all of them?
Thanks.

Comment on Updating File from Other Files

Replies are listed 'Best First'.
Re: Appending File from Other Files by Random_Walk (Prior) on Nov 29, 2004 at 17:54 UTC
If your source files are contstant size you can do this reasonably. Otherwise you have a problem as a file is just sequential bits. `# source files aaa bbbb cc ddd # concat file aaabbbbccddd # replace constant size 2nd file with xxxx aaaxxxxccddd # but if we replace it with a shorter or longer file aaaxxxbccddd #whoops aaaxxxxxcddd #oh dear` [download] You may be able to do some optimisation by not re-writing the part of the file before the changes, if you have freedom over the order the data is in perhaps you can move the most frequently changed files to the end of the stream so you are often not re-writing most of the file. If you also own the reader script you have more options open, a couple spring to mind: Replace file with linked list allow null values in the file Read direct from the source files Cheers, R.	[reply] [d/l]

Replies are listed 'Best First'.

Re: Appending File from Other Files
by Random_Walk (Prior) on Nov 29, 2004 at 17:54 UTC

If your source files are contstant size you can do this reasonably. Otherwise you have a problem as a file is just sequential bits.

# source files
aaa
bbbb
cc
ddd

# concat file
aaabbbbccddd

# replace constant size 2nd file with xxxx
aaaxxxxccddd

# but if we replace it with a shorter or longer file
aaaxxxbccddd #whoops
aaaxxxxxcddd #oh dear
[download]

You may be able to do some optimisation by not re-writing the part of the file before the changes, if you have freedom over the order the data is in perhaps you can move the most frequently changed files to the end of the stream so you are often not re-writing most of the file.

If you also own the reader script you have more options open, a couple spring to mind:

Replace file with linked list
allow null values in the file
Read direct from the source files

Cheers,
R.

[reply]
[d/l]