I would still think that a DB would be prime for this. Access is not too shabby at getting around DB's, and if nothing else, an export process should be able to keep a copy in Access format.

The problem you should be having is not so much disk read time (Although it should be ridiculous) but the actual retrieval of data from a 300 MB variable. I cannot understand why it would be risky to split the transactions. If they are transactions, the should be sequential and without dependency. (Don't you at least have to split the variable within your script??) I would think that even splitting transactions by some other method, like by date, would have some benefit.

If nothing else, sorting your current records and putting them in a DB would help. Then just update the other scripts to write to a DB not a file. If you still have one or two scripts that need the flat-file, do an export on transaction. Less if you need less. Even halfing the disk reads would have to help. A 300 MB dataset is roughly equivalent to 600 copies of Gulliver's travels. I personally panic when datasets get in the 30-40 MB range.

While caching might work, it is not a long term solution. Heck, just maintaining a 300 MB file in ram without swapping in Win2k means you need > than 1 GIG of RAM. If that Dataset grows too large.... Also, if you are noticing alot of HDD thrashing, it is most likely because of Swapping. For a std file read, the process is quick, and one-time. The problem is that as you load that 300 MB dataset into memory, many things need to be swapped out. Windows has a very aggressive swapping system and will swap well before memory is full. Also, you start to contend with growing and shrinking the swap if it is dynamic in size.

If the machine running this has less than 1 GIG of RAM, look at your swapping performance. Another test is to Manually set Virtual Memory to twice RAM and wait to see if the machine carps about out of memory. If so, you need more memory, more swap is a losing proposition.

~Hammy