Given the disparate processes needing access, and that you need read and write access. If you have the memory to maintain 300MB in RAM, then a ramdisc is by far your easiest and sanest option.

Ram discs have minimal overhead, and the only change required by the applications would be for the paths to change. Even that could be eliminated if you can use a symbolic link. The only downside is the risk of data loss in the event that the server crashes.

There are several ways that you could approach mitigating the data loss, some simplistic, other quite involved depending on the level of reliablility to you need.

An interesting thought, but I emphasis its nothing more than a thought, would be to create a partition/filesystem on disc the same size as the ram disc, and use mirroring to reflect the ramdisc onto the filesystem. Whether there is any mileage in the idea depends on which os, mirroring software etc and whether the latter can be configured to use a ram disk.

Attempting to use any form of caching is likely to be a problem as caching only really benefits you if the same sections of the data are being repeatedly accessed. Given all the different processes that would be vying for cache space in your scenario, your likely to slow things down with cache thrash rather than speed them up.

I would still think that a DB would be prime for this. Access is not too shabby at getting around DB's, and if nothing else, an export process should be able to keep a copy in Access format.

The problem you should be having is not so much disk read time (Although it should be ridiculous) but the actual retrieval of data from a 300 MB variable. I cannot understand why it would be risky to split the transactions. If they are transactions, the should be sequential and without dependency. (Don't you at least have to split the variable within your script??) I would think that even splitting transactions by some other method, like by date, would have some benefit.

If nothing else, sorting your current records and putting them in a DB would help. Then just update the other scripts to write to a DB not a file. If you still have one or two scripts that need the flat-file, do an export on transaction. Less if you need less. Even halfing the disk reads would have to help. A 300 MB dataset is roughly equivalent to 600 copies of Gulliver's travels. I personally panic when datasets get in the 30-40 MB range.

While caching might work, it is not a long term solution. Heck, just maintaining a 300 MB file in ram without swapping in Win2k means you need > than 1 GIG of RAM. If that Dataset grows too large.... Also, if you are noticing alot of HDD thrashing, it is most likely because of Swapping. For a std file read, the process is quick, and one-time. The problem is that as you load that 300 MB dataset into memory, many things need to be swapped out. Windows has a very aggressive swapping system and will swap well before memory is full. Also, you start to contend with growing and shrinking the swap if it is dynamic in size.

If the machine running this has less than 1 GIG of RAM, look at your swapping performance. Another test is to Manually set Virtual Memory to twice RAM and wait to see if the machine carps about out of memory. If so, you need more memory, more swap is a losing proposition.

~Hammy