Re^3: Perl solution for storage of large number of small files

So, do you actually recommend sqlite as storage for binary data?

Well, I don't recommend neither disrecommend it. I was only suggesting you should try another backend!

Which database is the best for a given problem, does not depend exclusively on the data structures but also on the usage pattern.

Anyway, if you need to access 2GB of data randomly, there is probably nothing you can do to stop disk trashing other than adding more RAM to your machine, so that all the disk sectors used for the database remain cached.

Comment on Re^3: Perl solution for storage of large number of small files

Replies are listed 'Best First'.
Re^4: Perl solution for storage of large number of small files by andye (Curate) on Apr 30, 2007 at 12:06 UTC
Hi isync and salva, interesting topic. Anyway, if you need to access 2GB of data randomly, there is probably nothing you can do to stop disk trashing other than adding more RAM to your machine, so that all the disk sectors used for the database remain cached. In this situation - more data than memory, but not loads more - I've found memory mapping works well. In my situation the data accesses were randomly scattered but with a non-uniform distribution - if that makes sense. I.e. although the access wasn't sequential, some data was accessed more often than others. So memory mapping meant that the often-access data stayed cached in ram. Any decent database should be able to do pretty much the same thing - as long as you configure it with a big query cache - although disk access will be slower than for memory mapping. The real problem comes if you're making a lot of changes to the data, which busts your cache... Best wishes, andye	[reply]
Re^5: Perl solution for storage of large number of small files by jbert (Priest) on Apr 30, 2007 at 14:31 UTC
Often-accessed data will stay in memory whether it is accessed via read() or mmap(). mmap() can be a more convenient interface, precisely because of the opposite effect, data on disk mapped by mmap() isn't automatically brought into memory until it is used, and then only the bits which are needed are brought in (subject to 4k page granularity). Whereas a successful read() will always bring the data to memory. This means you are perhaps less likely to have unwanted data in memory, but that's more to do with it taking more code to do the read() approach well than because mmap()'d data is more likely to stick in memory. The kernel might trigger different heuristics for the two different methods of access (such as readahead if you do a number of sequential reads or a big sequential memory access to an mmap'd area), but I'm not even sure of that - they might go through exactly the same code paths. I'd say that the biggest difference is the results of a read() are normally copied into a per-process buffer in the application, whereas multiple processes can in principle share the same copy of mmap'd data.	[reply]
Re^6: Perl solution for storage of large number of small files by andye (Curate) on Apr 30, 2007 at 14:51 UTC
Hi jbert, So can you read() a file that's bigger than memory? I thought you couldn't... hence mmap. Best wishes, andye	[reply]
Re^7: Perl solution for storage of large number of small files by jbert (Priest) on Apr 30, 2007 at 15:28 UTC