|Perl Monk, Perl Meditation|
How to subclass IO::Handle to properly get a low level file handle without having a file or memory?by Pickwick (Beadle)
|on Nov 02, 2016 at 10:49 UTC||Need Help??|
Pickwick has asked for the wisdom of the Perl Monks concerning the following question:
I have an app which accesses a PostgreSQL database and needs to read some large binary data out of it depending on some needed processing. This might be hundreds of MB or even some GB of data. Please no discussion about using file systems instead or such, it's the way it is now.
That data is simply files of various types, e.g. it might be a Zip container or some other kind of archive. Some of the needed processing is list the contents of the Zip, maybe even extract some members for further processing, maybe hash the stored data... In the end the data is read multiple times, but written only once to store it.
All of the Perl libs I use are able to work with file handles, some with IO::Handle, others with IO::String or IO::Scalar, some others only with low level file handles. So what I've done is create a subclass of IO::Handle and IO::Seekable which acts like a wrapper for the corresponding methods around DBD::Pg. In the CTOR I create a connection to the database, open some provided LOID for reading and store the handle provided by Postgres in the instance. My own handle object is then forwarded to whoever is able to work with such a file handle and can directly read and seek within the blob provided by Postgres.
The problem is libs which use low level file handles or low level file handle operations on IO::Handle. Digest::MD5 seems to be one, Archive::Zip another one. Digest::MD5 croaks and tells me that no handle has been provided, Archive::Zip on the other hand tries to create a new, own handle from mine, calls IO::Handle::fdopen and fails in my case.
I guess the problem is the low level copy of the handle, which takes my instance of the game.
So, is it even possible in my case to provide some IO::Handle which successfully can be used wherever a low level file handle is expected?
I mean, I don't have a real file handle, I have an object only where method calls are wrapped to their corresponding Postgres methods, for which a database handle is needed and such. All of that data needs to be stored somewhere, the wrapping needs to be done etc.
I tried to do what others are doing, like IO::String, which additionally uses tie for example. But in the end that use case is different, because Perl is able to create a real low level file handle to some internal memory on its own. Something which is not supported at all in my case. I need to keep my instance around, because only that knows of the handle to the database etc.
Using my handle like an IO::Handle by calling method read and such works like expected, but I would like to take it a bit further and be more compatible to whoever doesn't expects to work on IO::Handle objects. Much like IO::String or File::Temp can be used as low level file handles.