Persistence and handling large amounts of multi-level self-referential data

premchai21 has asked for the wisdom of the Perl Monks concerning the following question:

I am writing an OO program. It being OO (or some close approximation), it deals with its data in the form of objects. These objects are 2-3 levels deep in references, and refer to each other in a mad, free-form manner which would normally not be a problem, except...

... that this data may grow to be too large for memory to hold. Not only that, but also it may, nay, will, need to be saved to disk. Thus, I need persistence of some sort. My first instinct was to use a tied hash to store all the objects and use something like DB_File. However, DB_File stringifies everything, which is unacceptable. My next thought was to use MLDBM or similar (or write my own), which might work, except I'm not sure how well it handles circular structures. I think it dumps the whole structure into each slot.

Any thoughts?

Comment on Persistence and handling large amounts of multi-level self-referential data

Replies are listed 'Best First'.
Re: Persistence and handling large amounts of multi-level self-referential data by princepawn (Parson) on Sep 29, 2001 at 06:19 UTC
The new kid on the block in data freeze-thawing is Data::Denter and it does handle circular references perfectly. I'm not sure why stringification is bad for you, but it does use a readable (indented) format for data dumping.	[reply]
Re: Re: Persistence and handling large amounts of multi-level self-referential data by premchai21 (Curate) on Sep 29, 2001 at 07:34 UTC
I am aware of Data::Denter and in fact use it for simpler persistence situations.. By stringification I mean stringification of the reference, rather than the frozen version. Stringification of the frozen version is fine. However, I see no elegant way to use Data::Denter to do what I'm trying to do; that is, something like DB_File, with caching etc. If I just Denter the hash and dump that into a file (DB or otherwise), I run into the problem of possibly not having enough memory to read it back. It might be possible to write my own module that uses DB_File as a backend, but stores Dentered data to the DB_File, and Undents it when FETCHing. But then my circular references get clobbered.	[reply]
Re: Persistence and handling large amounts of multi-level self-referential data by perrin (Chancellor) on Sep 29, 2001 at 07:12 UTC
MLDBM is a good approach. You can use MLDBM::Sync for multi-process concurrent access. You could also look at Cache::Cache which provides various storage options. Finally, you might consider using SPOPS, which supports dbm files but would also allow you to move your objects to an RDBMS pretty simply later on.	[reply]
Tangram: Persistant Object Mapped to an RDBMS by Arguile (Hermit) on Sep 29, 2001 at 08:03 UTC
I'm not familiar enough with OO to be confident in my reply, but I ran across something a while back that might interest you: What is Tangram? Tangram is an object-relational mapper for Perl objects. It makes objects persist in relational databases, and provides powerful facilities for retrieving and filtering them. Tangram fully supports object-oriented programming, including polymorphism, multiple inheritance and collections. It does so in an orthogonal fashion, that is, it doesn't require your classes to implement support functions nor inherit from a utility class. See Also Tanagram Home WEBTechniques Column on Tanagram	[reply]
Re: Tanagram: Persistant Object Mapped to an RDBMS by rob_au (Abbot) on Sep 29, 2001 at 12:25 UTC
I looked into this collection of modules for a similar project where I had already built much of my object models and while the conversion of data types was fairly direct and straight-forward, documented examples proved to be incorrect and outdated and the whole experience left me software jaded with Tangram. In short, while Tangram is certainly an excellent modules for establishing and storing data relations, its documentation is horrid and examples erroneous in points. Approach with plenty of time to experiment ... Ooohhh, Rob no beer function well without!	[reply]
Re: Persistence and handling large amounts of multi-level self-referential data by premchai21 (Curate) on Sep 29, 2001 at 05:52 UTC
Update: Tested MLDBM, and it does work with multi-level circular structures. I think I might use that. I'm still looking for any suggestions, comments, etc. though.	[reply]
Re: Persistence and handling large amounts of multi-level self-referential data by lachoy (Parson) on Oct 03, 2001 at 06:32 UTC
You should definitely check out The POOP page for a rundown of different persistence schemes. This is definitely an area of lots of work, including by me :-) Chris M-x auto-bs-mode	[reply]

What is Tangram?

See Also