Never use Storable to save XS objects

The past 6 months I've been struggling with a weird bug I had in pVoice. The application itself ran fine, but when the application closed, 99% of the time I got an Access Violation in Wx::Bitmap, which tried to free the memory used by some Bitmap I had obviously created...or so I thought

Yesterday evening I used Data::Dumper to dump the datafile I use for pVoice, which showed me lots of entries like this:

                                [
                                  'wassen',
                                  'wassen',
                                  '1',
                                  'C:\\Mijn documenten\\PCS\\Verbs\\Gr
+ooming\\wash 1.jpg',
                                  bless( do{\(my $o = 12451544)}, 'Wx:
+:Bitmap' )
                                ],
[download]

I had seen entries like that before and never really worried about it. Indeed, once I tried to save the actual Bitmap in the Storable file, which didn't work, but I didn't care about having that entry in my datafile. I didn't use it, did I?

But yesterday it struck me. Was it just a coincidence that my Access Violations happened in Wx::Bitmap and that I had those lines in my datafile? Obviously not. Removing all references to those Wx::Bitmap objects, and saving the datastructure again solved the problem. So what has been going on here?

My theory is this: the code references are only valid for the time the application runs. When the datastructure is read in another run, the data the Wx::Bitmap entry refers to doesn't exist, but when the application closes, it will still try to free the memory, which is obviously not in use by my application, so an Access Violation is generated.

Moral of the story: never use Storable to save objects that aren't pure Perl, but consist of XS code.

Jouke Visser
Speaking at the 2004 O'Reilly Open Source Convention about pVoice

Comment on Never use Storable to save XS objects Download Code

Replies are listed 'Best First'.
Re: Never use Storable to save XS objects by ysth (Canon) on May 11, 2004 at 17:44 UTC
It depends; ideally modules mucking about as Wx::Bitmap seems to would provide Storable hooks to be properly serializable. (See Storable)	[reply]
Re: Re: Never use Storable to save XS objects by simonflk (Pilgrim) on May 11, 2004 at 18:09 UTC
ysth says: It depends; ideally modules mucking about as Wx::Bitmap seems to would provide Storable hooks to be properly serializable Nice, I'd missed/forgotten about those hooks. I don't think it'll work for Wx objects that are subclasses of Wx::Window. But it'd probably work fine for Wx::Bitmap. Here's some untested code for (de)?serialising Wx::Bitmaps... `my $bitmap_data = $wx_bitmap->GetImage->GetData; # serialise bitmap data # later... $wx_bitmap = new Wx::Image()->SetData($bitmap_data)->GetBitmap;` [download] -- simonflk	[reply] [d/l]
Re: Never use Storable to save XS objects by diotalevi (Canon) on May 11, 2004 at 20:23 UTC
Also be sure not to use Storable on closures. The lexical environment of the closure is not captured in the Storable object. Functions are deparsed into approximately equivalent source code so beware that you get the same functional thing back.	[reply]
Re: Never use Storable to save XS objects by simonflk (Pilgrim) on May 11, 2004 at 17:51 UTC
I've hit this in the past. In my case I was storing objects that sometimes contained DBI handles. It's a mistake I hope not to repeat. -- simonflk	[reply]
Re^2: Never use Storable to save XS objects by robins (Acolyte) on Sep 17, 2007 at 10:20 UTC
You shouldn't by any chance have some documentation about how you solved that hurdle? I have the DBI problem myself. My objects contain DBI handles, and I need to serialize them, but without the DBI code. Is there maybe a module in the DBI or Class namespace that can help with this? Any pointers appreciated...	[reply]
Re^3: Never use Storable to save XS objects by dragonchild (Archbishop) on Sep 17, 2007 at 13:56 UTC
I think you need to answer a couple of questions first: What do you expect to happen when you serialize the DB connection? Should the DB connection be closed? What should be stored in the serialized data structure? What do you expect to happen when you de-serialize the DB connection? Should the connection happen immediately? What about waiting until the first query because you may not actually use the DB connection at every given de-serialization. There are some further concerns: Should passwords be stored? If so, how? How should DBI connections using DBD::Excel or DBD::CSV work? What should happen if you de-serialize the thing and attempt to use the DB connection on a box that doesn't have the right things installed? And I'm sure that I'm missing some concerns. In other words, serializing stuff that uses external resources is non-trivial. I've run into the same problem with DBM::Deep and I don't have a really good way to solve it as of yet. My criteria for good software: Does it work? Can someone else come in, make a change, and be reasonably certain no bugs were introduced?	[reply]
Re^4: Never use Storable to save XS objects by robins (Acolyte) on Nov 08, 2007 at 18:00 UTC
Re^5: Never use Storable to save XS objects by dragonchild (Archbishop) on Nov 08, 2007 at 18:36 UTC
Re: Never use Storable to save XS objects by demerphq (Chancellor) on May 12, 2004 at 15:04 UTC
You should be looking at exploiting the freezer/toaster methods in Data::Dumper or Storable. Data::Dump::Streamer has even more sophisticated mechanisms for handling this so you may also want to look there. --- demerphq _{First they ignore you, then they laugh at you, then they fight you, then you win. -- Gandhi}	[reply] [d/l]