Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??

I originally developed this serialization/deserialization code for small configuration files. An important goal was to be able to save and restore arbitrary nested hash/array data structures, without updating the save/restore code every time I added new config data. It worked fine.

But now I'm working on a roguelike game, which needs to be able to save and restore much more substantial quantities of information. Performance instantly becomes a problem.

The save is a little slow; I could work around that probably, by storing each level in a separate file and updating it only when it changes. (Assuming nothing happens on levels where the player is not, this would mean only one level's worth of data would be saved most turns, two levels' worth on turns where the player changes levels.)

Restore, however, is unacceptably slow. It's so slow, I don't actually know how long it takes, because I've never let it run to completion.

Here's my code, such as it is:

sub readconfigfile { my ($cfgkey, $file) = @_; my $path = catfile($config{configfiles}{directory}, $file); if (-e $path) { open CFG, "<", $path or return nonfatalerror("Unable to read confi +g $path: $!"); #print " Opened config file $file\n"; my $slurp = join "", <CFG>; close CFG; $config{cfgkey}{fromfile} = $path; readconfigtext($config{$cfgkey}, $slurp); } } sub readconfigscalar { my ($text) = @_; return if not $text; my %closer = ("[" => "]", "{" => "}", '"' => '"', "'" => "'"); if ($text =~ m/^\s*([[{"'])/) { my ($type) = ($1); $text =~ s/^\s*([[{"'])//; if ($type eq "[") { return readconfiglist($text, $closer{$type}); } elsif ($type eq "{") { my ($value, $remainingtext) = readconfiglist($text, $closer{$typ +e}); return (+{ @$value }, $remainingtext); } else { return readconfigstring($text, $closer{$type}); } } else { my ($line) = $text =~ m/^(.*?)$/; nonfatalerror("Failed to parse config scalar: $line"); } } sub readconfiglist { my ($text, $closer) = @_; my @list; while ($text and not $text =~ m/^\s*[$closer]/) { if ($text =~ m/^\s*(?:[,]|[=][>])\s*/) { push @list, undef; } else { my ($value, $rest) = readconfigscalar($text); push @list, $value; $text = $rest; } $text =~ s/^\s*(?:[,]|[=][>])\s*//; } $text =~ s/^\s*[$closer]\s*//; return (\@list, $text); } sub readconfigstring { my ($text, $closer) = @_; my $string = ""; while ($text and not $text =~ m/^[$closer]/) { my ($substring) = $text =~ m/^([^"'\\])/; $string .= $substring; $text =~ s/^([^"'\\])//; if ($text =~ m/^[\\]([\\"'])/) { my ($escaped) = $1; $string .= $escaped; $text =~ s/^[\\]([\\"'])//; } } $text =~ s/^[$closer]//; return ($string, $text); } sub readconfigtext { my ($cfghash, $text) = @_; $text =~ s/^\s*(?:[#].*|)(?:$)\s*//; return if not $text; print " <" . length($text) . ">\n"; my %closer = ("[" => "]", "{" => "}", '"' => '"', "'" => "'"); if ($text =~ m/^\s*(\w+)\s*[=][>]?\s*([[{"'])/) { my ($key, $type) = ($1, $2); $text =~ s/^\s*(\w+)\s*[=][>]?\s*([[{"'])//; if ($type eq "[") { my ($value, $remainingtext) = readconfiglist($text, $closer{$typ +e}); if ($value) { $$cfghash{$key} = $value; readconfigtext($cfghash, $remainingtext); } } elsif ($type eq "{") { my ($value, $remainingtext) = readconfiglist($text, $closer{$typ +e}); if ($value) { $$cfghash{$key} = +{ %{$$cfghash{$key}}, @$value }; readconfigtext($cfghash, $remainingtext); } } else { my ($value, $remainingtext) = readconfigstring($text, $closer{$t +ype}); if ($value) { $$cfghash{$key} = $value; readconfigtext($cfghash, $remainingtext); } } } else { my ($line) = $text =~ m/^(.*?)$/; nonfatalerror("Failed to parse config text: $line"); } } sub cfgscalartostring { my ($scalar, $indentlevel) = @_; $indentlevel ||= ""; if ((ref $scalar) eq "ARRAY") { return "[" . (join ", ", map { cfgscalartostring($_, $indentlevel +. " ") } @$scalar) . "]\n$indentlevel"; } elsif ((ref $scalar) eq "HASH") { return "{" . (join ", ", map { my $k = $_; cfgscalartostring($k) . + " => " . cfgscalartostring($$scalar{$k}, $indentlevel . " ") } key +s %$scalar) . "}\n$indentlevel"; } elsif (not defined $scalar) { return ""; } elsif (not ref $scalar) { my $string = "" . $scalar; $string =~ s/[\\]/\\\\/; $string =~ s/(['"])/\\$1/; return '"' . $string . '"'; } } sub writeconfigfile { my ($cfgkey, $file) = @_; my $path = catfile($config{configfiles}{directory}, $file); open CFG, ">", $path or return nonfatalerror("Unable to write config + file $path: $!"); my $cfghash = $config{$cfgkey}; for my $key (keys %$cfghash) { print CFG "$key = " . cfgscalartostring($$cfghash{$key}) . "\n"; } close CFG; }

An example of a working program that uses this code for configuration purposes is available (for a limited time) on my scratchpad (a few Unicode characters seem to have got mangled to HTML entities, sorry) or here (no such caveat). Yes, it's a Tetris game. But it demonstrates how the code works.

I've thought about trying to rip out all the regex based parsing and replace it with substr() and such, but I don't know how much that will actually help. I also thought about trying to find a suitable serialization module on the CPAN that I can just use, but I would need something that preserves nested data structures containing both hashes and arrays intact. (I don't need to preserve the distinction between numbers and strings. If 1 becomes "1" that's fine with me. I go out of my way to avoid writing code that cares about the difference.)

It's crucial that I don't need to change the save/restore code every time I add some more data somewhere, because I expect to be doing a lot of that as this roguelike game develops. That code, in its current highly preliminary and not-entirely-working state, can be seen here. (If you comment out the save/restore stuff, it actually kind of sort of works, as far as it goes, though barely any features are implemented yet, and ingestion doesn't currently work due to an unrelated bug.)

I'd kind of like to avoid using Data::Dumper if possible, on the grounds that allowing configuration files to run arbitrary code gives me the willies. (If the game ever has, say, a public server, it'd be nice to let users copy/paste some of their config files from their local systems, such as the one for key bindings...)

How should I go about optimizing this (it needs to improve by at least a couple of orders of magnitude), or what should I replace it with?


In reply to Optimization Question (Data Serialization) by jonadab

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others surveying the Monastery: (5)
As of 2024-04-23 17:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found