in Re: Geo::ShapeFile memory problem he was on the right track but maybe not as strong as needed. try this

 
$shapefile = Geo::ShapeFile->new ("tabblock2010_42_pophu");
$shapefile->caching(        shp                => 0); 
$shapefile->caching(        dbf                => 0); 
$shapefile->caching(        shx                => 0); 
$shapefile->caching(        shapes_in_area     => 0);
[download]

Ive been playing, that worked for me. Mem usage is nice and flat now

This works. Thanks!

I've just uploaded a new dev version of Geo::ShapeFile to CPAN which allows users to turn caching off.

https://metacpan.org/release/SLAFFAN/Geo-ShapeFile-2.63_001

Once that's installed, change the object creation line to be:

It's an all-or-nothing approach, but should be sufficient for this use case. I might in future try to add a maximum cache size option.

Other ideas and pull requests welcome.

Shawn

CPAN testers was all green, so I have uploaded version 2.64.

https://metacpan.org/release/SLAFFAN/Geo-ShapeFile-2.64

Shawn

Being able to read right from one of these zip files without needing to unzip it to disk first would be a treat, but i realize it maybe a little too much to ask for.

In that kind of mode requiring me to read it sequentially would be no problem, i'd expect that kind of processing to be kinda common ie

  my $dir0='h:/active/tiger_data';
  my $sfips='42'; # pa 


# make $base->{$blockid10}{color} via 
# zipbyline reads a zip member sequentially 
#   as in http://search.cpan.org/~phred/Archive-Zip-1.59/lib/Archive/Z
+ip.pm#Low-level_member_data_reading 
#  my $sn=$fips2state->{$sfips}.'2010.sf1'; 
#  my $zf=$dir0.'/sf1/'.$sn.'.zip';
#  my $mf=$fips2state->{$fips}.'geo2010.sf1'; ; 
#  my $member=zipbyline_start($zf,$mf); 
#  while (my $line=zipbyline_read($member)){ 
#    ... pull out datums AREALAND AREAWATR POP100, create density 
#    } # line   
#  zipbyline_close($member); 
# sort by density, total POP100, 
#   break into deciles, 
#   assign a decile color to each $base->{$blockid10}{color}

  my $dir=$dir0.'/shapes';
  my $state='tabblock2010_'.$sfips.'_pophu';
  my $shapefn=$dir.'/'.$state.'/'.$state;
  my $imgfn=$dir0.'/gifs/'.$state.'.gif'; 

  # this points to the unzipped dir now,
  # be nice to just point to $dir.'/'.$state.'.zip' instead 
  my $sf = Geo::ShapeFile->new ($shapefn); 
  $sf->caching(shp                => 0); 
  $sf->caching(dbf                => 0); 
  $sf->caching(shx                => 0); 
  $sf->caching(shapes_in_area     => 0); 

  my $x_min=$sf->x_min(); 
  my $x_max=$sf->x_max(); 
  my $y_max=90-$sf->y_min(); # need to invert 90 is top 0 is bot 
  my $y_min=90-$sf->y_max(); 
  my $totalblocks = $sf->shapes();
#   $totalblocks=5000; 


  my $xsize=$x_max-$x_min; 
  my $ysize=$y_max-$y_min; 

  my $imgy=5000; 
  my $yscale=$imgy/$ysize; 


  my $pfx=-0.00923452628555483*($sf->y_min)+ 1.15467278754118; # proje
+ction factor 


  my $imgx=$yscale*$xsize*$pfx;
  my $xscale=$imgx/($xsize); 

  sub xproj { return (($_[0])-($x_min))*$xscale; } 
  sub yproj { return ((90-$_[0])-$y_min)*$yscale; } 

  # create a new image
  my $im = new GD::Image($imgx+1,$imgy+1);

  for my $si (1 .. $totalblocks) {
    my %attr = $sf->get_dbf_record($si);
    my $blockid10 = $attr{BLOCKID10};
    my $color=$base->{$blockid10}{color}; 
    unless ($color) {$color=$yellow;}
    my $polygon = $sf->get_shp_record($si);
     for my $pi (1 .. $polygon->num_parts) {
        my $part = $polygon->get_part($pi);
        my $poly = new GD::Polygon;
        for my $hash (@$part){ 
          $poly->addPt(xproj($hash->{X}),yproj($hash->{Y}));
          } 
        my $first=$part->[0]; 
        my $last =$part->[-1]; 
        if ($first->{X} ne $last->{X} || $first->{Y} ne $last->{Y} ) {
+ 
          $poly->addPt(xproj($first->{X}),yproj($first->{Y}));
          } 
        $im->filledPolygon($poly,$color);
        } # pi
    }  # si

  outlines (); 

  open (my $img,'>',$imgfn); binmode $img;print $img $im->gif;close $i
+mg;
  exit; 

sub outlines { 
  my $state='tl_2010_'.$sfips.'_county10';
  my $shapefn=$dir.'/'.$state.'/'.$state;
# this too points at an unzipped dir, 
# be nice to point at $dir.'/'.$state.'.zip' instead 
  my $sf = Geo::ShapeFile->new ($shapefn); 
  $sf->caching(shp                => 0); 
  $sf->caching(dbf                => 0); 
  $sf->caching(shx                => 0); 
  $sf->caching(shapes_in_area     => 0); 
  my $totalblocks = $sf->shapes();
  for my $si (1 .. $totalblocks) {
    my $polygon = $sf->get_shp_record($si);
    for my $pi (1 .. $polygon->num_parts) {
        my $part = $polygon->get_part($pi);
        my $poly = new GD::Polygon;
        for my $hash (@$part){ 
          $poly->addPt(xproj($hash->{X}),yproj($hash->{Y}));
          } 
        my $first=$part->[0]; 
        my $last =$part->[-1]; 
        if ($first->{X} ne $last->{X} || $first->{Y} ne $last->{Y} ) {
+ 
          $poly->addPt(xproj($first->{X}),yproj($first->{Y}));
          } 
        $im->openPolygon($poly,$black);
        } # pi
    }  # si 

  }
[download]

PA output at this sendspace link while it lasts
MO output at this sendspace link while it lasts

Thanks Huck,

Direct extraction from a zip file is potentially useful, but in my experience not a common use case. (Although perhaps that's because there are no tools to do so...).

Maybe there is a module that supports reading from archives as a file handle? Archive::Zip has readFromFileHandle() but that would need special handling in Geo::Shapefile each time data are accessed from file.

Also, one thing to watch for in any plotting code is holes in the polygons. I don't think the Tiger data have holes, but in the shapefile spec they are implied by vertex order instead of being explicitly flagged. https://www.esri.com/library/whitepapers/pdfs/shapefile.pdf

Shawn.

Great. Thank you.

I cannot discern what is eating your memory, but one thing you should do is move your variables declarations into the smallest possible scope. This will remove a lot of cruft from your code which may make it easier to spot the issue.

Eg:

my $shapefile = Geo::ShapeFile->new ("tabblock2010_42_pophu");
my $totalblocks = $shapefile->shapes();
for (1 .. $totalblocks) {
    my %attr = $shapefile->get_dbf_record($_);
    my $blockid = $attr{BLOCKID10};
    ...
[download]

Hope this helps !

Hi,

I dont see a mention of Scalar::Util::weaken in either Geo::ShapeFile or Tree::R , so thats an obvious bug, as it seems each leaf has a ->{root}, and thats a cycle

But I didn't check by running code

I've come to the same conclusion, and have opened an issue on github.

https://github.com/shawnlaffan/Geo-ShapeFile/issues/19

Test::LeakTrace also shows some leaked scalars (one per shape) that needs further investigation.

Shawn.