in reply to Re^3: Geo Package files
in thread Geo Package files

Thank you Marshall

Form print join ' ', unpack '(H2)*', $geo; I have much more sensible output to work with. Here are the first 16 hex 'characters' 

47 50 00 05 34 6c 00 00 50 b8 1e c5 2c 75 23 41

[download]
As per the spec, the first two are 0x4750. The version number is 0 which means version 1 which is what I would expect.

Next comes 05 - or as I said previously, this is 00000101

The bits correspond to RRXYEEEB - where EEE defines the envelope. So that is 010 = 2 which gives the envelope as being 2: envelope is [minx, maxx, miny, maxy, minz, maxz], 48 bytes. Also from this, B = 1 tells us that it's Little Endian.

The SRSID I worked out late last evening using 'V' to unpack it. It is 27700 which is the SRSID for OSGB 1936. That passes the sanity check as the data I am decoding comes from the Ordnance Survey.

This is where I got lost...
The 6 components of the envelope I tried to unpack with 'd6' and got: 

637590.385
642426.601
309577.58
310361.391000001
0
0

[download]
The last two seem reasonable as I am not expecting height data although I would not be surprised if it were included. However, the first and third give a location inland from the east coast (near Norwich if you know your UK geography) and not off the west coast as I would expect for a minimum bounding coordinate.

If I ue '(f<)6' I get even less sensible results: 

-2539.51953125
10.2161064147949
8.48770014272304e-08
10.2253313064575
126443847680
9.18094444274902

[download]

I have no idea of how many BLOB's you need to decode or what the performance implications are. I would try to get something functionally working and then worry about performance tweaks later. It could very well be that such tweaks are not necessary

There are 1.4 million BLOBs that need decoding! However, the decode process will be done once every few months (the dataset is updated monthly but doesn't change massively). The decoding process is not time critical. If it needs to run overnight then so be it.

Replies are listed 'Best First'.
Re^5: Geo Package files
by Marshall (Canon) on Mar 05, 2022 at 04:19 UTC
    Well, it appears to me something like this will work?:
    This geo thing is a complex format in its general case. 

    byte 1  ="G" 
byte 2 = "P'
byte 3 = 0  # means version 1 -> probably doesn't matter
byte 4 = 5  # flags: little endian, 32 bit (Intel) for each point, 
            # 6 values for each data set, 6x8 = 48 bytes each 
byte 5-8  = # srs_id = 0x0000346C = Unique identifier for each 
            # Spatial Reference System within a GeoPackage
            # I have no idea what that means?
byte 9-...  # start of data.. is here...
            # 
            #

    [download]
    Fetching and decoding 1.4 million points is no big deal. Will run in less than a minute. Also please understand that buzzwords like SRSID or OSGB mean nothing to me - I am clueless.
[reply]
[d/l]
      something like this will work?

      So far so good. And the srs_id passes the sanity check as it is the value I would expect.

      The problem comes when we get to the envelope. We know it is a double[] from the spec and we know it is Little Endian from the flag byte. The trouble I'm having is understanding how to take that information and translate it to a template for unpack. From the list in the documentation, the closest seems to be 'V' but that is a long not a float.

      unpack'ing the envelope with 'V' gives these values: 

      minx - 3307124816
maxx - 1092842796
miny - 867583392
maxy - 1092852469
minz - 1374389536
maxz - 1091757350

      [download]
      This makes no sense as min_x has to numerically less than max_x - it is only a co-ordinate system, albeit one in 3 dimensional irregular spherical space. Likewise with min_z and max_z, this is height above sea-level and the min has to numerically less than the max.

      So from the data it seems I am using the wrong unpack template of 'nCCV(V)6'.

      I've tried the two float options - 'f' and 'd' but they produce equally unrealistic values.

      Do you have any advice on how I go about translating from the information I know about the data to the template necessary to unpack it?

      buzzwords like SRSID or OSGB mean nothing to me

      Sorry!

      SRS - Spatial Reference System is the co-ordinate system used to identify where a point is on the Earth's surface. Many SRS's exist and none are perfect. In order that the data can be used, we need to know which SRS we are using

      OSGB - Ordnance Survey of Great Britain is the SRS that we most commonly use here in the UK. Because the UK is a relatively small country, OSGB mostly ignores the curvature of the Earth. OSGB is what we have found as the SRS ID in this GeoPackage so that's a sanity check that we are on the right path so far.

[reply]
[d/l]
[select]
        The WKB spec says it's an "8-byte IEEE double". I assumed that would imply "f" format specifier, but probably I was mistaken, and it should be "d", as this older post suggests. Data::IEEE754::Tools mentions "d>"…

        The "Floating point numbers" section in perlpacktut says "There is no such thing as a network representation for reals", which might mean that the "little-endian"-flag bit could be ineffective for these doubles.

[reply]
        Dealing with floats is messy. Among those issues is what the heck does a 64 bit float mean?

        We know that this format is in "little endian". Is your processor also "little endian", e.g. an Intel processor? If it is then maybe we don't have to write completely platform independent code?

        I think also relevant to this would be: Is your Perl version 32 or 64 bit? I am running 64 bit Perl on a 64 bit Windows platform. And I have a 64 bit GCC complier. On my 64 bit machine, in C, a simple float is 64 bits. I don't know for sure, but I would suspect that 64 bit Perl's representation of a simple float is also 64 bits? Added: I see trouble if you are using 32 bit Perl.

        I don't know where the 6 comes from in 'nCCV(V)6'? Each float is 8 bytes, not 6 bytes.

        From what I understand so far,
        index 8, length 8 bytes => minx
        index 16.length 8 bytes => maxx
        index 24,length 8 bytes => miny
        index 32,length 8 bytes => maxy
        index 40,length 8 bytes => minz
        index 48.length 8 bytes => maxz

        so,  my $minx = unpack ("d8", substr($geo,8,8)); may work??
        I am not sure if f8 would work also?

        Update: BTW, what is length ($geo)?

        Also, again, I request the binary data dump of $geo, NOT what you think that binary unpacks as.
        Dump should be 2 hex digits per byte, with a space between bytes. 0 should be 00 so that columns line up nicely. 16 bytes per line would be fine.

[reply]
[d/l]

In Section Seekers of Perl Wisdom