Re^2: Geo Package files

Thanks soonix - that has got me much further :)

But - I am not understanding how to map the GeoPackage spec to Perl's unpack. I've read perlpacktut although I cannot say I understand much of it. So I've made a start based on your information and after a lot of pondering I think I understand why C is the second and third parts of the template

I don't understand why we start with n
The spec says "byte2 magic = ox4750" - so it's 2 bytes or 16 bits. Why is it n and not v or even s?

Spec 2 says it is an "8-bit unsigned integer" so C fits the bill.

I'm guessing we know 3 is unsigned from looking at the bits laid out in the spec - is that about right?

Now I get completely lost...
There is an int32 between 3 and 4 so that's N or V - but how can we tell? I've tried unpacking with those templates options and get 879493120 and 27700 respectively. I cannot tell from those values which is right.

Next we have 4 double[] envelope
As spec 3 returns 00000101 (5), the envelope consists of 6 values 2: envelope is [minx, maxx, miny, maxy, minz, maxz], 48 bytes so I am using d6 in the template. The returned values don't seem to make sense!

Next is spec 5, the GeoPackageBinaryHeader - I'm took an educated guess this is a but the result didn't seem right so I've used N as it seems to be another flag byte.

Finally we have the geometry which I expect to be text of unknown length. So I tried a100 and A100 but they take me back to the gobbledegook and I cannot work out which one I might need to use.

This is what I have tried...

use DBD::SQLite;
use Data::Dumper;

use strict;
use warnings;

my $dbh = DBI->connect("dbi:SQLite:uri=file:osopenusrn_202203.gpkg?mod
+e=rwc");

my $tab = $dbh->prepare("SELECT * FROM openUSRN");
$tab->execute;
my $n = $tab->fetchrow_hashref;

my $geo = $n->{'geometry'};

my @test = unpack "nCCVd6C100", $geo;

foreach my $t(@test) {
    print "$t\n";
}
[download]

Am I heading in the right direction here or am I going down a blind alley with my reasoning behind the mapping from the specs to the unpack template?
Is there a simpler way to understand this mapping and work out the correct template?

Comment on Re^2: Geo Package files Select or Download Code

Replies are listed 'Best First'.
Re^3: Geo Package files by soonix (Chancellor) on Mar 04, 2022 at 20:39 UTC
n versus v is determined by the byte order. In the https://libgeos.org/specifications/wkb/ specification mentioned by swl, there is a section on Byte Order which mentions a flag stating the order used, and http://www.geopackage.org/spec120/#gpb_format says this flag is the lowest bit in the "flags" field (second C in the unpack template), so this determines whether to use big-endian (n and N) or big-endian (v and V). Luckily, this flag is a single byte, so that it itself isn't affected by byte-order ;-) Looking further at the flag byte, I see the the next three bits tell you how many doubles (probably f format, from Perl 5.10 upwards you can distinguish between big-endian "f>" and little-endian "f<") are in the `double[] envelope` array at the end of the GeoPackageBinaryHeader. Sorry I'm not really into this, so I can't dive very deep into it. Depending on what information comes up in the discussion, it might trigger me to go looking deeper, but don't bet on it :-)	[reply] [d/l]
Re^4: Geo Package files by Marshall (Canon) on Mar 04, 2022 at 22:46 UTC
Yes, this format tells us at the beginning how to interpret the following bytes. In my post here, we need to know the value of YY, the critical flag byte. I am not sure exactly what that byte says.	[reply]
Re^3: Geo Package files by Marshall (Canon) on Mar 04, 2022 at 19:52 UTC
I don't understand exactly what your binary BLOB (Binary Large OBject) contains. A simple "print $geo" doesn't mean much to me. I would start by making a hex dump of $geo. Something along the lines of: `join ' ', unpack '(H2)', $geo;` ... I would put either 10 or 16 bytes per line in the dump. It could be that some option on Data::Dump would work? You should wind up with something like: 47 50 XX YY... 0x47 means "G" and 0x50 means "P". The next byte looks like a version number - probably doesn't mean much to you. The very next flag byte means a lot* (emphasis added) and we need to know what that is in order to understand what "double[] envelope;" means. My code for dealing with binary headers like this usually has a substr() to select an range of bytes then it applies the appropriate unpack template upon that subset of bytes. I suspect that some relatively straightforward, special purpose code will be able to decode your specific blobs. Decoding this binary geo format in a general sense appears to be a "non-trivial" task. Your code will probably wind up depending upon the flag byte being a particular value - your code being specific to decoding segments with that particular set of flags. The code to decode the bulk of the BLOB depends critically upon knowing what the flag byte is. Lets see the first 16 bytes of this $geo blob in hex dump format... Also don't forget the ord() function. `print "Got G!\n" if ( ord(substr($geo,0,1)) == 0x47 );` I think will work. I have no idea of how many BLOB's you need to decode or what the performance implications are. I would try to get something functionally working and then worry about performance tweaks later. It could very well be that such tweaks are not necessary.	[reply] [d/l] [select]
Re^4: Geo Package files by Bod (Parson) on Mar 05, 2022 at 00:58 UTC
Thank you Marshall Form `print join ' ', unpack '(H2)', $geo;` I have much more sensible output to work with. Here are the first 16 hex 'characters' `47 50 00 05 34 6c 00 00 50 b8 1e c5 2c 75 23 41` [download] As per the spec, the first two are 0x4750. The version number is 0 which means version 1 which is what I would expect. Next comes 05 - or as I said previously, this is 00000101 The bits correspond to RRXYEEEB - where EEE defines the envelope. So that is 010 = 2 which gives the envelope as being `2: envelope is [minx, maxx, miny, maxy, minz, maxz], 48 bytes`. Also from this, B = 1 tells us that it's Little Endian. The SRSID I worked out late last evening using 'V' to unpack it. It is 27700 which is the SRSID for OSGB 1936. That passes the sanity check as the data I am decoding comes from the Ordnance Survey. This is where I got lost... The 6 components of the envelope I tried to unpack with 'd6' and got: `637590.385 642426.601 309577.58 310361.391000001 0 0` [download] The last two seem reasonable as I am not expecting height data although I would not be surprised if it were included. However, the first and third give a location inland from the east coast (near Norwich if you know your UK geography) and not off the west coast as I would expect for a minimum bounding coordinate. If I ue '(f<)6*' I get even less sensible results: `-2539.51953125 10.2161064147949 8.48770014272304e-08 10.2253313064575 126443847680 9.18094444274902` [download] I have no idea of how many BLOB's you need to decode or what the performance implications are. I would try to get something functionally working and then worry about performance tweaks later. It could very well be that such tweaks are not necessary There are 1.4 million BLOBs that need decoding! However, the decode process will be done once every few months (the dataset is updated monthly but doesn't change massively). The decoding process is not time critical. If it needs to run overnight then so be it.	[reply] [d/l] [select]
Re^5: Geo Package files by Marshall (Canon) on Mar 05, 2022 at 04:19 UTC
Well, it appears to me something like this will work?: This geo thing is a complex format in its general case. `byte 1 ="G" byte 2 = "P' byte 3 = 0 # means version 1 -> probably doesn't matter byte 4 = 5 # flags: little endian, 32 bit (Intel) for each point, # 6 values for each data set, 6x8 = 48 bytes each byte 5-8 = # srs_id = 0x0000346C = Unique identifier for each # Spatial Reference System within a GeoPackage # I have no idea what that means? byte 9-... # start of data.. is here... # #` [download] Fetching and decoding 1.4 million points is no big deal. Will run in less than a minute. Also please understand that buzzwords like SRSID or OSGB mean nothing to me - I am clueless.	[reply] [d/l]
Re^6: Geo Package files by Bod (Parson) on Mar 05, 2022 at 14:17 UTC
Re^7: Geo Package files by soonix (Chancellor) on Mar 05, 2022 at 18:07 UTC
Some notes below your chosen depth have not been shown here
Re^7: Geo Package files by Marshall (Canon) on Mar 07, 2022 at 23:50 UTC
Some notes below your chosen depth have not been shown here
Re^4: Geo Package files by hv (Prior) on Mar 05, 2022 at 14:21 UTC
A useful technique for dumping binary data is to use the "v" flag in sprintf. Eg: `% perl -E 'say sprintf "%v02X", "foo\x{0}"' 66.6F.6F.00` [download] See the section on "vector flag" in sprintf.	[reply] [d/l]
Fixed starting bytes (was: Re^4: Geo Package files) by Bod (Parson) on Mar 12, 2022 at 19:16 UTC
You should wind up with something like: 47 50 XX YY... 0x47 means "G" and 0x50 means "P" Just as a general question... Why would the file specification require 0x4750 at the start of the file? What does it add? Is it just there so that any processor of the file can fail quickly if it is passed a file that doesn't start with these two bytes or is there more to it than that?	[reply]
Re: Fixed starting bytes (was: Re^4: Geo Package files) by Marshall (Canon) on Mar 12, 2022 at 20:12 UTC
This is a common protocol feature. For example a .WAV file starts with WAVE - the letters RIFF can also appear. Using ASCII letters makes it easy to see that you have the right kind of format just by inspection (if you deal with ASCII often). Often binary dump views will display the ASCII if it is within normal "printable" character range as added info. This is also helpful to make sure that you are at a "proper beginning". Decoding the thing requires being certain that you are at a valid byte(0) because all field definitions are a delta to that byte's "address". So this is a cheap (very little bandwidth), "sanity check". This is also often seen with communication links and can assist with resynchronization when some data "goes missing".	[reply]