I am currently interested in fetching OpenStreetMap (OSM) data using Overpass API (see https://wiki.openstreetmap.org/wiki/Overpass_API/Language_Guide). This is data regarding roads and cities: traffic lights, roundabouts, amenities, residential addresses. The wealth of data is amazing, its correctness is also amazing judging from what I can verify with my locality. (I can't really thank enough those who came up with the idea and implemented it and collected the data)

I soon came to the, possibly wrong, conclusion that learning the API is hard. What I noticed is that a lot of people just ask how to implement a specific task, e.g. how to fetch all traffic lights data within a bounding box. And so, I soon gave up on implementing a monolithic package which abstracts the queries, do the fetching and cleans the data or exports it. Instead I wrote it so that it does not know of how to do specific queries (e.g. fetch traffic lights) but will do one if user supplies the query content (via a plugin).

I thought of the following architecture (all modules are in PerlOO):

Implement Geo::OSM::Overpass which deals with all the low level communication with OSM server and fetches data -- provided one supplies it with a query text (queries can be in XML or in their own format - QL).

Implement various plugins which take in an Geo::OSM::Overpass object, through which one can run a query or do an export on its last fetched data. Each plugin will know of its specific query content and just that. E.g. how to fetch traffic lights. It may have parameters for cleaning the data afterwards e.g. selecting only traffic lights on highways (or just implement another plugin to do that). The plugins base class will be Geo::OSM::Overpass::Plugin. Any plugin must use this as its parent class and implement the run() method.

For example, I have implemented the Geo::OSM::Overpass::Plugin::FetchTrafficLights to fetch traffic lights data. And also Geo::OSM::Overpass::Plugin::ParseXML to parse the last-query-result (XML) and convert it to a hashtable.

With this model I can get away with my minimal knowledge of Overpass API and let more knowledgable people create their own plugins for fetching this or that and submit them to CPAN without my mediation. At the same time, I have satisfied my basic and urgent need: fetch traffic lights information, but also keep the code expandable and maintanable for the future rather than making a one-off script.

Contrast this model with the model where all queries are implemented in a single module by the current maintainer via user feature-requests (and then scraping on the forums any information on how to convert it to Overpass query which probably will be sub-optimal anyway if you are not an Overpass whizkid *).

Here is an example:

use Geo::BoundingBox;
use Geo::OSM::Overpass;
use Geo::OSM::Overpass::Plugin;
use Geo::OSM::Overpass::Plugin::FetchTrafficLights;
use Geo::OSM::Overpass::Plugin::ParseXML;

my $engine = Geo::OSM::Overpass->new();
$engine->verbosity(2);
$engine->output_filename('xyz');
my $bbox = Geo::BoundingBox->new();
$bbox->centred_at(30.0, 23.0, 100); # centre at (lat=30,lon=23), 100m 
+x 100m square box
$engine->bounding_box($bbox);

my $tfp = Geo::OSM::Overpass::Plugin::FetchTrafficLights->new({'engine
+'=>$engine});
# fetch the data
$tfp->run() or die;
# save XML result to disk
$engine->save() or die;
# convert XML result to a perl hashtable
my $xmlp = Geo::OSM::Overpass::Plugin::ParseXML->new({'engine'=>$engin
+e});
my $hashmap_of_traffic_lights = $xmlp->run();

[download]

I am looking for comments and suggestions regarding this approach.

bw, bliako

*) After my initial success with fetching traffic lights, I pushed my luck to fetch roundabouts. And there all the weaknesses of the underlying OSM model broke: there are roundabouts, mini_roundabouts and circulars. These are nodes. However, the actual roundabouts are ways: a collection of nodes circling the roundabout structure. So, one has to know that or keep asking (see https://forum.openstreetmap.org/viewtopic.php?id=66178 on how the information comes out bit by bit it was like an interogation) and put the puzzle pieces together. Eventually I found out how and also how to ask for the centre coordinate. The kind bureaucrats on the forum are still debating what a roundabout is ...

In reply to OpenStreetMap API : Worker and Plugins architecture by bliako

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.