Node Locations on Ways

OpenStreetMap has a somewhat peculiar data model. OSM ways don’t contain the locations of the points they go through, but only references to nodes, which in turn contain their location. This model has some advantages, but it also has a major disadvantage: For almost all uses of OSM, like drawing maps or calculating routes, the software has first to find the locations of all nodes in a way, before it can do anything with it. This is cumbersome and potentially expensive. To do this quickly, the lookup needs to be done in main memory without any disk access, but to store the locations of all nodes of a full planet file we already need about 32GB RAM. But maybe we can create a shortcut, so that we can do this once, instead of everybody doing this again and again?

The newest git master branch of libosmium now contains code to add node location to the ways. This works for XML, PBF, and OPL files. It is enabled by using the locations_on_ways option on the file format. The IDs of the nodes are still in the ways, so no information is lost. For the XML Format there simply are additional lon and lat attributes on the nd elements:

<way id="123"...>
  <nd ref="12" lon="3.141" lat="5.926"/>
  <nd ref="13" lon="2.111" lat="1.222"/>
</way> 

PBF files are based on the extensible Google Protobuf Format, so it is easy to encode extra information in them. Libosmium uses delta encoded packed sint64 fields numbered 9 (lat) and 10 (lon) in addition to the field 8 (refs) containing the IDs of all the nodes. In addition the string “LocationsOnWays” is added to the list of optional features in the header so that it is easy to see whether this information is available. All programs that can read OSM XML or PBF files will just ignore this extra information if they don’t understand it, so we are completely backwards compatible.

The OPL format usually has a list of nodes in a way looking like this:

... Nn12,n13 ...

To add the locations this is extended to:

... Nn12x3.141y5.926,n13x2.111y1.222 ...

This format is not backwards compatible.

Of course adding all this data will make the files larger, but because in most applications all the nodes that don’t have any tags are only needed to assemble the way geometries, we can remove them which usually makes the generated file a bit smaller than the original file.

There is support in the master branch of the Osmium command line tool to create files like this. The command

osmium add-locations-to-ways germany.osm.pbf \
       -o germany-with-loc.osm.pbf

for instance will add the node locations to the ways of the germany.osm.pbf file. Running this for the whole planet is just as easy:

osmium add-locations-to-ways planet.osm.pbf \
       -i dense_mmap_array -o planet-with-loc.osm.pbf

You will need at least 32GB RAM to run this. By default this command will not output untagged nodes, if you want to keep them, use the -n option. For more details see the man page.

To convert between formats, you can use osmium cat:

osmium cat germany-with-loc.osm.pbf \
       -o germany-with-loc.osm.bz2 -f osm,locations_on_ways

You have to manually set the format option locations_on_ways as above or you will loose the extra information. (The add-locations-to-ways command will do this automatically, because it doesn’t make any sense without it.)

If you want to try it out, but don’t want to generate those files yourself, you can use these example files, I have created for you, both in PBF format: [Update: Files not available any more.]

Both files do not contain nodes without any tags to keep them smaller. These files are not updated, but if there is some interest I might consider offering regular planet downloads or so.

This is all experimental currently. Please tell me if and how you use it. I also need to document all this on the wiki as optional file format extensions.

One obvious missing piece is a way to update those files from change files. This is not really difficult to do. For each node in a change file, we have to store the location and then make sure all ways are updated accordingly. This is, of course, more expensive than simply applying the change files, but probably not really bad. I am planning to add this functionality to the osmium tool.