New flex output in osm2pgsql

Last week I wrote about Paying off technical debt in osm2pgsql. The work described there came about while I was adding the new “flex output” to osm2pgsql. In osm2pgsql terminology “outputs” are the parts of osm2pgsql responsible for converting the OSM data into a format suitable for the database and then adding it to the database. There are several outputs, the classic “pgsql” output, the “gazetteer” output used for Nominatim imports, and the “multi” output, an earlier attempt at adding more flexibility to osm2pgsql that never got much traction. The new “flex” output rethinks this job and adds, as the name suggest, a lot of flexibility. When I worked on it I needed changes in other parts of osm2pgsql outside the output structure itself, so that’s where the cleanup job I talked about last week started. In this blog post I want to write a little more about the flex output, the ideas behind it, and my work on it.

The original osm2pgsql is pretty opinionated on what an OSM database should look like. It filled, for instance, a specific set of database tables you could not change. You could change some aspects of the columns used in those tables through a config file. The way the data was converted from OSM format into the database table was pretty rigid, later Lua transforms were introduced which allowed some more ways of transforming the data. But for a project like OSM that prides itself on its open-endedness and with the many developments of the last years introducing more and more complex tagging and many different types of relations, this isn’t enough for all the “creative, productive, or unexpected ways” OSM can be used.

The “flex” output

The flex output has a different approach than the old code. Instead of having some kind of default setup that you can change through command line arguments and config files, the flex output starts with a blank slate. The user decides what database tables there should be and which columns of what types they should have. And the user decides for each and every OSM object how it should be handled, how the OSM data should be transformed and which data should end up in which table(s). This allows for much more flexibility and is arguably even easier to understand, because the user can look at the code and understand what it is doing instead of having to know which obscure option in combination with these other three options changes exactly which behaviour.

Of course it is not everybody’s cup of tea to write Lua code. But not everybody has to write a new config from scratch. I fully expect there to be some “standard” configurations users can either use as is or use as basis for some customizations. Having this in Lua code makes it much more accessible than doing this in C++.

Use cases

So why do you need that flexibility? OSM is an incredible database of facts about the world and you can create awesome specialized maps from the data. But with the old osm2pgsql this often meant a lot of the processing had to be done outside of osm2pgsql. Hopefully the flex output allows you to do more of the processing inline.

One annoying problem when using OSM data is how many different ways there are to map similar (or even the same) thing. Is it oneway=yes or oneway=true? You have to support both. And maybe, for your special map, it doesn’t matter whether something is a natural=wood or landuse=forest, you want show it in the same way. With the flex output it is trivial to unify all these in just the way you want so that it become easier to query the database when rendering.

Because it is now possible to add the same OSM feature to several database tables, it is also possible to prepare different versions of the same data for different zoom levels. (What’s missing here is more powerful geometry processing though, see below.)

The flexibility also makes osm2pgsql much more interesting for non-rendering use cases. Whether you want to create statistics of certain OSM tags or find mapping differences between different countries, I am sure you can find a database format that helps you answering those questions.

Geometry Processing

When you want to use OSM data for rendering you need to do some geometry processing. In the first step you convert OSM nodes, ways, and relations into points, linestrings, and polygons. This is simple for nodes, reasonably easy for ways, but quite complex for relations, because it depends on the type of relation and its members what kind of geometry you need to create. This conversion has always been the job of osm2pgsql. Part of it is configurable, part of it hardcoded magic. The flex output gives you a bit more flexibility here. You can decide for yourself, for instance, whether you want to split long linestrings into shorter pieces and exactly how long those pieces should be. Or you can decide whether you want boundary relations as linestrings or as polygons (or both, one in one table, the other in a different table). You can even decide not to store the geometry of an OSM object at all. Maybe you are doing tag statistics only anyway and don’t need the geometries.

Often you don’t want to use this geometry as is, but you need some kind of further transformation. A typical case are simplified geometries for lower zoom levels where you don’t need every crinkly little curve in a road, but just the rough form. This is something osm2pgsql doesn’t support. You can do these things in the database after the data import, but it would be really nice if osm2pgsql would support this in some way. Unfortunately it isn’t always clear how to best do this. Lets use a simple example: Calculating the center point of an area and storing that in the database instead of the area itself. A lot of people have asked for this and it seems a rather simple request. How hard can it be do calculate the center? But if you look into it you’ll see that it isn’t quite clear what the center of a polygon actually is. If you are drawing lower zoom level maps where you want to draw a point instead of the outline of a building, you can use the “centroid” as center. It doesn’t matter in this case that, for oddly shaped buildings, the centroid can lie outside the polygon it was created from! But if you want to use the center point as a point for drawing a label on a high zoom level map, it suddenly is important that the label isn’t outside the feature you are drawing. This can happen, for instance, for a U-shaped lake. So when you asked for a center point what you really asked for is a point that is in the center of a normal polygon, but inside the polygon in any case.

Osm2pgsql can not anticipate all the needs everybody can have on those geometry transformations. But maybe we can find a way to leverage all the functionality built into PostGIS. Hopefully without making everything too slow. This is definitely an area that needs some work. I have been experimenting with some ideas and the flex output already has a generalized framework to do geometry transformations, but it isn’t quite there yet. More ideas are needed here.

Two-stage processing

One of the often requested features for osm2pgsql is some way of combining tags from relations with the tags and geometry from their member ways. This is needed for proper rendering of highway shields for instance, or for rendering bus routes. In those cases some tags are on type=route relations and those tags need to be combined with the highway tags from the ways for the actual rendering.

Andy Allan of Thunderforest needed this for his cycle maps and other specialized maps. And he was willing to put up some money to allow me to work on this. Sarah Hoffmann originally had an idea how to add this in osm2pgsql and I wrote the code as part of the flex output to make this work.

Osm2pgsql processes OSM objects in order: first nodes, then ways, then relations. So by the time we are looking at the relations, the ways have already been dealt with. Information from the relations can’t be added to the ways any more. The idea now is to add another processing stage: When processing a relation, we can “remember” the tags from the relation and we can “mark” the member ways for second-stage processing. After the relations are done, we re-process all marked ways. Now the tags from the relations are available because we remembered them earlier and we can add them to the ways.

The flex output supports this two-stage processing today. But it doesn’t yet work for all object types and there are some issues when updating an existing database. So more work is needed on this.

There are performance implications when using two-stage processing. If too many OSM objects are processed again in the second step, things might get very slow. But that might be a price you are willing to pay for specialized maps. We still need more experiments to see what this feature can do and where its limits are.

Early users

Several people have already tried out the flex output and built experimental maps based on it (see blogposts from Marcos Dione [update 2022-07-17: now unavailable] and Adrien Pavie). And Paul Norman is working on a version of the openstreetmap-carto style using the flex output (branch on Github). It aims to bring the current functionality of the style sheet to the flex output as a first step, before we can explore new functionality.

Status

The flex output is available in the current “master” branch of osm2pgsql. It is marked as experimental and will be for a while. We want to be sure we can keep the API stable before we remove the experimental status.

Tell me about your experiences with the flex output. I’d be great to get some feedback about what works and what doesn’t.