Tile expiry and generalization

Several of the generalization algorithms I am working on are based on tiles. They collect all features inside a tile boundary and do something with them. This means that whenever something changes, we have to figure out which tiles are affected and re-process them. Luckily a very similar functionality is also needed for generating lists of tiles that need to be re-rendered. And support for that has been in osm2pgsql for a long time. We just need some additions to be able to re-purpose that code.

(This post is part of a series about generalization of OSM data.)

How tile expiry works

Tile expiry (for re-rendering or for re-processing generalization steps) works in the same way. For each change in the OSM data we need to figure out which tiles are affected. We do this based on the old geometry of the OSM object (if we are changing the object or deleting it) and the new geometry (if we are changing the object or creating a new one). Depending on the type of geometry we can calculate the tiles that need changing. For point and line geometries this is pretty straightforward. For polygon geometries it is a bit more complicated. Depending on the type of feature and the rendering or generalization we do with it, we might want to expire the full area or only the boundary of the polygon. If we draw only the outline of a large area (a country boundary for instance) it makes a big difference whether we have to re-render all tiles in that area or only a few around the border.

What do we need?

For the purposes of generalization we need two things. First we need more than one list of expired tiles. We need different expiry configuration for different database tables or layers so that we can expire them independently. The old code only allowed a single expired tiles list configured from the command line.

Second, it would be much more convenient to be able to put the list of expired tiles into a database table instead of a file, which is the only thing the old code supported. The generalization code can then read the tile coordinates from those tables again and process those tiles.

These changes are now available in the master branch. The configuration is now done in the Lua config file together with the other configurations. You can define different expire outputs and specify in which zoom level they work and where they write their data (files or database tables). For each geometry column you can specify which expire output to use and some more details.

The details are documented in the manual. For a working example look at this example Lua config.

I’ll leave it for a future blog post to explain how to actually use these expires outputs with the generalization functionality.

Future features

There are still several areas of improvement for tile expiry that need more work:

Expiry of polygons is made by bounding box of the polygon. For large polygons this can affect more tiles than strictly necessary.

We always expire based on the union of the before and after geometries. Even if only a small part of a geometry changed, we still expire the whole thing.

Even if the geometry of an object didn’t change, we always do the expire if anything in the OSM object changed. Osm2pgsql can not know whether a change in some tag actually changes the rendering of that feature. Maybe there is a way to improve this somehow.

Expiry improvement ideas are tracked in this issue.