This blog post is about the new OSM XRAY service. You can find a demo here. It uses vector tiles with all OSM data to really show all of OSM, mostly for debugging. But it is also interesting to just explore what’s there.

I have to import a lot of OSM planet files to test things due to my work on osm2pgsql. I have a test machine for that graciously provided by FOSSGIS e.V., the OSM Foundation local chapter for Germany. A few days ago I imported a planet file and osm2pgsql ran out of memory.

Most OpenStreetMap-based maps you see these days on the Internet are only updated every so often. Some may be updated daily, most are probably only updated every few weeks or months. And that is totally okay for most use cases. But there are some maps that are updated continuously so that changes in the OSM data appear as fast as possible, which is important to motivate OSM contributors and helps with quickly detecting and fixing problems. The most important map with continuous updates is the main map you see on www.openstreetmap.org.

In osm2pgsql there is that code that stores the raw data it gets from the input files and makes it available to other parts of osm2pgsql for further processing. This code is called, in osm2pgsql-speak, the “middle”. There are two implementations of this “middle”, one stores the data in RAM and doesn’t support updates. The other stores the data in the database and supports updates. Back in 2020 I started looking into that code to see what could be improved. In 2021 we released version 1.5.0 which contained a basically rewritten RAM middle. And now, in 2023, I finally got around to improving the database middle.

The osm2pgsql code contains a part called the “middle”. The middle is responsible for keeping track of all OSM objects read from the input file(s), their attributes, tags, and, most importantly, the relations between those objects. This is needed for several reasons.

Back in September we launched the new osm2pgsql.org website. At that time it was still quite rudimentary. Over the last months a big part of my work for the OSM Foundation was to work on that site.

Today we launched the new osm2pgsql.org website. This is the first visible result of the support by the OSM Foundation for osm2pgsql development. Over the next months I’ll keep working on osm2pgsql, payed by OSMF and other sponsors.

Today Sarah released the new osm2pgsql version 1.3.0 (release notes). After working on this for more than half a year now it is great to see the new version out there.

Last week I wrote about Paying off technical debt in osm2pgsql. The work described there came about while I was adding the new “flex output” to osm2pgsql. In osm2pgsql terminology “outputs” are the parts of osm2pgsql responsible for converting the OSM data into a format suitable for the database and then adding it to the database. There are several outputs, the classic “pgsql” output, the “gazetteer” output used for Nominatim imports, and the “multi” output, an earlier attempt at adding more flexibility to osm2pgsql that never got much traction. The new “flex” output rethinks this job and adds, as the name suggest, a lot of flexibility. When I worked on it I needed changes in other parts of osm2pgsql outside the output structure itself, so that’s where the cleanup job I talked about last week started. In this blog post I want to write a little more about the flex output, the ideas behind it, and my work on it.

Osm2pgsql is one of the work horses of the OpenStreetMap world. It is used to import OSM data into a database which is then used for rendering maps (among other uses). Osm2pgsql is behind the map you see on openstreetmap.org and many others. The program has been around since 2006 and has accumulated many changes and additions over time. And the code has grown more and more complex. Part of that is inevitable because the program does a lot more than when it was started. But part of that is complexity that doesn’t have to be there. For a few months now I have been working on reducing that complexity and want to tell you about that work.