[JT] Jochen Topf's Blog

osm2pgsql

Minutely updated map

Most OpenStreetMap-based maps you see these days on the Internet are only updated every so often. Some may be updated daily, most are probably only updated every few weeks or months. And that is totally okay for most use cases. But there are some maps that are updated continuously so that changes in the OSM data appear as fast as possible, which is important to motivate OSM contributors and helps with quickly detecting and fixing problems. The most important map with continuous updates is the main map you see on www.openstreetmap.org.

| Read more…

Tags: openstreetmap · osm2pgsql


Improving the middle of osm2pgsql

In osm2pgsql there is that code that stores the raw data it gets from the input files and makes it available to other parts of osm2pgsql for further processing. This code is called, in osm2pgsql-speak, the “middle”. There are two implementations of this “middle”, one stores the data in RAM and doesn’t support updates. The other stores the data in the database and supports updates. Back in 2020 I started looking into that code to see what could be improved. In 2021 we released version 1.5.0 which contained a basically rewritten RAM middle. And now, in 2023, I finally got around to improving the database middle.

| Read more…

Tags: openstreetmap · osm2pgsql


Osm2pgsql Middle Improvements

The osm2pgsql code contains a part called the “middle”. The middle is responsible for keeping track of all OSM objects read from the input file(s), their attributes, tags, and, most importantly, the relations between those objects. This is needed for several reasons.

| Read more…

Tags: OSMF · openstreetmap · osm2pgsql · software development


Osm2pgsql Work

Back in September we launched the new osm2pgsql.org website. At that time it was still quite rudimentary. Over the last months a big part of my work for the OSM Foundation was to work on that site.

| Read more…

Tags: OSMF · openstreetmap · osm2pgsql · software development


Osm2pgsql has a new Website

Today we launched the new osm2pgsql.org website. This is the first visible result of the support by the OSM Foundation for osm2pgsql development. Over the next months I’ll keep working on osm2pgsql, payed by OSMF and other sponsors.

| Read more…

Tags: OSMF · openstreetmap · osm2pgsql · software development


Osm2pgsql Release

Today Sarah released the new osm2pgsql version 1.3.0 (release notes). After working on this for more than half a year now it is great to see the new version out there.

| Read more…

Tags: openstreetmap · osm2pgsql · software development


New flex output in osm2pgsql

Last week I wrote about Paying off technical debt in osm2pgsql. The work described there came about while I was adding the new “flex output” to osm2pgsql. In osm2pgsql terminology “outputs” are the parts of osm2pgsql responsible for converting the OSM data into a format suitable for the database and then adding it to the database. There are several outputs, the classic “pgsql” output, the “gazetteer” output used for Nominatim imports, and the “multi” output, an earlier attempt at adding more flexibility to osm2pgsql that never got much traction. The new “flex” output rethinks this job and adds, as the name suggest, a lot of flexibility. When I worked on it I needed changes in other parts of osm2pgsql outside the output structure itself, so that’s where the cleanup job I talked about last week started. In this blog post I want to write a little more about the flex output, the ideas behind it, and my work on it.

| Read more…

Tags: openstreetmap · osm2pgsql · software development


Paying off technical debt in Osm2pgsql

Osm2pgsql is one of the work horses of the OpenStreetMap world. It is used to import OSM data into a database which is then used for rendering maps (among other uses). Osm2pgsql is behind the map you see on openstreetmap.org and many others. The program has been around since 2006 and has accumulated many changes and additions over time. And the code has grown more and more complex. Part of that is inevitable because the program does a lot more than when it was started. But part of that is complexity that doesn’t have to be there. For a few months now I have been working on reducing that complexity and want to tell you about that work.

| Read more…

Tags: openstreetmap · osm2pgsql · software development