Yesterday I released version 2.17.0 of the Osmium library. This version is noticeable faster when reading and writing PBF files.

The last few days saw releases of my Osmium library (2.15.0), the Python bindings for Osmium (PyOsmium, 2.15.0), the Osmium command line tool (1.10.0), and the OSMCoastline (2.2.1) program. In this blog post I want to highlight some of the changes.

After a long time I have finally released new versions of the Osmium command line tool and the Osmium C++ library. PyOsmium also got a new release to match the libosmium version.

Recently I released new versions of the Osmium library (libosmium version 2.13) and the Osmium command line tool (version 1.7) with some major improvements I want to talk about.

There is too much OSM data! Of course we always want more, more, more data. But when all you need is the data for your city or country, handling 34 GByte of (densly compressed) OSM data for the whole planet can be overwhelming. So one of the most often needed tools for working with OSM data is something that will cut out the region you are interested in from the huge planet file.

Its high time I write something about the news in the Osmium world. After 10 days or so of multiple releases in short succession things have quieted down some and I think we have something more stable now.

Yesterday I released version 2.9.0 of libosmium and version 1.4.0 of the Osmium command line tool. There are quite a few significant changes.

OpenStreetMap has a somewhat peculiar data model. OSM ways don’t contain the locations of the points they go through, but only references to nodes, which in turn contain their location. This model has some advantages, but it also has a major disadvantage: For almost all uses of OSM, like drawing maps or calculating routes, the software has first to find the locations of all nodes in a way, before it can do anything with it. This is cumbersome and potentially expensive. To do this quickly, the lookup needs to be done in main memory without any disk access, but to store the locations of all nodes of a full planet file we already need about 32GB RAM. But maybe we can create a shortcut, so that we can do this once, instead of everybody doing this again and again?

The number of objects in OpenStreetMap is growing and with it the number of unique IDs. For node IDs we have gone over 4 billion now. Soon they will not fit into an unsigned 32 bit integer any more (2³²=4294967296).

Matias Dahl has looked at historic OSM data and done some analysis on changing use of the amenity tag including some pretty cool graphs. Nice work and potentially very useful. And he used node-osmium for it. Now, how do we get something like this into a continuously updated service, maybe even into taginfo?

It has been a long time since I wrote something about Osmium in this blog. Considering that I have worked on Osmium and related software almost every day for many months now, this is a huge omission. I want to make up for this a little bit with this posting.

When I started using the ZSH shell years ago, it was because of its superior command line completion features. Since then, the Bash shell has improved a lot, and probably has the same features, but I am used to ZSH now. And when a command doesn’t have completion it feels cumbersome to use. So, of course, I wanted to have completion for the osmium tool.

Last weekend we had the SOTM-EU here in Karlsruhe. I had a lot of fun, but didn’t get enough sleep, something you’ll notice in the video from my Osmium talk. Oh well, not everybody can have their talk on the first day.

Over the last months I have been busy working on a project for Mapbox. As part of that project I have spent a lot of time improving Osmium. And not just a few changes here or there but a more or less complete redesign based on the experience of developing and using Osmium for the last nearly three years now.

Osmium has had support for assembling multipolygons from relations for a long time. But the code had been lifted from an earlier Geofabrik project and was rather messy, hard to extend, and it contained several memory leaks. In the last month or so I finally chewed through the whole thing bit by bit and made it prettier.

I have done some long-overdue refactoring of the Osmium code which simplifies Osmium itself and simplifies its use. And should make it faster to compile, because only those parts that are really needed are included.

I just noticed that someone has created official Debian packages for Osmium (libosmium-dev, osmjs). They have also found their way into the new Ubuntu 12.04 version. Unfortunately they are already outdated. It is fine if you only want to use osmjs, but if you do any development based on Osmium, I suggest you use the version from github.

For a while now I have been working on OSMCoastline adding features bit by bit. (See this blog article for some background info.) OSMCoastline now closes small gaps in the coastline and it can split up the huge polygons into smaller ones. You can create land polygons or water polygons depending on your needs and you can have it all in WGS84 or in the usual Mercator projection used for tiles web maps (EPSG 3857). So it does a lot more than the coastcheck application used for these things before. And the software is still a lot faster.

Today the April 2012 issue of the German computer magazine “iX” was in my mailbox. It contains my four and a half page article on how to create your own maps with OSM data. I describe how to use Osmium/osmjs software to extract the data and Tilemill to style and create the maps.

I gave two talks at the FOSSGIS 2012 conference last week. The first, OpenStreetMap in Zahlen und Karten, was not recorded and I will not release the slides as they will be difficult to understand and easy to mis-understand without my words. I will probably re-use the content for a few blog articles here in the future.

Next week is the FOSSGIS conference in Dessau. I’ll be speaking there about OpenStreetMap in Zahlen und Karten and Das Osmium-Framework. Together with Tim Alder I am also organising a community session to discuss the Wikipedia Multilingual Maps Project. See you in Dessau!

Every map of the world has to take into account that there are land areas and water areas. You could have huge polygons for each continent and paint them on a blue background. But handling those huge polygons is difficult. OpenStreetMap solves this problem by having ways tagged with “natural=coastline” and the convention that the land is always on the left side of this way. So a continent or an island is surrounded by one or more ways in counter-clockwise order. It is much easier to edit those coastlines instead of large multipolygon relations or something like it. Coastlines are the only case where OSM has this special rule.

The Karlsruhe Hacking Weekend is over. 18 people were there at least part of the time. I had a lot of fun and even got a little bit of work done: As mention I worked on Taginfo and on the second day I worked a bit on the statistics for my upcoming talk at the FOSSGIS conference. I am using Osmium and full-history planet dumps to see what has been happening in OSM. I’ll publish some code and graphics at some point.

The Osmium framework has been around for about a year and a half now and has grown to include a lot of functionality. But there is still much to to. In this blog post, I have collected some of those things. There is no particular order to the following list.

Osmium now includes OGR support. OGR is a popular Open Source library for converting different GIS storage formats into each other. It supports PostGIS, Shapefiles, Spatialite, and many other formats. I had started working on OGR integration many weeks ago, but didn’t quite finish it. I have now added the last bit: Support for OGR Multipolygons. This opens up a lot of new possibilities for Osmium users.

Osmium is getting bigger and bigger and doing more and more. But testing the code is still done manually and ad-hoc. Not good. I have used some testing frameworks in other languages, but never in C++. As usual, Boost comes to the rescue. I recently added some test code using the Boost Unit Test Framework which (at least for now) does all I need.

The organisers of the State of the Map Europe have published the programme and it looks very interesting. I will give a talk about Taginfo and a talk about Osmium, the two projects I currently spend my free time on.