This blog post is about the new OSM XRAY service. You can find a demo here. It uses vector tiles with all OSM data to really show all of OSM, mostly for debugging. But it is also interesting to just explore what’s there.

I have to import a lot of OSM planet files to test things due to my work on osm2pgsql. I have a test machine for that graciously provided by FOSSGIS e.V., the OSM Foundation local chapter for Germany. A few days ago I imported a planet file and osm2pgsql ran out of memory.

Most OpenStreetMap-based maps you see these days on the Internet are only updated every so often. Some may be updated daily, most are probably only updated every few weeks or months. And that is totally okay for most use cases. But there are some maps that are updated continuously so that changes in the OSM data appear as fast as possible, which is important to motivate OSM contributors and helps with quickly detecting and fixing problems. The most important map with continuous updates is the main map you see on www.openstreetmap.org.

In osm2pgsql there is that code that stores the raw data it gets from the input files and makes it available to other parts of osm2pgsql for further processing. This code is called, in osm2pgsql-speak, the “middle”. There are two implementations of this “middle”, one stores the data in RAM and doesn’t support updates. The other stores the data in the database and supports updates. Back in 2020 I started looking into that code to see what could be improved. In 2021 we released version 1.5.0 which contained a basically rewritten RAM middle. And now, in 2023, I finally got around to improving the database middle.

Taginfo has as its goal to find and collect all useful knowledge about OSM tags and make it accessible. But taginfo can’t do everything. That’s why it has always linked to tools that “do something with OSM tags”, the most popular one is probably Overpass Turbo.

About once a year I set aside a week to work on taginfo, one of my many projects that doesn’t get enough attention. And inevitably once I get started there are so many things I want to do that it takes two weeks instead of the originally allotted week. So this time I thought I plan with the two weeks from the start. You probably know where this goes: I spend more like three or four weeks on taginfo…

For the last six months I have been working on the project to add generalization support to osm2pgsql. This project was funded by the German Ministry of Education and Research (and organized by the Prototype Fund). Many thanks to them for giving me this opportunity!

Several of the generalization algorithms I am working on are based on tiles. They collect all features inside a tile boundary and do something with them. This means that whenever something changes, we have to figure out which tiles are affected and re-process them. Luckily a very similar functionality is also needed for generating lists of tiles that need to be re-rendered. And support for that has been in osm2pgsql for a long time. We just need some additions to be able to re-purpose that code.

Large waterways such as rivers and canals are important features on small scale maps. This is the fourth problem I wanted to work on described in this blog post.

Many maps show built-up areas of cities and other settlements in a different color on scales where you can’t show individual buildings. This gives the viewer an immediate idea on where the landscape is more urban and where it is more rural. This is the third problem I wanted to work on described in this blog post.

Most maps need little symbols and labels for settlements like cities and towns. Of course you can’t put all settlements on the map, so you have to select those you want to show. For that you need to “rate” the places in some way, so that you can show only the more important ones. And then you want to show a “reasonable” selection of places, not too many so the map isn’t too crowded, not too few to make the map look empty.

Like any project, the generalization project I am working on has its share of experiments. Not everything you try out will work in the end, or is fast enough, or actually needed, etc. These experiments often don’t end up in the progress reports or final reports, but without them we can’t create something new. I want to talk about three experiments related to raster data in PostgreSQL/PostGIS, one successful, one failed and one where the outcome is still open.

Maps almost always have lots of large and often complex polygons showing lakes, forests, urban areas, countries and so on. On a small scale map all you are going to see of some of these polygons is a few colored pixels. For that you don’t need the detailed data.

I want to write a bit about the processing architecture of osm2pgsql and how I plan to extend it to help with generalization.

In cartography you often have to find points that “represent” a polygon in some way, the exact outline of the polygon isn’t important, you just need a single position somewhere on that polygon, for instance to draw an icon there. In OpenStreetMap points of interest are sometimes modelled as polygons (usually building outlines) for example. A similar problem is finding a labelling point of a polygon, that is the point where to put the name label of a forest or a lake or a country.

To start the OSM generalization project I decided to look at some specific generalization problems with a wide variety of use cases and challenges, and with different types of solutions. It is obvious that I can’t implement every algorithm out there and solve every generalization problem, but I can take a sample of typical problems with different solutions and try to implement them. This should hopefully give us a better understanding of the problem space and a basis on which we can build later.

OpenStreetMap data is often incredibly detailed and the generated maps look amazing. But there is a problem for maps in smaller zoom levels/on smaller scales: There is often too much detail. Maps become slow to render and cluttered, the important information is hard to see. To solve this the map data needs to be generalized.

It’s two weeks after the State of the Map 2022 in Florence and its high time I write this blog post. I had a great time and can only agree with what Ilya said in the closing session, how great it was to see everyone again (or for the first time) after such a long dry spell (Ilya’s words were much better though).

In the last months I have been working on a study commissioned by the EWG about ideas to improve the OSM data model. This study is now finished and I have sent it to the EWG today. You can already find it on my website. The EWG might also publish it somewhere official after review. I am also going to talk about the study at the State of the Map conference in Florence on Friday, August 19, 2022. I am looking forward to lots of discussions at SotM and in the coming months.

Yesterday I released version 2.17.0 of the Osmium library. This version is noticeable faster when reading and writing PBF files.

The osm2pgsql code contains a part called the “middle”. The middle is responsible for keeping track of all OSM objects read from the input file(s), their attributes, tags, and, most importantly, the relations between those objects. This is needed for several reasons.

Back in September we launched the new osm2pgsql.org website. At that time it was still quite rudimentary. Over the last months a big part of my work for the OSM Foundation was to work on that site.

10 years ago I started the taginfo project. To celebrate this I am introducing some new features!

Today we launched the new osm2pgsql.org website. This is the first visible result of the support by the OSM Foundation for osm2pgsql development. Over the next months I’ll keep working on osm2pgsql, payed by OSMF and other sponsors.

Today Sarah released the new osm2pgsql version 1.3.0 (release notes). After working on this for more than half a year now it is great to see the new version out there.

Last week I wrote about Paying off technical debt in osm2pgsql. The work described there came about while I was adding the new “flex output” to osm2pgsql. In osm2pgsql terminology “outputs” are the parts of osm2pgsql responsible for converting the OSM data into a format suitable for the database and then adding it to the database. There are several outputs, the classic “pgsql” output, the “gazetteer” output used for Nominatim imports, and the “multi” output, an earlier attempt at adding more flexibility to osm2pgsql that never got much traction. The new “flex” output rethinks this job and adds, as the name suggest, a lot of flexibility. When I worked on it I needed changes in other parts of osm2pgsql outside the output structure itself, so that’s where the cleanup job I talked about last week started. In this blog post I want to write a little more about the flex output, the ideas behind it, and my work on it.

Osm2pgsql is one of the work horses of the OpenStreetMap world. It is used to import OSM data into a database which is then used for rendering maps (among other uses). Osm2pgsql is behind the map you see on openstreetmap.org and many others. The program has been around since 2006 and has accumulated many changes and additions over time. And the code has grown more and more complex. Part of that is inevitable because the program does a lot more than when it was started. But part of that is complexity that doesn’t have to be there. For a few months now I have been working on reducing that complexity and want to tell you about that work.

Since March the OSMData Service has been running on new servers sponsored by FOSSGIS e.V. I wrote about the switch in an earlier blog post. Now, half a year later, I want to write about my experiences with the new setup.

I was recently at the State of The Map conference in Heidelberg. The SoTM is always a lot of fun, lots of old friends to see again and new people to meet. And lots of talks to listen to, which I mostly skip, because there are so many interesting people around to talk to in person. And all the talks are available on video, so I can watch them in the weeks and months ahead. I also gave a talk myself, on OSM Data Processing with PostgreSQL / PostGIS, a tour de force through “everything” related to OSM and PostgreSQL/PostGIS. In an hour I covered a huge amount of ground from basics of relational databases and spatial data formats to the tools used to importing OSM data into the database. I am glad this talk is available on video, too, and I hope it will help people getting an overview over this huge topic in years to come. You can watch the talk and get the slides on my web page.

For many year now I have been running a service at <a openstreetmapdata.com, first alone and then with the help of Christoph Hormann. We offered processed OSM data for download, most importantly coastline and Antarctic icesheet data. And although the service was always free to use and the software mostly Open Source, this was always our show. For a long time I wanted to change this and turn this into a proper OSM community project, and, finally, we have taken a big step towards that.

The last few days saw releases of my Osmium library (2.15.0), the Python bindings for Osmium (PyOsmium, 2.15.0), the Osmium command line tool (1.10.0), and the OSMCoastline (2.2.1) program. In this blog post I want to highlight some of the changes.

There are many tools that help with quality assurance in OpenStreetMap and they come in many flavours. One popular class of these tools are web sites showing the OSM map in the background and (usually switchable) overlays with markers for different types of problematic OSM data. They usually allow you to jump directly into the OSM editors to fix the problems. The most prominent of these tools are Osmose, the OpenStreetMap Inspector, and KeepRight. But there are many more of these, often written for very specific use cases. All of these tools have the drawback that they are “closed systems”: The person developing the tool and/or running the web service decides which layers there are. The creation of those layers is often tied more or less closely to the tool showing the layers. At the recent Karlsruhe hack weekend I released a new tool, Osmoscope, of that same class, but with much more openness built in.

An international project like OpenStreetMap needs localized documentation. Ideally everybody can use OSM and contribute to OSM knowing only their own language. Of course we are far away from that goal and will probably never fully reach it, but it is worth the effort to try to get there. Taginfo did an important step into the right direction in the last weeks.

After a long time I have finally released new versions of the Osmium command line tool and the Osmium C++ library. PyOsmium also got a new release to match the libosmium version.

Recently I released new versions of the Osmium library (libosmium version 2.13) and the Osmium command line tool (version 1.7) with some major improvements I want to talk about.

In February of this year I started a community effort to fix broken multipolygons in OSM and switch the remaining old-style multipolygons (with the tags on the outer ways instead of on the relations) to a more modern tagging. Seven months later I am declaring this effort to be finished.

The Multipolygon Fixing Effort started a few weeks ago is going well. An amazing amount of work has already been done, but there is a lot more to do. I have started a blog (sort of) specifically to document what is happening with that effort. See you there!

There are a lot of (multi)polygons in OSM that are broken in one way or another. And we have to fix them. While some of the broken ones appear on the map just fine, some don’t appear and some mess up the map. And some of those that appear fine on the main OSM map will not show up on other maps where different software is used.

There is too much OSM data! Of course we always want more, more, more data. But when all you need is the data for your city or country, handling 34 GByte of (densly compressed) OSM data for the whole planet can be overwhelming. So one of the most often needed tools for working with OSM data is something that will cut out the region you are interested in from the huge planet file.

Its high time I write something about the news in the Osmium world. After 10 days or so of multiple releases in short succession things have quieted down some and I think we have something more stable now.

I am busy preparing for two events: Only two days now until the State of the Map 2016 in Brussels, where I’ll be giving a talk about taginfo. On Sunday I’ll be hosting an Area Workshop where we want to talk about the effort to fix broken and inefficiently tagged (multi)polygons in OSM. I’ll be around for (part of) the hack day, too.

Yesterday I released version 2.9.0 of libosmium and version 1.4.0 of the Osmium command line tool. There are quite a few significant changes.

The most requested features of taginfo is a view showing historical use of tags. I never had the time to add this, but fortunately there is now a solution: Martin Raifer created a service that shows the history of tags in nice graphs. You can see it at taghistory.raifer.tech. Some more background in a diary entry from Martin. Matthijs Melissen has already used this to find some interesting statistics.

About two years ago I introduced the taginfo “projects” feature: A way to connect taginfo with all the uses of OSM tags out there, to editors, special maps, routing services etc. By now we have about 50 projects that have published a project file describing their tag use. This data gets integrated into taginfo and helps mappers and users of OSM data to see who actually uses which tags. Today I have released some changes to taginfo that integrate this information even better.

OpenStreetMap has a somewhat peculiar data model. OSM ways don’t contain the locations of the points they go through, but only references to nodes, which in turn contain their location. This model has some advantages, but it also has a major disadvantage: For almost all uses of OSM, like drawing maps or calculating routes, the software has first to find the locations of all nodes in a way, before it can do anything with it. This is cumbersome and potentially expensive. To do this quickly, the lookup needs to be done in main memory without any disk access, but to store the locations of all nodes of a full planet file we already need about 32GB RAM. But maybe we can create a shortcut, so that we can do this once, instead of everybody doing this again and again?

The number of objects in OpenStreetMap is growing and with it the number of unique IDs. For node IDs we have gone over 4 billion now. Soon they will not fit into an unsigned 32 bit integer any more (2³²=4294967296).

I recently gave a talk about my experiences with the C++ Catch Unit Test Framework at our local C++ user group. The slides and video from the talk (in German) are now available on my website.

Matias Dahl has looked at historic OSM data and done some analysis on changing use of the amenity tag including some pretty cool graphs. Nice work and potentially very useful. And he used node-osmium for it. Now, how do we get something like this into a continuously updated service, maybe even into taginfo?

It has been a long time since I wrote something about Osmium in this blog. Considering that I have worked on Osmium and related software almost every day for many months now, this is a huge omission. I want to make up for this a little bit with this posting.

The recent London OSM hack weekend gave me the incentive to work on taginfo once more and we got quite a few things done on the weekend and the days after. Taginfo now works better on mobile, is faster and we have a very cool tag list feature for the wiki. Lets look at these in turn…

Recently I was working on some OSM software when I noticed a strange thing…

Only one week before SOTM-US starts. I’ll be there giving a talk at the conference and a workshop on hack day. Hope to see you there!

Three years ago I launched the OpenStreetMapData.com [Update: Discontinued June 2019, some data now available from osmdata.openstreetmap.de]. web site to host the OSM coastline data extracts I was generating. Since then I have been updating those extracts at least daily and offering them for download on that site. The coastline extracts are used in many OSM maps including the OSM standard style.

Last week I gave a talk about taginfo at the FOSSGIS 2015 conference. Thanks to the amazing video team all talks from the conference are available online in great quality. As always you can find my talk on my Talks page.

I just added a few features to taginfo: There is a new “Similar” tab on all key pages, see for instance all keys similar to the “highway” key. This makes it easy to find related keys and typos. The new Similar keys report shows keys that are commonly used together with similar keys that are rarely used. Often, but not always, the rare keys are just misspellings.

The next annual FOSSGIS conference on Open Source geographic information systems and OpenStreetMap will be held in Münster, Germany, from March 11th to 13th. I’ll be giving a talk there about taginfo.

I am a heavy git user. I am actively developing in about half a dozen repositories at the same time and have many more I occasionally check out or add to. I have files lying around that need to be added, changes to be committed, commits to be push, all in several branches, local and remote. And then there is the stuff I stashed months ago and forgot…

Taginfo has the “mission” of aggregating all available information about OSM tag usage and present it to the users. The main sources of information are the OSM database and the documentation in the OSM wiki. But there are many more sources of information to be tapped. Everybody who creates a map, or writes presets for an editor, everybody writing a data export program or a routing application has to decide which tags to use in what way. All this is useful information, but it is hard to come by. Everybody uses different programming languages and different config file formats. And all this “stuff” is spread out over many software repositories and web sites.

When I started using the ZSH shell years ago, it was because of its superior command line completion features. Since then, the Bash shell has improved a lot, and probably has the same features, but I am used to ZSH now. And when a command doesn’t have completion it feels cumbersome to use. So, of course, I wanted to have completion for the osmium tool.

Last weekend we had the SOTM-EU here in Karlsruhe. I had a lot of fun, but didn’t get enough sleep, something you’ll notice in the video from my Osmium talk. Oh well, not everybody can have their talk on the first day.

I spent a few days working on taginfo again. I fixed some bugs and refactored code here and there and added a heap of new features.

QGIS caches the data for the layers it is showing. This can be a bit annoying when the data changes and QGIS doesn’t know about this. What is missing is some kind of “reload layer data” function. Instead I have to remove layers and add them again all the time. Or, instead of reloading each layer, I reload the whole project. Still, this needs several clicks and after the reload you are back to the project extent instead of where you were before the reload.

Next week the (German language) FOSSGIS-Konferenz takes place in Berlin. For the first time since 2007 I am not giving a talk there, but I’ll be around all three days to learn new things and (re-)connect with people from the OSM and Open Source GIS communities. Lars Lingner and I are also organizing a BoF session for freelancers. We both are have been seeing more and more interest in OSM and GIS development and services and are often contacted for projects larger than one person can handle or where different skillsets are needed. We want to better connect the freelancers in these fields to to able to help each other out. So if you are working freelance or have a small company in the OSM or Open Source GIS fields, please come join us on Wednesday for a chat.

I have been writing lots of software for the OpenStreetMap project and there is always one problem: How to test it? There is a huge variety of data in OSM, every conceivable way of tagging, every way of getting the data slightly (or not so slightly) wrong, will sooner or later appear. We need to test file formats, OSM objects syntax and semantic, changing data, cases with failing referential integrity or invalid geometries and much much more. But just coming up with all those test cases is a huge task and implementing those tests is even more work.

In October 2010 I introduced taginfo. Since then I have updated it occasionally and added new functions. And over time it has become part of the “OSM landscape”.

On January 16 2014 I will give a talk about the OpenStreetMap data model at the HafenCity University in Hamburg. The talk will be in German. Everybody is welcome to attend, you don’t need to be a student there.

I have written before about my work on the software doing the assembly, repair, and conversion of the OSM coastline data. I wrote the Osmcoastline software and I run it daily to create shapefiles with land and water polygons available for download on OpenStreetMapData.com [Update: Discontinued June 2019, now available from osmdata.openstreetmap.de]. For a while this data has now been used on the main OSM map. Unlike with the old process, this new process usually makes sure the map is updated within a few days when the coastline changes.

A few weeks back I gave two talks at the State Of The Map 2013 conference. One about Osmium, the other called “Towards an Area Datatype for OSM”. The slides and videos from both talks are now available on my talks page. Thanks to Peter Barth for “liberating” the videos from the 8 hour long flash-only videos put up originally.

OpenStreetMap doesn’t allow multiple tags with the same tag key. But there is a believe out there in OSM land, that you can work around this by putting all the tag values you want into one tag using semicolons as separator characters. So if there is a bank with an ATM machine, people tag [amenity=bank;atm]. Or there is a road with several refs, then they tag them as [ref=I 70;US 40] and hope that it will magically work. And it does work in some cases. But in many cases it doesn’t. Let’s look into this a bit.

Over the last months I have been busy working on a project for Mapbox. As part of that project I have spent a lot of time improving Osmium. And not just a few changes here or there but a more or less complete redesign based on the experience of developing and using Osmium for the last nearly three years now.

Last week I was at the yearly FOSSGIS conference, the German language conference for Open Source GIS, Open Data and OpenStreetMap. This year it took place in Rapperswil, Switzerland near Zurich. I like the relaxed atmosphere and the good mixture of introductory talks and talks for power users.

I have had a personal web page for a long time. I started creating web pages in 1993 when I was a student at Freiburg University. At that time I didn’t have any way to publish it. But when I moved to Karlsruhe later that year I could put up my first real public home page on the university server there. In 1995 my web pages moved to the INKA web server and in 1998 they moved again to my new home in cyberspace on remote.org. In 2005 I added a web page for my “professional self” at www.jtic.de.

The yearly FOSSGIS conference, the German language conference for Open Source GIS, Open Data and OpenStreetMap, will take place from June 12th to June 14th 2013 in Rapperswil, Switzerland near Zurich.

A year ago I released the first version of the OSMCoastline program that extracts and assembles coastlines from OSM data. Since then I have learned more about coastlines in OSM than I ever wanted to know, fixed numerous coastline bugs myself and added more features to the software. OSMCoastline has been in production use at Geofabrik and the German map server for more six months now.

I have written before about the mapping of Antarctia in OSM. Since then there has been a lot of activity. Christoph Hormann has prepared better Antarctica data for import and has documented the planned import extensively on the wiki. Peter Körner has created a web map in Polar Stereographic Projection.

The Map Features wiki page is one of the oldest pages on the OSM wiki and it has been getting larger and larger over time. And more and more wiki templates are used to create it and its language variants. Very similar tables can also be found on many “Key:*” pages on the wiki to list the tags used with this key.

OpenStreetMap wants to map the whole world, but there is a whole continent that has been neglected a bit: Antarctica. Of course there isn’t much there besides rocks, ice, and penguins and not too many people live there. But still, it is a huge area and it should appear properly on our maps.

A map and routing service for the Kathmandu public transport system. And how the data was collected. This is why I love OSM.

It has been nearly a year since I did some serious work on taginfo. Taginfo works well, but there has been and still is a backlog of todo items. And I get new ideas on how to improve it all the time. So I spent the first days of the new year doing some work on it. A lot of this work has been “behind the scenes”, improving code here or there, so there isn’t actually that much to see. (If you are taking a look remember that you might have to clear your browser cache for all the goodies to show up properly.) But here are the larger improvements:

It is high time I’ll write something about the status of the Wikipedia Multilingual Maps project I have been working on for many months now.

One of the biggest problems with the OSM data model is that there are no real polygons. Polygons or multipolygons are “simulated” using closed ways or multipolygon relations and several other methods. These multipolygon relations are often broken. Some of the problems resulting from this non-existence of multipolygons are affecting many people, for instance with broken coastlines and country or other boundaries.

In a recent discussion on the OpenStreetMap talk mailing list around imports in France a point came up again that has been raised a few times: If we had some notion of layers in OSM data, maybe, some tasks such as imports would be easier or could be done in a better way. I want to look into this “layer” issue a little bit.

Yesterday Development Seed/MapBox announced that they received a grant of $575,000 to “improve the core infrastructure of OpenStreetMap”. This is a great development. We do have a lot of problems and there are many things that can be improved. The community has been working on all of these things over the years of course, but I do think that we lag behind the growth curve of OSM and any help is appreciated.

A while ago I wrote a small QGIS plugin, to control a running JOSM or other OSM editor that supports the remote control API from QGIS. I finally got around to add it to the official repository so that you can now install it with a few clicks from inside QGIS.

Osmium has had support for assembling multipolygons from relations for a long time. But the code had been lifted from an earlier Geofabrik project and was rather messy, hard to extend, and it contained several memory leaks. In the last month or so I finally chewed through the whole thing bit by bit and made it prettier.

I have done some long-overdue refactoring of the Osmium code which simplifies Osmium itself and simplifies its use. And should make it faster to compile, because only those parts that are really needed are included.

After setting up the MapQuest Render Stack and poking around in its code I consider using it for the Multilingual Maps project. Unlike Tirex it can run on a cluster of machines for reliability and load distribution. That will be important for a tile server working on “Wikipedia scale”.

Whatever way we’ll develop for rendering the multilingual tiles, we have to get the information about which language(s) the user wants from the user to the tileserver. Web browsers typically support the selection of an ordered list of preferred languages. This list is sent through the Accept-Language header to the web server. We could use this setting to determine the language for the labels. But in some situations people might not know about this setting or can’t change it. Maybe they are sitting in an Internet cafe in a foreign country. In that case it might be easier for them if they can just change the language setting on the web page. This is especially interesting if the web site knows which languages are available and only shows those options. The browser setting doesn’t know anything about actually available languages, it just has one large master list.

Occasionally people ask me why there is no comment function on this blog. There are two answers for that: First, the software I use to create this blog does not support comments. My blog posts are “compiled” into static web pages that I then upload to my public site. This has many advantages. The site is fast, I don’t need a database around, the files are easy to backup and can be kept under version control. But mainly it means I can sleep soundly, because I don’t have to update Wordpress or some other blogging software every time a new security vulnerability is found.

If we want to render multilingual maps by separating the “label layers” from the “base layers” as described in my last blog post, we have to make sure the rendering results don’t change. Or, at least, that they do not look too bad. Lets look into that.

As a proof-of-concept for the rendering of labels on demand I set up an OpenLayers map with two XYZ tile layers, one for the background and one for the labels. I configured the MapQuest render stack to deliver those two layers, the background layer is stored in the tile store, the label layer is never stored, but always rendered on demand.

Nearly everybody setting up an OSM tile server uses either the Apache/mod_tile/renderd or the Apache/mod_tile/Tirex combination. The renderd software has been around for while. When we needed something more flexible two years ago, Frederik Ramm and I created Tirex. To keep the development work needed small and make it easier to switch from renderd, Tirex uses the same Apache/mod_tile frontend as renderd. We had plans to add different frontends, but they never materialized. Apache/mod_tile is difficult to configure and not very flexible, but it does its job well enough. I wrote a proof-of-concept frontend using node.js, but that was never taken up either.

In Choosing a Language I have proposed to think about how to render labels in different languages into the tiles on the fly when they are requested.

One key issue of the multilingual map project is obviously how we choose which language labels to render into a map. There are two sides to consider:

(This article is a (modified) translation of a German language article I wrote for the German OpenStreetMap blog. It should have been translated months ago, but I am only now getting around to it. The project was a bit slow in starting but is gathering momentum now. It has progressed a bit since the original article.)

In the last months I have spent a lot of my spare time working on the OSMCoastline program mentioned several times in this blog already. It is by no means perfect yet, but it works and several people have tried it and used its output. But it is not the easiest program to use with all its options and you need a current planet file to work from, not everybody has one lying around. So I thought about how to make access to its output easier for everybody.

I just noticed that someone has created official Debian packages for Osmium (libosmium-dev, osmjs). They have also found their way into the new Ubuntu 12.04 version. Unfortunately they are already outdated. It is fine if you only want to use osmjs, but if you do any development based on Osmium, I suggest you use the version from github.

For a while now I have been working on OSMCoastline adding features bit by bit. (See this blog article for some background info.) OSMCoastline now closes small gaps in the coastline and it can split up the huge polygons into smaller ones. You can create land polygons or water polygons depending on your needs and you can have it all in WGS84 or in the usual Mercator projection used for tiles web maps (EPSG 3857). So it does a lot more than the coastcheck application used for these things before. And the software is still a lot faster.

Today the April 2012 issue of the German computer magazine “iX” was in my mailbox. It contains my four and a half page article on how to create your own maps with OSM data. I describe how to use Osmium/osmjs software to extract the data and Tilemill to style and create the maps.

I gave two talks at the FOSSGIS 2012 conference last week. The first, OpenStreetMap in Zahlen und Karten, was not recorded and I will not release the slides as they will be difficult to understand and easy to mis-understand without my words. I will probably re-use the content for a few blog articles here in the future.

Next week is the FOSSGIS conference in Dessau. I’ll be speaking there about OpenStreetMap in Zahlen und Karten and Das Osmium-Framework. Together with Tim Alder I am also organising a community session to discuss the Wikipedia Multilingual Maps Project. See you in Dessau!

Every map of the world has to take into account that there are land areas and water areas. You could have huge polygons for each continent and paint them on a blue background. But handling those huge polygons is difficult. OpenStreetMap solves this problem by having ways tagged with “natural=coastline” and the convention that the land is always on the left side of this way. So a continent or an island is surrounded by one or more ways in counter-clockwise order. It is much easier to edit those coastlines instead of large multipolygon relations or something like it. Coastlines are the only case where OSM has this special rule.

The Karlsruhe Hacking Weekend is over. 18 people were there at least part of the time. I had a lot of fun and even got a little bit of work done: As mention I worked on Taginfo and on the second day I worked a bit on the statistics for my upcoming talk at the FOSSGIS conference. I am using Osmium and full-history planet dumps to see what has been happening in OSM. I’ll publish some code and graphics at some point.

The Karlsruhe Hacking Weekend is in full swing. About 15 people are sitting around, munching apples, pizza and Gummi Bears and talking about things “somebody” should do “someday”. And we do some actual work, too. I fixed a few bugs in Taginfo, Sven and others are working on updating the German map style. Some people are talking about obscure Mapnik config options. No idea what the other people are doing.

I recently started writing my own RSS and Atom feed reader. I’ll write another blog post about why and how I am doing that. Today I want to focus just on one part of the job: Reading the RSS and Atom feeds.

The Osmium framework has been around for about a year and a half now and has grown to include a lot of functionality. But there is still much to to. In this blog post, I have collected some of those things. There is no particular order to the following list.

In recent Linux kernels something changed in the way CDROM/DVD ejects are handled. On my notebook with current Ubuntu (11.10) the result was that “eject -i” does not work any more to block the physical eject button. Thats very annoying as I kept hitting it accidentally. But I found a way to get it working again: Copy /lib/udev/rules.d/60-cdrom_id.rules to /etc/udev/rules.d and comment out the line with DISC_EJECT_REQUEST. Now “eject -i” works again.

Time for the next OpenStreetMap hack weekend. Hope to see you there.

Lately I have been working on OSM-derived data in QGIS and finding bugs in the OSM data in the process. To fix the data I have to go into an OSM editor (that’s JOSM for me), find the right position and do the actual fix. To make that process easier I added a button to the QGIS user interface that uses the JOSM remote control feature to tell the editor which area to load data for.

I spend a lot of time slicing and dicing OSM and other geodata. Sometimes that is done with custom software but often a database is involved somewhere. Recently that sometimes has been the SQLite/SpatiaLite duo which works great for simple things. But the heavy lifting has to be done with PostgreSQL/PostGIS which has much more “geo functionality” and is much faster when complex queries are used. So here comes the first tip:

While working with OpenStreetMap or other geodata I often stumble upon interesting (or just pretty) visualization of the data. I have started a collection on my web page (Updated 2018-08-10: Originally this linked to a flickr page). Some of these maps were created “on purpose”, some just appeared on my screen by accident. Maybe it inspires someone to create their own whacky visualizations.

Osmium now includes OGR support. OGR is a popular Open Source library for converting different GIS storage formats into each other. It supports PostGIS, Shapefiles, Spatialite, and many other formats. I had started working on OGR integration many weeks ago, but didn’t quite finish it. I have now added the last bit: Support for OGR Multipolygons. This opens up a lot of new possibilities for Osmium users.

More and more boundaries of administrative areas find their way into OpenStreetMap. In some countries there is already quite a lot of data for several levels of government. Unfortunately it is hard to edit and hard to use because relations must be used to model areas in OSM and relations are notoriously difficult to handle. Relations and with it the areas they represent are often broken as a result. Its therefore hard to get a complete set of administrative areas for a particular country and level of government.

Osmium is getting bigger and bigger and doing more and more. But testing the code is still done manually and ad-hoc. Not good. I have used some testing frameworks in other languages, but never in C++. As usual, Boost comes to the rescue. I recently added some test code using the Boost Unit Test Framework which (at least for now) does all I need.

Martijn van Exel is organizing a hack weekend on June 10-12 2011 at the Linuxhotel. I am planning to be there. After many hack weekends in London this ist the first such event in Essen after a long time. The first OSM event at the Linuxhotel was the Essen Developers Workshop I organized there four years ago. Has it already been that long?.

The organisers of the State of the Map Europe have published the programme and it looks very interesting. I will give a talk about Taginfo and a talk about Osmium, the two projects I currently spend my free time on.

Yesterday I gave a talk about Taginfo at the FOSSGIS2011 conference. The talk is in German, you can download a video (570 MB MP4). Many thanks to the volunteers who created the awesome videos of all the OSM talks at FOSSGIS!

It is quite amazing what can be done with good data and current map rendering technology. Mapnik and other renderers have continually pushed the envelope and there are many examples of beautiful maps out there.

The Unicode character set contains somewhat over one million code points from 0 to hex 10ffff. That wasn’t always so. Unicode started out with only 16 bit characters, or about 65000 code points. At some point it was decided that that wasn’t enough and the version 2.0 released in 1996 switched to the larger character set.

Taginfo always had an API so everybody can access all the juicy data in its database through simple HTTP queries. But that API was never used by anyone (as far as I can tell from a quick look at the log files). One reason was probably that I didn’t document the API.

In the past week two people contacted me independently who are working on Taginfo sites with local information for Ireland and Brazil. This prompted me to work on Taginfo again. Taginfo was never intended to be run on more than one web site, but it makes sense for other people to use it, too. So I incorporated some stuff that makes it easier to do that. Taginfo now has a new icon in the top left corner that can be changed for each instance, when you click on it you get some local information. Taginfo sites can add their address to the wiki.

I have wanted to play around with NodeJS for a while. I had some time on the weekend so I implemented a small tileserver for Tirex using NodeJS.

Even hotels that are proud of being different are similar in a way. Ideally there might be different experiences to be had, but somehow they are all standard. At least in the bathroom.

Some month ago I introduced Taginfo and mentioned there in passing that the statistics collection is done based on a C++ framework called Osmium. I promised another post and I am delivering on this promise now. I have been working on Osmium on and off over the last 6 months or so and, while it is far from perfect, I think it could be actually useful for other people now. So lets dive right in. What is Osmium and what is it good for?

The OpenStreetMap database currently contains nearly 900 Million nodes. Because some nodes have already been deleted the highest node ID is already past 1 Billion. IDs in the database are already 64 bit wide so that space will not run out. But lots of software only uses 32 bit IDs. There recently was a question on the German mailing list when the 32 bit would run out, so I want to give some answers here.

Yesterday Frederik Ramm send me a link to this wikipedia page. It contains 73 observations on (mostly obnoxious) behaviour in the Wikipedia and some recommendations how to deal with it. Almost all of that can be translated directly to OpenStreetMap and I recommend that every OSMer read it.

On the weekend there was (again) a discussion on the German OpenStreetMap mailinglist about how the system of coming up with new OSM tags and better defining existing tags could be improved (or whether it actually needs improving). This is a difficult subject. The “system” (for lack of a better word) is confusing and often leads to frustration. But it does basically work. OSM is growing, people are entering more and different kinds of data, usable maps are produced, finding routes based on OSM data works.

One of the things I like to do when on holiday is spending huge amounts of time and money in book shops and then more time reading the books. And inevitably the time runs out faster than the books, so my stack of unread books keeps growing. I recently added a new bookshelf conventiently located near my bed just for the unread books. There are just too many interesting things and interesting books about those things out there…

The OpenStreetMap project gets bigger every day. More people edit the data which leads to more data in the database and this to more people using the data. And with the growing database it gets harder and harder to see whats in there. Gone are the days when the common mapper would need to know maybe a dozen different tag keys, each with just a handful of typical values. Today people tag parking spaces and public transport, power lines and penguin pools in their local zoo. Some people are valiantly trying to keep up by documenting tags in the wiki, but sometimes that seems to be a lost cause. There are only about 550 wiki pages describing a key, but there are about 20,000 different keys in the database (about one third only used once, many will by typos) not to mention the 50 Million different tags. What are all these tags? How are they used and by whom? The endless discussions on the mailing lists about what tags to use are partly because the world is complex and fitting everything in our tagging model is not easy, but they are also in part a symptom of this confusing situation.