Osmium ToDo list

The Osmium framework has been around for about a year and a half now and has grown to include a lot of functionality. But there is still much to to. In this blog post, I have collected some of those things. There is no particular order to the following list.

Refactoring of multipolygon support: From quite early on Osmium has been able to assemble multipolygons from nodes, ways, and relations. This code came from a different Geofabrik internal application. It never fit well into the rest of Osmium and is in need of a refactoring. The code should be split up into easier to understand components and it should be more configurable. Policy decisions (like what to do with broken multipolygon geometries and which tags from the multipolygon relation and its member ways should end up in the multipolygon) should be defined in an extra policy class or something like it to make it more flexible. But before this can be tackled, we need some kind of test suite with lots of different valid and invalid multipolygons that can automatically test the code.

Rethink the Javascript integration: Osmium comes with support for the V8 Javascript engine. This allows for some great flexibility, but it also messes up the C++ code with lots of #ifdef sections and boilerplate code. The way the integration is done, everything thats needed on both the C++ and Javascript sides needs some special glue code. Even though I have written special C++ templates etc. to help with this, the integration is still too ugly and cumbersome. And it takes too much work to bring all the Osmium functionality to the Javascript side. One possible solution would be to integrate Osmium with Node.js which already has lots of helper function etc. to make this integration easier. As an added benefit we would get access to all the Node.js modules which help with everything from database access to web interaction. But I don’t know enough about Node.js to decide whether that really is the right way to go and how to do it. I could use the help of somebody more familiar with Node.js.

Integration with other languages: Currently the Javascript support is baked into Osmium, but no thought has been spent on other (scripting) languages. It might be nice to use Osmium from Ruby, Python, etc. I decided on the Javascript integration because of the performance of the V8 Javascript engine. But there are still lots of applications that could benefit from the integration with other languages.

Include ordering: In the early days when Osmium was small, the way to use it in your C++ program was to include one header file which would in turn include all the other Osmium header files. So with one include you had access to the full functionality. This turned out not to be so clever. Osmium is much bigger now and most programs will only need a subset of the functionality. Some parts might not even compile on your system, but that shouldn’t stop you from using other parts of Osmium. So at some point I decided to change this. Ideally you include the header files for exactly those parts that you need. Those header files would of course in turn include other parts of Osmium if they depend on them. This way compilation time stays down and you have no superfluous code in your application. I started to convert the include files to this schema, but I am not done yet. So currently we have a mix that needs cleaning up. This will probably also lead to some changes in the build process and of course all of this needs to be documented properly.

Multithreading: Osmium is currently singlethreaded. There are a lot of speed gains possible if we would use multithreading. Reading and writing OSM files for instance is relatively expensive and could be done in different threads from the main program. Somebody send me a patch once to make the PBF reader multithreaded based on OpenMP. I have not followed up on that because I am not sure it is the right way to go. There are different thread libraries around and Osmium should still work if somebody uses it in some other library. And the new C++11 standard published recently finally brings a standard, platform-independent threading model. Maybe that is the way to go? I don’t have much experience with C++ multithreading, so some input from others would be helpful here.

Convenience functions: Osmium is a low-level library. It gives you a lot of building blocks, but you have to assemble them yourself in your own C++ code. This gives you a lot of flexibility and thats good. But it also makes some tasks that should be simple more difficult than they could be. There could be convenience functions for instance that help you set up applications that read OSM files several times to do some complex task. I need some input from Osmium users on what functions are needed here.

Osmium application: Osmium is a library. It comes with some example programs that demonstrate how to use it, but they are (intentionally) simple and are not intended for normal use. There is no command line (or GUI) application that you can “just use” in your day-to-day work to, say, convert OSM files into different formats. Some of the tasks that Osmium can do are well covered by other programs like Osmosis, so maybe the need is not really there. But maybe it would be nice to have an Osmium application (or several of them), that would bring the Osmium features to the command line. This could be a different project just building on top of Osmium or it could be more integrated with the rest of Osmium.

Versioning: So far there is no versioning scheme for Osmium. Osmium users just check out the current version from Github and use it. But at some point we have to have versions and releases and git tags and packages etc. to make Osmium easier to use.

Changeset support: Nobody has asked for that yet, but at some point I would like to have support for changesets. They are a part of the OSM database and Osmium should be able to work with them.

String handling: OSM data uses huge amounts of small strings, mostly for tag keys and values, but also for relation member roles and user names. Using char * is cumbersome, but using std::string seems also suboptimal: An OSM planet file currently contains about 1.5 billion objects and about 700 million tags. Thats already about three billion memory allocations of (mostly) small strings for the user name, tag key, and tag value. It is worse because this data is copied around several times when parsing and writing OSM files etc. And often only a very small percentage of the data is actually used in the program. I am not exactly sure whether something can be done here, but I do see some room for improvements.

Exception handling: The exception handling in Osmium is a bit haphazard. Some parts of Osmium throw standard exception such as std::runtime_error, others have their own exception classes. And it is not always clear what exceptions from lower level libraries are passed on. The documentation is, aeh, patchy.

OSM file parsers: The parsers for OSM XML and PBF files work as push-parsers. You call them once with an open file and they will call back into your code with every object they find. Thats very easy to use and ok for most applications, but you can’t read several files at once, for instance to compare them. I would be nice to have pull-versions of the parsers, too.

Unit tests: When I started with Osmium I didn’t know how to do unit tests in C++ and when I had figured out how to use the Boost unit test framework I had already written a lot of code without tests to go with them. And who likes to write tests anyway? So now there is quite a lot of code without tests. At some point this situation needs to improve to make Osmium truly reliable.

Of course there is a lot more that needs to be done. But those are some of the larger tasks that came to mind. I’d appreciate any help with working on any of them. I’ll probably get around to do some of them myself, but others might just not get done until somebody else does it or somebody pays me to work on them.