Osmium news

Its high time I write something about the news in the Osmium world. After 10 days or so of multiple releases in short succession things have quieted down some and I think we have something more stable now.

Osmium isn’t just one piece of software and creating releases becomes increasingly difficult. From the bottom up (and only looking at the packages I am maintaining), there is first protozero, the low-level Protocol Buffers library which is used in libosmium, then libosmium itself, for which there are the NodeJS bindings node-osmium and the Python bindings pyosmium. Everything so far is just the building blocks needed for the real applications. Real applications are foremost the Osmium command line tool and then there is Osmcoastline and several others.

Most of this has to build on Linux, OSX, and Windows with several OS and compiler versions. Travis and Appveyor are a tremendous help with that but occasionally they have their own problems: Sometimes builds are slow or fail entirely for unknown reasons. If something fails, I have to investigate, fix build scripts or update dependencies.

On Linux, Debian packages are built by the ever helpful Debian maintainer, Bas Couwenberg, who most of the time updates the packages the day I release a new version. The Debian builds (for Stretch and Sid) as well as backports for Jessie are not only useful for users, they are also useful for me as a developer, because Debian runs automated builds and tests on many different architectures. Besides i686, there is i386, several versions of ARM, MIPS, and PowerPC builds and many more (see, for instance, the complete list for libosmium). Building on all these architectures often exposes sloppyness in the code and bugs in the low-level functions. This is especially important for protozero and libosmium, because both tend to fiddle around with bits and bytes to get the best performance. The problem with the Debian automated build is that it gets only triggered when a new version is packaged. A new version is only packaged if I release a new version. I can’t do a check before releasing a new version, which leads to lots of patch releases if there are problems. Which is what happened in the last few days. Not good. I do have an old i386 32bit machine around, which I occasionally try things on, but a complete build of everything takes hours. Tom Hughes has been building Fedora packages, maybe the Fedora infrastructure can be used to help out here.

Once the low-level library problems are all done, we move up to the NodeJS and Python bindings. They need to be built and released to their respective repositories, too. For node-osmium an NPM package has to be released and for Python a PyPI package (this btw is new, you can now easily install PyOsmium with the pip command line tool like any other Python package). Did I mention this all has to work with several NodeJS versions and Python2 and Python3?

And then there are the applications, but they create less problems usually. The Osmium command line tool is tied rather closely to libosmium. New features in the tool often need new features in the library, so I release new version together. Many other libosmium-based programs work fine with older versions so I only do releases as necessary. The Osmium tool and OSMCoastline are both packaged in Debian, so I have to keep that in mind.

And then there is the documentation. Some parts of it live in their own repository (Update 2019-02-21: This is not true any more) and need to be updated. And for libosmium, there is the API documentation generated using Doxygen and for PyOsmium the API documentation generated using Sphinx. Both need to be generated and everything has to be pushed to the web server.

All of this is only possible because there are many services I can use, mainly GitHub, Travis, Appveyor and the Debian and Fedora infrastructures. And a lot of this is scripted working more or less automatically. But there is still plenty to keep in mind and to do right and in the proper order, and a change log doesn’t write itself automatically either, so making new releases for the whole stack is still something that can take several days.

And all of this to deliver a new version of libosmium with fewer bugs and, again, a bit faster I/O. And it is easier to use thanks to more and better documented examples and clever checks that detect more programmer errors when Builders are used.