Using the MapQuest Render Stack for Multilingual Maps

After setting up the MapQuest Render Stack and poking around in its code I consider using it for the Multilingual Maps project. Unlike Tirex it can run on a cluster of machines for reliability and load distribution. That will be important for a tile server working on “Wikipedia scale”.

At some point I thought I might have to implement a completely new tile server, so I am happy that that’s not necessary. Still, there are several changes needed to the MapQuest Render Stack to make it usable in this context:

Map parameters: First, we need to get the language choice from the URL through the web server and the complete system to the renderer. Better, we need some generic feature that allows any kind of “map parameter” to be propagated through the system and taken into account in the renderer. So this needs changes in the URL parser in the tile_handler and changes to the protocol between the tile_handler, tile_broker, and worker. It also needs changes in the storage component, because different parameters lead to different tiles which must be stored.

Language choice: At some point we have to take the list of languages the user chose and decide which OSM tags to use for the rendering. This code should be encapsulated nicely and easy to change, maybe even in some kind of scripting language, because it will likely need special cases for many languages or language combinations, scripts, etc.

Temporary storage: If rendering the labels on the demand is fast enough, we still need to store them temporarily, so that they are only rendered once for each metatile. I could imagine a storage plugin for the render stack that uses memcached for that.

Render priority: We will also need some way of prioritizing requests for the labels layer. Otherwise long-running render jobs for the background layer can starve out the on-demand-rendering of the labels.

Distributed storage: There is an implementation for a distributed tile storage based on the Python Pylons project in the MapQuest Render Stack source. I have not tried it and am not sure about its capabilities and what changes might be necessary. But it seems that it does work for MapQuest, so why shouldn’t it work for Wikipedia? I could imagine a distributed tile storage based on Cassandra or some other NoSQL database, too. In time this has to be explored further.

Mapnik changes: We might need some small changes to Mapnik to change the datasource parameters as discussed here. Details need to be hashed out, but there is no big problem here and the solution should be generic enough that it would end up in the main Mapnik source.

In contrast to the original plan we do not need a special language rendering hint database. So that part has become considerably simpler. The changes to the render stack should be possible with a manageable amount of work.

The big question still is whether rendering of the labels on demand is fast enough. We will probably need a complete setup of the whole system on large machines to see whether this works in practice. Testing this on a system that’s not under a realistic load will probably not give us reliable numbers. But I am confident that we can make this work in some way. Maybe we have to distribute the PostgreSQL database over several machines (which we might want to do anyway). Or we could permanently store the rendered label tiles for the languages requested most often, but keep the ability to render other languages on demand. That would give us both fast tile delivery and complete flexibility. Of course this would need some further changes to the render stack…

So thats an outline of a plan here. Now I hope for some feedback of what will probably work and what not, what’s stupid and what’s clever and what other ideas are out there.

Comments can be directed to the Multilingual maps wiki page.