OSM Spyglass

Two years ago or so I started the OSM XRAY project, later I wrote about it in this blog post. Since then I have renamed this project to “OSM Spyglass” and I have kept working on it on and off.

At the State of the Map Europe 2025 in Dundee I gave a talk with the title “Everything Everywhere All At Once” about this project. You can see the video on Youtube. This got some people excited about the project, there is even some talk about putting the tool on OSMF infrastructure. Until this comes about the tool is now hosted at spyglass.jochentopf.com.

I am finally getting around to writing some more about what’s been happening since my first announcement and since the talk.

User Interface

I keep fiddling with the user interface. Optional globe view (not much to do for me now that Maplibre supports that out of the box), map is now resizable (horizontally), display of city names in some zoom levels, improved pop-up menus for keys and tags, and much more. Generally the UI has been getting faster and more reliable.

There are still some bugs to fix and plenty of possible improvements. And I’d be happy about feedback and ideas. Its quite a lot of information we are trying to show here in limited space, so good ideas on how to do that are needed.

Caching

In the first blog post I wrote about some caching that I implemented in the database. That did work but it turns out it is pretty useless. The user wants to access the newest data anway and we can keep up with minutely updates (at least in larger zoom levels), so I removed the caching completely for vector tiles and for high zoom rasters. Only raster images at zoom levels up to 10 are cached. Currently we can not deliver them fast enough otherwise.

Map updates

The database is updated from OSM using minutely diffs. We are usually about 3 to 5 minutes behind the OSM data, that’s just how long it takes the OSM servers to create the minutely diffs, push them out to their server and for our update job to download the data and to apply it to the database. It is unlikely we can improve on that much further. Spyglass shows the timestamp of the latest data it has in the bottom right corner. This timestamp is updated whenever new data is loaded, i.e. when you move the map or so.

Vector tiles are always generated on the fly from the current database, for higher zoom levels they contain all data, for medium zoom levels only “larger” objects are shown, i.e. long ways and larger areas. In small and medium zoom levels raster tiles are shown. They always contain all data. So for the medium zoom levels raster data in gray is overlayed with vector data in black (nodes and ways) or blue (relations). So you can see everything, but only click on the larger items.

Raster tiles in small zoom levels are only updated once per day, for zoom 0 to 7 this happens by taking the zoom level 8 tiles, and merging and rescaling them. I have spent quite some time on optimizing this. The first version happened in the database but only generated black-and-white tiles, the current version uses code written in Go which creates grayscale images which are much better than the black-and-white images. And it is much faster than the gdal tools I tried for this task. Gdal is a great tool, but, as an “all purpose tool”, it has to cope with all sorts of different data sources, projections etc. which makes it much slower than a specialized tool for a specific use case. It only takes a few minutes now to create the low zoom tiles from the zoom level 8 tiles. And they are not stored in the database any more but on disk which is easier and they are faster to use that way, too.

Rasters are still generated in the database from the data. That is, unfortunately, not as efficient as one might think. We don’t need to copy the data from the database into another process, and the cost of actually getting the data seems to be not that huge, but the rasterizing costs time. This is probably something that could be improved inside PostGIS, or maybe we have to get rid of this idea alltogether and move rendering outside the database. There is plenty of space to experiment and improve performance here.

Server

Originally I used pg_tileserv as server to create the vector tiles from the database on the fly. It could also be tricked into creating the raster tiles. But I also needed GeoJSON output and some other API endpoints. I experimented with pg_featureserv which did work, but having two servers with lots of specialized PL/pgSQL functions in the database plus an ever growing configuration for nginx (used as reverse proxy) became too complicated and error prone. So I decided to rewrite the server from scratch in Go. Turns out it is really easy to write robust and featureful HTTP servers in Go, it comes with everything you need; the only external library I am using is for accessing the database. And deployment is really easy: Just copy over one Go binary and restart the server, no extra configuration files or functions to update in the database etc.

Filters

Everything is done three times for nodes, ways, and relations. There are three sets of raster tiles, 3 sets of vector tiles. It is easy to switch those layers on and off in the UI. And then there is the key or tag filter. The vector tiles in higher zoom levels contain all the data, the filter is applied on the client, which is very fast. For raster tiles the filtering has to be done on the server which takes somewhat more time. Filtering is (silently) disabled on the small zoom levels, so you always see all data there. This isn’t great as a user experience, I’ll still have to figure out a way to make this transition more user friendly. Or, ideally, allow filtering on all zoom levels.

It is a lot of fun to zip around the map and look at far away places and how they are mapped. Try it out!. And if you have any problems or ideas, open an issue on Codeberg.