Out of memory

I have to import a lot of OSM planet files to test things due to my work on osm2pgsql. I have a test machine for that graciously provided by FOSSGIS e.V., the OSM Foundation local chapter for Germany. A few days ago I imported a planet file and osm2pgsql ran out of memory.

I was doing a non-slim import, which means that all node locations and all way nodes are stored in memory. Until recently this worked okay, but it seems the planet has grown beyond what fits into the 128 GB RAM that this machine has. This does not come as a big surprise, it was going to happen sooner or later — I wrote about this issue in my 2022 paper on the “Evolution of the OSM Data Model” (PDF).

I don’t have a bigger machine and I am not going to get one anytime soon. And other people are probably in the same position, so I added support for the -F/--flat-nodes option in non-slim mode to osm2pgsql. This means we can move nearly 100 GB worth of data into a file on disk instead of having it in memory. The PR was just merged. (You still need lots of memory, though, because without enough memory for caching, processing will become painfully slow anway.)