Category Archives: Engineering Working Group

A progress update on vector tiles from the Engineering Working Group

The OpenStreetMap Foundation’s Engineering Working Group has an update on the effort to create vector tiles for openstreetmap.org. Read on for why this work is important, what’s been done so far and how they are incorporating community feedback, and the technical details for those who want to know more.

The background

Currently, the openstreetmap.org website serves raster tiles, which are image tiles made up of pixels — think a downloaded image of part of a map. But the effort has begun to create vector tiles for the site, which will help improve how the map looks and how it works. You can read more about the background of the project here.

Vector tiles serve up maps as vectors: points, lines and polygons. They store geographic data (like what makes up OpenStreetMap) in a format that allows for dynamic styling and interactivity. For users, vector tiles will mean a new, modern-looking map style with seamless zoom on openstreetmap.org, the map can be updated more quickly when data changes, and it should perform better for users.

Looking further ahead, the most exciting part is what this vector tile project will make easy for volunteers and tile users: 3D maps, more efficient data mixing and matching and integration of other datasets, thematic styles, multilingual maps, different views for administrative boundaries, interactive points of interest, more accessible maps for vision-impaired users, and surely many other ideas that no one has come up with yet. You may recall that many of these have been long-term interests for people in the OSM community.

The plan

The goal of vector tiles project is to provide a vector tiles setup that can work for openstreetmap.org — that is, a worldwide, complex basemap site in heavy demand from users and services around the world, where the data underlying the map changes all the time.

Or to put it technically, to create a setup for a worldwide complex basemap under high load which requires minutely updates.

Paul Norman is leading the vector tiles project.

He is working on adding to his Tilekiln project which generates vector tiles from a PostgreSQL database (like OpenStreetMap’s), making use of the Shortbread schema, which is a data format for how to name layers & properties within a vector tile, and improving Themepark, which allows one to add OSM data to a Postgres database.

The work is split up into three steps:

1. First round of Tilekiln improvements and Shortbread Themepark improvements

2. Parallelism improvements

3. Shortbread publicly available in production

The first two steps are nearly done. Tilekiln now generates tiles in parallel, making it practical to generate tiles for the entire world. The next step is to start the deployment into OSMF hardware to prepare for production.

Technical details on step 1

For those interested in the technical details of what’s being worked on, there are five main components of the first step above.

1. Automated packaging of Tilekiln

2. Tilekiln metrics being published with a Prometheus exporter

3. Themepark Shortbread reviewed

4. A demo server running with minutely updates of Shortbread tiles, rendering tiles on-demand

5. Demo shown to community

Items 1 and 2 are complete without need of further discussion. For item 3, Paul found that the osm2pgsql Themepark Shortbread implementation needed more work than anticipated as it was missing a layer and had some issues.

Item 4 and 5 are complete. Paul’s demo server is running with minutely updates and the hardware requirements are more modest than expected.

The community has also been providing useful feedback, such as on Paul’s OSM Community Forum post.

The community offered a lot of suggestions, some of which have already been incorporated. The remaining, in-scope issues from the community are: Curved lines rendering as jagged and vector tiles being too large.

The jagged lines issue is due to how smooth curves are represented in vector tiles. It has mostly been addressed but similar issues are expected to crop up in the future. A target scale equivalent to the minimum scale of the standard tile layer has been set. Zooming in to an even lower scale is possible, but artifacts may start to appear.

Vector tile size will continue to be an issue that needs continual work, but the current tiles are particularly large. Since this part of the testing some changes have been made that cut the size in half. Tile size optimization will be an issue that needs ongoing work, as tile size is the biggest factor in user experience.

The tiles being produced are usable, but more work remains to be done. Now that the parallelism work is complete it’s possible to generate large sets of tiles in order to test, so Paul will be returning to working on the tile definitions to improve tile size and fix some remaining issues, but the current tiles are usable.

Background on the tools being used

Here is some information on the various tools used for this project.

Tilekiln is software written by Paul Norman for generating vector tiles from a PostgreSQL database. Alternatives are martin (or maybe t_rex). Tilekiln is in new development, although it uses a lot of standard PostgreSQL features to generate the vector tile data. Most OSM based maps (incl. osm-carto on osm.org) are generated from SQL database queries from a PostgreSQL database. Tilekiln generates vector tiles from similar queries. Tilekiln is new.

Themepark is part of the osm2pgsql suite of tools, to allow one to add OSM data to postgres, and share those processing steps between other projects. Many PostgreSQL based OSM map styles (like osm-carto) use osm2pgsql

osm2pgsql has been around for 15+ years in OSM, and is used in many many places. Although Paul has contributed code to it, he is not the main developer. osm2pgsql has gotten more advanced, and better, in the last few years. Part of the power is pre-processing the data, and Themepark is an attempt to make these pre-processing steps easier.

Shortbread is a “vector tile schema” created by Geofabrik. It’s a data format for how to name layers & properties within a vector tile.

This blog posts contains contributions from Adam Hoyle, Mikel Maron, Amanda McCann, Paul Norman, and Andrew Wiseman

The OpenStreetMap Foundation is a not-for-profit organisation, formed to support the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data for anyone to use and share. The OpenStreetMap Foundation owns and maintains the infrastructure of the OpenStreetMap project, is financially supported by membership fees and donations, and organises the annual, international State of the Map conference. Our volunteer Working Groups and small core staff work to support the OpenStreetMap project. Join the OpenStreetMap Foundation for just £15 a year or for free if you are an active OpenStreetMap contributor.

2024: announcing the year of the OpenStreetMap vector maps

OpenStreetMap will take a large leap forward with the introduction of vector tiles on openstreetmap.org this year. This is the first of a series of blog posts where we will share our progress.

To lead our vector tiles project, the OpenStreetMap Foundation has hired Paul Norman, a renowned figure in cartography and open data, whose journey with OpenStreetMap began in 2010 with a chance encounter on the xkcd forums. His role in the community took off with his work on OpenStreetMap Carto in 2013. His volunteer involvement with the OSM Foundation, including contributions to several working groups and a tenure on the OSMF board, highlights his commitment to the project. Professionally, he has held various influential positions at MapQuest, CartoDB, Wikimedia Foundation, and Amazon. Billions have seen the products of his work. To read more from Paul, visit his blog for technical deep dives into vector tiles, follow him on Mastodon or on Twitter.

Vector tiles represent a significant advancement in how map data is processed and presented. Unlike traditional raster tiles, which are static images with pixels, vector tiles are like the ‘SVGs’ of the mapping world: you get lines and points. This stores geodata in a format that allows for dynamic styling and interactivity, enabling the user to adapt the visual appearance of the map without altering the data. If that sounds like what you’ve seen on other maps, you are right! Vector tiles have become industry standard in interactive maps that, unlike openstreetmap.org, don’t get updated often, and where you can simply recalculate your whole database occasionally.

But the map displayed on openstreetmap.org are quite uniquely different! They get updated incrementally and constantly, a minute after you edit; it’s a critical part of the feedback loop to mappers – and how the author of this blog post got hooked in the first place. This is why we have to invest in our own vector tile software stack.

In the direct future, for users, this will mean a new, modern-looking map style with seamless zoom on openstreetmap.org. Looking further ahead, the most exciting part is what this vector tile project will make easy for volunteers and tile users: 3d maps, more efficient data mixing and matching and integration of other datasets, thematic styles, multilingual maps, different views for administrative boundaries, interactive points of interest, more accessible maps for vision-impaired users, and I’m sure many other ideas that no one has come up with yet. This technology is not just a leap in aesthetics, but also in functionality, enhancing the overall user experience.

In the 2021 community survey (page 15), there was no clear sentiment on what the foundation should do on vector tiles. We noticed a split in preferences: some advocated for volunteer-led development, others for professional engagement. The ecosystem has evolved since then, making it easier to build on top of existing software bricks. We see our project as a reasonable balance between the two most popular answers. Investing in core software is also part of our multi-year strategic plan.

The OpenStreetMap Foundation depends on donations to finish this project. If you would like to support our year of vector tiles specifically, you can donate and leave ‘vector tiles’ in the donation message. Every contribution, large or small, directly supports our ability to ensure that OpenStreetMap can be open, accessible and dynamic for all. Your support is not just a donation; it’s an investment in the future of open-source mapping.

We’re just at the beginning of this exciting journey. Stay tuned as we will delve deeper into the schema and style aspects in future blog posts.

EWG Project Deadline: “Adding the Ability to Mute Users”

The Engineering Working Group (EWG) would like to announce the deadline for the following project: Adding the ability to mute users on the openstreetmap.org website.

Project Deadline

The deadline for submitting a proposal will be March 13, 2023. After the submission deadline the EWG will resolve to award the bid within 2 weeks.

About the Project

Users who receive unwanted messages to their openstreetmap.org message inbox currently have to report the message writer and wait for an administrator to take action. This feature will make it possible for anyone to painlessly mute (ignore) private messages from another user.

For more details about the project, including how to apply and proposal requirements, please see the the Engineering Working Group Project Funding Repository on Github. Click on “Ability to mute other users” under the Projects section or visit this link for a list of deliverables.

Understanding the Project Funding Process

Before submitting a proposal make sure to also read the Engineering Working Group’s Project Funding Proposal Framework for a general overview of the process. Should you have any questions about the funding process please reach out to the Engineering Working group at engineering@osmfoundation.org.

About the Engineering Working Group

The Engineering Working Group is charged with, among other things, handling software development paid for by the OSMF, putting out calls for proposals on tasks of interests, offering a platform for coordination of software development efforts across the OSM ecosystem, and managing OSM’s participation in software mentorship programs.

The Engineering Working Group meets once every two weeks. Meetings are open to all and all are welcome. Questions? Please send an email to engineering@osmfoundation.org. We are a small group and are still welcoming new members!

Reminder: Call for Feedback on the Data Model

Data Model Study

The Engineering Working Group of the OpenStreetMap Foundation commissioned a study in the beginning of 2022 on how to improve the existing data model. Jochen Topf has delivered the results of this study, including recommendations on how to make the OpenStreetMap data model more computationally efficient and more accessible.

Two key suggestions have been made:

introducing an area datatype for representing polygons
getting rid of untagged nodes

Community Consultation

In order to decide the next steps in this process we want to have more discussions with the community of developers as the proposed changes impact OpenStreetMap software which directly or indirectly depends on the data model.

Join the Discussion

Potential benefits

Less Mess for Areas

Some mappers may be surprised to hear that OSM does not already have an Area data type. After all, the iD editor prominently features buttons for drawing points, lines and areas. Once mapped, these areas usually appear on the map as expected. The OSM wiki documents whether a tag is typically used on areas, and even Overpass Turbo lets you use areas in your query.

Behind the scenes, however, these areas are represented as ways or relations. Each tool working with OSM data uses its own set of rules to guess whether a particular way represents a line or an area. Making areas a proper part of the OSM data model would lead to a consistent interpretation across applications, enable the API to prevent broken areas from being uploaded, and may eventually lead to support for partial downloads of very large areas.

Keeping OSM Processing Accessible

Currently, ways are made up of references to nodes, and we rely on these references to determine how ways connect to each other. Resolving the coordinates to these node references is a costly process within the OpenStreetMap toolchain as it takes hours to days, even on capable hardware.

In the future, we might model ways as a simple list of coordinates – depending on the exact implementation we end up with. This would offer large performance benefits, but getting rid of untagged nodes would be a significant change.

At first glance, performance improvements may not seem particularly exciting. But how easy it is to work with our data directly impacts how useful OpenStreetMap is to the world at large. As Jochen observes: “The goal is to keep OSM as that great resource that can be used not only by multi-billion-dollar companies but by the student who wants to create a map of the world on their notebook or the activist with their donated second-hand computer.”

Better OSM History

Many mappers are disappointed when they realise how few things the history tab of the website can actually show. There are many tools, like OSMCha and Achavi, that offer much more, but still require a certain degree of proficiency to use them.

You might ask why, and the answer is very technical – the location of a single version of a way is, in many cases, not defined. It is the reason that change tracking remained an expert discipline with relatively newbie-unfriendly tools. By changing the data model we will move away from that barrier, and subsequently we can expect substantially better tools, but not before we get proper coordinates and versions for ways.

Minutely Vector Tiles Generation

While there are quite a number of matured vector tile generators nowadays, a couple of problems are still open.

One is which features shall go into the vector tiles for openstreetmap.org
The other is how to reconcile minutely updates with vector tiles for performance at an acceptable level.

That task gets an order of magnitude easier if you can not only truly parallelise the generation of tiles, but also elide the first expensive step to figure out to which tile a changed way belongs.

We might be able to find someone who encapsulates the raw computing power necessary to do this. But even if so, this is a highly nondesirable degree of dependence on that partner.

So yes, vector tiles for openstreetmap.org are in principle possible without this data model change, but at a so much higher cost that only specialized hardware will be able to keep up with minutely changes.

Have Your Say about the Future

Some kind of change is inevitable. The growth of the OSM database is outpacing speed improvements in hardware, and the ID-based model means that the whole process cannot be parallelized with full speedup. Keeping up with changes was easily possible in the past, but needs needs more and more tricks now. There is a point in the future where also specialized hardware will suffice to keep up with minutely changes.

However, there are many possible approaches to meeting this challenge. Now is the opportunity for the developer community to share your opinion about the way forward.

Engineering Working Group Call for Proposals

The Engineering Working Group (EWG) would like to announce a call for proposals for the following project: Adding the ability to mute users on the openstreetmap.org website.

About the project

For more details about the project, including how to apply and proposal requiements, please see the Engineering Working Group Project Funding Repository on Github. Click on “Ability to mute other users” under the Projects section or visit this link for for a list of deliverables.

Understanding the project funding process

About the Engineering Working Group

The Engineering Working group is charged with, among other things, handling software development paid for by the OSMF, putting out calls for proposals on tasks of interests, offering a platform for coordination of software development efforts across the OSM ecosystem, and managing OSM’s participation in software mentorship programs.

The Engineering Working Group meets once every 2 weeks. Meetings are open to all and all are welcome. Questions? Please send an email to engineering@osmfoundation.org. We are a small group and are still welcoming new members!

About OpenStreetMap