Category Archives: osm.org

A progress update on vector tiles from the Engineering Working Group

The OpenStreetMap Foundation’s Engineering Working Group has an update on the effort to create vector tiles for openstreetmap.org. Read on for why this work is important, what’s been done so far and how they are incorporating community feedback, and the technical details for those who want to know more.

The background

Currently, the openstreetmap.org website serves raster tiles, which are image tiles made up of pixels — think a downloaded image of part of a map. But the effort has begun to create vector tiles for the site, which will help improve how the map looks and how it works. You can read more about the background of the project here.

Vector tiles serve up maps as vectors: points, lines and polygons. They store geographic data (like what makes up OpenStreetMap) in a format that allows for dynamic styling and interactivity. For users, vector tiles will mean a new, modern-looking map style with seamless zoom on openstreetmap.org, the map can be updated more quickly when data changes, and it should perform better for users.

Looking further ahead, the most exciting part is what this vector tile project will make easy for volunteers and tile users: 3D maps, more efficient data mixing and matching and integration of other datasets, thematic styles, multilingual maps, different views for administrative boundaries, interactive points of interest, more accessible maps for vision-impaired users, and surely many other ideas that no one has come up with yet. You may recall that many of these have been long-term interests for people in the OSM community.

The plan

The goal of vector tiles project is to provide a vector tiles setup that can work for openstreetmap.org — that is, a worldwide, complex basemap site in heavy demand from users and services around the world, where the data underlying the map changes all the time.

Or to put it technically, to create a setup for a worldwide complex basemap under high load which requires minutely updates.

Paul Norman is leading the vector tiles project.

He is working on adding to his Tilekiln project which generates vector tiles from a PostgreSQL database (like OpenStreetMap’s), making use of the Shortbread schema, which is a data format for how to name layers & properties within a vector tile, and improving Themepark, which allows one to add OSM data to a Postgres database.

The work is split up into three steps:

1. First round of Tilekiln improvements and Shortbread Themepark improvements

2. Parallelism improvements

3. Shortbread publicly available in production

The first two steps are nearly done. Tilekiln now generates tiles in parallel, making it practical to generate tiles for the entire world. The next step is to start the deployment into OSMF hardware to prepare for production.

Technical details on step 1

For those interested in the technical details of what’s being worked on, there are five main components of the first step above.

1. Automated packaging of Tilekiln

2. Tilekiln metrics being published with a Prometheus exporter

3. Themepark Shortbread reviewed

4. A demo server running with minutely updates of Shortbread tiles, rendering tiles on-demand

5. Demo shown to community

Items 1 and 2 are complete without need of further discussion. For item 3, Paul found that the osm2pgsql Themepark Shortbread implementation needed more work than anticipated as it was missing a layer and had some issues.

Item 4 and 5 are complete. Paul’s demo server is running with minutely updates and the hardware requirements are more modest than expected.

The community has also been providing useful feedback, such as on Paul’s OSM Community Forum post.

The community offered a lot of suggestions, some of which have already been incorporated. The remaining, in-scope issues from the community are: Curved lines rendering as jagged and vector tiles being too large.

The jagged lines issue is due to how smooth curves are represented in vector tiles. It has mostly been addressed but similar issues are expected to crop up in the future. A target scale equivalent to the minimum scale of the standard tile layer has been set. Zooming in to an even lower scale is possible, but artifacts may start to appear.

Vector tile size will continue to be an issue that needs continual work, but the current tiles are particularly large. Since this part of the testing some changes have been made that cut the size in half. Tile size optimization will be an issue that needs ongoing work, as tile size is the biggest factor in user experience.

The tiles being produced are usable, but more work remains to be done. Now that the parallelism work is complete it’s possible to generate large sets of tiles in order to test, so Paul will be returning to working on the tile definitions to improve tile size and fix some remaining issues, but the current tiles are usable.

Background on the tools being used

Here is some information on the various tools used for this project.

Tilekiln is software written by Paul Norman for generating vector tiles from a PostgreSQL database. Alternatives are martin (or maybe t_rex). Tilekiln is in new development, although it uses a lot of standard PostgreSQL features to generate the vector tile data. Most OSM based maps (incl. osm-carto on osm.org) are generated from SQL database queries from a PostgreSQL database. Tilekiln generates vector tiles from similar queries. Tilekiln is new.

Themepark is part of the osm2pgsql suite of tools, to allow one to add OSM data to postgres, and share those processing steps between other projects. Many PostgreSQL based OSM map styles (like osm-carto) use osm2pgsql

osm2pgsql has been around for 15+ years in OSM, and is used in many many places. Although Paul has contributed code to it, he is not the main developer. osm2pgsql has gotten more advanced, and better, in the last few years. Part of the power is pre-processing the data, and Themepark is an attempt to make these pre-processing steps easier.

Shortbread is a “vector tile schema” created by Geofabrik. It’s a data format for how to name layers & properties within a vector tile.

This blog posts contains contributions from Adam Hoyle, Mikel Maron, Amanda McCann, Paul Norman, and Andrew Wiseman

The OpenStreetMap Foundation is a not-for-profit organisation, formed to support the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data for anyone to use and share. The OpenStreetMap Foundation owns and maintains the infrastructure of the OpenStreetMap project, is financially supported by membership fees and donations, and organises the annual, international State of the Map conference. Our volunteer Working Groups and small core staff work to support the OpenStreetMap project. Join the OpenStreetMap Foundation for just £15 a year or for free if you are an active OpenStreetMap contributor.

OAuth 1.0a and HTTP Basic Auth shutdown on OpenStreetMap.org

In 2024, the OSMF Operations Working Group (OWG) is retiring OAuth 1.0a and HTTP Basic Auth on OpenStreetMap.org. These are technical ways for applications to authenticate users with the OSM website or API. OAuth 1.0a and HTTP Basic Auth have been deprecated since 2023, as OAuth 2.0 is now the standard authorization method for most systems.

There are three key dates in the transition process:

March 1st, 2024: New OAuth 1.0a application registrations were disabled. Existing applications were not impacted. HTTP Basic Auth was not impacted.
May 1st, 2024: System administrators will start brownouts to find applications that are still using OAuth 1.0a or HTTP Basic Auth.
June 1st, 2024: OAuth 1.0a and HTTP Basic Auth will be shut down.

Retiring these authentication methods is necessary because of security concerns, and the complexity of maintaining so many authorization implementations, including ones that rely on unmaintained components.

How does this impact me as a developer?

If you are a developer of an application using OAuth 1.0a or HTTP Basic Auth to log in to the OpenStreetMap.org website, you might need to make some changes to switch to OAuth 2.0. Fortunately, this is a well-supported industry standard.

If your application only makes read calls to the API, authorization is optional. For rate-limiting purposes, it is still a good idea to add authorization to your requests, but it is not required. If your application is a website using OSM for logins, making use of OAuth 2.0 is much easier as it is much better supported because so many other sites use it. It also avoids problems like users ending up with many tokens in their list on the website.

If you are developing software that edits using the API and is run locally, you may need to make code changes. All common languages have libraries that deal with OAuth 2, and libraries are the preferred choice for any authorization. You can also use Zverik’s library for command-line tools, or write your own shell script of about a dozen lines.

You should be able to find lots of examples online of OAuth 2 client implementations in your language. If you want to get more detailed information or ask technical questions, please use the GitHub ticket. Here, the OWG also tracks the applications requiring modification to use OAuth 2.0.

How does this impact me as a mapper?

Most mappers will notice no change. The transition will not affect how you log in to your OSM account or use the website. iD and JOSM have supported OAuth 2.0 as the default authentication method for some time. If you use your OSM account to log in to a third-party site like the HOT Tasking Manager, MapRoulette, or HDYC, you will not be impacted as those sites have already moved to OAuth 2.0. Read-only API access does not require authorization at all.

Tom Hughes Recognized for OpenStreetMap Contributions on OpenUK’s 2024 Honours List

OpenUK celebrates Tom Hughes as one of the top open source contributors on their New Year’s Honours List. It is the fourth time the UK organisation for open technology publishes this list to honour open source influencers across the United Kingdom.

Tom is an OSM system administrator and one of the maintainers of the openstreetmap.org code. He has been maintaining and improving the website project since around 2006. His inclusion in OpenUK’s 2024 Honours List is a well-deserved recognition for almost two decades of hard work as a volunteer.

Congratulations Tom, and thank you for your unwavering dedication to OpenStreetMap!

Want to learn more about how Tom maintains the OpenStreetMap website project, and find out how you can help? Check out the “Maintaining OpenStreetMap.org” talk his fellow maintainer Andy Allan gave at State of the Map Europe 2023.

Towards an improved data model for OpenStreetMap

We all know and love the OpenStreetMap data model with its nodes, ways, and relations and the open tagging that has allowed OpenStreetMap to be so innovative. But the data model also shows its years and some improvements might be possible. There is a lot we don’t want to change. Especially the open tagging model has proven itself. We might think of some small improvements, but the core idea of allowing any number of key-value (string) tags has worked amazingly well.

But there are some pain points due to the way we organize our data. The biggest problem is that geographic location is only available to nodes and not higher-level geographic objects like ways and relations. This means that accessing the location of, say, a way always means we have to follow the references to the member nodes of that way. This makes processing OSM data extremely cumbersome and resource-intensive.
The other major pain point often talked about is the missing “area” datatype. We use work-arounds like closed ways and multipolygon relations, but that has always been problematic, because we can’t be sure that those objects are actually valid polygons.

The OSMF Engineering Working Group (EWG) has commissioned me (Jochen Topf) to write a study over the next months outlining the problems with our current data model, possible improvements and their impact on our systems. Any changes to our data model will, of course, have a large impact on our mappers, the data users, our software and on the whole OSM ecosystem. So the study will also suggest ways to move forward implementing those changes step by step.

You are welcome to contact me via email at jochen@topf.org if you want to discuss any of this. After my talk at the State of the Map in Milano in 2018 in which I already outlined some of the issues with the data model, I created the osm-data-model repository. Feel free to comment there. After my preliminary study I will expect there will be a more formal discussion process where we can decide as a community which (if any) changes we want and how we are going to implement them.

Meet Grant Slater, the OpenStreetMap Foundation’s new Senior Site Reliability Engineer

Thanks to the support of corporate donors, the OpenStreetMap Foundation has been able to hire its first employee, who is starting on 1 May 2022. Grant Slater and Guillaume Rischard, the Foundation’s chairman, sat down for a virtual chat.

Hi! Tell us about you?

Hi! I’m Grant Slater, and I’m the new Senior Site Reliability Engineer (SRE) working for the OpenStreetMap Foundation. I’m originally from South Africa, and now live in London (UK) with my wife Ingrida and our son Richard.

What do you do in OSM? Where do you like to map?

I’ve been mapping since 2006, mostly in the Southern Africa and in the United Kingdom. I have a strong interest in mapping the rail network of South Africa; holidays “back home” often involve booking railway trips across the country, with a GPS in hand.

My latest toy is an RTK GPS base station and rover. I’ll soon be mapping my neighbourhood with centimetre-level accuracy.

For the last 15 years, I’ve been part of the volunteer OpenStreetMap Operations Team who install and maintain the servers and infrastructure which runs the OpenStreetMap.org website and many other related services.

What are your plans for the new SRE job?

My main objective will be helping improve the reliability and security of the project’s technology and infrastructure.

One of my goals will be to improve the project’s long-term stability as we grow. OWG can’t work without volunteers, and I will be improving the Operation Team’s bus factor by also improving our processes, documentation, and by smoothing the path to onboarding new team members.

I will be helping to drive forward modernising the project’s infrastructure by reducing complexity, paying-down technical debt, and reducing our need to maintain undifferentiated heavy lifting, by tactically using Cloud and SaaS services, where suitable.

Is there anything else you’d like to say?

With time, I would like to see OpenStreetMap introduce new tools and services to improve our mappers’ access to opted-in passively collected data to improve the mappers’ ability to map and detect change.

Gamification! OpenStreetMap should always remain a fun and gratifying experience for all. We’re building an invaluable and unique dataset with far-reaching consequences for which we should be incredibly proud. Happy Mapping!

I would like to hear your feedback and suggestions, please email me osmfuture@firefishy.com

–

Grant gave a talk at State of the Map US (2013) – OSM Core Architecture and DevOps and is hoping to give an updated talk at State of the Map 2022 in Florence, Italy 19–21 August 2022.

https://www.openstreetmap.org/user/Firefishy
https://twitter.com/firefishy1
https://github.com/firefishy

A new discussion forum for OSM

The OpenStreetMap Foundation is launching a new communication platform for the OpenStreetMap community at community.openstreetmap.org. Interested? Give it a try and find out how you can get involved with the next steps!

Screenshot of the new forum’s “feedback” category

Behind the scenes, the new space is powered by Discourse, a modern open source solution for online conversations. It is fully integrated with OSM accounts to let you log in with your existing user name. Discourse is easy to use and offers all the conveniences you would expect from social online spaces in the 2020s. At the same time, it gives power users access to optional advanced features: Write and receive posts through email, subscribe to RSS feeds, and more!

Of course, this is not the first platform dedicated to mappers’ conversations about their favourite passion. But while the new community space has been set up in parallel to the existing platforms provided by the Foundation (notably the forum, help site and mailing lists), we hope that the sub-communities on those platforms will eventually migrate to the new site. Likewise, we encourage communities currently using proprietary tools to consider adopting the new site for their communication needs. In particular, we are prepared to work with regional communities and local chapters – let us know on the Discourse forum what you need for your community to make this space your new home for discussing your local activities!

Choosing a solution that is hosted on our own infrastructure and based open-source software ties into the OpenStreetMap Foundation’s commitment to open communication channels. To ensure productive conversations and a welcoming environment, please respect our recently updated etiquette guidelines when you participate in the new forum. We hope that all members of the OSM community feel right at home at community.osm.org!

ÖPNVKarte, a new featured layer on www.openstreetmap.org

*ÖPNVKarte, a new featured layer on osm.org. Tiles* *courtesy of MeMoMa*ps.

OpenStreetMap.org now has a new featured map layer displaying our rich data in a different way: ÖPNVKarte.

ÖPNVKarte is a public transport map displaying public transport routes of trains, trams, light rails, buses, ferries, subways and also points of interest for public transport travelers. Melchior Moos, the creator of the layer, hopes that the inclusion on the global site “will help and reward mappers from all over the world by making their public transport mapping efforts more visible”.

The ÖPNVKarte map key can be found on the ÖPNVKarte website https://öpnvkarte.de (also OpenBusMap.org). For those wondering about the name, it’s derived from German for “Public Transport Map”.

ÖPNVKarte is already featured on sites such as: openstreetmap.de, openstreetmap.no, Mappa-Mercia, Facilmap , OSM inspector and on some mobile applications.

The ÖPNVKarte layer is an addition to these featured layers:

“Standard”(OpenStreetMap Carto), which is the default layer on osm.org,
“Cycle Map” (tiles courtesy of Andy Allan),
“Transport Map” (tiles courtesy of Andy Allan) and
“Humanitarian” (tile style by HOT OSM, hosted by OSM France)

The OSM Operations Working Group is interested in new featured layers that highlight different aspects of OpenStreetMap and you can read about the criteria here.

OSM Operations Working Group

Do you want to translate this and other blogposts in another language..? Please send an email to communication@osmfoundation.org with subject: Helping with translations in [language]

The Operations Working group is one of the volunteer Working Groups of the OpenStreetMap Foundation. The OpenStreetMap Foundation is a not-for-profit organisation, formed to support the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data for anyone to use and share. The OpenStreetMap Foundation owns and maintains the infrastructure of the OpenStreetMap project, is financially supported by membership fees and donations, and organises the annual, international State of the Map conference. It has no full-time employees and it is supporting the OpenStreetMap project through the work of our volunteer Working Groups. Please consider becoming a member of the Foundation.

OpenStreetMap was founded in 2004 and is a international project to create a free map of the world. To do so, we, thousands of volunteers, collect data about roads, railways, rivers, forests, buildings and a lot more worldwide. Our map data can be downloaded for free by everyone and used for any purpose – including commercial usage. It is possible to produce your own maps which highlight certain features, to calculate routes etc. OpenStreetMap is increasingly used when one needs maps which can be very quickly, or easily, updated.

New licence for the “standard style” tiles from openstreetmap.org

One of the important products of the OpenStreetMap community is the standard style map layer on openstreetmap.org. This has been licenced on CC BY-SA 2.0 terms probably since it was first created. At the time OpenStreetMap data was licensed on the same terms, however 8 years ago when we changed the data licence to the ODbL 1.0 we didn’t change the tile licence leaving it as it is.

Over the years this had led to a situation in which use of the “tiles” (the individual images that make up the displayed map) has been subject to more legal restrictions than necessary and has inhibited use of the images in many projects which would have been completely in order otherwise. The absurdity of the situation may be more clear if you consider that you can take the CC0 “licenced” map style, OSM data and produce the same images only being restricted by the terms of the ODbL for “Produced Works”.

It has been clear for many years that the situation was untenable, creating friction with third parties (for example PLOS One) for no good reason and that we should move to an attribution only licence. However the main question the Licensing Working Group (LWG) hadn’t answered was, “which licence should the tiles have in the future?”. Given that licences are a dime a dozen, you would assume this to not be an issue, however outside of licences in use by government entities (that is the OGL and derivatives), there is no popular and well known “attribution-only” licence in use and we wanted to avoid creating our own if at all possible.

In 2019 we presented the OpenStreetMap Foundation board with a proposal to switch to CC BY 4.0 licence together with a waiver of those terms that go further than requiring attribution. Our reasoning was that using a well known label would be preferable and we would be waiving terms that both licensors and licensees in general ignore, so even if the legal intricacies were not understood, things would come out right.

The proposal had not even been discussed by the board when it was met with massive opposition by the community. While the LWG believes the reasoning behind the opposition to be incorrect, we resubmitted the proposal to the board earlier this year, this time simply stating that the tiles should be licenced as an ODbL “Produced Work” with no additional terms applied. This had already been discussed as a possible alternative and seems to be a well accepted solution to the issue. The relevant legal text can be found in 4.3 of the ODbL https://opendatacommons.org/licenses/odbl/1-0/index.html and essentially only requires attribution in a suitable form.

Last month the board accepted the proposal and has put the new licensing in to force per July 1st 2020. While we will be able to change the text on the English version of the openstreetmap.org “Copyright” page immediately, translations will lag a bit and we ask everybody for some patience till everything has been adapted.

Simon Poole for the Licensing Working Group

Do you want to translate this and other blogposts in another language..? Please send an email to communication@osmfoundation.org with subject: Helping with translations in [language]

The OpenStreetMap Foundation is a not-for-profit organisation, formed to support the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data for anyone to use and share. The OpenStreetMap Foundation owns and maintains the infrastructure of the OpenStreetMap project, is financially supported by membership fees and donations, and organises the annual, international State of the Map conference. It has no full-time employees and it is supporting the OpenStreetMap project through the work of our volunteer Working Groups. Please consider becoming a member of the Foundation.

Call for more featured tile layers on openstreetmap.org

Layers on openstreetmap.org

On openstreetmap.org you can switch between different “layers” offering different views of the same mapping data. Do this by clicking the layers icon (fourth button from top on right sidebar, highlighted on the image below).

The layers we currently have are:

“Standard” (OpenStreetMap Carto) is the default layer on osm.org
“Cycle Map” (OpenCycleMap by Thunderforest),
“Transport Map” (Transport by Thunderforest) and
“Humanitarian“

These views are rendered in different ways (being turned from raw data into a map image), although all of these use Mapnik software. Because the layers are running on different server infrastructure with different set-ups, one layer may be more out-of-date compared to another (recently mapped roads not showing up).

Call for more featured tile layers

We are interested in new featured layers that highlight different aspects of OpenStreetMap. In particular it would be nice to have submissions for layers that:

aren’t based on the Mapnik software stack,
are topographic maps,
are walking-focused maps, or
are non-raster maps, along with help integrating them to the website

No one style needs to meet all the above points, but they’re some areas we currently lack.

Criteria for layers

Proposed new tile layers that will be featured on the OSM website should follow the guidelines set out in the new tile layers policy.

Contact

If you interested, please find the contact email in the layers policy page.

Operations Working Group

Do you want to translate this and other blogposts in your language..? Please send us an email to communication@osmfoundation.org with subject: Helping with translations in [your language]

The map you see on OpenStreetMap.org is changing

If you head over to OpenStreetMap.org and click on the layers button on the right of the map, you are provided a selection of map layers to choose from. This is possible due to the nature of OpenStreetMap – by distributing open geographic data we enable others to produce a map from OpenStreetMap data in whatever style they require. The OSM website provides five such styles but there are many hundreds, if not thousands of other styles in use across the web.

We will shortly begin to roll out a new version of the ‘Standard’ (or default) map style replacing the current version. Although the new version is an evolution of the existing version, the changes to road colours and the display of railways will significantly help to improve the readability of the map. During this roll out you may see a patchwork of old and new map style for a few hours – please be patient whilst our servers work hard to update all corners of the map.

How will the style change?

The change of the map style will primarily effect the way roads and railways are displayed. As OpenStreetMap has grown over the last 11 years we have began to collect more and more information about our surrounding environment. The standard map style has adapted to display this information, but over time this has led to the road and rail network becoming harder to identify. For example, trunk roads (currently shown in green) can be very hard to see in heavily forested areas.

In the new map style “road colours [are] tuned to ensure that roads are well visible on all landcovers”, explains Mateusz Konieczny, who has developed the new style (project details) . Mateusz has worked with the OpenStreetMap community at every step of the journey from initial research to draft implementations to gather feedback on the design. He adds that “steady progression of hue and lightness for major road types (motorway, trunk, primary, secondary, tertiary) should make more intuitive which roads are more important”.

There’s been other recent changes to the style – why highlight this one?

You’re right. For the last few years there has been a new map style released every two to three weeks. As with all things OpenStreetMap there is an army of volunteers working to fix and improve the map style. It would be great to write about all of them but that’s simply not possible. So this change, being one of the larger ones and a Google Summer of Code project, gives us a great opportunity to pause and thank all those who have contributed to the map style. Thank you style maintainers!

What next for this map style?

In the short term, it’s likely to be small incremental tweaks so as to continuously improve the map. Every big change attracts new style maintainers who bring their own ideas and experience. There may be a few minor tweaks to the paths and roads based on their ideas.

There is also an updated version of Mapnik, the underlying toolkit used to convert the written style rules in to the final map you see on the OSM website. The latest version will help to fix a lot of bugs related to non-latin scripts/languages. In the longer term there is the option to repopulate the database used by the Standard map style so that it has access to all OpenStreetMap data tags, not just a limited few.

Can I help with development of the ‘Standard’ style?

Yes. The standard map style, which is known as OpenStreetMap-Carto is available on GitHub. Andy Allan, who re-energized the development of this map style in 2012 has given a number of talks at recent OpenStreetMap conferences. Two good starting points are his talk at State of the Map US 2015 in which he gave a progress update for the project, and the detailed workshop he ran at State of the Map EU 2014.

And what about the other map styles?

Well this change only affects the ‘Standard’ (or default) layer as seen on openstreetmap.org.
There are a number of alternative map styles available or you could make your own. Perhaps you want to produce your own personal map to highlight features that are important to you, or maybe you want the map to better match your company brand. This is all possible with OpenStreetMap data. You can even take the current ‘Standard’ as a starting point.