A Year of Infrastructure Progress: Site Reliability Engineer 2023/2024 Update

As the OpenStreetMap Foundation’s Senior Site Reliability Engineer (SRE), my focus in the OpenStreetMap Operations Team over the last year has been on driving efficiency, improving resiliency, and scaling our infrastructure to support the continued growth of the OpenStreetMap project. From cloud migration to server upgrades, we’ve made several improvements since last year to better position OpenStreetMap’s infrastructure to meet these resiliency and growth challenges.

Improving User Facing Services

Upgraded Rendering Services

The tile rendering infrastructure saw notable upgrades, including hardware and software optimisations, faster tile cache expiry to address vandalism, and automation to block non-attributing users. We now re-render low-zoom tiles daily, improving both performance and allowing a faster mapper feedback loop. The tile service is widely used and keeping up with demand is an ongoing challenge.

New Aerial Imagery Service

Launched a new aerial imagery service that supports GeoTIFF COGs. The service now hosts aerial.openstreetmap.org.za which is backed by 16TB of high-resolution imagery. The new service makes it easier to host additional imagery in the future.

Transition to Gmail Alternative & Spam Mitigation

After facing significant spam issues with the OSMF’s Google Workspace, I migrated OSMF email services to mailbox.org. This has reduced the spam volume and improved administrative efficiency. We’re also in the process of transitioning historical OSMF Google Docs data to a self hosted service.

Dealing with DDoS Attacks and Vandalism

This year, we faced several Distributed Denial of Service (DDoS) attacks, including a major DDoS for ransom incident, which was reported to law enforcement. These attacks tested our infrastructure, but we’ve implemented measures to strengthen our resilience and better protect against future threats.

We also dealt with large-scale vandalism that affected OpenStreetMap services. Thanks to the swift response and adjustments made by the Operations team, we’ve reinforced our infrastructure to better handle abuse and ensure continuous service.

Planet Data Hosting on AWS S3

With the OpenStreetMap Operations Team I’ve moved our planet data hosting to AWS S3 with mirrors in both the EU and US, allowing us to fully reinstate the back catalog of historical data. Through AWS’s OpenData sponsorship, replication diffs and planet data are now more accessible.

Making Systems Easier to Manage

Full AWS Infrastructure Management Using OpenTofu

With the OpenStreetMap Operations Team I’ve successfully migrated all manually managed AWS resources to Infrastructure-as-Code (IAC) using OpenTofu (formerly Terraform). This transition allowed us to improve cost efficiency, enhance security by adopting a least privilege IAM model, and gain better visibility into expenditures through detailed billing tags. Additionally, we’ve integrated S3 Storage Analytics to further optimise our costs, set up additional backups, and implemented enhanced lifecycle rules.

Improved Service Outage Alerting

We implemented SMS-based alerting for critical service outages, alongside a sponsored PagerDuty account. These improvements ensure quicker response times and better coordination during outages, with full integration with Prometheus/Alertmanager and Statuscake in the works.

Technical Debt reduction

This year, we made progress in reducing technical debt by moving several legacy services to more maintainable solutions. For instance, we containerised old services, including legacy State of the Map websites that were previously running poorly maintained WordPress installations. This transition has improved the scalability, security, and long-term maintainability of these services.

Additionally, we replaced our custom source installation of OTRS with a Znuny package installation from Debian. This shift simplifies upgrades and reduces the maintenance burden, ensuring the system remains up to date and secure without custom modifications.

Ensuring Infrastructure Resilience Despite Hardware Failures

Over the past year, we’ve maintained a resilient infrastructure even in the face of hardware failures. We replaced numerous disks and RAM, ensuring minimal disruption to services. Our bespoke monitoring system allows us to detect early signs of hardware failure, enabling us to act quickly and replace faulty components before they cause significant issues. This proactive approach has been key to maintaining system uptime and reliability.

Upgrading Infrastructure

Cross-Site Replication of Backups

To ensure robust disaster recovery, I’ve established cross-account, cross-region replication for AWS S3 backups, enabling point-in-time recovery. This safeguards critical data and services, even in the face of major failures, providing long-term peace of mind.

High Availability Infrastructure

Key hardware upgrades in our Amsterdam, Dublin, and OSUOSL sites improved performance, storage capacity, and network reliability. New switches were installed in 2022, and we’ve now finished setting up a high availability (HA) configurations to ensure improved service, which we have continued improve the setup by moving to dual diverse uplinks to our ISP for better resilience.

Debian Migration

We are migrating from Ubuntu to Debian 12 (Bookworm) as our standard distribution. All new servers now run on Debian. Our chef configuration management has been updated with test code to ensure ongoing compatibility. This transition marks a shift towards greater long-term stability and security. Mastodon post celebrating the transition.

Looking Ahead

The year ahead brings exciting new opportunities as we build on our progress. Key priorities for 2024 / 2025 include:

Engaging

Community Engagement & Outward Communication: Enhancing collaboration with the Communication Working Group (CWG) and improving our public-facing communication around service status and outages.

Improving Documentation and Onboarding: We’ll enhance onboarding documentation and conduct dedicated sessions to help new contributors get involved in operations more easily. This includes improving the reliability and coverage of our testing processes, ensuring smoother contributions and reducing the learning curve for new team members.

Planning and Optimizing

Capacity Planning for Infrastructure Growth: As OpenStreetMap and the demand on our services grow, we will ensure we can scale to meet demand. By anticipating future needs and balancing performance with cost-effective growth, we aim to maintain the service quality and availability our community expects.

Ongoing Cost Optimisation: We’ll continue to find ways to reduce costs by leveraging sponsorships like the AWS OpenData programme, ensuring sustainable operations.

Continuing to Reduce Technical Debt: We will continue simplifying our infrastructure by reducing the maintenance burden of legacy systems, such as increasing the use of containers. This will help streamline management tasks and allow us to focus on other improvements, making the infrastructure more efficient and scalable over time.

Continue Infrastructure Improvements

Implementation of High Availability Load Balancers: Rolling out the HA (VRRP + LVS + DSR) configuration for load balancers to improve system reliability and reduce potential downtime.

Finalising Prometheus Integration with PagerDuty: Completing the integration of Prometheus for monitoring and PagerDuty for streamlined alerting and incident response.

Complete the Transition to Full Debian Environment: Migrating all remaining services from Ubuntu to Debian for increased stability and security.

Enhancing Disaster Recovery & Backup Strategies: Further refining our recovery documentation and introducing additional backup measures across critical services are protected and recoverable in the event of failure.


The Days are Long but the Years are Short

OpenStreetMap at 20

by Steve Coast

This essay is re-posted with permission from Steve Coast’s Substack.

Two decades ago, I knew that a wiki map of the world would work. It seemed obvious in light of the success of Wikipedia and Linux. But I didn’t know that OpenStreetMap would work until much later.

I was showing someone new to OSM how to add data to the map. I would ask for a place they knew well, zoom in to that area and then find something to fix. The key was to get a quick win by showing the map before and after they had made it better. Get a little shot of dopamine for making the world a slightly better place.

This person asked to look at Cuba.

This presented a challenge, and I had to manage expectations. OpenStreetMap at the time had okay maps of major Western countries but my expectation, as I explained to them, was that Cuba would be a blank empty slate.

Cuba was doubly tricky not just because of economic factors that limit peoples free time and ability to contribute to open projects, but also the internet was (effectively if not actually) banned and computers illegal.

Zooming in to Cuba that day was the last time I was surprised by OSM, and when I stopped worrying about it working as a project: Cuba had roads, parks, hospitals and everything else imaginable already mapped.


OpenStreetMap has grown exponentially or quadratically over the last twenty years depending on the metric you’re interested in. My involvement has waxed and waned like soul mates oscillating between rapture and, inevitably, wanting the best for each other in our post-relationship new lives.

The story isn’t so much about the data and technology, and it never was. It’s the people.

Like John Boyd said, it’s the people then the ideas and then the technology. Not the other way around.

People: The people that wanted to map just weren’t in the existing camps by definition. They largely didn’t work in geography at all. They just wanted a way to make a map better. Governments, universities and companies had lists of reasons why public mapping wasn’t possible, but no actual solution.

Ideas: Allowing volunteers to edit a map in 2004 was simply anathema and bordering on unthinkable. Map data was supposed to be controlled, authorized and carefully managed by a priesthood of managers.

Technology: For those not in the industry, you might not know that OSM essentially did the opposite of what academic and the leading technology platforms at the time advocated. It needed a data model designed for volunteers not paid editors. So, we did tags not ontologies, and nodes and ways, not web feature service.

I do not know what I may appear to the world, but to myself I seem to have been only like a boy playing on the seashore, and diverting myself in now and then finding a smoother pebble or a prettier shell than ordinary, whilst the great ocean of truth lay all undiscovered before me. – Isaac Newton

OpenStreetMap managed to map the world and give the data away for free for almost no money at all. It managed to sidestep almost all the problems that Wikipedia has by virtue of only representing facts not opinions.

The project itself is remarkable. And it’s wonderful that so many are in love with it.


For me though, I’m far more fascinated with what are the other pebbles on the beach. What else can we make for almost no money that will radically change the world for the better?

If OpenStreetMap is a medium, what is the message?

For me it’s that we can go from nothing to something, or zero to one. Many of us love critiquing something that exists or maybe even improving it. But, my boyish naiveté was assuming that there were lots of other people out there also trying to build new things. Tautologically this simply can’t be true, for if everyone was making new things for any period of time we’d be much further along the various technological curves.

What stops us from doing new things? There seems to be a million reasons and two opposing forces keeping us in inaction: fear and vanity.

Fear of actually building something and showing it to people will push you from one side, and vainly falling in love with the idea itself will pull you the other way. These forces will perfectly balance like the tides. You’ll be stuck in the gravity well of some dead Lagrange point neither executing on the idea nor killing it.

Not everyone has ideas, but if you do, I encourage you to go do the thing.

When you do the thing, most likely you’ll have to kill it. New things tend to not work, or you have to change them drastically. OpenStreetMap’s first four or so major versions were all radically different from each other and relied on feedback from the world to make them into something that would work.

Killing the new things means you have to try many of them. This, too, is reflected in OSM, where I actually started about ten ideas at the time. OSM took off. One was taken over. The rest were strangled to death by reality meeting vanity.


So, celebrate all that we have achieved. It’s been amazing.

And then please turn the wheel and look to windward and consider how to kill it, by making something new or better.



–> Read this post at “Steve Coast’s Musings”

–> Sign OpenStreetMap’s 20th Birthday Card

–> Support or Join OpenStreetMap


The OpenStreetMap Foundation is a not-for-profit organisation, formed to support the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data for anyone to use and share. The OpenStreetMap Foundation owns and maintains the infrastructure of the OpenStreetMap project, is financially supported by membership fees and donations, and organises the annual, international State of the Map conference. Our volunteer Working Groups and small core staff work to support the OpenStreetMap project. Join the OpenStreetMap Foundation for just £15 a year or for free if you are an active OpenStreetMap contributor.

OSMF board election 2024 – Submission of self-nominations is open until 13 August 2024, 23:59:59 UTC

OpenStreetMap Foundation logo
Official OSM logo by Ken Vermette, CC-BY-SA 3.0 & trademarks apply.

Submission of self-nominations for the 2024 OSM Foundation board election is open. At least four of the seven board seats are available. Board members are volunteers and typically serve for two years until their seat is up for re-election.

You can put yourself forward as a board candidate by adding your information to the OSM wiki.

Submission of questions to candidates

Community members can submit questions to the candidates here, until 24 August 2024 at 23:59:59 UTC. The official set of questions, which candidates will be asked to answer, will be based on the questions from the community and previous years. The facilitator for the questioning process this year, selected by the board, is Brian Sperlongano.

Resources regarding the 2024 board election and the 18th Annual General Meeting

For more information regarding the 2024 board election and the 18th Annual General Meeting, please see the following links:

Key dates

  • List on the OSM wiki.
  • Monthly view – Please check the description of each event for exact dates and times — the OSM wiki is the authoritative source.

How you can help

Besides asking questions and voting, a few of the current and past board members have mentioned that the thought of being a candidate did not cross their mind until it was suggested to them. So, you might want to think if you’d like to run for the board or to propose being a candidate to others.


Do you want to translate this and other blog posts in another language..? Send an email to communication@osmfoundation.org with subject: Helping with translations in [your language]

The OpenStreetMap Foundation is a not-for-profit organisation, formed to support the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data for anyone to use and share. The OpenStreetMap Foundation owns and maintains the infrastructure of the OpenStreetMap project, is financially supported by membership fees and donations, and organises the annual, international State of the Map conference. Our volunteer Working Groups and small core staff work to support the OpenStreetMap project. Join the OpenStreetMap Foundation for just £15 a year or for free if you are an active OpenStreetMap contributor

Get notified about new blog posts: Subscribe to the English RSS feed (feeds also available for different languages)

SotM 2024: Call for Lightning talks

If you missed all SotM 2024 calls for participation, here is a chance to submit a presentation: Lightning talks are quick 5-minute talks about any topic related to OpenStreetMap. Since SotM 2024 is hybrid conference, there are two options for holding a lightning talk:

In-person lightning talk in Nairobi: These are traditional lightning talks held during one of the lightning talk sessions on-site at the conference. The registration for in-person lightning talks opens in-person when the conference starts in Nairobi. You can register for a slot directly at the conference venue.

Online pre-recorded lightning talks: We offer the opportunity to submit short videos (maximum 5 minutes). So if you don’t make it to Nairobi, you can still participate with a lightning talk and present your topic to a wider audience virtually.

Submission of online pre-recorded lightning talks is now open. We are looking forward to your videos on OpenStreetMap-related topics. The deadline to register and upload pre-recorded lightning talks is 20 August 2024 23:59:59 UTC https://www.timeanddate.com/worldclock/fixedtime.html?msg=SotM+2024+Prerecorded+Lightning+Talks+Deadline&iso=20240820T235959

For more details and to register a pre-recorded lightning talk, please visit the OpenStreetMap Wiki at https://wiki.openstreetmap.org/wiki/State_of_the_Map_2024/registration_lightning_talks

SotM 2024: Call for posters

Did you miss the call for general and academic presentations? Don’t worry!  you can still present your project or cartographic visualization at State of the Map 2024! The call for posters for SotM 2024 is now open!

Your poster could show how well your community is mapped; it could be a new beautiful style or a map. It could be a community project or statistics, or a poster explaining and inviting people to  OpenStreetMap. The important thing is that it is about OSM. We also welcome academic posters on research around OpenStreetMap data.

For inspiration, you can check out the SotM 2022 posters – 2022.stateofthemap.org/posters/

Submission Requirements

  • The poster must be in A0 format (841×1189 mm).
  • The poster must be related to OpenStreetMap.
  • The poster must be open, innovative and transparent (no copying).
  • The poster must be an original work (individual, collective or institutional).
  • The poster must be submitted under an open license (CC-BY-SA 3.0 or higher recommended or CC0).
  • Maximum of two entries per person, team or institution.

How to participate

  • Upload your poster via https://files.osmfoundation.org/s/5gMewMw6FTFHF7A
  • Maximum file size: 30-40 MB
  • Format: PDF
  • Please email sotm [at] openstreetmap.org with a description of your poster. For example, the background of the project or whatever you consider important to mention in the context of the poster – all that you would tell people if you show them your poster. We will publish this text together with the poster on the SotM website.
  • Please also mention the filename of the uploaded poster in your email, so that we can know which of the uploaded posters is yours.

Timeline

  • Deadline: 25 August 2024

The SotM team hopes to shortlist up to 20 posters that will be published on our website and some other SotM channels under CC BY SA 3.0 (or later)

The State of the Map Working Group

Do you want to translate this and other blogposts in your language…? Please email communication@osmfoundation.org with subject: Helping with translations in [your language]

The State of the Map conference is the annual, international conference of OpenStreetMap, organised by the OpenStreetMap Foundation. The OpenStreetMap Foundation is a not-for-profit organisation, formed to support the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data for anyone to use and share. The OpenStreetMap Foundation owns and maintains the infrastructure of the OpenStreetMap project, is financially supported by membership fees and donations, and organises the annual, international State of the Map conference. Our volunteer Working Groups and small core staff work to support the OpenStreetMap project. Join the OpenStreetMap Foundation for just £15 a year or for free if you are an active OpenStreetMap contributor.

OpenStreetMap was founded in 2004 and is an international project to create a free map of the world. To do so, we, thousands of volunteers, collect data about roads, railways, rivers, forests, buildings and a lot more worldwide. Our map data can be downloaded for free by everyone and used for any purpose – including commercial usage. It is possible to produce your own maps which highlight certain features, to calculate routes etc. OpenStreetMap is increasingly used when one needs maps which can be very quickly, or easily, updated.

Participating in SotM 2024; Programme and Tickets

The State of the Map Working Group is happy to announce that tickets and program are now accessible through the SotM 2024 website at https://2024.stateofthemap.org/programme/

Tickets

OSM contributors and enthusiasts can now purchase tickets to attend the SotM conference in person or online, via https://2024.stateofthemap.org/tickets. There are two different ways you can attend this year:

Nairobi Ticket

This is the ticket to attend the conference in person in Nairobi. This ticket includes meals at the conference’s breaks and social events. If you are contributing to OpenStreetMap as a hobbyist, you may choose the discounted Community Ticket. Otherwise please get a Regular Ticket.

Venueless Ticket

This is the ticket to attend the conference remotely. This will give you access to the conference talks, and a chat to interact with other participants and allows you to send questions to speakers during Q&A sessions.

Early Bird Ticket

We are offering limited Early bird tickets at a discounted price until 31 July 2024 at 23:59 UTC. Regular ticket price will apply after.

Program (General Track)

We thank everyone who submitted their proposal to speak and organize a session at SotM 2024! We received a lot of amazing proposals and we are delighted to share the first version of our programme for the General Track, now accessible via https://2024.stateofthemap.org/programme/. We will provide continuous updates on the programme (including the Academic Track Programme – https://2024.stateofthemap.org/calls/academic/).
Stay tuned for more updates about State of the Map 2024!

For any questions, do not hesitate to contact us at sotm [at] openstreetmap.org.

See you in Nairobi and online!

The State of the Map Working Group

Do you want to translate this and other blogposts in your language…? Please email communication@osmfoundation.org with subject: Helping with translations in [your language]

The State of the Map conference is the annual, international conference of OpenStreetMap, organised by the OpenStreetMap Foundation. The OpenStreetMap Foundation is a not-for-profit organisation, formed to support the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data for anyone to use and share. The OpenStreetMap Foundation owns and maintains the infrastructure of the OpenStreetMap project, is financially supported by membership fees and donations, and organises the annual, international State of the Map conference. Our volunteer Working Groups and small core staff work to support the OpenStreetMap project. Join the OpenStreetMap Foundation for just £15 a year or for free if you are an active OpenStreetMap contributor.

OpenStreetMap was founded in 2004 and is an international project to create a free map  of the world. To do so, we, thousands of volunteers, collect data about roads, railways, rivers, forests, buildings and a lot more worldwide. Our map data can be downloaded for free by everyone and used for any purpose – including commercial usage. It is possible to produce your own  maps which highlight certain features, to calculate routes etc. OpenStreetMap is increasingly used when one needs maps which can be very quickly, or easily, updated.

OSM Foundation board election 2024 – How to become a board candidate

OpenStreetMap Foundation logo
Official OSM logo by Ken Vermette, CC-BY-SA 3.0 , trademarks apply.


OpenStreetMap Foundation members will vote to elect a new board in October. There will be four seats available: of Arnalie Vicario, Craig Allan, Mateusz Konieczny and Sarah Hoffmann, whose board terms are ending. The terms of Dani Waltersdorfer, Guillaume Rischard and Roland Olbricht will continue. There are seven seats on the foundation board and board members are volunteers.

Self-nominations of board candidates will open on 28 July 2024 and you will be able to nominate yourself on this OpenStreetMap wiki page: https://wiki.osm.org/wiki/Foundation/AGM2024/Election_to_Board#List_of_board_candidates

You can create an account on the OSM wiki here and you will be able to add your name to the table that will be added on that date by editing the page here (please wait until 28 July 2024 to do so). Self-nominations will close on the 13th of August at 23:59 UTC.

Election timeline

The timeline of the election is here: https://wiki.osm.org/wiki/Foundation/AGM2024/Election_to_Board#Key_dates

What the board is/is not, rules and responsibilities and why run

Please read the links on the OSM wiki.

A lot of the foundation’s work is done by the volunteers of our Working Groups, and if you want to help the foundation, you can also look at joining those.

Who can become a board candidate

Any natural person may be elected to become a board member, provided that:

  • they have been a normal OSM Foundation member [1] or an associate member [2] during the full 180 days before the General Meeting, which will take place on 19 October 2024, and
  • have been a normal member for the full 28 days prior to the General Meeting, and
  • are willing to act as a board member, and
  • are permitted by law to do so.

[1] Normal members provide their full residential address and can vote on all issues. Their residential address may be disclosed to other members.
[2] Associate members provide just their country of residence - which may also be disclosed to other members - and can vote - but not on all issues. Additionally, they cannot be board candidates.

If you want to find out the type of your OpenStreetMap Foundation membership (normal or associate), please check the most recent approval/renewal membership email or email the volunteers of the Membership Working Group at membership@osmfoundation.org from the email account associated with your OSM Foundation membership. Please see the OSM wiki regarding changing your membership type.

Resources about the 2024 board election and Annual General Meeting

The main two pages that have the information about the 2024 board election and Annual General Meeting are:

Resolutions proposed by OSM Foundation members

OSM Foundation members can submit resolutions and ask the membership to vote on them. The resolutions need to be supported by at least 5% of members eligible to vote, in order to be added to the ballots. Please read: Companies Act 2006: Members’ power to require circulation of written resolution. The deadline for providing the supported resolutions will be the 24th of August.

How you can help

A few of the current and past board members have mentioned that the thought of being a candidate did not cross their mind until it was suggested to them. So, you might want to think if you’d like to run for the board or to suggest being a candidate to others.


Do you want to translate this and other blog posts in another language..? Send an email to communication@osmfoundation.org with subject: Helping with translations in [your language]

The OpenStreetMap Foundation is a not-for-profit organisation, formed to support the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data for anyone to use and share. The OpenStreetMap Foundation owns and maintains the infrastructure of the OpenStreetMap project, is financially supported by membership fees and donations, and organises the annual, international State of the Map conference. Our volunteer Working Groups and small core staff work to support the OpenStreetMap project. Join the OpenStreetMap Foundation for just £15 a year or for free if you are an active OpenStreetMap contributor

Get notified about new blog posts: Subscribe to the RSS feed

Invitation to State of the Map Europe 2024

The OpenStreetMap Poland Association is pleased to invite all members of the OSM community to the State of the Map Europe 2024 conference, to be held in Łódź, Poland, from 18–21 July 2024. In addition to many interesting presentations and talks, it will be an opportunity to meet, discuss and exchange experiences.

Conference tickets

Conference tickets can be purchased at: https://tobilet.pl/state-of-the-map-europe-2024.html

Traveling and accommodation

If you would like some advice on how to get to the conference, and suggestions for accommodation, we have compiled some information at: https://stateofthemap.eu/accommodation.html

If you have questions about traveling to Poland or you would like to ask something else related to the conference, you can write at: https://community.openstreetmap.org/t/invitation-to-the-state-of-the-map-europe-2024/114661

Volunteering

As you know, ’State of the Map Europe’ is a non-profit event, and for organizational purposes we need people who can help us on a volunteer basis.

During the event, we will need volunteers who will be ready to support us in Łódź — mainly on the premises of the Łódź University of Technology (unless other requirements are mentioned). To check what we need and what we offer in return, please see this page: https://stateofthemap.eu/volunteering.html For volunteers from outside Łódź, a limited accommodation will be available.

We are looking forward to seeing you at the conference!

OSM Poland board



OpenStreetMap Poland is a Local Chapter of the OpenStreetMap Foundation, which supports the OpenStreetMap project.

OpenStreetMap was founded in 2004 and is an international project to create a free map  of the world. To do so, we, thousands of volunteers, collect data about  roads, railways, rivers, forests, buildings and a lot more worldwide.  Our map data can be downloaded for free by everyone and used for any  purpose – including commercial usage. It is possible to produce your own  maps which highlight certain features, to calculate routes etc.  OpenStreetMap is increasingly used when one needs maps which can be very  quickly, or easily, updated.

OpenStreetMap at the UNMaps Conference 2024

OpenStreetMap foundation board member Sarah Hoffmann recently joined the 3rd annual UNMaps conference in Valencia, Spain to present about the OpenStreetMap project and participate in a panel discussion about crowd-sourced geodata.

The UNMaps conference brings together the GIS experts from the different entities of the UN, which was about a hundred participants in 2024. There were representatives from the UN secretariat, the different UN entities, like UNICEF and the International Criminal Court (ICC), and people from the UN peacekeeping missions. They also regularly invite external partners and contributors to present their work and join the discussions. Given the vital role that OSM already plays in their work, they asked the OSMF board to join the conference and represent the OpenStreetMap community. Also invited from the wider OSM community were the researchers from HeiGIT and the humanitarian mappers from HOT.

“It wasn’t necessary to explain to anyone who we are,” noted Sarah, “OpenStreetMap is well known and used on a daily basis within the UN. The maps produced by the UNMaps division for use within the UN heavily use OSM data, mainly replacing boundaries and names to follow the official policies of the UN.”

“Crowdsourcing Geospatial Information” Panelists from left to right: (host) Michael Montani (UNMappers; (panelists)
Benjamin Herfort (HeiGIT), Sarah Hoffmann (OSMF), Sam Colchester (HOT).

In many areas where the different UN entities operate, OSM is the best (and sometimes only) available source for geographic data. Thus, base maps using OpenStreetMap data were featured in almost every presentation. And it doesn’t stop there. UN users also contribute back to improve and complete the map in the areas where they use it, and the UN Mappers community supports the UN with organised mapping activities.

Sarah answered questions about OSM and OSM operations, including a lively discussion about iD editor as a response to the question: how do we deal with errors in the map data? In addition, Sarah joined a panel discussion on crowd-sourced data focussing on the questions of data quality and the importance of local knowledge in map data.

The discussions continued in the hallways, as well. Sarah noted that there were informal conversations about what kind of data is suitable for OSM and how to best connect with the community, as well as about what it’s like to be part of a world-wide community in peaceful cooperation to achieve a common goal.

“The conference was a great opportunity for getting to know each other,” said Sarah, “seeing OpenStreetMap data used so much in the daily operations of the UN is a great reminder how much our project has grown and how essential our data has become for so many different activities.”


The OpenStreetMap Foundation is a not-for-profit organisation, formed to support the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data for anyone to use and share. The OpenStreetMap Foundation owns and maintains the infrastructure of the OpenStreetMap project, is financially supported by membership fees and donations, and organises the annual, international State of the Map conference. Our volunteer Working Groups and small core staff work to support the OpenStreetMap project. Join the OpenStreetMap Foundation for just £15 a year or for free if you are an active OpenStreetMap contributor

A progress update on vector tiles from the Engineering Working Group

The OpenStreetMap Foundation’s Engineering Working Group has an update on the effort to create vector tiles for openstreetmap.org. Read on for why this work is important, what’s been done so far and how they are incorporating community feedback, and the technical details for those who want to know more.

The background

Currently, the openstreetmap.org website serves raster tiles, which are image tiles made up of pixels — think a downloaded image of part of a map. But the effort has begun to create vector tiles for the site, which will help improve how the map looks and how it works. You can read more about the background of the project here

Vector tiles serve up maps as vectors: points, lines and polygons. They store geographic data (like what makes up OpenStreetMap) in a format that allows for dynamic styling and interactivity. For users, vector tiles will mean a new, modern-looking map style with seamless zoom on openstreetmap.org, the map can be updated more quickly when data changes, and it should perform better for users.

Looking further ahead, the most exciting part is what this vector tile project will make easy for volunteers and tile users: 3D maps, more efficient data mixing and matching and integration of other datasets, thematic styles, multilingual maps, different views for administrative boundaries, interactive points of interest, more accessible maps for vision-impaired users, and surely many other ideas that no one has come up with yet. You may recall that many of these have been long-term interests for people in the OSM community.

The plan

The goal of vector tiles project is to provide a vector tiles setup that can work for openstreetmap.org — that is, a worldwide, complex basemap site in heavy demand from users and services around the world, where the data underlying the map changes all the time.

Or to put it technically, to create a setup for a worldwide complex basemap under high load which requires minutely updates.

Paul Norman is leading the vector tiles project.

He is working on adding to his Tilekiln project which generates vector tiles from a PostgreSQL database (like OpenStreetMap’s), making use of the Shortbread schema, which is a data format for how to name layers & properties within a vector tile, and improving Themepark, which allows one to add OSM data to a Postgres database.

The work is split up into three steps: 

1. First round of Tilekiln improvements and Shortbread Themepark improvements

2. Parallelism improvements

3. Shortbread publicly available in production

The first two steps are nearly done. Tilekiln now generates tiles in parallel, making it practical to generate tiles for the entire world. The next step is to start the deployment into OSMF hardware to prepare for production. 

Technical details on step 1

For those interested in the technical details of what’s being worked on, there are five main components of the first step above.

        1.        Automated packaging of Tilekiln

        2.        Tilekiln metrics being published with a Prometheus exporter

        3.        Themepark Shortbread reviewed

        4.        A demo server running with minutely updates of Shortbread tiles, rendering tiles on-demand

        5.        Demo shown to community

Items 1 and 2 are complete without need of further discussion. For item 3, Paul found that the osm2pgsql Themepark Shortbread implementation needed more work than anticipated as it was missing a layer and had some issues. 

Item 4 and 5 are complete. Paul’s demo server is running with minutely updates and the hardware requirements are more modest than expected. 

The community has also been providing useful feedback, such as on Paul’s OSM Community Forum post.

The community offered a lot of suggestions, some of which have already been incorporated. The remaining, in-scope issues from the community are: Curved lines rendering as jagged and vector tiles being too large.

The jagged lines issue is due to how smooth curves are represented in vector tiles. It has mostly been addressed but similar issues are expected to crop up in the future. A target scale equivalent to the minimum scale of the standard tile layer has been set. Zooming in to an even lower scale is possible, but artifacts may start to appear.

Vector tile size will continue to be an issue that needs continual work, but the current tiles are particularly large. Since this part of the testing some changes have been made that cut the size in half. Tile size optimization will be an issue that needs ongoing work, as tile size is the biggest factor in user experience.

The tiles being produced are usable, but more work remains to be done. Now that the parallelism work is complete it’s possible to generate large sets of tiles in order to test, so Paul will be returning to working on the tile definitions to improve tile size and fix some remaining issues, but the current  tiles are usable.

Background on the tools being used

Here is some information on the various tools used for this project.

Tilekiln is software written by Paul Norman for generating vector tiles from a PostgreSQL database. Alternatives are martin (or maybe t_rex). Tilekiln is in new development, although it uses a lot of standard PostgreSQL features to generate the vector tile data. Most OSM based maps (incl. osm-carto on osm.org) are generated from SQL database queries from a PostgreSQL database. Tilekiln generates vector tiles from similar queries. Tilekiln is new.

Themepark is part of the osm2pgsql suite of tools, to allow one to add OSM data to postgres, and share those processing steps between other projects. Many PostgreSQL based OSM map styles (like osm-carto) use osm2pgsql 

osm2pgsql has been around for 15+ years in OSM, and is used in many many places. Although Paul has contributed code to it, he is not the main developer. osm2pgsql has gotten more advanced, and better, in the last few years. Part of the power is pre-processing the data, and Themepark is an attempt to make these pre-processing steps easier.

Shortbread is a “vector tile schema” created by Geofabrik. It’s a data format for how to name layers & properties within a vector tile.

This blog posts contains contributions from Adam Hoyle, Mikel Maron, Amanda McCann, Paul Norman, and Andrew Wiseman

The OpenStreetMap Foundation is a not-for-profit organisation, formed to support the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data for anyone to use and share. The OpenStreetMap Foundation owns and maintains the infrastructure of the OpenStreetMap project, is financially supported by membership fees and donations, and organises the annual, international State of the Map conference. Our volunteer Working Groups and small core staff work to support the OpenStreetMap project. Join the OpenStreetMap Foundation for just £15 a year or for free if you are an active OpenStreetMap contributor.