Category Archives: Operations

Hardware and system administration related posts. Anything related to the Operations Working Group

Tile Usage Policy

Tile image by vidalia_11 is licensed CC-By-SA

We’ve had to block some uses of the OpenStreetMap Foundation tile servers. This article describes what is happening and why. This article also describes how you can adapt if you are affected.

We’re really proud of the increased popularity of OpenStreetMap. We’ve seen seven-plus years of project growth in every measurable area. As the project has grown we’ve learned and adapted in many ways. The use of our tile server has grown faster than every other aspect of the project. One way that we are adapting now, is by restricting how our map tiles may be used.

Slow sign by DaveCrosby is licensed CC-By-SA

You’ll still be able to use our tiles in creative and interesting ways but the volume of use will be limited. We need to limit access to our tile server to only those users who don’t overburden our resources.

Those users who make large demands on our tile server will be slowed down by our throttling mechanism. This throttling mechanism is rarely triggered by mappers.

Problematic applications may show this image instead of a map.

Those applications which make exceptional demands in aggregate from their users will be blocked. The tile usage policy is on the OSM wiki.

So what can you do about this? How can you get the wonderful OpenStreetMap tiles for your mobile device?

You’ll find more advice about potential tile source alternatives on the wiki.

Read on if you would like to know more about the history of the OpenStreetMap tile server.

OSM tile server background

We started creating rendered images of our map data as a way to encourage our data contributors. Mappers enjoy seeing the results of their surveys on the OpenStreetMap web site, and they can be inspired to map in their neighbourhood the things they see other mappers surveying in other places. Mappers loved the tile server when it first appeared. Potential users often looked at the map in 2006 and said, “Hey, why is there a huge blank spot where my town should be?” Some of those potential users became the long-time contributors that we all know and love.

As more contributors mapped more neighbourhoods, more blank spots started to fill in. More potential users became actual users, and OpenStreetMap tiles started to appear in more places. The tile server became even more popular when rapid updates were enabled. Rather than updating the map every week, parts of the map could update as contributors added data. If you remember the weekly updates you also remember that funny tingle you had the first time you mapped something and it appeared on the map immediately; it seemed like magic, didn’t it?

The OpenStreetMap Foundation has had a tile usage policy for some time. From September 2008 it has been explicitly stated that bulk downloading of tiles was discouraged. OpenStreetMap kept growing. More people came to understand the awesomeness of OSM tiles.

Also in 2008, the Ordnance Survey started serving map tiles to users through their OS Openspace program. In July of 2011, Ordnance Survey served their one billionth tile to a user.

OpenStreetMap serves a Billion tiles every eleven days.

So we know a thing or two about providing awesome maps to users. We do it all with the crowd-sourced data from our contributors around the world, the volunteered time of our sysadmins who keep our servers running, and the generous donations of servers and bandwidth and funds.

We’ve had to become more restrictive of the use of our tile server over time. We’ve limited how many tiles you can consume in a period of time. These restrictions only affected the most-demanding of tile consumers. The everyday mapper never ran into a problem getting tiles to add data to OSM. That allowed the growing number of users to continue to have access to OpenStreetMap tiles without our resources being monopolized by one or two bad actors.

More and more mobile applications started using OpenStreetMap tiles. Many of them included a bulk downloading method so that tiles could be saved on the device at home, rather than downloading tiles at a punitive data rate. That bulk-downloading has always been problematic for OpenStreetMap because a single user will consume hundreds of times the resources of an average user. There are so many applications using OSM tiles, with so many users making unreasonable demands on our resources that it is affecting the quality of service for the average user. And that’s not fair.

So we’ve started blocking the applications that are causing us the most trouble, in addition to blocking users with problematic specific behaviours. We regret it, in a way. After all, we map because we want people to be able to use our data. But our resources have to be used in a way that everybody can share. We can’t have a small number of people consuming all of our resources.

So that’s why some people have started to see the “prohibited” tiles on their maps. Overuse. Or mobile applications that are causing overuse by a group.

More new servers

Dragon sculpture on the Dragon Bridge in Ljubljana. Photo CC-By-SA, dani_7C3


The OpenStreetMap Foundation, and the Operations Working Group, would like to thank Nokia UK Limited for the donation of some of their redundant server hardware. This hardware has found new purpose in the form of “soup“[1] and “fiddlestick“[2], two new web front end servers. A third server “eustace“[3] will be used initially as a trial web statistics server.

The web front-end servers, soup and fiddlestick, replace puff and fuchur who had performed that role since 2008. Web front-end servers in OpenStreetMap provide the data browser and data layer, as well as user diaries and other “social” functions.

Eustace will debut in a new role for OpenStreetMap by collecting web statistics. The OpenStreetMap Foundation wants to know more about how users experience the OSM web site in an effort to improve the way that OSM services are delivered.

[1] Character from The Clangers, a UK children’s TV programme.
[2] Strangewood (1999): Fiddlestick, a small musically emotive dragon.
[3] Turns into a dragon in The Voyage of the Dawn Treader (Chronicles of Narnia) after slipping on a gold bracelet.

Introducing Zark

Zark, during installation.

Zark is the newest OpenStreetMap server. Give Zark a warm welcome. Continuing the in the tradition of naming OSM servers after dragons, the name “Zark” is taken from the Eidolon Chronicles/Shadow World books by Jane Johnson.

The first task for Zark will be to serve as a trial / evaluation server for the OWL – OpenStreetMap Watch List service. OWL’s popularity on the dev server has lead to performance problems and long update delays. After more than a year of development and increasing popularity of OWL’s ability to follow local changes without distracting “Big” changesets, moving OWL to Zark will make this service even more effective for mappers.

Many thanks to bitfolk.com for donating this server.

Q3 2011 Service reductions

Photo by strollers on Flicker, CC-By-SA

Some secondary OpenStreetMap services will be unavailable during the following dates, due to power maintenance work at our server hosting:

  • 2nd and 3rd July 2011. (Saturday & Sunday)
  • 30th and 31st July 2011. (Saturday & Sunday)
  • 3rd and 4th September 2011. (Saturday & Sunday)
  • and 1 weekday (to be confirmed) during August 2011.

On each of these weekends, the servers will be shut down at 7:30 UTC, and will be started again after works are completed, which will be Sunday 16:30 UTC at the latest, but may be earlier. We do not yet have a precise date for the August maintenance.

The following services will be unavailable during these periods:

As a temporary replacement for XAPI/jXAPI, consider using third-party services such as that provided by MapQuest

Additional information will be posted to the following page: Power_Maintenance_Q3_2011

The following services are not affected:

  • www.openstreetmap.org web site
  • API and Editing
  • planet.openstreetmap.org and diffs.
  • tile serving for existing tiles (details below)
  • Wiki
  • Nominatim (search)
  • mailing lists
  • subversion and git (source code repositories)
  • trac (bug-tracker)
  • donate.openstreetmap.org

Maps will still be viewable on the openstreetmap.org homepage and on other people’s websites. We’ll be serving tiles from a back-up tile server. However rendering engines will be deactivated, meaning that new rendering of map updates will not take place during the maintenance period, some requests for tiles will fail where no cached copy is available, and tile response times may be slower than normal.

Downtime announcement: 23 June 2011

Thursday 23rd June 2011 7:30am (GMT/UTC+0) the API and map data editing on www.openstreetmap.org will be unavailable. The maintenance period is expected to last for 12 hours.

The following services will be unavailable during the maintenance period:
API, editing features of www.openstreetmap.org and planet.openstreetmap.org including replication diffs.

The wiki, mailing lists and help.openstreetmap.org will be unaffected.

Technical: Some of the core servers are being re-located to another data-centre.

Additional information will be posted on the wiki as the maintenance window approaches:
http://wiki.openstreetmap.org/wiki/Servers/June_2011_Maintenance

OpenStreetMap tile layer guidelines

mosaic of Modesto California, shown in four different tile styles.

Mosaic of Modesto, California, in four tile layer styles.

The image above was inspired, in part, by Firefishy’s April Fools Day joke, OpenWhateverMap. While such a map is unlikely to be used for an ordinary application, it does serve to show the wide variety of renderings of OpenStreetMap data that are created by the OpenStreetMap community and displayed on the OpenStreetMap web site. It also serves as an illustration of one of the frequently asked questions about the OpenStreetMap.org web site.

How are maps added to the OpenStreetMap web site?

Until December 2005, the OpenStreetMap web site did not have a map shown at all.

The OpenStreetMap main page in November 2005 had no map. (Courtesy Internet Archive)

Starting in December 2005, maps have appeared on the OpenStreetMap web site. Today, the slippy map is the most prominent feature of the OpenStreetMap web page. Currently there are four rendering layers that can selected on the front page as well as one overlay layer.

The OpenStreetMap Foundation now has published guidelines for tile layers to be considered for inclusion on the OpenStreetMap web page. If you know of an interesting rendering of OSM data, and you think it should be considered as a Featured Layer, add your suggestion to the featured tiles score card and start the discussion.

The proposed layers will be evaluated for suitability by the Technical Working Group, periodically, and the tile layer that you recommend could be featured on the OpenStreetMap web site.

Working Group Summary

We’ve fallen behind on the Working Group Summaries as the snow got deeper.  Let’s try again.  These summaries do not replace the formal minutes of the working groups, they are intended to be a quick and easy summary of recent topics before the working groups.

Data Working Group

The Data Working Group met most recently on 20 January 2011. They responded to a report of inappropriately sourced data in Ecuador.  In the end that data was removed and the Ecuadoran community was notified on talk-ec@.

Technical Working Group

TWG met on 26 Jan 2011. Tile server loads are increasing. One major cause is the continued growth of data in well-mapped areas. A node moved in central Berlin, for example, causes re-rendering of a very complex meta-tile.  TWG is considering 5 minute expiry time to reduce the amount of re-rendering during an editing session. A new power supply is required for one server.  Most equipment has been transitioned to the new power management hardware.

The TWG also updated the near-, mid- and long-term hardware maintenance plan.

State of the Map Working Group

SotMWG met on 26 Jan 2011 and have released their Call for Papers for Denver.  The SotMWG is still accepting members.  Would you like to help organize the OpenStreetMap conference this year?   In particular, please offer your help to team@stateofthemap.org if you can help translate the Call for Papers. The SotM logo contest ends soon.

Communication Working Group

The CWG meeting of 24 Jan 2011 was canceled due to scheduling issues.

License Working Group

The LWG meeting of 25 Jan 2011 was canceled due to illness.

Strategic Working Group

The SWG met on 27 Jan 2011 and formalized the initial membership and operating practices of the SWG.  There was a review of the proposed budgeting process.