Category Archives: Operations

Hardware and system administration related posts. Anything related to the Operations Working Group

Database maintenance

Updates included below the original post.

OpenStreetMap database unavailable on 31 March 2013

There is a current problem with the OpenStreetMap database and so the data API is currently unavailable due to hardware failures.

The current issues were first observed early on 31 March 2013 when the primary database server became non-responsive. The system administrators are aware of the problem and will resolve it as soon as possible. A complicating factor is the observation of a holiday at the hosting facility that prevents physical access to the server until Tuesday morning.

With the database unavailable, editing is not possible. Other database-related activities, such a new-user-signups, are also unavailable.

Other non-database-related services should operate normally. Tiles are being served with current data.

The system administrators will continue to take steps towards the return of normal operation. Given the aforementioned holiday at the hosting facility, normal operation may not return until some time Tuesday, 02 April 2013.

2013-03-31 1810 UTC – Admin on site

One of the OpenStreetMap Foundation hard working system administrators is currently on-site at the hosting location and working on a solution.

Thank you to the facility and everybody involved for permitting access during the holiday.

Corrective actions are in progress. Services are not yet restored. Stay tuned.

2013-03-31 1900 UTC – Return to normal operation

The database has been returned to normal operation. All OpenStreetMap services should be operating as expected.

Thanks again to the admin team and our hosting providers for the prompt resolution to this unscheduled outage.

Scheduled Maintenance – 08 March 2013

Some OpenStreetMap services will be off-line on 08 March 2013 for scheduled maintenance.

The OpenStreetMap Foundation, Operations Working Group advises that on 08 March 2013, scheduled maintenance of OpenStreetMap services will occur from approximately 0900H to 1600H UTC. The following service changes are expected:

API read-only from 0900H to 1500H UTC
Diffs off-line from 0900H to 1500H UTC
Wiki off-line/read-only from 1000H to 1600H UTC

The scheduled maintenance includes upgrades for the database server and improvements for the wiki server. Some more specifics given on the wiki. Actual service times may vary within the service windows described above.

Thank you, in advance, for your patience and understanding during the maintenance window.

Thank you, as always, to the Operations Working Group, for their tireless efforts that keep OpenStreetMap services working smoothly.

Two More New Tile Servers

Thanks to generous donations and active members of the OpenStreetMap community, OpenStreetMap infrastructure continues to grow.

A new tile server, Trogdor, has been added to the OSM tile cache network. Located in Amsterdam, The Netherlands, Trogdor is currently serving tiles to IP addresses from The Netherlands, Belgium and several other central European and central African countries.

A second new tile server, Ridgeback, has also been added to the OpenStreetMap tile cache network. Located in Oslo, Norway, Ridgeback is currently serving tiles to IP addresses from Finland, Iceland, the Faroe Islands and several others.

The list of countries served by any tile server will change over time due to expansion of the tile server network, loading, maintenance activities and other factors.

Map tiles are delivered to users based on their GeoDNS location. The OpenStreetMap Foundation seeks additional distributed tile servers. If you would like to donate a tile server and hosting, please see the Tile CDN requirements page on the wiki.

We would like to thank Blix Solutions AS for this generous donation to OpenStreetMap infrastructure.

The OpenStreetMap Foundation is a not-for-profit organization, formed in the UK to support the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data and to providing geospatial data for anyone to use and share. The OpenStreetMap Foundation owns and maintains the infrastructure of the OpenStreetMap project. You can support OpenStreetMap by donating to the OpenStreetMap Foundation.

Photo Credit. This photo of the Oslo tile cache server is kindly provided by Blix Solutions AS, licensed CC-By-SA and used by permission.

New aerial imagery – South Africa

A small sample of the South Africa aerial imagery provided by CD:NGI, the mapping agency of the government of South Africa.

OpenStreetMap contributors in South Africa have negotiated with the South African government to arrange the donation of some aerial imagery to be used in creating and editing data for OpenStreetMap. Chief Directorate: National Geo-Spacial Information, or CD:NGI, part of the South African Department of Rural Development and Land Reform, have provided aerial imagery and OSM contributor Grant Slater, has prepared it for use in our editors. There is 3TB of imagery and covers all of South Africa.

Thanks as always to the many people who helped make this happen: the fine folks at CD:NGI, the OSM contributors who carried the data between continents, and the mappers who continue to make OSM data better, more up to date and accurate every day.

You can find more information about this aerial imagery in the thread on the talk-ZA mailing list.

Do you want to donate aerial imagery or other data to OpenStreetMap? Contact the OSM Foundation Communication Working Group for more details. communication@osmfoundation.org

New Tile Server in Pau, France

Thanks to generous donations and active members of the OpenStreetMap community, OpenStreetMap infrastructure continues to grow.

A new tile server, Lurien, has been added to the OSM tile cache network. Located in Pau, Pyrénées-Atlantiques, France, Lurien is currently serving tiles to IP addresses from France, Spain, Portugal, Andorra, Gibraltar, Italy, Monaco, San Marino and Vatican.

Lurien, highlighted.

We would like to thank PauLLA with support of Université de Pau et des Pays de l’Adour (UPPA) for the server and connectivity and Communauté d’Agglomération de Pau Pyrénées (CDAPP) for the data centre hosting. We would also like to thank OpenStreetMap contributor Christophe Merlet for arranging the donation.

New Tile Server in Moscow

For the second time this month, the OpenStreetMap Foundation has a generous donation of OSM infrastructure to announce.

A new tile server, gorynych, has been added to the OSM tile cache network.

We would like to thank Yandex for providing the new tile server in Moscow. We’d also like to thank the local OpenStreetMap community, especially Dmitry, who was instrumental in the Yandex donation.

Yandex is the leading internet company in Russia, operating the most popular search engine and the most visited website. Yandex also has demonstrated support for Open projects by hosting a mirror for Linux distributions and other F/LOSS projects and a jabber server.

New Australian caching server

Posted to the talk-au mailing list some news from Grant Slater and the sysadmin team:

Have you noticed faster tiles this week? Australia now has a map caching server located in Brisbane. The server is used to speed up the standard tile.osm.org “Mapnik” map style.

Browsing the map on http://www.openstreetmap.org/ should now be more responsive. This new server, named ‘bunyip’, first started providing tiles on Tuesday.

We thank Kris Amy for providing the server and hosting. Thank you, Kris!

If anyone experiences any issues or hiccups, please let me know.

Technical:

Server specs are here

OpenStreetMap tile servers use GeoDNS (PowerDNS with Geo backend) to locate the closest tile server. The DNS regions / cache-server can be viewed here.

To check which server you are being directed to use the command:
nslookup tile.openstreetmap.org
or
dig tile.openstreetmap.org

Regards
Grant – Part of OSM sysadmin team

OpenStreetMap data license is ODbL

As of 9am (UK time), today, 12 September 2012, OpenStreetMap is now licensed under the Open Database Licence.

Thank you Open Data Commons for making legal tools available for the Open Data community.

[Update: Friday, 14 September 2012 approximately 18:30 UTC]
The XML-format Planet file took longer to generate than expected. It is now available in the new directory structure at http://planet.osm.org/planet/. You may experience slow downloads due to demand at this time.

Additional PBF-format files will follow as soon as they are completely generated.

Change to ODbL imminent

Hello OpenStreetMap-pers,

The change to ODbL is imminent. No, Really. We mean it.

At long last we are at the end of the license change process. After four years of consultation, debate, revision, improvement, revision, debate, improvement, implementation, coding and mapping, mapping, mapping, it comes down to this final step. And this final step is an easy one, because we have all pitched in to do the hard work in advance. The last step is so easy, it will be a picnic.

On Wednesday, 12 September 2012[1], generation of the next Planet file will begin. At that point, the API will switch over to ODbL and OpenStreetMap will be an ODbL-licensed Open Data project. API transactions and diffs consumed after that point will consist of ODbL-licensed OpenStreetMap data.

About thirty hours later, that newly-generated planet file will be available from planet.openstreetmap.org for you to consume with your renderers, routers, QA systems, convertors and re-imaginers.

You won’t want to mix ODbL diffs with old license planets or diffs. Purge and reload your systems with the ODbL planet. Then consume the ODbL diffs. Planet will have a new directory structure. We’re taking this opportunity to rationalize the layout of planet directories a bit. You should find it easier to understand afterwards. This also means that you won’t accidentally mix data of different licenses.

Mappers

Mappers shouldn’t see a difference and won’t have to change their mapping. Continue to improve OpenStreetMap by mapping from your own survey observations and using OSM-approved external sources. Never copy from other maps.

Data consumers

If you consume OpenStreetMap data and publish it, we have some guidance for you on the wiki. You’ll want to consider your obligations under the new license and then proceed to purge your old data and switch to the new. Many consumers, such as custom renderers, will only need to update their attribution of OpenStreetMap to the new simplified attribution.

Data consumers may time their upgrades to the new planet and diffs at their convenience.

Best regards and happy mapping,
The Communication Working Group

[1] in case of rain, we won’t cancel this picnic, just reschedule it for the subsequent Wednesday, 19 September 2012.

Image credit

Automated redactions complete

Over the past week the license change redaction bot has made automated redactions, sweeping across our entire worldwide dataset. The whole globe was covered yesterday. There has been substantial technical effort involved in developing and running the software to make those changes, and a fair degree of uncertainty about how long it would take, so this is a significant milestone. Congratulations and thanks are in order to all those who helped achieve it, and especially to Matt Amos, Andy Allan, Gnonthgol and MonkZ who carried out most of the coding work.

The data now in the live OpenStreetMap database is largely in a state where it can be declared ODbL licensed, however the license hasn’t changed yet. We will be posting a further update when this is imminent.

More than 99% of the data has been retained, and in most places, the difference is barely noticeable. There are, however, some areas of our map where the redaction was concentrated, in particular Poland and Australia. Though we would of course have preferred to retain this data, we do respect the original contributors’ decision, and we thank them for their past involvement in the project.

Fortunately the OSM community’s response in these areas has been magnificent and we believe we will be back to having a high-quality dataset in these areas in a short space of time. If, as an OSM mapper, you would like to help – by using aerial imagery and other sources, or ideally, from personal knowledge and survey – then please do get involved. Essentially we’re left with some new blank spots on the map, and can respond with the process we know and love. Use your favourite mapping techniques, go explore and fill in the blanks. (Note that you must not copy from CC-BY-SA datasets or map views based on the pre-redaction data; please treat this like you would any other incompatibly copyrighted map data.) We can begin this process in earnest now while final preparations for license changeover are made.