Category Archives: Downtime

Database maintenance

Updates included below the original post.

OpenStreetMap database unavailable on 31 March 2013

There is a current problem with the OpenStreetMap database and so the data API is currently unavailable due to hardware failures.

The current issues were first observed early on 31 March 2013 when the primary database server became non-responsive. The system administrators are aware of the problem and will resolve it as soon as possible. A complicating factor is the observation of a holiday at the hosting facility that prevents physical access to the server until Tuesday morning.

With the database unavailable, editing is not possible. Other database-related activities, such a new-user-signups, are also unavailable.

Other non-database-related services should operate normally. Tiles are being served with current data.

The system administrators will continue to take steps towards the return of normal operation. Given the aforementioned holiday at the hosting facility, normal operation may not return until some time Tuesday, 02 April 2013.

2013-03-31 1810 UTC – Admin on site

One of the OpenStreetMap Foundation hard working system administrators is currently on-site at the hosting location and working on a solution.

Thank you to the facility and everybody involved for permitting access during the holiday.

Corrective actions are in progress. Services are not yet restored. Stay tuned.

2013-03-31 1900 UTC – Return to normal operation

The database has been returned to normal operation. All OpenStreetMap services should be operating as expected.

Thanks again to the admin team and our hosting providers for the prompt resolution to this unscheduled outage.

Scheduled Maintenance – 08 March 2013

Some OpenStreetMap services will be off-line on 08 March 2013 for scheduled maintenance.

The OpenStreetMap Foundation, Operations Working Group advises that on 08 March 2013, scheduled maintenance of OpenStreetMap services will occur from approximately 0900H to 1600H UTC. The following service changes are expected:

  • API read-only from 0900H to 1500H UTC
  • Diffs off-line from 0900H to 1500H UTC
  • Wiki off-line/read-only from 1000H to 1600H UTC

The scheduled maintenance includes upgrades for the database server and improvements for the wiki server. Some more specifics given on the wiki. Actual service times may vary within the service windows described above.

Thank you, in advance, for your patience and understanding during the maintenance window.

Thank you, as always, to the Operations Working Group, for their tireless efforts that keep OpenStreetMap services working smoothly.

API Read – Write returns

The sysadmin team completed the data base migration to the new DB server on schedule during the the morning of 04 April 2012. The API is now back to normal, Read – Write operation. Now the final steps of the license upgrade will proceed as outlined in the March – April service schedule announcement

Other items of possible interest as the license upgrade process proceeds:

osm.org map tile generation will recommence within the next few hours.

Replication diffs during the license upgrade period have started after community requests. These cc-by-sa data replication diffs are found in the redaction-period directory on planet. planet.openstreetmap.org/redaction-period. These diffs will only serve the period up until the switch to the new license. Mappers have requested these diffs for the redaction period. General consumers of OSM data may choose to consume these diffs or not at their discretion.

ODbL diffs will be located in another directory to be announced in future.

During the redaction period it is recommended that editors save their work early and often to reduce the chances of, and the complexity of conflicts with the back ground redaction process.

Service schedule March – April 2012

The long awaited and eagerly anticipated license upgrade is coming soon, the conclusion of a multi-year process. To minimise disruption to OpenStreetMap mappers and users, we’re taking the opportunity to install our new database server (funded by your generous contributions) at the same time – reducing the total amount of downtime needed.

Please be aware of the following service schedule and the list of dates further on in this article. The license upgrade will start with the database server migration. All times and dates are subject to change: our volunteers are working flat out on this, so thanks in advance for your patience and support.

Mappers

There will be a several-day period of limited API availability. The API will be Read Only while the database is moved to the new database server, ramoth. This new database server was funded by your contributions during the December 2011 fund raising campaign. No map editing will be possible while the API is Read Only.

During the remainder of the upgrade, the API should operate normally. Please postpone bulk edits where possible, until after the license upgrade is complete. As always with system improvements, your patience while you find items to refine is appreciated. Consider monitoring the friendly OSM IRC chat channel, #osm on irc.oftc.net, if you have questions.

We would ask mappers who have not yet agreed (or otherwise) to the new terms to log into OSM before the downtime starts on 1st April (0800 UTC) and signal their intention. We are pleased that the vast majority of OSM data will be unaffected by the license change, and thank all the mappers who have thus far consented to their data being distributed under the new license.

Data consumers

The typical planet file scheduled for this week has been postponed. The final old license planet file will be created from 01 April 2012 data. It will be published once the planet file generation completes which may be delayed by a few days.

The old license replication diff service will stop when the database goes to Read Only mode for the server migration. A new license replication diff service will begin at the completion of the license upgrade from a new address. There will be another message with details for using the new diffs when you are ready to start consuming them.

Important Dates

These dates and times are subject to change without notice.

  • 1st April: Enter Read-only mode. 8am UTC
  • 4th April: End of downtime. Enter Read-write mode on new server. Our estimate is that this will be in the morning, but could be subject to change.
  • 5th-6th April: bbox-based live-data tests of rebuild logic.
  • 7th April: Start automated processing of all remaining non-clean objects.
  • 9th April: Progress report and estimation of remaining license upgrade time.
  • To be Determined: On completion of the processing, subject to satisfaction with the outcome we can re-declare the dataset to be ODbL. Immediately afterwards, a first new license planet file will be generated/published, and diff creation will resume.

Thanks

Thanks, as always, to the many people who make OpenStreetMap great. These people include: the countless mappers who have improved the data, the operations volunteers who make so many things “just work”, the programmers who make participating in OSM easier every day, the donors who provide the hardware and hosting we rely upon.

Many of you ‘overlap’ in more than one of these areas. Please be aware that the thanks are cumulative. :-)

[As excerpted from Dermot McNally's announcement. Context added from additional sources.]

Database downtime – 20 March 2012

On Tuesday 20th of March 2012 between 13:45 and 16:15 (GMT / UTC) the
primary database server will unavailable due to emergency maintenance.

The following services will be affected:

  • www.openstreetmap.org web site will not allow user login or edits (Potlatch). [1]
  • API and map database editing (using JOSM, Merkaartor etc.) will be unavailable.
  • planet.openstreetmap.org will be available but no new diffs will be generated during the outage.
  • Forum (no logins)
  • trac (bug-tracker, no logins)
  • help.openstreetmap.org (no logins)

Other services will not be affected – all of the following are
expected to function normally:

  • tile serving (“View The Map” & “Export”)
  • Wiki
  • Nominatim (search)
  • mailing lists
  • subversion and git (source code repositories)
  • donate.openstreetmap.org

Technical: Database Server Smaug: Replacing faulty motherboard.
Supplier Engineer Onsite. We have contingency hardware available.

1: Maps will still be viewable on the openstreetmap.org homepage and
on other people’s websites.

Sincerely
Grant Slater
On behalf of the OpenStreetMap sysadmin team

Q3 2011 Service reductions

Photo by strollers on Flicker, CC-By-SA

Some secondary OpenStreetMap services will be unavailable during the following dates, due to power maintenance work at our server hosting:

  • 2nd and 3rd July 2011. (Saturday & Sunday)
  • 30th and 31st July 2011. (Saturday & Sunday)
  • 3rd and 4th September 2011. (Saturday & Sunday)
  • and 1 weekday (to be confirmed) during August 2011.

On each of these weekends, the servers will be shut down at 7:30 UTC, and will be started again after works are completed, which will be Sunday 16:30 UTC at the latest, but may be earlier. We do not yet have a precise date for the August maintenance.

The following services will be unavailable during these periods:

As a temporary replacement for XAPI/jXAPI, consider using third-party services such as that provided by MapQuest

Additional information will be posted to the following page: Power_Maintenance_Q3_2011

The following services are not affected:

  • www.openstreetmap.org web site
  • API and Editing
  • planet.openstreetmap.org and diffs.
  • tile serving for existing tiles (details below)
  • Wiki
  • Nominatim (search)
  • mailing lists
  • subversion and git (source code repositories)
  • trac (bug-tracker)
  • donate.openstreetmap.org

Maps will still be viewable on the openstreetmap.org homepage and on other people’s websites. We’ll be serving tiles from a back-up tile server. However rendering engines will be deactivated, meaning that new rendering of map updates will not take place during the maintenance period, some requests for tiles will fail where no cached copy is available, and tile response times may be slower than normal.

Downtime announcement: 23 June 2011

Thursday 23rd June 2011 7:30am (GMT/UTC+0) the API and map data editing on www.openstreetmap.org will be unavailable. The maintenance period is expected to last for 12 hours.

The following services will be unavailable during the maintenance period:
API, editing features of www.openstreetmap.org and planet.openstreetmap.org including replication diffs.

The wiki, mailing lists and help.openstreetmap.org will be unaffected.

Technical: Some of the core servers are being re-located to another data-centre.

Additional information will be posted on the wiki as the maintenance window approaches:
http://wiki.openstreetmap.org/wiki/Servers/June_2011_Maintenance